nix-env uses a ton of memory

[kstenerud]

Issue description

nix-env uses so much memory that it fails on allocation when querying packages; far more than should be for a simple package manager.

Steps to reproduce

On a system with 2gb or less:

[demo@nixos:~]$ nix-env -qa
GC Warning: Out of memory - trying to allocate requested amount (336 bytes)...
GC Warning: Header allocation failed: dropping block
GC Warning: Failed to expand heap by 8388608 bytes
GC Warning: Failed to expand heap by 65536 bytes
GC Warning: Out of Memory! Heap size: 736 MiB. Returning NULL!
error: out of memory

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the
results.

• system: "x86_64-linux"
• host os: Linux 4.14.32, NixOS, 18.03.131807.489a14add9a (Impala)
• multi-user?: yes
• sandbox: no
• version: nix-env (Nix) 2.0
• channels(demo): "nixpkgs-18.09pre134800.e42ae4e586c"
• channels(root): "nixos-18.03.131807.489a14add9a"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs

[edolstra]

Unfortunately Nix is not a simple package manager: it implements a lazy purely functional language, the evaluation of which uses a lot of memory. It has some design issues that make it hard to reduce memory usage (in particular the use of a monolithic pkgs set combined with the .override feature makes garbage collection largely ineffective).

Also, Nixpkgs has seen a lot of growth such that evaluation now takes > 3 times as much memory as two years ago (https://hydra.nixos.org/job/nixpkgs/trunk/metrics/metric/nix-env.qa.allocations). (This may also be a result of the use of certain new abstractions in Nixpkgs, e.g. stdenv.mkDerivation is a lot more complicated than two years ago.)

[katyo]

When upgrading to NixOS 18.03 I had to add 6Gb swap on my laptop with 4Gb RAM to be able install some packages like KiCAD for example :(
Installation takes about 1.5 hours because using swap.
Of couse I also configured zram, because without zram I wont be able to complete installation.
The future of Nix on the middle/low-end systems in the mist...

[vcunat]

@katyo: I suspect your case is because there were no binaries for it, nix tried to build it locally, and /tmp/ in now in RAM by default.

[katyo]

@vcunat I already use distributed build a some time ago because my laptop not so much powerful. Also I relocated /tmp on a disk But it don't helps to reduce memory usage.

[vcunat]

@katyo: what kind of command do you use? If it's nix-env, be sure to use one with -A selectors, to save on performance (and it's saner in other ways, too).

[eliasp]

Ran into the same issue on a device with 8GB memory:

nonroot@box $ nix search mkpasswd
GC Warning: Failed to expand heap by 402653184 bytes
warning: using cached results; pass '-u' to update the cache
Attribute name: nixpkgs.perlPackages.StringMkPasswd
Package name: perl-String-MkPasswd
Version: 0.05
Description:

Attribute name: nixpkgs.mkpasswd
Package name: mkpasswd
Version: 5.3.0
Description: Overfeatured front-end to crypt, from the Debian whois package
nonroot@box $ nix-env -i mkpasswd
GC Warning: Failed to expand heap by 402653184 bytes

I also happened occasionally that my system simply started hanging out of the blue, which I suspect was caused by system.autoUpgrade.enable = true; which might suffer from a related issue - disabled it for now to see whether it still happens.

[vcunat]

@eliasp: almost all people really want to prefer -A for nix-env. You got

Attribute name: nixpkgs.mkpasswd

so you just nix-env -iA nixpkgs.mkpasswd. I haven't seen nixos-rebuild suffering from this, as it uses attributes "inside".

[oxij]

@edolstra

Also, Nixpkgs has seen a lot of growth such that evaluation now takes > 3 times as much memory as two years ago This may also be a result of the use of certain new abstractions in Nixpkgs, e.g. `stdenv.mkDerivation` is a lot more complicated than two years ago.

I read changes of the significant bumps in that graph that were not immediately fixed several revisions later and all but the very last one were either changes to haskell packages or melpa. So my guess would be that we now have ~3x more packages in release.nix that two years ago, not that `stdenv.mkDerivation` became 3x slower. The very latest bump is interesting, though, I'm not sure what causes it, it's either #44423 or #44517, bisecting now. I think a much more useful metric would be "nix-env -qa avg allocations per result produced", not the absolute value.