nix-env uses a ton of memory

[kstenerud]

Issue description

nix-env uses so much memory that it fails on allocation when querying packages; far more than should be for a simple package manager.

Steps to reproduce

On a system with 2gb or less:

[demo@nixos:~]$ nix-env -qa
GC Warning: Out of memory - trying to allocate requested amount (336 bytes)...
GC Warning: Header allocation failed: dropping block
GC Warning: Failed to expand heap by 8388608 bytes
GC Warning: Failed to expand heap by 65536 bytes
GC Warning: Out of Memory! Heap size: 736 MiB. Returning NULL!
error: out of memory

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the
results.

• system: "x86_64-linux"
• host os: Linux 4.14.32, NixOS, 18.03.131807.489a14add9a (Impala)
• multi-user?: yes
• sandbox: no
• version: nix-env (Nix) 2.0
• channels(demo): "nixpkgs-18.09pre134800.e42ae4e586c"
• channels(root): "nixos-18.03.131807.489a14add9a"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs

[edolstra]

Unfortunately Nix is not a simple package manager: it implements a lazy purely functional language, the evaluation of which uses a lot of memory. It has some design issues that make it hard to reduce memory usage (in particular the use of a monolithic pkgs set combined with the .override feature makes garbage collection largely ineffective).

Also, Nixpkgs has seen a lot of growth such that evaluation now takes > 3 times as much memory as two years ago (https://hydra.nixos.org/job/nixpkgs/trunk/metrics/metric/nix-env.qa.allocations). (This may also be a result of the use of certain new abstractions in Nixpkgs, e.g. stdenv.mkDerivation is a lot more complicated than two years ago.)

[katyo]

When upgrading to NixOS 18.03 I had to add 6Gb swap on my laptop with 4Gb RAM to be able install some packages like KiCAD for example :(
Installation takes about 1.5 hours because using swap.
Of couse I also configured zram, because without zram I wont be able to complete installation.
The future of Nix on the middle/low-end systems in the mist...

[vcunat]

@katyo: I suspect your case is because there were no binaries for it, nix tried to build it locally, and /tmp/ in now in RAM by default.

[katyo]

@vcunat I already use distributed build a some time ago because my laptop not so much powerful. Also I relocated /tmp on a disk But it don't helps to reduce memory usage.

[vcunat]

@katyo: what kind of command do you use? If it's nix-env, be sure to use one with -A selectors, to save on performance (and it's saner in other ways, too).

[eliasp]

Ran into the same issue on a device with 8GB memory:

nonroot@box $ nix search mkpasswd
GC Warning: Failed to expand heap by 402653184 bytes
warning: using cached results; pass '-u' to update the cache
Attribute name: nixpkgs.perlPackages.StringMkPasswd
Package name: perl-String-MkPasswd
Version: 0.05
Description:

Attribute name: nixpkgs.mkpasswd
Package name: mkpasswd
Version: 5.3.0
Description: Overfeatured front-end to crypt, from the Debian whois package
nonroot@box $ nix-env -i mkpasswd
GC Warning: Failed to expand heap by 402653184 bytes

I also happened occasionally that my system simply started hanging out of the blue, which I suspect was caused by system.autoUpgrade.enable = true; which might suffer from a related issue - disabled it for now to see whether it still happens.

[vcunat]

@eliasp: almost all people really want to prefer -A for nix-env. You got

Attribute name: nixpkgs.mkpasswd

so you just nix-env -iA nixpkgs.mkpasswd. I haven't seen nixos-rebuild suffering from this, as it uses attributes "inside".

[oxij]

@edolstra

Also, Nixpkgs has seen a lot of growth such that evaluation now takes > 3 times as much memory as two years ago This may also be a result of the use of certain new abstractions in Nixpkgs, e.g. `stdenv.mkDerivation` is a lot more complicated than two years ago.

I read changes of the significant bumps in that graph that were not immediately fixed several revisions later and all but the very last one were either changes to haskell packages or melpa. So my guess would be that we now have ~3x more packages in release.nix that two years ago, not that `stdenv.mkDerivation` became 3x slower. The very latest bump is interesting, though, I'm not sure what causes it, it's either #44423 or #44517, bisecting now. I think a much more useful metric would be "nix-env -qa avg allocations per result produced", not the absolute value.

[davidak]

@edolstra can you give a status update? are there plans to fix the design problems? Maybe is the effort with flakes related? This seem to be a big issue considering that the project should grow in terms of package count.

[chrissound]

@vcunat many thanks for the suggestion to use nix-env -iA. Perhaps we can mention as a message when nix-env runs out of memory?

[newskooler]

I am just getting started with nix and following the Nix manual.
On chapter 9, one of the suggested lines to execute is this nix-env -qa, so when I run it I get the same error:

some_user@some_server:~$ nix-env -qa
GC Warning: Failed to expand heap by 16777216 bytes
...
GC Warning: Failed to expand heap by 16777216 bytes
GC Warning: Failed to expand heap by 262144 bytes
GC Warning: Out of Memory! Heap size: 500 MiB. Returning NULL!
error: out of memory

I am experiencing the same issue on a freshly installed Nix (I did not install any packages, just followed the manual linked above).

For more info:

some_user@some_server::~$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 4.15.0-66-generic, Ubuntu, 18.04.3 LTS (Bionic Beaver)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.4`
 - channels(root): `"nixpkgs-20.09pre221814.10100a97c89"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

I am running a 1GB RAM 1VCPU server.

Does nix have minimum server requirements? If yes, what are they?

P.S. I also run out of memory with commands for specific packages as well such as nix-env -qa firefox or even package installation like nix-env -i subversion

[AleXoundOS]

I am running a 1GB RAM 1VCPU server.

Does nix have minimum server requirements? If yes, what are they?

I'm not very experienced, but I think it's 3 GiB RAM nowadays (if not evaluating/building nix expressions remotely, i.e. run nixos-rebuild --target-host HOST from another machine).

P.S. I also run out of memory with commands for specific packages as well such as nix-env -qa firefox or even package installation like nix-env -i subversion

P.S.
It may sound quite contradictory (and regardless of the RAM usage) but nix-env is very unlikely the tool you want to provision your server with. As it does not comprise the best of declarative features NixOS can provide. Perhaps, it's one of the reasons nix-env can receive less care and performance fixes.

[windware-ono]

Package managers shouldn't eat all the RAM there is. No other package managers eat as much trying to install packages.

Not everyone wants to start from NixOS but try out Nix package management system and try to replace the current one but right now it get's OOM'ed trying to install several packages with 1.5GB of memory left (including swap).

[katyo]

Package managers shouldn't eat all the RAM there is. No other package managers eat as much trying to install packages.

Not everyone wants to start from NixOS but try out Nix package management system but try to replace the current one but right now it get's OOM'ed trying to install several packages with 1.5GB of memory left (including swap).

It because nix isn't an ordinary package manager. It's configuration philosophy. Of course it eats too much RAM in spite of lazy evaluation of nix expressions. This is serious issue which can stop nix usage specifically in virtual environments and clouds where available RAM usually restricted.

I hope this problem can be (and should be) solved. May be special methods of caching can help to work around it (for example by splitting result of evaluation to hot/cold parts in RAM/disk respectively).

[pickfire]

Same goes for me when trying out nix with 2GB RAM, nix eats up half the memory while g++ taking only 20% of the memory, I wonder why it does not frees up memory for the compiler but takes it up itself, up to the point I have to quit firefox to fully build the application, I think this may be an issue in older raspberry pis.

[xcube16]

I am having this same issue with an 8GB laptop. When I run nix-index it stops at + querying available packages and a nix-env process slowly eats up all the memory until the OOM killer gets it. I am able to install packages and do upgrades just fine, so I don't know why nix-index needs over 7GB.

[stale]

I marked this as stale due to inactivity. → More info

[samrose]

I could be wrong, but from my experience if you every package pre-built and cached, and if you have 3GB or more of memory, you do not tend to run into this issue.

[samrose]

Also, avoiding using nix-env is a good practice too. nixos-rebuild is much better

[ghost]

running nix-env -qaP on new installation

[klarkc]

Unfortunately Nix is not a simple package manager: it implements a lazy purely functional language, the evaluation of which uses a lot of memory. It has some design issues that make it hard to reduce memory usage (in particular the use of a monolithic pkgs set combined with the .override feature makes garbage collection largely ineffective).

@edolstra I'm curious about this, has Nix language garbage collection and its inefficient, or it was not implemented because it's inefficient? I mean, Haskell GHC have garbage collection and Haskell is also a lazy purely functional language, and I normally don't see this kind of problem with Haskell compiled programs.

[edolstra]

I said ineffective, not inefficient. Large circular data structures like pkgs mean that there isn't much that the garbage collector can do.

[matu3ba]

@edolstra Just curious: Is there analysis of "the most evil packages" yet or any tooling to analyze dependencies semi-automatically? I have not seen links in the issue yet that go into direction of "fixing the cause of the problem".

[nartamonov]

I am just getting started with nix and following the Nix manual. On chapter 9, one of the suggested lines to execute is this nix-env -qa, so when I run it I get the same error:

Got the same behaviour, tried nix-env -qaP firefox from the chapter 4 of Nix Manual and got out of memory. Virtual machine has 2GB of RAM.

I'm new to NixOS and learning it now. Honestly, the emergence of such problem with basic command after an hour of study is not what I expected. Otherwise, I very like the whole idea of NixOS but my confidence was shaken.

[klarkc]

Don't worry @nartamonov, I've been using Nix since for a while now and haven't had any major issues except for some rare out of memory events when building on my modest laptop. However, there are new features currently being developed that aim to improve performance, such as file deduplication, ca-derivations, and RFCs.

[doronbehar]

Got the same behaviour, tried nix-env -qaP firefox from the chapter 4 of Nix Manual

Although officially considered unstable, I recommend nix search.

[Zahrun]

I don't know if this is the right place, but I just noticed nixos-rebuild is taking near to 7Go on my system

Is there any perspective to reduce that?

[2xsaiko]

I'm not familiar with Nix's memory model so I have no idea if this is theoretically possible and, if so, how much would have to change in Nix's code, but since Nix is a functional language, none of the evaluated expressions that are stuck in memory actually have to be kept around since they can always be recalculated, right? So, in case of low memory, is it possible to drop the least frequently referenced (or other heuristic) expressions from memory to free up space for what is currently needing the memory, and next time they are used, evaluate them again from scratch? Of course, this would come with decreased performance, but it would be better than running out of memory altogether.

[matu3ba]

Zig master has now a declarative build system and handles user-provided memory limits (but not yet cgroups/lockdown for strict enforcing and graceful retries etc).
I'm quite surprised to hear that nobody implemented user-providable memory quotas with graceful fallbacks yet.

[mrtolkien]

I was thinking of using nix-env to install some dependencies on my machines as I often juggle between Linux and Mac OS and nixpkgs is by far the most up-to-date packages repository as well as making clean installs compared to brew.

But like many people here nix-env crashes my 1Gb servers and forces me to fallback to other package managers with outdated/missing packages.

Is there any way to limit the RAM usage of nix-env somehow, particularly when only installing binary packages with specific versions?

[chrissound]

I was thinking of using nix-env to install some dependencies on my machines as I often juggle between Linux and Mac OS and nixpkgs is by far the most up-to-date packages repository as well as making clean installs compared to brew.

But like many people here nix-env crashes my 1Gb servers and forces me to fallback to other package managers with outdated/missing packages.

Is there any way to limit the RAM usage of nix-env somehow, particularly when only installing binary packages with specific versions?

A simple work around is to just use a swap. Which can be done with a few commands. I do this all the time on a server with 0 issues.

[mrtolkien]

Yes, I could make it work with the -A option or using a swap, but nix-env --install xxx crashing any machine without access to 4Gb+ of RAM + swap still feels like a bug more than a feature.

I definitely have very little knowledge about the nix system as a whole so I'm unsure as to where the RAM usage is coming from. If it's linked to data structures within the nix programming language taking up too much RAM, could using sqlite and querying it instead of storing packages information in RAM help? Sorry if that's a stupid idea.

[7c6f434c]

-i without -A is something that has not aged well… but even -iA or building a profile with buildEnv is still expensive.

On my VPS I define a toolset using an expression with buildEnv, evaluate it locally on my laptop, and then do closure-copying.

For some things I actually fetch the build data from Hydra, and then do stuff by output path (not attribute or name), which won't work except via substituters, but as I am asking Hydra for what is the latest stuff available in the cache…

[eode]

Is it planned to fix this?

[mdehollander]

I have been also running into this memory issue on a opensuse tumbleweed machine with nix installed. Installing packages worked using the -iA flag, but how to update them?
Running nix-channel --update and nix-env --upgrade has the same memory problems. And there is no nixos-rebuild or home-manager available, as those are the other suggested upgrade options.

[2xsaiko]

@mdehollander nix-env is pretty much the worst in terms of anything, including memory usage, I think. If you want to use imperative package management, you could try nix profile, because it implements updates in a way that doesn't have to evaluate the entire nixpkgs and also isn't very fragile (nix-env stores packages as their package names instead of attr path which is not good since those don't have an index and aren't even necessarily unique).

Otherwise, if you want to do declarative package management and don't want to use home manager, I suppose you could make an env yourself with pkgs.buildEnv along with a rebuild script to build the expression for it and link it to a known path. Then add its bin to the PATH environment. The expression might look something like this:

let
  pkgs = import <nixpkgs> {};
in pkgs.buildEnv {
  name = "path";
  paths = with pkgs; [
    nix
    zsh
    # more packages
  ];
  pathsToLink = ["/bin"]; # plus anything else you might want
}

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/reduce-nix-env-u-memory-usage/37954/2

[tobiasBora]

I'm also hitting this issue with nix-review, that uses nix-env under the hood… 11Gb of RAM used for a really basic package, even before compilation starts. Is there any alternative here or plans to fix this?