New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory usage in eval #8621
Comments
We are aware that Nix evaluation tends to consume significant amounts of memory.
|
I want to add the boehm garbage collector is a conservative collector that does not allow heap compaction. I was hoping to spark some interest in assessing the mark-region algorithm as a possible new garbage collection algorithm for nix because it allows for heap compaction. There are some existing implementations in rust (immix) and c (whippet). In particular the whippet implementation seems relevant to nix because it has zero dependencies and has boehm-compatible api. |
@jsoo1 Interesting! Would you be interested in giving whippet a try? I've added notes about gc. |
@roberth sweet! Yes I would be interested! I was planning on setting aside some time for it if there seemed to be interest from the team. |
Let's move the discussion of replacing the GC over to #8626 |
Thanks for the summary! Since there's already memory leaks, I'm wondering if the gc is working as expected and maybe just improve the gc makes no sence if the most memory usage is by the leaked memory. |
I don't expect the GC itself to be broken, and I don't expect many leaks from it being conservative either. |
I ran into this while upgrading from NixOS 23.05 to 23.11 on my cloud VM with 2G of RAM. nix-build itself took 1G of that, and also there were some server services running, taking up about 500M, leaving only 500M for the actual derivation builds. Naturally it OOM'd kind of a lot. I worked around that by taking the derivation file paths from the That would not solve the original problem, and looking into a different GC still sounds valuable, but it might make the problem less acute for a portion of affected users. |
Regarding freeing the expressions, a starting point would be #5747 (comment), but also making sure to destruct If you have really small machines to deploy to, you might want to use |
|
CC @astro FYI While learning nix and nix flakes, this command freezed my dear and at that point mostly idle 16GB laptop, eating >10GB:
shortened output:
The output is actually from a run after I found https://github.com/rfjakob/earlyoom - You might want to recommend this nice tool somewhere! Please don't get this issue site tracked by me. I just thought it might be interesting to mention earlyoom in this issue and have an example on how to reliably eat a lot of memory. |
NixOS/rfcs#163 may reduce memory use for NixOS, by virtue of not having to load service modules that aren't used. It's one solution among potentially others, such as #9650 for cases like |
Just eval my nixos profile takes about 1G ram. It's kind of too much for me. And when running something like
nixpkgs-review
, nix will just take more and more and more ram.Is this by design?
Or is there any way I can reduce the memory usage?
nix-env
run bynixpkgs-review
The text was updated successfully, but these errors were encountered: