Nixpkgs performance regression #3594

edolstra · 2014-08-14T17:15:59Z

Doing nix-env -qa on master now takes 2.81s on my system, while on the 14.04 branch it was only 1.72s, a 63% increase. Meanwhile the number of packages only increased by 9%.

We should do a bisect to figure out what caused the slowdown.

The text was updated successfully, but these errors were encountered:

madjar · 2014-10-06T09:42:49Z

@edolstra I wrote a small script to graph the evolution of the evaluation time, but I discovered it was constant. I tried on another computer (one with a SSD), and I have found a difference between master and 14.04.

Can you confirm that you have a SSD and that the evaluation is IO-bound otherwise?

Anyway, I'll use my script on my SSD-computer and show you the results.

edolstra · 2015-03-04T21:54:08Z

Another big regression: nix-env -qa on the 14.12 branch takes 1.89s on my system, but 3.03s on master. More than 50% increase.

With nix-env -qa --out-path, it's even worse: up from 13.4s to 34.0s, and max resident size going 861 MB to 1933 MB.

A quick git bisect points at 62bfe75, but there may be other contributors.

edolstra · 2015-03-04T21:54:51Z

This may be related to the OOMs while evaluating the nixos:trunk-combined jobset.

edolstra · 2015-03-05T10:03:02Z

The OOMs may have started here: b2fdcf8

offlinehacker · 2015-03-05T10:09:05Z

Yes, it could be because of nodePackages.
On Mar 5, 2015 11:03 AM, "Eelco Dolstra" notifications@github.com wrote:

The OOMs may have started here: b2fdcf8
b2fdcf8

—
Reply to this email directly or view it on GitHub
#3594 (comment).

offlinehacker · 2015-03-05T10:15:49Z

I actually tought that this could happen after i automatically detect platforms on which node packages should be built. The fact is these few exported node packages have a lot of dependencies and recursive dependency avoidance algorithm is not so trivial. So if anyone could propose optimization or we don't export those packages.

offlinehacker · 2015-03-05T10:29:32Z

I will try to optimizem it for sake of evaluation in following days.

domenkozar · 2015-03-07T18:28:09Z

@offlinehacker can we revert meanwhile?

offlinehacker · 2015-03-07T18:30:43Z

Meh, i will comment out platforms, don't revert or it will break node
packages probably.
On Mar 7, 2015 7:28 PM, "Domen Kožar" notifications@github.com wrote:

@offlinehacker https://github.com/offlinehacker can we revert meanwhile?

—
Reply to this email directly or view it on GitHub
#3594 (comment).

vcunat · 2015-03-07T21:46:06Z

I think meta.platform changes should have no practical impact on performance of evaluation (for nix-env -qa).

lucabrunox · 2015-03-10T11:07:49Z

Reverting b2fdcf8 doesn't solve the problem for me.

lucabrunox · 2015-03-10T12:39:30Z

Forget about the above comment, I'm able to nix-instantiate nixos/release-combined.nix -A tested of commit ed75d61 (6 hours ago). It took about 2 minutes and 6.5gb of ram.
Also a commit of like 10 weeks ago requires about the same amount of time and ram, maybe slightly less time.

Perhaps I should try evaluating the whole release-combined instead of only "tested", not sure I will be able to do it with my ram :(

lucabrunox · 2015-03-10T12:46:43Z

I'm also able to evaluate the whole nix-instantiate nixos/release-combined.nix

domenkozar · 2015-03-13T18:06:17Z

@edolstra could you help out with the hydra issue? I have to revert back to 14.12 because VirtualBox update is not in the current master channel

domenkozar · 2015-03-15T20:04:39Z

I can evaluate master with reverting b2fdcf8, without that it gets killed by OOM (8GB of ram).

@offlinehacker I'm reverting the commit if you don't find a reasonable argument/solution in a day. We haven't updated unstable channel in 17 days and nixpkgs in 40 days. This is slowing down development of NixOS project quite a bit.

offlinehacker · 2015-03-16T02:15:47Z

Ok, i will take time tomorrow(today to be more precise) to fix this, if i
don't figure it out, i will revert it in a way that it will not break all
nodePackages.
On Mar 15, 2015 9:04 PM, "Domen Kožar" notifications@github.com wrote:

I can evaluate master with reverting b2fdcf8
b2fdcf8,
without that it gets killed by OOM (8GB of ram).

@offlinehacker https://github.com/offlinehacker I'm reverting the
commit if you don't find a reasonable argument/solution in a day. We
haven't updated unstable channel in 17 days and nixpkgs in 40 days. This is
slowing down development of NixOS project quite a bit.

—
Reply to this email directly or view it on GitHub
#3594 (comment).

domenkozar · 2015-03-16T20:48:21Z

Hmmm, I can also evaluate master with 6.5GB of ram. @edolstra is it because hydra uses old kernel? I'm using 3.18.

domenkozar · 2015-03-17T07:56:52Z

I've pushed 1e4ba02.

PS: /nix/store/kdfywafkcyhnws66i06va6vn83xihzmi-hydra-0.1pre1234-abcdef/bin/hydra-eval-jobs /home/ielectric/dev/nixpkgs/nixos/release-combined.nix >/dev/null gets killed by OOM on my 8GB of ram.

offlinehacker · 2015-03-18T01:01:21Z

How can i test performance issues, i was trying to fix this today, i don't have 8GB of ram to try to evaluate everything. If i do nix-env -f default.nix -i nodejs-bower-1.3.12 i don't get any GC errors. I would like someone to help me debug the problem, not just revert some commits and then break a lot of packages. I will try to debug this tomorrow on 32G server, i have to install nix there first.

lucabrunox · 2015-03-18T09:13:31Z

Will try with hydra-eval-jobs instead of nix-build.

peti · 2015-03-18T09:57:17Z

@offlinehacker, in all fairness, it's not like someone "just revert some commits".

Also, note that @iElectric has posted the exact command-line he's using to re-produce the problem.

lucabrunox · 2015-03-18T10:09:29Z

So with a commit of 6 days ago 0bc51ea (because of binary cache) I can reproduce the problem with hydra-eval-jobs. But I don't think it's something related to node packages, honestly. Node packages were evaluated just fine. The fact that's the commit where hydra stopped evaluating, is not a 1:1 indication it's a node packages problem. @iElectric you wasn't very clear, were you able to evaluate the jobset after reverting the commits, while you weren't before?

Here it stopped when evaluating tests, right after the i3wm test. The problem is just it needs more ram nowadays, I believe. Nothing related to node packages. E.g. the evaluation hit 7.9gb of ram out of 8gb for a long time, so the GC was a lot aggressive on keeping below the max ram. At some point however it just needed more ram, wasn't there, and stopped. That's it, at least what I observed.

domenkozar · 2015-03-20T12:57:56Z

There we go, first eval after a month: http://hydra.nixos.org/eval/1178620

domenkozar · 2015-03-20T20:06:05Z

@offlinehacker with 0461f35 you could try applying nodePackages fixes again and see how much it increases memory usage

offlinehacker · 2015-03-21T16:46:00Z

@iElectric nice, so with current master i get 4.45s for nix-env -f default.nix -qa and the same value on origin/master, so i guess that's and i can succesfully evaluate nix-instantiate nixos/release-combined.nix on my laptop with 4G ram and if i compare with origin/master it is almost same memory usage. I'm reverting your reverts now. If there's something i'm missing, please let me know.

domenkozar · 2015-03-21T19:21:41Z

With nodePackages problematic commits (master):

Memory usage: 5149MB (105% increase)
Time to evaluate: 244s (16% increase)

Without the problematic commits (51d4656):

Memory usage: 2508MB
Time to evalute: 209s

PS: tested with

`which time` -f "%P %M" /nix/store/kdfywafkcyhnws66i06va6vn83xihzmi-hydra-0.1pre1234-abcdef/bin/hydra-eval-jobs /home/ielectric/dev/nixpkgs/nixos/release-combined.nix >/dev/null

domenkozar · 2015-03-21T19:22:06Z

@edolstra do you think the regression is bad enough not to have such commits?

offlinehacker · 2015-03-21T19:30:18Z

Well the real problem is that i don't have any idea what's going on in hydra-eval-jobs that's causing such memory usage, what can i easily make is disable building node packages on hydra.

domenkozar · 2015-03-21T19:32:07Z

That might be a good solution as build time is very quick for npm/iojs

offlinehacker · 2015-03-21T19:34:07Z

Not so quick anymore in comparisson how fast npm is now.

offlinehacker · 2015-03-21T19:34:53Z

Ok, is there any better way to disable builds on hydra as removing platforms?

domenkozar · 2015-03-21T19:36:15Z

By removing recurseIntoAttrs at https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix#L1791 (and the same for iojsPackages)

offlinehacker · 2015-03-21T19:36:31Z

Ah, ok 👍

domenkozar · 2015-03-21T20:07:45Z

2273MB 👍

vcunat · 2015-06-09T18:50:45Z

So I looked into this, and bisecting was fast enough (about 12 steps between master and the merge-base against 14.12): 8fb5494 add kde-applications-14.12.1.

On my notebook there's a distinct jump on that single commit – all I've seen before was faster and all after was slower than the point of jump (from the ~11 commits tried in-between).

$ git co 8fb5494309d
$ NIX_PATH= time nix-env -f . -qa > /dev/null
9.10user 0.42system 0:09.54elapsed 99%CPU (0avgtext+0avgdata 569920maxresident)k
0inputs+0outputs (0major+143823minor)pagefaults 0swaps

$ git co HEAD^
$ NIX_PATH= time nix-env -f . -qa > /dev/null
4.41user 0.37system 0:04.79elapsed 99%CPU (0avgtext+0avgdata 567820maxresident)k
0inputs+0outputs (0major+143340minor)pagefaults 0swaps

ATM I don't see what makes it so slow, /cc @ttuegel. Just removing recurseIntoAttrs for kde doesn't help noticeably for me (though it removes quite a few packages).

ttuegel · 2015-06-09T19:03:14Z

@vcunat It's probably because that commit adds ~2.8k lines of Nix code between two files. Are you sure you removed recurseIntoAttrs for kdeApps_14_12?

vcunat · 2015-06-09T19:04:34Z

Yes, sure (I did check that KDE packages disappeared from the list). A better test case is:

command time nix-instantiate -A kde4.kdelibs

... counting second and further runs, as the first one can create derivations. For me it jumps from ~0.26s to ~3.9s!

ttuegel · 2015-06-09T19:08:40Z

That makes sense. kde4 now gets its kdelibs from kdeApps_stable because that is where the KDE team releases updates. So, evaluating kde4.kdelibs requires first parsing all of kdeApps_stable (note: parsing, not evaluating). I think the best way forward here is to split up the expressions into individual files for optimum laziness, and we probably need to do the same for the other large package collections.

vcunat · 2015-06-09T19:10:33Z

Would you look at it? (To avoid duplicating effort; though I'd be unlikely to do it today anyway.)

ttuegel · 2015-06-09T19:19:20Z

Sure, I'll take a look.

One remark I can make off-hand is that something is different about the KDE packages because evaluating a package from haskellPackages (which is a much larger collection) takes only a fraction of a second on my system.

vcunat · 2015-06-09T19:50:04Z

I'd expect some of the many transformations in there is/are expensive, as the kde expressions are built in a rather complex way.

vcunat · 2015-06-25T16:13:41Z

Sounds great! Thanks a lot :-)

domenkozar · 2015-12-28T08:59:37Z

We're back to 7.5G of memory usage for nixos/release-combined.nix

domenkozar · 2015-12-28T09:10:19Z

I removed recurseIntoAttrs for node 5.0 and 4.3 packages and it's now down to 4.6G.

Two reasons for this change: - most of 5.0 packages don't build yet - node packages are memory intensive and block Hydra evaluation (Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS) PS: Removing node packages from evaluation goes from 7.5G down to 4.6G for whole nixos release job. See #3594 and #11865

edolstra added this to the 14.10 milestone Aug 14, 2014

vcunat added the enhancement label Aug 14, 2014

guyzmo mentioned this issue Feb 22, 2015

nixpkgs search hanging the system #6515

Closed

edolstra modified the milestones: 15.05, 14.12 Mar 4, 2015

edolstra added the 1.severity: blocker label Mar 4, 2015

aszlig mentioned this issue Mar 6, 2015

'nixops deploy' exits with 'Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS' NixOS/nix#287

Closed

vcunat mentioned this issue Mar 16, 2015

Update moderncv SHA #6747

Closed

vcunat mentioned this issue May 19, 2015

nodePackages derivations not listed with nix-env --query #7888

Closed

vcunat mentioned this issue Jun 9, 2015

Huge performance regression in NixOS evaluation #8152

Closed

ttuegel closed this as completed in 99d2018 Jun 25, 2015

hqurve mentioned this issue Feb 3, 2022

nix search does not show Node packages #157851

Closed

Nixpkgs performance regression #3594

Nixpkgs performance regression #3594

Comments

edolstra commented Aug 14, 2014

madjar commented Oct 6, 2014

edolstra commented Mar 4, 2015

edolstra commented Mar 4, 2015

edolstra commented Mar 5, 2015

offlinehacker commented Mar 5, 2015

offlinehacker commented Mar 5, 2015

offlinehacker commented Mar 5, 2015

domenkozar commented Mar 7, 2015

offlinehacker commented Mar 7, 2015

vcunat commented Mar 7, 2015

lucabrunox commented Mar 10, 2015

lucabrunox commented Mar 10, 2015

lucabrunox commented Mar 10, 2015

domenkozar commented Mar 13, 2015

domenkozar commented Mar 15, 2015

offlinehacker commented Mar 16, 2015

domenkozar commented Mar 16, 2015

domenkozar commented Mar 17, 2015

offlinehacker commented Mar 18, 2015

lucabrunox commented Mar 18, 2015

peti commented Mar 18, 2015

lucabrunox commented Mar 18, 2015

domenkozar commented Mar 20, 2015

domenkozar commented Mar 20, 2015

offlinehacker commented Mar 21, 2015

domenkozar commented Mar 21, 2015

domenkozar commented Mar 21, 2015

offlinehacker commented Mar 21, 2015

domenkozar commented Mar 21, 2015

offlinehacker commented Mar 21, 2015

offlinehacker commented Mar 21, 2015

domenkozar commented Mar 21, 2015

offlinehacker commented Mar 21, 2015

domenkozar commented Mar 21, 2015

vcunat commented Jun 9, 2015

ttuegel commented Jun 9, 2015

vcunat commented Jun 9, 2015

ttuegel commented Jun 9, 2015

vcunat commented Jun 9, 2015

ttuegel commented Jun 9, 2015

vcunat commented Jun 9, 2015

vcunat commented Jun 25, 2015

domenkozar commented Dec 28, 2015

domenkozar commented Dec 28, 2015