system.autoUpgrade causes high cpu and memory load

[rkoe]

Describe the bug
When using system.autoUpgrade.enable = true;, NixOS gets upgraded automatically.
If some packages have to be compiled, this may cause a very high CPU load and use several GB of memory, and make the machine unusable until the upgrade is done. (here recently: several minutes of full CPU load on all CPUs and ~5GB of memory).

Would it be possible to (a) restrict the CPU-usage, e.g. via nice and/or less parallel build, so the machine is still usable, (b) reduce the memory usage somehow or (c) skip large builds from autoUpgrade and notice root?

To Reproduce

• Install packages with no current binary-cache, so that they have to be compiled locally.
• set system.autoUpgrade.enable = true;
• Wait for the next upgrade.

Expected behavior
Less resource usage / less impact of the upgrade-process on the current use of the machine.

[aanderse]

It sounds like you want to use a builder to build packages on another machine so your primary machine doesn't have to. You can do that.

If you're interested in limiting system resources used by the build I think you can do that too using some systemd options on the service running your update. This seems like a reasonable place to start.

[hedning]

Nix also supports a --no-fallback option which will disable local building if the binary cache fails. This should be passable to system.autoUpgrade.flags. Might be what you're looking for.

[rkoe]

Thanks.

For a production-machines, a builder-machine is definitely sensible.
But for "normal" machines, this would be too much. I'll try configuring systemd; but I think this could be done directly in configurations.nix.

--no-fallback sounds nice, but I cannot find any documentation; there's only some documentation for --fallback for nix-env, which should be false by default according to the documentation. Did I miss the relevant documentation, or is the documentation outdated?

So, my enhancement-suggestion would be:

• change NixOS to run autoupdate with a lower priority via "nice" by default
• maybe add an option to "system.autoUpgrade" to limit the CPU-resources to e.g. 1 CPU (if this cannot be done with system.autoUpgrade.flags)
• add something about this (at least for me unexpected) behaviour and solutions to the documentation.

[aanderse]

Yeah I would think you could just modify the systemd.services.nixos-upgrade service config and setup what you need.

[hedning]

Ah, sorry, you'll need to provide --option fallback false. The nix command introduced a bunch of options which can also be passed to nixos-rebuild. nix also supports the short forms like --no-fallback, but that's apparently not supported by nixos-rebuild 🤷‍♂️

Edit: agreed that it makes sense to at least have an easy toggle to limit the cpu/ram (might be sensible as a default too).

[pasqui23]

I already have

nix={

    daemonIONiceLevel=7;
    daemonNiceLevel=19;
};

however if the rebuild include some big package,the system still degrades in quality and even makes polkit timeout before any prompt appears ,so I don't think that nice levels are sufficient

[aanderse]

Did anyone try playing around with the systemd.services.nixos-upgrade service? I feel like using this as a starting place is going to yield results... but have not tried myself.

[stale]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

[pasqui23]

Still valid for me

[aanderse]

@pasqui23 did you want to fiddle around with the systemd service at all? See my previous comment:

Did anyone try playing around with the systemd.services.nixos-upgrade service? I feel like using this as a starting place is going to yield results... but have not tried myself.

[pasqui23]

I can see what I can do when I find some free time

[pasqui23]

Also there are other services that use {,IO}Nice instead of {CPU,IO}Weight,if this work they should use the latter

[pasqui23]

First I need to test whether CPUWeight does indeed improve with responsiveness

[stale]

I marked this as stale due to inactivity. → More info

[pasqui23]

It's still a problem that the nix daemon will make the system unresponsive during heavy builds, unlike other services that consume high CPU load like boinc in my experience.

[pasqui23]

And after several months of

      CPUWeight=20;
      IOWeight=CPUWeight;

I can say that while it helps it doesn't fix it completely.

[stale]

I marked this as stale due to inactivity. → More info

[pasqui23]

Even on a much beefier laptop it still get noticebly less responsive when doing a big update

[stale]

I marked this as stale due to inactivity. → More info

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/system-autoupgrade-nearly-halts-my-system-even-though-nixos-rebuild-doesnt/23820/1

[Yeshey]

Did anyone try playing around with the systemd.services.nixos-upgrade service? I feel like using this as a starting place is going to yield results... but have not tried myself.

I added this to my configuration:

  systemd.services.nixos-upgrade.serviceConfig = {
    # https://unix.stackexchange.com/questions/436791/limit-total-memory-usage-for-multiple-instances-of-systemd-service
    # This doesn't prevent it from using swap
    MemoryHigh = [ "500M" ];
    MemoryMax = [ "2048M" ];

    # https://unix.stackexchange.com/questions/494843/how-to-limit-a-systemd-service-to-play-nice-with-the-cpu
    CPUWeight = [ "20" ];
    CPUQuota = [ "85%" ];
    IOWeight = [ "20" ];
  };

After a nixos-rebuild I can see the changes in the service cat /etc/systemd/system/nixos-upgrade.service.
I'll report back on how well this works once I figure out a way to test that.

my autoUpgrade option is set like this rn:

  # Auto Upgrade
  system.autoUpgrade = {
    enable = true;
    # dates = "22:14";
    flake = "${location}#${host}"; # my flake online uri is for example github:yeshey/nixos-config#laptop
    flags = [
      "--upgrade" # upgrade NixOS to the latest version in your chosen channel
      "--option fallback false" # fallback false should force it to use pre-built packages (https://github.com/NixOS/nixpkgs/issues/77971)
      "--update-input nixos-hardware --update-input home-manager --update-input nixpkgs" # To update all the packages
      # "--commit-lock-file" # commit the new lock file with git
    ];
    allowReboot = false; # set to false
  };

[Yeshey]

Did anyone try playing around with the systemd.services.nixos-upgrade service? I feel like using this as a starting place is going to yield results... but have not tried myself.

I added this to my configuration:

  systemd.services.nixos-upgrade.serviceConfig = {
    # https://unix.stackexchange.com/questions/436791/limit-total-memory-usage-for-multiple-instances-of-systemd-service
    # This doesn't prevent it from using swap
    MemoryHigh = [ "500M" ];
    MemoryMax = [ "2048M" ];

    # https://unix.stackexchange.com/questions/494843/how-to-limit-a-systemd-service-to-play-nice-with-the-cpu
    CPUWeight = [ "20" ];
    CPUQuota = [ "85%" ];
    IOWeight = [ "20" ];
  };

After a nixos-rebuild I can see the changes in the service cat /etc/systemd/system/nixos-upgrade.service. I'll report back on how well this works once I figure out a way to test that.

my autoUpgrade option is set like this rn:

  # Auto Upgrade
  system.autoUpgrade = {
    enable = true;
    # dates = "22:14";
    flake = "${location}#${host}"; # my flake online uri is for example github:yeshey/nixos-config#laptop
    flags = [
      "--upgrade" # upgrade NixOS to the latest version in your chosen channel
      "--option fallback false" # fallback false should force it to use pre-built packages (https://github.com/NixOS/nixpkgs/issues/77971)
      "--update-input nixos-hardware --update-input home-manager --update-input nixpkgs" # To update all the packages
      # "--commit-lock-file" # commit the new lock file with git
    ];
    allowReboot = false; # set to false
  };

Ok, so, this made things better, and it seems to keep things responsive when performing an upgrade, but the option "--option fallback false" doesn't seem to be preventing it from building from source, so what ends up happening even in more powerful machines is it falls back to building from source when it doesn't find pre built binaries in https://cache.nixos.org, so it might decide to build whole applications or even kernels in the background, witch in most machines even without restricting memory and CPU would take a couple hours, so with these restrictions it ends up never finishing the job before a reboot in a normal "home desktop PC".
I say it was building from source because I saw a bunch of building messages when checking the logs of nixos-upgrade with journalctl -u nixos-upgrade.service:

jan 17 04:40:05 nixos systemd[1]: Starting NixOS Upgrade...
jan 17 04:40:27 nixos nixos-upgrade-start[606306]: building Nix...
jan 17 04:40:37 nixos nixos-upgrade-start[606306]: building the system configuration...
jan 17 04:41:58 nixos nixos-upgrade-start[606375]: these 17 derivations will be built:
jan 17 04:41:58 nixos nixos-upgrade-start[606375]:   /nix/store/n078dxz6ckp8lyp3jj68x4596gmaswn1-webkitgtk-2.38.3+abi=4.0.drv
jan 17 04:41:58 nixos nixos-upgrade-start[606375]:   /nix/store/rqij6ljdhiswgvn55yq8mzwh8qx7dx6j-lutris-unwrapped-0.5.12.drv
jan 17 04:41:58 nixos nixos-upgrade-start[606375]:   /nix/store/va03sqh6qdxpid2za7rmqz52prx83j1n-lutris-usr-target.drv
jan 17 04:41:58 nixos nixos-upgrade-start[606375]:   /nix/store/cah8m7vxgv2pk2wndqz9kza0zq4np63v-lutris-fhs.drv
jan 17 04:41:58 nixos nixos-upgrade-start[606375]:   /nix/store/g59gnidyp6k7zqvpw6ijhmjq9mswaahg-lutris.drv
jan 17 04:41:58 nixos nixos-upgrade-start[606375]:   /nix/store/0ah63q5c6l6yl3g66p77ysfbmwin66j0-lutris.drv
jan 17 04:41:58 nixos nixos-upgrade-start[606375]:   /nix/store/mm4kpfppayhppkjh1hp7400vz2cl8rbd-system-path.drv
jan 17 04:41:58 nixos nixos-upgrade-start[606375]:   /nix/store/2pjsh66aj8wz3v84p17f0ma8md51kpr1-unit-systemd-fsck-.service.drv
jan 17 04:41:58 nixos nixos-upgrade-start[606375]:   /nix/store/az65bzh40855rbv127k5zfghv57mb80k-dbus-1.drv
jan 17 04:41:58 nixos nixos-upgrade-start[606375]:   /nix/store/ppmkj7vnq8ky3qhfwj7vdxvr375g26ri-unit-dbus.service.drv
jan 17 04:41:58 nixos nixos-upgrade-start[606375]:   /nix/store/4iq1blf8ys5ppj4v5y9180v92n4qibml-user-units.drv
jan 17 04:41:58 nixos nixos-upgrade-start[606375]:   /nix/store/6qxkz9v1vpnpxklfnkwbv7mpqqcx00s6-unit-dbus.service.drv
jan 17 04:41:58 nixos nixos-upgrade-start[606375]:   /nix/store/9wqahrglk2w9wv8nhlw5fr32h5j372pw-unit-polkit.service.drv
jan 17 04:41:58 nixos nixos-upgrade-start[606375]:   /nix/store/whmhsdz342rrq7lr46j1k0d4if3c1waz-unit-accounts-daemon.service.drv
jan 17 04:41:58 nixos nixos-upgrade-start[606375]:   /nix/store/ssmcwfvfaqp12flcpas70c6h52iqss33-system-units.drv
jan 17 04:41:58 nixos nixos-upgrade-start[606375]:   /nix/store/50mpy6p7qyvgws64m19y1c2chdvdab91-etc.drv
jan 17 04:41:58 nixos nixos-upgrade-start[606375]:   /nix/store/97gfry7ibqpnvhv3pphnlb7idsic3vzp-nixos-system-nixos-23.05pre442253.befc83905c9.drv
jan 17 04:41:59 nixos nixos-upgrade-start[606375]: waiting for lock on '/nix/store/0gva2jivdzqm734yzilxd404s80xhana-webkitgtk-2.38.3+abi=4.0-dev', '/nix/store/1jg7a7pnyvjigzmsk4p>
jan 17 13:06:31 nixos nixos-upgrade-start[606375]: waiting for lock on '/nix/store/kr1gs0dsqkj4971p09d4kwcrlqvm7azz-lutris-unwrapped-0.5.12-dist', '/nix/store/lampv20amrkyrkfs5ab>
jan 17 13:06:51 nixos nixos-upgrade-start[606375]: waiting for lock on '/nix/store/zpsgsw6i3qrlasms6wq9f7z8qykfdn83-lutris-usr-target'...
jan 17 13:06:56 nixos nixos-upgrade-start[606375]: waiting for lock on '/nix/store/pcxlill54hj77gjjmhla8im7ph1dbg1b-lutris-fhs'...
jan 17 13:07:11 nixos nixos-upgrade-start[606375]: building '/nix/store/mm4kpfppayhppkjh1hp7400vz2cl8rbd-system-path.drv'...
jan 17 13:07:59 nixos nixos-upgrade-start[606375]: building '/nix/store/az65bzh40855rbv127k5zfghv57mb80k-dbus-1.drv'...
jan 17 13:08:00 nixos nixos-upgrade-start[606375]: building '/nix/store/whmhsdz342rrq7lr46j1k0d4if3c1waz-unit-accounts-daemon.service.drv'...
jan 17 13:08:00 nixos nixos-upgrade-start[606375]: building '/nix/store/9wqahrglk2w9wv8nhlw5fr32h5j372pw-unit-polkit.service.drv'...
jan 17 13:08:00 nixos nixos-upgrade-start[606375]: building '/nix/store/2pjsh66aj8wz3v84p17f0ma8md51kpr1-unit-systemd-fsck-.service.drv'...
jan 17 13:08:01 nixos nixos-upgrade-start[606375]: building '/nix/store/6qxkz9v1vpnpxklfnkwbv7mpqqcx00s6-unit-dbus.service.drv'...
jan 17 13:08:01 nixos nixos-upgrade-start[606375]: building '/nix/store/ppmkj7vnq8ky3qhfwj7vdxvr375g26ri-unit-dbus.service.drv'...
jan 17 13:08:02 nixos nixos-upgrade-start[606375]: building '/nix/store/ssmcwfvfaqp12flcpas70c6h52iqss33-system-units.drv'...
jan 17 13:08:02 nixos nixos-upgrade-start[606375]: building '/nix/store/4iq1blf8ys5ppj4v5y9180v92n4qibml-user-units.drv'...
jan 17 13:08:04 nixos nixos-upgrade-start[606375]: building '/nix/store/50mpy6p7qyvgws64m19y1c2chdvdab91-etc.drv'...

So what we need is something that actually prevents it from building or upgrading a package if it isn't found in https://cache.nixos.org, the --option fallback false seems to have not done the trick, or I didn't apply it correctly

[aanderse]

Just to clarify: you're saying you don't want your system to update if any package isn't available in the cache, right?

[Yeshey]

Just to clarify: you're saying you don't want your system to update if any package isn't available in the cache, right?

I don't know nix enough to know what's possible or easy to implement or configure, but I'd say ideally it'd update packages that it finds in cache (or that are simply pre-built binaries?) and skip the ones it doesn't find instead of falling back to building them from source.

A toggle or way to compleatly prevent the update if any package is not found in cache would also be a solution, but a less desireable one by my view

I imagine that this is also made worse because I'm on unstable

its also causing some embarrassment with falkes, because it is updating my lock file even without finishing the job.

[aanderse]

ideally it'd update packages that it finds in cache (or that are simply pre-built binaries?) and skip the ones it doesn't find instead of falling back to building them from source.

Generally speaking your system is updated or not updated with no middle state - this is the value behind NixOS.

A toggle or way to compleatly prevent the update if any package is not found in cache would also be a solution, but a less desireable one by my view

If those are the options then this is the solution you should aim for.