tarball job on Hydra eats too much RAM

[vcunat]

Background

The tarball job (e.g. in this jobset; code) does various evaluation checks on nixpkgs by calling nix.

It's two derivations (linearized) using just single thread (maybe except nix's GC). However, some steps require huge amount of RAM, which causes practical issues, also frequently on current hydra.nixos.org.

RAM consumed now

🥇 checking eval-release.nix step is far the worst, max. around 36 GiB, as measured by time package+command.

🥈 generating packages.json step gets around 18 GiB.

Others use around 10 GiB or less, which seems... bearable.

Running in practice

Given enough RAM, the build takes 2x 10-15 minutes (roughly). If not, result depends on your machine. The current builders on hydra.nixos.org usually manage to keep CPUs relatively busy with other work, but jobs that "touch lots of RAM or disk" can get slowed down extremely due to (un)swapping and time out – mainly the tarball jobs themselves but not exclusively.

---

I believe that strict evaluation checks are nice to have, even as blocker for channels, so I'd prefer to preserve at least similar level of strictness.

[vcunat]

For a stop-gap workaround I'd probably mark these two steps with big-parallel tag, which should bring back reliability at the expense of some idle CPUs.

These machines run 2 jobs only and have 64 GiB of physical RAM. (I hope we always get the same amount of RAM or larger.)

[vcunat]

Stop-gap implemented as PR #229132