hydra: increase max_output_size from 2GB to 3GB

[bobvanderlinden]

A potential solution for NixOS/nixpkgs#159612

Currently the Hydra nixos-unstable channel is clogged due to the Gnome ISO being just a tad bit bigger than 2GB. This results in the ISO build itself succeeding, but hydra marking the build as failing because of a failing validation check for the size. This has been the case for 6 days now. I'm suggesting this PR as a short-term solution. There are some critical CVE's that were fixed, but are not coming through right now (NixOS/nixpkgs#160354).

The validation check for the output size seems to be here: https://github.com/NixOS/hydra/blob/f6e86efc9fc4f132cc183bc1c139fbaeeaa19238/src/hydra-queue-runner/build-remote.cc#L472-L475

maxOutputSize is initialized with configuration option max_output_size here: https://github.com/NixOS/hydra/blob/d0bc0d0edafef4201a33391eae243c0e3c6925d3/src/hydra-queue-runner/hydra-queue-runner.cc#L44. Note that its default is 2 << 30, thus 2GB.

Note that I don't run Hydra myself and have never done anything with nixos-org-configurations before, so this is a bit of a 'change in the dark' for me. I wanted to propose this as a short-term (temporary?) solution until the discussions and long-term solutions in NixOS/nixpkgs#159612 have been concluded

Feel free to edit this PR/commit if needed.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-should-the-hydra-output-size-limit-be/17761/6

[Kranzes]

If only there was an option to increase the output size for specific builds/jobs...

[bobvanderlinden]

If only there was an option to increase the output size for specific builds/jobs...

Could be a cool feature for hydra at a later time. Maybe worthwhile to create a separate issue? For now, changing the global limit should be enough to avoid the blockade on nixos-unstable.

[vcunat]

changing the global limit should be enough to avoid the blockade on nixos-unstable

Yes, except that we have also another issue blocking us for a week – failing evaluation.

[bobvanderlinden]

changing the global limit should be enough to avoid the blockade on nixos-unstable

Yes, except that we have also another issue blocking us for a week – failing evaluation.

Do you have an issue/link?

[vcunat]

I'm not aware of any.

[bobvanderlinden]

@grahamc @edolstra @garbas it seems you often review PRs on this repo and maintenance in this area. Could you review this one as well?

[grahamc]

I think it'd be good to raise the limit more slowly (say 2.5, 2.25G) but I'm happy enough to merge as-is. If someone wanted to submit a follow-up I'd merge that too :).

[bobvanderlinden]

Thank you! 👍 I see that a next build of nixos-unstable has just started. Awaiting with 🤞