🦦 NixOS 23.05 — Zero Hydra Failures

[RaitoBezarius]

Hi, we are Ryan Lahfa & Martin Weinelt, the release managers for NixOS 23.05 🦦 ("Stoat").

Today we want to invite everyone to participate in the Zero Hydra Failures Project, wherein we prepare the package set for the upcoming release, up until its public release at the end of may.

There are only two more upcoming dates that we'd like to mention in that context:

• 2023-05-22: Branch-off
• 2023-05-31: 23.05 Release

The complete timeline can be found in the

• 🦦 NixOS 23.05 — Release Schedule #223562.

The mission

Every time we plan to do a release, we take time to stabilize the master branch and later on the release branch.
Our goal here is to reduce the number of failing jobs on the nixpkgs:trunk nixos:trunk-combined jobsets as much as possible before branch-off. We call this the "Zero Hydra Failure" campaign.

Besides aiming for zero failed jobs, we also strive to again provide all packages that were available in the previous release.

Changes need to always target the master branch. Take note that the branch-off will occur on 2023-05-22, after which ZHF changes will need to be tagged with the backport: release-23.05 label to land in the stable release.

Jobsets

The relevant jobsets to check for failing jobs are:

• nixos:trunk-combined (Linux packages & NixOS tests)
• nixpkgs:trunk (Linux & Darwin packages)

Workflow

Finding broken packages

Eval reports

Evaluation reports provide a structural overview of the most impactful failing builds. They originated at https://github.com/nix-community/nix-review-tools and were automated over at https://github.com/malob/nix-review-tools-reports.

1. Navigate to https://malob.github.io/nix-review-tools-reports/
2. Open the relevant jobset: see the previous section on which jobset to select based on what you are looking at (Linux, Darwin or NixOS tests).
3. Browse the latest reports for build failures
   
4. Follow the links to the build failure on hydra

ZERO Hydra Failures

The platform automatically crawls Hydra and lists packages by maintainer and lists the most important dependencies (failing packages with the most dependants). It also graphs the general trend per platform.

1. Navigate to https://zh.fail

For the record, we started ZHF here:

Latest Linux evaluation (completely built):	[1794693](https://hydra.nixos.org/eval/1794693) on 2023-05-07 15:51:32 (UTC)
Latest Darwin evaluation (completely built):	[1794694](https://hydra.nixos.org/eval/1794694) on 2023-05-07 16:12:27 (UTC)
Failing builds on aarch64-darwin:	739
Failing builds on aarch64-linux:	1781
Failing builds on i686-linux:	612
Failing builds on x86_64-darwin:	825
Failing builds on x86_64-linux:	1909
Total failed builds	5866

For comparison, last's release ZHF started there:

Latest Linux evaluation (completely built):	1784776 on 2022-11-05 07:52:49 (UTC)
Latest Darwin evaluation (completely built):	1784782 on 2022-11-05 09:24:59 (UTC)
Failing builds on aarch64-linux:	1706
Failing builds on i686-linux:	262
Failing builds on x86_64-darwin:	1057
Failing builds on x86_64-linux:	2695
Total failed builds	5720

Check on packages you maintain

1. Clone nixpkgs and checkout the master branch
2. Run

   nix-build maintainers/scripts/build.nix --argstr maintainer <name>
   

Alternatively: you can check https://zh.fail/failed/overview.html also.

Hydra

Hydra is nixpkgs CI platform, where all active branches are built and pushed into the cache, after which channels can originate from its build results.

1. Open the nixpkgs:trunk jobset
2. Select the latest evaluation
   
3. Directly failing jobs are marked with a red cross, while transitively failing ones are greyed out.
   
4. Use the search form to scope the package list to things relevant to you and that you can test.
   

Submit fixes

6. Search through PRs to make sure none provided a fix yet. If there is one, please take the time and help review the change.

7. If there is no open PR, troubleshoot why it's failing and fix it.

8. Pull Request the fix against the master branch and wait potential review & change requests

   • Add the 0.kind: ZHF Fixes label, so people can better browse these fixes
   • If your PR causes more than ~500 rebuilds, it is generally preferred to target staging to avoid compute churn for users on master.
   • If no reviewer is automatically added to your PR, check the Git history or the maintainers and ping them (in the pull request) or add them (if you have the rights) as reviewers
   • If, after a while, no one reviewed the PR, you can post it in https://discourse.nixos.org/t/prs-ready-for-review/3032/2183 to get more attention
   • If, after an (extra) while, nothing really happened, you can drop a line in the NixOS development channel or mention @NixOS/nixos-release-managers on the PR

Backporting

After 2023-05-22

10. Apply the relevant backport label to land the fix in the release branch

    • Changes to master get backported into release-23.05
    • Changes to staging get backported into staging-23.05

11. If the backport action fails, follow the manual backporting steps. Make sure to use git cherry-pick -x <rev> on all commits intended for backport.

    • @jonringer created a video covering the backport process.

---

Always link back to this issue by mentioning the issue number in the description of your pull request:

ZHF: #230712

If your PR receives no reviews or does not get merged, feel free to

• add the 0.kind: ZHF Fixes label, so people can better browse these fixes
• request or mention @NixOS/nixos-release-managers on the PR

Broken packages

Everything we cannot fix in time will need to be marked broken on the respective platforms, so that Hydra will not retry builds over and over, thereby wasting compute resources.

Set meta.broken and add a reference and/or explanation, like this:

meta = {
  # ref to issue/explanation
  broken = stdenv.isDarwin; # only broken on darwin
  # broken = true; # broken on all platforms.
};

Orphaned packages

You can read about failing packages without a maintainer here: https://zh.fail/failed/by-maintainer/_.html (orphaned packages).

If you're new to NixOS, adopting an orphaned package is a great way to get involved and contribute to the community. By doing so, you'll not only help improve the overall quality of the NixOS ecosystem, but you'll also gain valuable experience working with Nix, the language and tool that powers the package management system.

By adopting an orphaned package, you'll be taking on a responsibility that can be both challenging and rewarding. You'll need to understand the package's code and dependencies, make sure it builds and works correctly, and respond to any issues or pull requests that come up. This process can be a great learning experience, as you'll be exposed to a wide variety of programming languages and libraries.

Moreover, by adopting an orphaned package, you'll be making a tangible impact on the NixOS community. Your contributions will be greatly appreciated by users who depend on that package, and you'll be helping to ensure that NixOS releases are as stable and up-to-date as possible.

Closing

This is a great way to help NixOS, and it is a great time for new contributors to start their nixpkgs adventure. 🥳

As with the feature freeze issue, please keep the discussion here to a minimum so we don't ping all maintainers (although relevant comments can of course be added here if they are directly ZHF-related) and ping one of the release managers (@mweinelt, @RaitoBezarius) in the respective issues.

Extra thanks to @JulienMalka and @dasJ who fixed https://zh.fail in time (~ yesterday-today) for ZHF, and made it insanely faster! (~6 hours to ~15 mn)

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/zero-hydra-failures-23-05-edition/27959/1

[RaitoBezarius]

Reserved post.

cc @NixOS/nixpkgs-committers @NixOS/nixpkgs-maintainers

[Krutonium]

Did you intend to say November instead of May?

[RaitoBezarius]

NixOS 23.05 has been released! https://discourse.nixos.org/t/nixos-23-05-released/28649
We will take care of the rest of the PRs, don't hesitate to keep 23.05 even better after ZHF :).