Debugging Channel Updates

Does the tested job exist?

Go to the page for the job: https://hydra.nixos.org/job/nixos/trunk-combined/tested/ if you see "This job is not a member of the latest evaluation of its jobset. This means it was removed or had an evaluation error." then it has disappeared.

If the reason it broke is not obvious, pick any failing build for the most recent evaluation of that jobset and click Actions -> Reproduce Locally and download the script.

Edit the script from:

args+=("$nixExprInputDir/nixos/release-combined.nix" -A 'the-job-you-picked')

to:

args+=("$nixExprInputDir/nixos/release-combined.nix" -A 'tested')

and run it with --dir scratch so it does not have to perform a fresh clone each test run:

./reproduce.sh --dir ./scratch --show-trace

with any luck, you'll get a trace and a failure you can use to bisect.

---

SSH to the bastion to fetch logs:

journalctl -eu update-nixos-unstable.service

---

Bastion's Disk is Full

/scratch partition

It is likely /scratch/hydra-mirror/nixos-files.sqlite. If this file is very large (20+G) feel free to delete it. The update jobs will automatically recreate it.