Support for NixOS tests

[cleeyv]

NixOS tests are a powerful tool for writing automated VM-based integration tests of complex and high-level software functionality. It would be great to have support for them in Ngipkgs. One of the main questions is where they would run. On nixpkgs they run on hydra so hopefully we can do the same on hydra.ngi0.nixos.org.

Some links to relevant documentation:
https://nixos.org/manual/nixos/stable/index.html#sec-nixos-tests
https://nix.dev/tutorials/nixos/integration-testing-using-virtual-machines

[lorenzleutgeb]

We've started using NixOS tests for pretalx. It's still work in progress...

ngipkgs/flake.nix

Lines 63 to 67 in 66a4995

	test-pretalx =
	pkgs.nixosTest
	(import ./tests/pretalx/pretalx.nix {
	inherit ngipkgsModule;
	});

We put the test itself under tests/pretalx/.
https://github.com/ngi-nix/ngipkgs/blob/66a4995ea708a7fb92f580c790d0b0f18c9ea523/tests/pretalx/pretalx.nix

Then, nix flake check will execute the test. For just executing the test individually, we use nix build -L .#checks.x86_64-linux.test-pretalx and to debug the VM nix build -L .#checks.x86_64-linux.test-pretalx.driverInteractive && ./result/bin/nixos-test-driver.

Of course, as we accumulate more tests, we might switch to a pattern analogous to configurations and packages, with something like tests/all-tests.nix.

Just noting this here, in case someone else works in that direction...

[cleeyv]

Thanks for sharing this, @lorenzleutgeb ! There was also recently discussion of this on matrix recently where you shared these additional details that I think are very helpful for anyone using this for local interactive testing of a web service:

This will drop you into a shell, where you can enter start_all(). Then you'll get a QEMU VM. You can log in as root without a password. After some booting (it's quite slow), you should first get HTTP 502 for curl localhost (as pretalx is still initializing), and once pretalx is up you'll get some Website at curl localhost. To use your browser, in the "driver shell", run server.forward_port(8000, 80) or so (first argument is your host, second argument is the testing VM port).

[cleeyv]

In their major PR #38 yesterday, @ngi-nix/algae has improved considerably the support in ngipkgs for NixOS tests! There is now a tests directory containing an all-tests.nix, with a few different examples in it already.

After discussion with @ngi-nix/algae, in the end we decided not to include the nixosTests in checks because they take so long and it would slow down CI significantly.

Also, I learned yesterday that there is a README with documentation for pretalx, including the kind of testing docs I had referenced earlier: https://github.com/ngi-nix/ngipkgs/tree/main/pkgs/pretalx#testing

This recent work covers a lot of what I had in mind for integration of NixOS tests into ngipkgs, however there is still the major CI piece of getting these tests running on a hydra instance. My next step will be evaluating what it might entail to get this working for ngipkgs on https://hydra.ngi0.nixos.org.

[cleeyv]

The remaining integration of nixos-tests into the hydra CI has been completed with commit 1153203 which adds the nixosTests output tohydraJobs in the flake. Hydra reports that the test is building successfully, https://hydra.ngi0.nixos.org/eval/1274 and the logs show that the test is running: https://hydra.ngi0.nixos.org/build/3029/nixlog/385

[lorenzleutgeb]

I'd like to remind ourselves, that the output nixosTests was chosen relatively ad hoc by @ngi-nix/algae and should be reconsidered. If possible, I'd like to use passthrus, just like in nixpkgs. I refactored the way tests are defined and wired in #52 and am happy to have your comments.