contrib: add direct tests for agenix

[ryantm]

These tests are MUCH faster than the NixOS tests.

[n8henrie]

Looking through, it looks like you've migrated the tests to an installcheck on the derivation, which means they would run when the CLI was instantiated (?) / built / better word. Would we then remove the nix flake check process and just install the binary in CI -- if it installs, that means the tests passed?

EDIT: Premature submission. More coming.

[n8henrie]

Compared to the existing approach, I think this is fine. Speeding things up test cycles is nice for local development, though I don't much care if things take a minute in CI. Potential downsides include that running in a VM ensures a reproducible environment, both locally and in CI, whereas it looks like these may (?) be impure -- please help me learn if I'm wrong here.

Compared to #159, there are a few things I like about the python module approach:

• importing from a python module lets me use a whole host of tools that I'm accustomed to when writing tests -- (limited) type checking, syntax highlighting, autocomplete
• while it will still depend on mutable state, using a python module should make is easier to break out individual tests and make them more functional, if desired -- for example isolating a failure to a specific test in a live environment would be just tester = AgenixTester(vars); tester.run_specific_test()
• I presume that if we ever wanted to split out tests into separate checks checks.${system}.rekeying_words, the python module would make this easier. Might not be much desired to ever do this though!
• I believe we could add some doctests to the tests in the python module, making a simple way to "test the test", if desired.

All that said -- the python module approach does add some complexity, take longer, and many of the above points are admittedly premature optimization! It looks like either approach should make it easier and more reliable to test the CLI as $USER, which I think is the first priority.

If the python module approach feels too far from the beaten path, too complex / brittle, or it's just not your preference, I won't take offense!

[n8henrie]

Also, I meant to ask -- other than speed and more directly testing user installs, are there other benefits that you envision from testing this way that I might have missed?

[ryantm]

Mainly I'm concerned about the speed of the NixOS tests.

This typically runs in the nix build sandbox, so it doesn't end up messing with the user's environment.

Another totally different path we could go is to merge our efforts with https://github.com/yaxitech/ragenix and stop coding bash, and use Rust's testing tools.

[n8henrie]

Another totally different path we could go is to merge our efforts with https://github.com/yaxitech/ragenix and stop coding bash, and use Rust's testing tools.

I'm a big rust fanboy (though not terribly good at it yet, it's probably what I'm most focusing on learning at this point).

What about #23 ?

[ryantm]

My concern with #23 is that I don't want to force everyone to change their secrets.nix file to a TOML file.