Modularize Tests

[erikarvstedt]

This improves debugging and experimenting by making it easy to compose fine-grained
scenarios that have specific tests and features enabled.

Scenarios can now be defined like this:

# A node with two services. The corresponding tests are automatically enabled.
simple = {
  services.electrs.enable = true;
  services.clightning.enable = true;
};

# Full netns scenario, but with only two tests enabled
netnsTwoTests = {
   imports = [ scenarios.netns ];
   tests = mkForce {
     btcpayserver = true;
     netns-isolation = true;
   };
 };

Custom scenarios can conveniently be added by setting the env variable scenarioOverridesFile.

For undefined scenarios the tests are run with an adhoc scenario where services.<scenario> is enabled:

run-tests.sh -s clightning
run-tests.sh -s electrs

Also, the test output now includes the subtest name and duration and subtests use nested logging, which improves the html log.

Containers

Scenarios can now be run in containers (for now, without the python test suite) for super fast
debugging and experimenting.
Like in the examples, creating NixOS containers needs root permissions.

## Run single command in container, delete container afterwards
run-tests.sh -s bitcoind container --run c systemctl status bitcoind

## Start shell session in container, delete container afterwards
run-tests.sh -s bitcoind container
# Run command in container
c systemctl status bitcoind
# Run via SSH
cssh systemctl status bitcoind
# enter container
c
# enter via SSH
cssh

Review helper

Use the following to audit 4548576 (extra-container: pre-release -> 0.4)

tmp=$(mktemp -d -p /tmp)
mkdir -p $tmp/{old,new}
fetchRelease() { curl -SL https://github.com/erikarvstedt/extra-container/archive/$1.tar.gz | tar xz --strip-components=1 -C $2;}
fetchRelease 6cced2c26212cc1c8cc7cac3547660642eb87e71 $tmp/old
fetchRelease 0.4 $tmp/new
git diff --no-index $tmp/{old,new}
rm -rf $tmp

[erikarvstedt]

@jonasnick, @nixbitcoin do you use a systemd-based distro other than NixOS for developing? I'll try adding extra-container support for these platforms then. (Or, preferably, I'd help you switch distros. 😸)

[jonasnick]

Looks interesting! I'm running ArchLinux which is systemd based. So far qemu was sufficient for me but I wouldn't say no to testing with fast containers.

[jonasnick]

This is excellent! Concept ACK

I don't see why the tests suddenly don't need to be executed in the corresponding namespace anymore, or was that always the case?

@nixbitcoin since you wrote a large part of the existing tests, would you have time to look at this as well?

[jonasnick]

Perhaps mention that this is NixOS only?

[erikarvstedt]

I'll have a shot today at making extra-container work on all systemd-based distros.

[nixbitcoin]

This would also be interesting for me, since I'm running Debian and couldn't test the container yet. Planning to switch to NixOS this year.

[erikarvstedt]

or was that always the case?

Yes.

[erikarvstedt]

@nixbitcoin it could be the case that the kernel in Debian 10.6 (latest release) is too old to support seccomp system call filters in container services, leading to failures when starting nix-bitcoin services.
I couldn't find a working, reasonably recent Debian Vagrant image, so could you check if this works?

run-tests.sh -s bitcoind container --run c systemctl status bitcoind

[nixbitcoin]

I couldn't find a working, reasonably recent Debian Vagrant image, so could you check if this works?

I'm getting /tmp/extra-container/bin/extra-container: line 21: errEcho: command not found

[erikarvstedt]

@nixbitcoin, thanks, I'll fix this in extra-container.
Probably the type -P machinectl check is failing on your system.
Fix this with sudo apt install systemd-container.

[nixbitcoin]

./run-tests.sh -s bitcoind container --run c systemctl status bitcoind succeeded on Debian 10.6. I had to reinstall nix as a multi-user install.

[erikarvstedt]

Awesome, thanks for testing. I've added a multi-user installation check to extra-container.

Use this to audit the extra-container update:

tmp=$(mktemp -d -p /tmp)
mkdir -p $tmp/{old,new}
fetchAndCheck() {
    local url=https://github.com/erikarvstedt/extra-container/archive/$1.tar.gz
    curl -sSL $url | tar xz --strip-components=1 -C $2;
    echo "sha256 for $1 is $(nix hash-path --base32 "$2")"
}
# old: 0lqdrd8pvsb5jf7v124n8r5dv9dikkqbzy8082wsdrgymn8s577b (sha256)
# new: 0gdy2dpqrdv7f4kyqz88j34x1p2fpav04kznv41hwqq88hmzap90 (sha256)
fetchAndCheck 0.5-pre-1 $tmp/old
fetchAndCheck 0.5-pre   $tmp/new
git diff --no-index $tmp/{old,new}
rm -rf $tmp

Meanwhile, I always keep updating the unsquashed branch.

[nixbitcoin]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

ACK 1cc432a1363eaafcb200b4ad3a4e7cad989fde26
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEV3o0Un8+KoXoD+Fk3RH5rVMIs7oFAl+JqWAACgkQ3RH5rVMI
s7psRw//SijO6jLSG0PvlTocU46FVGc2daJL1ZUf2aBcGAVfxe428t9fKjFqkccB
k3WK3qgF8T+Jlw2Jg61geihSu7ghJ2UJC2AG59Sq3jHopX+golwnEJTJV8ESWajO
o5ZVeA/zYtwPnN8yCjGpCSgHvhm2tlCWf0YUcMzAuwSp3qKLnlXlaF7HsMCTIxRU
vePnbzPJFSakc2SVW7tRaaycvMGpdKy4WEohv0kH0L5nvrbK3W5vmFi812UaFrxC
gMADX0wpoJm1CdZYvTWEIN3EC11x+SXkwn9wD8CxEX+MgDtUYXtc//JghQxDLUif
Dcr8J5JyFxK9Iyqv6I/wjQUpFmk4ELRH4JPUMaDhV6Pwu4Q+0tVqinwj8kWv2E2y
CdkehkETawWMtVL/lNtIpytcp3X+CDXx8kaYOO5Ul6uFUicgPsiOMSaeup4zcmWd
6aeZr6TyQzT3OhnIkbuxoNfFxzqdmaFNe5BUSNw/9DdSxUQLay4h0fceqiBdUdyc
2NOjYtg+7vHFpnx/0essWPWO8vqIMdb/5G0Tdzu5bVj4FpW8LPncMS6pAxoCjZO6
wHCoN0W9UH966fgIAMw7+WDu1q60IFy36/CAcyh9OQ9xzZwXE/YEthpwyZhLF6Eq
kVSK5K1M/tmaR2yVnSONPen2pAV6/72vLWqFnVOTB7xx5KXE/wM=
=ekbv
-----END PGP SIGNATURE-----

[jonasnick]

ACK 1cc432a