Testing functions with "passthru" tests.

[roberth]

Functions producing derivations can have their own tests attribute, just like a derivation can have a tests attribute using mkDerivation's passthru.tests. While a plain lambda can't have attributes, a callable attrset with __functor, as produced by callPackage for example, can.

While some functions are already tested indirectly by having calls in the package tree (mkDerivation has quite some coverage this way...), some functions have surprisingly little. Oddly, the fetchers are among those, because they often don't even have to run, as their outputs are already available.

With NixOS/nixpkgs#136022 we can use a trick to run fetchers whenever their implementations change, which is great for testing. However, ofborg can't run them yet. It uses nix-build (or a similar command) to run the tests, but it tries to invoke the __functor function instead of going directly for the attribute. nix build does not suffer from this problem, but alternatively, we can introduce a packageTests attribute in Nixpkgs as an alias for the expected behavior. I've elided it from that PR, but it could be reintroduced for ofborg specifically. See NixOS/nixpkgs#136022 (comment)

So it seems that all ofborg needs to do to support running these tests is to either
a. switch to nix build, or
b. switch from *.tests (eqv. *.passthru.tests if it insists?) to packageTests.*

[ShamrockLee]

AFAIK, we currently put NixOS tests and (other) package tests of functions under pkgs.tests, such as tests.trivial-builders.<name>.

It would be much more intuitive to move tests specific to a function under <function>.passthru.tests.

[ShamrockLee]

Update:

As callables are plain attribute sets that won't update themselves with their passthru attributes (the way packages returned by stdenv.mkDerivation do), <function>.tests is more sensible.