Add recurseOnHydra function

[matthewbauer]

Motivation for this change

This has similar behavior to recurseIntoAttrs. The main difference is
that nix-env -qa will not find the attributes. These attribute sets will be built
on Hydra but not listed on nix-env -qa. Most of the time this is what
is actually wanted. On the downside they become less discoverable
in nix-env -qa output.

This will significantly improve times to run ‘nix-env -qa’.

$ time nix-env -qa -f before >/dev/null
8.319 secs

$ time nix-env -qa -f after >/dev/null
3.805 secs

[dezgeg]

But that also means those packages are all undiscoverable now?

[matthewbauer]

But that also means those packages are all undiscoverable now?

Yes but you can always do nix-env -qa -f . -A pythonDocs if you know the attribute name. There's definitely a tradeoff. If this were accepted we would probably want to add more things to package.json like was done here: NixOS/nixos-homepage#201

[LnL7]

I'm not sure if all this belongs in all-packages or release.nix.

/cc @grahamc Does this impact the ofborg evaluation checker?

[LnL7]

does release.nix recurse these? they are referenced by the constituent jobsets so this would cause evaluation errors if that's not the case

[matthewbauer]

Sorry my bad

[LnL7]

it definitively shouldn't be recursed for nix-env tho, nobody is going to want to install cc-wrapper-test. 😄

[matthewbauer]

Yeah I kind of like just putting it in release.nix now that I think about it.

I think there is definitely a case to be made for including these into nix-env -qa - I just thought this would be worth proposing after seeing the dramatic savings in nix-env -qa.

[LnL7]

We already hide some stuff and 8s is definitively too slow IMHO. What about adding something like a config.recurseLanguages allowing users to enable querying for language specific package sets if they prefer that behaviour and don't mind the performance.

[grahamc]

Unfortunately this has broken evaluation, it seems to never terminate.
With this PR, running the following command with the file at https://github.com/NixOS/ofborg/blob/released/ofborg/src/outpaths.nix fails to ever finish evaluating, maybe check it out?

NIX_PATH=nixpkgs=. nix-env --query --available --no-name --attr-path --out-path --show-trace --option restrict-eval true --option build-timeout 3600 --argstr system x86_64-linux -f .gc-of-borg-outpaths.nix --arg checkMeta false

[grahamc]

Note: a second commit in this series also has been evaluting "forever" due to the same problem:

[grahamc]

My concern (beyond it breaking nix-env / ofborg) is now it isn't an easy list of packages which are hidden (oh you want haskell? here. oh, you want r? here.), but a massive list of packages. We haven't suitably solved the discoverability problem with the easy to remember list, and I think we should try to solve that before expanding the list of secret, hidden, hard to find, hard for new users, package list.

[FRidh]

Most of the time this is what is actually wanted.

Sometimes, perhaps, but in case of the package sets that's not true; users want to be able to find packages.

This will significantly improve times to run ‘nix-env -qa’

Search times for users are improved with the nix search command which uses a cache.

But that also means those packages are all undiscoverable now?

Correct

[grahamc]

11:59 does "all of nixpkgs" for ofborg include haskellPackages?

11:59 That's why I'm wondering if we could split it up
11:59 Like, everything sans haskellPackages, then just haskellPackages

an example of how the current situation is very easy to remember

[matthewbauer]

Closing for now. I think a better approach is to slowly move recurseIntoAttrs logic into an overlay. This kind of information just kind of gets lost in all-packages.nix.