make --search aware of extensions

[klust]

Problem description

EasyBuild installs many Python, Perl and R packages as part of either the base package or a bundle. It is possible to configure EasyBuild to define the extensions in the module file so that users can search for a particular package in an installation using module spider. Lmod will then show which module contains the package as an extension.

However, this still leaves the user with the problem of locating the EasyConfig file that should be used to install the package. E.g., a user who needs NumPy should know that it is part of SciPy-bundle, and eb --search cannot be used to search for NumPy. This often leads to users thinking a package is not yet supported by EasyBuild while in fact it is.

The current solution is to go to an easyconfig installation on the system and use grep -ri to search for the package, hoping that it will turn up in an easyconfig file.

Proposed extension

Provide a search function in the eb command that also searches for extensions and can also tell that a certain package can be installed as an extension of another package.

I assume this may be more demanding then it looks as the search can no longer be based on the name of the easyconfig file alone but all files need to be parsed, so some caching mechanism may be needed also, or we may need two search commands, a quicker one that only searches for names of packages and a slower one that also searches for extensions?

Note: It should probably no only search for extensions, but also for components of a bundle to be complete.

[ocaisa]

We do have an index, that could probably be extended to include extensions and components. The slower search capability would then involve building the index first

[boegel]

EasyBuild releases come with a pre-built index (included in the easybuild-easyconfigs source tarball we push to PyPI), so that's not a big issue imho.
It'll probably building a local search index (for example when using develop branch) a bit slower, but that's probably acceptable.

Next to making --search aware of extensions, it would be nice to also make --list-software aware of extensions, since that would imply that the list of supported software also includes extensions...
The downside would be that it would make that page significantly larger, and it's already quite slow to load.

[ocaisa]

This could also come in useful for creating a script that autotranslates bioconda conda recipes (like https://github.com/bioconda/bioconda-recipes/blob/master/recipes/agouti/meta.yaml) to easyconfigs. We could use this search capability to find deps and pin versions.