Dealing with proliferation of package variants vs. ease of selection

[h-vetinari]

Crossposting from conda/conda#11053 because it affects conda-forge in many ways.

More and more packages come in various different flavours, independently of relatively fixed things like the python version. A non-exhaustive list:

• cpu vs cuda (vs. eventually ROCm or Metal)?
• blas-flavour
• mpi-flavour
• openmp-enabled or not
• possibly for CPU arches
• builds for different licenses (links omitted)

I opened that issue on conda to find out if there might be a better solution than tacking on more and more things onto buildstrings. 😅

[jakirkham]

Yes virtual packages. Particularly custom ones for extensibility ( conda/conda#10131 ). They can detect system features, which either allow or disallow particular variants.

[h-vetinari]

Virtual packages are not the answer to everything though... A virtual package IMO makes sense for something (effectively) immutable like OSX, CPU Arch, GPU version, but not for things like BLAS, MPI, OpenMP, etc... And even with virtual packages, I might want to override them and install the CPU version despite having a GPU, or install the SSE version despite being AVX2-capable.

[h-vetinari]

Though looking closer at conda/conda#10131, maybe I'm overlooking how the plugin-approach might overcome these issues/limitations? If so, could you elaborate?

[jakirkham]

Mainly thinking about auto-detection and constraint of OS and hardware features. With the existing virtual packages there are already ways to override these. Presumably that could be done for new ones

The other things (BLAS, MPI, etc.) are up to the user to decide. What alternative would you propose?

[h-vetinari]

The other things (BLAS, MPI, etc.) are up to the user to decide. What alternative would you propose?

My point was exactly about exposing these variants to the user, and ideally in a more user-friendly way than pattern-matching build strings. I have nothing against package variants, I think they're a great asset of conda-forge. I just wish we'd have a better UI for it.

[jakirkham]

@ngam, this might be best as a new issue on the webpage (this) repo (though it does seem like an interesting discussion to have)

[jakirkham]

It's diving deeper into CUDA functionality. We may want to invite others familiar with that use case to discuss further. While generally within this issue, it is a bit of a subcase. So we may not want to get too into the weeds on that case in this issue (and so move it to another one)

To be clear am happy to see that discussion move forward, just trying to keep things organized

[jakirkham]

FWIW the virtual package plugin architecture is being worked on in PR ( conda/conda#11854 ). Once complete virtual packages could be created independently and used as dependencies in metapackages to cutdown on the configuration users themselves would have to do.