Optional Dependencies for conda v2

[travishathaway]

This is a second attempt to draft a CEP for optional dependencies for use by conda compatible package managers (e.g. conda and mamba).

Please see the CEP itself for more information.

[h-vetinari]

This is amazing to see, thanks so much for picking this up!

Just one question/comment about the syntax.

I haven't yet thought through the implications of the virtual packages in detail, but it looks like a very elegant approach!

[h-vetinari]

How does the parsing here work? It looks like it would break the *args,**kwargs pattern proposed in conda/conda#11053 (which you're referencing already).

I'm also wondering if we really need two ways to do this? package_name[optional_a,optional_b] looks perfectly alright to me, and matches with what pip is doing.

While it's out of scope for this CEP, I'd really like to keep the door open to eventually extend the syntax to be able to use (exposition only, all kwargs need a proper design)

conda install package_name[optional_a,optional_b,cuda=True,license=no_gpl,cpu=avx512,...]

It seems to me that if we add the extras= way to specify optional dependencies, we don't gain much and risk making it much harder to eventually solve that problem.

[nicornk]

+1 for following the pip syntax.

[wolfv]

Can packages in the repodata also depend on other packages + optional dependencies?

[jaimergp]

Points to take into account here:

• How will this be specified in the conda-build recipes
• How will this be exposed in repodata.json
• How will this be reflected in the info/ metadata folder inside the package

[jaimergp]

I don't know how to feel about the new syntax for an arguably odd case. Why do we need it to begin with? If one wants to remove the extras, they could use the same spec à la package_name[optional_a]? And if you just want to remove the extras, they can simply spell them out.

[jonas-w]

Agreed, but I think there are good use cases for something like the “meta-packages” to exist:

1. Install package_name[optional_deps]
2. package_name gets an update and the extra “optional_deps” adds another dependency
3. You upgrade the (automatically, with the other packages) package_name@optional_deps package and also get the new dependency that was added with the upgrade.

Otherwise, if there are no “virtual meta-packages” (or anything that keeps track of the extras you installed) the user wouldn't get the new dependency.

Also, another thing that I would love to have, but which would differ from the way pip does it (at least to my knowledge), is to not stop at optional dependencies, but go further with optional modules/files.
Let's say my project has the following structure:

src/package/...
src/package/core/...
src/package/feature_a/...
src/package/feature_b/...

And in the meta.yaml I could configure it like this (just an example):

# if files get specifically included in an extra,
# they would be excluded from the "core" package
extras:
    feature_a:
        include_files: ["feature_a/"]
        dependencies: ["optional1",..]
    feature_b:
        include_files: ["feature_b/"]
        dependencies: ["optional2",...]
# maybe also something like feature_c
    feature_c:
        include_extras: ["feature_a","feature_b"]
# optionally also include further files/dependencies etc.

Then conda install package, would only include /core.
And conda install package[feature_a] would include /core and /feature_a and feature_a's dependencies.
And conda install package[feature_c] would include everything.

[mbargull]

This CEP was mentioned by @chenghlee in a discussion at https://matrix.to/#/!ugIkjExjZULQyVDDNz:gitter.im/$vMqrBnqcqeOfqNq27A4epLJ_xDuIjXiu-59XN_KeBbk?via=gitter.im&via=matrix.org

I didn't yet read the text or discussion much, but just wanted to copy my points from the chat:

There are multiple steps; main points (list not necessarily exhaustive):

1. Define the actual semantics on what "optional"/"extra" would mean (ask different people and you'd get different answers...).
   I.e., figure out how this fits into the general dependency resolution and what defaults should be (and look out for typical corner cases, e.g., differences between env create/install/update.)
2. Define the data storage/communication layer, i.e., "repodata CEP update" and stuff.
3. Define the user interface for this; e.g., (amongst other things) how would a user say "install foo with bar".
   • This cannot be install foo[bar] since we have [...] already reserved for "match spec".
   • Would have to be foo[extra=bar]/foo[extras=[bar]] or the like if this would use he match spec syntax; could be something entirely different, too, though.

[xhochy]

I don't understand why we need this feature. We already use in several cases multi-output recipes for optional dependencies where a package[feature] on the pip side is translated to a package-feature package. For the sake of simplicity, I would prefer to avoid to define an additional way of doing it and keep the current way of doing it.

[h-vetinari]

We already use in several cases multi-output recipes for optional dependencies where a package[feature] on the pip side is translated to a package-feature package.

Yes, but it's a crutch, and a persistent paper cut that people run into and contributes substantially to friction and user-frustration for requirement-incompatibilities between PyPI and conda. Even a bunch of conda-forge recipes get this wrong (and don't even get a warning that this doesn't do what's expected!). It should IMO absolutely be fixed just based on those two points alone.

If we zoom out a little bit further, this "string-based work arounds instead of sufficiently expressive syntax/metadata" is a constant source of pain (the other huge pain point on this being the completely overloaded build strings), which I mention here because IMO both should be solved with the same syntax.

conda install package[feature,build_number=1,cuda=True]

would be orders of magnitude more user-friendly than the following jumble of symbols

conda install package-feature=*=cuda*_1

and that's even before getting into conda/conda#11442, conda/conda-build#4498, conda/conda-libmamba-solver#446, etc.

[xhochy]

My main concern is that I want to avoid is adding an attribute to repodata.json as this will take longer to propagate to all implementations.

It would be nice if the problem could be solved by mapping the square bracket syntax to plain packages, i.e. make this a client-side convention.

One example could be to map jaxlib[build_number=1,cuda,docs] to jaxlib=*=*_1 jaxlib-cuda jaxlib-docs.