Optional groups of dependencies

[jbednar]

I'm submitting a...

• feature request

Issues #793, #1696, and #3299 from different users all request that conda support optional dependencies. One interpretation of this request was implemented in conda 4.4, namely constraints for optional dependencies: packages listed under run_constrained are constrained to the specified version, but only if they are installed explicitly by the user.

The other interpretation of optional dependencies, i.e. like pip's groups of optional requirements that are specified using brackets like matplotlib[qt], remains unsupported. @pelson, @astrofrog, and others have provided arguments why it should be supported, but the response so far has been that conda metapackages cover this type of functionality. I'd like to challenge that claim.

It's true that a conda metapackage is in some sense an equivalent way to handle the functionality of pip's bracketed options, but in practice there are major different implications for the different approaches taken by pip and conda.

In pip, it's simple for a package maintainer to provide a long list of sets of options, such as the various optional backends available for matplotlib. No matter how many such options are provided, the result is only a single pip package to prepare and track for a given release, and only a single package that shows up in a listing of the environment. Thus in pip, listing such a set of relevant optional packages is very cheap for the developer, and also for the user, as it doesn't result in anything in the user's environment for them to track.

In conda, having to create a metapackage and track it in repositories, environments, licensing reviews, and so on very strongly discourages making such groups of optional dependencies. Sure, it can be done, but it's no longer trivial; it has substantial maintenance costs for the project maintainer and management costs for the user that the pip approach does not.

Perhaps more importantly for me as a package maintainer, this difference between conda and pip makes it much more difficult to maintain packages for both pip and conda, and creates ambiguities about what a package name means in any installation that combines both pip and conda. Specifically, pip's approach encourages the main package for a given library to be lean, without optional extras, while conda making it difficult to specify optional groups of things to install encourages packages to be fat, with any optional packages included as dependencies, even if they aren't strictly required for a minimal installation.

So e.g. we end up with some package "pack" on pip, with "pack[recommended]" installing the most typical dependencies, "pack[all]" installing lots of extra things, and "pack" installing only the bare minimum. On conda we maybe get "pack" installing the most typical dependencies, sometimes "pack-core" (a dependency of "pack") installing the bare minimum, and "pack-extras" installing the extra stuff.

So far the only problem is that we had to make and track three packages (and all their arch variants) where one would do. But more seriously, if a user installs "otherpack", which depends on "pack", she's now going to get a different set of dependencies if "otherpack" is installed with pip than if it's installed with conda (due to "pack" having different dependencies in the two cases). This leads to all sorts of trouble!

If conda would support a similar notation for installable groups of optional packages, project maintainers could safely support similar options on pip and conda without these constraints always being in tension, and users would see more predictable behavior.

[jni]

Thanks @jbednar for eloquently arguing for this feature. I just want to add scikit-image to the list of projects where we would like this.

[hmaarrfk]

I'm just going to share what I learned these last few weeks:

• conda outputs probably do what you want in terms of having a single recipe. The cost to pay will be approximately equal to an "additional tab" and the choice of a naming convention. I'll try to put an example up for scikit-image in the next few weeks. For example, you might have the following outputs section

outputs:
  - name: scikit-image
    # Most of your original recipe would go here.
  - name: scikit-image.test
    requirements:
       run:
       - {{ pin_subpackage('scikit-image', exact=True) }}
       - pytest
  - name: scikit-image.docs
    requirements:
       run:
       - {{ pin_subpackage('scikit-image', exact=True) }}
       - numpydoc
       - sphinx
       - something else
  - name: scikit-image.all
    requirements:
       run:
       - {{ pin_subpackage('scikit-image', exact=True) }}
       - {{ pin_subpackage('scikit-image.test', exact=True) }}
       - {{ pin_subpackage('scikit-image.doc', exact=True) }}

• People are understandably scared to break dependencies, so making pack, mean a minimum set of requirements is unlikely to fly (although I would wholly support it). That said, people seemed quite open to creating split packages that would pull in additional dependencies.

That said, we would probably need a good naming convention. The square brackets of pip are kinda nice, but maybe something like a . might do for conda? Dashes are being proposed for things like build variants which might mean something compiled with different options and be mutually exclusive as opposed to additional dependencies which are additive. I would much rather have dashes indicate the use of more recent instruction sets 🐳

@jakirkham @msarahan @jjhelmus any ideas on a good naming convention for optional dependencies that won't break things.

[Tagar]

@hmaarrfk

I'll try to put an example up for scikit-image in the next few weeks.

Let me know if you have an example handy how to use conda outputs here?

[hmaarrfk]

See the opencv package

[Tagar]

@hmaarrfk
something like this -
https://github.com/skvark/opencv-python/blob/45a77d6c6e24519de68e08f71742f078526623c4/setup.py#L64-L67
?
opencv has two packages ['cv2', 'cv2.data']
How do you reference this from conda?

[hmaarrfk]

https://github.com/conda-forge/opencv-feedstock

[h-vetinari]

Is there any plan to support anything similar to pip install package[options]?

See this boa proposal (which broadly proposes to follow pip syntax) and this conda issue (which proposes a new syntax that would be compatible with optional packages).

[leofang]

xref: #2984

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!

[jbednar]

This issue is old, but still relevant!

[wolfv]

Yes, let's finally fix it! :)