Pinning subset requirements

[xhochy]

When developing, we will install much more dependencies at runtime. We still would like to pin the runtime and the development/CI dependencies to the exact same versions. This should result in two lockfiles where one is a subset of the other.

Given the following environment.yml:

name: nyc-taxi-fare-prediction-deployment-example
channels:
  - conda-forge
  - nodefaults
dependencies:
  - click
  - jupyterlab  # [dev]

and the following command: conda-lock -f environment.yml --subset "core|dev=0"

I would expect two files conda-linux-64.lock and conda-linux-64-core.lock where in the latter jupyterlab and its dependencies are omitted but all other packages are the same version.

Does this sound like a reasonable proposal? If so, I would start working on this.

[mariusvniekerk]

so the way we can do this atm is by passing multiple files using the compound specifications

conda-lock -f base.yml -f specific.yml -p linux-64 --filename-format "specific-{platform}.lock"

[mariusvniekerk]

in the general case the additional dependency can invalidate the entire solve though so not sure we can just have these be atomic

[hadim]

Just a small comment to say that such a feature would be awesome to have on the conda and mamba side also when creating an env. See conda/conda#10398 for something a bit different.

I really like this idea of selectors to specify subset env.

[maresb]

Rather than merging the specifications successively, I'd expect that a more accurate approximation would be:

1. First install all dependencies
2. Export as conda-linux-64.lock
3. Uninstall dev dependencies
4. Export as conda-linux-64-core.lock

Is there any progress on this? Anything I could do to help?

[xhochy]

The most pressing thing where help is needed is to find a way how to specify this in a way that is compatible/accepted by the tools in the ecosystem (conda, mamba, conda-lock).

We don't need to "install" anything actually. The solver output of conda / mamba for the larger set of dependencies should be sufficient to already pin the subsets. You get a tree and you can leave out all branches that don't end up in one of the remaining directly listed dependencies.

[maresb]

We don't need to "install" anything actually.

Right, I actually meant "install" in the sense of "ask the conda/mamba solver which packages would be required to install"

You get a tree and you can leave out all branches that don't end up in one of the remaining directly listed dependencies.

Is it really a tree? I assume nodes would be packages, but then what would be the unique parent function? In any case it looks like I'd need to invest some time into understanding the data structures.

@xhochy, did you ever start working on this? Any partial progress to share?

[xhochy]

@xhochy, did you ever start working on this? Any partial progress to share?

No, the important starting point for me would be to have consensus on the specification. Implementing itself is (from my perspective) a less complicated issue.

[maresb]

By "specification" do you mean a way to indicate dev dependencies such as using # [dev] as a label? Or do you mean something else?

[xhochy]

By "specification" do you mean a way to indicate dev dependencies such as using # [dev] as a label?

Yes.

[maresb]

I don't understand exactly what you mean by --subset "core|dev=0", but I'd trust your judgement, and it sounds like you have something clever in mind. I really like your proposal, and so do 7 others according to the 👍's. That looks quite positive to me.

What sort of consensus are you looking for exactly? I wonder if we could somehow push the issue?

[xhochy]

I don't understand exactly what you mean by --subset "core|dev=0"

This is the one thing I really don't like about my suggestion. It is hard to specify the subset on the CLI in an understandable manner. --subset "core|dev=0" boils down to:

• --subset: In addition to a full lock, also pin for this sub selection of packages
• core: Name that selection core
• dev=0: The subset is selected by setting the environment marker dev to false.

Also, I had a discussion with @mariusvniekerk outside this issue where he expressed a certain dislike regarding selectors. @mariusvniekerk Can you maybe expand that here a bit?

[mariusvniekerk]

My general problem with selectors is that they are an abnormal way to add logic into yaml.

Additionally conda lock supports reading deps from a few more sources that exist outside of the conda ecosystem.

I'm perfectly fine with just sticking a few more keys into the yaml (and the pyproject.toml etc) and parsing it correctly.

Maybe something like

conda-lock:
   package-subsets:
       core: [foo] 

As a means of expressing named package subsets.

[maresb]

@mariusvniekerk, to clarify, would your [foo] be a list of packages, each of which must also occur in dependencies?

[mariusvniekerk]

Yep those packages and obviously all their deps from the resulting graph

[mariusvniekerk]

Additionally a specification like this allows us to rather easily support it entirely on the command-line as well

conda-lock --package-subset core=foo,bar --package-subset test=foo,bar,pytest