Proposal: pins/run_constrained support

[nicoddemus]

Problem

To ensure all projects use the same "blessed" versions, we use a strategy where we include other devenv.yml files which exist solely to specify package versions, for example:

name: app
includes:
   - {{ eden }}/environment.devenv.yml
   - {{ eden }}/pytest.devenv.yml
dependencies:
   - cogapp ==0.3

With the other files being:

# eden/environment.devenv.yml
name: eden
dependencies:
  - artifacts_cache ==3.0.0
  - attrs ==21.2.0
  - black ==21.12b0
environment:
  - PYTHONPATH: {{ root }}/source/python

# eden/pytest.devenv.yml
dependencies:
  - pytest ==6.2.5
  - pytest-mock ==3.0.0
  - pytest-lazy-fixture ==0.6.0

The purpose here is to make sure we are using the same versions, without putting all dependencies in eden/environment.devenv.yml, as that would bloat all environments that include it.

This works, but it has some problems:

• Versions are scattered in many devenv.yml files, which makes it difficult to update versions as often there are conflicts, and we need to hunt down the offending pins.
• Often, for brevity and avoiding having an explosion of devenv.yml files, we put a lot of packages in the same file because they are often related, but that's not always true or convenient. For example, when we include pytest.devenv.yml above, we will get pytest-lazy-fixture whether we use it or not.

Proposal

We add a new section in the devenv.yml files, named pins. This contains a number of packages and pins (mandatory), which will be used when processing the dependencies of the chain of devenv.yml files. Any dependency in devenv files which has a pin in the pins section, will be pinned accordingly in the final environment.yml file generated.

For example, we can have pins directly into eden/environment.devenv.yml, which is included by all other projects:

name: eden
pins:
  - artifacts_cache ==3.0.0
  - attrs ==21.2.0
  - black ==21.12b0
  - cogapp ==0.3
  - pytest ==6.2.5
  - pytest-mock ==3.0.0
  - pytest-lazy-fixture ==0.6.0
dependencies:
  - artifacts_cache
  - attrs
  - black

pins doesn't mean that dependency will be included, but if any of the dependencies include a package in pins which is not itself pinned, then it will be pinned accordingly.

name: app
includes:
   - {{ eden }}/environment.devenv.yml   
dependencies:
   - cogapp
   - pytest

Having a pin file, all dependencies sections in devenv.yml files can now leave things unpinned, with the pin being managed by the pins section.

Notes

Dependencies with pins

If a dependencies section specifies a pinned package, then the generated file will contain both pins:

name: eden
pins:
  - artifacts_cache ==3.0.0

name: app
includes:
   - {{ eden }}/environment.devenv.yml   
dependencies:
   - artifacts_cache >=2.4.0

In this example the generated environment.yml will contain the entry artifacts_cache: >=2.4.0, ==3.0.0, which conda will be able to solve. If they conflict for, conda will fail as expected.

Multiple pins

It is possible to have multiple pins sections in a chain of devenv.yml files, we should error out if different pin sections try to pin the same library (this from the POV of conda-devenv, we probably should have a single pins section in one file).

Usage in eden

We probably will have a separate pins.devenv.yml file, to avoid polluting eden/environment.devenv:

# eden/pins.devenv.yml
pins:
  - artifacts_cache ==3.0.0
  - attrs ==21.2.0
  - black ==21.12b0
  - cogapp ==0.3
  - pytest ==6.2.5
  - pytest-mock ==3.0.0
  - pytest-lazy-fixture ==0.6.0

name: eden
includes:
  - {{ root }}/pins.devenv.yml
dependencies:
  - artifacts_cache
  - attrs
  - black

What about "lock files"

Locking is a different use case/concern, and does not conflict with this proposal.

(note: much of the ideas and solutions here were result from discussions with @tadeu)

[prusse-martin]

As I understand this will not affect transitive dependencies.
We could document this behavior but I think it a bit strange.
Say we have:

name: foo
pins:
  - pytest ==6.2.5
  - pytest-mock ==3.0.0
  - pytest-lazy-fixture ==0.6.0
dependencies:
  - pytest-lazy-fixture

As I see it will respect pytest-lazy-fixture ==0.6.0 but not pytest ==6.2.5 (and pytest is a dependency of pytest-lazy-fixture).

[nicoddemus]

As I understand this will not affect transitive dependencies.

Good point.

From your example, with the original proposal, it would generate:

# environment.yml
name: foo
dependencies:
  - pytest-lazy-fixture ==0.6.0

And indeed pytest would be unpinned.

We should resolve/find that pytest-lazy-fixture depends on pytest, and output pytest ==6.2.5 to the environment.yml as well (as well all the other transitive dependencies).

Not sure how to tackle this in an efficient manner though. 🤔

[prusse-martin]

pins = ...  # {package: version_spec, ...}
dependencies = ...  # {package, [version_spec, version_spec, ...], ...}
updated_pins_in_deps =True
while updated_pins_in_deps:
  updated_pins_in_deps = False
  new_dependencies = conda_solve(dependencies)
  transitive_deps = set(new_dependencies) - set(dependencies)
  for dep in transitive_deps:
    if dep in pins:
      updated_pins_in_deps = True
      dependencies[dep].append(pins.pop(dep))

[nicoddemus]

Unless I'm missing something, we need to do a full blown solve of the environment, correct?

Because the same package might have different dependencies depending on its version (say pytest 6 depends on type_extensions, pytest 7 does not), so we need to account for that.

If the only solution is in that direction, then probably best to rethink the whole solution with lock files in mind (or even scrap this idea altogether in favor of using full lock files).

[prusse-martin]

... we need to do a full blown solve of the environment, correct?

I think so. That or fetching the repo data and parsing it, that could be more performant but we will need to handle channel priority and that kind of conda configurations.

[nicoddemus]

Unless I'm mistaken, it is not just a matter of parsing versions, we do need to solve the full environment in order to obtain a reliable list of versions and dependencies.

For example, if I depend on package X, I need to find its pinned version, say 1.1, and from that, I need to be able to fully solve its dependencies for version 1.1, which in turn brings a whole world of possible dependencies and their versions, which in turn need to be fully solved again.

Just to make it clear that we are no longer in the realm of "just list some packages and pin them according to a list of pins", we do need to solve the full environment in order to produce a reliable list of dependencies.

For that reason I think we need to rethink this in terms of lock files, possibly going straight to a lock file solution, instead of this which would be an interim solution.

[prusse-martin]

You are right, do get it done properly we need the actual solve.

[nicoddemus]

Thanks for the inputs @prusse-martin, I'll close this then. 👍