Add an option to install the oldest supported versions of dependencies

[fjarri]

Imagine you're adding a requests dependency to your projects, as it's done in the tutorial. Currently this will result in "requests>=2.27.1" in the dependency list. This is most probably an overkill: it's quite likely your library will work just as well with requests at 2.20 or 2.10, or maybe even just 2.0, in which case you'd want to set that as a limit to reduce the possibility of conflicts for the users of your library. But if you set "requests>=2.0", 2.27.1 will still be installed on install (unless there are some limitations from other dependencies). For the testing purposes, it would be great to be able to resolve the dependencies as close to the minimally allowed versions as possible (I understand that it may not be possible to set all dependencies exactly at the minimum versions, so some kind of distance measure will have to be introduced). This way you could run that on CI and be sure that all your bounds actually make sense.

Another use case is when you start using a new feature of some dependency, but forget to bump the bound. Everything works on your end, but then a user with that dependency at a lower (but still allowed) version gets an error because the feature is missing.

[frostming]

If you are a project author and add requests as a dependency, how can PDM know what is the lower bound of requests looking at the command pdm add requests? Is it 0.0.0? Or does PDM need to perform some code analysis to find it out?
IMHO that is too much for a package manager. You, who write the code, should be responsible to set any bound if that is important: pdm add requests>=2.10

I understand your intentions behind this feature request, but this is not realistic because there is no way to know the minimum required version unless you explicitly give it.

[frostming]

Oh I might misread the feature request, there is another request to resolve the version to the minimum required. This part can be realized though.

So:

1. pdm add requests -> saving requests>=2.27.1: Not going to change
2. requests>=2.0 -> resolving to requests==2.0: Can be worked out.

Think of it more, what should happen if run pdm update or pdm update <package_name>?

[fjarri]

there is another request to resolve the version to the minimum required.
requests>=2.0 -> resolving to requests==2.0: Can be worked out.

Yes, pretty much this. As far as I understand currently the resolver tries to find a solution with the newest possible versions, I would like an option where it would try to find the oldest possible versions, given the constraints in the pyproject.toml. So it probably should be an option to lock (which uses pyproject.toml and changes/creates the lock file), and not affect the behavior of update (which modifies pyproject.toml).

[frostming]

FYI update also changes the lock file, trying to find the latest version of given packages.

[fjarri]

Yes, that's true, but it would be counter-intuitive to have this as an option of update.

[frostming]

I thought of this feature again and found that it may produce broken lock files.

Say you add a dependency without a lower bound like requests, with the "prefer oldest" strategy, requests 0.2.0 will be picked since it doesn't have any python requires and PDM thinks it satisfies our constraints. But it doesn't because it doesn't even run on Python 3.

[fjarri]

Perhaps some safety checks can be implemented? E.g. only minimize versions for the packages with explicit lower bounds, and only use those versions with explicit Python bounds. After all, it is only supposed to be used for testing purposes.

[frostming]

Yes, you would say you clearly know what you are doing and will remember to set lower bounds every time. But what if it comes to sub-dependencies? You don't know what sub-dependency is missing a lower bound then a broken and legacy version will be pinned.

[frostming]

Or do you know other package managers that do the same so I can learn from it?

[fjarri]

But what if it comes to sub-dependencies? You don't know what sub-dependency is missing a lower bound then a broken and legacy version will be pinned.

Well, wouldn't that achieve the goal, which is to see if given bounds can produce a bad lock file? Edit: or, rather, allow the package to be installed on a system with already installed packages satisfying the bounds, but making your package non-operational.

Also, perhaps, this feature does not need to minimize sub-dependencies? We're only interested in explicit first-level dependencies, the ones we actually use. We want to see if even they're at their lower bound our package still works; but what they install as sub-dependencies is their business.

Or do you know other package managers that do the same so I can learn from it?

No, unfortunately I don't.

[frostming]

Well, I will leave this for discussion until enough people show their interest in this feature(by voting).

[CharString]

Perhaps some safety checks can be implemented?

The real solution would ask for something that works like git bisect and uses your project's tests. I'm not great with combinatorics, but my intuition says this explodes in complexity with multiple (sub-) dependencies.

But this is something interesting. Not something that should be implemented in pdm nor the res resolution library we will replace pip with. But definitely a use case it could enable in a plugin.

[fjarri]

As an example of this functionality, Rust's cargo has a -Z minimal-versions option (currently only available in nightly):

    -Z minimal-versions        -- Resolve minimal dependency versions instead of maximum

and there's a corresponding plugin

[frostming]

I think the situation Rust is facing is much better than the Python community. It enforces relatively strict rules on the package versions.

[fjarri]

It enforces relatively strict rules on the package versions.

I wouldn't say it's enforced, but for whatever reason semver is much more popular in Rust ecosystem. And, of course, the dependencies not living in a flat namespace also helps. But that doesn't mean that such resolution method won't be useful in Python.

[houbie]

I notice more sloppiness regarding dependency management in the Python ecosystem compared with Java and Nodejs.
IMO it's better to create awareness than to provide workarounds that eventually will blow up in your face.

That said, I do realise that you want to keep the barrier low in tutorials and at the same time keep them stable over time.
For this reason I always use pyprojectx in my tutorials and blogs. In combination with PDM, it can relieve your users from installing required tools (like PDM itself) and/or dependencies (example here).

Disclaimer: I am the author of pyprojectx.

[fjarri]

While I can see the benefits of using this tool, I am not sure what does it have to do with this issue. You cannot force a lockfile on the users of a library because they will have their own lockfiles. That's why it is important to ensure your minimum bounds are actually valid.

[frostming]

Please go to #2033 for further discussion on this issue.

[inigohidalgo]

xref: completed in #2327