Request: developers should be able to turn off dependency upper-bound calculations

[joelb123]

• [x ] I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

Poetry assumes that dependencies have an upper bound of the next major version from semantic versioning. I do not see a way to turn off this assumption. I also see no way to distinguish between upper bounds based on affirmative information (an explicit upper bound that indicates the package will break when using with an existing higher version of a dependency) and upper bounds based on presumption that major versions break untested code---a presumption that is usually incorrect.

Hard-wiring the upper bounds of non-existing future dependencies has the effect of making packages more brittle, not less, in the event they are used in an environment where they are not the only dependency. It's important to realize that the concept of the next-major-version-numbered dependency is a fiction until it appears. The next major version may break things, but it usually does not. As a developer, I want my package to be useful as long as possible, even if I am no longer actively maintaining the package. Being able to distinguish between the case of "might be broken by a future dependency" and "definitely breaks with an existing dependency version" are very different things. When a future user wishes to install my package, she should be able to try it with current versions, if her other dependencies dictate those versions, without having to edit out the upper bounds. If she does have to manually edit out the upper bounds, she will have no way of knowing which ones are potential/fictional and which are real.

For example, take poetry itself, which currently has the dependency of 'keyring>=20.0.1,<21.0.0' under python>3.5. My preferred method of installation is in my base environment. While you may find that preference distasteful for the way you work, you can't argue with my taste. On my distro, keyring 21 has been the only choice for a while now, and it seems to work just fine for me. So when I'm writing the build for poetry, I have to edit out the upper bounds of keyring and a number of other superseded packages. I do this with the big-boy warranty that I might break poetry, which I'm okay with, and which I find preferable to not being able to install poetry at all in that environment or to installing it in a virtualenv.

[abn]

@joelb123 I have not groked your entire description, will re-read it later once I have some mroe time. But based on parsing your last bit, I am assuming you mean the bounds written to the wheel metadata.

As a quick example, taking poetry, we currently define the dependency on keyring conservatively.

keyring = [
    { version = "^18.0.1", python = "~2.7" },
    { version = "^20.0.1", python = "~3.5" },
    { version = "^21.2.0", python = "^3.6" }
]

This translates to the wheel metadata as:

Requires-Dist: keyring (>=18.0.1,<19.0.0); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: keyring (>=20.0.1,<21.0.0); python_version >= "3.5" and python_version < "3.6"
Requires-Dist: keyring (>=21.2.0,<22.0.0); python_version >= "3.6" and python_version < "4.0"

The different versions here are because keyring < 21.0.0 does not support python 3.5 and keyring < 19.0.0 does not support python 3.

If we apply the following change hower, we remove the upper bound.

diff --git a/pyproject.toml b/pyproject.toml
index 5ef380e..ccafdd5 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -54,7 +54,7 @@ functools32 = { version = "^3.2.3", python = "~2.7" }
 keyring = [
     { version = "^18.0.1", python = "~2.7" },
     { version = "^20.0.1", python = "~3.5" },
-    { version = "^21.2.0", python = "^3.6" }
+    { version = ">=21.2.0", python = "^3.6" }
 ]
 # Use subprocess32 for Python 2.7
 subprocess32 = { version = "^3.5", python = "~2.7" }

The metadata now looks like this.

Requires-Dist: keyring (>=18.0.1,<19.0.0); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: keyring (>=20.0.1,<21.0.0); python_version >= "3.5" and python_version < "3.6"
Requires-Dist: keyring (>=21.2.0); python_version >= "3.6" and python_version < "4.0"

This has to do with how the version is specified for the dependency.

[NickleDave]

I know this issue a bit stale but I want to agree with @joelb123 and echo the points made.

As someone developing a library, not an application, I would love to use poetry for its convenience, and recommend that other people start using it, but unfortunately poetry is preventing me from leaving my dependencies "open-ended", so I can't use it.
I would like to specify a lower-bound, but I don't want to force any one that might care to use my libraries to meet some upper-bound.

Nowhere is this more obvious than for Python itself.
The default given by poetry init is "^3.x", that is ">=3.x, <4.0".
But Python 4.0 is nonexistent and even if there were plans for it to exist, there's no reason to expect it will obey semver and induce breaking changes.
This is especially a problem if I switch to using Poetry from another tool (e.g. setup.py file) where I have never specified "<4.0".
If I try to leave my Python version without an upper bound, I will suddenly find I can't install some of my dependencies if they have declared an upper bound (perhaps because they also switched to poetry and unthinkingly used the defaults).

An example is here
nedbat/coveragepy#1021
and I have similar problems with my own set of tools here
#3747
Please see comment here with links to Twitter threads describing unintended effects this can have
nedbat/coveragepy#1021 (comment)

I don't need an extra feature that lets me turn this on and off, I just need the solver to stop being uneccesarily strict.
If someone specifies ">3.x, <4.0", and I just specify ">3.x", then just give me something less than 4.0 instead of failing completely because of a non-existent Python 4.0

[joelb123]

Thanks, @NickleDave, this issue is still very much alive for me. I don't think that @abn ever quite grokked my description, and yours is better. But let me try again.

Telling whether the next major-numbered release is compatible with code is the proper function of testing, not of poetry pretending to know the future based on semver. In the field where I code (bioinformatics), code can usefully go a long time without getting attention from the developer. I should be able to run dependabot and turn on automatic merges of code that passes tests and have the dependencies get updated without my attention if I so choose.

This problem occurs a lot in code with many dependencies and seems to require manual intervention from me each time. One of the dependencies I use in an application, pyarrow, went in a period of months from 1.X to 2.0 to 3.0. Nothing in my code was broken by the version changes, but poetry update refused to take notice of the new versions until I manually ">"ed them in the pyproject file.

I like poetry, and I'm considering simply editing the lock file until this problem is addressed, but that requires me to know which dependencies actually specify "<major" because they have tested and can't accept the next major release and which were themselves developed with poetry making this unjustified assumption.

[dimbleby]

poetry is preventing me from leaving my dependencies "open-ended",

No it's not. Declare your dependency as foo = ">=3.0.0", if that's what you want to do.

Is this issue just asking that the default via poetry add be different? That was discussed (and denied) in #3427.

I don't see that there's anything else here, in which case this should be closed.

[NickleDave]

Hi @dimbleby you are right.

I raised several points out of frustration but that just distracted from the question raised by the person who opened the issue, adding noise to discussion here. My mistake

A more in-depth discussion for those who are interested is here: https://iscinumpy.dev/post/bound-version-constraints/

It's not my issue to close but my impression is that the situation @joelb123 describes will not apply to most poetry users.

[joelb123]

I’ve switched to using “>=X” for 90% of my development work. It’s a minor nuisance editing it by hand. But I’m guessing I’m not alone in having updates of dependencies like scipy refused.

…

On Jul 31, 2022, at 3:55 PM, David Nicholson ***@***.***> wrote: Hi @dimbleby <https://github.com/dimbleby> you are right. I raised several points out of frustration but that just distracted from the question raised by the person who opened the issue, adding noise to discussion here. My mistake A more in-depth discussion for those who are interested is here: https://iscinumpy.dev/post/bound-version-constraints/ <https://iscinumpy.dev/post/bound-version-constraints/> It's not my issue to close but my impression is that the situation @joelb123 <https://github.com/joelb123> describes will not apply to most poetry users. — Reply to this email directly, view it on GitHub <#2731 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALKZIDM62NWTP5EFWDERFLVW3Y6TANCNFSM4PJAJJOA>. You are receiving this because you were mentioned.

[neersighted]

I'm going to close this as ultimately I think there is nothing to be done here. This is a discussion of the merits of semver, and for better and for worse Poetry encourages its use with custom specifiers like ^, but you are free to specify any version range that makes sense to you. Library authors should decide and control what version ranges they permit as that is essentially part of the public API, regardless of what tool they use.

There may be users who prefer a different default for Poetry's version constraints (and we even have discussed a hypothetical library mode that would change some defaults, like making a constraint-free poetry add use >=), but at the end of the day any constraints that are too tight are an issue you'll have to take up with the library you depend on regardless of what tooling they use to build the library.

[woutervh]

I'm going to close this as ultimately I think there is nothing to be done here.

For future reference, as poetry gains adoption, this will not age well.
We are creating a giant dependency hell caused by poetry-defaults and poetry-maintainers not realizing the problem of misreporting compatibilities.

but at the end of the day any constraints that are too tight are an issue you'll have to take up with the library you depend on
Good luck with that. It took a much-used library like isort only 4 years to fix a single char to stop misreporting its compatibility.

[KOLANICH]

https://github.com/KOLANICH-tools/unpin.py - my tool to unpin version constraints from prebuilt wheels.

[KOLANICH]

For future reference, as poetry gains adoption

Poetry will not gain adoption. It has gained some adoption because it looked new, shiny and fashionable. Now it doesn't, and its technical drawbacks outweight everything else except the the fact some people already using it and don't want to do pointless changes once again.

[joelb123]

This problem is becoming more and more frequent in my work as a computational biologist. It is for this reason I cannot recommend poetry for any new project and project templates.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.