ENH: Don't restrict the upper Python version or use "<4" #24810
Comments
|
That sounds like everything is working as designed. You cannot install numpy 1.26.0 into python>=3.13. |
|
This works as intended "on paper" by the specification, but it doesn't work in the real world in practice because most projects and libraries don't restrict the Python version from above within the minor version (3.x). Restriction from below is reasonable and necessary, restriction from above is almost meaningless in the vast majority of cases and it only gets in the way. |
|
The upper version caps in numpy releases are new but scipy has been doing it for a while, see scipy/scipy#17957 and links therein for more context. I think poetry not allowing installation because there are possible version mismatches on different versions of python from what is installed is a strange decision. There's not much NumPy can do to change how poetry handles upper version bounds. I think you will have to make your version bounds a bit more complicated from here on out, matching what NumPy does, see the poetry docs. |
|
I understand the position of SciPy dev team and all their pain with the ABI, but capping the upper Python version affects a lot of projects and libraries. It just breaks what was working. Everyone's gonna have to do the same things: capping the upper Python version. Why? However, NumPy and SciPy have worked for years in thousands of packages without capping the upper Python version. I understand what the developers are trying to convey, but it's basically shifting problems from the sick head to the healthy head. Imagine you're using a bunch of dependencies that need NumPy and suddenly it all breaks just because NumPy restricted the upper Python version. All your package management is going to hell. Because it's very difficult to get all authors of all these packages to follow the same rule (because it's very inconvenient and meaningless in their case). We can all just write tests and specify Trove classifiers for the versions of Python that we support, from the minimal version all the way up to the latest current version instead of suffer the headache of upgrading the upper Python version every time for all our projects and libraries. |
espdev
changed the title
This is either a Poetry bug, or the Poetry devs want you to do this. The Poetry docs seem to hint at the latter. I'm not sure though, since the Poetry devs did not participate in the relevant discussions even when pinged (as far as I know). There's also the matter of propagating caps on other dependencies. E.g., SciPy has for a long time capped NumPy with the |
Because it is the right thing for a library like NumPy (EDIT: And I like optimistic pins for smaller libs). It is very unlikely (probably already known to be true) that 3.13 will Going to be the trigger happy person here and close this. This is a discussion for poetry devs/users to figure out. |
Any modern package management tool: pip with strict dependency resolving, PDM, Poetry will work problematic with restriction of upper Python version. I'm not the first and I guess I'm not the last person to raise this issue. Package management in Python has always been an issue. Modern package management tools give us a chance to fix things, or at least do so with less pain, but things break down again because of, I think, misuse and misunderstanding of the right concepts. Good luck, everybody. |
Yup, and I even agree with them! But as far as I remember all of those discussions give reasons that don't really apply to this situation. So, if you can argue that there is a decent chance that NumPy compiles with Python 3.13, then you can get out those arguments. I don't think that was true for any of the recent Python minor releases and Python isn't exactly slowing down when it comes to changing things these days. From my prespective those discussions might be an argument to look at your own But honestly, this annoyance is really only about poetry. You say |
|
When poetry says It does NOT mean to change: to It means to change it to: Which resolves just fine! If a future user is trying to use whatever outdated version of your package exists when 3.13 comes out, then it'll break because |
Isn't this what the metadata for numpy 1.26 already states? Why the repetition? |
|
@mattip, I think that was what is recommended for the downstream project. |
|
Thanks for the clarification @BeRT2me, that's quite useful - and seems much more reasonable as behavior than what the initial issue description here suggested.
My guess is that it'll help Poetry do quicker/better solves. It is already in the numpy metadata, but Poetry cannot know that without a PyPI REST API call to query that metadata. If you look at Spack for example, it has the numpy version ranges for |
|
The Python ecosystem does not support upper caps in Requires-Python. I've proposed fixing this, but nothing has been done: https://discuss.python.org/t/requires-python-upper-limits/12663. TL;DR: This field was designed to allow old Python versions to be dropped via backsolving. This is great for lower caps but wrong for upper caps. What's worse, locking solvers like Poetry and PDM solve for all Python version at once, and they don't have a way to specify a separate range for the locking solver and the package public metadata. So every single Poetry/PDM user will now have to put <3.13 or they will get NumPy 1.25 forever, and if they do, they now will also require <3.13, even if they don't need to change code (most user code doesn't change on a Python update), even if they don't lock their NumPy version! If you run pip in CPython 3.13, it backsolves, since that's what this field was designed for: $ pyenv install 3.13-dev
$ ~/.pyenv/versions/3.13-dev/bin/pip install numpy
Collecting numpy
Downloading numpy-1.25.2.tar.gz (10.8 MB)
...PS: The current holdup in moving forward fixing this at the ecosystem level is there's no way to tell if a specifier set failed because the version was too new, too old, or somewhere in the middle of a complicated expression (like You can see a long discussion on Numba and they finally dropped their added upper cap, and only have a check with a nicer error message in setup.py, and have been very happy with the results. I assume you could either make the meson file error out if Python was unsupported, or an option could be added to meson-python to add a nice error. (The whole point is to get a nice error, otherwise you could just let the build go and see the real error). |
I'd never thought of that, but it's not really the correct solution, even if it's possibly better than the Python limit since it lets your dependencies avoid the mess; it's still producing broken metadata all to get Poetry's locking solver happy. If you depend on NumPy, you do want it to install on 3.13 if a user installs your package. By the time 3.13 comes out, there will be a version of NumPy supporting that, so you just want that, you don't want no NumPy at all. It would only be correct if you were capping or pinning NumPy. The correct solution (which isn't possible, sadly) would be to have separate bounds for making the lock file and the package metadata. You probably only want to solve for existing Python versions, but your package metadata can (and usually should) allow for future Python and NumPy versions. |
|
@mattip @rgommers The repetition is because otherwise you're technically breaking the provided rules by having both I'm not saying I agree with what NumPy has done here, but I believe my suggestion is a far better approach to the Poetry issue than capping your own project's version just because you use a library that made a controversial choice. Providing a nice error message via an
That is possible! Which puts both The published METADATA then includes: Which is reasonable - given the circumstances. |
Poetry does this as well, this difference is If I wanted Poetry to act like pip, I'd do Personally, I'd rather be warned: "Hey, add the proper python marker to numpy if you're going to specify that as a dependency without thinking about the version" - rather than having it arbitrarily break the rules I've specified and been given, or have it output distro Metadata that doesn't actually match my pyproject.toml. |
|
TBH, while I don't care what we do (the one argument that is somewhat convincing to me is that there is in practice almost no gain in adding the upper bound). But, I find most of this rather impenetrable collection of arguments... My understanding:
I don't care about removing the upper bound, but I am still unhappy about what comes across as tooling forcing libraries to not provide correct information. The original issue still sounds to me like "Poetry is making my life hard because NumPy is providing correct metadata". You cannot resolve a fixed NumPy version for future Python at this time. That is clearly correct! I suspect this is what you mean with:
So what does it take for this being the correct solution? Isn't this a poetry issue? Why can't poetry use no bounds in the metadata, but lock things to |
|
@seberg I think you're mostly asking the right questions, and yes it's a Poetry / "design of Python packaging" problem. For context: there are still significant gaps in the conceptual model for metadata of Python packages. On the one hand, PyPA folks will tell you that sdists/PyPI is more flexible/generic than conda/debian/etc, and the metadata applies in any environment - and that that's why PyPI is special and other distros should rely on / pull from PyPI. On the other hand, when you then add metadata that is (a) correct, and (b) also in fact needed in other distros, there's a "oh but this is special because pip/poetry does it like X". Example of where this exact metadata for Python version support is necessary and now actually manually guessed by peeking at what CPython versions we release wheels for: See https://pypackaging-native.github.io/key-issues/pypi_metadata_handling/ for more connected issues.
I think at some point we're going to have to bite that bullet (or multiple bullets). The "wheel metadata can't differ from sdist metadata" that you (@henryiii) are preparing a PEP for, as well as this "sdist metadata isn't actually generic" will need to be resolved in a consistent conceptual manner sooner or later (probably later). |
|
Clearly, it is true that for the majority of libraries it is impossible to give a correct upper version. And I can only agree that upper version limits on Python must be useless at least in almost all cases until some large changes happen. For the right now, it sounds a like removing the upper bound is the pragmatic thing because for "reasons" it's sometimes problematic and it doesn't really help anyone in practice. What would help me be more convinced would probably be (one or both):
|
|
@seberg I believe it is exactly the behavior that poetry wants, because it can't do more than follow the rules you've given it. Their error message says python property, not python version. Just because people expect it to read their minds, doesn't mean it knows what you want without specification. Could they do a better job of hand-holding the user through some of these tough configurations? Absolutely, but that's for them to figure out. A hypothetical to back this up. Say NumPy stopped all development today and never pushed a single update.
It seems silly to expect Poetry to implicitly choose that I say: "Trust me bro, I can see the future and I know NumPy will come out with a 3.13 compatible version by the time you use my library in 3.13." ... This is essentially what people are asking for.
If I have a pyproject.toml like: And I run
|
|
Nice explanation, thanks @BeRT2me. I think it's aligned then: it's what Poetry wants and what NumPy wants too. If you use CPython from its main branch or an alpha/beta, we want you to install NumPy's |
|
Could we have a definitive comment on what entries downstream users should put in their pyproject.toml files? There's a lot of comments and it's not clear on which of those it should be. |
From a lockfile perspective, yes, since there is no way to specify versions that work. But it is not correct from a metadata one! Your project will work with 3.13 (or is assumed to work)! It may fail to resolve valid depdencies, but failing earlier is just harmful. If poetry wants to encode this type of thing into metadata, it needs a better way to do that. That needs a poetry issue to point to and if that issue isn't actionable at this time, that is fine. What rubs me (and probably others) wrong about this is that a big chunk of the discussion (when it isn't about how to fix things) to me comes across as "NumPy is doing a terrible thing! It does exactly what poetry wants it to do and it makes my life as a poetry user hard. So NumPy should not do this.". |
|
And to be clear: I agree that peotry forcing things that belong into lockfiles to end up into metadata is absolutely terrible because it forces completely nonsense version bounds on libraries just because they have dependencies. So, yes, I think that is probably enough of a reason to change what NumPy does, but can we also shout at poetry very clearly? |
Poetry states to specify the python requirement via the It then provides links to how to do this using either "Python restricted dependencies" or "environment markers". The very next entry in their documentation is for Multiple constraints dependencies, with which you can add NumPy as a |
Interesting, I hadn't seen that section before. It specifier exactly what Poetry wants its users to do here, which is indeed declare different version ranges of Python for different NumPy versions. And that seems to equally apply to lower bounds? If we dropped Python 3.8 support in NumPy 1.25.0 and your package supports numpy 1.24.0 and up, it seems to want: [tool.poetry.dependencies]
numpy = [
{version = "~1.24.0", python = ">=3.8"},
{version = "^1.25.0", python = ">=3.9"}
]? |
Any library that has a hard pin like this deserves to have its users blow up with compilation failures. If I set
Neither of these issues are solved by adding an upper version to NumPy... The hypothetical end user being affected in any way by NumPy not having an upper version set... is a logical fallacy, in every scenario, something else causes their issue. |
|
@rgommers For development, Poetry will always choose the highest compatible version for all dependencies.
However, If NumPy had not implemented a Maximum python version.
If I do like you suggest, and have set The version used for local development now appears to arbitrarily change depending on the version of python being used. Installing If I make a further mistake and do: Then I'm now developing with The change appears, to the casual user, to have caused Poetry to arbitrarily refuse to update to The question becomes, "If I'm only ever going to be developing with the highest compatible version of NumPy - Currently If the API changes enough, I trust NumPy to do a Major version bump, but otherwise I expect a compatible version to exist for my users by that point in time... making me jump through hoops to express that trust seems like an odd decision by NumPy.
|
|
I have quite a hard time following these arguments, but regarding "Poetry's decision has been to follow the rules": that means "its own rules" I assume (which, given the question sentence below, rely or try to force strict SemVer versioning). Because, as @seberg said right at the start, these bounds are correct. It is Poetry's design that is at fault here, and making life harder than it should for its users - bold caps don't change that. I just woke up to a question about a build failure in a really old SciPy version that didn't have any upper bounds, and had to spend 15 minutes reverse engineering upper bounds based on release dates of various versions to add the right bounds into metadata in Spack. So I'm not feeling particularly charitable on this point right now - these upper bounds are (a) correct, and (b) very useful information that I don't want to have to reverse engineer or get right with trial and error years later.
You cannot trust NumPy, or CPython, or pretty much any other major Python package, to stricly follow SemVer. That is just not how it works in the Python ecosystem. There are deprecations and removals that happen between minor versions, and that is not going to change. Is that indeed the actual problem in the Poetry design here? It's designed with the assumption Python follows SemVer, while that is clearly not the case? |
I think there are two problems? The first one is that poetry has no way to limit the Python version, but rather downgrades NumPy, because it thinks that NumPy 1.25.x supports Python 3.13. (A problem that will go away in a few years, but until then might mean libs lock in old versions of NumPy which just makes them less compatible, but poetry thinks it is more compatible.) I also thought that one problem with Poetry is that:
Which gives us the danger of proliferation of incorrect upper bounds in downstream libraries and that could end up being a nightmare for end-users/packagers when Python 3.13 comes out? |
|
A few quick comments:
Footnotes
|
|
I will reopen this since I think we decided to remove it (PRs welcome). (I honestly still can't really condense all of these arguments into what and where the big problems are and what are more arguments along something being infeasible but actually working as designed. I also still think that if this is so clear cut, there should be a link to documentation that says so, rather than meandering discussions with 100+ posts.) |
|
I found a simple example where this leads to problems: If you create a new poetry project using Python 3.12 and then install spacy with the command A new user would probably be stuck at that point, because the solution of adding Python version constraints in the pyproject.toml is not obvious at all, and there is nothing hinting at it in the log. I think it's great that numpy will solve this problem soon 👍. (There was a discussion in the poetry issues, and IIRC they said it would be too difficult to implement a fix for this specific scenario.) The only other alternative would be convincing every single project using numpy to put into their pyproject.toml, which doesn't seem sustainable and also would have to be updated each year. Edit: And installing pandas is similarly confusing, with |
While the upper version is technically correct for the released version of NumPy (we are sure it will not work on Python 3.13) advertising it creates some problems, mostly for locking resolvers. They try to guess correct versions for non-released Python versions... This is probably an ecosystem or just "reasons", but it seems less useful than trouble to do the correct advertising here. See numpygh-24810 for *way* too much discussion about the why (and you will still be confused afterwards probably, so...). This needs to be fixed or at least documented clearer upstream by PyPA or similar, but... Closes numpygh-24810
While the upper version is technically correct for the released version of NumPy (we are sure it will not work on Python 3.13) advertising it creates some problems, mostly for locking resolvers. They try to guess correct versions for non-released Python versions... This is probably an ecosystem or just "reasons", but it seems less useful than trouble to do the correct advertising here. See numpygh-24810 for *way* too much discussion about the why (and you will still be confused afterwards probably, so...). This needs to be fixed or at least documented clearer upstream by PyPA or similar, but... Closes numpygh-24810
|
I've hit with this poetry lock issue myself again with different packages and definitely not to ignite the discussion again. I'm with @seberg in not understanding the counter arguments and I am also not knowledgeable on the subject. So please let's not restart. However I'd like to invite to the issues we will have when 3.13 or further versions break SciPy/NumPy and let the folks see exactly what we are talking about. Please leave any emoji to this one so I can ping you later and discuss exactly what we mean and how we should solve it when it happens. If possible I'd like to invite poetry devs too. Because it is much easier to look at a postmortem instead and it seems to me that counter argument owners never saw our codebase breaking with every major release. Maybe stable ABI work will save us then but anyways. Fingers crossed. |
Proposed new feature or change:
Hello,
NumPy 1.26.0 restricts the Python version from above to 3.13.
This makes it impossible to install the package in projects which do not restrict the Python version from above and where a modern package manager like Poetry is used.
For example:
The project Python version is specified as:
>=3.10,<4NumPy Python version is specified as:
>=3.9,<3.13When we try to add numpy dependency to the project we get a dependency resolving error:
Why restrict the Python version from above? Is backwards compatibility not working anymore?
NumPy 1.25.0 restricts the Python version as:
Python >=3.9The text was updated successfully, but these errors were encountered: