Please reconsider the Python support policy

[robsdedude]

The Problem

Initially when discovering pyo3, I was really positively surprised to find that it even supported legacy Python versions. However, recently support for Python 3.7 was dropped.

This is a big pain for me developing an extensions module for a Python library. The library follows SemVer and has an LTS branch. That means support for Python versions is kept even after they go EOL.

How to tackle it in Python

It's somewhat straight forward to deal with python dependencies that drop support for said EOL Python versions. Dependencies declared in pyproject.toml can be different versions for depending on the Python version. E.g.,

dependencies = [
    "Sphinx >= 8.1.3; python_version <= '3.10'",  # dropped support for Python 3.10 in 8.2.0
    "Sphinx >= 9.0.4; python_version == '3.11'",  # dropped support for Python 3.11 in 9.1.0
    "Sphinx >= 9.1.0; python_version >= '3.12'",
]

Trying to solve it it with build.rs

In Rust this isn't possible. Cargo isn't aware of the Python version at the time of having to select the pyo3 version to build against. So I'd have to stick to the newest pyo3 version which is supporting the oldest Python version I want to support. That's an option, but an unfortunate one as it means users of my library will not have access to security fixes (such as the recent possible oob read that got fixed in 0.29.0) even if they are on a non-EOL Python version.

Using build.rs to check which Python version to build for doesn't seem to be an option either as Cargo doesn't respect feature selection in build.rs:

Note that this does not affect Cargo’s dependency resolution. This cannot be used to enable an optional dependency, or enable other Cargo features.
—https://doc.rust-lang.org/cargo/reference/build-scripts.html#rustc-cfg

Trying to solve it with Cargo features

Maturin does allow to select package features depending on the Python version. Something like

# pyproject.toml
# [...]
[tool.maturin]
features = [
    { feature = "pyo3_0_28", python-version = "<3.8" },
    { feature = "pyo3_latest", python-version = ">=3.8" },
]
# [...]

But that doesn't get me anywhere either because adding this to my Cargo.toml

# [...]
[features]
pyo3_latest = ["pyo3"]
pyo3_0_28 = ["dep:pyo3_0_28"]

[dependencies]
pyo3 = { version = "0.29.0", optional = true }
pyo3_0_28 = { package = "pyo3", version = "0.28.3", optional = true  }
# [...]

Leaves me with the following compilation error

error: failed to select a version for `pyo3-ffi`.
    ... required by package `pyo3 v0.28.3`
    ... which satisfies dependency `pyo3_0_28 = "^0.28.3"` of package `neo4j-rust-ext v0.1.0 (/home/rouven/Neo4j/code/neo4j-python-driver-rust-ext)`
versions that meet the requirements `=0.28.3` are: 0.28.3

package `pyo3-ffi` links to the native library `python`, but it conflicts with a previous package which links to `python` as well:
package `pyo3-ffi v0.29.0`
    ... which satisfies dependency `pyo3-ffi = "=0.29.0"` of package `pyo3 v0.29.0`
    ... which satisfies dependency `pyo3 = "^0.29.0"` of package `neo4j-rust-ext v0.1.0 (/home/rouven/Neo4j/code/neo4j-python-driver-rust-ext)`
Only one package in the dependency graph may specify the same links value. This helps ensure that only one copy of a native library is linked in the final binary. Try to adjust your dependencies so that only one package uses the `links = "python"` value. For more information, see https://doc.rust-lang.org/cargo/reference/resolver.html#links.

failed to select a version for `pyo3-ffi` which could resolve this conflict

[LilyFirefly]

Why are your library's users on really old versions of Python? PyO3 looks at the cryptography library's download statistics for old Python versions to decide when to drop EOL Pythons. See #4581 for the most recent discussion.

If your library is a commercial project, maybe it would be worthwhile maintaining a fork of PyO3 (or paying someone to do it for you) which maintains support for old versions or backports fixes from new versions.

[robsdedude]

Thanks for your input.

For me this is about SemVer. I very much dislike how Python and much of its ecosystem started to not care about SemVer any longer. I want to offer a stable library API with fixes as needed (LTS), and despite how hard it is I want to follow SemVer.

The question of how many users are still using 3.7 is interesting, but no matter the answer it wouldn't change what I'm aiming for. The LTS was cut with 3.7 support in it so I want to maintain it. The other option, of course, would be to EOL the LTS version and cut a new one. But I feel like there should be some path forward keeping support for older Python versions while receiving fixes for newer ones. To achieve the goal of SemVer, I'm fine with having to do some version dances depending on the Python version (like I already do with Python dependencies). I'm also fine with conditional code-paths. But I just don't see any way whatsoever to support a wide range of Python versions using pyo3 (besides very hacky things like editing/copying the Cargo.toml in some pre-processing step, or absolute behemoth approaches like maintaining a pyo3 fork as you suggested).

[LilyFirefly]

The other option, of course, would be to EOL the LTS version and cut a new one.

I think this is probably your best option. Python 3.7, 3.8 and 3.9 are all EOL and not receiving security fixes from upstream. Similarly, your users won't receive any new security fixes from PyO3 until you upgrade.