RFC: Use a capsule-based API with a stable ABI for global, cross-extension functionality.

[adamreichold]

This does not work (meaning build) due to multiple issues but I wanted to propose something concrete here so we can discuss whether this roundabout way is something we would want to pursue.

As this it is proposed, this should mean that the first extension built using PyO3 that calls ensure_global_api creates the module containing its PanicException type object and the corresponding capsule API. All other PyO3-based extensions would then use this capsule to create panic exceptions based on the same internals as first one.

Downstream code should then be able to import pyo3.PanicException to catch all panic exceptions raised by an PyO3-based extension. (At least if it is new enough to use the capsule API.)

[adamreichold]

The types do not work out at all for now, but I wanted to discuss the concrete approach first before trying plough through the trampolines.

[davidhewitt]

(I had intended to include GILProtected in 0.18.2, unfortunately bungled the rebase. We could push out a 0.18.3 however I'm tempted to just get on with some new features for 0.19 and get that out the door faster instead :) )

[adamreichold]

(I had intended to include GILProtected in 0.18.2, unfortunately bungled the rebase. We could push out a 0.18.3 however I'm tempted to just get on with some new features for 0.19 and get that out the door faster instead :) )

Currently, the only reason I would see for a 0.18.3 is the PyDateTime type object SNAFU.

[adamreichold]

Any comments w.r.t. the proposed direction? Do want this solution for cross-extension functionality? Are there alternatives? Are we okay with a static for caching the capsule pointer? Again, are there alternatives?

[davidhewitt]

There are the following questions / alternatives in my mind:

• Functionality: I can imagine this API may grow to also contain various infrastructure types beyond the exception:
  • pyo3.Future type which could be a wrapper of Box<dyn Future>
  • pyo3.Iterator type which could be a wrapper of Box<dyn Iterator<Item = PyObject>>
  • This may give us power to build designs which we can't easily ship at the moment.
• Discoverability: we could publish a pyo3 package on PyPI. The upside of doing this is that then downstream code can import pyo3 immediately without needing to first import a module which is built in PyO3. I quite strongly believe downstream users will frequently be confused if import pyo3 is path-dependent and they can't install it, so I think publishing is probably necessary.
  • We can also ship .pyi type annotations for PanicException and future APIs.
  • This capsule-based solution would likely still be used to build that pyo3 package?
  • Downside of shipping on PyPI is that pyo3-built extensions may need to update their Python dependencies to list our PyO3 version, creating more churn.
• Versioning: this current draft crashes if there are conflicting PyO3 versions. I wonder if we can namespace them to support multiple versions. We may also want to version per portion of the API individually rather than at the global level.
  • I think the main goal is to ensure downstream code can be as agnostic as possible about individual PyO3 versions.
  • It would also be quite helpful if PyO3-built packages using different PyO3 versions can coexist. This might be at odds with pyo3 being installable from PyPI.
    • Perhaps the install from PyPI defines the top-level API which is importable, and we do as best we can to make it forward and backward-compatible with other PyO3 versions?
      • Maybe it's even possible to make that work in a way where downstream crates do not need to require or pin to a specific pyo3 package version as a Python dependency?
  • As an idea, pyo3.PanicException can be itself subclassed by versioned types, e.g. pyo3.v0_20.PanicException, pyo3.PanicExceptionV1, pyo3.PanicExceptionV2. That way we could freely rebuild the actual concrete type per version while users can still just write except PanicException and not care about versioning.
• I think the static falls under the usual story that at right now it's a necessary evil, however when we have a way to store this in module data then we should do so and remove the static.

[adamreichold]

This capsule-based solution would likely still be used to build that pyo3 package?

Yes, I think this is the case. If we have a separately installable pyo3 module, it would mainly give it the responsibility to install the capsule API instead of lazily installing it by whichever PyO3-built extension is loaded first.

The only alternative I see to provide a pure Python API which we use internally, but that would still have the same versioning issues and might have additional performance issues on top of that.

this current draft crashes if there are conflicting PyO3 versions. I wonder if we can namespace them to support multiple versions. We may also want to version per portion of the API individually rather than at the global level.

It would also be quite helpful if PyO3-built packages using different PyO3 versions can coexist. This might be at odds with pyo3 being installable from PyPI.

The idea is that the capsule API stays backwards compatible, i.e. whenever an older than current version is installed, this might mean degraded functionality, but not imply crashes. (The check could stay version < 1 indefinitely which basically means the version is not a random zero but initialized. But an extension built against version 5 should be able to use version 1 with degraded functionality. Similarly, an extension built against version 1 should be able to version 5 without any issues.)

Of course this implies additional effort to implement this graceful degradation as the API evolves and it makes the path dependency worse (i.e. the available functionality depends on which extension is loaded first).

Only if we need to add new but universally required functionality will there be a hard incompatibility. (And the check would change to e.g. version < 42 if we introduce necessary functionality in version 42.)

I think working hard on a minimal and highly compatible capsule API is preferable to for example provide multiple namespaced API as this complicates an already complex design further but still could not rule incompatibilities like multiple PanicExceptions. (Splitting might reduce the blast radius of incompatibilities, but it would not avoid them.) Finally as you point out, this does not fit well with the Python approach to dependency management which would not allow multiple pyo3 modules in a single dependency tree AFAIU.

I think the observation that other foundational projects like NumPy were able to evolve their capsule API in a compatible manner suggests that this is possible. There is the danger of ossification for which NumPy is probably an example as well, but to me this mainly implies that we must put a lot of consideration into adding anything to the cross-extension API, i.e. try to keep as much functionality as possible extension-private.

So in summary, I think providing a separately installable PyO3-built pyo3 package to be published on PyPI would be a good thing. It would also simplify thinking about the capsule API as the self-referentialness of installing and using it from a single code base would go away. But we should still aim for that package to provide a single backward-compatible capsule API.

If we do go for this, the next step would be to setup a Python package implemented using PyO3 which we can publish to PyPI in the repository here? Or do we want a separate GitHub project? (More self contained, but harder to evolve in parallel. But then again coupling should be loose.)

[adamreichold]

One thing I noticed that if we provide a separate pyo3 package which installs the capsule API, we have to be very very careful to not do something that would call into that capsule API until it is installed when that pyo3 package is initialized. Or we would need to not use PyO3 to implement the pyo3 package?

[adamreichold]

Making the pyo3 package pure Python and caching a reference to the constructor of a PanicException defined in that Python package and just calling that via the normal Python calling convention would have a certain simplicity going for it, wouldn't it? The performance should also not be such a problem as long as we talk about raising exceptions?

[davidhewitt]

Making the pyo3 package pure Python and caching a reference to the constructor of a PanicException defined in that Python package and just calling that via the normal Python calling convention would have a certain simplicity going for it, wouldn't it? The performance should also not be such a problem as long as we talk about raising exceptions?

Definitely would be simpler. Would that rule out replacing it with a capsule-based version later? Any older PyO3 versions which just do import pyo3 could potentially fall-back to using a private type if the import fails.

[davidhewitt]

One thing I noticed that if we provide a separate pyo3 package which installs the capsule API, we have to be very very careful to not do something that would call into that capsule API until it is installed when that pyo3 package is initialized. Or we would need to not use PyO3 to implement the pyo3 package?

Can you provide an example of how things might go wrong? I haven't thought super hard about this, however it's not readily apparent to me what the kind of failure you're anticipating here looks like.

[adamreichold]

Can you provide an example of how things might go wrong? I haven't thought super hard about this, however it's not readily apparent to me what the kind of failure you're anticipating here looks like.

Well let's say initialization of the pyo3 extension panics and wants to create a PanicException but alas it cannot because it panicked before the capsule API used to instantiate a PanicException was installed. We could silently fall back to an internal type, but as a general approach, this would probably mask a lot of configuration/deployment errors which only come to light when except PanicException: does not work, i.e. at a really bad time on the production system. So we might need to limit that fallback to the pyo3 extension itself, but that brings us back to the self-referential dance showcased here which would be nice to avoid by using a canonical source for the capsule API.

Definitely would be simpler. Would that rule out replacing it with a capsule-based version later? Any older PyO3 versions which just do import pyo3 could potentially fall-back to using a private type if the import fails.

I think we could always upgrade to a capsule API later and hence think that starting with a pure Python pyo3 package (and by extension pure Python PanicException class) would probably be the best next step. We can get this published, integrate it into version 0.20.0 and gain some general experience with the approach without getting bogged down in the technicalities of the capsule API evolution.

[davidhewitt]

Ah right I see, yes. There are options like aborting or falling back silently but they're all quite messy, so something it would be nice to learn about first.

I'm in favour of starting with a pure-Python package and we can learn from there.

So I think just a couple questions to consider:

• Do we call the PyPI package pyo3 or something like pyo3_runtime? (pyo3_runtime.PanicException is the current name I gave the type object.) I am happy to consider arguments for both or any other suggestion.
  • It might be a good idea to go for pyo3 and pyo3.PanicException to make a clean break from the non-importable pyo3_runtime.PanicException which will exist for 0.19 and older.
• Versioning: do we version this separately from the Rust? If downstream packages put bounds on, say, pyo3==0.20, they can't be installed with packages depending on pyo3==0.21. Maybe we document to always use lower bounds pyo3>=0.20 and promise to users we will never break compatibility? That's really hard, pyo3>=0.20,<1? Same compatbility problem I guess when we release 1.0 and end users can't have compatible PyO3 runtimes.
  • Maybe use calendar versioning for this package rather than semver? That way there's no "breaking" releases and it's just about what runtime features the end user needs. pyo3_runtime>=2023.06 ? pyo3>=2023.06 ?

[davidhewitt]

... I think I quite like going for pyo3 with calendar versioning - it's relatively obvious what it means. It has the great property that pyo3>=2023.05.31 is sufficiently far enough from the Rust version of 0.19.0 that users are likely to pay attention to our documentation if we ask them to always put just a lower bound.

[adamreichold]

I am not really familiar with calendar version. From what you describe, is there a difference between >=0.20 and promise to never break and >=2023.06? Does that somehow allow more breakage?

Personally, I would probably prefer to version this separately from PyO3 (the library), e.g. we just start at 1.0 with a standard bound >=1.0,<2, i.e. we can break if we ever need to but I don't see us doing that without a very very good reason, e.g. not "just" because PyO3 (the library) hits 1.0. Version numbers being independent would also match that I would envision pyo3 (the package) evolving much slower than PyO3 (the library).

[davidhewitt]

The difference between >=0.20 looking like a semantic version and >=2023.06 would be that calendar version bumps don't imply any restrictions or changes on breakage, it just labels the release date of the package.

If we go with this package being 1.0 and followed semantic versioning I think it would be less confusing for users if we called it something different like pyo3_runtime or pyo3_core. Imagine we release the crate PyO3 1.0 a couple years from now and we're on version 1.7 of the pyo3 PyPI package. I suspect many users may trip up and try to install the older PyPI package.

Regardless of which combination we go for, I've observed that one nice thing about documenting lower bounds is that we only need to care about backwards-compatibility of the PyPI package with older PyO3 versions. We don't need to gracefully degrade PyO3 to use older versions of the PyPI package, because we can always document "extensions built with PyO3 0.20 require pyo3_core>=1.0" (or >=2023.06). As long as downstream users have set their bounds appropriately, we'll always have a suitably new package.

... Just trying to write that I realise how confusing it is to write pyo3>=1.0 and PyO3 0.20 in the same sentences, so I'd be quite keen to call the PyPI package something different unless we either:

• version and release it alongside the PyO3 crate, so everything is in sync
• (maybe) use calendar versioning, if we think the distinction that semver used for Rust crate, calver for Python package can be clear enough for users to follow.

[adamreichold]

If we go with this package being 1.0 and followed semantic versioning I think it would be less confusing for users if we called it something different like pyo3_runtime or pyo3_core. Imagine we release the crate PyO3 1.0 a couple years from now and we're on version 1.7 of the pyo3 PyPI package. I suspect many users may trip up and try to install the older PyPI package.

Not invested in naming the PyPI package pyo3, pyo3_core or pyo3_runtime for example would be fine by me. pyo3_runtime fits best IMHO because the stuff in there is support infrastructure, not core types like Py or the Python marker.

I definitely think that being able to break pyo3_runtime at some point in the future if required is worth the disconnect between versions of the PyPI package and the Rust crate. (I actually don't think this is significantly worse than using calver for the PyPI package.)

Regardless of which combination we go for, I've observed that one nice thing about documenting lower bounds is that we only need to care about backwards-compatibility of the PyPI package with older PyO3 versions. We don't need to gracefully degrade PyO3 to use older versions of the PyPI package, because we can always document "extensions built with PyO3 0.20 require pyo3_core>=1.0" (or >=2023.06). As long as downstream users have set their bounds appropriately, we'll always have a suitably new package.

I think this less an effect of lower bounds, but rather of Python accepting only one version of a package per dependency tree. Because the downgrade is required here because different versions of PyO3 could be combined to built extensions loaded into a single Python interpreter and each would have to be able to cope with the capsule API installed by the others. With any Python package like pyo3_runtime this is not possible as it will be loaded only in one version. Of course, the price to pay is that users have to manage this themselves in every extension built using PyO3.

[davidhewitt]

So I think we're converging to the decision that the next step will be to push a pyo3_runtime package with version 1.0? Shall we keep it in this repository for ease of development?

I definitely think that being able to break pyo3_runtime at some point in the future if required is worth the disconnect between versions of the PyPI package and the Rust crate.

My gut feeling is that even if we reserve the capability to do so, we'll never want to break the package in practice, because then ecosystem of packages using PyO3 is forcibly split by the runtime version they need. I can imagine we'd be really popular for putting the whole ecosystem in dependency hell 🙃

I think this less an effect of lower bounds, but rather of Python accepting only one version of a package per dependency tree. Because the downgrade is required here because different versions of PyO3 could be combined to built extensions loaded into a single Python interpreter and each would have to be able to cope with the capsule API installed by the others. With any Python package like pyo3_runtime this is not possible as it will be loaded only in one version. Of course, the price to pay is that users have to manage this themselves in every extension built using PyO3.

Yes, this is essentially what I meant. As long as the bounds are set correctly for pyo3_runtime on downstream extension manifests then the extension with highest runtime requirement will win, and that runtime should support all the extensions needing older versions.

[adamreichold]

So I think we're converging to the decision that the next step will be to push a pyo3_runtime package with version 1.0? Shall we keep it in this repository for ease of development?

Yes, I think the ease of changing this together with the usage sites would be worth it.

I'll close this issue then because this not how we will proceed. I still a pure-Python class PanicException would be a good first export of pyo3_runtime though.

[davidhewitt]

I still a pure-Python class PanicException would be a good first export of pyo3_runtime though.

Agreed. (I'm up for us including this in 0.20, shall we track it in an issue?)

[davidhewitt]

I just came across https://docs.python.org/3/c-api/init.html#c.PyInterpreterState_GetDict, which looks like it might be useful for storing panic exception which is shared across extensions. I don't think it would offer importable symbols for downstream code though.

[adamreichold]

Indeed, I still think a pyo3_runtime published on PyPI opens up whole new avenues of integration between extensions and would not avoid doing that just because we have alternative ways of stashing per-interpreter or per-module state somewhere.