refactor(bindings/python): generate type stubs with pyo3 introspection by chitralverma · Pull Request #7824 · apache/opendal

chitralverma · 2026-06-25T13:06:20Z

Which issue does this PR close?

Closes #.

Rationale for this change

The Python binding generates its type stubs with pyo3-stub-gen. This has a few issues:

just stub-gen has not worked on main for months. pyo3-stub-gen 0.18 changed the binding's module layout (in a stale, unmerged PR), later bumps moved it to 0.23, and the generator was never re-run. The committed stubs went stale, so the break stayed hidden. Regenerating now fails with a wrong-module error.
It relies on #[gen_stub_*] macros and override_type / override_return_type attributes spread across the Rust code, kept in sync by hand.
It ships an extra stub_gen binary and a crate dependency.

PyO3 0.29 can describe the extension through its experimental-inspect feature, and maturin --generate-stubs turns that into .pyi files. This is the direction PyO3 is moving in. It removes the extra dependency, the binary, and the hand-maintained macros, and derives the stubs straight from the built binary.

What changes are included in this PR?

Switch the generator

Drop pyo3-stub-gen, the stub_gen binary, and define_stub_info_gatherer!. Enable pyo3/experimental-inspect. The lib is cdylib-only now (the rlib was only for the deleted binary).
Remove the #[gen_stub_*] attributes and convert the override_* attributes into inline #[pyo3(signature = (arg: "T") -> "R")] annotations.
Update the codegen template to emit a lean Scheme enum; regenerate src/services.rs (−5.3k lines).

Generate stubs at the public paths

maturin --generate-stubs writes under opendal/_opendal/. scripts/postprocess_stubs.py moves them to opendal/<name>.pyi, where type checkers resolve from opendal.operator import ….
Wire just stub-gen and run ruff's fixers in just fmt.

Guard against drift

Add a stubs CI job that runs just stub-gen and git diff --exit-code, so stale stubs fail CI. Release wheels ship the committed python/opendal tree as-is, so it must stay fresh.

Are there any user-facing changes?

Yes. The changes are to the generated stubs only; runtime behavior is unchanged.

Pros

Stubs derive from the built binary, so they stay aligned with the Rust source and the runtime types, and cannot silently drift.
Stub generation is fast — it reads the compiled binary instead of running a separate generator pass.
Richer, more consistent stubs: enum members (EntryMode.*, Scheme.*) and several dunders are now described, and Metadata.last_modified is correctly datetime | None (was datetime, which could not represent None).
Less code to maintain: no #[gen_stub_*] macros, no override_* attributes, no stub_gen binary, one fewer dependency.
Standard tooling: generation goes through maturin and PyO3 directly, the future-facing path as experimental-inspect matures.
Freshness is enforced by the new CI drift check, so the published stubs always match the code.

Cons

The per-service typed constructors are gone. The old stubs had 104 @overload def __new__(...) entries (one per service, with a typed scheme=Literal[...] and per-service config kwargs); they are now a single generic __new__(scheme: str | Scheme, **kwargs). PyO3 introspection has no overload support. A follow-up will regenerate these from Rust.
scripts/postprocess_stubs.py applies two temporary fixups: it injects the imports PyO3 omits for forward-ref annotations, and it substitutes a hand-written exceptions.pyi while the generated one is incomplete (create_exception! types are not introspectable). These are PyO3 limitations and should be removed as experimental-inspect improves in future releases.

Please add the breaking-changes label for the dropped constructor overloads.

AI Usage Statement

Replace pyo3-stub-gen with native PyO3 introspection (experimental-inspect) for type-stub generation. - Cargo.toml: add pyo3 experimental-inspect; drop pyo3-stub-gen dep and the stub_gen bin; lib crate-type cdylib-only (rlib was only for the deleted bin). - Delete src/bin/stub_gen.rs; drop define_stub_info_gatherer! and the pyo3_stub_gen use from lib.rs. - Strip all #[gen_stub*] attrs across capability/errors/file/operator/ metadata/layers; convert override_type/override_return_type to inline #[pyo3(signature = (arg: "T") -> "R")] annotations. errors.rs uses pyo3::create_exception. - python.j2: drop module_doc!/gen_methods_from_python! stub-gen wiring, keep the Scheme enum + impl_enum_to_str!. python.rs: keep the helper fns/registrations for a follow-up PR; only add a trailing-newline guard to the generated services.rs. Stub layout, the generation pipeline, and the handling of exceptions and per-service constructors are handled separately.

- stub-gen: generate services bindings, build with maturin --generate-stubs, run the stub post-processor, then `just fmt`. - fmt: ruff format then `ruff check --unsafe-fixes --fix` (fix must run after format), so generated stubs are normalised in place. - ruff.toml: keep only the *.pyi ignores that are not auto-fixable (PYI021 to retain introspected docstrings, ANN401, ANN003, TID252); the rest are handled by the fmt fixers. - Use --manifest-path instead of -m for maturin commands.

Delete the public opendal/*.pyi generated by pyo3-stub-gen. Stubs are now produced by pyo3 introspection under opendal/_opendal/ and resolved via the package's TYPE_CHECKING imports.

Explain that typing.Self (3.11+) is unavailable at the >=3.10 floor, so the introspected stub annotation uses typing_extensions.Self.

Add the type stubs generated by pyo3 introspection at the public ``opendal/<name>.pyi`` paths so ``from opendal.operator import X`` and friends resolve for type checkers and IDEs. ``__init__.py`` imports the native ``_opendal`` extension first (which registers the submodules in ``sys.modules``) and re-exports the public classes and submodules.

Rewrite the stub post-processor to move the introspected stubs from ``opendal/_opendal/`` to the public ``opendal/<name>.pyi`` paths (where type checkers resolve them), in addition to injecting forward-ref imports and the hand-written exceptions stub. Run ruff's fixers silently in ``just fmt`` so the relocated stubs are normalised.

Regenerate the service bindings from the updated codegen template: a lean `Scheme` enum plus `impl_enum_to_str!`, dropping the pyo3-stub-gen `gen_methods_from_python!` constructor overloads.

Add a `stubs` job that runs `just stub-gen` and `git diff --exit-code`, failing the build when the committed codegen or introspection stubs are stale. Release wheels ship the committed python/opendal tree verbatim, so it must stay fresh.

Copilot

Pull request overview

This PR refactors the Python binding’s type-stub generation pipeline to use PyO3’s introspection (experimental-inspect) via maturin --generate-stubs, removing the legacy pyo3-stub-gen-based workflow and reworking the binding code to express signature/type info through #[pyo3(signature = ...)]. It also adds a CI drift check to ensure committed .pyi stubs stay in sync.

Changes:

Replace pyo3-stub-gen + stub_gen binary with maturin --generate-stubs, and remove the stub-gen macros/attributes from Rust sources.
Add scripts/postprocess_stubs.py to relocate generated stubs to public module paths and apply temporary fixups (imports + exceptions stub substitution).
Regenerate service scheme bindings and committed .pyi files; add a CI job to fail if regenerated stubs differ from committed ones.

Reviewed changes

Copilot reviewed 24 out of 26 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
dev/src/generate/python.rs	Ensures generated `services.rs` ends with a trailing newline.
dev/src/generate/python.j2	Updates the generator template to emit a lean `Scheme` and removes `pyo3_stub_gen` usage.
bindings/python/src/services.rs	Regenerated services enum and string conversion mappings (large shrink).
bindings/python/src/operator.rs	Replaces `pyo3_stub_gen` overrides with `#[pyo3(signature=...)]` annotations for stubs.
bindings/python/src/metadata.rs	Removes stub-gen attributes; relies on PyO3/jiff conversion for datetime types.
bindings/python/src/lib.rs	Drops stub-gen gatherer wiring; exports `Scheme` directly.
bindings/python/src/layers.rs	Removes stub-gen attributes and relies on PyO3 signatures/introspection.
bindings/python/src/file.rs	Adds/updates `#[pyo3(signature=...)]` annotations for generated stubs.
bindings/python/src/errors.rs	Switches exception macro import to `pyo3::create_exception`.
bindings/python/src/capability.rs	Removes stub-gen attribute wrapper on `Capability`.
bindings/python/src/bin/stub_gen.rs	Deletes the legacy stub generator binary.
bindings/python/scripts/postprocess_stubs.py	New stub post-processor to relocate stubs and patch known PyO3 introspection gaps.
bindings/python/ruff.toml	Adjusts `.pyi` per-file ignores for the new generated stub shape.
bindings/python/python/opendal/types.pyi	Updated generated stubs for `Entry`, `Metadata`, `EntryMode`, `PresignedRequest`.
bindings/python/python/opendal/services.pyi	Updated generated stubs for `Scheme`.
bindings/python/python/opendal/layers.pyi	Updated generated stubs for layers.
bindings/python/python/opendal/file.pyi	Updated generated stubs for `File` / `AsyncFile`.
bindings/python/python/opendal/exceptions.pyi	Replaced with simplified stub output (via postprocess hook).
bindings/python/python/opendal/capability.pyi	Updated generated stubs for `Capability`.
bindings/python/python/opendal/_opendal.pyi	New stub for the extension package `_opendal` namespace.
bindings/python/python/opendal/init.py	Adjusts import order to ensure `_opendal` registers submodules before re-exports.
bindings/python/pyproject.toml	Removes `ty` exclusion of `.pyi` stubs.
bindings/python/justfile	Rewires `stub-gen` to use maturin + postprocess; expands `fmt` to fix stubs via ruff.
bindings/python/Cargo.toml	Drops `pyo3-stub-gen` dep and `stub_gen` bin; enables `pyo3/experimental-inspect`.
.github/workflows/ci_bindings_python.yml	Adds a `stubs` job that regenerates stubs and fails on drift.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

`maturin --generate-stubs` reads the introspection data from the built library. On Linux, `[tool.maturin] strip = true` strips that data out, so stub generation fails with "No module named _opendal found" (it happened to survive the strip on macOS). Drop the maturin strip; release wheels are still stripped via `[profile.release]`.

`just stub-gen` ends in `just fmt`, which runs taplo and hawkeye. Install both in the stubs job so stub regeneration completes (matching the tools used elsewhere in CI).

Use the prebuilt hawkeye installer instead of `cargo install`, which builds from source and is slow.

…stubs - readinto requires a writable buffer; type it `bytearray | memoryview` instead of `bytes | bytearray` (`bytes` is rejected at runtime). - __aenter__ returns an awaitable, so annotate it `Awaitable[Self]` rather than `Self`.

chitralverma · 2026-06-25T15:32:59Z

@erickguan here a big one, mostly deletions.

Xuanwo

Love this, thank you!

erickguan · 2026-06-26T08:26:13Z

This is a really nice PR. Deleting code feels always great.

chitralverma added 8 commits June 25, 2026 14:32

refactor(bindings/python): remove old stub-gen .pyi stubs

d359cab

Delete the public opendal/*.pyi generated by pyo3-stub-gen. Stubs are now produced by pyo3 introspection under opendal/_opendal/ and resolved via the package's TYPE_CHECKING imports.

docs(bindings/python): note why __aenter__ uses typing_extensions.Self

60687ba

Explain that typing.Self (3.11+) is unavailable at the >=3.10 floor, so the introspected stub annotation uses typing_extensions.Self.

feat(bindings/python): regenerate services.rs for pyo3 introspection

31aae18

Regenerate the service bindings from the updated codegen template: a lean `Scheme` enum plus `impl_enum_to_str!`, dropping the pyo3-stub-gen `gen_methods_from_python!` constructor overloads.

ci(bindings/python): add stub drift check

06191b1

Add a `stubs` job that runs `just stub-gen` and `git diff --exit-code`, failing the build when the committed codegen or introspection stubs are stale. Release wheels ship the committed python/opendal tree verbatim, so it must stay fresh.

chitralverma requested a review from messense as a code owner June 25, 2026 13:06

Copilot AI review requested due to automatic review settings June 25, 2026 13:06

chitralverma requested review from Xuanwo, Zheaoli and tisonkun as code owners June 25, 2026 13:06

dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. releases-note/refactor The PR does a refactor on code or has a title that begins with "refactor" labels Jun 25, 2026

Copilot started reviewing on behalf of chitralverma June 25, 2026 13:06 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

Comment thread bindings/python/src/file.rs Outdated

Comment thread bindings/python/src/file.rs Outdated

Comment thread bindings/python/src/file.rs

Comment thread bindings/python/justfile

chitralverma added 5 commits June 25, 2026 18:51

ci(bindings/python): install taplo and hawkeye for the stubs job

818161c

`just stub-gen` ends in `just fmt`, which runs taplo and hawkeye. Install both in the stubs job so stub regeneration completes (matching the tools used elsewhere in CI).

ci(bindings/python): install hawkeye from prebuilt binary

51ca36c

Use the prebuilt hawkeye installer instead of `cargo install`, which builds from source and is slow.

Merge branch 'main' into refactor/python-introspect

27b7e56

Xuanwo approved these changes Jun 25, 2026

View reviewed changes

dosubot Bot added the lgtm This PR has been approved by a maintainer label Jun 25, 2026

Xuanwo merged commit 7014914 into apache:main Jun 25, 2026
39 checks passed

chitralverma deleted the refactor/python-introspect branch June 25, 2026 18:21

chitralverma mentioned this pull request Jul 8, 2026

feat(bindings/python): typed service configs via from_config #7883

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor(bindings/python): generate type stubs with pyo3 introspection#7824

refactor(bindings/python): generate type stubs with pyo3 introspection#7824
Xuanwo merged 13 commits into
apache:mainfrom
chitralverma:refactor/python-introspect

chitralverma commented Jun 25, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chitralverma commented Jun 25, 2026

Uh oh!

Xuanwo left a comment

Uh oh!

Uh oh!

erickguan commented Jun 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

chitralverma commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

AI Usage Statement

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chitralverma commented Jun 25, 2026

Uh oh!

Xuanwo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

erickguan commented Jun 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

chitralverma commented Jun 25, 2026 •

edited

Loading