Add native_enum: expose Rust enums as Python enum.Enum subclasses#6020
Open
wakita181009 wants to merge 7 commits intoPyO3:mainfrom
Open
Add native_enum: expose Rust enums as Python enum.Enum subclasses#6020wakita181009 wants to merge 7 commits intoPyO3:mainfrom
wakita181009 wants to merge 7 commits intoPyO3:mainfrom
Conversation
Introduce `#[py_native_enum]` / `#[derive(NativeEnum)]` macros that expose a fieldless Rust enum to Python as a true `enum.Enum` subclass via the functional API, without touching the C-API enum machinery (PyO3#991). Runtime (`src/native_enum/`): - `spec.rs` – `NativeEnumSpec`, `NativeEnumBase`, `VariantValue` - `base_cache.rs` – per-interpreter `PyOnceLock` cache for base classes (`enum.Enum`, `IntEnum`, `StrEnum`, `Flag`, `IntFlag`) and `enum.auto` - `construct.rs` – `build_native_enum`: builds a Python enum subclass from a spec without caching - `trait_def.rs` – `NativeEnum` trait with `py_enum_class`, `to_py_member`, `from_py_member` Macro (`pyo3-macros-backend/src/native_enum.rs`): - Parses enum-level attrs (`base`, `rename`, `module`) and variant-level attrs (`rename`, `value`); supports integer discriminants - Caches the Python class in a function-local `static PyOnceLock<Py<PyType>>` in the generated `py_enum_class`, constructing it once per interpreter session - `from_py_member` checks `isinstance(obj, cached_class)` for type safety - Auto-derives `IntoPyObject`, `IntoPyObject for &T`, `FromPyObject` Public API additions to `pyo3-macros/src/lib.rs`: - `#[proc_macro_derive(NativeEnum, attributes(native_enum))]` - `#[proc_macro_attribute] py_native_enum`
- tests/test_native_enum.rs: Rust integration tests covering the full enum protocol (isinstance, name/value, len/iter/contains, lookup by name and value), class and member identity, IntoPyObject/FromPyObject roundtrips, all five base classes (Enum/IntEnum/Flag/IntFlag/StrEnum), rename at class and variant level, module attribute, qualname via direct build_native_enum call, and VariantValue::Str - pytests/src/native_enums.rs + pytests/tests/test_native_enums.py: Python-side tests for the same surface area, exposing Color/Status/ Permission/Bits/Size to Python and verifying enum protocol from pytest - tests/ui/invalid_native_enum_base.rs: UI test for unknown base name - tests/ui/native_enum_generic.rs: UI test for generic/lifetime params (also adds the validation logic in pyo3-macros-backend)
- Apply ruff formatting to test_native_enums.py (line wrapping) - Disambiguate `pyclass` doc link with `macro@pyclass` prefix
Proc-macro crates cannot resolve intra-doc links to other crates. Replace `pyo3::native_enum::NativeEnum` with a docs.rs URL.
The test-introspection CI job generates .pyi stubs and compares them against expected files in pytests/stubs/. The native_enums module was missing its stub file. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merging this PR will improve performance by 12.89%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| 🆕 | native_enum_from_py_member |
N/A | 9.1 µs | N/A |
| 🆕 | native_enum_int_enum_class |
N/A | 423.9 ns | N/A |
| 🆕 | native_enum_int_enum_from_py_member |
N/A | 10.4 µs | N/A |
| 🆕 | native_enum_int_enum_to_py_member |
N/A | 2.5 µs | N/A |
| 🆕 | native_enum_py_enum_class |
N/A | 394.7 ns | N/A |
| 🆕 | native_enum_to_py_member |
N/A | 2.4 µs | N/A |
| ⚡ | bench_pyclass_create |
4.6 µs | 4.1 µs | +12.89% |
Comparing wakita181009:native-enum (c0f95ea) with main (f57bda7)
Footnotes
-
1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports. ↩
`&str: FromPyObject` is gated by `#[cfg(any(Py_3_10, not(Py_LIMITED_API)))]`, making it unavailable under abi3-py39. Use `String` which is always available.
Align with `#[pyclass(name = "...")]` convention. Both enum-level and variant-level attributes now use `name` instead of `rename`. Remove unused `kw::rename` custom keyword.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #2887. Adds #[py_native_enum] / #[derive(NativeEnum)] — an opt-in way to expose a fieldless Rust enum as a real Python enum.Enum subclass, without touching
the existing #[pyclass] enum path.
Discussed in #5991 (RFC + benchmarks).
How it works
The macro uses Python's functional enum API at runtime to construct the class:
equivalent to what the macro generates at first use:
Color = enum.Enum("Color", [("Red", auto()), ("Green", auto()), ("Blue", auto())])
Two layers of caching keep things both correct and fast, both backed by PyOnceLock:
session (generated by the derive macro).
Caching the generated class is essential for correctness, not just performance: without it, each to_py_member call would produce a new class object, and
isinstance(x, Color) would return False for values converted in a previous call.
This makes repeated conversions as cheap as a reference clone (~59 ns, on par with #[pyclass] enum at ~58 ns — see #5991 for the full benchmark table).
Usage
IntoPyObject and FromPyObject are auto-derived, so the enum works directly in #[pyfunction] signatures.
Supported bases
Enum (default) · IntEnum · StrEnum (Python 3.11+) · Flag · IntFlag
Supported attributes
Design decisions
silently break the Rust ↔ Python name mapping.
The derive macro generates a per-type static to provide caching. Manual implementers must supply their own — this is documented with an example.
Known limitations / out of scope for this PR
(RFC: Opt-in `native_enum` mode — expose `#[pyclass]` enums as real `enum.Enum` subclasses #5991)).
means #[pyfunction] fn f(c: Color) -> Color requires an explicit derive.