Skip to content

Add native_enum: expose Rust enums as Python enum.Enum subclasses#6020

Open
wakita181009 wants to merge 7 commits intoPyO3:mainfrom
wakita181009:native-enum
Open

Add native_enum: expose Rust enums as Python enum.Enum subclasses#6020
wakita181009 wants to merge 7 commits intoPyO3:mainfrom
wakita181009:native-enum

Conversation

@wakita181009
Copy link
Copy Markdown

@wakita181009 wakita181009 commented May 7, 2026

Summary

Closes #2887. Adds #[py_native_enum] / #[derive(NativeEnum)] — an opt-in way to expose a fieldless Rust enum as a real Python enum.Enum subclass, without touching
the existing #[pyclass] enum path.

Discussed in #5991 (RFC + benchmarks).

How it works

The macro uses Python's functional enum API at runtime to construct the class:

equivalent to what the macro generates at first use:

Color = enum.Enum("Color", [("Red", auto()), ("Green", auto()), ("Blue", auto())])

Two layers of caching keep things both correct and fast, both backed by PyOnceLock:

  1. Base class cache — enum.Enum, enum.IntEnum, etc. are imported once per interpreter and reused across all enums that share the same base (base_cache.rs).
  2. Generated class cache — the Python class for each Rust enum type is stored in a per-type static PyOnceLock<Py>, constructed only once per interpreter
    session (generated by the derive macro).

Caching the generated class is essential for correctness, not just performance: without it, each to_py_member call would produce a new class object, and
isinstance(x, Color) would return False for values converted in a previous call.

This makes repeated conversions as cheap as a reference clone (~59 ns, on par with #[pyclass] enum at ~58 ns — see #5991 for the full benchmark table).

Usage

use pyo3::py_native_enum;

#[py_native_enum]                         // defaults to enum.Enum                                                                                                   
enum Color { Red, Green, Blue }
                                                                                                                                                                       
#[py_native_enum(base = "IntEnum")]                                                                                                                                  
enum Status { Active = 1, Inactive = 2 }                                                                                                                             
                                                                                                                                                                   
#[py_native_enum(base = "Flag")]                          
enum Permission {
   Read  = 1,                                                                                                                                                       
   Write = 2,                                                                                                                                                       
   Exec  = 4,                                                                                                                                                       
}

IntoPyObject and FromPyObject are auto-derived, so the enum works directly in #[pyfunction] signatures.

Supported bases

Enum (default) · IntEnum · StrEnum (Python 3.11+) · Flag · IntFlag

Supported attributes

  • Enum-level: base, rename, module, crate
  • Variant-level: rename, value

Design decisions

  • StrEnum with implicit values uses the variant name as-is rather than enum.auto(), because Python's StrEnum.generate_next_value lowercases the name, which would
    silently break the Rust ↔ Python name mapping.
    • py_enum_class is uncached by default in the trait. A static PyOnceLock cannot live in a trait default method (it would be shared across all implementing types).
      The derive macro generates a per-type static to provide caching. Manual implementers must supply their own — this is documented with an example.
  • No unsafe code — all Python C API interaction goes through PyO3's safe wrappers.
    • No unwrap/panic/expect in library code — all fallible operations use Result + ?.

Known limitations / out of scope for this PR

Introduce `#[py_native_enum]` / `#[derive(NativeEnum)]` macros that expose
a fieldless Rust enum to Python as a true `enum.Enum` subclass via the
functional API, without touching the C-API enum machinery (PyO3#991).

Runtime (`src/native_enum/`):
- `spec.rs`       – `NativeEnumSpec`, `NativeEnumBase`, `VariantValue`
- `base_cache.rs` – per-interpreter `PyOnceLock` cache for base classes
  (`enum.Enum`, `IntEnum`, `StrEnum`, `Flag`, `IntFlag`) and `enum.auto`
- `construct.rs`  – `build_native_enum`: builds a Python enum subclass
  from a spec without caching
- `trait_def.rs`  – `NativeEnum` trait with `py_enum_class`,
  `to_py_member`, `from_py_member`

Macro (`pyo3-macros-backend/src/native_enum.rs`):
- Parses enum-level attrs (`base`, `rename`, `module`) and variant-level
  attrs (`rename`, `value`); supports integer discriminants
- Caches the Python class in a function-local `static PyOnceLock<Py<PyType>>`
  in the generated `py_enum_class`, constructing it once per interpreter session
- `from_py_member` checks `isinstance(obj, cached_class)` for type safety
- Auto-derives `IntoPyObject`, `IntoPyObject for &T`, `FromPyObject`

Public API additions to `pyo3-macros/src/lib.rs`:
- `#[proc_macro_derive(NativeEnum, attributes(native_enum))]`
- `#[proc_macro_attribute] py_native_enum`
- tests/test_native_enum.rs: Rust integration tests covering the full
  enum protocol (isinstance, name/value, len/iter/contains, lookup by
  name and value), class and member identity, IntoPyObject/FromPyObject
  roundtrips, all five base classes (Enum/IntEnum/Flag/IntFlag/StrEnum),
  rename at class and variant level, module attribute, qualname via
  direct build_native_enum call, and VariantValue::Str

- pytests/src/native_enums.rs + pytests/tests/test_native_enums.py:
  Python-side tests for the same surface area, exposing Color/Status/
  Permission/Bits/Size to Python and verifying enum protocol from pytest

- tests/ui/invalid_native_enum_base.rs: UI test for unknown base name
- tests/ui/native_enum_generic.rs: UI test for generic/lifetime params
  (also adds the validation logic in pyo3-macros-backend)
wakita181009 and others added 3 commits May 8, 2026 01:18
- Apply ruff formatting to test_native_enums.py (line wrapping)
- Disambiguate `pyclass` doc link with `macro@pyclass` prefix
Proc-macro crates cannot resolve intra-doc links to other crates.
Replace `pyo3::native_enum::NativeEnum` with a docs.rs URL.
The test-introspection CI job generates .pyi stubs and compares
them against expected files in pytests/stubs/. The native_enums
module was missing its stub file.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented May 7, 2026

Merging this PR will improve performance by 12.89%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 104 untouched benchmarks
🆕 6 new benchmarks
⏩ 1 skipped benchmark1

Performance Changes

Benchmark BASE HEAD Efficiency
🆕 native_enum_from_py_member N/A 9.1 µs N/A
🆕 native_enum_int_enum_class N/A 423.9 ns N/A
🆕 native_enum_int_enum_from_py_member N/A 10.4 µs N/A
🆕 native_enum_int_enum_to_py_member N/A 2.5 µs N/A
🆕 native_enum_py_enum_class N/A 394.7 ns N/A
🆕 native_enum_to_py_member N/A 2.4 µs N/A
bench_pyclass_create 4.6 µs 4.1 µs +12.89%

Comparing wakita181009:native-enum (c0f95ea) with main (f57bda7)

Open in CodSpeed

Footnotes

  1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports.

`&str: FromPyObject` is gated by `#[cfg(any(Py_3_10, not(Py_LIMITED_API)))]`,
making it unavailable under abi3-py39. Use `String` which is always available.
Align with `#[pyclass(name = "...")]` convention. Both enum-level and
variant-level attributes now use `name` instead of `rename`. Remove
unused `kw::rename` custom keyword.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

enums generated by Pyo3 don't actually subclass Python's Enum

1 participant