Skip to content

[STUBGEN][RUST] Add the Rust backend with opaque object bindings - #738

Merged
tlopex merged 4 commits into
apache:mainfrom
Seven-Streams:main-dev/2026-09-03/stubgen_rust_opaque
Sep 4, 2026
Merged

[STUBGEN][RUST] Add the Rust backend with opaque object bindings#738
tlopex merged 4 commits into
apache:mainfrom
Seven-Streams:main-dev/2026-09-03/stubgen_rust_opaque

Conversation

@Seven-Streams

@Seven-Streams Seven-Streams commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Register --target rust in tvm-ffi-stubgen. Every reflected object becomes an opaque Rust binding: a #[repr(C)] struct that embeds only its parent, a reference wrapper, read-only Deref, impl_object_upcast! along the ancestor chain, and one accessor per reflected field that reads through the C ABI getter (FieldGetter). The object's bytes are never reproduced, so the binding is correct for every registered type. Three directives (field, nullable, enum) shape the accessor types. A CMake STUB_TARGET option and an example project with its generated module checked in round it out.

Rebased on main after #736; depends on nothing else.

Motivation

This is the first Rust step of the series that replaces #609 (after the layout classifier in #730 and the directive channel in #736). Starting with the opaque form keeps "never generate a wrong layout" true from the first Rust PR. The follow-ups attach the classifier and upgrade the types whose layout the registry can prove to full field mirrors, then add allocators.

The opaque form is not a stopgap. Polymorphic types, types with unreflected bytes, and types the registry has no layout for stay opaque forever, and reading their fields through the reflected getter is the only correct option. It is exactly what tvm-rust-ext hand-writes today for IterVar, Axis, TileLayout, BufferRegion, SourceName, and Source; this PR generates that shape from directives alone.

Changes

  • python/tvm_ffi/stub/rust_generator/ (new)
    • consts.py: FFI-origin to Rust-type map, ffi.* to crate-root rewrite, declared directive names, keyword table.
    • utils.py: RustUse / RustImports (per-file use collector plus directives), render_rust_type (returns None where the crate has no mirror), rust_ident.
    • directives.py: grammar of field: K.f -> T, nullable: K.f, enum: K.f -> Name(i32) { A=0, ... }; malformed payloads are rejected with the line number.
    • codegen.py: _ObjectRenderer (module-tree name resolution, accessors, the #[repr(transparent)] enum newtype with TryFrom<i64>, Deref, upcasts), plus the import section, --init scaffold and pub mod stitching carried over from [FEAT][STUBGEN] Add Rust code generation backend #609.
    • generator.py: the Generator protocol adapter, declaring {import-object, field, nullable, enum}.
  • stub/generator.py: registers "rust".
  • cmake/Utils/Library.cmake: STUB_TARGET, passed as --target.
  • examples/rust_stubgen/: a polymorphic C++ object with two registered global functions, a Rust program that constructs it through the FFI and reads it through the generated accessors, and the generated mod.rs with an enum directive.

Decisions worth a look:

  • A field without a Rust mirror (Union, Dict, List, tuple, void*, an unmapped bare origin) gets an accessor returning Any. Nothing is skipped; there is no UnsupportedTypeError.
  • A builtin ffi.* parent has no generated <Leaf>Obj, so such a type embeds the object header and upcasts only to generated ancestors.
  • A use whose leaf is already taken is spelled in full at the use site instead of failing.
  • No allocator is generated: construction goes through the registered global functions, as in the example.

Not carried over from #609: the builder, DerefMut, same_as / downcast, method generation, the offset warnings, and every Rust runtime change.

Testing

tests/python/test_stubgen_rust.py (31 cases): use modelling and collisions; value-type rendering including origins without a mirror; the three directive grammars and their rejections; goldens for a root object (header base, Any fallback, r#type), same-module and cross-module derived objects, a builtin-parent object, and the hand-written IterVar of tvm-rust-ext reproduced from directives; import section, --init scaffold and module-tree stitching; _stage_3 over a registered type with an enum directive; the CLI in --init mode over the testing. prefix, run twice to confirm idempotence. Full Python suite passes on the rebased branch.

The example builds end to end: cmake --build regenerates rust/src/generated/rust_stubgen/mod.rs, and cargo run prints a=1 b=2 kind=PairKind(1) and sum=3 against the C++ library.

@Seven-Streams
Seven-Streams marked this pull request as ready for review September 3, 2026 21:09
Signed-off-by: yuchuan <yuchuan.7streams@gmail.com>
@Seven-Streams
Seven-Streams force-pushed the main-dev/2026-09-03/stubgen_rust_opaque branch 2 times, most recently from c31bbb8 to fb6d4ae Compare September 4, 2026 02:08
A type whose parent is a builtin such as `ffi.IntEnum` used to embed the
bare `tvm_ffi::Object` header, because the crate has no `<Leaf>Obj` for
the builtin. `derive(Object)` derives `TYPE_DEPTH` from the embedded base,
and `is_instance_of` indexes the ancestor table with that depth, so
`demo.Color : ffi.IntEnum` got depth 1 instead of 3 and no subtype of it
could be cast to `Color` from `Any`.

The import section now defines a header-only stand-in per builtin
ancestor below `ffi.Object` (`FfiEnumObj`, `FfiIntEnumObj`, ...), once
per file, and the object embeds the last one, so the derived depth
matches the registry. The mirrors carry no fields, `Deref` or upcast.

Signed-off-by: yuchuan <yuchuan.7streams@gmail.com>
Signed-off-by: yuchuan <yuchuan.7streams@gmail.com>
Signed-off-by: yuchuan <yuchuan.7streams@gmail.com>
@tlopex
tlopex merged commit 94133a5 into apache:main Sep 4, 2026
9 checks passed
tlopex pushed a commit that referenced this pull request Sep 4, 2026
## Summary

Attach the layout classifier (#730) to the Rust backend. A type whose
layout the registry proves is now bound *complete*: a `#[repr(C)]`
struct with every physical field at its reflected offset and width, a
`const` size/alignment assertion, and a lossless allocator
(`<Leaf>Obj::new` crate-private, `<Leaf>::new` public). Everything else
keeps the opaque form of #738. Three new directives: `opaque` vetoes a
reproducible layout, `upcast` adds a hand-written typed view,
`custom-new` renames the generated allocator to `from_complete_fields`
so a hand-written `new` can own the name.

## Motivation

This is what tvm-rust-ext's `STUBGEN_FEEDBACK.md` asks for: ordinary
data nodes allocated in Rust from their complete fields, polymorphic and
unreflected-byte types kept opaque. Generating `ir.` / `tirx.` /
`arith.` / `target.` from tvm-rust-ext's `libtvm_compiler.so` gives 131
types (98 complete, 33 opaque) that compile against the crate without
edits; the differences from the hand-written bindings are exactly what
the directives cover.

## Changes

- `stub/layout.py`: `classify(..., unmirrored=...)` and the `no-mirror`
reason, so types under a builtin parent (`ffi.Enum`, ...) stay opaque:
their base is only a header-only stand-in.
- `rust_generator/codegen.py`: classify each object with its ancestors;
mirror fields (scalars by reflected width, `Optional<T>` as `Option<T>`
or `tvm_ffi::Optional<T>` by payload, directive widths checked against
the field size); render the complete struct and its allocators; `upcast`
and `custom-new` handling.
- `rust_generator/directives.py`, `consts.py`: the three directives and
the width tables.
- `rust_generator/utils.py`: a reflected field named `base` or `data` is
spelled `base_` / `data_`, since those are the generated struct's own
members (TVM has `tirx.Ramp.base`, `tirx.DeclBuffer.data`).
- `examples/rust_stubgen/`: `IntPair` is now a plain data object,
allocated from Rust with the generated `IntPair::new` and read back by
C++.

## Testing

`test_stubgen_rust.py` (36 cases), `test_stubgen.py`,
`test_stub_layout.py`: 121 passed; ruff clean. The example regenerates
an identical `mod.rs` and `cargo run` prints `a=1 b=2 kind=PairKind(1)`
and `sum=3`.

---------

Signed-off-by: yuchuan <yuchuan.7streams@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants