[STUBGEN][RUST] Add the Rust backend with opaque object bindings - #738
Merged
tlopex merged 4 commits intoSep 4, 2026
Merged
Conversation
Seven-Streams
marked this pull request as ready for review
September 3, 2026 21:09
Signed-off-by: yuchuan <yuchuan.7streams@gmail.com>
Seven-Streams
force-pushed
the
main-dev/2026-09-03/stubgen_rust_opaque
branch
2 times, most recently
from
September 4, 2026 02:08
c31bbb8 to
fb6d4ae
Compare
A type whose parent is a builtin such as `ffi.IntEnum` used to embed the bare `tvm_ffi::Object` header, because the crate has no `<Leaf>Obj` for the builtin. `derive(Object)` derives `TYPE_DEPTH` from the embedded base, and `is_instance_of` indexes the ancestor table with that depth, so `demo.Color : ffi.IntEnum` got depth 1 instead of 3 and no subtype of it could be cast to `Color` from `Any`. The import section now defines a header-only stand-in per builtin ancestor below `ffi.Object` (`FfiEnumObj`, `FfiIntEnumObj`, ...), once per file, and the object embeds the last one, so the derived depth matches the registry. The mirrors carry no fields, `Deref` or upcast. Signed-off-by: yuchuan <yuchuan.7streams@gmail.com>
Signed-off-by: yuchuan <yuchuan.7streams@gmail.com>
tlopex
approved these changes
Sep 4, 2026
tlopex
pushed a commit
that referenced
this pull request
Sep 4, 2026
## Summary Attach the layout classifier (#730) to the Rust backend. A type whose layout the registry proves is now bound *complete*: a `#[repr(C)]` struct with every physical field at its reflected offset and width, a `const` size/alignment assertion, and a lossless allocator (`<Leaf>Obj::new` crate-private, `<Leaf>::new` public). Everything else keeps the opaque form of #738. Three new directives: `opaque` vetoes a reproducible layout, `upcast` adds a hand-written typed view, `custom-new` renames the generated allocator to `from_complete_fields` so a hand-written `new` can own the name. ## Motivation This is what tvm-rust-ext's `STUBGEN_FEEDBACK.md` asks for: ordinary data nodes allocated in Rust from their complete fields, polymorphic and unreflected-byte types kept opaque. Generating `ir.` / `tirx.` / `arith.` / `target.` from tvm-rust-ext's `libtvm_compiler.so` gives 131 types (98 complete, 33 opaque) that compile against the crate without edits; the differences from the hand-written bindings are exactly what the directives cover. ## Changes - `stub/layout.py`: `classify(..., unmirrored=...)` and the `no-mirror` reason, so types under a builtin parent (`ffi.Enum`, ...) stay opaque: their base is only a header-only stand-in. - `rust_generator/codegen.py`: classify each object with its ancestors; mirror fields (scalars by reflected width, `Optional<T>` as `Option<T>` or `tvm_ffi::Optional<T>` by payload, directive widths checked against the field size); render the complete struct and its allocators; `upcast` and `custom-new` handling. - `rust_generator/directives.py`, `consts.py`: the three directives and the width tables. - `rust_generator/utils.py`: a reflected field named `base` or `data` is spelled `base_` / `data_`, since those are the generated struct's own members (TVM has `tirx.Ramp.base`, `tirx.DeclBuffer.data`). - `examples/rust_stubgen/`: `IntPair` is now a plain data object, allocated from Rust with the generated `IntPair::new` and read back by C++. ## Testing `test_stubgen_rust.py` (36 cases), `test_stubgen.py`, `test_stub_layout.py`: 121 passed; ruff clean. The example regenerates an identical `mod.rs` and `cargo run` prints `a=1 b=2 kind=PairKind(1)` and `sum=3`. --------- Signed-off-by: yuchuan <yuchuan.7streams@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Register
--target rustintvm-ffi-stubgen. Every reflected object becomes an opaque Rust binding: a#[repr(C)]struct that embeds only its parent, a reference wrapper, read-onlyDeref,impl_object_upcast!along the ancestor chain, and one accessor per reflected field that reads through the C ABI getter (FieldGetter). The object's bytes are never reproduced, so the binding is correct for every registered type. Three directives (field,nullable,enum) shape the accessor types. A CMakeSTUB_TARGEToption and an example project with its generated module checked in round it out.Rebased on
mainafter #736; depends on nothing else.Motivation
This is the first Rust step of the series that replaces #609 (after the layout classifier in #730 and the directive channel in #736). Starting with the opaque form keeps "never generate a wrong layout" true from the first Rust PR. The follow-ups attach the classifier and upgrade the types whose layout the registry can prove to full field mirrors, then add allocators.
The opaque form is not a stopgap. Polymorphic types, types with unreflected bytes, and types the registry has no layout for stay opaque forever, and reading their fields through the reflected getter is the only correct option. It is exactly what tvm-rust-ext hand-writes today for
IterVar,Axis,TileLayout,BufferRegion,SourceName, andSource; this PR generates that shape from directives alone.Changes
python/tvm_ffi/stub/rust_generator/(new)consts.py: FFI-origin to Rust-type map,ffi.*to crate-root rewrite, declared directive names, keyword table.utils.py:RustUse/RustImports(per-fileusecollector plus directives),render_rust_type(returnsNonewhere the crate has no mirror),rust_ident.directives.py: grammar offield: K.f -> T,nullable: K.f,enum: K.f -> Name(i32) { A=0, ... }; malformed payloads are rejected with the line number.codegen.py:_ObjectRenderer(module-tree name resolution, accessors, the#[repr(transparent)]enum newtype withTryFrom<i64>,Deref, upcasts), plus the import section,--initscaffold andpub modstitching carried over from [FEAT][STUBGEN] Add Rust code generation backend #609.generator.py: theGeneratorprotocol adapter, declaring{import-object, field, nullable, enum}.stub/generator.py: registers"rust".cmake/Utils/Library.cmake:STUB_TARGET, passed as--target.examples/rust_stubgen/: a polymorphic C++ object with two registered global functions, a Rust program that constructs it through the FFI and reads it through the generated accessors, and the generatedmod.rswith anenumdirective.Decisions worth a look:
Union,Dict,List,tuple,void*, an unmapped bare origin) gets an accessor returningAny. Nothing is skipped; there is noUnsupportedTypeError.ffi.*parent has no generated<Leaf>Obj, so such a type embeds the object header and upcasts only to generated ancestors.usewhose leaf is already taken is spelled in full at the use site instead of failing.Not carried over from #609: the builder,
DerefMut,same_as/downcast, method generation, the offset warnings, and every Rust runtime change.Testing
tests/python/test_stubgen_rust.py(31 cases):usemodelling and collisions; value-type rendering including origins without a mirror; the three directive grammars and their rejections; goldens for a root object (header base,Anyfallback,r#type), same-module and cross-module derived objects, a builtin-parent object, and the hand-writtenIterVarof tvm-rust-ext reproduced from directives; import section,--initscaffold and module-tree stitching;_stage_3over a registered type with anenumdirective; the CLI in--initmode over thetesting.prefix, run twice to confirm idempotence. Full Python suite passes on the rebased branch.The example builds end to end:
cmake --buildregeneratesrust/src/generated/rust_stubgen/mod.rs, andcargo runprintsa=1 b=2 kind=PairKind(1)andsum=3against the C++ library.