Skip to content

Flat: a resolved model with direct access, not a stream of elements - #232

Open
milyin wants to merge 7 commits into
language-integrationfrom
flat-model
Open

Flat: a resolved model with direct access, not a stream of elements#232
milyin wants to merge 7 commits into
language-integrationfrom
flat-model

Conversation

@milyin

@milyin milyin commented Jul 29, 2026

Copy link
Copy Markdown
Owner

Stage L0.5 of #229. #227 produced a Vec<Element>; this makes it a model that
can be asked questions, and closes the flat API so its references mean something.

Targets language-integration, not main.

Why

Three things about L0's output were wrong for what consumes it next.

It was a stream, so nobody could ask it anything. Every later stage needs to
look an item up by name. L1's checklist carried "elements are indexed by name" as
an afterthought; it is a property of the model.

References were unresolved. A parameter naming Storage yielded
TypeKind::Named { id } and nothing more. Whether that name denoted anything
surfaced much later as ResolveError::Unresolved from the fixed-point resolver —
the "one source-language fact, several authorities" #211 exists to end. Names live
in one flat namespace, so resolution is decidable at parse time.

Element mixed two levels. Function | Struct | Variant | Enum | Const put
type declarations beside functions and constants.

And the module was called language while the thing it models is the flat API.

What

pub enum Element { Function(Function), Type(Type), Constant(Constant), Unsupported(Unsupported) }
pub enum Type    { Struct(Struct), Variant(Variant), Enum(Enum), Opaque(Opaque) }

let flat = Flat::builder().source(PREBINDGEN_OUT_DIR).build()?;
flat.function("storage_get");        // answers by name
flat.resolve(&id);                   // a reference leads to its declaration

The type reference becomes TypeRef, freeing Type for the declaration.
Struct::fields drops its OptionNone was the opaque case, which is now its
own entity.

Opacity is declared

#[prebindgen] pub type X = path; yields Type::Opaque. This reverses #227,
where a marked alias became Unsupported. It is how a foreign or crate-private
type gets a name in the flat API without any claim about its contents — and it
is the prerequisite for requiring references to resolve. A marked tuple struct
declares the same thing, unchanged from #227.

References resolve at parse time

A third pass walks every TypeRef — through Option, Vec, &, Result,
arrays, callback arguments and generic arguments alike. An item naming a type the
flat API does not declare becomes Element::Unsupported with
ItemError::UnresolvedType: deferred, not fatal, like every other refusal, so an
item no binding declares stays harmless. A path-qualified name gets its own
diagnosis, since #[prebindgen] pub type foreign::Option = .. is not a spelling
that exists.

Note the two resolutions stay distinct: this says a name denotes something,
while an adapter's resolver still decides whether it supplied a converter for it.

An out-parameter is a mode of borrowing

MaybeUninit is only meaningful behind a &mut, so uninitialized-ness is a
property of the borrow, not of a type. Ref carries the mode and absorbs it:

Ref { mode: RefMode, inner: Box<TypeRef> }
enum RefMode { Shared, Exclusive, Out }   // &T, &mut T, &mut MaybeUninit<T>

One axis, three values, and inner is always the borrowed value's type — so the
combinations that mean nothing at a boundary cannot be written down. Owned,
returned or in a field, uninitialized storage promises nothing; &MaybeUninit<T>
promises a readable T that may not be one. Both are refused, each naming why.

Out rather than Uninit because it names the boundary role every destination
language has (C's T *out) — the fact an adapter acts on. cbindgen was already
classifying MaybeUninit itself, so this moves that decision into the frontend per
#211; it is also the one foreign generic no alias can name, a generic alias being a
generic binder.

Closing the example flat APIs

Every referenced type had to become a declaration. Two idioms, chosen by what the
type is rather than by its Rust shape:

  • A handle gets a marked alias. Storage, the three callback handlers,
    Token, TokenGc, Summary, Archive, Report, EscapeProbe,
    StorageError, Calculator move into a private handles module with
    #[prebindgen] pub type X = handles::X;. The alias is transparent, so every
    signature still reads the same. Error in both crates was already an alias and
    needed only the attribute — the shape zenoh-flat's 26 re-exports will take.
  • A public newtype stays a marked struct. Millis, Celsius, Percent,
    Label are not handles: they cross by convert!, and covertest-helpers both
    constructs them and reads .0. An alias names the type but not the
    tuple-struct constructor, which lives in the value namespace — hiding them broke
    that downstream, which was the useful signal.

That second distinction also protects the goldens: a marked struct enters
registry.structs, and write.rs emits on_struct for any declared type there —
so marking the handles as structs would have changed generated output. A marked
alias is invisible to the registry (Item::Type no longer reaches passthrough
either, since an opaque declaration is not code to copy into a binding and its
target is routinely crate-private).

The closure is asserted, not assumed. covertest-kotlin's build script runs
Flat over both its sources and fails if anything is unsupported — the only place
the helper crate's references to perftest-flat's types resolve, since it cannot
mark them itself. Verified by deliberately unmarking Storage: the build fails,
naming all twelve referencing functions and the fix.

Verification

502 unit tests and 17 doctests pass on --all --all-features and on default
features; clippy clean; the boundary ledger unmoved. examples/regen-check.sh
reports byte-identical — the source crates were restructured and the model
rebuilt, and generation did not move. covertest-kotlin's JVM run passes all 47
sections.

Out of scope, and will break

zenoh-flat, zenoh-flat-c and zenoh-flat-jni are separate repos. Their 28
unmarked types (26 zenoh aliases plus Duration and Cow<'_, [u8]>) need the same
treatment before they parse. Cow<'_, [u8]> has no alias spelling, so
zbytes_to_bytes needs either the MaybeUninit treatment or a signature change.

Noticed while surveying: zenoh-flat-c/build.rs:539 declares zbytes_as_bytes,
which no longer exists in zenoh-flat (renamed zbytes_to_bytes) — that repo is
already stale against zenoh-flat HEAD.

Refs #211. Umbrella: #229.

🤖 Generated with Claude Code

milyin and others added 7 commits July 30, 2026 01:28
`core::language` modelled one thing and was named for another. What it parses is
the **flat API** — the single flat namespace a `#[prebindgen]` crate exports — so
`Language` becomes `Flat` and `api/core/language/` becomes `api/core/flat/`.

Mechanical, and separated from the model changes that follow so those arrive as a
readable diff. The boundary ledger's skipped-path constant and header move with
the directory; the count does not change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`Element` mixed two levels: `Function | Struct | Variant | Enum | Const` set type
declarations beside functions and constants, when the kinds a binding
distinguishes are a function, a type, and a constant. Types now group under
`Element::Type`, and the type *reference* — which held the name `Type` — becomes
`TypeRef`, so a declaration and a use site stop sharing a word.

`Opaque` becomes the entity for a type whose contents do not cross, and it
arrives two ways:

* `#[prebindgen] pub type X = path;` — this **reverses** #227, where a marked
  alias was `Unsupported`. It is now how a handle enters the flat API
  deliberately: a foreign or crate-private type gets a name here without any
  claim about its contents. That is what makes the API closable, and it is the
  prerequisite for requiring references to resolve.
* a marked tuple struct, whose fields no adapter has ever crossed — unchanged
  acceptance, now named for what it always meant.

So `Struct::fields` drops its `Option`. `None` was the opaque case; an empty list
now means the source wrote a struct with no fields, which is a different thing.

`MaybeUninit<T>` joins the grammar as `TypeKind::Uninit`. It is a boundary
concept — an out-parameter whose slot the caller supplies and the callee fills —
and cbindgen already models it as exactly that, so this moves a classification
out of the adapter and into the frontend, per #211. It is also the one foreign
generic that no alias could name, a generic alias being a generic binder.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two changes that belong together, because the first is what makes the second
decidable.

**The model is addressed by name, not iterated.** `FlatBuilder` collects and
`build` hands over a `Flat` — `function(name)`, `declared_type(name)`,
`constant(name)`, `element(name)`, plus iterators over each kind. Names are
unique across the whole model, so a name is a complete address, and that is what
every later stage wants: an adapter asks what a declared name *is* rather than
scanning a list. L1 carried this as a checklist bullet; it is really a property
of the model.

Two types rather than one, because a half-built model should not be the same type
as a resolved one — `Source::builder()` sets the precedent.

**References resolve at parse time.** A third pass walks every `TypeRef` — through
`Option`, `Vec`, `&`, `Result`, arrays, callback arguments and generic arguments
alike — and an item naming a type the flat API does not declare becomes
`Element::Unsupported` with `ItemError::UnresolvedType`. Deferred, not fatal, like
every other refusal: an item no binding declares stays harmless.

This is what a marked type alias bought. A dangling name previously surfaced far
downstream as an unresolved *converter*, from whichever adapter happened to look
first — the "one fact, several authorities" #211 exists to end. Note the two
remain distinct: resolution here says a name denotes something, while an
adapter's resolver still decides whether it supplied a converter for it.

A path-qualified name gets its own diagnosis, since `#[prebindgen] pub type
foreign::Option = ..` is not a spelling that exists — marked items live in one
flat namespace of bare names.

Also: `Item::Type` no longer reaches the registry's passthrough. An opaque
declaration states something about the API's surface and is not code to copy into
the binding; its target is routinely crate-private, so re-emitting it would not
compile.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every type a marked signature named had to become a declaration for resolution to
mean anything. Two idioms, chosen by what the type actually is rather than by its
Rust shape:

**A handle gets a marked alias.** `Storage`, the three callback handlers,
`Token`, `TokenGc`, `Summary`, `Archive`, `Report`, `EscapeProbe`,
`StorageError`, and example-flat's `Calculator` move into a private `handles`
module, with `#[prebindgen] pub type X = handles::X;` at the top level. The alias
is transparent, so every signature still says `Storage`. `Error` in both crates
was already an alias and only needed the attribute — which is exactly the shape
zenoh-flat's 26 zenoh re-exports will take.

**A public newtype stays a marked struct.** `Millis`, `Celsius`, `Percent` and
`Label` are not handles: they cross by `convert!`, and covertest-helpers both
constructs them and reads `.0`. Hiding them behind an alias broke that
downstream, which is the useful signal — a type alias names the type, not the
tuple-struct constructor, and the constructor lives in the value namespace where
the struct is defined. In-crate construction of the relocated handlers is
qualified `handles::PayloadHandler(..)` for the same reason.

Marking these as structs rather than aliases matters for a second reason: a
marked struct enters `registry.structs`, and `write.rs` emits `on_struct` for any
declared type there — so marking the *handles* as structs would have changed
generated output. The alias route is invisible to the registry, which is why the
goldens hold.

**And the closure is asserted, not assumed.** covertest-kotlin's build script now
runs `Flat` over both sources and fails if anything is unsupported. It is the
right place: only there do the helper crate's references to perftest-flat's types
resolve, since it cannot mark them itself. Verified by deliberately unmarking
`Storage` — the build fails naming all twelve referencing functions and the fix.

Generation is byte-identical (`examples/regen-check.sh`) and the JVM covertest
passes all 47 sections.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The model is now indexed and resolved, which takes two bullets off L1 — elements
indexed by name, and the entry point that shares one parser — and adds a
prerequisite L0 did not have: the flat API has to be closed for resolution to mean
anything.

Also records what is left open: zenoh-flat and its two consumers are separate
repos whose 28 unmarked types need the same treatment, and `Cow<'_, [u8]>` has no
alias spelling.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`clippy::ptr_arg` under CI's no-default-features run: the pass only mutates
elements in place, so a slice is the honest signature. My local checks used
--all-features only; CI runs three clippy configurations.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`TypeKind::Uninit` wrapped a type, but uninitialized-ness is a property of the
**borrow**: my own doc said `MaybeUninit` is "only meaningful behind a `&mut`",
which is the argument against modelling it as a type at all.

So `Ref` carries the mode, and the `MaybeUninit` is absorbed into it:

    Ref { mode: RefMode, inner: Box<TypeRef> }
    enum RefMode { Shared, Exclusive, Out }

`&T`, `&mut T`, `&mut MaybeUninit<T>` — one axis, three values, and `inner` is
always the borrowed *value's* type. One variant fewer than the `mutable` flag plus
a wrapper, and the combinations that mean nothing at a boundary can no longer be
written down: uninitialized storage owned, returned or in a field promises nothing
a destination language can use, and `&MaybeUninit<T>` promises a readable `T` that
may not be one. Both are refused, each naming why.

`Out` rather than `Uninit` because it names the boundary role every destination
language has — C's `T *out` — which is the fact an adapter acts on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant