Schema for user-defined note types (WIT + Wasm)

[greenhat]

#814
#1204

Problem

A note's storage (protocol NoteStorage, formerly NoteInputs) is a bare Vec<Felt>.
The Rust SDK gives it a type — #[note] struct P2idNote { target_account_id: AccountId } — and derives the on-chain decoder from it.
But the off-chain side that creates the note has no access to that type: today every consumer hand-mirrors the felt layout (to_core_felts, P2idNoteStorage, p2ide's "must match the MASM" index constants), and nothing catches drift.

Goals

1. Single source of truth: the #[note] struct. The macro that derives the on-chain decoder also emits the schema; they cannot drift.
2. The schema travels inside the .masp package, so any consumer — Rust client, CLI, TS wallet, explorer — can, given only the package:
   • build NoteStorage from named, human-readable inputs (strings), with validation;
   • decode a received note's storage back into named, displayable values.
3. A Rust off-chain macro generates the exact typed struct (with host-side protocol types) plus encode/decode from the embedded schema.
4. Hybrid split: type definition lives in the format; string parsing / validation / display lives in code, bound by type identity.

Design

The schema is WIT: the #[note] macro renders the storage struct (and its nested #[export_type] types) as a WIT record/variant document and embeds it in the package. WIT contributes only the type structure and identity — the felt layout is defined by a normative felt-repr mapping over WIT type constructs (the same To/FromFeltRepr rules the derived on-chain decoder already implements). String parsing, validation, and display bind off-chain to fully-qualified WIT type names (miden:base/core-types.account-id) through code — the hybrid: types in data, behavior in code. That code comes in two forms: native codecs for protocol-defined leaf types, shipped with the schema reader, and — for user-defined types — an optional author-written codec bundled in the package as a sandboxed host-wasm component (structural field-by-field UX always works without it).

End to end: the #[note] expansion renders the schema WIT (the same expansion that derives the decoder, so drift is impossible) into a Wasm custom section; the frontend carries it into the .masp as the note_storage_schema section — the exact pipeline AccountComponentMetadata already rides — beside the optional note_codec component section; consumers read it through the runtime API or the from_project!/from_package! bindings macros.

The schema document

For the p2id example, the section contains (self-contained, multi-package WIT text):

package miden:base@1.0.0 {   // bundled with the SDK; the emitter embeds the referenced subset
    interface core-types {
        record felt { inner: f32 }
        record word { a: felt, b: felt, c: felt, d: felt }
        record account-id { prefix: felt, suffix: felt }
        // … only the transitively referenced subset
    }
}

package example:p2id-note-schema@0.1.0 {
    interface note-storage {
        use miden:base/core-types@1.0.0.{account-id};

        /// P2ID note storage.
        record p2id-note {
            /// The account allowed to consume this note.
            target-account-id: account-id,
        }

        /// Marks the root type of the note's storage.
        type storage = p2id-note;
    }
}

Conventions:

• Interface is named note-storage; the root is marked by the alias type storage = <root> so nested helper records can coexist. The package name derives from the note's component package (<ns>:<name>-schema@<version>).
• Dependency packages (miden:base core types, subset-pruned or whole) are embedded in the same document, so the section is resolvable with zero external inputs. The miden:base definitions come from the WIT bundled with the SDK (sdk/base-macros/wit/miden.wit, embedded as SDK_WIT_SOURCE — the same copy the macros already resolve core types against), so the emitter copies from the one canonical source rather than restating anything. Rust doc comments on the struct/fields are carried into WIT doc comments — wallet-facing descriptions for free.
• The section is plain UTF-8 WIT text: miden pkg inspect can print it verbatim; wit-parser (Rust) and the JS component tooling parse it as-is.

Off-chain consumption

Runtime API (protocol repo, beside NoteStorage/NoteScript)

let schema = NoteStorageSchema::from_package(&package)?;   // wit-parser resolve + surface validation

// build direction (CLI / wallet / tests)
let mut b = schema.builder();
b.set("target-account-id", "mtst1qz…")?;    // registry: account-id ⇒ bech32|hex → [prefix, suffix]
let storage: NoteStorage = b.build()?;       // completeness + range validation
let recipient = NoteRecipient::new(serial, NoteScript::from_package(&package)?, storage);

// decode direction (displaying a received note)
let view = schema.decode(note.storage())?;   // named value tree; Display via registry

This answers "how is a Note constructed from user strings": package → (NoteScript, NoteStorageSchema); schema + string map → storage; plus serial/assets/metadata → Note. Tag derivation and asset expectations are per-note logic and stay in code (standards keep their typed builders); the schema deliberately doesn't model them in v1.

Builder and view key on the schema's field names as written in the WIT (kebab-case; snake_case accepted and normalized). The string-codec registry keys on WIT fqns and reuses existing protocol parsers (AccountId::parse hex|bech32, Word::parse, TokenSymbol::new, decimal/hex felts). Multi-felt-capable:

trait ConsumerTypeCodec {
    fn parse(&self, s: &str) -> Result<Vec<Felt>, _>;      // must match the type's felt width
    fn display(&self, felts: &[Felt]) -> String;
    fn validate(&self, felts: &[Felt]) -> Result<(), _>;
}

Only consumers see this trait, and it is felt-level and object-safe by necessity: registry entries must be uniform over types the consumer's program has never heard of. Its implementors are the native standard codecs (tier 1 below) and the adapter wrapping a bundled codec component (tier 2). The miden:note-codec world is exactly this trait's shape crossing the component boundary, plus a ty discriminator so one component covers all of its package's types. The author-side face is a different, typed trait (AuthorTypeCodec, see Author DX) — the boundary erases types, so the two sides cannot share one trait.

Where codecs live

Codecs are keyed by WIT fqn strings, not Rust types, so the orphan rule never constrains placement:

1. Standard leaf codecs — for types defined by the protocol's own WIT (felt, word, account-id, asset, token-symbol, …). These ship with the schema reader in the protocol repo as thin adapters over parsers it already owns (AccountId::parse, Word::parse, TokenSymbol::new, decimal/hex felts). Protocol-owned types are the only codecs that may live centrally.

2. Custom-type codecs — bundled in the .masp. The whole point of the schema is user-defined types, and their string syntax/validation is author knowledge, so the code must travel with the package: as a sandboxed host-side wasm component (ordinary wasm executed by the consumer — wasmtime in Rust clients, jco/browser wasm in TS wallets — not Miden-VM code) in a dedicated section (note_codec), implementing a standard world:

   package miden:note-codec@1.0.0;
   
   world note-codec {
       /// Host-context felt: the canonical field-element value (< p), carried as u64.
       /// Deliberately NOT miden:base/core-types.felt — that record's `inner: f32` is a
       /// type-level fiction only midenc honors (it rewrites f32 to the VM's native
       /// 64-bit element; no float ever exists at runtime). This component runs on
       /// standard runtimes (wasmtime, jco), where f32 is a real IEEE 754 float:
       /// 32 bits too small for a Goldilocks element, and subject to NaN
       /// canonicalization on lift/lower.
       type felt = u64;
   
       // WIT fqns this component covers
       export supported-types: func() -> list<string>;
       export parse: func(ty: string, input: string) -> result<list<felt>, string>;
       export display: func(ty: string, value: list<felt>) -> string;
       export validate: func(ty: string, value: list<felt>) -> result<_, string>;
   }

Author DX: a sibling codec crate. The note crate is hostile territory for codec code — no_std, nightly feature gates, SDK macros emitting guest boilerplate (#[panic_handler], #[global_allocator] — which would collide with std), while a codec wants String and ordinary parsing crates. So the codec lives in a separate host crate next to the note crate, and the note crate has zero codec awareness: no features, no cfgs, and nothing to strip from the MASM build because nothing codec-related is ever part of that compilation.

examples/dex-note/          # guest: no_std + SDK + #[note] — unchanged, codec-agnostic
examples/dex-note-codec/    # host: ordinary std crate, plain cargo build

(The example is a hypothetical dex note with custom types — p2id needs no codec crate at all, since account-id is a standard leaf.) The codec crate builds with plain cargo for the sandbox wasm target (no miden cfg) and depends only on the miden-note-codec runtime crate (the AuthorTypeCodec trait, from_project!/from_package!/#[note_codec]/export_codecs!, world glue via wit-bindgen, boundary canonicality checks) — transitively just miden-field/miden-field-repr in native mode; not the note crate, not the SDK, not miden-protocol.

What makes this work without type duplication: the low-level type stack is already dual-backed. miden-field's Felt is the f32 hijack only under cfg(all(target_family = "wasm", miden)) and a real Felt(Goldilocks) otherwise, and miden-field-repr is generic over it — so a normal build gets correct felt semantics plus the very same To/FromFeltRepr machinery the guest uses. The typed view of the storage struct is regenerated from the schema rather than imported from the note crate:

// in dex-note-codec
miden_note_codec::from_project!("../dex-note");
// ↑ path to the note PROJECT directory — artifact and profile discovery is the macro's job
// → host-profile root storage struct + all custom component types in its closure
//   (native Felt fields, felt-repr derives, a WIT-fqn const per type), crate-local;
//   also records the schema — including which type is the root — for export_codecs!

#[note_codec]                            // opt-in: this type gets custom string syntax
impl AuthorTypeCodec for LimitPrice {         // generated type is crate-local ⇒ orphan rule satisfied
    fn parse(s: &str) -> Result<Self, String> { … }   // String, format!, parser deps — all fine here
    fn display(&self) -> String { … }
    fn validate(&self) -> Result<(), String> { … }
}

miden_note_codec::export_codecs!();      // no type list — covers the #[note_codec]-marked impls

Artifact resolution — distinct macros, not keyword modes. from_project!("../dex-note") takes the note project directory, never an artifact path — build profiles stay out of source code; the macro finds the freshest built package across profiles under the note's target dir (sound because the schema is profile-invariant: it derives from the struct definition, not from codegen), and a missing artifact is a compile error naming the fix. from_package!("vendor/dex_note.masp") reads an exact artifact path for packages that are obtained rather than built in-tree; from_registry!("acme:dex-note@1.2") is the natural future extension.

AuthorTypeCodec is the author-side face of ConsumerTypeCodec — one codec concept, two representations on either side of the type-erasing component boundary. Inside the component the author's Rust types exist, so the trait is typed (parse → Self, display(&self)) and felts never appear in author code. export_codecs! lowers it to the world's felt-level shape mechanically (fqn dispatch + the generated To/FromFeltRepr impls). Outside, the consumer wraps the instantiated component in an adapter implementing ConsumerTypeCodec per reported fqn. The author never sees ConsumerTypeCodec; the consumer never sees AuthorTypeCodec.

Single source of truth is preserved transitively — guest struct → schema in the package → generated host struct — so layout drift surfaces as compile errors in the codec crate on rebuild, never as silent disagreement.

Orchestration. miden-project.toml points at the codec crate (e.g. [note] codec-crate = "../dex-note-codec"); cargo miden build sequences: guest build (schema section attached) → codec build + componentize → attach bytes as the note_codec section (cargo-miden already orchestrates multi-artifact builds for dependency .masp materialization). The codec crate is entirely optional — omit it and the package ships without the section, falling back to structural UX.

Codegen macro (typed Rust clients)

// the dex-note example again — its schema contains custom types
miden_note_bindings::from_project!("../dex-note");   // or from_package!("vendor/dex_note.masp")
// expands to:
pub struct DexNote {
    pub target: miden_protocol::account::AccountId,  // standard leaf → protocol type
    pub price: LimitPrice,                           // custom type → generated structurally
}
pub struct LimitPrice { pub num: u64, pub den: u64 }
impl DexNote {
    pub fn to_note_storage(&self) -> NoteStorage;
    pub fn from_note_storage(&NoteStorage) -> Result<Self, _>;
    pub fn from_str_values(values: &BTreeMap<String, String>, codecs: &CodecRegistry) -> Result<Self, _>;
    pub fn validate_with(&self, codecs: &CodecRegistry) -> Result<(), _>;  // advisory; the script is the authority
}
impl LimitPrice {
    pub fn display_with(&self, codecs: &CodecRegistry) -> String;          // structural fallback without codec
}

// build direction, custom codec included:
let mut codecs = CodecRegistry::default();     // native codecs for the standard leaves
codecs.load_from_package(&package)?;           // instantiates the bundled note_codec component
let note = DexNote::from_str_values(&values, &codecs)?;
// "target" parses natively (bech32 → AccountId);
// "price" = "1.5 MID/BTC" routes to the component: parse("…/limit-price", …) → felts → LimitPrice

// decode direction (received note): same registry, opposite direction
let incoming = DexNote::from_note_storage(note_storage)?;
incoming.validate_with(&codecs)?;                    // custom fields → codec validate(fqn, felts)
let price = incoming.price.display_with(&codecs);    // codec display(fqn, felts) → "1.5 MID/BTC"

All three codec calls are one mechanism: generated code encodes the field to felts via its felt-repr impl and invokes the registry entry by fqn — parse inbound, display/validate outbound. Without a codec entry, display falls back to structural rendering and validate to structural checks (range/canonicality); the note script remains the only authoritative validator either way. For a schema with no custom types (p2id), the macro drops the codecs parameters entirely — the schema shapes the generated API. And a consumer that skips the codec still constructs LimitPrice { num, den } field-by-field; the registry serves only the human-facing paths (strings in, strings out).

/cc @bobbinth