diff --git a/CLAUDE.md b/CLAUDE.md index 8d62539..47f5769 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -15,7 +15,8 @@ Standard Go tooling applies once code exists: `go build ./...`, `go test ./...`, ## What Morphic is A spec-to-SDK compiler: any API spec (OpenAPI, Swagger 2.0, TypeSpec, Smithy, GraphQL, AsyncAPI, -Protobuf) → **one spec-agnostic intermediate representation (IR)** → idiomatic SDKs and docs. +Protobuf, Erlang/OTP module specs) → **one spec-agnostic intermediate representation (IR)** → +idiomatic SDKs and docs. Pipeline: **frontends** (spec → IR) → **IR passes** (IR → IR) → **backends** (IR → artifacts). ## The documents are the spec — read them first diff --git a/docs/architecture.md b/docs/architecture.md index 4ac7f8e..2d7401e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,8 +1,8 @@ # Morphic Architecture Morphic turns any API specification — OpenAPI/Swagger, TypeSpec, Smithy, GraphQL, AsyncAPI, -Protobuf — into idiomatic SDKs and docs through a single spec-agnostic intermediate -representation (IR). This document defines the pipeline, the package layout, and the contracts +Protobuf, Erlang/OTP module specs — into idiomatic SDKs and docs through a single spec-agnostic +intermediate representation (IR). This document defines the pipeline, the package layout, and the contracts between stages. The IR itself is specified in [`ir-design.md`](./ir-design.md). ``` @@ -14,6 +14,7 @@ between stages. The IR itself is specified in [`ir-design.md`](./ir-design.md). GraphQL ──▶ └───────────────────────────────────────┘ AsyncAPI │ Protobuf ▼ + Erlang/OTP ┌── IR passes (IR → IR) ──┐ │ validate · link · dedup │ │ filter · version-slice │ @@ -86,7 +87,7 @@ Internal phases every frontend follows (each format implements them its own way) Frontends are registered in a registry keyed by detected format; the engine sniffs the source format and dispatches. Milestone 1 ships the OpenAPI 3.x frontend only; the frontend registry, -provenance model, and IR are built for all seven from day one. +provenance model, and IR are built for all eight from day one. ### 2.2 IR passes (IR → IR) @@ -102,11 +103,22 @@ can enable: - **dedup** — structurally identical anonymous types are merged (by content hash), with ID aliases retained so provenance survives. - **filter** — include/exclude operations and types by pattern (Kiota-style path filtering), - followed by reachability trimming of orphaned types. + followed by reachability trimming of orphaned types. Filtering serves *surface reduction* + (a smaller SDK). Regenerating only one service of an existing SDK is a different problem and + never uses a filtered document: global decisions (dedup, shared files, naming) must stay + byte-identical across scoped runs, so the backend consumes the full document plus a scope + option and gates emission itself (a lesson oagen recorded after trying the filtered route). - **version-slice** — project a document carrying availability metadata into a concrete per-version snapshot (the TypeSpec versioning model: timeline stored, snapshot consumed). - **overlay** — user-supplied IR patches (rename hints, pagination declarations for specs that - can't express them, doc overrides) applied as data, not code. + can't express them, doc overrides) applied as data, not code. IR overlays are language-neutral + and each entry carries provenance (user-authored vs tool-inferred, mirroring the `Inferred` + marker) so automated overlay-generation loops stay auditable. Two related hooks live + deliberately elsewhere: *source-document* patching (e.g. the OpenAPI Overlay spec) is a + frontend option applied before lowering — some fixes must land before naming/hoisting + heuristics consume the broken shape — and *per-target-language* naming/compat overlays are a + backend input keyed by IR ID, so one IR document drives different compat baselines per + language. Passes operate on the IR only; they know nothing about source formats or target languages. @@ -117,7 +129,11 @@ except for the boundary they impose on the IR; the internal shape mirrors what K converged on independently: 1. **Plan** — compute language-*neutral* per-operation and per-type decisions once - (is-paginated, body presence, idempotency, error taxonomy) so templates contain no policy. + (is-paginated, body presence, idempotency, error taxonomy, primary-content/response + selection, return-shape classification (model / void / list-of-models with elementwise + deserialization), pagination item-type unwrap, parameter-passing shape (positional vs + options-bag)) so templates contain no policy. This list is the decision set oagen's + emitters demonstrably needed, computed once and shared across all languages. 2. **Refine** — per-language lowering: reserved words, casing, union representation strategy (native union vs sealed interface vs wrapper class), enum strategy for open enums, interface extraction. Everything a refiner needs must exist un-lowered in the IR — this is the IR's @@ -127,6 +143,24 @@ converged on independently: Language-specific naming (casing, reserved words, import layout) lives here exclusively. The IR carries source names, wire names, and naming hints — never camelCase/PascalCase renderings. +Two further backend-side stages are named here so their obligations shape the contracts, even +though both are post-milestone scope: + +- **Write/integrate** — regenerating into a live repo alongside hand-written code requires: + deterministic file paths, a generation manifest (spec/emitter/config hashes, sorted file + list, and an entity → generated-symbol map **keyed by IR stable IDs**, which survive path + and name churn), file-header provenance that gates pruning (never delete a file lacking the + generated-by header), ignore-region markers for hand-written islands, and additive-only + merging. Additive-only writers also need a staleness check — (previous-revision entities − + current entities) ∩ files-on-disk — to surface dead code that pruning cannot touch. +- **Surface verification** — per-language extractors project existing SDK source into a + neutral API-surface model; the same projection of generated output is diffed against it. + Change records are *neutral*; breaking/additive severity is an injectable per-language + policy function (parameter names are public API in PHP, arity in Go, almost nothing in + JS), consistent with principle 6. Behavioral changes (defaults) are a separate channel + from structural ones. Backends propagate IR IDs into manifests and reports so findings + correlate across languages without name-matching heuristics. + ### 2.4 Runtime/SDK policy is a separate input Retry, timeout, telemetry, error-class taxonomy, user-agent — the *behavioral* configuration of @@ -134,6 +168,18 @@ generated SDKs — is a backend input alongside the IR, not part of it. The IR d policy describes the SDK. (oagen embeds both in one root; we keep the trees separate so the same IR document can drive SDKs, docs, and mock servers without dragging SDK opinions along.) +The canonical policy-input vocabulary, taken from oagen's production `SdkBehavior` (the best +real-world enumeration available): **retry** (retryable status codes, max attempts, full backoff +strategy with jitter), **timeouts** (defaults + env override), **error taxonomy** (status → +logical exception-kind map, client/server catch-alls, doc-URL template), **telemetry** (request- +ID and client-telemetry headers), **logging** (a closed lifecycle-event list), **user-agent** +construction (identifier template, app-info enrichment), **idempotency injection** (header name, +auto-generate rules), **pagination pacing** (auto-page delay), and **request guards** (option +keys that must not appear as params — misuse detection). Delivery model: canonical defaults + +deep-partial per-backend/per-project overrides. Precedence rule: *declared IR facts win over +policy defaults* — `ErrorCase.Retryable/Fault` and `Operation.Idempotency` come from the spec; +policy fills in where the spec is silent, never the reverse. + ## 3. Go package layout ``` @@ -143,7 +189,7 @@ morphic/ ├── frontend/ # Layer 1 — frontend contract + registry. │ ├── openapi/ # OpenAPI 3.x → IR (milestone 1). │ ├── swagger/ # 2.0 lift → openapi frontend (future). -│ ├── typespec/ smithy/ graphql/ asyncapi/ protobuf/ (future) +│ ├── typespec/ smithy/ graphql/ asyncapi/ protobuf/ otp/ (future) ├── pass/ # Layer 1 — IR → IR passes (validate, dedup, filter, slice, overlay). ├── backend/ # Layer 2 — backend contract, plan layer, registry (future). ├── engine/ # Layer 3 — orchestration: sniff format, run frontend, passes, backends. @@ -164,7 +210,10 @@ Every IR node carries a `Provenance` (source format, file, JSON pointer or line/ source name, and an `Inferred` marker naming the heuristic when applicable). Every stage returns `[]Diagnostic{Severity, Code, Message, Provenance}`. Codes are stable strings (`openapi/unresolved-ref`, `ir/dangling-type-ref`, `pass/discriminator-missing-variant`) so CI -can allowlist. Nothing in the pipeline writes to stderr; the CLI renders diagnostics. +can allowlist. A mature allowlist entry is keyed by (diagnostic code × entity ID), requires a +human rationale, supports expiry tied to a release, and rejects wildcards — narrowness is +validated, so approvals cannot rot into blanket suppressions. Nothing in the pipeline writes to +stderr; the CLI renders diagnostics. ## 5. Testing strategy @@ -176,6 +225,11 @@ can allowlist. Nothing in the pipeline writes to stderr; the CLI renders diagnos - **Round-trip property**: `parse → serialize → deserialize → deep-equal` for every corpus document. - **Architecture test**: import-graph assertions for the layering rules above. +- **Wire-conformance harness** (from milestone 3): expected request shapes (method, path, + query, body keys) are derived *from the IR alone* — offline, deterministic, shared across + every language backend; generated SDKs run under HTTP interception and the requests are + diffed. Request-side mismatches block; response-side mismatches inform. The decisive test of + a generated SDK is the bytes it puts on the wire, not whether it compiles. ## 6. Milestones @@ -187,5 +241,5 @@ can allowlist. Nothing in the pipeline writes to stderr; the CLI renders diagnos 4. **Second family frontend (TypeSpec or Smithy)** — proves the spec-agnostic claim: richer-than- OpenAPI concepts (interfaces, custom scalars, lifecycle visibility, declared pagination) flow through untouched IR code. -5. **Event-shaped frontend (AsyncAPI)** — proves channels/messages/bindings; then GraphQL and - Protobuf. +5. **Event-shaped frontend (AsyncAPI)** — proves channels/messages/bindings; then GraphQL, + Protobuf, and Erlang/OTP (the actor-protocol frontend: behaviours → operations + channels). diff --git a/docs/ir-design.md b/docs/ir-design.md index c336db5..7a3289f 100644 --- a/docs/ir-design.md +++ b/docs/ir-design.md @@ -17,7 +17,7 @@ spec — field names and shapes here are normative, receiver methods and helpers (`ir-spec-matrix.md`). A backend may ignore a capability; a frontend must never drop one. Only frontends are staged over time — the IR's capability surface is complete from day one, so shipping the OpenAPI 3.x frontend first never forces an IR schema change when TypeSpec, - Smithy, GraphQL, AsyncAPI, or Protobuf frontends land. + Smithy, GraphQL, AsyncAPI, Protobuf, or Erlang/OTP frontends land. 2. **Un-lowered.** Composition, unions, visibility, discriminators, encodings, and streaming stay in source-semantic form. Flattening and language fitting happen in backends. 3. **Stable IDs, flat registries.** All named entities live in flat, ID-keyed registries and @@ -58,16 +58,27 @@ type Document struct { Name string // API title Version string // API version string (source-declared) Docs Docs + Contact *Contact // {Name, URL, Email} (OpenAPI/AsyncAPI info.contact) + License *License // {Name, Identifier, URL} (OpenAPI/AsyncAPI info.license) + TermsOfService string Services []Service // ≥1; multi-service documents are normal (TypeSpec, stitching) Types map[TypeID]TypeDef // the type registry — the only owner of TypeDefs - Channels map[ChannelID]Channel // event/messaging layer (AsyncAPI, webhooks, subscriptions) + Channels map[ChannelID]Channel // event/messaging layer (AsyncAPI, webhooks, subscriptions, OTP processes) + Messages map[MessageID]Message // message registry — messages are reused across channels and + // referenced by identity from operations and replies (AsyncAPI 3) Auth map[AuthID]AuthScheme // auth scheme registry Servers []Server // endpoint templates + TagDefs []TagDef // tag metadata registry: {Name, Docs}; tag *membership* stays + // []string on the tagged nodes (OpenAPI/AsyncAPI tag objects) Versions []string // ordered version labels when availability metadata is used Extensions Extensions Diagnostics []Diagnostic // accumulated by frontend + passes; not part of API meaning Sources []SourceInfo // input files: format, path, content hash } + +type Contact struct { Name, URL, Email string } +type License struct { Name, Identifier, URL string } +type TagDef struct { Name string; Docs Docs } ``` A `Document` is self-contained: no node references anything outside it. @@ -81,7 +92,7 @@ A `Document` is self-contained: no node references anything outside it. ```go type TypeID string // e.g. "t/openapi/components/schemas/User" or "t/anon/paths/~1users/get/responses/200/content/application~1json" type OpID string // operation identity, same construction -type ChannelID, AuthID, PropID string +type ServiceID, ChannelID, MessageID, AuthID, PropID string ``` IDs are opaque to consumers but constructed deterministically by frontends from the source @@ -90,6 +101,10 @@ snapshots, cachable, diffable across spec revisions). IDs are never derived from and never rewritten by renames. The `dedup` pass may alias two structurally identical anonymous types; aliases are recorded so both IDs stay resolvable. +Every named entity has an ID — including services (Thrift `service B extends A`, WSDL 2.0 +interface extension, and Cap'n Proto interface inheritance all reference services by identity) +and messages (AsyncAPI reuses one named message across channels, operations, and replies). + This is the direct answer to oagen's name-keyed registry (silent collision merging, string-rewrite ref fixing) and Kiota's name-keyed children (collision reconciliation logic), and adopts the intent of TCGC's `crossLanguageDefinitionId`. @@ -101,6 +116,8 @@ type Naming struct { Source string // exactly as written in the spec ("user_id", "$ref name", GraphQL field) Canonical string // IR-normalized identifier in neutral form: lower_snake words, no casing opinions Hint string // for anonymous types only: context-derived suggestion ("connection_domain") + Aliases []string // alternate names for schema-resolution matching (Avro aliases). Versionless — + // rename *history* tied to version labels lives in Availability.RenamedFrom. } ``` @@ -121,10 +138,13 @@ Nullability lives on the reference, not the target type, because the same type i position and not another. Combined with `Property.Required` this yields the four distinct states (required/optional × nullable/non-null) that OpenAPI 3.1, TypeSpec, and GraphQL all distinguish. Frontends normalize every source spelling to this one bit: OAS 3.0 `nullable: true`, OAS 3.1 -`type: [T, "null"]`, TypeSpec `T | null`, GraphQL's absence-of-`!`, proto3 explicit presence. +`type: [T, "null"]`, TypeSpec `T | null`, GraphQL's absence-of-`!`. A oneOf/anyOf/union whose only distinction is a null variant becomes a plain nullable `TypeRef`, never a union node. +Protobuf field *presence* is **not** nullability — protobuf has no null. Presence disciplines +lower to `Property.Presence` (§5.1), keeping `Nullable` strictly about wire-null. + --- ## 4. The type graph @@ -136,18 +156,39 @@ type TypeKind string // "primitive" | "scalar" | "model" | "union" | "enum" | "l type TypeCommon struct { ID TypeID Name Naming + Namespace []string // the type's declared logical namespace (proto package, Avro namespace, + // Thrift/XSD/Cap'n Proto scopes); independent of Service.Namespace Anonymous bool // hoisted inline type Docs Docs - Tags []string // free-form labels (Smithy @tags, OpenAPI tag membership on types via policy) + Tags []string // free-form labels (Smithy @tags, OpenAPI tag membership on types via policy); + // tag metadata lives once in Document.TagDefs Sensitive bool // whole-type redaction (Smithy @sensitive on shapes); Property.Secret is the per-use form + Access string // "" = public; "internal" = not part of the exported SDK surface + // (protobuf editions export/local, TCGC @access(internal), Smithy @internal via policy) Deprecation *Deprecation Availability *Availability Usage UsageFlags // computed by a pass: Input | Output | Error | Multipart | … - Instantiation *TemplateInstantiation // provenance for monomorphized generics (TypeSpec templates): - // {Template string; Args []TypeRef} — naming hint + cross-version identity + WireNameByFormat map[string]string // type-level serialized-name overrides per mime + // (TypeSpec @encodedName on models/enums/scalars) + MediaTypeHint string // declared default content type when the type is a body + // (TypeSpec @mediaTypeHint, Smithy @mediaType on string/blob shapes) + XML *XMLHints // type-level XML wire shape: root element name/namespace + // (OpenAPI xml object, Smithy @xmlName/@xmlNamespace on shapes, TypeSpec @Xml.*) + Examples []Example // typed example values attached to the type + // (TypeSpec @example, OpenAPI schema-level examples) + Instantiation *TemplateInstantiation // provenance for monomorphized generics (TypeSpec templates) Extensions Extensions Provenance Provenance } + +type TemplateInstantiation struct { + Template string + Args []TemplateArg // instances are identified by type AND value arguments +} +type TemplateArg struct { + Type *TypeRef // exactly one of Type/Value is set + Value *Value // TypeSpec `valueof` template parameters +} ``` ### 4.1 Primitives @@ -163,6 +204,7 @@ type PrimKind string // int8 int16 int32 int64 uint8 uint16 uint32 uint64 // integer (arbitrary precision — JSON Schema "integer", TypeSpec "integer") // float32 float64 +// float (arbitrary precision *binary* floating point — TypeSpec "float", supertype of float32/64) // number (arbitrary precision — JSON Schema "number", TypeSpec "numeric") // decimal decimal128 // date time datetime datetime_offset duration @@ -173,14 +215,18 @@ type PrimKind string The set is the union of TypeSpec's intrinsic scalars, JSON Schema's types, and Protobuf's needs. Protobuf's `fixed32`/`sfixed64`/`sint*` are **encodings of** `uint32`/`int64`/…, not distinct primitives — they lower to `Encoding` (§5.3). There is no `void` type: absence of a body/return -is a `nil *TypeRef`. +is a `nil *TypeRef`. TypeSpec `safeint` is not a primitive either — its canonical lowering is +`Scalar{Base: integer, Constraints: ±(2^53−1)}`. ### 4.2 Scalars (named restricted/extended primitives) ```go type Scalar struct { TypeCommon - Base TypeRef // primitive or another scalar — extension chains preserved + Base *TypeRef // primitive or another scalar — extension chains preserved. + // nil = opaque scalar with implementation-defined representation + // (GraphQL custom scalars declare no base; backends map nil-base + // scalars to their opaque-scalar strategy, not a fabricated chain) Constraints *Constraints Encoding *Encoding } @@ -200,13 +246,27 @@ type Model struct { Implements []TypeRef // interface conformance, N-ary (GraphQL `implements A & B`); targets are Abstract models Mixins []TypeRef // composition without subtyping (Smithy mixins, TypeSpec spread provenance, extra allOf entries) AdditionalProps *AdditionalProps // map-like catch-all alongside declared properties + Additional AdditionalMode // openness of the property set beyond declared + AdditionalProps Abstract bool // cannot be instantiated directly; fields may be typed by it and // resolve to a conforming concrete model (GraphQL interface types) + Positional bool // properties serialize positionally as a tuple ordered by WireID + // (Erlang records: tag at element 1, fields at 2..N; WireID is the + // 1-based element index, matching element/2) + ExtensionRanges []WireIDRange // wire-ID ranges reserved for third-party extension fields + // (protobuf `extensions 100 to 199`) Discriminator *Discriminator // set on the polymorphic base DiscriminatorValue string // set on each subtype: its wire tag value InputOnly bool // GraphQL input types; distinct identity from output types } +type AdditionalMode string +// "" — unspecified (open by JSON Schema default) +// "closed" — no properties beyond the declared set (additionalProperties: false, +// closed-by-construction records) +// "closed_after_composition" — closed once composition is resolved (unevaluatedProperties: false) + +type WireIDRange struct { From, To int } + type AdditionalProps struct { Value TypeRef Key *TypeRef // nil = string keys @@ -215,10 +275,21 @@ type AdditionalProps struct { type PatternProps struct { Pattern string; Value TypeRef } type Discriminator struct { - Property PropID // which property carries the tag + // Exactly one of Property / PropertyName / Index locates the tag: + Property PropID // model hierarchies: the property that carries the tag + PropertyName string // unions: wire name of the tag property (it exists on no single + // model, so there is no PropID to point at; TypeSpec @discriminated + // discriminatorPropertyName, OpenAPI discriminator on oneOf) + Index *int // positional: 0-based tuple element carrying the tag Literal + // (Erlang tagged tuples {ok, V} | {error, R}; JSON arrays with a + // const head via prefixItems) Mapping map[string]TypeID // wire value → subtype; nil mapping = infer by type name + Default TypeID // variant to use when the tag is absent/unrecognized + // (OpenAPI 3.2 defaultMapping); zero = none Envelope string // "" = tag inline in the variant object; // "object" = {kind, value} wrapper (TypeSpec @discriminated envelope) + EnvelopeValueName string // wire name of the envelope's value property (default "value"; + // TypeSpec envelopePropertyName); meaningful only when Envelope=="object" Inferred bool // discovered heuristically (const-property detection), not declared } ``` @@ -251,18 +322,37 @@ type Union struct { type Variant struct { Name Naming // named variants (TypeSpec, Smithy, protobuf oneof fields); Hint-only for bare oneOf members Type TypeRef - WireID int // protobuf oneof field number; 0 = none + WireName string // serialized tag when it differs from Name.Source + // (Smithy @jsonName on union members, protobuf oneof json_name) + WireID *int // protobuf oneof field number, Cap'n Proto/Avro ordinal; nil = none + // (pointer because 0 is a legal ordinal in several formats) + XML *XMLHints // @xmlName/@xmlNamespace on union members + Event *EventInfo // event-stream metadata when the union is a stream's event set Docs Docs Deprecation *Deprecation + Availability *Availability + Examples []Example Extensions Extensions } + +type EventInfo struct { + ContentType string // per-event content type (TypeSpec @Events.contentType) + Terminal bool // receiving this event ends the stream (TypeSpec @SSE.terminalEvent) +} ``` One node covers the whole design space from the capability matrix: untagged `anyOf` (`Exclusive=false`), untagged `oneOf` (`Exclusive=true`), discriminated `oneOf` (`+Discriminator`), natively tagged unions (`WireTagged=true`). Variant identity survives so backends can generate named accessors, sealed interfaces, or wrapper types per their refiner -strategy. Unions never degrade to optional-field merges in the IR. +strategy. Unions never degrade to optional-field merges in the IR — that includes GraphQL +`@oneOf` input objects, which are spec-level tagged input unions and lower here, not to models. + +**Tag-mode conventions** (normative): `WireTagged=true` with a `Discriminator` means the tag is +a property inside the variant payload (GraphQL `__typename`). `WireTagged=true` with **nil** +`Discriminator` means the union is *key-tagged*: the wire shape is a single-key object whose key +is the variant's wire name (Smithy unions, proto3-JSON oneof, GraphQL `@oneOf` inputs). +`WireTagged=false` means the variant is inferred by validation (JSON oneOf/anyOf). ### 4.5 Enums @@ -273,29 +363,40 @@ type Enum struct { Members []EnumMember Closed bool // false = open/extensible: unknown values must be representable Flags bool // bitfield semantics + FallbackMember string // wire name of the member to substitute when an unknown value is read + // (Avro enum `default` symbol); "" = none } type EnumMember struct { Name Naming Value Value // typed value, matches ValueType - WireName string // serialized form when it differs from Value (rare) + WireName string // serialized form when it differs from Value (rare) Docs Docs Deprecation *Deprecation + Availability *Availability // members appear/disappear across versions (TypeSpec @added on EnumMember) + Examples []Example Extensions Extensions } ``` -`Closed` defaults per source semantics: JSON Schema enums are closed, Smithy and proto3 enums are -open. Backends that can't express open enums (plain Go consts, TS string literals) lower via -their refiners — the bit must survive to that point (Kiota's string-only closed enums are the -counterexample). +`Closed` defaults per source semantics: JSON Schema enums are closed; Smithy enums are open; +protobuf enums are open or closed **per file syntax / editions feature** (proto2 closed, proto3 +open, editions `enum_type` per enum) — the frontend lowers the *resolved* value. Backends that +can't express open enums (plain Go consts, TS string literals) lower via their refiners — the +bit must survive to that point (Kiota's string-only closed enums are the counterexample). +Duplicate member values are legal (protobuf `allow_alias`); slice order preserves which name is +canonical for serialization, and the validate pass must not reject them. ### 4.6 Containers and the rest ```go -type List struct { TypeCommon; Elem TypeRef; Constraints *Constraints } // minItems/maxItems/uniqueItems +type List struct { TypeCommon; Elem TypeRef; Constraints *Constraints; Encoding *Encoding } + // Constraints: minItems/maxItems/uniqueItems. + // Encoding: container-level wire encoding — protobuf packed vs expanded repeated + // fields ("packed"/"expanded"); stacks with the element's own encoding + // (repeated sint64 [packed=false] = zigzag element + expanded list) type MapT struct { TypeCommon; Key TypeRef; Value TypeRef } // Record/additionalProperties-only/proto map -type Tuple struct { TypeCommon; Elems []TypeRef } // prefixItems, TypeSpec tuples +type Tuple struct { TypeCommon; Elems []TypeRef } // prefixItems, TypeSpec tuples, Erlang tuples type Literal struct { TypeCommon; Value Value } // const / single-value enum / discriminator pins type External struct{ TypeCommon; Identity string; Package string; MinVersion string } // resolve to a well-known library type (TCGC external) type Any struct { TypeCommon } // schemaless @@ -303,18 +404,45 @@ type Any struct { TypeCommon } // sc Containers are real type nodes with IDs (hoisted like all anonymous types), not flags on references — Kiota's `CollectionKind` flag couldn't express nested collections or constrained -lists cleanly. +lists cleanly. `External` also carries opaque runtime handles that no target can structurally +model (`erlang:pid`, `erlang:fun` — see §4.8). ### 4.7 Validation-only schema constructs — a documented boundary -JSON Schema's `not`, `if`/`then`/`else`, and `dependentSchemas` express *validation logic*, not -data shape; no target language's type system represents them, and none of the other six formats -has an equivalent. The IR deliberately does not model them structurally. Frontends preserve them -verbatim in `Extensions` (`openapi:not`, `openapi:if-then-else`, …) and emit an `info` +JSON Schema's `not`, `if`/`then`/`else`, `dependentSchemas`, `contains`/`minContains`/ +`maxContains`, and `unevaluatedItems` express *validation logic*, not data shape; no target +language's type system represents them, and none of the other source formats has an equivalent. +The IR deliberately does not model them structurally. Frontends preserve them verbatim in +`Extensions` (`openapi:not`, `openapi:if-then-else`, `openapi:contains`, …) and emit an `info` diagnostic, so the information is never silently lost and a future validation-oriented backend -(request validators, mock servers) can still consume them. This is the one intentional carve-out -from "lossless means structural": losslessness is satisfied by verbatim preservation, and the -carve-out is explicit rather than accidental. +(request validators, mock servers) can still consume them. The one structural carve-back: +`unevaluatedProperties: false` is *shape* (a closed model after composition) and lowers to +`Model.Additional = closed_after_composition`; other `unevaluated*` forms stay verbatim. +`$dynamicRef` is resolved per reference site by frontend expansion (dynamic scope is static per +document); irreducible cases are preserved verbatim with a diagnostic. + +This is the one intentional carve-out from "lossless means structural": losslessness is +satisfied by verbatim preservation, and the carve-out is explicit rather than accidental. + +### 4.8 Degraded source constructs — documented lowerings + +A few source constructs have no faithful structural target in any SDK language. Each has a +*normative* degraded lowering plus verbatim preservation, so degradation is a decision, not an +accident: + +- **TypeSpec `never`-typed properties/variants** — the compiler does *not* remove them; + frontends delete them and emit an `info` diagnostic. No `never` node exists. +- **TypeSpec `StringTemplate` types** (interpolating non-literal types) — degrade to `string` + + `Extensions["typespec:string-template"]` + `info` diagnostic. +- **Erlang bit-sized binaries** (`<<_:M, _:_*N>>`) — `bytes` + `Min/MaxLength` when + byte-aligned, plus `Extensions["erlang:bit-size"] = {base, unit}` + `info` diagnostic. +- **Erlang `fun()` types** — `External{Identity: "erlang:fun"}` + full spec text in + `Extensions["erlang:fun-spec"]` + `warning` diagnostic (non-portable member). No function + type node exists. +- **Erlang heterogeneous map association lists** (`#{atom() => a(), integer() => b()}`) — + lower to union-typed `AdditionalProps` + `Extensions["erlang:map-assocs"]` verbatim. +- **Erlang `-opaque`** — lowered structurally (lossless) + `Extensions["erlang:opaque"] = true`; + backend policy decides whether to expose or wrap. --- @@ -328,26 +456,51 @@ type Property struct { Name Naming WireName string // serialized name; defaults to Name.Source WireNameByFormat map[string]string // per-mime overrides (TypeSpec @encodedName json/xml) - WireID int // protobuf field number / thrift id; 0 = none + WireID *int // protobuf field number / thrift id / tuple element index + // (1-based when Model.Positional); nil = none — pointer because + // 0 is a legal ordinal in Cap'n Proto/FlatBuffers/Avro + ExtensionOf string // "" = the model's own field; else the fully-qualified declaring + // scope of a third-party extension field (protobuf `extend`) — + // directs qualified JSON naming and registry-based accessors Type TypeRef Required bool // presence on the wire; orthogonal to Type.Nullable + Presence PresenceKind // wire-presence discipline where the format distinguishes more + // than required/optional (protobuf) + ClientOptional bool // wire-required but clients MUST treat as optional + // (Smithy @clientOptional; implicit for @input structure members) + DefaultAdded bool // default was added post-publication; generators may ignore it + // for backward compatibility (Smithy @addedDefault) Visibility Visibility // lifecycle set; zero value = visible in all Default *Value Constraints *Constraints Encoding *Encoding Args []Parameter // parameterized fields: GraphQL field arguments on any property, // at any depth — not just operation entry points. Empty elsewhere. - Flatten bool // property's fields hoisted into parent on the wire (Smithy/TCGC flatten) + Flatten bool // property's fields hoisted into parent on the wire (Smithy/TCGC + // flatten; also set on the synthetic property wrapping a hoisted + // protobuf oneof — oneof members are wire-level top-level fields) EventHeader bool // in an event-stream event model: member travels in the frame // header, not the event payload (Smithy @eventHeader) + EventPayload bool // member is the *raw* frame payload — blob/string as bytes, + // structure as protocol document (Smithy @eventPayload). + // Mutually exclusive with EventHeader; at most one per model. Secret bool // redact in logs/docs (TypeSpec @secret, format:password) XML *XMLHints // XML wire shape when it diverges from the JSON-implied shape + Examples []Example // property-level examples (TypeSpec @example, OpenAPI) Docs Docs Deprecation *Deprecation Availability *Availability Extensions Extensions Provenance Provenance } + +type PresenceKind string +// "" — format default (JSON world: Required/Nullable say everything) +// "implicit" — absence ⇔ default value; unset is unobservable; zero values are not serialized +// (proto3 no-label, editions IMPLICIT) +// "explicit" — unset is distinguishable from default-valued (hazzers/pointers; proto2 optional, +// proto3 `optional`, editions EXPLICIT) +// "required" — must be present on the wire (proto2 required, editions LEGACY_REQUIRED) ``` **Scope rule for `Args`:** parameterized properties are only legal on models reachable from a @@ -359,22 +512,27 @@ graph serializes conventionally in every backend. ```go type Lifecycle = string // OPEN set. Canonical values: "create" | "read" | "update" | "delete" | "query". - // TypeSpec's new visibility system allows arbitrary user-defined visibility + // TypeSpec's visibility system allows arbitrary user-defined visibility // classes; frontends lower custom classes as ":" strings so - // nothing is dropped. Backends treat unknown classes as opaque filters. + // nothing is dropped. Filters evaluate per class, not across the flat union; + // backends treat unknown classes as opaque filters. type Visibility struct { - Only []Lifecycle // empty = visible in all lifecycles + Only []Lifecycle // empty = visible in all lifecycles (unless None) + None bool // visible in NO lifecycle: excluded from every projection + // (TypeSpec @invisible) — distinct from the zero value } ``` `readOnly` lowers to `Only: [read]` (plus delete/query per OpenAPI semantics); `writeOnly` to `Only: [create, update]`; GraphQL input-vs-output types to `create/update` vs `read`; TypeSpec `@visibility` maps directly. One logical model therefore produces N wire shapes; the projection -(`ModelShape(model, lifecycle)`, with PATCH additionally making properties optional) is a -computed traversal in backends' plan layer — the IR stores the single logical model plus the -visibility facts, never the projected variants. This is TypeSpec's `MetadataInfo` design with -storage and computation split. +(`ModelShape(model, lifecycle)`, with PATCH additionally making properties optional *unless the +binding disables it* — §8.1 `PatchImplicitOptionality`) is a computed traversal in backends' +plan layer — the IR stores the single logical model plus the visibility facts, never the +projected variants. Operations can override which filter applies to their request/response +(§7.2 `ParameterVisibility`/`ReturnTypeVisibility`). This is TypeSpec's `MetadataInfo` design +with storage and computation split. ### 5.3 Constraints and encodings @@ -384,9 +542,12 @@ type Constraints struct { Min, Max *BigVal ExclusiveMin, ExclusiveMax bool MultipleOf *BigVal - // string + Precision, Scale *int64 // decimal digit bounds (Avro decimal, XSD totalDigits/fractionDigits, + // OData Edm.Decimal) + // string & bytes — length constraints apply to both (Avro fixed(N) = bytes, MinLength=MaxLength=N) MinLength, MaxLength *int64 Pattern string // ECMA-262 regex as written; backends translate or drop with a diagnostic + PatternMessage string // human-readable validation message (TypeSpec @pattern's second arg) // collections MinItems, MaxItems *int64 UniqueItems bool @@ -395,30 +556,41 @@ type Constraints struct { type Encoding struct { Name string // "rfc3339" | "unixTimestamp" | "base64" | "base64url" | "iso8601" | - // "seconds" | "http-date" | "varint" | "zigzag" | "fixed" | format strings + // "seconds" | "http-date" | "varint" | "zigzag" | "fixed" | + // "packed" | "expanded" | "delimited" | format strings + // (packed/expanded are container encodings on List.Encoding; "delimited" = + // protobuf group / editions DELIMITED message encoding) WireType *TypeRef // the on-wire primitive when it differs from the logical type // (utcDateTime encoded as int32; bytes as base64 string) + MediaType string // content media type of the value itself (Smithy @mediaType on string/blob, + // JSON Schema contentMediaType); "" = none } ``` The logical-type / encoding-name / wire-type triple is TCGC's reification of TypeSpec `@encode` and also absorbs OpenAPI `format` and Protobuf's `sint*/fixed*` wire variants. Encoding attaches -at the scalar definition or overrides at the property — property wins. +at the scalar definition or overrides at the property — property wins. Protobuf editions features +lower here per element after the frontend resolves the feature cascade (descriptors expose +resolved values): `field_presence` → `Presence`, `enum_type` → `Closed`, +`repeated_field_encoding` → `List.Encoding`, `message_encoding` → `"delimited"`; remaining axes +(`utf8_validation`, `json_format`) → namespaced Extensions. ### 5.4 XML hints XML-capable formats (OpenAPI's `xml` object, Smithy's `xmlName`/`xmlAttribute`/`xmlNamespace`/ -`xmlFlattened` traits) describe an XML wire shape that diverges from the JSON-implied one. This -is typed, not an extension, because two formats express it and backends must act on it: +`xmlFlattened` traits, TypeSpec's `@Xml.*` decorators) describe an XML wire shape that diverges +from the JSON-implied one. This is typed, not an extension, because multiple formats express it +and backends must act on it. Hints attach at both levels: `TypeCommon.XML` (root element +name/namespace of the shape itself) and `Property.XML` (per-use overrides; property wins). ```go type XMLHints struct { Name string // element/attribute name override Namespace string // namespace URI Prefix string - Attribute bool // serialize as attribute, not element + NodeType string // "" | "element" | "attribute" | "text" | "cdata" | "none" + // (OpenAPI 3.2 nodeType; "text" covers Smithy httpPayload text) Wrapped bool // list items wrapped in a container element - Text bool // value is the element's text content (Smithy httpPayload text) } ``` @@ -431,20 +603,34 @@ separate from the type graph (TypeSpec's Type-vs-Value split): ```go type Value struct { - Kind ValueKind // "null" | "bool" | "string" | "number" | "bytes" | "list" | "object" | "ref" + Kind ValueKind // "null" | "bool" | "string" | "number" | "bytes" | "symbol" | + // "list" | "object" | "ref" | "ctor" Bool bool - Str string + Str string // payload for "string" AND "symbol" Num BigVal // arbitrary precision, decimal string form Bytes []byte // base64 in JSON form List []Value Object []Field // ordered; Field{Name string; Value Value} Ref *ValueRef // reference to a declared constant: {Type TypeID; Member string} // (TypeSpec enum-member defaults, references to named consts) + Ctor *CtorValue // constructor-built value (TypeSpec scalar constructors) +} + +type CtorValue struct { + Scalar TypeID // the scalar whose constructor is invoked + Name string // constructor name ("fromISO", "now", custom inits) + Args []Value } ``` `BigVal` is a decimal-string wrapper: no float64 round-tripping anywhere in the IR. +`symbol` is an interned-symbol value distinct from `string` (Erlang atoms: on the native wire +`ok` ≠ `<<"ok">>`). Backends without a symbol concept render symbols as strings — explicitly, +not accidentally. `ctor` captures values built by named scalar constructors +(`utcDateTime.now()`, `plainDate.fromISO("2024-05-06")`) — inherently non-literal, so frontends +must not fold them. + --- ## 7. Service layer @@ -453,36 +639,57 @@ type Value struct { ```go type Service struct { + ID ServiceID Name Naming Docs Docs + Version string // per-service version string (Smithy service version); the + // document-level Version remains the API title's version Namespace []string // source namespace path (TypeSpec/Smithy/proto package) + Extends []ServiceID // client-visible service inheritance (Thrift service extends, + // WSDL 2.0 interface extension, Cap'n Proto interface inheritance); + // inherited operations are walked, never copied Groups []OperationGroup // hierarchical; a group ≈ TypeSpec interface / Smithy resource / tag Auth []AuthRequirement // service-level default (OR-of-ANDs, §9) CommonErrors []ErrorCase // errors every operation can return (Smithy service-level errors) + Protocols []ProtocolDecl // declared serde/protocol conventions the service speaks + // (Smithy @protocolDefinition traits like aws.protocols#restJson1) + Renames map[TypeID]Naming // per-service shape presentation names (Smithy service `rename`); + // the TypeID — and Naming on the type — are unchanged Servers []int // indexes into Document.Servers scoped to this service Extensions Extensions Provenance Provenance } +type ProtocolDecl struct { + Name string // e.g. "aws.restJson1", "grpc" + Options Extensions // per-protocol options kept raw (the Channel.Bindings pattern) +} + type OperationGroup struct { Name Naming Docs Docs Groups []OperationGroup // nesting: Smithy resources, sub-clients Operations []Operation Resource *ResourceInfo // Smithy resource semantics when declared + Availability *Availability // groups (TypeSpec interfaces) are versionable Extensions Extensions } type ResourceInfo struct { Identifiers []Property // resource identity fields Properties []Property // resource state fields (Smithy 2.0 resource properties) - Lifecycle map[string]OpID // "create"|"read"|"update"|"delete"|"list" → op + Lifecycle map[string]OpID // "create"|"put"|"read"|"update"|"delete"|"list" → op + // (put = create-or-replace with client-provided identifier) + NoReplace bool // put may create but not replace (Smithy @noReplace) + InstanceOps []OpID // declared non-lifecycle instance operations (require identifiers) + CollectionOps []OpID // declared collectionOperations — the split drives + // sub-client shape and is a declared fact, not a heuristic } ``` OpenAPI frontends build groups from tags (policy-controllable: tag-based vs path-prefix-based); TypeSpec from interfaces/namespaces; Smithy from resources; GraphQL yields three groups -(query/mutation/subscription); Protobuf one group per `service`. +(query/mutation/subscription); Protobuf one group per `service`; Erlang/OTP one group per module. ### 7.2 Operation — the protocol-neutral core @@ -499,6 +706,10 @@ type Operation struct { Responses []Response // ordered; success + alternative successes Errors []ErrorCase // declared failure shapes + OneWay bool // fire-and-forget: no response EVER exists (OTP cast, AsyncAPI + // send-without-reply, Thrift oneway, JSON-RPC notifications). + // Distinct from a response with no body (an ack still exists); + // validate rejects OneWay && len(Responses) > 0 Streaming StreamingMode // none | client | server | bidi (derived summary of the two below) RequestStream *StreamDetail // client→server streaming semantics, when present ResponseStream *StreamDetail // server→client streaming semantics, when present @@ -508,6 +719,9 @@ type Operation struct { // safe = no side effects (Smithy @readonly, HTTP GET semantics) Auth []AuthRequirement // override of service default; empty slice ≠ nil (empty = explicitly public) Tags []string + ParameterVisibility []Lifecycle // visibility filter override for the request view + // (TypeSpec @parameterVisibility); nil = protocol default + ReturnTypeVisibility []Lifecycle // filter override for the response view; nil = protocol default OverloadOf *OpID // TypeSpec @overload grouping Bindings OpBindings // §8 — how the core maps onto concrete protocols @@ -522,22 +736,36 @@ type Parameter struct { Required bool Default *Value Constraints *Constraints + ValueFrom *PropPath // the parameter's value is derived from a location in the + // outgoing/incoming message (AsyncAPI parameter `location` + // runtime expressions) — SDKs may auto-fill it; nil = caller-supplied Docs Docs Deprecation *Deprecation + Availability *Availability + Examples []Example Extensions Extensions // NOTE: no location here — path/query/header is HTTP-binding detail (§8.1) } type Payload struct { Contents []Content // one per media type / message schema — all kept + Extensions Extensions } type Content struct { MediaType string // "application/json", "multipart/form-data", "" for non-HTTP + SchemaFormat string // schema *language* the type graph was lowered from, media-type + // form (AsyncAPI multiFormatSchema: Avro/Protobuf/RAML/…); + // "" = source-native. The verbatim source schema is preserved in + // Extensions (e.g. "asyncapi:schema") for schema-registry workflows Type TypeRef + Item *TypeRef // element shape of a sequential stream declared per media type + // (OpenAPI 3.2 itemSchema for SSE/JSONL/json-seq); nil = not sequential + ItemEncoding map[string]PartEncoding // per-item encoding for sequential media types (3.2 itemEncoding) Encoding map[string]PartEncoding // multipart/form: per-property (part) wire config, keyed by PropID File *FileInfo // body is a file upload/download (TypeSpec file bodies, binary payloads) Examples []Example + Extensions Extensions } type PartEncoding struct { @@ -550,14 +778,26 @@ type PartEncoding struct { } // TypeSpec tuple-form multipart lowers to a synthesized model whose properties are the parts. -type FileInfo struct { IsText bool; FilenameParam string } +type FileInfo struct { + IsText bool // textual vs binary contents + Contents *TypeRef // declared contents scalar chain (string/bytes extensions); nil = bytes + ContentTypes []string // declared *allowed* content-type set (TypeSpec File<"image/png" | + // "image/jpeg">); runtime Content-Type comes from the file value + ContentTypeDefault string // default when the file value carries none + FilenameLocation string // "content-disposition" (default) | "path" | "header" + FilenameWireName string // wire name when FilenameLocation is path/header +} type Response struct { Name Naming // for formats with named outputs; Hint elsewhere Conditions ResponseConditions // HTTP status codes/ranges; empty for RPC single-response Payload *Payload // nil = no body Headers []Property // response metadata fields + StatusCodeProp *PropPath // output member populated from the runtime HTTP status line + // (Smithy @httpResponseCode, TypeSpec non-literal @statusCode); + // the member is suppressed from the body Docs Docs + Extensions Extensions } type ResponseConditions struct { @@ -567,8 +807,14 @@ type ResponseConditions struct { type ErrorCase struct { Type TypeRef // an error-flagged model Conditions ResponseConditions + Fault string // "" | "client" | "server" — protocol-neutral fault classification + // (Smithy @error; OpenAPI 4XX/5XX is its HTTP lowering). Drives + // exception hierarchies and default status synthesis Retryable *bool // Smithy @retryable; nil = unknown + Throttling *bool // retryable specifically due to throttling — distinct backoff class + // (Smithy @retryable(throttling: true)); nil = unknown Docs Docs + Extensions Extensions } ``` @@ -583,6 +829,10 @@ Design notes: counterexample). The plan layer picks a primary for SDK ergonomics; the IR doesn't. - **Errors reference error models** (`Model` with `UsageFlags.Error`), with status conditions and range collapsing exactly as Kiota's error mappings (`404`, `4XX`, catch-all). +- **`OneWay` is on the neutral core**, like `Streaming` and `Idempotency`, because + fire-and-forget is protocol-independent — an SDK that blocks awaiting a reply on a cast is + wrong regardless of transport. The AsyncAPI frontend sets it for send-operations without a + reply, rather than leaving one-way-ness inferable only from binding fields. ### 7.3 Pagination, long-running, streaming @@ -595,16 +845,26 @@ type Pagination struct { Items *PropPath // where result items live in the response — a PATH, not a name NextCursor *PropPath // continuation source in the response NextLink *PropPath + PrevLink *PropPath // server-driven navigation links beyond next + FirstLink *PropPath // (TypeSpec @prevLink/@firstLink/@lastLink, OpenAPI links) + LastLink *PropPath TotalCount *PropPath } -type PropPath struct{ Root TypeRef; Segments []PropID } // survives envelope nesting (TypeSpec paging lesson) +type PropPath struct { + Root *TypeRef // the type the path roots in; nil = determined by context + // (the enclosing response body, message payload, …) + In string // "" = body/payload | "header" — continuation tokens and reply addresses + // can live in response/message headers, not just bodies + Segments []PropID +} type ParamPath struct{ Param string; Segments []PropID } type LongRunning struct { FinalStateVia string // "operation-location" | "status-monitor" | "original-uri" | … - PollingOperation *OpID // declared poll op (TypeSpec @pollingOperation) - FinalOperation *OpID // declared final-result op (TypeSpec @finalOperation) + PollingOperation *OpID // declared poll op (Azure.Core @pollingOperation — a library + // convention, not core TypeSpec) + FinalOperation *OpID // declared final-result op (Azure.Core @finalOperation) PollingType *TypeRef FinalType *TypeRef ResultPath *PropPath @@ -615,16 +875,22 @@ type StreamingMode string // "none" | "client" | "server" | "bidi" type StreamDetail struct { Events *TypeRef // stream element type when it differs from the payload content type — // for event streams this is a WireTagged Union of event models; within an - // event model, Property.EventHeader marks frame-header members and the - // remaining members form the event payload (Smithy @eventHeader/@eventPayload) + // event model, Property.EventHeader marks frame-header members and + // Property.EventPayload marks a raw-payload member (Smithy + // @eventHeader/@eventPayload); per-event content types and terminal + // events live on Variant.Event Initial *TypeRef // initial-request / initial-response message preceding the stream // (Smithy event stream initial messages); nil = none + RequiresLength bool // the streamed content must have a known finite length up front + // (Smithy @requiresLength) — changes the generated parameter type } ``` Streaming mode is on the operation core because it is protocol-independent (gRPC streams, SSE, WebSocket, Smithy event streams all project onto it); the wire mechanism (SSE vs chunked vs -WS frames vs length-prefixed events) lives in the binding. +WS frames vs length-prefixed events) lives in the binding. Smithy **waiters** +(`smithy.waiters#waitable`) are *not* folded into `LongRunning` — acceptor lists and JMESPath +matchers are preserved verbatim in `Operation.Extensions` (§15). --- @@ -632,10 +898,16 @@ WS frames vs length-prefixed events) lives in the binding. ```go type OpBindings struct { - HTTP *HTTPBinding // OpenAPI/Swagger ops, TypeSpec @route, Smithy http traits - RPC *RPCBinding // Protobuf/gRPC, Smithy RPC protocols - Message *MessageBinding // AsyncAPI operations, GraphQL subscriptions, webhooks - GraphQL *GraphQLBinding // query/mutation/subscription field binding + HTTP []HTTPBinding // OpenAPI/Swagger ops, TypeSpec @route, Smithy http traits. + // A slice: one operation may carry several HTTP mappings — + // gRPC transcoding's additional_bindings — with the primary first + RPC *RPCBinding // Protobuf/gRPC, Smithy RPC protocols, JSON-RPC + Message *MessageBinding // AsyncAPI operations, webhooks + GraphQL *GraphQLBinding // query/mutation/subscription field binding. GraphQL subscriptions + // bind here + streaming fields on the core — NOT via MessageBinding + // (GraphQL defines no channel; synthesizing one is deployment-aware + // policy, marked Inferred, never a frontend default) + OTP *OTPBinding // Erlang/OTP behaviour operations (§8.5) } ``` @@ -646,24 +918,47 @@ HTTP and RPC; gRPC with HTTP transcoding). ```go type HTTPBinding struct { - Method string // GET, POST, … uppercase + Method string // as sent on the wire (OpenAPI 3.2 additionalOperations keys + // carry exact capitalization; QUERY and custom methods are legal) URITemplate string // RFC 6570 — the one true path representation (Kiota + TypeSpec agree) HostPrefix string // endpoint host prefix, may contain {param} labels (Smithy @endpoint) + SharedRoute bool // multiple operations legally share method+path, disambiguated by + // request content (TypeSpec @sharedRoute) — validate must not + // reject the duplicate, and single-route backends must merge ParamBindings []HTTPParamBinding RequestContentTypes []string // priority-ordered + ResponseBodyPath *PropPath // HTTP response body = this sub-field of the response type + // (gRPC transcoding response_body); nil = the whole payload SuccessStatus map[int]int // response index → primary status (denormalized convenience; conditions are the truth) + Compression *RequestCompression // client MUST compress the request body (Smithy @requestCompression) + ChecksumRequired bool // client MUST send a payload checksum (Smithy @httpChecksumRequired) + PatchImplicitOptionality *bool // nil = protocol default (PATCH projections make properties + // optional); false = disabled (TypeSpec @patch implicitOptionality) IsWebhook bool // OpenAPI 3.1 webhooks: direction is inbound Callbacks []Callback // out-of-band operations keyed by runtime expressions + Extensions Extensions } +type RequestCompression struct { Encodings []string } // priority-ordered ("gzip", …) + type HTTPParamBinding struct { Param string // Operation.Params name it binds - Location HTTPLocation // path | query | header | cookie | body | body_property | host + ParamPath []PropID // nested source field within the logical param, when the binding + // targets a sub-field of a message-typed param (gRPC transcoding + // {book.name}, dotted query params); empty = the whole param + Location HTTPLocation // path | query | querystring | header | cookie | body | + // body_property | host // host = param fills a HostPrefix label (Smithy @hostLabel) + // querystring = the whole query string serialized from one schema + // (OpenAPI 3.2 in: querystring, combined with ContentType) WireName string Style string // simple | form | label | matrix | deepObject | pipe/space-delimited Explode *bool AllowReserved bool + PathPattern string // multi-segment path pattern constraint for this param + // (gRPC transcoding {name=shelves/*/books/*}); "" = single segment. + // The URI template uses reserved expansion; backends that cannot + // validate the pattern drop it with a diagnostic Prefix string // map-typed param spread as prefixed wire entries: // prefixed headers (Smithy @httpPrefixHeaders) or catch-all // query maps (@httpQueryParams with Prefix "") @@ -674,18 +969,24 @@ type HTTPParamBinding struct { type Callback struct { Expression string; Operations []OpID } ``` -Every logical parameter is bound exactly once (validated). TypeSpec's `HttpProperty` role+path -flattening is the model here: nested `@header`/`@body` annotations resolve to explicit -`(role, wire name, path)` bindings rather than restructured models. +Every logical parameter is bound exactly once **per non-host location; a `host` binding is +additive** — Smithy `@hostLabel` members expand into the host prefix *and* still serialize at +their modeled location, by spec. TypeSpec's `HttpProperty` role+path flattening is the model +here: nested `@header`/`@body` annotations resolve to explicit `(role, wire name, path)` +bindings rather than restructured models. ### 8.2 RPC ```go type RPCBinding struct { - System string // "grpc" | "smithy-rpc" | "connect" | … + System string // "grpc" | "smithy-rpc" | "connect" | "jsonrpc" | … FullMethod string // "/pkg.Service/Method" InputType *TypeRef // the request message type params fold into (nil = synthesize from Params) + ParamStructure string // "" | "by_name" | "by_position" | "either" — how params serialize + // (JSON-RPC positional vs named; OpenRPC paramStructure). Param order + // is already source order per design rule 5; this is the mode IdempotencyLevel string + Extensions Extensions } ``` @@ -695,16 +996,35 @@ type RPCBinding struct { type MessageBinding struct { Channel ChannelID Direction MsgDirection // send | receive (application perspective, AsyncAPI 3 semantics) - Messages []MessageRef // which of the channel's messages this operation uses - ReplyTo *ChannelID + Messages []MessageID // which of the channel's messages this operation uses + // (must be a subset — validated) + Reply *Reply // request-reply semantics; nil = none. A send-op with no Reply and + // no Responses is one-way (set Operation.OneWay) + Bindings map[string]Extensions // operation-level protocol bindings kept raw + // (kafka groupId/clientId — constrain SDK client config) + Extensions Extensions +} + +type Reply struct { + Channel *ChannelID // static reply channel; nil when the address is dynamic-only. + // (An AsyncAPI reply channel's own address is null by spec) + Address *PropPath // dynamic reply address: where in the *request* message the reply + // destination lives, e.g. In:"header", Segments:[replyTo] + // (AsyncAPI Operation Reply Address runtime expressions) + Messages []MessageID // reply payload message set + Docs Docs } type Channel struct { ID ChannelID Name Naming - Address string // topic/routing key/path, may contain {params} + Address *string // topic/routing key/path, may contain {params}. + // nil = unknown/runtime-assigned address (load-bearing: reply + // channels and dynamic topics; SDKs expose a runtime address arg) + Docs Docs + Tags []string Params []Parameter - Messages []Message + Messages []MessageID // the channel's message set — messages live in Document.Messages Servers []int Bindings map[string]Extensions // protocol-specific ("kafka", "amqp", "ws", "mqtt") kept as namespaced raw config Extensions Extensions @@ -712,19 +1032,29 @@ type Channel struct { } type Message struct { + ID MessageID Name Naming Payload Payload - Headers []Property - CorrelationID *PropPath + Headers *TypeRef // header *schema* — an object-constrained model hoisted into the + // type registry like any anonymous type (headers can be named, + // composed, even Avro-defined; and $message.header#/… paths need + // a type to resolve against). Backends compute flat lists per §4.3 + CorrelationID *PropPath // In: "header" | "" (payload) ContentType string + Tags []string Docs Docs + Deprecation *Deprecation + Examples []Example // correlated header+payload example pairs (Example.Headers + .Value) + Bindings map[string]Extensions // message-level protocol bindings (kafka message key, …) Extensions Extensions + Provenance Provenance } ``` Kafka/AMQP/MQTT binding minutiae stay as namespaced raw extension config rather than modeled structs — they are protocol deployment detail, not API shape, and modeling them structurally -would chase every AsyncAPI binding spec revision. +would chase every AsyncAPI binding spec revision. The raw-bindings channel exists at every +level that AsyncAPI defines bindings: server, channel, operation, and message. ### 8.4 GraphQL @@ -732,16 +1062,58 @@ would chase every AsyncAPI binding spec revision. type GraphQLBinding struct { Kind string // "query" | "mutation" | "subscription" FieldPath []string // entry-point field (nesting for namespaced schemas) + Extensions Extensions } ``` GraphQL lowering: schema types → models (`InputOnly` for inputs), interfaces → `Abstract` -models with implementors listing them in `Implements`, unions → `WireTagged` unions -discriminated by `__typename`, entry-point fields → operations (field args → `Params`, field -type → response), nested field arguments → `Property.Args` at any depth. Arbitrary -client-composed selection sets are out of scope by design: Morphic generates SDK surface, and -the full type graph — including per-field arguments — is retained so a backend can still offer -query builders. +models with implementors listing them in `Implements` (including interfaces implementing +interfaces), unions → `WireTagged` unions discriminated by `__typename`, **`@oneOf` input +objects → `Union{Exclusive: true, WireTagged: true}` with a variant per field** (never a model +of optional properties — that is the union-to-optional-fields collapse rule 2 forbids), custom +scalars → `Scalar` with `Base = nil`, entry-point fields → operations (field args → `Params`, +field type → response), nested field arguments → `Property.Args` at any depth, subscriptions → +`GraphQLBinding{Kind: "subscription"}` + `StreamingMode: server` + `ResponseStream`. + +Directive conventions (normative): applications are ordered and repeatable, so +`Extensions["graphql:@"]` holds an **ordered JSON array** of application argument objects +— a singleton array for a single application. Directive *definitions* are preserved verbatim at +document level under `Extensions["graphql:directive-definitions"]`. Type-extension assembly +(`extend type` across files): the node's `Provenance` points at the original definition, each +member's `Provenance` at its defining occurrence; the occurrence list goes to +`Extensions["graphql:extends"]` when SDL round-trip fidelity is wanted. + +Arbitrary client-composed selection sets are out of scope by design: Morphic generates SDK +surface, and the full type graph — including per-field arguments — is retained so a backend can +still offer query builders. + +### 8.5 Erlang/OTP + +```go +type OTPBinding struct { + Behaviour string // "gen_server" | "gen_statem" | "gen_event" + Kind string // "call" (synchronous request→reply) | "cast" (fire-and-forget; the + // operation also sets OneWay) | "info" (raw message send) + Process ChannelID // the channel modeling the target process: Address = registered name + // (nil Address = unregistered/runtime pid); registration kind + // (local/global/via) in Channel.Bindings["otp"] + RequestTag *Value // tag of the request tuple (a symbol Value, e.g. 'get'); + // nil = the whole term is the request +} +``` + +An OTP frontend consumes module type information (`-spec`/`-type` on behaviour callbacks and +API functions). Lowering conventions: a gen_server ≈ one `Channel`; `handle_call`/`handle_cast` +APIs are `Operation`s with an `OTPBinding`; unsolicited `handle_info` messages are channel +`Messages` consumed via `MessageBinding{Direction: receive}`; a gen_event manager is a Channel +with one message shape per event. Tagged-tuple protocols (`{ok, V} | {error, R}`) are Unions of +Tuples discriminated by `Discriminator.Index`; records are `Model{Positional: true}` with the +record tag as an index-1 `Literal`-typed property. Splitting a tagged reply union into +`Responses` vs `Errors` (`{error, Reason}` is in-band data, not transport failure) is injectable +frontend policy, marked `Inferred`. Erlang `string()` is `[char()]` and must lower as a List of +a char-ranged scalar — never as PrimKind `string`. Delayed replies (`noreply` + +`gen_server:reply/2`) are invisible at the protocol surface and need no representation; call +timeouts and `multi_call`/`send_request` machinery are SDK runtime policy, not IR. --- @@ -751,36 +1123,63 @@ query builders. type AuthScheme struct { ID AuthID Name Naming - Kind AuthKind // apiKey | http_basic | http_bearer | oauth2 | openid_connect | mutual_tls | custom + Kind AuthKind + // AuthKind: apiKey | http_basic | http_bearer | oauth2 | openid_connect | mutual_tls | + // user_password | x509 | symmetric_encryption | asymmetric_encryption | + // sasl_plain | sasl_scram_sha256 | sasl_scram_sha512 | sasl_gssapi | custom + // The SASL family, user_password, and x509 are how Kafka/AMQP clients authenticate + // (AsyncAPI securitySchemes). AsyncAPI `httpApiKey` lowers to apiKey; AsyncAPI `apiKey` + // (transport user/password slot) lowers to apiKey with In: user|password. X509 is distinct + // from mutual_tls (certificate as credential vs mutual verification) — frontends must not + // conflate them. + Docs Docs + Deprecation *Deprecation // apiKey - In string // header | query | cookie + In string // header | query | cookie | user | password KeyName string // http - Scheme string // bearer, basic, digest… + Scheme string // bearer, basic, digest…; also legal for apiKey (Smithy @httpApiKeyAuth scheme) BearerFormat string // oauth2 Flows []OAuthFlow // {Kind: authorization_code|client_credentials|implicit|password|device, - // AuthorizationURL, TokenURL, RefreshURL, Scopes map[string]string} + // AuthorizationURL, TokenURL, RefreshURL, Scopes map[string]string, + // Extensions} — device flow's deviceAuthorizationUrl rides AuthorizationURL + OAuth2MetadataURL string // RFC 8414 authorization-server metadata (OpenAPI 3.2) // openid OpenIDConnectURL string Extensions Extensions + Provenance Provenance } type AuthRequirement struct { // one *option* Schemes []SchemeUse // ALL must be satisfied together } type SchemeUse struct { Scheme AuthID; Scopes []string } -// []AuthRequirement on a service/operation = OR across options (TypeSpec/OpenAPI security semantics) +// []AuthRequirement on a service/operation/server = OR across options (TypeSpec/OpenAPI security +// semantics). Option order is PRIORITY order — clients pick the first supported option (Smithy +// @auth is priority-ordered; the Smithy frontend materializes the alphabetical default when +// @auth is absent). Frontends whose source has no ordering semantics emit source order. +// An empty option (AuthRequirement{Schemes: []}) inside a non-empty list means "no auth is one +// acceptable choice" (TypeSpec NoAuth in a union, Smithy @optionalAuth, OpenAPI's empty security +// requirement) — distinct from Operation.Auth = [] (explicitly public). ``` ## 10. Servers ```go type Server struct { + Name Naming // servers are named entities (AsyncAPI name-keyed servers, + // OpenAPI 3.2 Server.name) URLTemplate string // may contain {variables} Description Docs - Variables []ServerVariable // {Name, Default, Enum []string, Docs} + Variables []ServerVariable // {Name, Default, Enum []string, Docs, Extensions} Protocol string // "https" default; "kafka", "wss", … for messaging servers + ProtocolVersion string // e.g. Kafka "3.5", AMQP "0-9-1" (AsyncAPI) + Tags []string + Auth []AuthRequirement // server-scoped security — AsyncAPI's primary auth placement + // (broker connections authenticate per server; different + // servers of one service may require different schemes) + Bindings map[string]Extensions // server-level protocol bindings kept raw Extensions Extensions } ``` @@ -789,19 +1188,29 @@ type Server struct { ```go type Availability struct { - Added string // version label from Document.Versions - Removed string + Added []string // version labels from Document.Versions, ordered — multiple entries + Removed []string // support add/remove/re-add cycles (v1 present, v2 removed, v3 re-added) Deprecated string RenamedFrom []VersionedName // {Version, Name} TypeChangedFrom []VersionedType // {Version, Type TypeRef} — property/return type changed // across versions (TypeSpec @typeChangedFrom) + RequiredChanged []VersionedBool // {Version, WasRequired} — optionality flipped at a version + // (TypeSpec @madeOptional/@madeRequired); the slice pass + // reconstructs Required for older snapshots from it } +type VersionedBool struct { Version string; WasRequired bool } ``` The IR stores the **timeline** (TypeSpec model); the `version-slice` pass produces a concrete snapshot document per version for consumption. Backends always receive snapshots — they never interpret availability themselves. Formats without versioning semantics simply leave this nil. +`Availability` attaches everywhere versioning decorators can: `TypeCommon`, `Property`, +`Operation`, `Parameter`, `EnumMember`, `Variant`, and `OperationGroup` — a versioned enum +member or union variant must disappear from pre-`Added` snapshots. Smithy `@since` (free-form, +no version registry) lowers to `Added` only when the frontend synthesizes labels; otherwise it +is preserved as an extension. + ## 12. Docs, deprecation, examples, extensions ```go @@ -814,17 +1223,37 @@ type Docs struct { type Deprecation struct { Message, Since, RemovalVersion string } -type Example struct { Name string; Summary string; Value *Value; ExternalURL string } +type Example struct { + Name string + Summary string + Description string + Value *Value // single-value examples (schemas, properties, parameters); + // for message examples: the payload + Headers *Value // message examples: the correlated header values (AsyncAPI message + // examples are header+payload PAIRS — never split them) + Input *Value // operation scenarios: paired input ↔ output/error + Output *Value // (Smithy @examples, TypeSpec @opExample parameters/returnType) + Error *ErrorExample // the scenario ends in this error instead of Output + ExternalURL string + Extensions Extensions +} +type ErrorExample struct { Type TypeRef; Content Value } +// Field legality is contextual (validated): Value/Headers on types, properties, parameters, +// contents, and messages; Input/Output/Error on operations. An Example never mixes the two arms. type Extensions map[string]RawValue // keys are namespaced by origin: "openapi:x-rate-limit", "smithy:aws.api#arn", -// "graphql:@key", "typespec:@myOrg/decorator", "asyncapi:bindings.kafka" +// "graphql:@key", "typespec:@myOrg/decorator", "asyncapi:bindings.kafka", "erlang:opaque" // RawValue is the source JSON preserved verbatim. ``` Extensions are the lossless escape hatch: any source metadata without a first-class IR node survives, namespaced so two formats' extensions never collide, and typed passes can promote -well-known extensions (`x-ms-pagination` → `Pagination`) without losing the original. +well-known extensions (`x-ms-pagination` → `Pagination`) without losing the original. For the +escape hatch to hold, **every node that can carry source metadata has an `Extensions` field** — +including `Response`, `ErrorCase`, `Payload`, `Content`, `Example`, `ServerVariable`, +`OAuthFlow`, and every binding struct (new spec revisions land fields on exactly these objects). +Per-file metadata (protobuf file options) is keyed by path in `Document.Extensions`. ## 13. Provenance & diagnostics @@ -853,26 +1282,37 @@ How each format's distinctive concepts land in the IR (full details live with ea | Format | Lowering highlights | |---|---| -| **OpenAPI 3.x** | components/schemas → registry (IDs from pointers); inline schemas hoisted with hints; `allOf` → Base/Mixins per §4.3; `oneOf`/`anyOf` → Union (Exclusive bit), null-variant → Nullable ref; `discriminator` → Discriminator; `nullable`/type-arrays → Nullable; readOnly/writeOnly → Visibility; parameters → Params + HTTPBinding locations w/ style/explode; requestBody/responses all content types → Payload.Contents; per-status responses/default → Conditions + ranges; webhooks → MessageBinding or HTTPBinding.IsWebhook; callbacks → Callbacks; links → extensions (promotable later); securitySchemes/security → Auth OR-of-ANDs; servers+variables → Servers; `xml` object → XMLHints; `not`/`if-then-else`/`dependentSchemas` → verbatim Extensions per §4.7; `patternProperties` → AdditionalProps.Patterns; `x-*` → namespaced Extensions; pagination only via injectable policy, marked Inferred | +| **OpenAPI 3.x** | components/schemas → registry (IDs from pointers); inline schemas hoisted with hints; `allOf` → Base/Mixins per §4.3; `oneOf`/`anyOf` → Union (Exclusive bit), null-variant → Nullable ref; `discriminator` → Discriminator (3.2 `defaultMapping` → Discriminator.Default); `nullable`/type-arrays → Nullable; readOnly/writeOnly → Visibility (schema-level readOnly pushed down to referencing properties, residue → Extensions + diagnostic); `additionalProperties: false` → Additional=closed, `unevaluatedProperties: false` → closed_after_composition; parameters → Params + HTTPBinding locations w/ style/explode, 3.2 `in: querystring` → querystring location; requestBody/responses all content types → Payload.Contents; 3.2 `itemSchema`/`itemEncoding` → Content.Item/ItemEncoding; per-status responses/default → Conditions + ranges; webhooks → HTTPBinding.IsWebhook; callbacks → Callbacks; links → extensions (promotable later); securitySchemes/security → Auth OR-of-ANDs, 3.2 device flow + `oauth2MetadataUrl` → Flows/OAuth2MetadataURL; servers+variables (3.2 named) → Servers; tags (3.2 parent/kind) → groups + TagDefs; info contact/license → Document; schema `example(s)` → Examples; `xml` object (incl. 3.2 nodeType) → XMLHints at type and property level; `not`/`if-then-else`/`dependentSchemas`/`contains`/`unevaluated*` → verbatim Extensions per §4.7; `patternProperties` → AdditionalProps.Patterns; `x-*` → namespaced Extensions (legal on every object — hence Extensions on every node); `$ref`-adjacent sibling keywords (3.1) and ref-target annotations merge onto the referencing Property/Parameter with **use-site precedence**, applied uniformly (oagen's ad-hoc per-site patching is the counterexample); a oneOf/anyOf whose variants are all string consts normalizes to a closed `Enum` in a `pass/` normalization — not in the frontend — so per-variant `Docs` survive until the collapse is chosen; mutually-exclusive parameter groups (`x-mutually-exclusive-parameter-groups`) stay as namespaced Extensions, and their documented *promotion* (no dedicated node needed) is a pass that synthesizes one logical `Parameter` typed by a `Union` of variant models, bound via `HTTPParamBinding.ParamPath` per field; pagination only via injectable policy, marked Inferred | | **Swagger 2.0** | lifted to OpenAPI 3.x shape first (body/formData → Payload; host/basePath/schemes → Servers; consumes/produces → content types), then the OpenAPI lowering runs | -| **TypeSpec** | consumed post-check (monomorphized, `isFinished`); template instances → TypeCommon.Instantiation; models → Model w/ Base + spread provenance → Mixins; scalars → Scalar chains; `@encode` → Encoding triple; unions w/ named variants → Union, `@discriminated` envelope → Discriminator.Envelope; `| null` → Nullable; visibility enums (incl. custom classes) → Visibility; interfaces → OperationGroups; `@overload` → OverloadOf; `@service` → Service; versioning decorators incl. `@typeChangedFrom` → Availability timeline; `@list`/`@pageItems` paths → Pagination PropPaths; `@pollingOperation`/`@finalOperation` → LongRunning; multipart w/ parts → Content.Encoding/PartEncoding, file bodies → FileInfo; `@error` → UsageFlags.Error; values/consts incl. enum-member refs → Values channel | -| **Smithy 2.0** | structures → Model, mixins → Mixins; `document` → Any; unions → WireTagged Union; enum/intEnum → Enum (open by default); `@sparse` → element Nullable; traits: constraints → Constraints, `@paginated` → Pagination (declared), `@retryable` → ErrorCase.Retryable, `@readonly` → Idempotency safe, `@idempotent`/`@idempotencyToken` → Idempotency, `@sensitive` → Sensitive/Secret, `@tags` → Tags; `@streaming` blob → StreamDetail; event streams → StreamDetail.Events union + Property.EventHeader + Initial messages; service-level errors → Service.CommonErrors; resources → OperationGroup + ResourceInfo (identifiers, properties, lifecycle map); http traits → HTTPBinding incl. `@endpoint`/`@hostLabel` → HostPrefix/host location, `@httpPrefixHeaders`/`@httpQueryParams` → Prefix bindings; `@jsonName` → WireName; xml traits → XMLHints; other traits → namespaced Extensions | -| **GraphQL** | §8.4; interfaces → Abstract models + Implements; field arguments → Property.Args; input objects → `InputOnly` models; non-null wrapping → Required/Nullable; deprecation w/ reason; custom scalars → Scalar; directives → Extensions | -| **AsyncAPI** | servers w/ protocols → Servers; channels/messages → Channels/Messages; operations send/receive → Operation + MessageBinding; correlation IDs → PropPath; protocol bindings → Channel.Bindings raw; message payload schemas share the same JSON-Schema lowering as OpenAPI | -| **Protobuf** | messages → Model w/ WireIDs; oneof → WireTagged Exclusive Union w/ variant WireIDs; enums → open Enum w/ int32 values; map/repeated → MapT/List; scalar wire variants (sint/fixed) → Encoding; services/rpcs → Service/Operation + RPCBinding; streaming modifiers → StreamingMode; options → Extensions; reserved ranges → Extensions (guarded by validate pass) | +| **TypeSpec** | consumed post-check (monomorphized, `isFinished`); template instances → TypeCommon.Instantiation incl. value args → TemplateArg; models → Model w/ Base + spread provenance → Mixins; scalars → Scalar chains, constructors in values → Value.Ctor; `@encode`/`@format` → Encoding triple; `@encodedName` → WireNameByFormat at property AND type level; unions w/ named variants → Union, `@discriminated` → Discriminator.PropertyName/Envelope/EnvelopeValueName; `| null` → Nullable; visibility classes (incl. custom, `@invisible` → Visibility.None) → Visibility, op overrides → ParameterVisibility/ReturnTypeVisibility; `@patch` implicitOptionality → HTTPBinding.PatchImplicitOptionality; interfaces → OperationGroups (versionable); `@overload` → OverloadOf; `@sharedRoute` → SharedRoute; `@service` → Service; versioning decorators incl. `@typeChangedFrom`/`@madeOptional`/`@madeRequired` and add/remove cycles → Availability timeline (on members/variants/params too); pagination decorators incl. prev/first/last links and header continuation tokens → Pagination PropPaths (In:"header"); Azure.Core `@pollingOperation`/`@finalOperation` → LongRunning; multipart w/ parts → Content.Encoding/PartEncoding, `Http.File` → FileInfo (content-type set, contents chain, filename location); streams/SSE → StreamDetail + Variant.Event (contentType, terminal); `@error` → UsageFlags.Error; `@example`/`@opExample` → Examples (Input/Output pairs); `@pattern` message → Constraints.PatternMessage; `@mediaTypeHint` → TypeCommon.MediaTypeHint; `never` members deleted + diagnostic per §4.8; TCGC client-shaping decorators (`@clientName`, `@access`, `@usage`, `@scope`, `@override`, …) → namespaced Extensions consumed by backend policy, never IR semantics; values/consts incl. enum-member refs → Values channel | +| **Smithy 2.0** | structures → Model, mixins → Mixins (non-structure mixins flattened — spec-sanctioned); `document` → Any; unions → WireTagged Union, member `@jsonName` → Variant.WireName; enum/intEnum → Enum (open by default); `@sparse` → element Nullable; traits: constraints → Constraints, `@paginated` → Pagination (declared), `@retryable` → ErrorCase.Retryable + Throttling, `@error` fault → ErrorCase.Fault, `@readonly` → Idempotency safe, `@idempotent`/`@idempotencyToken` → Idempotency, `@sensitive` → Sensitive/Secret, `@tags` → Tags, `@clientOptional`/`@input` → Property.ClientOptional (+InputOnly), `@addedDefault` → DefaultAdded, root-shape `@default` pushed down to properties w/ provenance; `@streaming` blob → StreamDetail (+`@requiresLength` → RequiresLength); event streams → StreamDetail.Events union + Property.EventHeader/EventPayload + Initial messages; service-level errors → Service.CommonErrors; protocol traits → Service.Protocols; service `rename`/`version` → Service.Renames/Version; resources → OperationGroup + ResourceInfo (identifiers, properties, lifecycle incl. put/@noReplace, instance vs collection ops); http traits → HTTPBinding incl. `@endpoint`/`@hostLabel` → HostPrefix/host location (additive binding), `@httpPrefixHeaders`/`@httpQueryParams` → Prefix bindings, `@httpResponseCode` → Response.StatusCodeProp, `@requestCompression` → Compression, `@httpChecksumRequired` → ChecksumRequired; `@auth` order → priority-ordered Auth, `@optionalAuth` → empty option; `@jsonName` → WireName; `@mediaType` → Encoding.MediaType; xml traits → XMLHints at type and property level; `@examples` → Examples (Input/Output/Error); waiters + rules-engine traits → verbatim Extensions (§15); `smithy.api#Unit` → nil payload / shared empty Model for tag-only variants; other traits → namespaced Extensions | +| **GraphQL** | §8.4; interfaces (incl. interface hierarchies) → Abstract models + Implements; field arguments → Property.Args; input objects → `InputOnly` models, `@oneOf` inputs → WireTagged Exclusive Union; non-null wrapping → Required/Nullable (all `[T!]!` combinations via per-layer list nodes); defaults on args AND input fields → Default (list-input coercion normalized); custom scalars → Scalar{Base: nil}, `@specifiedBy` → Extensions; deprecation w/ reason at every location incl. args/input fields; directives → ordered-array Extensions convention + document-level definitions | +| **AsyncAPI** | servers w/ protocols/protocolVersion/security → Servers (named, w/ Auth); channels → Channels (Address nil = unknown), messages → Document.Messages registry referenced by ID; operations send/receive → Operation + MessageBinding, no-reply sends → OneWay; reply objects → Reply (static channel + dynamic PropPath address + message set); correlation IDs → PropPath (In: header|payload); message headers → hoisted header model (Headers *TypeRef); message examples → Example{Headers, Value} pairs; multi-format payload schemas → Content.SchemaFormat + verbatim schema in Extensions (Avro payloads lower through the type graph: record→Model, enum→closed Enum w/ FallbackMember, fixed→bytes w/ length, decimal→Constraints.Precision/Scale, aliases→Naming.Aliases); parameter `location` → Parameter.ValueFrom; traits applied by merge w/ provenance; protocol bindings at server/channel/operation/message level → raw Bindings maps; payload JSON-Schema lowering shared with OpenAPI | +| **Protobuf** | messages → Model w/ WireIDs; oneof → WireTagged Exclusive Union w/ variant WireIDs + Flatten on the synthetic wrapper property (oneof members are top-level wire fields; synthetic oneofs from proto3 `optional` do NOT lower to unions — they are presence markers); presence disciplines → Property.Presence (never Nullable); enums → Enum open/closed per syntax/edition, allow_alias → duplicate member values; map/repeated → MapT/List (packed/expanded → List.Encoding); groups / editions DELIMITED → nested Model + Encoding "delimited"; scalar wire variants (sint/fixed) → Encoding; `extend` fields → Property.ExtensionOf, extension ranges → Model.ExtensionRanges; well-known types → External (wrapper types → External or nullable primitive per injectable policy); custom options → namespaced Extensions (proto-JSON), file options keyed by path in Document.Extensions; services/rpcs → Service/Operation + RPCBinding; streaming modifiers → StreamingMode; gRPC transcoding (google.api.http) → additional HTTPBinding entries (slice), path patterns → PathPattern, nested bindings → ParamPath, response_body → ResponseBodyPath; reserved ranges → Extensions (guarded by validate pass) | +| **Erlang/OTP** | §8.5; `-type`/`-spec` type language → type graph: tuples → Tuple, tagged-tuple unions → Union + Discriminator.Index, records → Model{Positional} w/ 1-based WireIDs, atoms → symbol Values / Literal types / `Scalar{Base: string, Encoding: "erlang:atom"}`, integer ranges → integer Scalar + Min/Max, maps `:=`/`=>` → Properties(Required)/AdditionalProps w/ typed keys, `pid()`/`port()`/`reference()`/`fun()` → External, `string()` → List of char scalar, parametrized types monomorphized w/ Instantiation; behaviours: gen_server call/cast → Operations + OTPBinding (cast → OneWay), info → channel Messages received, gen_event → Channel w/ N messages, gen_statem states → `Extensions["otp:states"]`; registered process → Channel.Address, registration kind → Channel.Bindings["otp"]; reply-union Errors split → injectable Inferred policy; bit-sized binaries/map assoc lists/`-opaque` → §4.8 degraded lowerings; `-deprecated` → Deprecation; `-doc`/EDoc → Docs | ## 15. Deliberate exclusions - **Language names/casings** — backends own identifier rendering entirely (IR stores neutral word sequences + wire names). - **SDK runtime policy** (retry/timeout/telemetry/error-class taxonomy) — a separate backend - input, never IR (§2.4 of architecture.md). + input, never IR (§2.4 of architecture.md). OTP call timeouts and `multi_call`/`send_request` + machinery fall here too. - **Generator plan artifacts** (request builders, executor pairs, per-visibility model variants, primary-response selection) — computed views in backends, not stored. - **Structured modeling of transport-deployment minutiae** (Kafka partition configs, AMQP exchange args) — preserved as raw namespaced extensions. +- **Smithy rules-engine traits** (`@endpointRuleSet`, context params) — runtime endpoint + resolution machinery; preserved as raw namespaced extensions on the service. +- **Smithy waiters** (`smithy.waiters#waitable`) — preserved verbatim in `Operation.Extensions`, + never folded into `LongRunning`; promoted to a typed node only if a second format lands. +- **TCGC-style client-shaping decorators** (`@clientName`, `@access`, `@usage`, `@scope`, …) — + per-language SDK-surface policy, preserved as namespaced extensions for backend policy layers. - **Arbitrary GraphQL persisted queries** — the type graph and entry points are retained; query composition is a backend/runtime feature. +- **Capability/interface-typed fields** (Cap'n Proto capability passing) — requires + services-as-types; out of scope for a data-SDK compiler. +- **Function types** — Erlang funs degrade per §4.8; no target language marshals closures. ## 16. Open questions (tracked for implementation) @@ -889,8 +1329,19 @@ How each format's distinctive concepts land in the IR (full details live with ea runs but churn when a schema moves within its source file. TCGC solves cross-version correlation with name-based `crossLanguageDefinitionId`s. Likely resolution: keep pointer IDs as the structural identity and add a derived, name-based correlation key computed by a pass — - decide once IR diffing between spec revisions is actually built. + decide once IR diffing between spec revisions is actually built. Evidence from oagen's + differ/compat stack: (a) diffing IR-document-to-IR-document is the right plane (oagen already + does this); (b) name-based correlation alone still misses renames — plan a third *structural* + tier (content-hash / field-set similarity, shared with the `dedup` pass) as the fallback when + both pointer and name churn. Without stable identity, rename detection costs hundreds of + lines of best-effort heuristics downstream (see prior-art.md §1). 6. **Response discrimination placement** — `ResponseConditions` (status ranges) is HTTP-shaped - living on the neutral core. Alternatives: move conditions into `HTTPBinding` alongside - parameter locations, leaving responses as an ordered named list. Revisit when the RPC or - messaging frontend lands and either validates or strains the current shape. + living on the neutral core. Fresh evidence that it sits awkwardly: RPC single-response and OTP + call replies both use empty conditions, and `Response.StatusCodeProp` is likewise + HTTP-flavored. Alternatives: move conditions into `HTTPBinding` alongside parameter locations, + leaving responses as an ordered named list. Revisit when the RPC or messaging frontend lands + and either validates or strains the current shape. +7. **Semantic nullability (GraphQL)** — the `@semanticNonNull` RFC ("null only on error") is a + third nullability state `TypeRef.Nullable` cannot express. Pre-spec today: preserved as + `Extensions["graphql:@semanticNonNull"]`. If it merges into the spec, add a `NullKind` (or a + `NullOnlyOnError` bool) rather than churning every frontend early. diff --git a/docs/ir-spec-matrix.md b/docs/ir-spec-matrix.md index 6ef362b..6a501e4 100644 --- a/docs/ir-spec-matrix.md +++ b/docs/ir-spec-matrix.md @@ -3,59 +3,68 @@ What each source specification format can express, and what the Morphic IR must therefore be able to represent without loss. The IR is designed against the **union** of these capabilities, not the intersection — a generator target may ignore a capability, but the IR must never drop one. +Erlang/OTP is included as a frontend target: its "spec" is module type information +(`-spec`/`-type` on behaviour callbacks) plus the gen_server/gen_statem/gen_event message +protocols. Legend: ✅ native concept · ⚠ expressible indirectly · — absent -| Capability | OpenAPI 3.x | Swagger 2.0 | TypeSpec | Smithy 2.0 | GraphQL | AsyncAPI | Protobuf | -|---|---|---|---|---|---|---|---| -| Named object types | ✅ components.schemas | ✅ definitions | ✅ model | ✅ structure | ✅ type/input | ✅ schemas | ✅ message | -| Inline/anonymous types | ✅ | ✅ | ✅ | — (all named) | ⚠ | ✅ | ⚠ nested | -| Inheritance / base types | ⚠ allOf | ⚠ allOf | ✅ extends | ⚠ mixins | ✅ interfaces | ⚠ allOf | — | -| Mixins / spread | — | — | ✅ spread | ✅ mixins | — | ⚠ traits | — | -| Tagged unions | ⚠ oneOf+discriminator | — | ✅ discriminated union | ✅ union | ⚠ union+__typename | ⚠ oneOf | ✅ oneof | -| Untagged unions | ✅ oneOf/anyOf | — | ✅ union | — | — | ✅ oneOf | — | -| Intersection | ✅ allOf | ✅ allOf | ⚠ & (model is) | — | — | ✅ allOf | — | -| Negation | ✅ not (3.1) | — | — | — | — | ✅ not | — | -| Enums (string) | ✅ | ✅ | ✅ named members | ✅ enum | ✅ | ✅ | ⚠ | -| Enums (numeric, valued) | ✅ | ✅ | ✅ | ✅ intEnum | — | ✅ | ✅ | -| Open enums (unknown values allowed) | ⚠ anyOf trick | — | ⚠ union w/ string | ✅ (enums are open by default) | — | ⚠ | ✅ (proto3 semantics) | -| Custom scalars | ⚠ type+format | ⚠ | ✅ scalar extends | ⚠ traits | ✅ scalar | ⚠ | — | -| Wire encoding hints (@encode / format) | ✅ format | ✅ format | ✅ @encode | ✅ timestampFormat | — | ✅ | ✅ fixed/zigzag | -| Field wire IDs (numeric tags) | — | — | — | — | — | — | ✅ field numbers | -| Wire name ≠ model name | ✅ (property key) | ✅ | ✅ @encodedName | ✅ jsonName | — | ✅ | ✅ json_name | -| Optionality vs nullability distinct | ✅ (3.1) | ⚠ | ✅ | ✅ | ✅ | ✅ | ⚠ presence | -| Defaults | ✅ | ✅ | ✅ | ✅ | ✅ args | ✅ | ✅ proto2 | -| Constraints (min/max/pattern…) | ✅ | ✅ | ✅ decorators | ✅ traits | ⚠ directives | ✅ | ⚠ protovalidate | -| readOnly/writeOnly / visibility | ✅ | ✅ readOnly | ✅ @visibility lifecycle | — | ✅ input vs output types | — | — | -| Recursive types | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | -| Maps / additionalProperties | ✅ | ✅ | ✅ Record | ✅ map | — | ✅ | ✅ map | -| Tuples | ✅ prefixItems (3.1) | — | ✅ | — | — | ✅ | — | -| Literal types | ✅ const | ⚠ single enum | ✅ | — | — | ✅ | — | -| Operations grouped by service/interface | ⚠ tags | ⚠ tags | ✅ interface/namespace | ✅ service/resource | ✅ Query/Mutation/Subscription | ⚠ | ✅ service | -| Resource hierarchy (CRUDL) | — | — | ⚠ @autoRoute | ✅ resource | — | — | — | -| HTTP binding (method/path/status) | ✅ | ✅ | ✅ @route/@get… | ✅ http traits | — | ⚠ ws binding | ⚠ transcoding | -| Param styles (explode, matrix…) | ✅ style/explode | ⚠ collectionFormat | ✅ | ✅ | — | — | — | -| Multiple content types per body | ✅ | ⚠ consumes | ✅ @header contentType | ⚠ | — | ✅ | — | -| Multipart/form encoding | ✅ encoding | ✅ formData | ✅ multipart | — | — | — | — | -| Per-status error types | ✅ responses | ✅ | ✅ @error models | ✅ errors list | ⚠ errors union | — | ⚠ status codes | -| Streaming: server (SSE/chunk) | ⚠ text/event-stream | — | ✅ streams | ✅ eventstream | ✅ subscription | ✅ | ✅ stream | -| Streaming: client / bidi | — | — | ✅ | ✅ | — | ✅ | ✅ | -| Events / pub-sub channels | ✅ webhooks (3.1) | — | ⚠ | — | ✅ subscriptions | ✅ channels | — | -| Callbacks | ✅ callbacks | — | — | — | — | ⚠ reply | — | -| Pagination (first-class) | ⚠ x-* / links | — | ✅ @list/@pageItems | ✅ paginated trait | ⚠ connections | — | ⚠ AIP-158 | -| Long-running operations | ⚠ x-* | — | ✅ @pollingOperation | ⚠ | — | — | ✅ LRO (google) | -| Idempotency | ⚠ verb semantics | ⚠ | ⚠ | ✅ idempotent/@idempotencyToken | — | — | ✅ idempotency_level | -| Auth schemes | ✅ securitySchemes | ✅ | ✅ @useAuth | ✅ auth traits | ⚠ | ✅ | ⚠ | -| Per-op auth override (AND/OR) | ✅ security | ✅ | ✅ | ✅ | — | ✅ | — | -| Servers / endpoints | ✅ servers+vars | ✅ host | ✅ @server | ✅ endpoint | ⚠ | ✅ servers+protocols | — | -| Protocol bindings (kafka/amqp/…) | — | — | — | — | — | ✅ bindings | — | -| Versioning (added/removed) | — | — | ✅ @added/@removed | — | — | — | ✅ reserved | -| Deprecation w/ message | ✅ deprecated | ✅ | ✅ #deprecated | ✅ @deprecated | ✅ @deprecated(reason) | ✅ | ✅ | -| Examples | ✅ | ✅ | ⚠ | ✅ trait | — | ✅ | — | -| Docs: summary + description | ✅ | ✅ | ✅ @doc/@summary | ✅ @documentation | ✅ description | ✅ | ✅ comments | -| Vendor extensions / traits / directives | ✅ x-* | ✅ x-* | ✅ decorators | ✅ traits | ✅ directives | ✅ x-* | ✅ options | -| Field arguments (parameterized fields) | — | — | — | — | ✅ | — | — | -| Client-selectable response shape | — | — | — | — | ✅ selection sets | — | — | +| Capability | OpenAPI 3.x | Swagger 2.0 | TypeSpec | Smithy 2.0 | GraphQL | AsyncAPI | Protobuf | Erlang/OTP | +|---|---|---|---|---|---|---|---|---| +| Named object types | ✅ components.schemas | ✅ definitions | ✅ model | ✅ structure | ✅ type/input | ✅ schemas | ✅ message | ✅ -record/-type | +| Inline/anonymous types | ✅ | ✅ | ✅ | — (all named) | ⚠ | ✅ | ⚠ nested | ✅ type exprs | +| Inheritance / base types | ⚠ allOf | ⚠ allOf | ✅ extends | ⚠ mixins | ⚠ interfaces (conformance, not inheritance) | ⚠ allOf | — | — | +| Mixins / spread | — | — | ✅ spread | ✅ mixins | — | ⚠ traits | — | — | +| Tagged unions | ⚠ oneOf+discriminator | — | ✅ discriminated union | ✅ union | ⚠ union+__typename · ✅ @oneOf inputs (draft) | ⚠ oneOf | ✅ oneof | ✅ tagged tuples | +| Untagged unions | ✅ oneOf/anyOf | — | ✅ union | — | — | ✅ oneOf | — | ✅ \| | +| Intersection | ✅ allOf | ✅ allOf | ⚠ & (model is) | — | — | ✅ allOf | — | — | +| Negation | ✅ not | — | — | — | — | ✅ not | — | — | +| Enums (string) | ✅ | ✅ | ✅ named members | ✅ enum | ✅ | ✅ | ⚠ | ⚠ atom unions | +| Enums (numeric, valued) | ✅ | ✅ | ✅ | ✅ intEnum | — | ✅ | ✅ | ⚠ int unions | +| Open enums (unknown values allowed) | ⚠ anyOf trick | — | ⚠ union w/ string | ✅ (enums are open by default) | — | ⚠ | ✅ open (proto3/editions) / closed (proto2, per-enum feature) | ⚠ atom() fallback | +| Custom scalars | ⚠ type+format | ⚠ | ✅ scalar extends | ⚠ traits | ✅ scalar | ⚠ | — | ✅ -type/-opaque | +| Wire encoding hints (@encode / format) | ✅ format | ✅ format | ✅ @encode | ✅ timestampFormat | — | ✅ | ✅ fixed/zigzag/packed/delimited | — (ETF fixed) | +| Field wire IDs (numeric tags) | — | — | — | — | — | — | ✅ field numbers | ⚠ tuple positions | +| Wire name ≠ model name | ✅ (property key) | ✅ | ✅ @encodedName | ✅ jsonName (incl. union members) | — | ✅ | ✅ json_name | — | +| Optionality vs nullability distinct | ✅ (3.1) | ⚠ | ✅ | ⚠ presence only (null via @sparse collections) | ✅ | ✅ | ⚠ presence (3-state: implicit/explicit/required) | ⚠ :=/=> + 'undefined' | +| Defaults | ✅ | ✅ | ✅ | ✅ | ✅ args + input fields | ✅ | ✅ proto2 | ⚠ record fields | +| Constraints (min/max/pattern…) | ✅ | ✅ | ✅ decorators | ✅ traits | ⚠ directives (convention only) | ✅ | ⚠ protovalidate | ⚠ ranges, bit sizes | +| readOnly/writeOnly / visibility | ✅ | ✅ readOnly | ✅ @visibility classes | — | ✅ input vs output types | ⚠ (JSON Schema readOnly) | — | — | +| Recursive types | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| Maps / additionalProperties | ✅ | ✅ | ✅ Record | ✅ map | — | ✅ | ✅ map | ✅ :=/=> | +| Tuples | ✅ prefixItems (3.1) | — | ✅ | — | — | ✅ | — | ✅ native | +| Literal types | ✅ const | ⚠ single enum | ✅ | — | — | ✅ | — | ✅ atoms/ints | +| Operations grouped by service/interface | ✅ tags (3.2 parent/kind) | ⚠ tags | ✅ interface/namespace | ✅ service/resource | ✅ Query/Mutation/Subscription | ⚠ | ✅ service | ✅ module | +| Resource hierarchy (CRUDL) | — | — | ⚠ @autoRoute | ✅ resource (incl. put, instance vs collection ops) | — | — | — | — | +| HTTP binding (method/path/status) | ✅ | ✅ | ✅ @route/@get… | ✅ http traits | — | ⚠ ws binding | ⚠ transcoding | — | +| Param styles (explode, matrix…) | ✅ style/explode | ⚠ collectionFormat | ✅ | ✅ | — | — | — | — | +| Multiple content types per body | ✅ | ⚠ consumes | ✅ @header contentType | ⚠ | — | ✅ | — | — | +| Multipart/form encoding | ✅ encoding | ✅ formData | ✅ multipart | — | — | — | — | — | +| Per-status error types | ✅ responses | ✅ | ✅ @error models | ✅ errors list (client/server fault) | — | — | ⚠ status codes | ⚠ {error, R} variants | +| Streaming: server (SSE/chunk) | ✅ itemSchema/sequential media types (3.2) | — | ✅ streams | ✅ eventstream | ✅ subscription | ✅ | ✅ stream | ⚠ info streams | +| Streaming: client / bidi | — | — | ✅ client · ⚠ bidi | ✅ | — | ✅ | ✅ | ⚠ cast/info flows | +| Events / pub-sub channels | ✅ webhooks (3.1) | — | ✅ events/sse | — | ✅ subscriptions | ✅ channels | — | ✅ gen_event/info | +| Callbacks / request-reply | ✅ callbacks | — | — | — | — | ✅ reply (static + dynamic address) | — | ⚠ From-reply | +| Pagination (first-class) | ⚠ x-* / links | — | ✅ @list/@pageItems + prev/first/last links | ✅ paginated trait | ⚠ connections | — | ⚠ AIP-158 | — | +| Long-running operations | ⚠ x-* | — | ⚠ Azure.Core @pollingOperation | ⚠ smithy.waiters | — | — | ⚠ google.longrunning | ⚠ send_request | +| Idempotency | ⚠ verb semantics | ⚠ | — | ✅ idempotent/@idempotencyToken | — | — | ✅ idempotency_level | — | +| Auth schemes | ✅ securitySchemes | ✅ | ✅ @useAuth | ✅ auth traits | — | ✅ (wide: SASL/X509/userPassword; attaches to servers) | ⚠ | — | +| Per-op auth override (AND/OR) | ✅ security | ✅ | ✅ | ⚠ OR only, priority-ordered | — | ✅ | — | — | +| Servers / endpoints | ✅ servers+vars (3.2 named) | ✅ host | ✅ @server | ⚠ @endpoint hostPrefix only | — | ✅ named servers+protocols+security | — | ⚠ nodes/registry | +| Protocol bindings (kafka/amqp/…) | — | — | — | — | — | ✅ bindings | — | ✅ behaviours | +| Versioning (added/removed) | — | — | ✅ @added/@removed | ⚠ @since | — | — | — | — | +| Deprecation w/ message | ✅ deprecated | ✅ | ✅ #deprecated | ✅ @deprecated | ✅ @deprecated(reason) | ✅ | ✅ | ✅ -deprecated | +| Examples | ✅ | ✅ | ✅ @example/@opExample | ✅ trait (input/output/error scenarios) | — | ✅ (header+payload pairs) | — | — | +| Docs: summary + description | ✅ | ✅ | ✅ @doc/@summary | ✅ @documentation | ✅ description | ✅ | ✅ comments | ✅ -doc/EDoc | +| Vendor extensions / traits / directives | ✅ x-* | ✅ x-* | ✅ decorators | ✅ traits | ✅ directives (ordered, repeatable) | ✅ x-* | ✅ options | ⚠ module attributes | +| One-way (fire-and-forget) operations | — | — | — | — | — | ✅ send w/o reply | — | ✅ cast | +| Positional wire encoding (records as tuples) | ⚠ prefixItems | — | ⚠ tuples | — | — | ⚠ items array | — | ✅ records/tuples | +| Symbol/atom literal values | — | — | — | — | — | — | — | ✅ atoms | +| Unsolicited server-initiated messages | ⚠ webhooks | — | — | — | ⚠ subscriptions | ✅ channels | — | ✅ info | +| Multi-format payload schemas | — | — | — | — | — | ✅ schemaFormat (Avro/Protobuf/RAML) | — | — | +| Third-party field extensions / extension ranges | — | — | — | — | — | — | ✅ extend/extensions | — | +| Field arguments (parameterized fields) | — | — | — | — | ✅ | — | — | — | +| Client-selectable response shape | — | — | — | — | ✅ selection sets | — | — | — | ## Consequences for the IR @@ -63,21 +72,48 @@ Legend: ✅ native concept · ⚠ expressible indirectly · — absent optional discriminator (property name + value→variant mapping), whether the union is *tagged on the wire* (protobuf oneof, Smithy union — the wire format itself encodes the variant) vs *untagged* (JSON oneOf — variant inferred by validation), and open vs closed semantics - (anyOf ≈ open, oneOf ≈ exactly-one). -2. **Optionality ≠ nullability.** Four distinct states exist (required non-null, required nullable, - optional non-null, optional nullable); the IR keeps `Required` on the property and `Nullable` - on the type reference. + (anyOf ≈ open, oneOf ≈ exactly-one). Discrimination is not always property-based: Erlang + tagged tuples (and JSON arrays with a `const` head) discriminate by *position*, so the + discriminator needs a tuple-index form; GraphQL `@oneOf` and Smithy unions are *key-tagged* + (variant wire name is the object key). +2. **Optionality ≠ nullability ≠ presence.** Four distinct states exist (required non-null, + required nullable, optional non-null, optional nullable); the IR keeps `Required` on the + property and `Nullable` on the type reference. Protobuf adds a third axis — the *presence + discipline* (implicit/explicit/required) — which is not nullability (protobuf has no null) + and lives on its own field. 3. **Visibility must be lifecycle-based**, not a readOnly boolean: TypeSpec and GraphQL both need - per-usage (create/read/update/query) property filtering. OpenAPI readOnly/writeOnly lowers into it. + per-usage (create/read/update/query) property filtering, and TypeSpec's "visible in *no* + lifecycle" (`@invisible`) is a distinct state from "unrestricted". OpenAPI readOnly/writeOnly + lowers into it. 4. **Operations are protocol-neutral cores + protocol bindings.** The same operation node serves - HTTP (OpenAPI), RPC (protobuf/Smithy), and messaging (AsyncAPI) by attaching different bindings. - Streaming direction (unary, client, server, bidi) lives on the core, not the binding. + HTTP (OpenAPI), RPC (protobuf/Smithy), messaging (AsyncAPI), and actor protocols (OTP) by + attaching different bindings. Streaming direction (unary, client, server, bidi) and + one-way-ness (fire-and-forget: OTP cast, AsyncAPI send-without-reply, Thrift oneway, JSON-RPC + notifications) live on the core, not the binding. One operation may carry *several* bindings + of the same protocol (gRPC transcoding `additional_bindings`). 5. **Field identity is three names + an optional wire ID**: source name, IR canonical name, - wire/serialized name, and numeric tag (protobuf). Generators derive language names; the IR never - stores camelCase/PascalCase variants. + wire/serialized name, and numeric tag. The wire ID must admit zero (Cap'n Proto/FlatBuffers/ + Avro ordinals start at 0), and union *variants* need wire names too (Smithy `@jsonName` on + union members). Generators derive language names; the IR never stores camelCase/PascalCase + variants. 6. **Extensions are preserved, namespaced by origin** (`openapi:x-foo`, `smithy:aws.api#arn`, - `graphql:@key`), so no source metadata is lost and later generators can opt into them. + `graphql:@key`), so no source metadata is lost and later generators can opt into them. The + escape hatch only holds if *every* node that can carry source metadata has an extensions + slot — new spec revisions land fields on exactly the objects one forgets (responses, examples, + security schemes, server variables). 7. **GraphQL's parameterized fields and selection sets** don't map to fixed operations; the IR models schema entry-points as operations with arguments, and keeps the full type graph so a generator can offer query-building. This is deliberately lossy for arbitrary client queries — acceptable because Morphic generates SDK surface, not persisted queries. +8. **Messages need identity.** AsyncAPI reuses one named message across channels, operations, + and replies — messages live in a flat ID-keyed registry like every other named entity, and + request-reply carries both a static reply channel and a *dynamic* reply address (a path into + the request message's headers). +9. **Symbols are not strings.** Erlang atoms are a distinct term class (`ok` ≠ `<<"ok">>` on the + wire); the Values channel carries a symbol kind so backends degrade to strings explicitly, + never accidentally. +10. **Formats beyond this matrix already shape the IR.** Thrift/WSDL/Cap'n Proto service + inheritance (services carry IDs and an extends list), Avro/XSD/OData decimal + precision+scale, Avro aliases and enum fallback members, JSON-RPC positional params, and + per-type namespaces (proto packages, Avro fullnames) are all held by the IR so those + frontends never force a schema change. diff --git a/docs/prior-art.md b/docs/prior-art.md index 1bb99b5..4adc00e 100644 --- a/docs/prior-art.md +++ b/docs/prior-art.md @@ -9,6 +9,8 @@ each. This is the evidence base behind `architecture.md` and `ir-design.md`. oagen parses OpenAPI 3.x into a small structural IR (`ApiSpec` → services → operations, plus a flat model/enum registry) and hands that IR to per-language emitter plugins. +Every claim in this section has been verified against the source (parser, IR, engine, +and the differ/compat/verify subsystems added since the original survey). ### Worth adopting @@ -21,23 +23,83 @@ plus a flat model/enum registry) and hands that IR to per-language emitter plugi - **Models referenced by name/handle, never embedded** — a flat registry makes deduplication, traversal, and serialization trivial and makes recursive types a non-issue. - **A "resolved plan" layer between IR and templates**: semantic rendering decisions - (is-paginated, has-body, idempotent-post…) are computed once, language-neutrally, so string + (is-paginated, has-body, idempotent-post, primary-response selection, return-shape + classification, parameter-passing shape…) are computed once, language-neutrally, so string templates contain no policy. - **Runtime SDK policy (retry, telemetry, error taxonomy) kept in a separate tree** from the structural API description. +- **Policy addressed by wire identity, not derived names** — consumer operation hints are keyed + `"METHOD /path"`, never by generated method name, because derived names churn. Independent + convergence on stable source identity; Morphic's IDs make the same key durable across path + renames too. The hint vocabulary itself (rename, remount, split-union-body into typed wrapper + methods, constant defaults, client-config-derived fields, URL-builder ops) is a concrete + inventory of what a client-shaping backend policy input must express. +- **Group metadata over an untouched wire list** — mutually-exclusive parameter groups + (`x-mutually-exclusive-parameter-groups`) are modeled as grouping metadata whose variants + share the parameter objects by identity, while the flat wire arrays stay authoritative for + serialization. Ergonomic structure layered on wire truth, neither duplicated nor destroyed. +- **A generation manifest + provenance-gated integration** — regenerating into a live repo + rests on: a manifest (spec/emitter/config hashes, sorted file list, operation → generated + symbol map), file-header provenance that gates pruning (never delete a file lacking the + generated-by header), ignore-region markers for hand-written islands, and additive-only + AST merges. Manifests should key entities by stable ID — oagen's `"METHOD /path"` keys break + on path renames. +- **Wire-conformance smoke testing with a spec-only offline baseline** — the expected request + shape (method, path, query, body keys) is derived from the spec alone and diffed against the + generated SDK running under HTTP interception; request-side mismatches block, response-side + inform. The decisive test of a generator is bytes-on-the-wire, not compilation. +- **A behavioral-change diff channel** — default-value changes are surfaced separately from + structural signature changes (same signature, different runtime behavior), and every compat + finding carries *drift provenance* (which pipeline stage caused it) plus, where detectable, + a spec-level *remediation* hint (e.g. "new schema forks the old — extend instead"). +- **Narrow, expiring diagnostic approvals** — intentional breaking changes are allowlisted per + (symbol × change category) with a required reason, optional expiry and tracker link, and + wildcards rejected. The shape a mature CI allowlist takes. ### Mistakes to avoid | oagen behavior | Morphic decision | |---|---| -| Models keyed by *derived string names*; collisions resolved silently ("keep largest"), refs fixed by string rewriting | Stable synthetic IDs (source-pointer-derived); names are presentation metadata | -| Inline-schema naming logic duplicated across ≥4 call sites that must stay byte-identical | One hoisting pass, keyed by source-pointer identity | +| Models keyed by *derived string names*; collisions resolved silently ("keep largest"), refs fixed by string rewriting. Same failure class for hoisted inline enums (name-colliding enums silently adopt the first's member set) | Stable synthetic IDs (source-pointer-derived); names are presentation metadata | +| Inline-schema naming logic duplicated across ≥6 call sites that must stay byte-identical — worse, *inferred discriminator mappings are name strings* that must reproduce the hoisting pass's naming byte-for-byte across files or they dangle | One hoisting pass, keyed by source-pointer identity; `Discriminator.Mapping` points at IDs, severing the naming coupling | +| Reference nodes bake in the *target's* kind (`enum` vs `model` ref), so no reference can be built without global knowledge of all targets — the root cause of the two-pass module-global state | One `TypeRef{Target TypeID}`; kind lives on the registry entry | | `allOf` eagerly flattened to a field list — the inheritance relationship is lost | Composition preserved un-lowered (base + mixin provenance kept) | -| oneOf branches without discriminator merged as *optional fields* — exclusivity constraint lost | Unions always survive as union nodes | -| Pagination/envelope/discriminator detection via hardcoded name lists | Heuristics are injectable frontend policy, never IR semantics | -| Failures are `console.warn`; unresolved refs degrade to `unknown` silently | Typed diagnostics with severity + provenance, collected on the IR document | -| Single "primary response" privileged; media-type multiplicity collapsed by fixed priority | All responses and all content types are first-class | -| Mutable module-global parser state (non-reentrant) | Frontends are pure `(source, options) → (IR, diagnostics)` | +| oneOf merged as *optional fields* at top-level-schema and request-body positions — even when explicitly discriminated, for argument-spreading ergonomics (variant `required` discarded); only property-level unions survive as union nodes | Unions always survive as union nodes; ergonomic flattening is a backend plan decision | +| Pagination/envelope detection via hardcoded name lists (with a fabricated `after` cursor param when none exists); single-resource envelope unwrapping applied *destructively* — the wrapper key vanishes from the IR with no recorded path | Heuristics are injectable frontend policy, never IR semantics; unwrap decisions are recorded as `PropPath`s, never applied to the stored shape | +| Discriminator *inference* is structural (const-property across variants — sound), but its output is welded to derived names (see above); name lists appear as tie-break preference order | Structural inference marked `Inferred`; output keyed by ID | +| Failures are `console.warn`; unresolvable model refs left dangling with a warning; unrecognized schema shapes degrade to `unknown` — silently when anonymous | Typed diagnostics with severity + provenance, collected on the IR document | +| Single "primary response" privileged (all 2xx kept, but inline-model extraction and pagination classification follow whichever iterates last); media-type multiplicity collapsed by fixed priority | All responses and all content types are first-class | +| Mutable module-global parser state (non-reentrant; concurrent parses race) | Frontends are pure `(source, options) → (IR, diagnostics)` | +| Collision-cascade operation renaming: same-named ops renamed in place by appending path context — adding one endpoint can rename *existing* SDK methods | Names are presentation; identity is the ID; naming collisions are a backend policy concern | +| Name sanitization accretes hardcoded domain word-lists (acronym sets, `Json`-fork collapse passes doing registry surgery via string rewriting) | Acronym/cleanup policy is injectable per-frontend; the registry is never edited by string rewrite | +| Catch-all `additionalProperties` smuggled as a magic-named synthetic field (collides with a real property of that name); `patternProperties` truncated to the first pattern | `AdditionalProps{Value, Key, Patterns}` is its own channel | +| `$ref`-site vs ref-target annotation merging patched ad-hoc for parameters only; model fields typed by `$ref` still lose target defaults | One documented precedence rule (use-site overrides target), applied uniformly in the frontend | + +### The diff/compat/verify subsystems — identity re-derived by heuristic + +oagen grew an IR-to-IR spec differ, a cross-language backwards-compat checker (per-language +extractors parse SDK source into a neutral API-surface model, diffed under per-language policy), +and a self-correcting overlay loop. Three structural lessons: + +- **Everything correlates by display name, and it costs hundreds of lines of heuristics.** + The differ matches models/operations by name, so a rename is indistinguishable from + remove+add; the compat layer then *re-derives* identity structurally (field-set superset + matching, enum value-set equality, Jaccard-similarity overlay matching ≥0.6). Its symbol + model even has a dual-key design (`id` + name) intended for rename detection — but `id` is + derived from the name, so the rename branch is dead. Cross-language change rollup finally + reinvents a spec-level identity (`conceptualChangeId`). Morphic's pointer-derived IDs make + rename detection a lookup; the compat experience adds one refinement — when both pointer + *and* name churn, a third *structural* correlation tier (content-hash / field-set + similarity, shared with the `dedup` pass) is the fallback (see ir-design.md open question 5). +- **Breaking-ness is per-language policy, not a property of a change.** The same spec change + is breaking in PHP (param names are public API), soft-risk in Kotlin, invisible in Node. + oagen's two-stage split — neutral change records, then `(change × language) → severity` + policy — is the right architecture for any future Morphic diff pass. +- **The comparison plane is an extracted "SDK surface", not the IR.** Live SDK source and + generated output are both projected into a neutral surface model and diffed there; the IR's + roles are to generate one side and to scope the baseline. Nothing in the ~20-file compat + subsystem needed information Morphic's IR lacks — the strongest validation the audit + produced. ---