From 38cc1e0343496c82a95de28814fe7fa1b95e1b31 Mon Sep 17 00:00:00 2001 From: OmarAlJarrah Date: Mon, 20 Jul 2026 00:20:11 +0300 Subject: [PATCH 1/2] docs: extend IR to the full cross-format capability surface MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The IR draft could not losslessly host several constructs that the target source formats declare, which would have forced schema changes the moment those frontends landed. Close the gaps found by auditing the model against Smithy 2.0, TypeSpec (incl. http/streams/sse/events/versioning/rest/xml and the new visibility system), Protobuf (proto2/proto3/editions + gRPC transcoding), GraphQL (current spec + draft), AsyncAPI 3.0, OpenAPI 3.1/3.2, and a sweep of Thrift/Avro/JSON-RPC/WSDL/Cap'n Proto — and add Erlang/OTP (gen_server/gen_statem/gen_event message-protocol surface) as a supported frontend target. Headline model changes: - Operation.OneWay: fire-and-forget ops (OTP cast, AsyncAPI send-without- reply, Thrift oneway, JSON-RPC notifications) are distinct from "response with no body" - messages become registry citizens (MessageID, Document.Messages); AsyncAPI request-reply gains Reply with dynamic reply addresses - new OTPBinding (call/cast/info against a process channel); positional discrimination (Discriminator.Index) and Model.Positional for tagged tuples and records; symbol values for atoms - services get IDs, inheritance, versions, per-service renames, and protocol declarations; servers get names, auth, and protocol versions - OpBindings.HTTP becomes a slice (gRPC transcoding additional_bindings) - Property gains Presence (protobuf tri-state), ClientOptional/DefaultAdded (Smithy evolution model), EventPayload; WireID becomes *int (0 is a legal ordinal in Cap'n Proto/FlatBuffers/Avro) - Visibility.None (@invisible), type-level XML/wire-name/media-type hints, richer Example (input/output/error scenarios, header+payload pairs), Availability on enum members/variants/params with add/remove cycles and optionality flips - AuthKind covers the SASL/X509/userPassword families; Extensions and Provenance land on every node that can carry source metadata The capability matrix gains an Erlang/OTP column, six new capability rows, and ~25 cell corrections; lowering summaries are rewritten per format and documented conventions added for degraded constructs (never-typed members, bit-sized binaries, waiters, endpoint rules, GraphQL directive ordering). --- CLAUDE.md | 3 +- docs/architecture.md | 13 +- docs/ir-design.md | 642 ++++++++++++++++++++++++++++++++++------- docs/ir-spec-matrix.md | 156 ++++++---- 4 files changed, 649 insertions(+), 165 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 8d62539..47f5769 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -15,7 +15,8 @@ Standard Go tooling applies once code exists: `go build ./...`, `go test ./...`, ## What Morphic is A spec-to-SDK compiler: any API spec (OpenAPI, Swagger 2.0, TypeSpec, Smithy, GraphQL, AsyncAPI, -Protobuf) → **one spec-agnostic intermediate representation (IR)** → idiomatic SDKs and docs. +Protobuf, Erlang/OTP module specs) → **one spec-agnostic intermediate representation (IR)** → +idiomatic SDKs and docs. Pipeline: **frontends** (spec → IR) → **IR passes** (IR → IR) → **backends** (IR → artifacts). ## The documents are the spec — read them first diff --git a/docs/architecture.md b/docs/architecture.md index 4ac7f8e..77330fc 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,8 +1,8 @@ # Morphic Architecture Morphic turns any API specification — OpenAPI/Swagger, TypeSpec, Smithy, GraphQL, AsyncAPI, -Protobuf — into idiomatic SDKs and docs through a single spec-agnostic intermediate -representation (IR). This document defines the pipeline, the package layout, and the contracts +Protobuf, Erlang/OTP module specs — into idiomatic SDKs and docs through a single spec-agnostic +intermediate representation (IR). This document defines the pipeline, the package layout, and the contracts between stages. The IR itself is specified in [`ir-design.md`](./ir-design.md). ``` @@ -14,6 +14,7 @@ between stages. The IR itself is specified in [`ir-design.md`](./ir-design.md). GraphQL ──▶ └───────────────────────────────────────┘ AsyncAPI │ Protobuf ▼ + Erlang/OTP ┌── IR passes (IR → IR) ──┐ │ validate · link · dedup │ │ filter · version-slice │ @@ -86,7 +87,7 @@ Internal phases every frontend follows (each format implements them its own way) Frontends are registered in a registry keyed by detected format; the engine sniffs the source format and dispatches. Milestone 1 ships the OpenAPI 3.x frontend only; the frontend registry, -provenance model, and IR are built for all seven from day one. +provenance model, and IR are built for all eight from day one. ### 2.2 IR passes (IR → IR) @@ -143,7 +144,7 @@ morphic/ ├── frontend/ # Layer 1 — frontend contract + registry. │ ├── openapi/ # OpenAPI 3.x → IR (milestone 1). │ ├── swagger/ # 2.0 lift → openapi frontend (future). -│ ├── typespec/ smithy/ graphql/ asyncapi/ protobuf/ (future) +│ ├── typespec/ smithy/ graphql/ asyncapi/ protobuf/ otp/ (future) ├── pass/ # Layer 1 — IR → IR passes (validate, dedup, filter, slice, overlay). ├── backend/ # Layer 2 — backend contract, plan layer, registry (future). ├── engine/ # Layer 3 — orchestration: sniff format, run frontend, passes, backends. @@ -187,5 +188,5 @@ can allowlist. Nothing in the pipeline writes to stderr; the CLI renders diagnos 4. **Second family frontend (TypeSpec or Smithy)** — proves the spec-agnostic claim: richer-than- OpenAPI concepts (interfaces, custom scalars, lifecycle visibility, declared pagination) flow through untouched IR code. -5. **Event-shaped frontend (AsyncAPI)** — proves channels/messages/bindings; then GraphQL and - Protobuf. +5. **Event-shaped frontend (AsyncAPI)** — proves channels/messages/bindings; then GraphQL, + Protobuf, and Erlang/OTP (the actor-protocol frontend: behaviours → operations + channels). diff --git a/docs/ir-design.md b/docs/ir-design.md index c336db5..ad76956 100644 --- a/docs/ir-design.md +++ b/docs/ir-design.md @@ -17,7 +17,7 @@ spec — field names and shapes here are normative, receiver methods and helpers (`ir-spec-matrix.md`). A backend may ignore a capability; a frontend must never drop one. Only frontends are staged over time — the IR's capability surface is complete from day one, so shipping the OpenAPI 3.x frontend first never forces an IR schema change when TypeSpec, - Smithy, GraphQL, AsyncAPI, or Protobuf frontends land. + Smithy, GraphQL, AsyncAPI, Protobuf, or Erlang/OTP frontends land. 2. **Un-lowered.** Composition, unions, visibility, discriminators, encodings, and streaming stay in source-semantic form. Flattening and language fitting happen in backends. 3. **Stable IDs, flat registries.** All named entities live in flat, ID-keyed registries and @@ -58,16 +58,27 @@ type Document struct { Name string // API title Version string // API version string (source-declared) Docs Docs + Contact *Contact // {Name, URL, Email} (OpenAPI/AsyncAPI info.contact) + License *License // {Name, Identifier, URL} (OpenAPI/AsyncAPI info.license) + TermsOfService string Services []Service // ≥1; multi-service documents are normal (TypeSpec, stitching) Types map[TypeID]TypeDef // the type registry — the only owner of TypeDefs - Channels map[ChannelID]Channel // event/messaging layer (AsyncAPI, webhooks, subscriptions) + Channels map[ChannelID]Channel // event/messaging layer (AsyncAPI, webhooks, subscriptions, OTP processes) + Messages map[MessageID]Message // message registry — messages are reused across channels and + // referenced by identity from operations and replies (AsyncAPI 3) Auth map[AuthID]AuthScheme // auth scheme registry Servers []Server // endpoint templates + TagDefs []TagDef // tag metadata registry: {Name, Docs}; tag *membership* stays + // []string on the tagged nodes (OpenAPI/AsyncAPI tag objects) Versions []string // ordered version labels when availability metadata is used Extensions Extensions Diagnostics []Diagnostic // accumulated by frontend + passes; not part of API meaning Sources []SourceInfo // input files: format, path, content hash } + +type Contact struct { Name, URL, Email string } +type License struct { Name, Identifier, URL string } +type TagDef struct { Name string; Docs Docs } ``` A `Document` is self-contained: no node references anything outside it. @@ -81,7 +92,7 @@ A `Document` is self-contained: no node references anything outside it. ```go type TypeID string // e.g. "t/openapi/components/schemas/User" or "t/anon/paths/~1users/get/responses/200/content/application~1json" type OpID string // operation identity, same construction -type ChannelID, AuthID, PropID string +type ServiceID, ChannelID, MessageID, AuthID, PropID string ``` IDs are opaque to consumers but constructed deterministically by frontends from the source @@ -90,6 +101,10 @@ snapshots, cachable, diffable across spec revisions). IDs are never derived from and never rewritten by renames. The `dedup` pass may alias two structurally identical anonymous types; aliases are recorded so both IDs stay resolvable. +Every named entity has an ID — including services (Thrift `service B extends A`, WSDL 2.0 +interface extension, and Cap'n Proto interface inheritance all reference services by identity) +and messages (AsyncAPI reuses one named message across channels, operations, and replies). + This is the direct answer to oagen's name-keyed registry (silent collision merging, string-rewrite ref fixing) and Kiota's name-keyed children (collision reconciliation logic), and adopts the intent of TCGC's `crossLanguageDefinitionId`. @@ -101,6 +116,8 @@ type Naming struct { Source string // exactly as written in the spec ("user_id", "$ref name", GraphQL field) Canonical string // IR-normalized identifier in neutral form: lower_snake words, no casing opinions Hint string // for anonymous types only: context-derived suggestion ("connection_domain") + Aliases []string // alternate names for schema-resolution matching (Avro aliases). Versionless — + // rename *history* tied to version labels lives in Availability.RenamedFrom. } ``` @@ -121,10 +138,13 @@ Nullability lives on the reference, not the target type, because the same type i position and not another. Combined with `Property.Required` this yields the four distinct states (required/optional × nullable/non-null) that OpenAPI 3.1, TypeSpec, and GraphQL all distinguish. Frontends normalize every source spelling to this one bit: OAS 3.0 `nullable: true`, OAS 3.1 -`type: [T, "null"]`, TypeSpec `T | null`, GraphQL's absence-of-`!`, proto3 explicit presence. +`type: [T, "null"]`, TypeSpec `T | null`, GraphQL's absence-of-`!`. A oneOf/anyOf/union whose only distinction is a null variant becomes a plain nullable `TypeRef`, never a union node. +Protobuf field *presence* is **not** nullability — protobuf has no null. Presence disciplines +lower to `Property.Presence` (§5.1), keeping `Nullable` strictly about wire-null. + --- ## 4. The type graph @@ -136,18 +156,39 @@ type TypeKind string // "primitive" | "scalar" | "model" | "union" | "enum" | "l type TypeCommon struct { ID TypeID Name Naming + Namespace []string // the type's declared logical namespace (proto package, Avro namespace, + // Thrift/XSD/Cap'n Proto scopes); independent of Service.Namespace Anonymous bool // hoisted inline type Docs Docs - Tags []string // free-form labels (Smithy @tags, OpenAPI tag membership on types via policy) + Tags []string // free-form labels (Smithy @tags, OpenAPI tag membership on types via policy); + // tag metadata lives once in Document.TagDefs Sensitive bool // whole-type redaction (Smithy @sensitive on shapes); Property.Secret is the per-use form + Access string // "" = public; "internal" = not part of the exported SDK surface + // (protobuf editions export/local, TCGC @access(internal), Smithy @internal via policy) Deprecation *Deprecation Availability *Availability Usage UsageFlags // computed by a pass: Input | Output | Error | Multipart | … - Instantiation *TemplateInstantiation // provenance for monomorphized generics (TypeSpec templates): - // {Template string; Args []TypeRef} — naming hint + cross-version identity + WireNameByFormat map[string]string // type-level serialized-name overrides per mime + // (TypeSpec @encodedName on models/enums/scalars) + MediaTypeHint string // declared default content type when the type is a body + // (TypeSpec @mediaTypeHint, Smithy @mediaType on string/blob shapes) + XML *XMLHints // type-level XML wire shape: root element name/namespace + // (OpenAPI xml object, Smithy @xmlName/@xmlNamespace on shapes, TypeSpec @Xml.*) + Examples []Example // typed example values attached to the type + // (TypeSpec @example, OpenAPI schema-level examples) + Instantiation *TemplateInstantiation // provenance for monomorphized generics (TypeSpec templates) Extensions Extensions Provenance Provenance } + +type TemplateInstantiation struct { + Template string + Args []TemplateArg // instances are identified by type AND value arguments +} +type TemplateArg struct { + Type *TypeRef // exactly one of Type/Value is set + Value *Value // TypeSpec `valueof` template parameters +} ``` ### 4.1 Primitives @@ -163,6 +204,7 @@ type PrimKind string // int8 int16 int32 int64 uint8 uint16 uint32 uint64 // integer (arbitrary precision — JSON Schema "integer", TypeSpec "integer") // float32 float64 +// float (arbitrary precision *binary* floating point — TypeSpec "float", supertype of float32/64) // number (arbitrary precision — JSON Schema "number", TypeSpec "numeric") // decimal decimal128 // date time datetime datetime_offset duration @@ -173,14 +215,18 @@ type PrimKind string The set is the union of TypeSpec's intrinsic scalars, JSON Schema's types, and Protobuf's needs. Protobuf's `fixed32`/`sfixed64`/`sint*` are **encodings of** `uint32`/`int64`/…, not distinct primitives — they lower to `Encoding` (§5.3). There is no `void` type: absence of a body/return -is a `nil *TypeRef`. +is a `nil *TypeRef`. TypeSpec `safeint` is not a primitive either — its canonical lowering is +`Scalar{Base: integer, Constraints: ±(2^53−1)}`. ### 4.2 Scalars (named restricted/extended primitives) ```go type Scalar struct { TypeCommon - Base TypeRef // primitive or another scalar — extension chains preserved + Base *TypeRef // primitive or another scalar — extension chains preserved. + // nil = opaque scalar with implementation-defined representation + // (GraphQL custom scalars declare no base; backends map nil-base + // scalars to their opaque-scalar strategy, not a fabricated chain) Constraints *Constraints Encoding *Encoding } @@ -200,13 +246,27 @@ type Model struct { Implements []TypeRef // interface conformance, N-ary (GraphQL `implements A & B`); targets are Abstract models Mixins []TypeRef // composition without subtyping (Smithy mixins, TypeSpec spread provenance, extra allOf entries) AdditionalProps *AdditionalProps // map-like catch-all alongside declared properties + Additional AdditionalMode // openness of the property set beyond declared + AdditionalProps Abstract bool // cannot be instantiated directly; fields may be typed by it and // resolve to a conforming concrete model (GraphQL interface types) + Positional bool // properties serialize positionally as a tuple ordered by WireID + // (Erlang records: tag at element 1, fields at 2..N; WireID is the + // 1-based element index, matching element/2) + ExtensionRanges []WireIDRange // wire-ID ranges reserved for third-party extension fields + // (protobuf `extensions 100 to 199`) Discriminator *Discriminator // set on the polymorphic base DiscriminatorValue string // set on each subtype: its wire tag value InputOnly bool // GraphQL input types; distinct identity from output types } +type AdditionalMode string +// "" — unspecified (open by JSON Schema default) +// "closed" — no properties beyond the declared set (additionalProperties: false, +// closed-by-construction records) +// "closed_after_composition" — closed once composition is resolved (unevaluatedProperties: false) + +type WireIDRange struct { From, To int } + type AdditionalProps struct { Value TypeRef Key *TypeRef // nil = string keys @@ -215,10 +275,21 @@ type AdditionalProps struct { type PatternProps struct { Pattern string; Value TypeRef } type Discriminator struct { - Property PropID // which property carries the tag + // Exactly one of Property / PropertyName / Index locates the tag: + Property PropID // model hierarchies: the property that carries the tag + PropertyName string // unions: wire name of the tag property (it exists on no single + // model, so there is no PropID to point at; TypeSpec @discriminated + // discriminatorPropertyName, OpenAPI discriminator on oneOf) + Index *int // positional: 0-based tuple element carrying the tag Literal + // (Erlang tagged tuples {ok, V} | {error, R}; JSON arrays with a + // const head via prefixItems) Mapping map[string]TypeID // wire value → subtype; nil mapping = infer by type name + Default TypeID // variant to use when the tag is absent/unrecognized + // (OpenAPI 3.2 defaultMapping); zero = none Envelope string // "" = tag inline in the variant object; // "object" = {kind, value} wrapper (TypeSpec @discriminated envelope) + EnvelopeValueName string // wire name of the envelope's value property (default "value"; + // TypeSpec envelopePropertyName); meaningful only when Envelope=="object" Inferred bool // discovered heuristically (const-property detection), not declared } ``` @@ -251,18 +322,37 @@ type Union struct { type Variant struct { Name Naming // named variants (TypeSpec, Smithy, protobuf oneof fields); Hint-only for bare oneOf members Type TypeRef - WireID int // protobuf oneof field number; 0 = none + WireName string // serialized tag when it differs from Name.Source + // (Smithy @jsonName on union members, protobuf oneof json_name) + WireID *int // protobuf oneof field number, Cap'n Proto/Avro ordinal; nil = none + // (pointer because 0 is a legal ordinal in several formats) + XML *XMLHints // @xmlName/@xmlNamespace on union members + Event *EventInfo // event-stream metadata when the union is a stream's event set Docs Docs Deprecation *Deprecation + Availability *Availability + Examples []Example Extensions Extensions } + +type EventInfo struct { + ContentType string // per-event content type (TypeSpec @Events.contentType) + Terminal bool // receiving this event ends the stream (TypeSpec @SSE.terminalEvent) +} ``` One node covers the whole design space from the capability matrix: untagged `anyOf` (`Exclusive=false`), untagged `oneOf` (`Exclusive=true`), discriminated `oneOf` (`+Discriminator`), natively tagged unions (`WireTagged=true`). Variant identity survives so backends can generate named accessors, sealed interfaces, or wrapper types per their refiner -strategy. Unions never degrade to optional-field merges in the IR. +strategy. Unions never degrade to optional-field merges in the IR — that includes GraphQL +`@oneOf` input objects, which are spec-level tagged input unions and lower here, not to models. + +**Tag-mode conventions** (normative): `WireTagged=true` with a `Discriminator` means the tag is +a property inside the variant payload (GraphQL `__typename`). `WireTagged=true` with **nil** +`Discriminator` means the union is *key-tagged*: the wire shape is a single-key object whose key +is the variant's wire name (Smithy unions, proto3-JSON oneof, GraphQL `@oneOf` inputs). +`WireTagged=false` means the variant is inferred by validation (JSON oneOf/anyOf). ### 4.5 Enums @@ -273,29 +363,40 @@ type Enum struct { Members []EnumMember Closed bool // false = open/extensible: unknown values must be representable Flags bool // bitfield semantics + FallbackMember string // wire name of the member to substitute when an unknown value is read + // (Avro enum `default` symbol); "" = none } type EnumMember struct { Name Naming Value Value // typed value, matches ValueType - WireName string // serialized form when it differs from Value (rare) + WireName string // serialized form when it differs from Value (rare) Docs Docs Deprecation *Deprecation + Availability *Availability // members appear/disappear across versions (TypeSpec @added on EnumMember) + Examples []Example Extensions Extensions } ``` -`Closed` defaults per source semantics: JSON Schema enums are closed, Smithy and proto3 enums are -open. Backends that can't express open enums (plain Go consts, TS string literals) lower via -their refiners — the bit must survive to that point (Kiota's string-only closed enums are the -counterexample). +`Closed` defaults per source semantics: JSON Schema enums are closed; Smithy enums are open; +protobuf enums are open or closed **per file syntax / editions feature** (proto2 closed, proto3 +open, editions `enum_type` per enum) — the frontend lowers the *resolved* value. Backends that +can't express open enums (plain Go consts, TS string literals) lower via their refiners — the +bit must survive to that point (Kiota's string-only closed enums are the counterexample). +Duplicate member values are legal (protobuf `allow_alias`); slice order preserves which name is +canonical for serialization, and the validate pass must not reject them. ### 4.6 Containers and the rest ```go -type List struct { TypeCommon; Elem TypeRef; Constraints *Constraints } // minItems/maxItems/uniqueItems +type List struct { TypeCommon; Elem TypeRef; Constraints *Constraints; Encoding *Encoding } + // Constraints: minItems/maxItems/uniqueItems. + // Encoding: container-level wire encoding — protobuf packed vs expanded repeated + // fields ("packed"/"expanded"); stacks with the element's own encoding + // (repeated sint64 [packed=false] = zigzag element + expanded list) type MapT struct { TypeCommon; Key TypeRef; Value TypeRef } // Record/additionalProperties-only/proto map -type Tuple struct { TypeCommon; Elems []TypeRef } // prefixItems, TypeSpec tuples +type Tuple struct { TypeCommon; Elems []TypeRef } // prefixItems, TypeSpec tuples, Erlang tuples type Literal struct { TypeCommon; Value Value } // const / single-value enum / discriminator pins type External struct{ TypeCommon; Identity string; Package string; MinVersion string } // resolve to a well-known library type (TCGC external) type Any struct { TypeCommon } // schemaless @@ -303,18 +404,45 @@ type Any struct { TypeCommon } // sc Containers are real type nodes with IDs (hoisted like all anonymous types), not flags on references — Kiota's `CollectionKind` flag couldn't express nested collections or constrained -lists cleanly. +lists cleanly. `External` also carries opaque runtime handles that no target can structurally +model (`erlang:pid`, `erlang:fun` — see §4.8). ### 4.7 Validation-only schema constructs — a documented boundary -JSON Schema's `not`, `if`/`then`/`else`, and `dependentSchemas` express *validation logic*, not -data shape; no target language's type system represents them, and none of the other six formats -has an equivalent. The IR deliberately does not model them structurally. Frontends preserve them -verbatim in `Extensions` (`openapi:not`, `openapi:if-then-else`, …) and emit an `info` +JSON Schema's `not`, `if`/`then`/`else`, `dependentSchemas`, `contains`/`minContains`/ +`maxContains`, and `unevaluatedItems` express *validation logic*, not data shape; no target +language's type system represents them, and none of the other source formats has an equivalent. +The IR deliberately does not model them structurally. Frontends preserve them verbatim in +`Extensions` (`openapi:not`, `openapi:if-then-else`, `openapi:contains`, …) and emit an `info` diagnostic, so the information is never silently lost and a future validation-oriented backend -(request validators, mock servers) can still consume them. This is the one intentional carve-out -from "lossless means structural": losslessness is satisfied by verbatim preservation, and the -carve-out is explicit rather than accidental. +(request validators, mock servers) can still consume them. The one structural carve-back: +`unevaluatedProperties: false` is *shape* (a closed model after composition) and lowers to +`Model.Additional = closed_after_composition`; other `unevaluated*` forms stay verbatim. +`$dynamicRef` is resolved per reference site by frontend expansion (dynamic scope is static per +document); irreducible cases are preserved verbatim with a diagnostic. + +This is the one intentional carve-out from "lossless means structural": losslessness is +satisfied by verbatim preservation, and the carve-out is explicit rather than accidental. + +### 4.8 Degraded source constructs — documented lowerings + +A few source constructs have no faithful structural target in any SDK language. Each has a +*normative* degraded lowering plus verbatim preservation, so degradation is a decision, not an +accident: + +- **TypeSpec `never`-typed properties/variants** — the compiler does *not* remove them; + frontends delete them and emit an `info` diagnostic. No `never` node exists. +- **TypeSpec `StringTemplate` types** (interpolating non-literal types) — degrade to `string` + + `Extensions["typespec:string-template"]` + `info` diagnostic. +- **Erlang bit-sized binaries** (`<<_:M, _:_*N>>`) — `bytes` + `Min/MaxLength` when + byte-aligned, plus `Extensions["erlang:bit-size"] = {base, unit}` + `info` diagnostic. +- **Erlang `fun()` types** — `External{Identity: "erlang:fun"}` + full spec text in + `Extensions["erlang:fun-spec"]` + `warning` diagnostic (non-portable member). No function + type node exists. +- **Erlang heterogeneous map association lists** (`#{atom() => a(), integer() => b()}`) — + lower to union-typed `AdditionalProps` + `Extensions["erlang:map-assocs"]` verbatim. +- **Erlang `-opaque`** — lowered structurally (lossless) + `Extensions["erlang:opaque"] = true`; + backend policy decides whether to expose or wrap. --- @@ -328,26 +456,51 @@ type Property struct { Name Naming WireName string // serialized name; defaults to Name.Source WireNameByFormat map[string]string // per-mime overrides (TypeSpec @encodedName json/xml) - WireID int // protobuf field number / thrift id; 0 = none + WireID *int // protobuf field number / thrift id / tuple element index + // (1-based when Model.Positional); nil = none — pointer because + // 0 is a legal ordinal in Cap'n Proto/FlatBuffers/Avro + ExtensionOf string // "" = the model's own field; else the fully-qualified declaring + // scope of a third-party extension field (protobuf `extend`) — + // directs qualified JSON naming and registry-based accessors Type TypeRef Required bool // presence on the wire; orthogonal to Type.Nullable + Presence PresenceKind // wire-presence discipline where the format distinguishes more + // than required/optional (protobuf) + ClientOptional bool // wire-required but clients MUST treat as optional + // (Smithy @clientOptional; implicit for @input structure members) + DefaultAdded bool // default was added post-publication; generators may ignore it + // for backward compatibility (Smithy @addedDefault) Visibility Visibility // lifecycle set; zero value = visible in all Default *Value Constraints *Constraints Encoding *Encoding Args []Parameter // parameterized fields: GraphQL field arguments on any property, // at any depth — not just operation entry points. Empty elsewhere. - Flatten bool // property's fields hoisted into parent on the wire (Smithy/TCGC flatten) + Flatten bool // property's fields hoisted into parent on the wire (Smithy/TCGC + // flatten; also set on the synthetic property wrapping a hoisted + // protobuf oneof — oneof members are wire-level top-level fields) EventHeader bool // in an event-stream event model: member travels in the frame // header, not the event payload (Smithy @eventHeader) + EventPayload bool // member is the *raw* frame payload — blob/string as bytes, + // structure as protocol document (Smithy @eventPayload). + // Mutually exclusive with EventHeader; at most one per model. Secret bool // redact in logs/docs (TypeSpec @secret, format:password) XML *XMLHints // XML wire shape when it diverges from the JSON-implied shape + Examples []Example // property-level examples (TypeSpec @example, OpenAPI) Docs Docs Deprecation *Deprecation Availability *Availability Extensions Extensions Provenance Provenance } + +type PresenceKind string +// "" — format default (JSON world: Required/Nullable say everything) +// "implicit" — absence ⇔ default value; unset is unobservable; zero values are not serialized +// (proto3 no-label, editions IMPLICIT) +// "explicit" — unset is distinguishable from default-valued (hazzers/pointers; proto2 optional, +// proto3 `optional`, editions EXPLICIT) +// "required" — must be present on the wire (proto2 required, editions LEGACY_REQUIRED) ``` **Scope rule for `Args`:** parameterized properties are only legal on models reachable from a @@ -359,22 +512,27 @@ graph serializes conventionally in every backend. ```go type Lifecycle = string // OPEN set. Canonical values: "create" | "read" | "update" | "delete" | "query". - // TypeSpec's new visibility system allows arbitrary user-defined visibility + // TypeSpec's visibility system allows arbitrary user-defined visibility // classes; frontends lower custom classes as ":" strings so - // nothing is dropped. Backends treat unknown classes as opaque filters. + // nothing is dropped. Filters evaluate per class, not across the flat union; + // backends treat unknown classes as opaque filters. type Visibility struct { - Only []Lifecycle // empty = visible in all lifecycles + Only []Lifecycle // empty = visible in all lifecycles (unless None) + None bool // visible in NO lifecycle: excluded from every projection + // (TypeSpec @invisible) — distinct from the zero value } ``` `readOnly` lowers to `Only: [read]` (plus delete/query per OpenAPI semantics); `writeOnly` to `Only: [create, update]`; GraphQL input-vs-output types to `create/update` vs `read`; TypeSpec `@visibility` maps directly. One logical model therefore produces N wire shapes; the projection -(`ModelShape(model, lifecycle)`, with PATCH additionally making properties optional) is a -computed traversal in backends' plan layer — the IR stores the single logical model plus the -visibility facts, never the projected variants. This is TypeSpec's `MetadataInfo` design with -storage and computation split. +(`ModelShape(model, lifecycle)`, with PATCH additionally making properties optional *unless the +binding disables it* — §8.1 `PatchImplicitOptionality`) is a computed traversal in backends' +plan layer — the IR stores the single logical model plus the visibility facts, never the +projected variants. Operations can override which filter applies to their request/response +(§7.2 `ParameterVisibility`/`ReturnTypeVisibility`). This is TypeSpec's `MetadataInfo` design +with storage and computation split. ### 5.3 Constraints and encodings @@ -384,9 +542,12 @@ type Constraints struct { Min, Max *BigVal ExclusiveMin, ExclusiveMax bool MultipleOf *BigVal - // string + Precision, Scale *int64 // decimal digit bounds (Avro decimal, XSD totalDigits/fractionDigits, + // OData Edm.Decimal) + // string & bytes — length constraints apply to both (Avro fixed(N) = bytes, MinLength=MaxLength=N) MinLength, MaxLength *int64 Pattern string // ECMA-262 regex as written; backends translate or drop with a diagnostic + PatternMessage string // human-readable validation message (TypeSpec @pattern's second arg) // collections MinItems, MaxItems *int64 UniqueItems bool @@ -395,30 +556,41 @@ type Constraints struct { type Encoding struct { Name string // "rfc3339" | "unixTimestamp" | "base64" | "base64url" | "iso8601" | - // "seconds" | "http-date" | "varint" | "zigzag" | "fixed" | format strings + // "seconds" | "http-date" | "varint" | "zigzag" | "fixed" | + // "packed" | "expanded" | "delimited" | format strings + // (packed/expanded are container encodings on List.Encoding; "delimited" = + // protobuf group / editions DELIMITED message encoding) WireType *TypeRef // the on-wire primitive when it differs from the logical type // (utcDateTime encoded as int32; bytes as base64 string) + MediaType string // content media type of the value itself (Smithy @mediaType on string/blob, + // JSON Schema contentMediaType); "" = none } ``` The logical-type / encoding-name / wire-type triple is TCGC's reification of TypeSpec `@encode` and also absorbs OpenAPI `format` and Protobuf's `sint*/fixed*` wire variants. Encoding attaches -at the scalar definition or overrides at the property — property wins. +at the scalar definition or overrides at the property — property wins. Protobuf editions features +lower here per element after the frontend resolves the feature cascade (descriptors expose +resolved values): `field_presence` → `Presence`, `enum_type` → `Closed`, +`repeated_field_encoding` → `List.Encoding`, `message_encoding` → `"delimited"`; remaining axes +(`utf8_validation`, `json_format`) → namespaced Extensions. ### 5.4 XML hints XML-capable formats (OpenAPI's `xml` object, Smithy's `xmlName`/`xmlAttribute`/`xmlNamespace`/ -`xmlFlattened` traits) describe an XML wire shape that diverges from the JSON-implied one. This -is typed, not an extension, because two formats express it and backends must act on it: +`xmlFlattened` traits, TypeSpec's `@Xml.*` decorators) describe an XML wire shape that diverges +from the JSON-implied one. This is typed, not an extension, because multiple formats express it +and backends must act on it. Hints attach at both levels: `TypeCommon.XML` (root element +name/namespace of the shape itself) and `Property.XML` (per-use overrides; property wins). ```go type XMLHints struct { Name string // element/attribute name override Namespace string // namespace URI Prefix string - Attribute bool // serialize as attribute, not element + NodeType string // "" | "element" | "attribute" | "text" | "cdata" | "none" + // (OpenAPI 3.2 nodeType; "text" covers Smithy httpPayload text) Wrapped bool // list items wrapped in a container element - Text bool // value is the element's text content (Smithy httpPayload text) } ``` @@ -431,20 +603,34 @@ separate from the type graph (TypeSpec's Type-vs-Value split): ```go type Value struct { - Kind ValueKind // "null" | "bool" | "string" | "number" | "bytes" | "list" | "object" | "ref" + Kind ValueKind // "null" | "bool" | "string" | "number" | "bytes" | "symbol" | + // "list" | "object" | "ref" | "ctor" Bool bool - Str string + Str string // payload for "string" AND "symbol" Num BigVal // arbitrary precision, decimal string form Bytes []byte // base64 in JSON form List []Value Object []Field // ordered; Field{Name string; Value Value} Ref *ValueRef // reference to a declared constant: {Type TypeID; Member string} // (TypeSpec enum-member defaults, references to named consts) + Ctor *CtorValue // constructor-built value (TypeSpec scalar constructors) +} + +type CtorValue struct { + Scalar TypeID // the scalar whose constructor is invoked + Name string // constructor name ("fromISO", "now", custom inits) + Args []Value } ``` `BigVal` is a decimal-string wrapper: no float64 round-tripping anywhere in the IR. +`symbol` is an interned-symbol value distinct from `string` (Erlang atoms: on the native wire +`ok` ≠ `<<"ok">>`). Backends without a symbol concept render symbols as strings — explicitly, +not accidentally. `ctor` captures values built by named scalar constructors +(`utcDateTime.now()`, `plainDate.fromISO("2024-05-06")`) — inherently non-literal, so frontends +must not fold them. + --- ## 7. Service layer @@ -453,36 +639,57 @@ type Value struct { ```go type Service struct { + ID ServiceID Name Naming Docs Docs + Version string // per-service version string (Smithy service version); the + // document-level Version remains the API title's version Namespace []string // source namespace path (TypeSpec/Smithy/proto package) + Extends []ServiceID // client-visible service inheritance (Thrift service extends, + // WSDL 2.0 interface extension, Cap'n Proto interface inheritance); + // inherited operations are walked, never copied Groups []OperationGroup // hierarchical; a group ≈ TypeSpec interface / Smithy resource / tag Auth []AuthRequirement // service-level default (OR-of-ANDs, §9) CommonErrors []ErrorCase // errors every operation can return (Smithy service-level errors) + Protocols []ProtocolDecl // declared serde/protocol conventions the service speaks + // (Smithy @protocolDefinition traits like aws.protocols#restJson1) + Renames map[TypeID]Naming // per-service shape presentation names (Smithy service `rename`); + // the TypeID — and Naming on the type — are unchanged Servers []int // indexes into Document.Servers scoped to this service Extensions Extensions Provenance Provenance } +type ProtocolDecl struct { + Name string // e.g. "aws.restJson1", "grpc" + Options Extensions // per-protocol options kept raw (the Channel.Bindings pattern) +} + type OperationGroup struct { Name Naming Docs Docs Groups []OperationGroup // nesting: Smithy resources, sub-clients Operations []Operation Resource *ResourceInfo // Smithy resource semantics when declared + Availability *Availability // groups (TypeSpec interfaces) are versionable Extensions Extensions } type ResourceInfo struct { Identifiers []Property // resource identity fields Properties []Property // resource state fields (Smithy 2.0 resource properties) - Lifecycle map[string]OpID // "create"|"read"|"update"|"delete"|"list" → op + Lifecycle map[string]OpID // "create"|"put"|"read"|"update"|"delete"|"list" → op + // (put = create-or-replace with client-provided identifier) + NoReplace bool // put may create but not replace (Smithy @noReplace) + InstanceOps []OpID // declared non-lifecycle instance operations (require identifiers) + CollectionOps []OpID // declared collectionOperations — the split drives + // sub-client shape and is a declared fact, not a heuristic } ``` OpenAPI frontends build groups from tags (policy-controllable: tag-based vs path-prefix-based); TypeSpec from interfaces/namespaces; Smithy from resources; GraphQL yields three groups -(query/mutation/subscription); Protobuf one group per `service`. +(query/mutation/subscription); Protobuf one group per `service`; Erlang/OTP one group per module. ### 7.2 Operation — the protocol-neutral core @@ -499,6 +706,10 @@ type Operation struct { Responses []Response // ordered; success + alternative successes Errors []ErrorCase // declared failure shapes + OneWay bool // fire-and-forget: no response EVER exists (OTP cast, AsyncAPI + // send-without-reply, Thrift oneway, JSON-RPC notifications). + // Distinct from a response with no body (an ack still exists); + // validate rejects OneWay && len(Responses) > 0 Streaming StreamingMode // none | client | server | bidi (derived summary of the two below) RequestStream *StreamDetail // client→server streaming semantics, when present ResponseStream *StreamDetail // server→client streaming semantics, when present @@ -508,6 +719,9 @@ type Operation struct { // safe = no side effects (Smithy @readonly, HTTP GET semantics) Auth []AuthRequirement // override of service default; empty slice ≠ nil (empty = explicitly public) Tags []string + ParameterVisibility []Lifecycle // visibility filter override for the request view + // (TypeSpec @parameterVisibility); nil = protocol default + ReturnTypeVisibility []Lifecycle // filter override for the response view; nil = protocol default OverloadOf *OpID // TypeSpec @overload grouping Bindings OpBindings // §8 — how the core maps onto concrete protocols @@ -522,22 +736,36 @@ type Parameter struct { Required bool Default *Value Constraints *Constraints + ValueFrom *PropPath // the parameter's value is derived from a location in the + // outgoing/incoming message (AsyncAPI parameter `location` + // runtime expressions) — SDKs may auto-fill it; nil = caller-supplied Docs Docs Deprecation *Deprecation + Availability *Availability + Examples []Example Extensions Extensions // NOTE: no location here — path/query/header is HTTP-binding detail (§8.1) } type Payload struct { Contents []Content // one per media type / message schema — all kept + Extensions Extensions } type Content struct { MediaType string // "application/json", "multipart/form-data", "" for non-HTTP + SchemaFormat string // schema *language* the type graph was lowered from, media-type + // form (AsyncAPI multiFormatSchema: Avro/Protobuf/RAML/…); + // "" = source-native. The verbatim source schema is preserved in + // Extensions (e.g. "asyncapi:schema") for schema-registry workflows Type TypeRef + Item *TypeRef // element shape of a sequential stream declared per media type + // (OpenAPI 3.2 itemSchema for SSE/JSONL/json-seq); nil = not sequential + ItemEncoding map[string]PartEncoding // per-item encoding for sequential media types (3.2 itemEncoding) Encoding map[string]PartEncoding // multipart/form: per-property (part) wire config, keyed by PropID File *FileInfo // body is a file upload/download (TypeSpec file bodies, binary payloads) Examples []Example + Extensions Extensions } type PartEncoding struct { @@ -550,14 +778,26 @@ type PartEncoding struct { } // TypeSpec tuple-form multipart lowers to a synthesized model whose properties are the parts. -type FileInfo struct { IsText bool; FilenameParam string } +type FileInfo struct { + IsText bool // textual vs binary contents + Contents *TypeRef // declared contents scalar chain (string/bytes extensions); nil = bytes + ContentTypes []string // declared *allowed* content-type set (TypeSpec File<"image/png" | + // "image/jpeg">); runtime Content-Type comes from the file value + ContentTypeDefault string // default when the file value carries none + FilenameLocation string // "content-disposition" (default) | "path" | "header" + FilenameWireName string // wire name when FilenameLocation is path/header +} type Response struct { Name Naming // for formats with named outputs; Hint elsewhere Conditions ResponseConditions // HTTP status codes/ranges; empty for RPC single-response Payload *Payload // nil = no body Headers []Property // response metadata fields + StatusCodeProp *PropPath // output member populated from the runtime HTTP status line + // (Smithy @httpResponseCode, TypeSpec non-literal @statusCode); + // the member is suppressed from the body Docs Docs + Extensions Extensions } type ResponseConditions struct { @@ -567,8 +807,14 @@ type ResponseConditions struct { type ErrorCase struct { Type TypeRef // an error-flagged model Conditions ResponseConditions + Fault string // "" | "client" | "server" — protocol-neutral fault classification + // (Smithy @error; OpenAPI 4XX/5XX is its HTTP lowering). Drives + // exception hierarchies and default status synthesis Retryable *bool // Smithy @retryable; nil = unknown + Throttling *bool // retryable specifically due to throttling — distinct backoff class + // (Smithy @retryable(throttling: true)); nil = unknown Docs Docs + Extensions Extensions } ``` @@ -583,6 +829,10 @@ Design notes: counterexample). The plan layer picks a primary for SDK ergonomics; the IR doesn't. - **Errors reference error models** (`Model` with `UsageFlags.Error`), with status conditions and range collapsing exactly as Kiota's error mappings (`404`, `4XX`, catch-all). +- **`OneWay` is on the neutral core**, like `Streaming` and `Idempotency`, because + fire-and-forget is protocol-independent — an SDK that blocks awaiting a reply on a cast is + wrong regardless of transport. The AsyncAPI frontend sets it for send-operations without a + reply, rather than leaving one-way-ness inferable only from binding fields. ### 7.3 Pagination, long-running, streaming @@ -595,16 +845,26 @@ type Pagination struct { Items *PropPath // where result items live in the response — a PATH, not a name NextCursor *PropPath // continuation source in the response NextLink *PropPath + PrevLink *PropPath // server-driven navigation links beyond next + FirstLink *PropPath // (TypeSpec @prevLink/@firstLink/@lastLink, OpenAPI links) + LastLink *PropPath TotalCount *PropPath } -type PropPath struct{ Root TypeRef; Segments []PropID } // survives envelope nesting (TypeSpec paging lesson) +type PropPath struct { + Root *TypeRef // the type the path roots in; nil = determined by context + // (the enclosing response body, message payload, …) + In string // "" = body/payload | "header" — continuation tokens and reply addresses + // can live in response/message headers, not just bodies + Segments []PropID +} type ParamPath struct{ Param string; Segments []PropID } type LongRunning struct { FinalStateVia string // "operation-location" | "status-monitor" | "original-uri" | … - PollingOperation *OpID // declared poll op (TypeSpec @pollingOperation) - FinalOperation *OpID // declared final-result op (TypeSpec @finalOperation) + PollingOperation *OpID // declared poll op (Azure.Core @pollingOperation — a library + // convention, not core TypeSpec) + FinalOperation *OpID // declared final-result op (Azure.Core @finalOperation) PollingType *TypeRef FinalType *TypeRef ResultPath *PropPath @@ -615,16 +875,22 @@ type StreamingMode string // "none" | "client" | "server" | "bidi" type StreamDetail struct { Events *TypeRef // stream element type when it differs from the payload content type — // for event streams this is a WireTagged Union of event models; within an - // event model, Property.EventHeader marks frame-header members and the - // remaining members form the event payload (Smithy @eventHeader/@eventPayload) + // event model, Property.EventHeader marks frame-header members and + // Property.EventPayload marks a raw-payload member (Smithy + // @eventHeader/@eventPayload); per-event content types and terminal + // events live on Variant.Event Initial *TypeRef // initial-request / initial-response message preceding the stream // (Smithy event stream initial messages); nil = none + RequiresLength bool // the streamed content must have a known finite length up front + // (Smithy @requiresLength) — changes the generated parameter type } ``` Streaming mode is on the operation core because it is protocol-independent (gRPC streams, SSE, WebSocket, Smithy event streams all project onto it); the wire mechanism (SSE vs chunked vs -WS frames vs length-prefixed events) lives in the binding. +WS frames vs length-prefixed events) lives in the binding. Smithy **waiters** +(`smithy.waiters#waitable`) are *not* folded into `LongRunning` — acceptor lists and JMESPath +matchers are preserved verbatim in `Operation.Extensions` (§15). --- @@ -632,10 +898,16 @@ WS frames vs length-prefixed events) lives in the binding. ```go type OpBindings struct { - HTTP *HTTPBinding // OpenAPI/Swagger ops, TypeSpec @route, Smithy http traits - RPC *RPCBinding // Protobuf/gRPC, Smithy RPC protocols - Message *MessageBinding // AsyncAPI operations, GraphQL subscriptions, webhooks - GraphQL *GraphQLBinding // query/mutation/subscription field binding + HTTP []HTTPBinding // OpenAPI/Swagger ops, TypeSpec @route, Smithy http traits. + // A slice: one operation may carry several HTTP mappings — + // gRPC transcoding's additional_bindings — with the primary first + RPC *RPCBinding // Protobuf/gRPC, Smithy RPC protocols, JSON-RPC + Message *MessageBinding // AsyncAPI operations, webhooks + GraphQL *GraphQLBinding // query/mutation/subscription field binding. GraphQL subscriptions + // bind here + streaming fields on the core — NOT via MessageBinding + // (GraphQL defines no channel; synthesizing one is deployment-aware + // policy, marked Inferred, never a frontend default) + OTP *OTPBinding // Erlang/OTP behaviour operations (§8.5) } ``` @@ -646,24 +918,47 @@ HTTP and RPC; gRPC with HTTP transcoding). ```go type HTTPBinding struct { - Method string // GET, POST, … uppercase + Method string // as sent on the wire (OpenAPI 3.2 additionalOperations keys + // carry exact capitalization; QUERY and custom methods are legal) URITemplate string // RFC 6570 — the one true path representation (Kiota + TypeSpec agree) HostPrefix string // endpoint host prefix, may contain {param} labels (Smithy @endpoint) + SharedRoute bool // multiple operations legally share method+path, disambiguated by + // request content (TypeSpec @sharedRoute) — validate must not + // reject the duplicate, and single-route backends must merge ParamBindings []HTTPParamBinding RequestContentTypes []string // priority-ordered + ResponseBodyPath *PropPath // HTTP response body = this sub-field of the response type + // (gRPC transcoding response_body); nil = the whole payload SuccessStatus map[int]int // response index → primary status (denormalized convenience; conditions are the truth) + Compression *RequestCompression // client MUST compress the request body (Smithy @requestCompression) + ChecksumRequired bool // client MUST send a payload checksum (Smithy @httpChecksumRequired) + PatchImplicitOptionality *bool // nil = protocol default (PATCH projections make properties + // optional); false = disabled (TypeSpec @patch implicitOptionality) IsWebhook bool // OpenAPI 3.1 webhooks: direction is inbound Callbacks []Callback // out-of-band operations keyed by runtime expressions + Extensions Extensions } +type RequestCompression struct { Encodings []string } // priority-ordered ("gzip", …) + type HTTPParamBinding struct { Param string // Operation.Params name it binds - Location HTTPLocation // path | query | header | cookie | body | body_property | host + ParamPath []PropID // nested source field within the logical param, when the binding + // targets a sub-field of a message-typed param (gRPC transcoding + // {book.name}, dotted query params); empty = the whole param + Location HTTPLocation // path | query | querystring | header | cookie | body | + // body_property | host // host = param fills a HostPrefix label (Smithy @hostLabel) + // querystring = the whole query string serialized from one schema + // (OpenAPI 3.2 in: querystring, combined with ContentType) WireName string Style string // simple | form | label | matrix | deepObject | pipe/space-delimited Explode *bool AllowReserved bool + PathPattern string // multi-segment path pattern constraint for this param + // (gRPC transcoding {name=shelves/*/books/*}); "" = single segment. + // The URI template uses reserved expansion; backends that cannot + // validate the pattern drop it with a diagnostic Prefix string // map-typed param spread as prefixed wire entries: // prefixed headers (Smithy @httpPrefixHeaders) or catch-all // query maps (@httpQueryParams with Prefix "") @@ -674,18 +969,24 @@ type HTTPParamBinding struct { type Callback struct { Expression string; Operations []OpID } ``` -Every logical parameter is bound exactly once (validated). TypeSpec's `HttpProperty` role+path -flattening is the model here: nested `@header`/`@body` annotations resolve to explicit -`(role, wire name, path)` bindings rather than restructured models. +Every logical parameter is bound exactly once **per non-host location; a `host` binding is +additive** — Smithy `@hostLabel` members expand into the host prefix *and* still serialize at +their modeled location, by spec. TypeSpec's `HttpProperty` role+path flattening is the model +here: nested `@header`/`@body` annotations resolve to explicit `(role, wire name, path)` +bindings rather than restructured models. ### 8.2 RPC ```go type RPCBinding struct { - System string // "grpc" | "smithy-rpc" | "connect" | … + System string // "grpc" | "smithy-rpc" | "connect" | "jsonrpc" | … FullMethod string // "/pkg.Service/Method" InputType *TypeRef // the request message type params fold into (nil = synthesize from Params) + ParamStructure string // "" | "by_name" | "by_position" | "either" — how params serialize + // (JSON-RPC positional vs named; OpenRPC paramStructure). Param order + // is already source order per design rule 5; this is the mode IdempotencyLevel string + Extensions Extensions } ``` @@ -695,16 +996,35 @@ type RPCBinding struct { type MessageBinding struct { Channel ChannelID Direction MsgDirection // send | receive (application perspective, AsyncAPI 3 semantics) - Messages []MessageRef // which of the channel's messages this operation uses - ReplyTo *ChannelID + Messages []MessageID // which of the channel's messages this operation uses + // (must be a subset — validated) + Reply *Reply // request-reply semantics; nil = none. A send-op with no Reply and + // no Responses is one-way (set Operation.OneWay) + Bindings map[string]Extensions // operation-level protocol bindings kept raw + // (kafka groupId/clientId — constrain SDK client config) + Extensions Extensions +} + +type Reply struct { + Channel *ChannelID // static reply channel; nil when the address is dynamic-only. + // (An AsyncAPI reply channel's own address is null by spec) + Address *PropPath // dynamic reply address: where in the *request* message the reply + // destination lives, e.g. In:"header", Segments:[replyTo] + // (AsyncAPI Operation Reply Address runtime expressions) + Messages []MessageID // reply payload message set + Docs Docs } type Channel struct { ID ChannelID Name Naming - Address string // topic/routing key/path, may contain {params} + Address *string // topic/routing key/path, may contain {params}. + // nil = unknown/runtime-assigned address (load-bearing: reply + // channels and dynamic topics; SDKs expose a runtime address arg) + Docs Docs + Tags []string Params []Parameter - Messages []Message + Messages []MessageID // the channel's message set — messages live in Document.Messages Servers []int Bindings map[string]Extensions // protocol-specific ("kafka", "amqp", "ws", "mqtt") kept as namespaced raw config Extensions Extensions @@ -712,19 +1032,29 @@ type Channel struct { } type Message struct { + ID MessageID Name Naming Payload Payload - Headers []Property - CorrelationID *PropPath + Headers *TypeRef // header *schema* — an object-constrained model hoisted into the + // type registry like any anonymous type (headers can be named, + // composed, even Avro-defined; and $message.header#/… paths need + // a type to resolve against). Backends compute flat lists per §4.3 + CorrelationID *PropPath // In: "header" | "" (payload) ContentType string + Tags []string Docs Docs + Deprecation *Deprecation + Examples []Example // correlated header+payload example pairs (Example.Headers + .Value) + Bindings map[string]Extensions // message-level protocol bindings (kafka message key, …) Extensions Extensions + Provenance Provenance } ``` Kafka/AMQP/MQTT binding minutiae stay as namespaced raw extension config rather than modeled structs — they are protocol deployment detail, not API shape, and modeling them structurally -would chase every AsyncAPI binding spec revision. +would chase every AsyncAPI binding spec revision. The raw-bindings channel exists at every +level that AsyncAPI defines bindings: server, channel, operation, and message. ### 8.4 GraphQL @@ -732,16 +1062,58 @@ would chase every AsyncAPI binding spec revision. type GraphQLBinding struct { Kind string // "query" | "mutation" | "subscription" FieldPath []string // entry-point field (nesting for namespaced schemas) + Extensions Extensions } ``` GraphQL lowering: schema types → models (`InputOnly` for inputs), interfaces → `Abstract` -models with implementors listing them in `Implements`, unions → `WireTagged` unions -discriminated by `__typename`, entry-point fields → operations (field args → `Params`, field -type → response), nested field arguments → `Property.Args` at any depth. Arbitrary -client-composed selection sets are out of scope by design: Morphic generates SDK surface, and -the full type graph — including per-field arguments — is retained so a backend can still offer -query builders. +models with implementors listing them in `Implements` (including interfaces implementing +interfaces), unions → `WireTagged` unions discriminated by `__typename`, **`@oneOf` input +objects → `Union{Exclusive: true, WireTagged: true}` with a variant per field** (never a model +of optional properties — that is the union-to-optional-fields collapse rule 2 forbids), custom +scalars → `Scalar` with `Base = nil`, entry-point fields → operations (field args → `Params`, +field type → response), nested field arguments → `Property.Args` at any depth, subscriptions → +`GraphQLBinding{Kind: "subscription"}` + `StreamingMode: server` + `ResponseStream`. + +Directive conventions (normative): applications are ordered and repeatable, so +`Extensions["graphql:@"]` holds an **ordered JSON array** of application argument objects +— a singleton array for a single application. Directive *definitions* are preserved verbatim at +document level under `Extensions["graphql:directive-definitions"]`. Type-extension assembly +(`extend type` across files): the node's `Provenance` points at the original definition, each +member's `Provenance` at its defining occurrence; the occurrence list goes to +`Extensions["graphql:extends"]` when SDL round-trip fidelity is wanted. + +Arbitrary client-composed selection sets are out of scope by design: Morphic generates SDK +surface, and the full type graph — including per-field arguments — is retained so a backend can +still offer query builders. + +### 8.5 Erlang/OTP + +```go +type OTPBinding struct { + Behaviour string // "gen_server" | "gen_statem" | "gen_event" + Kind string // "call" (synchronous request→reply) | "cast" (fire-and-forget; the + // operation also sets OneWay) | "info" (raw message send) + Process ChannelID // the channel modeling the target process: Address = registered name + // (nil Address = unregistered/runtime pid); registration kind + // (local/global/via) in Channel.Bindings["otp"] + RequestTag *Value // tag of the request tuple (a symbol Value, e.g. 'get'); + // nil = the whole term is the request +} +``` + +An OTP frontend consumes module type information (`-spec`/`-type` on behaviour callbacks and +API functions). Lowering conventions: a gen_server ≈ one `Channel`; `handle_call`/`handle_cast` +APIs are `Operation`s with an `OTPBinding`; unsolicited `handle_info` messages are channel +`Messages` consumed via `MessageBinding{Direction: receive}`; a gen_event manager is a Channel +with one message shape per event. Tagged-tuple protocols (`{ok, V} | {error, R}`) are Unions of +Tuples discriminated by `Discriminator.Index`; records are `Model{Positional: true}` with the +record tag as an index-1 `Literal`-typed property. Splitting a tagged reply union into +`Responses` vs `Errors` (`{error, Reason}` is in-band data, not transport failure) is injectable +frontend policy, marked `Inferred`. Erlang `string()` is `[char()]` and must lower as a List of +a char-ranged scalar — never as PrimKind `string`. Delayed replies (`noreply` + +`gen_server:reply/2`) are invisible at the protocol surface and need no representation; call +timeouts and `multi_call`/`send_request` machinery are SDK runtime policy, not IR. --- @@ -751,36 +1123,63 @@ query builders. type AuthScheme struct { ID AuthID Name Naming - Kind AuthKind // apiKey | http_basic | http_bearer | oauth2 | openid_connect | mutual_tls | custom + Kind AuthKind + // AuthKind: apiKey | http_basic | http_bearer | oauth2 | openid_connect | mutual_tls | + // user_password | x509 | symmetric_encryption | asymmetric_encryption | + // sasl_plain | sasl_scram_sha256 | sasl_scram_sha512 | sasl_gssapi | custom + // The SASL family, user_password, and x509 are how Kafka/AMQP clients authenticate + // (AsyncAPI securitySchemes). AsyncAPI `httpApiKey` lowers to apiKey; AsyncAPI `apiKey` + // (transport user/password slot) lowers to apiKey with In: user|password. X509 is distinct + // from mutual_tls (certificate as credential vs mutual verification) — frontends must not + // conflate them. + Docs Docs + Deprecation *Deprecation // apiKey - In string // header | query | cookie + In string // header | query | cookie | user | password KeyName string // http - Scheme string // bearer, basic, digest… + Scheme string // bearer, basic, digest…; also legal for apiKey (Smithy @httpApiKeyAuth scheme) BearerFormat string // oauth2 Flows []OAuthFlow // {Kind: authorization_code|client_credentials|implicit|password|device, - // AuthorizationURL, TokenURL, RefreshURL, Scopes map[string]string} + // AuthorizationURL, TokenURL, RefreshURL, Scopes map[string]string, + // Extensions} — device flow's deviceAuthorizationUrl rides AuthorizationURL + OAuth2MetadataURL string // RFC 8414 authorization-server metadata (OpenAPI 3.2) // openid OpenIDConnectURL string Extensions Extensions + Provenance Provenance } type AuthRequirement struct { // one *option* Schemes []SchemeUse // ALL must be satisfied together } type SchemeUse struct { Scheme AuthID; Scopes []string } -// []AuthRequirement on a service/operation = OR across options (TypeSpec/OpenAPI security semantics) +// []AuthRequirement on a service/operation/server = OR across options (TypeSpec/OpenAPI security +// semantics). Option order is PRIORITY order — clients pick the first supported option (Smithy +// @auth is priority-ordered; the Smithy frontend materializes the alphabetical default when +// @auth is absent). Frontends whose source has no ordering semantics emit source order. +// An empty option (AuthRequirement{Schemes: []}) inside a non-empty list means "no auth is one +// acceptable choice" (TypeSpec NoAuth in a union, Smithy @optionalAuth, OpenAPI's empty security +// requirement) — distinct from Operation.Auth = [] (explicitly public). ``` ## 10. Servers ```go type Server struct { + Name Naming // servers are named entities (AsyncAPI name-keyed servers, + // OpenAPI 3.2 Server.name) URLTemplate string // may contain {variables} Description Docs - Variables []ServerVariable // {Name, Default, Enum []string, Docs} + Variables []ServerVariable // {Name, Default, Enum []string, Docs, Extensions} Protocol string // "https" default; "kafka", "wss", … for messaging servers + ProtocolVersion string // e.g. Kafka "3.5", AMQP "0-9-1" (AsyncAPI) + Tags []string + Auth []AuthRequirement // server-scoped security — AsyncAPI's primary auth placement + // (broker connections authenticate per server; different + // servers of one service may require different schemes) + Bindings map[string]Extensions // server-level protocol bindings kept raw Extensions Extensions } ``` @@ -789,19 +1188,29 @@ type Server struct { ```go type Availability struct { - Added string // version label from Document.Versions - Removed string + Added []string // version labels from Document.Versions, ordered — multiple entries + Removed []string // support add/remove/re-add cycles (v1 present, v2 removed, v3 re-added) Deprecated string RenamedFrom []VersionedName // {Version, Name} TypeChangedFrom []VersionedType // {Version, Type TypeRef} — property/return type changed // across versions (TypeSpec @typeChangedFrom) + RequiredChanged []VersionedBool // {Version, WasRequired} — optionality flipped at a version + // (TypeSpec @madeOptional/@madeRequired); the slice pass + // reconstructs Required for older snapshots from it } +type VersionedBool struct { Version string; WasRequired bool } ``` The IR stores the **timeline** (TypeSpec model); the `version-slice` pass produces a concrete snapshot document per version for consumption. Backends always receive snapshots — they never interpret availability themselves. Formats without versioning semantics simply leave this nil. +`Availability` attaches everywhere versioning decorators can: `TypeCommon`, `Property`, +`Operation`, `Parameter`, `EnumMember`, `Variant`, and `OperationGroup` — a versioned enum +member or union variant must disappear from pre-`Added` snapshots. Smithy `@since` (free-form, +no version registry) lowers to `Added` only when the frontend synthesizes labels; otherwise it +is preserved as an extension. + ## 12. Docs, deprecation, examples, extensions ```go @@ -814,17 +1223,37 @@ type Docs struct { type Deprecation struct { Message, Since, RemovalVersion string } -type Example struct { Name string; Summary string; Value *Value; ExternalURL string } +type Example struct { + Name string + Summary string + Description string + Value *Value // single-value examples (schemas, properties, parameters); + // for message examples: the payload + Headers *Value // message examples: the correlated header values (AsyncAPI message + // examples are header+payload PAIRS — never split them) + Input *Value // operation scenarios: paired input ↔ output/error + Output *Value // (Smithy @examples, TypeSpec @opExample parameters/returnType) + Error *ErrorExample // the scenario ends in this error instead of Output + ExternalURL string + Extensions Extensions +} +type ErrorExample struct { Type TypeRef; Content Value } +// Field legality is contextual (validated): Value/Headers on types, properties, parameters, +// contents, and messages; Input/Output/Error on operations. An Example never mixes the two arms. type Extensions map[string]RawValue // keys are namespaced by origin: "openapi:x-rate-limit", "smithy:aws.api#arn", -// "graphql:@key", "typespec:@myOrg/decorator", "asyncapi:bindings.kafka" +// "graphql:@key", "typespec:@myOrg/decorator", "asyncapi:bindings.kafka", "erlang:opaque" // RawValue is the source JSON preserved verbatim. ``` Extensions are the lossless escape hatch: any source metadata without a first-class IR node survives, namespaced so two formats' extensions never collide, and typed passes can promote -well-known extensions (`x-ms-pagination` → `Pagination`) without losing the original. +well-known extensions (`x-ms-pagination` → `Pagination`) without losing the original. For the +escape hatch to hold, **every node that can carry source metadata has an `Extensions` field** — +including `Response`, `ErrorCase`, `Payload`, `Content`, `Example`, `ServerVariable`, +`OAuthFlow`, and every binding struct (new spec revisions land fields on exactly these objects). +Per-file metadata (protobuf file options) is keyed by path in `Document.Extensions`. ## 13. Provenance & diagnostics @@ -853,26 +1282,37 @@ How each format's distinctive concepts land in the IR (full details live with ea | Format | Lowering highlights | |---|---| -| **OpenAPI 3.x** | components/schemas → registry (IDs from pointers); inline schemas hoisted with hints; `allOf` → Base/Mixins per §4.3; `oneOf`/`anyOf` → Union (Exclusive bit), null-variant → Nullable ref; `discriminator` → Discriminator; `nullable`/type-arrays → Nullable; readOnly/writeOnly → Visibility; parameters → Params + HTTPBinding locations w/ style/explode; requestBody/responses all content types → Payload.Contents; per-status responses/default → Conditions + ranges; webhooks → MessageBinding or HTTPBinding.IsWebhook; callbacks → Callbacks; links → extensions (promotable later); securitySchemes/security → Auth OR-of-ANDs; servers+variables → Servers; `xml` object → XMLHints; `not`/`if-then-else`/`dependentSchemas` → verbatim Extensions per §4.7; `patternProperties` → AdditionalProps.Patterns; `x-*` → namespaced Extensions; pagination only via injectable policy, marked Inferred | +| **OpenAPI 3.x** | components/schemas → registry (IDs from pointers); inline schemas hoisted with hints; `allOf` → Base/Mixins per §4.3; `oneOf`/`anyOf` → Union (Exclusive bit), null-variant → Nullable ref; `discriminator` → Discriminator (3.2 `defaultMapping` → Discriminator.Default); `nullable`/type-arrays → Nullable; readOnly/writeOnly → Visibility (schema-level readOnly pushed down to referencing properties, residue → Extensions + diagnostic); `additionalProperties: false` → Additional=closed, `unevaluatedProperties: false` → closed_after_composition; parameters → Params + HTTPBinding locations w/ style/explode, 3.2 `in: querystring` → querystring location; requestBody/responses all content types → Payload.Contents; 3.2 `itemSchema`/`itemEncoding` → Content.Item/ItemEncoding; per-status responses/default → Conditions + ranges; webhooks → HTTPBinding.IsWebhook; callbacks → Callbacks; links → extensions (promotable later); securitySchemes/security → Auth OR-of-ANDs, 3.2 device flow + `oauth2MetadataUrl` → Flows/OAuth2MetadataURL; servers+variables (3.2 named) → Servers; tags (3.2 parent/kind) → groups + TagDefs; info contact/license → Document; schema `example(s)` → Examples; `xml` object (incl. 3.2 nodeType) → XMLHints at type and property level; `not`/`if-then-else`/`dependentSchemas`/`contains`/`unevaluated*` → verbatim Extensions per §4.7; `patternProperties` → AdditionalProps.Patterns; `x-*` → namespaced Extensions (legal on every object — hence Extensions on every node); pagination only via injectable policy, marked Inferred | | **Swagger 2.0** | lifted to OpenAPI 3.x shape first (body/formData → Payload; host/basePath/schemes → Servers; consumes/produces → content types), then the OpenAPI lowering runs | -| **TypeSpec** | consumed post-check (monomorphized, `isFinished`); template instances → TypeCommon.Instantiation; models → Model w/ Base + spread provenance → Mixins; scalars → Scalar chains; `@encode` → Encoding triple; unions w/ named variants → Union, `@discriminated` envelope → Discriminator.Envelope; `| null` → Nullable; visibility enums (incl. custom classes) → Visibility; interfaces → OperationGroups; `@overload` → OverloadOf; `@service` → Service; versioning decorators incl. `@typeChangedFrom` → Availability timeline; `@list`/`@pageItems` paths → Pagination PropPaths; `@pollingOperation`/`@finalOperation` → LongRunning; multipart w/ parts → Content.Encoding/PartEncoding, file bodies → FileInfo; `@error` → UsageFlags.Error; values/consts incl. enum-member refs → Values channel | -| **Smithy 2.0** | structures → Model, mixins → Mixins; `document` → Any; unions → WireTagged Union; enum/intEnum → Enum (open by default); `@sparse` → element Nullable; traits: constraints → Constraints, `@paginated` → Pagination (declared), `@retryable` → ErrorCase.Retryable, `@readonly` → Idempotency safe, `@idempotent`/`@idempotencyToken` → Idempotency, `@sensitive` → Sensitive/Secret, `@tags` → Tags; `@streaming` blob → StreamDetail; event streams → StreamDetail.Events union + Property.EventHeader + Initial messages; service-level errors → Service.CommonErrors; resources → OperationGroup + ResourceInfo (identifiers, properties, lifecycle map); http traits → HTTPBinding incl. `@endpoint`/`@hostLabel` → HostPrefix/host location, `@httpPrefixHeaders`/`@httpQueryParams` → Prefix bindings; `@jsonName` → WireName; xml traits → XMLHints; other traits → namespaced Extensions | -| **GraphQL** | §8.4; interfaces → Abstract models + Implements; field arguments → Property.Args; input objects → `InputOnly` models; non-null wrapping → Required/Nullable; deprecation w/ reason; custom scalars → Scalar; directives → Extensions | -| **AsyncAPI** | servers w/ protocols → Servers; channels/messages → Channels/Messages; operations send/receive → Operation + MessageBinding; correlation IDs → PropPath; protocol bindings → Channel.Bindings raw; message payload schemas share the same JSON-Schema lowering as OpenAPI | -| **Protobuf** | messages → Model w/ WireIDs; oneof → WireTagged Exclusive Union w/ variant WireIDs; enums → open Enum w/ int32 values; map/repeated → MapT/List; scalar wire variants (sint/fixed) → Encoding; services/rpcs → Service/Operation + RPCBinding; streaming modifiers → StreamingMode; options → Extensions; reserved ranges → Extensions (guarded by validate pass) | +| **TypeSpec** | consumed post-check (monomorphized, `isFinished`); template instances → TypeCommon.Instantiation incl. value args → TemplateArg; models → Model w/ Base + spread provenance → Mixins; scalars → Scalar chains, constructors in values → Value.Ctor; `@encode`/`@format` → Encoding triple; `@encodedName` → WireNameByFormat at property AND type level; unions w/ named variants → Union, `@discriminated` → Discriminator.PropertyName/Envelope/EnvelopeValueName; `| null` → Nullable; visibility classes (incl. custom, `@invisible` → Visibility.None) → Visibility, op overrides → ParameterVisibility/ReturnTypeVisibility; `@patch` implicitOptionality → HTTPBinding.PatchImplicitOptionality; interfaces → OperationGroups (versionable); `@overload` → OverloadOf; `@sharedRoute` → SharedRoute; `@service` → Service; versioning decorators incl. `@typeChangedFrom`/`@madeOptional`/`@madeRequired` and add/remove cycles → Availability timeline (on members/variants/params too); pagination decorators incl. prev/first/last links and header continuation tokens → Pagination PropPaths (In:"header"); Azure.Core `@pollingOperation`/`@finalOperation` → LongRunning; multipart w/ parts → Content.Encoding/PartEncoding, `Http.File` → FileInfo (content-type set, contents chain, filename location); streams/SSE → StreamDetail + Variant.Event (contentType, terminal); `@error` → UsageFlags.Error; `@example`/`@opExample` → Examples (Input/Output pairs); `@pattern` message → Constraints.PatternMessage; `@mediaTypeHint` → TypeCommon.MediaTypeHint; `never` members deleted + diagnostic per §4.8; TCGC client-shaping decorators (`@clientName`, `@access`, `@usage`, `@scope`, `@override`, …) → namespaced Extensions consumed by backend policy, never IR semantics; values/consts incl. enum-member refs → Values channel | +| **Smithy 2.0** | structures → Model, mixins → Mixins (non-structure mixins flattened — spec-sanctioned); `document` → Any; unions → WireTagged Union, member `@jsonName` → Variant.WireName; enum/intEnum → Enum (open by default); `@sparse` → element Nullable; traits: constraints → Constraints, `@paginated` → Pagination (declared), `@retryable` → ErrorCase.Retryable + Throttling, `@error` fault → ErrorCase.Fault, `@readonly` → Idempotency safe, `@idempotent`/`@idempotencyToken` → Idempotency, `@sensitive` → Sensitive/Secret, `@tags` → Tags, `@clientOptional`/`@input` → Property.ClientOptional (+InputOnly), `@addedDefault` → DefaultAdded, root-shape `@default` pushed down to properties w/ provenance; `@streaming` blob → StreamDetail (+`@requiresLength` → RequiresLength); event streams → StreamDetail.Events union + Property.EventHeader/EventPayload + Initial messages; service-level errors → Service.CommonErrors; protocol traits → Service.Protocols; service `rename`/`version` → Service.Renames/Version; resources → OperationGroup + ResourceInfo (identifiers, properties, lifecycle incl. put/@noReplace, instance vs collection ops); http traits → HTTPBinding incl. `@endpoint`/`@hostLabel` → HostPrefix/host location (additive binding), `@httpPrefixHeaders`/`@httpQueryParams` → Prefix bindings, `@httpResponseCode` → Response.StatusCodeProp, `@requestCompression` → Compression, `@httpChecksumRequired` → ChecksumRequired; `@auth` order → priority-ordered Auth, `@optionalAuth` → empty option; `@jsonName` → WireName; `@mediaType` → Encoding.MediaType; xml traits → XMLHints at type and property level; `@examples` → Examples (Input/Output/Error); waiters + rules-engine traits → verbatim Extensions (§15); `smithy.api#Unit` → nil payload / shared empty Model for tag-only variants; other traits → namespaced Extensions | +| **GraphQL** | §8.4; interfaces (incl. interface hierarchies) → Abstract models + Implements; field arguments → Property.Args; input objects → `InputOnly` models, `@oneOf` inputs → WireTagged Exclusive Union; non-null wrapping → Required/Nullable (all `[T!]!` combinations via per-layer list nodes); defaults on args AND input fields → Default (list-input coercion normalized); custom scalars → Scalar{Base: nil}, `@specifiedBy` → Extensions; deprecation w/ reason at every location incl. args/input fields; directives → ordered-array Extensions convention + document-level definitions | +| **AsyncAPI** | servers w/ protocols/protocolVersion/security → Servers (named, w/ Auth); channels → Channels (Address nil = unknown), messages → Document.Messages registry referenced by ID; operations send/receive → Operation + MessageBinding, no-reply sends → OneWay; reply objects → Reply (static channel + dynamic PropPath address + message set); correlation IDs → PropPath (In: header|payload); message headers → hoisted header model (Headers *TypeRef); message examples → Example{Headers, Value} pairs; multi-format payload schemas → Content.SchemaFormat + verbatim schema in Extensions (Avro payloads lower through the type graph: record→Model, enum→closed Enum w/ FallbackMember, fixed→bytes w/ length, decimal→Constraints.Precision/Scale, aliases→Naming.Aliases); parameter `location` → Parameter.ValueFrom; traits applied by merge w/ provenance; protocol bindings at server/channel/operation/message level → raw Bindings maps; payload JSON-Schema lowering shared with OpenAPI | +| **Protobuf** | messages → Model w/ WireIDs; oneof → WireTagged Exclusive Union w/ variant WireIDs + Flatten on the synthetic wrapper property (oneof members are top-level wire fields; synthetic oneofs from proto3 `optional` do NOT lower to unions — they are presence markers); presence disciplines → Property.Presence (never Nullable); enums → Enum open/closed per syntax/edition, allow_alias → duplicate member values; map/repeated → MapT/List (packed/expanded → List.Encoding); groups / editions DELIMITED → nested Model + Encoding "delimited"; scalar wire variants (sint/fixed) → Encoding; `extend` fields → Property.ExtensionOf, extension ranges → Model.ExtensionRanges; well-known types → External (wrapper types → External or nullable primitive per injectable policy); custom options → namespaced Extensions (proto-JSON), file options keyed by path in Document.Extensions; services/rpcs → Service/Operation + RPCBinding; streaming modifiers → StreamingMode; gRPC transcoding (google.api.http) → additional HTTPBinding entries (slice), path patterns → PathPattern, nested bindings → ParamPath, response_body → ResponseBodyPath; reserved ranges → Extensions (guarded by validate pass) | +| **Erlang/OTP** | §8.5; `-type`/`-spec` type language → type graph: tuples → Tuple, tagged-tuple unions → Union + Discriminator.Index, records → Model{Positional} w/ 1-based WireIDs, atoms → symbol Values / Literal types / `Scalar{Base: string, Encoding: "erlang:atom"}`, integer ranges → integer Scalar + Min/Max, maps `:=`/`=>` → Properties(Required)/AdditionalProps w/ typed keys, `pid()`/`port()`/`reference()`/`fun()` → External, `string()` → List of char scalar, parametrized types monomorphized w/ Instantiation; behaviours: gen_server call/cast → Operations + OTPBinding (cast → OneWay), info → channel Messages received, gen_event → Channel w/ N messages, gen_statem states → `Extensions["otp:states"]`; registered process → Channel.Address, registration kind → Channel.Bindings["otp"]; reply-union Errors split → injectable Inferred policy; bit-sized binaries/map assoc lists/`-opaque` → §4.8 degraded lowerings; `-deprecated` → Deprecation; `-doc`/EDoc → Docs | ## 15. Deliberate exclusions - **Language names/casings** — backends own identifier rendering entirely (IR stores neutral word sequences + wire names). - **SDK runtime policy** (retry/timeout/telemetry/error-class taxonomy) — a separate backend - input, never IR (§2.4 of architecture.md). + input, never IR (§2.4 of architecture.md). OTP call timeouts and `multi_call`/`send_request` + machinery fall here too. - **Generator plan artifacts** (request builders, executor pairs, per-visibility model variants, primary-response selection) — computed views in backends, not stored. - **Structured modeling of transport-deployment minutiae** (Kafka partition configs, AMQP exchange args) — preserved as raw namespaced extensions. +- **Smithy rules-engine traits** (`@endpointRuleSet`, context params) — runtime endpoint + resolution machinery; preserved as raw namespaced extensions on the service. +- **Smithy waiters** (`smithy.waiters#waitable`) — preserved verbatim in `Operation.Extensions`, + never folded into `LongRunning`; promoted to a typed node only if a second format lands. +- **TCGC-style client-shaping decorators** (`@clientName`, `@access`, `@usage`, `@scope`, …) — + per-language SDK-surface policy, preserved as namespaced extensions for backend policy layers. - **Arbitrary GraphQL persisted queries** — the type graph and entry points are retained; query composition is a backend/runtime feature. +- **Capability/interface-typed fields** (Cap'n Proto capability passing) — requires + services-as-types; out of scope for a data-SDK compiler. +- **Function types** — Erlang funs degrade per §4.8; no target language marshals closures. ## 16. Open questions (tracked for implementation) @@ -891,6 +1331,12 @@ How each format's distinctive concepts land in the IR (full details live with ea as the structural identity and add a derived, name-based correlation key computed by a pass — decide once IR diffing between spec revisions is actually built. 6. **Response discrimination placement** — `ResponseConditions` (status ranges) is HTTP-shaped - living on the neutral core. Alternatives: move conditions into `HTTPBinding` alongside - parameter locations, leaving responses as an ordered named list. Revisit when the RPC or - messaging frontend lands and either validates or strains the current shape. + living on the neutral core. Fresh evidence that it sits awkwardly: RPC single-response and OTP + call replies both use empty conditions, and `Response.StatusCodeProp` is likewise + HTTP-flavored. Alternatives: move conditions into `HTTPBinding` alongside parameter locations, + leaving responses as an ordered named list. Revisit when the RPC or messaging frontend lands + and either validates or strains the current shape. +7. **Semantic nullability (GraphQL)** — the `@semanticNonNull` RFC ("null only on error") is a + third nullability state `TypeRef.Nullable` cannot express. Pre-spec today: preserved as + `Extensions["graphql:@semanticNonNull"]`. If it merges into the spec, add a `NullKind` (or a + `NullOnlyOnError` bool) rather than churning every frontend early. diff --git a/docs/ir-spec-matrix.md b/docs/ir-spec-matrix.md index 6ef362b..6a501e4 100644 --- a/docs/ir-spec-matrix.md +++ b/docs/ir-spec-matrix.md @@ -3,59 +3,68 @@ What each source specification format can express, and what the Morphic IR must therefore be able to represent without loss. The IR is designed against the **union** of these capabilities, not the intersection — a generator target may ignore a capability, but the IR must never drop one. +Erlang/OTP is included as a frontend target: its "spec" is module type information +(`-spec`/`-type` on behaviour callbacks) plus the gen_server/gen_statem/gen_event message +protocols. Legend: ✅ native concept · ⚠ expressible indirectly · — absent -| Capability | OpenAPI 3.x | Swagger 2.0 | TypeSpec | Smithy 2.0 | GraphQL | AsyncAPI | Protobuf | -|---|---|---|---|---|---|---|---| -| Named object types | ✅ components.schemas | ✅ definitions | ✅ model | ✅ structure | ✅ type/input | ✅ schemas | ✅ message | -| Inline/anonymous types | ✅ | ✅ | ✅ | — (all named) | ⚠ | ✅ | ⚠ nested | -| Inheritance / base types | ⚠ allOf | ⚠ allOf | ✅ extends | ⚠ mixins | ✅ interfaces | ⚠ allOf | — | -| Mixins / spread | — | — | ✅ spread | ✅ mixins | — | ⚠ traits | — | -| Tagged unions | ⚠ oneOf+discriminator | — | ✅ discriminated union | ✅ union | ⚠ union+__typename | ⚠ oneOf | ✅ oneof | -| Untagged unions | ✅ oneOf/anyOf | — | ✅ union | — | — | ✅ oneOf | — | -| Intersection | ✅ allOf | ✅ allOf | ⚠ & (model is) | — | — | ✅ allOf | — | -| Negation | ✅ not (3.1) | — | — | — | — | ✅ not | — | -| Enums (string) | ✅ | ✅ | ✅ named members | ✅ enum | ✅ | ✅ | ⚠ | -| Enums (numeric, valued) | ✅ | ✅ | ✅ | ✅ intEnum | — | ✅ | ✅ | -| Open enums (unknown values allowed) | ⚠ anyOf trick | — | ⚠ union w/ string | ✅ (enums are open by default) | — | ⚠ | ✅ (proto3 semantics) | -| Custom scalars | ⚠ type+format | ⚠ | ✅ scalar extends | ⚠ traits | ✅ scalar | ⚠ | — | -| Wire encoding hints (@encode / format) | ✅ format | ✅ format | ✅ @encode | ✅ timestampFormat | — | ✅ | ✅ fixed/zigzag | -| Field wire IDs (numeric tags) | — | — | — | — | — | — | ✅ field numbers | -| Wire name ≠ model name | ✅ (property key) | ✅ | ✅ @encodedName | ✅ jsonName | — | ✅ | ✅ json_name | -| Optionality vs nullability distinct | ✅ (3.1) | ⚠ | ✅ | ✅ | ✅ | ✅ | ⚠ presence | -| Defaults | ✅ | ✅ | ✅ | ✅ | ✅ args | ✅ | ✅ proto2 | -| Constraints (min/max/pattern…) | ✅ | ✅ | ✅ decorators | ✅ traits | ⚠ directives | ✅ | ⚠ protovalidate | -| readOnly/writeOnly / visibility | ✅ | ✅ readOnly | ✅ @visibility lifecycle | — | ✅ input vs output types | — | — | -| Recursive types | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | -| Maps / additionalProperties | ✅ | ✅ | ✅ Record | ✅ map | — | ✅ | ✅ map | -| Tuples | ✅ prefixItems (3.1) | — | ✅ | — | — | ✅ | — | -| Literal types | ✅ const | ⚠ single enum | ✅ | — | — | ✅ | — | -| Operations grouped by service/interface | ⚠ tags | ⚠ tags | ✅ interface/namespace | ✅ service/resource | ✅ Query/Mutation/Subscription | ⚠ | ✅ service | -| Resource hierarchy (CRUDL) | — | — | ⚠ @autoRoute | ✅ resource | — | — | — | -| HTTP binding (method/path/status) | ✅ | ✅ | ✅ @route/@get… | ✅ http traits | — | ⚠ ws binding | ⚠ transcoding | -| Param styles (explode, matrix…) | ✅ style/explode | ⚠ collectionFormat | ✅ | ✅ | — | — | — | -| Multiple content types per body | ✅ | ⚠ consumes | ✅ @header contentType | ⚠ | — | ✅ | — | -| Multipart/form encoding | ✅ encoding | ✅ formData | ✅ multipart | — | — | — | — | -| Per-status error types | ✅ responses | ✅ | ✅ @error models | ✅ errors list | ⚠ errors union | — | ⚠ status codes | -| Streaming: server (SSE/chunk) | ⚠ text/event-stream | — | ✅ streams | ✅ eventstream | ✅ subscription | ✅ | ✅ stream | -| Streaming: client / bidi | — | — | ✅ | ✅ | — | ✅ | ✅ | -| Events / pub-sub channels | ✅ webhooks (3.1) | — | ⚠ | — | ✅ subscriptions | ✅ channels | — | -| Callbacks | ✅ callbacks | — | — | — | — | ⚠ reply | — | -| Pagination (first-class) | ⚠ x-* / links | — | ✅ @list/@pageItems | ✅ paginated trait | ⚠ connections | — | ⚠ AIP-158 | -| Long-running operations | ⚠ x-* | — | ✅ @pollingOperation | ⚠ | — | — | ✅ LRO (google) | -| Idempotency | ⚠ verb semantics | ⚠ | ⚠ | ✅ idempotent/@idempotencyToken | — | — | ✅ idempotency_level | -| Auth schemes | ✅ securitySchemes | ✅ | ✅ @useAuth | ✅ auth traits | ⚠ | ✅ | ⚠ | -| Per-op auth override (AND/OR) | ✅ security | ✅ | ✅ | ✅ | — | ✅ | — | -| Servers / endpoints | ✅ servers+vars | ✅ host | ✅ @server | ✅ endpoint | ⚠ | ✅ servers+protocols | — | -| Protocol bindings (kafka/amqp/…) | — | — | — | — | — | ✅ bindings | — | -| Versioning (added/removed) | — | — | ✅ @added/@removed | — | — | — | ✅ reserved | -| Deprecation w/ message | ✅ deprecated | ✅ | ✅ #deprecated | ✅ @deprecated | ✅ @deprecated(reason) | ✅ | ✅ | -| Examples | ✅ | ✅ | ⚠ | ✅ trait | — | ✅ | — | -| Docs: summary + description | ✅ | ✅ | ✅ @doc/@summary | ✅ @documentation | ✅ description | ✅ | ✅ comments | -| Vendor extensions / traits / directives | ✅ x-* | ✅ x-* | ✅ decorators | ✅ traits | ✅ directives | ✅ x-* | ✅ options | -| Field arguments (parameterized fields) | — | — | — | — | ✅ | — | — | -| Client-selectable response shape | — | — | — | — | ✅ selection sets | — | — | +| Capability | OpenAPI 3.x | Swagger 2.0 | TypeSpec | Smithy 2.0 | GraphQL | AsyncAPI | Protobuf | Erlang/OTP | +|---|---|---|---|---|---|---|---|---| +| Named object types | ✅ components.schemas | ✅ definitions | ✅ model | ✅ structure | ✅ type/input | ✅ schemas | ✅ message | ✅ -record/-type | +| Inline/anonymous types | ✅ | ✅ | ✅ | — (all named) | ⚠ | ✅ | ⚠ nested | ✅ type exprs | +| Inheritance / base types | ⚠ allOf | ⚠ allOf | ✅ extends | ⚠ mixins | ⚠ interfaces (conformance, not inheritance) | ⚠ allOf | — | — | +| Mixins / spread | — | — | ✅ spread | ✅ mixins | — | ⚠ traits | — | — | +| Tagged unions | ⚠ oneOf+discriminator | — | ✅ discriminated union | ✅ union | ⚠ union+__typename · ✅ @oneOf inputs (draft) | ⚠ oneOf | ✅ oneof | ✅ tagged tuples | +| Untagged unions | ✅ oneOf/anyOf | — | ✅ union | — | — | ✅ oneOf | — | ✅ \| | +| Intersection | ✅ allOf | ✅ allOf | ⚠ & (model is) | — | — | ✅ allOf | — | — | +| Negation | ✅ not | — | — | — | — | ✅ not | — | — | +| Enums (string) | ✅ | ✅ | ✅ named members | ✅ enum | ✅ | ✅ | ⚠ | ⚠ atom unions | +| Enums (numeric, valued) | ✅ | ✅ | ✅ | ✅ intEnum | — | ✅ | ✅ | ⚠ int unions | +| Open enums (unknown values allowed) | ⚠ anyOf trick | — | ⚠ union w/ string | ✅ (enums are open by default) | — | ⚠ | ✅ open (proto3/editions) / closed (proto2, per-enum feature) | ⚠ atom() fallback | +| Custom scalars | ⚠ type+format | ⚠ | ✅ scalar extends | ⚠ traits | ✅ scalar | ⚠ | — | ✅ -type/-opaque | +| Wire encoding hints (@encode / format) | ✅ format | ✅ format | ✅ @encode | ✅ timestampFormat | — | ✅ | ✅ fixed/zigzag/packed/delimited | — (ETF fixed) | +| Field wire IDs (numeric tags) | — | — | — | — | — | — | ✅ field numbers | ⚠ tuple positions | +| Wire name ≠ model name | ✅ (property key) | ✅ | ✅ @encodedName | ✅ jsonName (incl. union members) | — | ✅ | ✅ json_name | — | +| Optionality vs nullability distinct | ✅ (3.1) | ⚠ | ✅ | ⚠ presence only (null via @sparse collections) | ✅ | ✅ | ⚠ presence (3-state: implicit/explicit/required) | ⚠ :=/=> + 'undefined' | +| Defaults | ✅ | ✅ | ✅ | ✅ | ✅ args + input fields | ✅ | ✅ proto2 | ⚠ record fields | +| Constraints (min/max/pattern…) | ✅ | ✅ | ✅ decorators | ✅ traits | ⚠ directives (convention only) | ✅ | ⚠ protovalidate | ⚠ ranges, bit sizes | +| readOnly/writeOnly / visibility | ✅ | ✅ readOnly | ✅ @visibility classes | — | ✅ input vs output types | ⚠ (JSON Schema readOnly) | — | — | +| Recursive types | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| Maps / additionalProperties | ✅ | ✅ | ✅ Record | ✅ map | — | ✅ | ✅ map | ✅ :=/=> | +| Tuples | ✅ prefixItems (3.1) | — | ✅ | — | — | ✅ | — | ✅ native | +| Literal types | ✅ const | ⚠ single enum | ✅ | — | — | ✅ | — | ✅ atoms/ints | +| Operations grouped by service/interface | ✅ tags (3.2 parent/kind) | ⚠ tags | ✅ interface/namespace | ✅ service/resource | ✅ Query/Mutation/Subscription | ⚠ | ✅ service | ✅ module | +| Resource hierarchy (CRUDL) | — | — | ⚠ @autoRoute | ✅ resource (incl. put, instance vs collection ops) | — | — | — | — | +| HTTP binding (method/path/status) | ✅ | ✅ | ✅ @route/@get… | ✅ http traits | — | ⚠ ws binding | ⚠ transcoding | — | +| Param styles (explode, matrix…) | ✅ style/explode | ⚠ collectionFormat | ✅ | ✅ | — | — | — | — | +| Multiple content types per body | ✅ | ⚠ consumes | ✅ @header contentType | ⚠ | — | ✅ | — | — | +| Multipart/form encoding | ✅ encoding | ✅ formData | ✅ multipart | — | — | — | — | — | +| Per-status error types | ✅ responses | ✅ | ✅ @error models | ✅ errors list (client/server fault) | — | — | ⚠ status codes | ⚠ {error, R} variants | +| Streaming: server (SSE/chunk) | ✅ itemSchema/sequential media types (3.2) | — | ✅ streams | ✅ eventstream | ✅ subscription | ✅ | ✅ stream | ⚠ info streams | +| Streaming: client / bidi | — | — | ✅ client · ⚠ bidi | ✅ | — | ✅ | ✅ | ⚠ cast/info flows | +| Events / pub-sub channels | ✅ webhooks (3.1) | — | ✅ events/sse | — | ✅ subscriptions | ✅ channels | — | ✅ gen_event/info | +| Callbacks / request-reply | ✅ callbacks | — | — | — | — | ✅ reply (static + dynamic address) | — | ⚠ From-reply | +| Pagination (first-class) | ⚠ x-* / links | — | ✅ @list/@pageItems + prev/first/last links | ✅ paginated trait | ⚠ connections | — | ⚠ AIP-158 | — | +| Long-running operations | ⚠ x-* | — | ⚠ Azure.Core @pollingOperation | ⚠ smithy.waiters | — | — | ⚠ google.longrunning | ⚠ send_request | +| Idempotency | ⚠ verb semantics | ⚠ | — | ✅ idempotent/@idempotencyToken | — | — | ✅ idempotency_level | — | +| Auth schemes | ✅ securitySchemes | ✅ | ✅ @useAuth | ✅ auth traits | — | ✅ (wide: SASL/X509/userPassword; attaches to servers) | ⚠ | — | +| Per-op auth override (AND/OR) | ✅ security | ✅ | ✅ | ⚠ OR only, priority-ordered | — | ✅ | — | — | +| Servers / endpoints | ✅ servers+vars (3.2 named) | ✅ host | ✅ @server | ⚠ @endpoint hostPrefix only | — | ✅ named servers+protocols+security | — | ⚠ nodes/registry | +| Protocol bindings (kafka/amqp/…) | — | — | — | — | — | ✅ bindings | — | ✅ behaviours | +| Versioning (added/removed) | — | — | ✅ @added/@removed | ⚠ @since | — | — | — | — | +| Deprecation w/ message | ✅ deprecated | ✅ | ✅ #deprecated | ✅ @deprecated | ✅ @deprecated(reason) | ✅ | ✅ | ✅ -deprecated | +| Examples | ✅ | ✅ | ✅ @example/@opExample | ✅ trait (input/output/error scenarios) | — | ✅ (header+payload pairs) | — | — | +| Docs: summary + description | ✅ | ✅ | ✅ @doc/@summary | ✅ @documentation | ✅ description | ✅ | ✅ comments | ✅ -doc/EDoc | +| Vendor extensions / traits / directives | ✅ x-* | ✅ x-* | ✅ decorators | ✅ traits | ✅ directives (ordered, repeatable) | ✅ x-* | ✅ options | ⚠ module attributes | +| One-way (fire-and-forget) operations | — | — | — | — | — | ✅ send w/o reply | — | ✅ cast | +| Positional wire encoding (records as tuples) | ⚠ prefixItems | — | ⚠ tuples | — | — | ⚠ items array | — | ✅ records/tuples | +| Symbol/atom literal values | — | — | — | — | — | — | — | ✅ atoms | +| Unsolicited server-initiated messages | ⚠ webhooks | — | — | — | ⚠ subscriptions | ✅ channels | — | ✅ info | +| Multi-format payload schemas | — | — | — | — | — | ✅ schemaFormat (Avro/Protobuf/RAML) | — | — | +| Third-party field extensions / extension ranges | — | — | — | — | — | — | ✅ extend/extensions | — | +| Field arguments (parameterized fields) | — | — | — | — | ✅ | — | — | — | +| Client-selectable response shape | — | — | — | — | ✅ selection sets | — | — | — | ## Consequences for the IR @@ -63,21 +72,48 @@ Legend: ✅ native concept · ⚠ expressible indirectly · — absent optional discriminator (property name + value→variant mapping), whether the union is *tagged on the wire* (protobuf oneof, Smithy union — the wire format itself encodes the variant) vs *untagged* (JSON oneOf — variant inferred by validation), and open vs closed semantics - (anyOf ≈ open, oneOf ≈ exactly-one). -2. **Optionality ≠ nullability.** Four distinct states exist (required non-null, required nullable, - optional non-null, optional nullable); the IR keeps `Required` on the property and `Nullable` - on the type reference. + (anyOf ≈ open, oneOf ≈ exactly-one). Discrimination is not always property-based: Erlang + tagged tuples (and JSON arrays with a `const` head) discriminate by *position*, so the + discriminator needs a tuple-index form; GraphQL `@oneOf` and Smithy unions are *key-tagged* + (variant wire name is the object key). +2. **Optionality ≠ nullability ≠ presence.** Four distinct states exist (required non-null, + required nullable, optional non-null, optional nullable); the IR keeps `Required` on the + property and `Nullable` on the type reference. Protobuf adds a third axis — the *presence + discipline* (implicit/explicit/required) — which is not nullability (protobuf has no null) + and lives on its own field. 3. **Visibility must be lifecycle-based**, not a readOnly boolean: TypeSpec and GraphQL both need - per-usage (create/read/update/query) property filtering. OpenAPI readOnly/writeOnly lowers into it. + per-usage (create/read/update/query) property filtering, and TypeSpec's "visible in *no* + lifecycle" (`@invisible`) is a distinct state from "unrestricted". OpenAPI readOnly/writeOnly + lowers into it. 4. **Operations are protocol-neutral cores + protocol bindings.** The same operation node serves - HTTP (OpenAPI), RPC (protobuf/Smithy), and messaging (AsyncAPI) by attaching different bindings. - Streaming direction (unary, client, server, bidi) lives on the core, not the binding. + HTTP (OpenAPI), RPC (protobuf/Smithy), messaging (AsyncAPI), and actor protocols (OTP) by + attaching different bindings. Streaming direction (unary, client, server, bidi) and + one-way-ness (fire-and-forget: OTP cast, AsyncAPI send-without-reply, Thrift oneway, JSON-RPC + notifications) live on the core, not the binding. One operation may carry *several* bindings + of the same protocol (gRPC transcoding `additional_bindings`). 5. **Field identity is three names + an optional wire ID**: source name, IR canonical name, - wire/serialized name, and numeric tag (protobuf). Generators derive language names; the IR never - stores camelCase/PascalCase variants. + wire/serialized name, and numeric tag. The wire ID must admit zero (Cap'n Proto/FlatBuffers/ + Avro ordinals start at 0), and union *variants* need wire names too (Smithy `@jsonName` on + union members). Generators derive language names; the IR never stores camelCase/PascalCase + variants. 6. **Extensions are preserved, namespaced by origin** (`openapi:x-foo`, `smithy:aws.api#arn`, - `graphql:@key`), so no source metadata is lost and later generators can opt into them. + `graphql:@key`), so no source metadata is lost and later generators can opt into them. The + escape hatch only holds if *every* node that can carry source metadata has an extensions + slot — new spec revisions land fields on exactly the objects one forgets (responses, examples, + security schemes, server variables). 7. **GraphQL's parameterized fields and selection sets** don't map to fixed operations; the IR models schema entry-points as operations with arguments, and keeps the full type graph so a generator can offer query-building. This is deliberately lossy for arbitrary client queries — acceptable because Morphic generates SDK surface, not persisted queries. +8. **Messages need identity.** AsyncAPI reuses one named message across channels, operations, + and replies — messages live in a flat ID-keyed registry like every other named entity, and + request-reply carries both a static reply channel and a *dynamic* reply address (a path into + the request message's headers). +9. **Symbols are not strings.** Erlang atoms are a distinct term class (`ok` ≠ `<<"ok">>` on the + wire); the Values channel carries a symbol kind so backends degrade to strings explicitly, + never accidentally. +10. **Formats beyond this matrix already shape the IR.** Thrift/WSDL/Cap'n Proto service + inheritance (services carry IDs and an extends list), Avro/XSD/OData decimal + precision+scale, Avro aliases and enum fallback members, JSON-RPC positional params, and + per-type namespaces (proto packages, Avro fullnames) are all held by the IR so those + frontends never force a schema change. From c702f8ba489b6fd2156fb8b5df36429677490f60 Mon Sep 17 00:00:00 2001 From: OmarAlJarrah Date: Mon, 20 Jul 2026 00:32:09 +0300 Subject: [PATCH 2/2] docs: fold oagen source-audit findings into prior art and pipeline docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The original prior-art survey of oagen was written from a high-level study; auditing the actual source (parser, engine, and the differ/compat/verify subsystems the survey never covered) verified every §1 claim, sharpened four that were imprecise, and surfaced new lessons. None require IR schema changes — the strongest result is negative: nothing in oagen's ~25k-line codebase needed information the IR lacks, while its compat stack repeatedly re-derives by heuristic (field-set matching, Jaccard overlay correlation) the stable spec-level identity the IR carries by design. - prior-art.md §1: verified/sharpened mistake rows (discriminated request bodies are also flattened; discriminator inference is structural but welded to derived names; unwrapping applied destructively), new rows (ref-kind-on-reference forcing global state, collision-cascade renames, name-list accretion), new worth-adopting entries (wire-identity-keyed hints, group-metadata-over-wire-list, generation manifests, spec-only smoke baselines, behavioral-change channel, narrow expiring approvals), and a new subsection on the diff/compat/verify subsystems - architecture.md: real-world plan-layer decision list; canonical policy- input vocabulary with a declared-facts-win precedence rule; named write/integrate and surface-verification stages; filter-vs-scoped- regeneration distinction; pre-IR document patching vs IR overlay vs backend overlays; allowlist shape; wire-conformance test bullet - ir-design.md: $ref-sibling merge precedence and union-of-consts normalization placement in the OpenAPI lowering row; documented promotion path for mutually-exclusive parameter groups; open question 5 gains the structural third correlation tier --- docs/architecture.md | 61 +++++++++++++++++++++++++++++++--- docs/ir-design.md | 9 +++-- docs/prior-art.md | 78 +++++++++++++++++++++++++++++++++++++++----- 3 files changed, 134 insertions(+), 14 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 77330fc..2d7401e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -103,11 +103,22 @@ can enable: - **dedup** — structurally identical anonymous types are merged (by content hash), with ID aliases retained so provenance survives. - **filter** — include/exclude operations and types by pattern (Kiota-style path filtering), - followed by reachability trimming of orphaned types. + followed by reachability trimming of orphaned types. Filtering serves *surface reduction* + (a smaller SDK). Regenerating only one service of an existing SDK is a different problem and + never uses a filtered document: global decisions (dedup, shared files, naming) must stay + byte-identical across scoped runs, so the backend consumes the full document plus a scope + option and gates emission itself (a lesson oagen recorded after trying the filtered route). - **version-slice** — project a document carrying availability metadata into a concrete per-version snapshot (the TypeSpec versioning model: timeline stored, snapshot consumed). - **overlay** — user-supplied IR patches (rename hints, pagination declarations for specs that - can't express them, doc overrides) applied as data, not code. + can't express them, doc overrides) applied as data, not code. IR overlays are language-neutral + and each entry carries provenance (user-authored vs tool-inferred, mirroring the `Inferred` + marker) so automated overlay-generation loops stay auditable. Two related hooks live + deliberately elsewhere: *source-document* patching (e.g. the OpenAPI Overlay spec) is a + frontend option applied before lowering — some fixes must land before naming/hoisting + heuristics consume the broken shape — and *per-target-language* naming/compat overlays are a + backend input keyed by IR ID, so one IR document drives different compat baselines per + language. Passes operate on the IR only; they know nothing about source formats or target languages. @@ -118,7 +129,11 @@ except for the boundary they impose on the IR; the internal shape mirrors what K converged on independently: 1. **Plan** — compute language-*neutral* per-operation and per-type decisions once - (is-paginated, body presence, idempotency, error taxonomy) so templates contain no policy. + (is-paginated, body presence, idempotency, error taxonomy, primary-content/response + selection, return-shape classification (model / void / list-of-models with elementwise + deserialization), pagination item-type unwrap, parameter-passing shape (positional vs + options-bag)) so templates contain no policy. This list is the decision set oagen's + emitters demonstrably needed, computed once and shared across all languages. 2. **Refine** — per-language lowering: reserved words, casing, union representation strategy (native union vs sealed interface vs wrapper class), enum strategy for open enums, interface extraction. Everything a refiner needs must exist un-lowered in the IR — this is the IR's @@ -128,6 +143,24 @@ converged on independently: Language-specific naming (casing, reserved words, import layout) lives here exclusively. The IR carries source names, wire names, and naming hints — never camelCase/PascalCase renderings. +Two further backend-side stages are named here so their obligations shape the contracts, even +though both are post-milestone scope: + +- **Write/integrate** — regenerating into a live repo alongside hand-written code requires: + deterministic file paths, a generation manifest (spec/emitter/config hashes, sorted file + list, and an entity → generated-symbol map **keyed by IR stable IDs**, which survive path + and name churn), file-header provenance that gates pruning (never delete a file lacking the + generated-by header), ignore-region markers for hand-written islands, and additive-only + merging. Additive-only writers also need a staleness check — (previous-revision entities − + current entities) ∩ files-on-disk — to surface dead code that pruning cannot touch. +- **Surface verification** — per-language extractors project existing SDK source into a + neutral API-surface model; the same projection of generated output is diffed against it. + Change records are *neutral*; breaking/additive severity is an injectable per-language + policy function (parameter names are public API in PHP, arity in Go, almost nothing in + JS), consistent with principle 6. Behavioral changes (defaults) are a separate channel + from structural ones. Backends propagate IR IDs into manifests and reports so findings + correlate across languages without name-matching heuristics. + ### 2.4 Runtime/SDK policy is a separate input Retry, timeout, telemetry, error-class taxonomy, user-agent — the *behavioral* configuration of @@ -135,6 +168,18 @@ generated SDKs — is a backend input alongside the IR, not part of it. The IR d policy describes the SDK. (oagen embeds both in one root; we keep the trees separate so the same IR document can drive SDKs, docs, and mock servers without dragging SDK opinions along.) +The canonical policy-input vocabulary, taken from oagen's production `SdkBehavior` (the best +real-world enumeration available): **retry** (retryable status codes, max attempts, full backoff +strategy with jitter), **timeouts** (defaults + env override), **error taxonomy** (status → +logical exception-kind map, client/server catch-alls, doc-URL template), **telemetry** (request- +ID and client-telemetry headers), **logging** (a closed lifecycle-event list), **user-agent** +construction (identifier template, app-info enrichment), **idempotency injection** (header name, +auto-generate rules), **pagination pacing** (auto-page delay), and **request guards** (option +keys that must not appear as params — misuse detection). Delivery model: canonical defaults + +deep-partial per-backend/per-project overrides. Precedence rule: *declared IR facts win over +policy defaults* — `ErrorCase.Retryable/Fault` and `Operation.Idempotency` come from the spec; +policy fills in where the spec is silent, never the reverse. + ## 3. Go package layout ``` @@ -165,7 +210,10 @@ Every IR node carries a `Provenance` (source format, file, JSON pointer or line/ source name, and an `Inferred` marker naming the heuristic when applicable). Every stage returns `[]Diagnostic{Severity, Code, Message, Provenance}`. Codes are stable strings (`openapi/unresolved-ref`, `ir/dangling-type-ref`, `pass/discriminator-missing-variant`) so CI -can allowlist. Nothing in the pipeline writes to stderr; the CLI renders diagnostics. +can allowlist. A mature allowlist entry is keyed by (diagnostic code × entity ID), requires a +human rationale, supports expiry tied to a release, and rejects wildcards — narrowness is +validated, so approvals cannot rot into blanket suppressions. Nothing in the pipeline writes to +stderr; the CLI renders diagnostics. ## 5. Testing strategy @@ -177,6 +225,11 @@ can allowlist. Nothing in the pipeline writes to stderr; the CLI renders diagnos - **Round-trip property**: `parse → serialize → deserialize → deep-equal` for every corpus document. - **Architecture test**: import-graph assertions for the layering rules above. +- **Wire-conformance harness** (from milestone 3): expected request shapes (method, path, + query, body keys) are derived *from the IR alone* — offline, deterministic, shared across + every language backend; generated SDKs run under HTTP interception and the requests are + diffed. Request-side mismatches block; response-side mismatches inform. The decisive test of + a generated SDK is the bytes it puts on the wire, not whether it compiles. ## 6. Milestones diff --git a/docs/ir-design.md b/docs/ir-design.md index ad76956..7a3289f 100644 --- a/docs/ir-design.md +++ b/docs/ir-design.md @@ -1282,7 +1282,7 @@ How each format's distinctive concepts land in the IR (full details live with ea | Format | Lowering highlights | |---|---| -| **OpenAPI 3.x** | components/schemas → registry (IDs from pointers); inline schemas hoisted with hints; `allOf` → Base/Mixins per §4.3; `oneOf`/`anyOf` → Union (Exclusive bit), null-variant → Nullable ref; `discriminator` → Discriminator (3.2 `defaultMapping` → Discriminator.Default); `nullable`/type-arrays → Nullable; readOnly/writeOnly → Visibility (schema-level readOnly pushed down to referencing properties, residue → Extensions + diagnostic); `additionalProperties: false` → Additional=closed, `unevaluatedProperties: false` → closed_after_composition; parameters → Params + HTTPBinding locations w/ style/explode, 3.2 `in: querystring` → querystring location; requestBody/responses all content types → Payload.Contents; 3.2 `itemSchema`/`itemEncoding` → Content.Item/ItemEncoding; per-status responses/default → Conditions + ranges; webhooks → HTTPBinding.IsWebhook; callbacks → Callbacks; links → extensions (promotable later); securitySchemes/security → Auth OR-of-ANDs, 3.2 device flow + `oauth2MetadataUrl` → Flows/OAuth2MetadataURL; servers+variables (3.2 named) → Servers; tags (3.2 parent/kind) → groups + TagDefs; info contact/license → Document; schema `example(s)` → Examples; `xml` object (incl. 3.2 nodeType) → XMLHints at type and property level; `not`/`if-then-else`/`dependentSchemas`/`contains`/`unevaluated*` → verbatim Extensions per §4.7; `patternProperties` → AdditionalProps.Patterns; `x-*` → namespaced Extensions (legal on every object — hence Extensions on every node); pagination only via injectable policy, marked Inferred | +| **OpenAPI 3.x** | components/schemas → registry (IDs from pointers); inline schemas hoisted with hints; `allOf` → Base/Mixins per §4.3; `oneOf`/`anyOf` → Union (Exclusive bit), null-variant → Nullable ref; `discriminator` → Discriminator (3.2 `defaultMapping` → Discriminator.Default); `nullable`/type-arrays → Nullable; readOnly/writeOnly → Visibility (schema-level readOnly pushed down to referencing properties, residue → Extensions + diagnostic); `additionalProperties: false` → Additional=closed, `unevaluatedProperties: false` → closed_after_composition; parameters → Params + HTTPBinding locations w/ style/explode, 3.2 `in: querystring` → querystring location; requestBody/responses all content types → Payload.Contents; 3.2 `itemSchema`/`itemEncoding` → Content.Item/ItemEncoding; per-status responses/default → Conditions + ranges; webhooks → HTTPBinding.IsWebhook; callbacks → Callbacks; links → extensions (promotable later); securitySchemes/security → Auth OR-of-ANDs, 3.2 device flow + `oauth2MetadataUrl` → Flows/OAuth2MetadataURL; servers+variables (3.2 named) → Servers; tags (3.2 parent/kind) → groups + TagDefs; info contact/license → Document; schema `example(s)` → Examples; `xml` object (incl. 3.2 nodeType) → XMLHints at type and property level; `not`/`if-then-else`/`dependentSchemas`/`contains`/`unevaluated*` → verbatim Extensions per §4.7; `patternProperties` → AdditionalProps.Patterns; `x-*` → namespaced Extensions (legal on every object — hence Extensions on every node); `$ref`-adjacent sibling keywords (3.1) and ref-target annotations merge onto the referencing Property/Parameter with **use-site precedence**, applied uniformly (oagen's ad-hoc per-site patching is the counterexample); a oneOf/anyOf whose variants are all string consts normalizes to a closed `Enum` in a `pass/` normalization — not in the frontend — so per-variant `Docs` survive until the collapse is chosen; mutually-exclusive parameter groups (`x-mutually-exclusive-parameter-groups`) stay as namespaced Extensions, and their documented *promotion* (no dedicated node needed) is a pass that synthesizes one logical `Parameter` typed by a `Union` of variant models, bound via `HTTPParamBinding.ParamPath` per field; pagination only via injectable policy, marked Inferred | | **Swagger 2.0** | lifted to OpenAPI 3.x shape first (body/formData → Payload; host/basePath/schemes → Servers; consumes/produces → content types), then the OpenAPI lowering runs | | **TypeSpec** | consumed post-check (monomorphized, `isFinished`); template instances → TypeCommon.Instantiation incl. value args → TemplateArg; models → Model w/ Base + spread provenance → Mixins; scalars → Scalar chains, constructors in values → Value.Ctor; `@encode`/`@format` → Encoding triple; `@encodedName` → WireNameByFormat at property AND type level; unions w/ named variants → Union, `@discriminated` → Discriminator.PropertyName/Envelope/EnvelopeValueName; `| null` → Nullable; visibility classes (incl. custom, `@invisible` → Visibility.None) → Visibility, op overrides → ParameterVisibility/ReturnTypeVisibility; `@patch` implicitOptionality → HTTPBinding.PatchImplicitOptionality; interfaces → OperationGroups (versionable); `@overload` → OverloadOf; `@sharedRoute` → SharedRoute; `@service` → Service; versioning decorators incl. `@typeChangedFrom`/`@madeOptional`/`@madeRequired` and add/remove cycles → Availability timeline (on members/variants/params too); pagination decorators incl. prev/first/last links and header continuation tokens → Pagination PropPaths (In:"header"); Azure.Core `@pollingOperation`/`@finalOperation` → LongRunning; multipart w/ parts → Content.Encoding/PartEncoding, `Http.File` → FileInfo (content-type set, contents chain, filename location); streams/SSE → StreamDetail + Variant.Event (contentType, terminal); `@error` → UsageFlags.Error; `@example`/`@opExample` → Examples (Input/Output pairs); `@pattern` message → Constraints.PatternMessage; `@mediaTypeHint` → TypeCommon.MediaTypeHint; `never` members deleted + diagnostic per §4.8; TCGC client-shaping decorators (`@clientName`, `@access`, `@usage`, `@scope`, `@override`, …) → namespaced Extensions consumed by backend policy, never IR semantics; values/consts incl. enum-member refs → Values channel | | **Smithy 2.0** | structures → Model, mixins → Mixins (non-structure mixins flattened — spec-sanctioned); `document` → Any; unions → WireTagged Union, member `@jsonName` → Variant.WireName; enum/intEnum → Enum (open by default); `@sparse` → element Nullable; traits: constraints → Constraints, `@paginated` → Pagination (declared), `@retryable` → ErrorCase.Retryable + Throttling, `@error` fault → ErrorCase.Fault, `@readonly` → Idempotency safe, `@idempotent`/`@idempotencyToken` → Idempotency, `@sensitive` → Sensitive/Secret, `@tags` → Tags, `@clientOptional`/`@input` → Property.ClientOptional (+InputOnly), `@addedDefault` → DefaultAdded, root-shape `@default` pushed down to properties w/ provenance; `@streaming` blob → StreamDetail (+`@requiresLength` → RequiresLength); event streams → StreamDetail.Events union + Property.EventHeader/EventPayload + Initial messages; service-level errors → Service.CommonErrors; protocol traits → Service.Protocols; service `rename`/`version` → Service.Renames/Version; resources → OperationGroup + ResourceInfo (identifiers, properties, lifecycle incl. put/@noReplace, instance vs collection ops); http traits → HTTPBinding incl. `@endpoint`/`@hostLabel` → HostPrefix/host location (additive binding), `@httpPrefixHeaders`/`@httpQueryParams` → Prefix bindings, `@httpResponseCode` → Response.StatusCodeProp, `@requestCompression` → Compression, `@httpChecksumRequired` → ChecksumRequired; `@auth` order → priority-ordered Auth, `@optionalAuth` → empty option; `@jsonName` → WireName; `@mediaType` → Encoding.MediaType; xml traits → XMLHints at type and property level; `@examples` → Examples (Input/Output/Error); waiters + rules-engine traits → verbatim Extensions (§15); `smithy.api#Unit` → nil payload / shared empty Model for tag-only variants; other traits → namespaced Extensions | @@ -1329,7 +1329,12 @@ How each format's distinctive concepts land in the IR (full details live with ea runs but churn when a schema moves within its source file. TCGC solves cross-version correlation with name-based `crossLanguageDefinitionId`s. Likely resolution: keep pointer IDs as the structural identity and add a derived, name-based correlation key computed by a pass — - decide once IR diffing between spec revisions is actually built. + decide once IR diffing between spec revisions is actually built. Evidence from oagen's + differ/compat stack: (a) diffing IR-document-to-IR-document is the right plane (oagen already + does this); (b) name-based correlation alone still misses renames — plan a third *structural* + tier (content-hash / field-set similarity, shared with the `dedup` pass) as the fallback when + both pointer and name churn. Without stable identity, rename detection costs hundreds of + lines of best-effort heuristics downstream (see prior-art.md §1). 6. **Response discrimination placement** — `ResponseConditions` (status ranges) is HTTP-shaped living on the neutral core. Fresh evidence that it sits awkwardly: RPC single-response and OTP call replies both use empty conditions, and `Response.StatusCodeProp` is likewise diff --git a/docs/prior-art.md b/docs/prior-art.md index 1bb99b5..4adc00e 100644 --- a/docs/prior-art.md +++ b/docs/prior-art.md @@ -9,6 +9,8 @@ each. This is the evidence base behind `architecture.md` and `ir-design.md`. oagen parses OpenAPI 3.x into a small structural IR (`ApiSpec` → services → operations, plus a flat model/enum registry) and hands that IR to per-language emitter plugins. +Every claim in this section has been verified against the source (parser, IR, engine, +and the differ/compat/verify subsystems added since the original survey). ### Worth adopting @@ -21,23 +23,83 @@ plus a flat model/enum registry) and hands that IR to per-language emitter plugi - **Models referenced by name/handle, never embedded** — a flat registry makes deduplication, traversal, and serialization trivial and makes recursive types a non-issue. - **A "resolved plan" layer between IR and templates**: semantic rendering decisions - (is-paginated, has-body, idempotent-post…) are computed once, language-neutrally, so string + (is-paginated, has-body, idempotent-post, primary-response selection, return-shape + classification, parameter-passing shape…) are computed once, language-neutrally, so string templates contain no policy. - **Runtime SDK policy (retry, telemetry, error taxonomy) kept in a separate tree** from the structural API description. +- **Policy addressed by wire identity, not derived names** — consumer operation hints are keyed + `"METHOD /path"`, never by generated method name, because derived names churn. Independent + convergence on stable source identity; Morphic's IDs make the same key durable across path + renames too. The hint vocabulary itself (rename, remount, split-union-body into typed wrapper + methods, constant defaults, client-config-derived fields, URL-builder ops) is a concrete + inventory of what a client-shaping backend policy input must express. +- **Group metadata over an untouched wire list** — mutually-exclusive parameter groups + (`x-mutually-exclusive-parameter-groups`) are modeled as grouping metadata whose variants + share the parameter objects by identity, while the flat wire arrays stay authoritative for + serialization. Ergonomic structure layered on wire truth, neither duplicated nor destroyed. +- **A generation manifest + provenance-gated integration** — regenerating into a live repo + rests on: a manifest (spec/emitter/config hashes, sorted file list, operation → generated + symbol map), file-header provenance that gates pruning (never delete a file lacking the + generated-by header), ignore-region markers for hand-written islands, and additive-only + AST merges. Manifests should key entities by stable ID — oagen's `"METHOD /path"` keys break + on path renames. +- **Wire-conformance smoke testing with a spec-only offline baseline** — the expected request + shape (method, path, query, body keys) is derived from the spec alone and diffed against the + generated SDK running under HTTP interception; request-side mismatches block, response-side + inform. The decisive test of a generator is bytes-on-the-wire, not compilation. +- **A behavioral-change diff channel** — default-value changes are surfaced separately from + structural signature changes (same signature, different runtime behavior), and every compat + finding carries *drift provenance* (which pipeline stage caused it) plus, where detectable, + a spec-level *remediation* hint (e.g. "new schema forks the old — extend instead"). +- **Narrow, expiring diagnostic approvals** — intentional breaking changes are allowlisted per + (symbol × change category) with a required reason, optional expiry and tracker link, and + wildcards rejected. The shape a mature CI allowlist takes. ### Mistakes to avoid | oagen behavior | Morphic decision | |---|---| -| Models keyed by *derived string names*; collisions resolved silently ("keep largest"), refs fixed by string rewriting | Stable synthetic IDs (source-pointer-derived); names are presentation metadata | -| Inline-schema naming logic duplicated across ≥4 call sites that must stay byte-identical | One hoisting pass, keyed by source-pointer identity | +| Models keyed by *derived string names*; collisions resolved silently ("keep largest"), refs fixed by string rewriting. Same failure class for hoisted inline enums (name-colliding enums silently adopt the first's member set) | Stable synthetic IDs (source-pointer-derived); names are presentation metadata | +| Inline-schema naming logic duplicated across ≥6 call sites that must stay byte-identical — worse, *inferred discriminator mappings are name strings* that must reproduce the hoisting pass's naming byte-for-byte across files or they dangle | One hoisting pass, keyed by source-pointer identity; `Discriminator.Mapping` points at IDs, severing the naming coupling | +| Reference nodes bake in the *target's* kind (`enum` vs `model` ref), so no reference can be built without global knowledge of all targets — the root cause of the two-pass module-global state | One `TypeRef{Target TypeID}`; kind lives on the registry entry | | `allOf` eagerly flattened to a field list — the inheritance relationship is lost | Composition preserved un-lowered (base + mixin provenance kept) | -| oneOf branches without discriminator merged as *optional fields* — exclusivity constraint lost | Unions always survive as union nodes | -| Pagination/envelope/discriminator detection via hardcoded name lists | Heuristics are injectable frontend policy, never IR semantics | -| Failures are `console.warn`; unresolved refs degrade to `unknown` silently | Typed diagnostics with severity + provenance, collected on the IR document | -| Single "primary response" privileged; media-type multiplicity collapsed by fixed priority | All responses and all content types are first-class | -| Mutable module-global parser state (non-reentrant) | Frontends are pure `(source, options) → (IR, diagnostics)` | +| oneOf merged as *optional fields* at top-level-schema and request-body positions — even when explicitly discriminated, for argument-spreading ergonomics (variant `required` discarded); only property-level unions survive as union nodes | Unions always survive as union nodes; ergonomic flattening is a backend plan decision | +| Pagination/envelope detection via hardcoded name lists (with a fabricated `after` cursor param when none exists); single-resource envelope unwrapping applied *destructively* — the wrapper key vanishes from the IR with no recorded path | Heuristics are injectable frontend policy, never IR semantics; unwrap decisions are recorded as `PropPath`s, never applied to the stored shape | +| Discriminator *inference* is structural (const-property across variants — sound), but its output is welded to derived names (see above); name lists appear as tie-break preference order | Structural inference marked `Inferred`; output keyed by ID | +| Failures are `console.warn`; unresolvable model refs left dangling with a warning; unrecognized schema shapes degrade to `unknown` — silently when anonymous | Typed diagnostics with severity + provenance, collected on the IR document | +| Single "primary response" privileged (all 2xx kept, but inline-model extraction and pagination classification follow whichever iterates last); media-type multiplicity collapsed by fixed priority | All responses and all content types are first-class | +| Mutable module-global parser state (non-reentrant; concurrent parses race) | Frontends are pure `(source, options) → (IR, diagnostics)` | +| Collision-cascade operation renaming: same-named ops renamed in place by appending path context — adding one endpoint can rename *existing* SDK methods | Names are presentation; identity is the ID; naming collisions are a backend policy concern | +| Name sanitization accretes hardcoded domain word-lists (acronym sets, `Json`-fork collapse passes doing registry surgery via string rewriting) | Acronym/cleanup policy is injectable per-frontend; the registry is never edited by string rewrite | +| Catch-all `additionalProperties` smuggled as a magic-named synthetic field (collides with a real property of that name); `patternProperties` truncated to the first pattern | `AdditionalProps{Value, Key, Patterns}` is its own channel | +| `$ref`-site vs ref-target annotation merging patched ad-hoc for parameters only; model fields typed by `$ref` still lose target defaults | One documented precedence rule (use-site overrides target), applied uniformly in the frontend | + +### The diff/compat/verify subsystems — identity re-derived by heuristic + +oagen grew an IR-to-IR spec differ, a cross-language backwards-compat checker (per-language +extractors parse SDK source into a neutral API-surface model, diffed under per-language policy), +and a self-correcting overlay loop. Three structural lessons: + +- **Everything correlates by display name, and it costs hundreds of lines of heuristics.** + The differ matches models/operations by name, so a rename is indistinguishable from + remove+add; the compat layer then *re-derives* identity structurally (field-set superset + matching, enum value-set equality, Jaccard-similarity overlay matching ≥0.6). Its symbol + model even has a dual-key design (`id` + name) intended for rename detection — but `id` is + derived from the name, so the rename branch is dead. Cross-language change rollup finally + reinvents a spec-level identity (`conceptualChangeId`). Morphic's pointer-derived IDs make + rename detection a lookup; the compat experience adds one refinement — when both pointer + *and* name churn, a third *structural* correlation tier (content-hash / field-set + similarity, shared with the `dedup` pass) is the fallback (see ir-design.md open question 5). +- **Breaking-ness is per-language policy, not a property of a change.** The same spec change + is breaking in PHP (param names are public API), soft-risk in Kotlin, invisible in Node. + oagen's two-stage split — neutral change records, then `(change × language) → severity` + policy — is the right architecture for any future Morphic diff pass. +- **The comparison plane is an extracted "SDK surface", not the IR.** Live SDK source and + generated output are both projected into a neutral surface model and diffed there; the IR's + roles are to generate one side and to scope the baseline. Nothing in the ~20-file compat + subsystem needed information Morphic's IR lacks — the strongest validation the audit + produced. ---