feat(schema): add semantic IR and symbol ID infrastructure

[hardbyte]

Summary

Adds semantic IR infrastructure (#96), fixes Python codegen for flattened tagged-union fields (#123), makes Python a first-class codegen backend with namespace classes, typed errors, field descriptions, and SemanticSchema-driven code generation.

Semantic IR Infrastructure (reflectapi-schema)

Module	Purpose
symbol.rs	SymbolId / SymbolKind — stable unique identifiers for all schema symbols
ids.rs	ensure_symbol_ids() — canonical ID assignment with cross-typespace disambiguation
semantic.rs	Immutable SemanticSchema, SemanticType, SymbolTable, ResolvedTypeReference
normalize.rs	NormalizationPipeline + Normalizer (&Schema → SemanticSchema)

Schema type changes: id: SymbolId on all types, Normalizer::normalize(&Schema), original_name on SemanticType preserving pre-normalization qualified names.

Python Codegen — Now Driven by SemanticSchema

The Python codegen uses SemanticSchema as the primary driver:

• Type iteration: semantic.types() (deterministic BTreeMap order, replaces manual topological sort)
• Import detection: SemanticType pattern matching
• Function ordering: semantic.functions()
• Raw Schema kept for rendering (concrete field/variant data)

Python Codegen — Wire-Compatible Flatten (#123)

For #[serde(flatten)] on internally-tagged enums, generates per-variant models merging parent fields + tag + variant fields into a discriminated union RootModel.

Python Codegen — First-Class DX

• Namespace classes mirroring Rust module structure (auth.UsersSignInRequest)
• Field descriptions via Field(description="...")
• Typed error returns — ApiResponse[OutputType, ErrorType] in method signatures
• Typed error deserialization — ApplicationError.typed_error as Pydantic model
• Typed list responses — list[Model] via TypeAdapter (was returning raw dicts)
• Fast JSON parsing — Pydantic's Rust-based validate_json(bytes)
• Factory classes removed — direct construction via namespace types (-13% file size)
• Docstring escaping for backslashes, triple-quotes, Python keywords in method/parameter names

Python Runtime Fixes

• TypeAdapter for all response validation (handles list[Model], generics, unions)
• error_model parameter on _make_request for typed error deserialization
• ApiResponse[T, E] generic with both success and error type parameters
• validate_json(bytes) fast path for Pydantic's Rust-based parser

Other Changes

• Architecture documentation (docs/architecture.md)
• Fixed dead README links
• Pin mdbook 0.4.x in CI (fixes 3-month doc build failure)
• Merged Andrey/refactoring cleanup #122 askama removal
• 21 new edge case snapshot tests

Real-World Validation

Partly's core-server (284 endpoints, 78K-line schema):

• 47K-line Python client (was 68K before factory removal)
• Valid Python, imports in ~0.65s
• Authenticates against live API, typed responses and errors work
• list[BillingCurrencyListItem] returns Pydantic models
• ApplicationError.typed_error = CustomerGetErrorCustomerNotFoundVariant

Test Coverage

• 220 tests total (0 failures)
• 166 demo snapshot tests (21 new edge case tests)
• 37 schema crate tests
• All 3 CI workflows green (including doc build)

[claude]

⚠️ Code review skipped — your organization's overage spend limit has been reached.

Code review is billed via overage credits. To resume reviews, an organization admin can raise the monthly limit at claude.ai/admin-settings/claude-code.

Once credits are available, reopen this pull request to trigger a review.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 86c00edfb8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[hardbyte]

@claude review

[hardbyte]

@claude review

[claude]

🔴 The find_map at python.rs:462 returns only the FIRST flattened internally-tagged enum field; if a struct has two such fields (valid in Rust when they use different tag names), the second enum's variants are never generated and are silently dropped from the Python output. Any consumer deserializing such a struct will face a mismatch: the Rust type has two independent discriminated unions flattened in, but the Python model only reflects one of them.

Extended reasoning...

What the bug is and how it manifests

In render_struct_with_flatten (python.rs lines 457–475), the iterator chains .filter(|f| f.flattened()).find_map(...) to locate a flattened internally-tagged enum field. find_map short-circuits on the first match and returns Option<(field, enum_def, tag)>. Only that single enum_def is ever passed to render_struct_with_flattened_internal_enum. If a struct has a second flattened internally-tagged enum field — valid Rust with serde when the two enums use distinct tag field names (e.g. type and kind) — find_map never sees it.

The specific code path that triggers it

Inside render_struct_with_flattened_internal_enum, the loop at lines 561–578 iterates over all flattened fields:

for field in struct_def.fields.iter().filter(|f| f.flattened()) {
    let type_name = resolve_flattened_type_name(&field.type_ref);
    if let Some(reflectapi_schema::Type::Struct(_)) = schema.get_type(type_name) {
        // expand struct fields into base_fields
    }
    // Enum fields are handled below as variants   <-- misleading comment
}

The comment says "Enum fields are handled below as variants", but "below" refers only to the for variant in &enum_def.variants loop, which iterates over the variants of the ONE enum that was found by find_map. A second flattened internally-tagged enum field is neither expanded into base_fields nor iterated as a variant block. It is completely skipped.

Why existing code does not prevent it

The function signature render_struct_with_flattened_internal_enum(... enum_def: &Enum ...) accepts a single enum. There is no mechanism to pass, receive, or render a second enum. The test suite (test_flatten_internally_tagged_enum_field) uses a struct with exactly one flattened enum, so the missing second-enum path is never exercised.

What the impact would be

Given a Rust struct:

struct Combined {
    id: String,
    #[serde(flatten)] action: ActionKind,   // internal tag "type"
    #[serde(flatten)] status: StatusKind,   // internal tag "kind"
}

The generated Python model would contain only the ActionKind discriminated union variants. Every variant that comes from StatusKind — including its tag field "kind" — is absent from the Python output. Any Python code receiving a wire message with {"id":"1","type":"Create","kind":"Active",...} would fail to deserialize or would silently ignore the kind and all status-related fields.

Step-by-step proof

1. struct_def has two flattened fields: action: ActionKind (internal tag type) and status: StatusKind (internal tag kind).
2. .filter(|f| f.flattened()).find_map(...)) evaluates action first. ActionKind matches Representation::Internal, so find_map returns Some((action_field, action_enum_def, "type")) immediately.
3. status: StatusKind is never evaluated.
4. render_struct_with_flattened_internal_enum receives enum_def = ActionKind and generates CombinedCreate, CombinedDelete, etc. — no CombinedActive, CombinedInactive variants.
5. The inner loop at line 562 skips status because it is an Enum (not a Struct) and the comment defers to code that never runs for it.
6. Result: StatusKind's variants are entirely absent from the Python output.

How to fix it

Collect ALL flattened internally-tagged enums (not just the first), then either: (a) generate a cross-product of variant combinations, which is complex but wire-accurate; or (b) for each additional internally-tagged enum beyond the first, fall back to the standard field emission path used for non-internal enums, with a documented limitation. At minimum, a warning or error should be surfaced when multiple flattened internally-tagged enums are detected, rather than silently generating incorrect output.

[claude]

🔴 The Python codegen emits model_rebuild() calls for Union type aliases (e.g., ReflectapiDemoTestsSerdeValue = Union[...]) alongside real BaseModel subclasses inside a single try/except AttributeError block. Union aliases have no model_rebuild() method, so the call always raises AttributeError. Because all calls share one block, any Union alias that sorts alphabetically before a real BaseModel subclass will silently abort the entire try block, leaving the real model's forward references unresolved. The fix is to either wrap each model_rebuild() call in its own try/except block, or filter the list to exclude Union aliases.

Extended reasoning...

What the bug is and how it manifests

The Python codegen in reflectapi/src/codegen/python.rs (around line 1320-1332) collects all rendered type names, sorts them alphabetically, and emits them in a single try/except AttributeError block. In the snapshot flatten_untagged_enum_field-5.snap (lines 148-160), ReflectapiDemoTestsSerdeValue is a plain Python Union type alias, not a Pydantic BaseModel subclass:

ReflectapiDemoTestsSerdeValue = Union[
    ReflectapiDemoTestsSerdeValueNum, ReflectapiDemoTestsSerdeValueText
]

Union type aliases in Python are typing special forms and have no model_rebuild() method. Calling .model_rebuild() on them always raises AttributeError.

The specific code path that triggers the latent bug

Types are sorted alphabetically before the block is emitted (sorted_type_names.sort() in python.rs). In the tested snapshot, Cell (C) sorts before Value (V), so ReflectapiDemoTestsSerdeCell.model_rebuild() runs first and succeeds, and then the AttributeError from ReflectapiDemoTestsSerdeValue.model_rebuild() is caught. This specific case is harmless.

However, the structural defect is that all calls share one try/except block. Consider any schema where a Union alias name sorts alphabetically before a real BaseModel/RootModel subclass — for example, an 'AValue = Union[...]' alias and a 'BModel(BaseModel)' class. The sequence would be: (1) AValue.model_rebuild() raises AttributeError, (2) the except block catches it and execution exits the entire try block, (3) BModel.model_rebuild() is never called.

Why existing code does not prevent it

The comment 'Some types may not have model_rebuild method' shows the author anticipated this case, but the single-block structure is the defect. The only reason the tested snapshots work is that all real models happen to sort before the Union aliases in the current test cases. With 'from future import annotations' active (which this generated file uses), Pydantic defers annotation evaluation and depends on model_rebuild() being called to resolve forward references in complex schemas. Any schema where a Union alias sorts before a real model relying on forward reference resolution will silently produce broken Pydantic models.

Step-by-step proof for the latent ordering failure

Suppose a schema produces 'AValueUnion = Union[AVariant1, AVariant2]' and 'class BModel(BaseModel): field: SomeForwardRef'. In the single try/except block (alphabetical order): AValueUnion.model_rebuild() raises AttributeError, the except catches it and exits the block, BModel.model_rebuild() never runs, and SomeForwardRef remains an unresolved string annotation in BModel.

How to fix it

Option 1 (simplest): wrap each call in its own try/except so that a failure on a Union alias does not abort subsequent real model rebuilds. Option 2: filter the type name list at codegen time to exclude Union type aliases, only emitting model_rebuild() calls for actual BaseModel/RootModel subclasses.

[claude]

🔴 SemanticEnum.variants is BTreeMap<SymbolId, SemanticVariant> (semantic.rs:103), which sorts variants alphabetically by name rather than by declaration order. For #[serde(untagged)] enums, serde tries variants in declaration order and picks the first successful deserialization — any downstream codegen backend iterating SemanticEnum.variants will silently use the wrong order, causing incorrect deserialization when an alphabetically-earlier variant can absorb input intended for a later one. Fix by using IndexMap<SymbolId, SemanticVariant> or Vec to preserve insertion order.

Extended reasoning...

What the bug is and how it manifests

SemanticEnum.variants is typed as BTreeMap<SymbolId, SemanticVariant> (semantic.rs line 103). SymbolId derives Ord by field order: kind, then path: Vec, then disambiguator. All variants within the same enum share kind=Variant, and their path ends with the variant name — so the BTreeMap sorts them alphabetically by variant name, not by the order they appear in the source.

The specific code path that triggers it

In normalize.rs, build_semantic_enum (around line 1134) iterates enm.variants() — which returns variants in their raw declaration order (preserved in Vec) — and inserts each into a BTreeMap<SymbolId, SemanticVariant> keyed by SymbolId. The BTreeMap then re-sorts by SymbolId::Ord, discarding the position metadata. The Normalizer::build_semantic_enum code is:

for variant in enm.variants() {
    let semantic_variant = self.build_semantic_variant(variant)?;
    variants.insert(variant.id.clone(), semantic_variant);  // BTreeMap re-sorts
}

Why existing code does not prevent it

The raw Enum.variants field is Vec, which preserves declaration order. That order is available at the point build_semantic_enum iterates enm.variants(). However, the result is inserted into a BTreeMap which re-sorts by SymbolId. There is no assertion, test, or fallback that checks whether BTreeMap ordering matches declaration order.

What the impact would be

For #[serde(untagged)] enums, serde's contract is: try variants in declaration order, use the first that deserializes successfully. A codegen backend that iterates SemanticEnum.variants (the natural, intended API) will silently produce a client that applies variants in alphabetical order instead. This leads to incorrect deserialization for any untagged enum where two variants can both deserialize a given input — the wrong variant is selected with no error.

Example: an enum declared as [Integer(i64), Float(f64)] — both variants can deserialize the JSON value 42. Serde (declaration order) picks Integer. A backend using SemanticEnum.variants iteration (alphabetical) tries Float first and picks Float. The generated client silently deserializes a different type than Rust would.

The new test case added in this PR includes test_flatten_untagged_enum_field with enum Value { Num { value: f64 }, Text { text: String } }. Alphabetically Num < Text, which happens to match declaration order here. But for any enum where declaration order differs from alphabetical order, the bug manifests.

How to fix it

Replace BTreeMap<SymbolId, SemanticVariant> with an insertion-order-preserving collection:

• IndexMap<SymbolId, SemanticVariant> from the indexmap crate — preserves insertion order, provides O(1) keyed lookup
• Vec — simplest, no key-based lookup without an auxiliary index

The same fix is needed for SemanticVariant.fields and SemanticStruct.fields for correctness with positional (unnamed) fields.

Step-by-step proof

1. Define an untagged enum: variants declared as [Integer(i64), Float(f64)].
2. Normalizer::build_semantic_enum inserts Integer (SymbolId path=["MyEnum","Integer"]) then Float (path=["MyEnum","Float"]) into BTreeMap.
3. BTreeMap sorts by path lexicographically: "Float" < "Integer", so Float entry comes first in iteration.
4. A codegen backend calls semantic_enum.variants.values() and emits: try Float, then try Integer.
5. For JSON input 42: both would match — Float wins because it was tried first. Rust's serde would have picked Integer (declaration order). The generated client deserializes a different type silently.

[hardbyte]

@claude review

[github-actions]

📖 Documentation Preview: https://reflectapi-docs-preview-pr-124.partly.workers.dev

Updated automatically from commit 1dd3496