Broken link to file spec on home page - #71
Merged
Conversation
Seth Juarez (sethjuarez)
approved these changes
Aug 27, 2024
Seth Juarez (sethjuarez)
added a commit
that referenced
this pull request
Aug 5, 2026
One line on Property. `named_collection_vectors.json` requires that an
immediate primitive value in a name-keyed collection -- `inputs: { "city":
"Seattle" }` -- load as:
{ name: "city", kind: "string", default: "Seattle" }
with `example` absent. The vector header states it directly: "Immediate
primitive Property values infer kind and default without leaking
direct-coercion example semantics."
Without this declaration the emitter falls back to the @Coerce target,
which for Property is `example` (properties.tsp:17 and siblings). That is
correct for direct coercion and wrong for the collection shorthand -- one
@Coerce set cannot express both contexts, which is why typra #77 added
@entryShorthand for the collection case specifically.
Measured, all four executable backends, this commit alone (parent is the
0.4.23 bump):
runtime before after
rust (4 targets) 2 / 2 failed 4 passed / 0 failed
go (13 vectors) 9 / 4 failed 13 passed / 0 failed
ts (14 tests) 10 / 4 failed 14 passed / 0 failed
csharp(13 vectors) 9 / 4 failed 13 passed / 0 failed
Full-suite regression check at this commit:
rust cargo test --no-fail-fast 908 passed / 0 failed (24 targets)
go go test -count=1 ./... ok, no failures
ts npm test 1692 passed / 2 failed
csharp dotnet test 1305 passed / 49 failed
Rust is fully green for the first time in this effort. The 4 named-
collection failures cleared in every backend and nothing else changed
state.
Residual failures, all pre-existing and none related to this change:
ts 2 model-enrichment assertions in packages/openai and
packages/foundry, unrelated to the emitted models
csharp 48 typra #53 (fixture generator omits required model-typed
fields) and 1 typra #71
Not verified: Python still does not build (typra #43), so its regenerated
loader is unexecuted. Java and Swift have no runtime in this repository.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Seth Juarez (sethjuarez)
added a commit
that referenced
this pull request
Aug 5, 2026
…dels Fifth and final executable backend for the named-collection contract, after Go (ba4866e), TypeScript (36c9619) and C# (1a0c3e2). Rust was this contract's only executable home for most of this effort. Measured: .venv/Scripts/python -m pytest tests/test_named_collection_vectors.py 13 passed / 0 failed ruff check All checks passed! ruff format 1 file already formatted Python confirms the two-part #46 fix (emitter 0.4.23 plus the @entryShorthand("default") declaration in e2152ef) holds in a fifth independent backend. All five now pass the contract: rust 4 targets 0 failed go 13 vectors 0 failed ts 14 tests 0 failed csharp 13 vectors 0 failed python 13 vectors 0 failed Also corrects the record on Python's status. This effort has been carrying "Python does not build (typra #43)" as a standing assumption; it is stale. Measured full suite on this commit: .venv/Scripts/python -m pytest tests/ -q 1468 passed / 1 failed / 17 skipped / 56 deselected (11m48s) The single failure is test_spec_vectors.py::test_wire_vector [kind_to_json_type_mapping], and it is not a Python defect: ValueError: tools[0].parameters[4].items: missing required field spec/vectors/wire/wire_vectors.json declares that parameter as {"name": "an_array", "kind": "array"} with no items, so the vector requires ArrayProperty.items to be optional. It cannot be made optional -- schema/model/core/properties.tsp:80 must stay `items:` because `items?:` emits uncompilable Rust (typra #71). Verified by experiment: applying items?: and regenerating produces E0308 at property.rs:113 and :247, and the change was reverted. C# fails the identical vector: SpecVectorWireTests.OpenAI_Chat_WireFormat (name: "kind_to_json_type_mapping") with "tools.parameters.items: missing required field". So #71 blocks a canonical spec vector wherever wire_vectors.json is executed, rather than costing one isolated test. Not verified: Java and Swift have no runtime in this repository. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Seth Juarez (sethjuarez)
added a commit
that referenced
this pull request
Aug 5, 2026
Three of the four crates in the `runtime/rust` workspace did not compile.
Every "Rust is green" figure reported in this effort so far — including my
own 908 passed / 0 failed — was measured with `cargo test` run from inside
`runtime/rust/prompty`, which covers exactly one of four workspace members.
`cargo test --workspace` never ran because the build failed first.
Measured before (this commit's parent):
cd runtime/rust && cargo build --workspace
prompty-anthropic: 13 errors
prompty-openai: 19 errors
prompty-foundry: 6 errors (only visible once the other two compiled)
Attribution: pre-existing, NOT caused by the 0.4.23 emitter bump. Verified by
checking out 1a0c3e2 (the commit before the bump) and rebuilding — identical
34 error lines. The breakage is fallout from the earlier `agent.tsp`
optionality changes, which altered generated shapes but were never propagated
to these handwritten provider crates.
Five generated-shape changes had to be absorbed:
Prompty.model Model -> Option<Model>
Model.api_type apiType -> Option<apiType>
Property.enum_values Option<Vec<Value>> -> Vec<Value>
PropertyKind::Union Vec<Property> -> Option<Vec<Property>> (both arms)
Model.options reached through the now-optional Model
Note for future readers: `api_type` is `Option<apiType>` where `apiType` is a
generated enum, not a String. `.as_deref()` does not compile; the correct
accessor is `.as_ref().map(|t| t.as_str())`.
Also recorded because the assumption is tempting and wrong:
`ToolKind::Function.parameters` is `Vec<Property>` and is NOT optional
(prompty/src/model/tools/tool.rs:23). I added `.flatten()` on that basis and
the compiler rejected it; both sites were reverted.
Design decision — an absent model is treated as an empty model, not an error.
`build_embedding_args` / `build_image_args` return a bare `Value` and cannot
signal failure, so `model_id()` returns `""` for `None`. That routes into the
pre-existing empty-id fallbacks (text-embedding-ada-002, dall-e-3, gpt-4o),
preserving behaviour exactly for every test that does supply a model.
`resolve_connection` likewise returns `&Value::Null`, which the existing
`conn.get("kind")...unwrap_or("")` already handles.
Measured after:
cd runtime/rust && cargo test --workspace --no-fail-fast
1142 passed / 3 failed
234 tests that had never once executed now run. The 3 remaining failures are
addressed as follows: 2 are stale test fixtures, fixed in the next commit;
1 is typra #71 and is not fixable here.
cargo clippy --workspace --all-targets exit 0
cargo fmt -p prompty-openai -p prompty-anthropic -p prompty-foundry --check exit 0
NOT verified:
- `cargo fmt --all` reformats the generated `prompty` crate (~300 files). That
churn was reverted and is deliberately not part of this commit. Use the
per-package form above; do not run `cargo fmt --all` in this repo until the
emitter's output is rustfmt-clean.
- No provider crate is exercised against a live endpoint here; all coverage is
wire-format and vector tests.
- The Go, TypeScript, Python and C# runtimes are untouched by this commit and
were not re-run for it.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Seth Juarez (sethjuarez)
added a commit
that referenced
this pull request
Aug 5, 2026
Two tests failed with `context: missing required field`:
prompty-anthropic test_context_response_is_portable_without_native_continuation
prompty-foundry test_context_response_is_portable_without_native_continuation
Both called `ModelInvocationRequest::load_from_value(&json!({}), ...)`, i.e.
they asked the loader to accept an empty document.
Attribution, by vector authority. I grepped all of `spec/vectors/` for
`ModelInvocationRequest`: there is no vector governing it anywhere. With no
vector authorizing an empty document, the schema is the only authority, and
`schema/model/pipeline/invocation.tsp:119` declares `context` required. So
these were stale fixtures asking for something invalid, not evidence of an
emitter or schema defect. Fixed at the test-data layer.
The minimal schema-valid request supplies the six required
`ModelInvocationContextSnapshot` fields (id, sessionId, turnId, invocationId,
iteration, contextState). `messages` (= #[]), `stablePrefixMessages` (= 0),
`decisions?` and `metadata?` all default. `InvocationContextState` is fully
defaulted, so `"contextState": {}` is valid — confirmed at invocation.tsp:35-41.
Measured:
cd runtime/rust && cargo test --workspace --no-fail-fast
before: 1142 passed / 3 failed
after: 1144 passed / 1 failed
The single remaining failure is `kind_to_json_type_mapping`
(prompty-openai --test wire_vectors), which is typra #71 and is not fixable
in this repo. Vector `spec/vectors/wire/wire_vectors.json` §7.1.4 declares
`{"name": "an_array", "kind": "array"}` with no `items`; the generated
`property.rs:15` emits `Array { items: serde_json::Value }` as non-optional
while both the loader (:113) and the saver (:247) already honour optionality.
The same vector fails identically in Python and C#, making it a three-runtime
block on one canonical vector.
NOT verified:
- No live-provider call is made by either test; this is loader-shape coverage
only.
- I did not add a vector for `ModelInvocationRequest`. The absence of one is
what licensed this fix, and closing that gap is a separate piece of work.
- Go, TypeScript, Python and C# were not re-run for this commit; nothing here
can affect them.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Seth Juarez (sethjuarez)
added a commit
that referenced
this pull request
Aug 6, 2026
`spec/vectors/wire/wire_vectors.json` holds 27 canonical provider request-body
contracts (22 OpenAI, 5 Anthropic). Only Rust executed them. TypeScript ships
`buildChatArgs` / `buildResponsesArgs` / `buildEmbeddingArgs` / `buildImageArgs`
in `packages/openai/src/wire.ts` and `buildChatArgs` in
`packages/anthropic/src/wire.ts` — the exact surface those vectors govern — so
this was an unported lock, not a missing feature.
Ported from the Rust reference implementations:
packages/openai/tests/wire-vectors.test.ts
<- runtime/rust/prompty-openai/tests/wire_vectors.rs
packages/anthropic/tests/wire-vectors.test.ts
<- runtime/rust/prompty-anthropic/tests/vectors.rs
Two deliberate divergences from the Rust reference, both in the direction of
closing coverage gaps rather than reproducing them:
- Vector selection is data-driven (filter on `input.provider`) instead of a
hand-maintained `wire_test!(...)` list. Rust's list must be edited by hand
whenever a vector is added, so a new vector silently goes unexecuted. That is
the same failure mode that left this entire file unported.
- Each suite asserts its own vector count (22 openai / 5 anthropic / 27 total).
Data-driven selection introduces the opposite hazard — a filter that matches
nothing is vacuously green — and the count assertion is the guard against it.
The Anthropic suite went green on the first run, so it was mutation-verified to
prove it is not vacuous:
stop separating system messages from the conversation array
-> anthropic_system_separate fails (1 failed / 5 passed)
drop the required max_tokens default
-> anthropic_max_tokens_required and anthropic_tool_wire fail
(2 failed / 4 passed)
restore
-> 6 passed
The OpenAI suite needed no synthetic mutation: it caught a real defect on its
first execution (see below), which is stronger evidence that it is live.
Measured:
cd runtime/typescript
npm test 1701 passed / 0 failed
-> 1729 passed / 1 failed
npm run build exit 0
npm run lint exit 0
The +28 is exactly the 22 OpenAI vectors plus the 5 Anthropic vectors plus the
Anthropic count guard. The one failure is the OpenAI count guard's sibling —
`kind_to_json_type_mapping` — and it is a known emitter defect, not a defect in
this change:
tools[0].parameters[4].items: missing required field
at _ArrayProperty.load (core/src/model/core/property.ts:256)
That vector declares `{"name": "an_array", "kind": "array"}` with no `items`,
per wire_vectors.json §7.1.4. The generated `ArrayProperty.load` treats `items`
as required while the corresponding save path already honours its optionality.
This is typra #71, already open and already failing the same single vector in
Rust, Python and C#. Porting the vector here raises its blast radius from three
runtimes to four. The test is deliberately left failing rather than skipped, so
all four runtimes clear together when #71 ships.
NOT verified / known remaining gaps:
- Go has no provider layer at all (`runtime/go/prompty` contains only `model/`),
so wire_vectors is "feature absent" there rather than unported. Same for C#.
- No live provider call is made; this is request-body construction only.
- Rust, Go, Python and C# are untouched by this commit and were not re-run.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Seth Juarez (sethjuarez)
added a commit
that referenced
this pull request
Aug 6, 2026
Bump alone, no schema or handwritten change, so the regeneration can be attributed on its own. Measured before (0.4.23) and after (0.4.25), every command run in full: | runtime | command | 0.4.23 | 0.4.25 | |---------|----------------------------------------------------|----------|----------| | Rust | cargo test --workspace --no-fail-fast | 1144 / 1 | 1144 / 1 | | TS | npm test | 1729 / 1 | 1729 / 1 | | Go | go test ./... -count=1 | ok / 0 | ok / 0 | | C# | dotnet test | 1305 /49 | 1353 / 1 | C# is the only movement: +48 passing. Those 48 were typra #53 and are cleared by this release. The regeneration also emits +462 lines of new C# conversion tests across 12 files, all passing. Every remaining failure in every runtime is now the same single vector, `kind_to_json_type_mapping` (spec/vectors/wire/wire_vectors.json, §7.1.4), i.e. typra #71. Verified by name in Rust, TypeScript and C#: Rust tests/wire_vectors.rs kind_to_json_type_mapping TS tests/wire-vectors.test.ts kind_to_json_type_mapping C# SpecVectorWireTests OpenAI_Chat_WireFormat(name: "kind_to_json_type_mapping") Diff shape: 312 files, 13810 insertions / 4084 deletions. That count is misleading and is almost entirely rustfmt reflow -- 0.4.25 wraps the generated Rust that 0.4.23 emitted on single lines. Generated Go and TypeScript output is byte-identical to 0.4.23; the only non-Rust source changes are the 12 new C# test files and the pin itself. schema/model/connection/connection.tsp shows 2 deletions. That is `npm run build` running format:tsp, not a semantic edit. NOT verified in this commit: - Python. The suite takes ~12 min and was still running; its 0.4.23 baseline was 1468 / 1 (the 1 being #71). Reported separately. - Whether 0.4.25's remaining behaviour changes beyond #53 and rustfmt are exercised by any vector we execute. The runtime numbers only prove no regression in what we do execute. - No real-provider calls were made; all suites are offline/mocked. This bump does not fix #71 on its own by design. 0.4.25 carries typra #78, which makes the emitter honour declared optionality in variant declarations; the schema must then declare `items` optional for it to take effect. That schema line is deliberately a separate commit. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.