Skip to content

Structured output: constrain an answer to a JSON schema - #10

Merged
pathscale merged 1 commit into
masterfrom
feat/structured-output
Jul 28, 2026
Merged

Structured output: constrain an answer to a JSON schema#10
pathscale merged 1 commit into
masterfrom
feat/structured-output

Conversation

@pathscale

Copy link
Copy Markdown
Owner

The capability behind a code-review interface, kept as a primitive. The template (review prompts, a findings shape, diff handling) belongs a layer up in the consumer; this is the part only this crate can express, because it is a CLI flag.

Missed 0.2.0: it was pushed to #9 after that PR had already been merged, so it lands as 0.2.1.

let outcome = run(&Request::new(Agent::Codex, "Alice is 30 years old.")
    .schema(r#"{"type":"object",
                "properties":{"name":{"type":"string"},"age":{"type":"integer"}},
                "required":["name","age"],"additionalProperties":false}"#))
    .await?;

assert_eq!(outcome.structured.unwrap()["name"], "Alice");

The three differ, and the difference is hidden

Delivery Where the value comes back
Claude --json-schema <inline> its own structured_output field
Codex --output-schema <FILE> the answer text
Copilot none

Codex needs a file, so the runner writes one and removes it through a guard, meaning cancel and timeout clean up too. Its answer is parsed only when a schema was asked for, so prose is never reinterpreted as data. Copilot 1.0.75 has no schema support, so asking is Error::Unsupported rather than prose presented as data.

Both shapes verified against the real CLIs, including a test that no schema file is left in the temp directory.

Two things found while building it

Codex requires strict schemas. It sends yours to a provider that demands "additionalProperties": false on every object and rejects the request with a 400 before the model runs:

additionalProperties is required to be supplied and to be false

Claude accepts the looser form, so a schema that works there can still fail on Codex. Documented on the builder and in the README, since the error comes from the provider rather than anything this crate could check.

That rejection exposed a reporting bug. Error::Failed took the first line of stderr as the cause, but CLIs open with progress chatter. The report read "Reading additional input from stdin...", which explains nothing, while the real cause sat in Codex's JSON error event on stdout. Now a line that looks like an error wins over one that narrates, and stdout is consulted when stderr only narrates.

That misdirection cost me three rounds of diagnosis on this PR, which is exactly what it would cost a user reading the error.

112 unit tests, clippy clean, cargo package verifies, live schema tests pass on Claude and Codex.

A consumer building anything on top of this, a code review being the case in
hand, wants findings back as data. Without a schema it has to parse prose,
which is a guess about formatting the model never promised.

`Request::schema` constrains the answer and `Outcome::structured` carries the
parsed value. The template that uses it, review prompts, a findings shape, diff
handling, belongs a layer up in the consumer: this is the capability, not the
policy, and only this crate can express it because it is a CLI flag.

The three agents differ and the difference is hidden rather than leaked:

- Claude takes the schema inline and reports the conforming value in its own
  `structured_output` field.
- Codex reads it from a file, so the runner writes one for the run and removes
  it afterwards through a guard, meaning cancel and timeout clean up too. Its
  answer is the JSON itself, parsed only when a schema was asked for, so prose
  is never reinterpreted as data.
- Copilot 1.0.75 has no schema support at all, so asking is Error::Unsupported
  rather than prose presented as data.

Both shapes verified live against the real CLIs.

Two things found while building it, both worth keeping:

Codex sends the schema to a provider requiring `additionalProperties: false` on
every object, and rejects the request with a 400 before the model runs
otherwise. Claude accepts the looser form, so a schema that works there can
still fail on Codex. Documented on the builder and in the README, since the
error arrives from the provider rather than from anything this crate could
check.

That rejection also exposed a reporting bug. Error::Failed took the first line
of stderr as the cause, but CLIs open with progress chatter: the report read
"Reading additional input from stdin...", which explains nothing, while the real
cause sat in Codex's JSON error event on stdout. A line that looks like an error
now wins over one that narrates, and stdout is consulted when stderr only
narrates. That misdirection cost me three rounds of diagnosis here, which is
exactly what it would cost a user.
@pathscale
pathscale merged commit 377cf95 into master Jul 28, 2026
2 checks passed
@pathscale
pathscale deleted the feat/structured-output branch July 28, 2026 23:26
@pathscale

Copy link
Copy Markdown
Owner Author

This PR now has two commits. Please wait for both before merging — the second landed after the first push, which is the pattern that stranded work on #5, #8 and #9.

Second commit: a real bug, now observable

45fbf13 fixes something I had flagged as unverified and turned out to be wrong.

The auth classifier's Copilot branch was inference. I said so at the time, because a bogus COPILOT_GITHUB_TOKEN still fell through to the stored credential and I could not produce a logged-out run to look at.

Pointing Copilot at an empty HOME produces one, and none of the phrases matched:

Error: No authentication information found.

Claude says "Not logged in". Codex says "not logged in". Copilot shares no vocabulary with either. So a missing Copilot login was reported as a generic Error::Failed with no login hint, which is precisely the case Error::NotAuthenticated exists for. Its verbatim wording is now in the list and pinned by a test.

What that says about the rest: the phrases that were observed are right, and the one that was inferred was wrong in a way no amount of reasoning would have caught. The remaining unobserved phrases are still guesses. They stay because a false negative there costs a worse error message rather than a wrong result, but they are not evidence.

Two other things worth recording from the same experiment:

  • An unauthenticated Copilot run exits 1 immediately, before any model call, so it costs nothing. A probe is only expensive in the authenticated case, which is the one you were trying to confirm, so AuthStatus still reports Unknown for Copilot rather than spending a turn.
  • No sudo was needed. A fresh HOME was enough, which also means Copilot's credential lookup is HOME-scoped rather than keychain-scoped in the way I had assumed.

113 unit tests, clippy clean, cargo package verifies.

@pathscale

Copy link
Copy Markdown
Owner Author

Third commit: the streaming fix (0210a52). This PR now has three; please wait for all of them.

Streaming existed but was effectively off with the settings a caller gets for free.

Two causes. Format defaulted to Json, under which nothing is observable until the turn ends — a twenty-minute run reported nothing for twenty minutes, which is indistinguishable from a hang. And Request::session pinned that format, so the multi-turn path a chat UI always uses could not stream at all. That is the worst possible place for it to be off, and it was my doing: I picked Json because it was the cheaper format carrying a session id, optimising for the wrong thing.

Stream is now the default. Verified against the live CLI that it carries everything Json does, session_id and structured_output included, so the change costs only parsing. session() now validates the format rather than overriding it, which is all it ever needed to do.

Claude also needed --include-partial-messages. Without it Claude emits only completed messages, so text arrives a paragraph at a time. With it, Claude streams tokens the way Copilot already did — meaning the abstraction was making Claude the coarser of the two, which is backwards.

Claude sends both the deltas and the finished message they build up to, so emitting both would show every answer twice. The finished copy is dropped once deltas have been seen, detected rather than configured: deltas always precede it, so seeing one proves the copy is redundant. Suppression covers text and thinking only, never tool calls, which the deltas do not duplicate.

Proven live, not by reading flags:

streamed 4 chunks for a 254 byte answer

with no .format() call. The test also asserts the concatenated chunks do not exceed the final answer, which is exactly what would happen if the finished message were emitted on top of them.

Full live suite: 14/14. 117 unit tests, clippy clean, cargo package verifies.

Not in scope, noted for later: system/status and system/thinking_tokens would give a "thinking… N tokens" indicator, and --brief exposes a SendMessage tool the agent can use to narrate deliberately — that one already arrives as Event::ToolCall { name: "SendMessage" }, so it needs no plumbing at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant