fix(runtime): unwrap claude CLI JSON envelope in Start() Raw#722
Merged
Conversation
ClaudeAdapter.Start ran `claude -p --output-format json` and returned
StartResult{Raw: result.Stdout} verbatim — the whole CLI result envelope
({"type":"result",...,"result":"<answer>"}), not the assistant's answer.
The sole consumer of Start().Raw is realSkillOptABDeliver's forked-session
path (empty RuntimeRef → Start), which feeds skillopt synth (challenger +
judge), skillopt ab --judge, skillopt binary, and the live-AB challenger.
JSON consumers (synth/ab) then parsed the envelope's outer object, found no
context/question/rubric, and silently rejected every item (bad_rubric, zero
scores). kimi already unwraps at kimi.go:56; codex returns prose. Claude was
the only enveloped Start.
Fix at the adapter (right altitude): reuse the existing parseClaudeJSONResult
helper (the same one the Deliver path uses to build Summary) to extract the
envelope's "result" into Raw. Fail-open — fall back to raw stdout when stdout
is not the envelope or has no result field, so unwrap never errors.
RuntimeRef behavior is unchanged. All other Start() call sites read only
RuntimeRef, so they are unaffected.
Closes #721
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jerryfane
added a commit
that referenced
this pull request
Jul 7, 2026
… judge answers (#724) (#726) Two composing halves of the same live `skillopt synth` failure (2026-07-08): 1. codex Start().Raw was the full `codex exec --json` transcript (thread.started banner, turn events, reasoning items), not the assistant's answer — the codex flavor of the claude bug fixed in #722 (b53269f). Forked-session consumers (skillopt synth/ab) that parse Start().Raw as the assistant's answer saw the whole transcript and could not find the challenger item's context/question/ rubric. Fix at the adapter (same altitude as #722): reuse parseCodexJSONResult — the exact parser the Deliver path uses to build Summary — to surface the joined agent_message text as Raw, failing open to the raw stdout when the stream carries no agent_message (older CLI, plain-text fallback, unexpected shape). RuntimeRef is untouched. 2. synthJudgePrompt embedded both weak/strong answers verbatim with no cap. Even after the unwrap a genuinely long answer, combined with both halves embedded, blew ARG_MAX on the kimi judge exec. capSynthAnswer now truncates each embedded answer to synthMaxAnswerBytes (12KB) with a byte-accurate "[truncated N bytes]" marker, so judge prompts stay small regardless of runtime verbosity. Under-limit answers are embedded byte-identical with no marker. Tests: TestCodexStartUnwrapsJSONTranscript (adapter unit: agent_message text vs transcript fallback, RuntimeRef preserved); TestRealSkillOptABDeliverUnwrapsCodex- TranscriptForSynth (consumer regression through the real delivery seam, codex flavor of #722's claude test); TestSynthJudgePromptCapsAnswers (cap + marker + verbatim-under-limit + exact-at-limit). Closes #724 Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Trace (issue #721)
First live
gitmoot skillopt synthrun (weak=codex, strong/challenger/judge=claude) rejected all 3 itemsbad_rubric, weak/strong scores exactly 0.0 every round, empty stderr — 9/9 silent failures.Root cause:
ClaudeAdapter.Startrunsclaude -p --output-format jsonand returnedStartResult{Raw: result.Stdout}— stdout is claude's CLI result envelope ({"type":"result",...,"result":"<answer>"}), not the answer text (internal/runtime/adapter.go:637-642).realSkillOptABDeliverreturnsstarted.Rawverbatim on the forked-session path (empty RuntimeRef → Start) (internal/cli/skillopt_ab.go:163).extractSynthJSONObjectthen grabs the FIRST balanced{...}= the envelope;parseSynthGeneratedItemfinds no context/question/rubric →bad_rubric,continue, no stderr. Judge verdicts fail identically.Contrast: kimi's Start already unwraps (
StartResult{Raw: content}, kimi.go:56); codex returns prose stdout. Claude was the only enveloped Start.Fix
At the adapter (right altitude).
ClaudeAdapter.Startnow reuses the existingparseClaudeJSONResulthelper — the same one the Deliver path uses to buildSummary— to extract the envelope'sresultfield intoRaw. Fail-open: falls back to raw stdout when stdout is not the envelope, has noresultfield, orresultis empty (e.g. an older CLI's plain-text--output-formatfallback), so unwrap never errors. No new parser was added.RuntimeRefbehavior is byte-identical. kimi/codex untouched (kimi already unwraps; codex is prose).Consumers audited (Start().Raw)
The only reader of
Start().RawisrealSkillOptABDeliver's forked-session path (skillopt_ab.go:163). Everything downstream routes through it:skillopt synth— challenger and judge (skillOptSynthDeliver = realSkillOptABDeliver). JSON-parsed → was the hard failure. Fixed.skillopt ab --judge(skillOptABJudgeDeliver, both single + jury paths). JSON-parsed. Fixed.skillopt abvariant deliver +skillopt binary. Answer used as prose; envelope leak was cosmetic/quality, now clean.Non-empty RuntimeRef paths already went through
Deliver→result.Summary(already unwrapped) and are unaffected.All other
adapter.Startcall sites —agent restart,agent_dispatchinstance start, daemon temp/ephemeral workers,skilloptagent start — read onlystarted.RuntimeRef, never.Raw. Unaffected by the unwrap.How tested (deterministic, no LLM)
TestClaudeStartUnwrapsJSONEnvelope, internal/runtime): fake subprocess runner returns a realistic claude JSON envelope →Start().Raw== innerresulttext; plain-answer envelope → the answer; non-JSON stdout → falls back to full stdout; emptyresult→ fallback; envelope withoutresult→ fallback.RuntimeRefasserted identical in every case.TestRealSkillOptABDeliverUnwrapsClaudeEnvelopeForSynth, internal/cli): drives the REALrealSkillOptABDeliver+ a realClaudeAdapterwired to a fake runner (viareplaceRuntimeFactory) on the forked-session path, then asserts the delivered answer is the unwrapped inner JSON and thatparseSynthGeneratedItemsucceeds on it — the exactbad_rubricparse that failed live.Gates (go1.26.4, GOTOOLCHAIN-pinned, pipefail + explicit exit checks)
go build ./...→ 0go vet ./internal/runtime/ ./internal/cli/→ 0go test -count=1 ./internal/runtime/→ok 0.084sgo test -count=1 ./internal/cli/→ok 208.926sgo test -race -count=1 ./internal/runtime/→ok 1.114sgofmt -lon all changed files → cleanCloses #721
🤖 Generated with Claude Code