Skip to content

fix(runtime): unwrap claude CLI JSON envelope in Start() Raw#722

Merged
jerryfane merged 1 commit into
mainfrom
fix/721-claude-start-envelope
Jul 7, 2026
Merged

fix(runtime): unwrap claude CLI JSON envelope in Start() Raw#722
jerryfane merged 1 commit into
mainfrom
fix/721-claude-start-envelope

Conversation

@jerryfane

Copy link
Copy Markdown
Owner

Trace (issue #721)

First live gitmoot skillopt synth run (weak=codex, strong/challenger/judge=claude) rejected all 3 items bad_rubric, weak/strong scores exactly 0.0 every round, empty stderr — 9/9 silent failures.

Root cause:

  • ClaudeAdapter.Start runs claude -p --output-format json and returned StartResult{Raw: result.Stdout} — stdout is claude's CLI result envelope ({"type":"result",...,"result":"<answer>"}), not the answer text (internal/runtime/adapter.go:637-642).
  • realSkillOptABDeliver returns started.Raw verbatim on the forked-session path (empty RuntimeRef → Start) (internal/cli/skillopt_ab.go:163).
  • extractSynthJSONObject then grabs the FIRST balanced {...} = the envelope; parseSynthGeneratedItem finds no context/question/rubric → bad_rubric, continue, no stderr. Judge verdicts fail identically.

Contrast: kimi's Start already unwraps (StartResult{Raw: content}, kimi.go:56); codex returns prose stdout. Claude was the only enveloped Start.

Fix

At the adapter (right altitude). ClaudeAdapter.Start now reuses the existing parseClaudeJSONResult helper — the same one the Deliver path uses to build Summary — to extract the envelope's result field into Raw. Fail-open: falls back to raw stdout when stdout is not the envelope, has no result field, or result is empty (e.g. an older CLI's plain-text --output-format fallback), so unwrap never errors. No new parser was added. RuntimeRef behavior is byte-identical. kimi/codex untouched (kimi already unwraps; codex is prose).

Consumers audited (Start().Raw)

The only reader of Start().Raw is realSkillOptABDeliver's forked-session path (skillopt_ab.go:163). Everything downstream routes through it:

  • skillopt synth — challenger and judge (skillOptSynthDeliver = realSkillOptABDeliver). JSON-parsed → was the hard failure. Fixed.
  • skillopt ab --judge (skillOptABJudgeDeliver, both single + jury paths). JSON-parsed. Fixed.
  • skillopt ab variant deliver + skillopt binary. Answer used as prose; envelope leak was cosmetic/quality, now clean.
  • live-AB challenger (agent_dispatch_live_ab.go) — presented to the user for a pick as prose (not JSON-parsed); previously showed the raw envelope, now shows the answer.

Non-empty RuntimeRef paths already went through Deliverresult.Summary (already unwrapped) and are unaffected.

All other adapter.Start call sitesagent restart, agent_dispatch instance start, daemon temp/ephemeral workers, skillopt agent start — read only started.RuntimeRef, never .Raw. Unaffected by the unwrap.

How tested (deterministic, no LLM)

  1. Unit (TestClaudeStartUnwrapsJSONEnvelope, internal/runtime): fake subprocess runner returns a realistic claude JSON envelope → Start().Raw == inner result text; plain-answer envelope → the answer; non-JSON stdout → falls back to full stdout; empty result → fallback; envelope without result → fallback. RuntimeRef asserted identical in every case.
  2. Consumer regression (TestRealSkillOptABDeliverUnwrapsClaudeEnvelopeForSynth, internal/cli): drives the REAL realSkillOptABDeliver + a real ClaudeAdapter wired to a fake runner (via replaceRuntimeFactory) on the forked-session path, then asserts the delivered answer is the unwrapped inner JSON and that parseSynthGeneratedItem succeeds on it — the exact bad_rubric parse that failed live.
  3. Both tests were verified to fail on the pre-fix code (leaked envelope) and pass after.

Gates (go1.26.4, GOTOOLCHAIN-pinned, pipefail + explicit exit checks)

  • go build ./... → 0
  • go vet ./internal/runtime/ ./internal/cli/ → 0
  • go test -count=1 ./internal/runtime/ok 0.084s
  • go test -count=1 ./internal/cli/ok 208.926s
  • go test -race -count=1 ./internal/runtime/ok 1.114s
  • gofmt -l on all changed files → clean

Closes #721

🤖 Generated with Claude Code

ClaudeAdapter.Start ran `claude -p --output-format json` and returned
StartResult{Raw: result.Stdout} verbatim — the whole CLI result envelope
({"type":"result",...,"result":"<answer>"}), not the assistant's answer.

The sole consumer of Start().Raw is realSkillOptABDeliver's forked-session
path (empty RuntimeRef → Start), which feeds skillopt synth (challenger +
judge), skillopt ab --judge, skillopt binary, and the live-AB challenger.
JSON consumers (synth/ab) then parsed the envelope's outer object, found no
context/question/rubric, and silently rejected every item (bad_rubric, zero
scores). kimi already unwraps at kimi.go:56; codex returns prose. Claude was
the only enveloped Start.

Fix at the adapter (right altitude): reuse the existing parseClaudeJSONResult
helper (the same one the Deliver path uses to build Summary) to extract the
envelope's "result" into Raw. Fail-open — fall back to raw stdout when stdout
is not the envelope or has no result field, so unwrap never errors.
RuntimeRef behavior is unchanged. All other Start() call sites read only
RuntimeRef, so they are unaffected.

Closes #721

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@jerryfane jerryfane merged commit b53269f into main Jul 7, 2026
1 check passed
@jerryfane jerryfane deleted the fix/721-claude-start-envelope branch July 7, 2026 14:35
jerryfane added a commit that referenced this pull request Jul 7, 2026
… judge answers (#724) (#726)

Two composing halves of the same live `skillopt synth` failure (2026-07-08):

1. codex Start().Raw was the full `codex exec --json` transcript (thread.started
   banner, turn events, reasoning items), not the assistant's answer — the codex
   flavor of the claude bug fixed in #722 (b53269f). Forked-session consumers
   (skillopt synth/ab) that parse Start().Raw as the assistant's answer saw the
   whole transcript and could not find the challenger item's context/question/
   rubric. Fix at the adapter (same altitude as #722): reuse parseCodexJSONResult
   — the exact parser the Deliver path uses to build Summary — to surface the
   joined agent_message text as Raw, failing open to the raw stdout when the
   stream carries no agent_message (older CLI, plain-text fallback, unexpected
   shape). RuntimeRef is untouched.

2. synthJudgePrompt embedded both weak/strong answers verbatim with no cap. Even
   after the unwrap a genuinely long answer, combined with both halves embedded,
   blew ARG_MAX on the kimi judge exec. capSynthAnswer now truncates each embedded
   answer to synthMaxAnswerBytes (12KB) with a byte-accurate "[truncated N bytes]"
   marker, so judge prompts stay small regardless of runtime verbosity. Under-limit
   answers are embedded byte-identical with no marker.

Tests: TestCodexStartUnwrapsJSONTranscript (adapter unit: agent_message text vs
transcript fallback, RuntimeRef preserved); TestRealSkillOptABDeliverUnwrapsCodex-
TranscriptForSynth (consumer regression through the real delivery seam, codex
flavor of #722's claude test); TestSynthJudgePromptCapsAnswers (cap + marker +
verbatim-under-limit + exact-at-limit).

Closes #724

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

claude Start() leaks the CLI JSON envelope as Raw — silently breaks skillopt synth/ab JSON consumers (bad_rubric, zero scores)

1 participant