fix(synthesis): tolerate string list fields in map-reduce partials; synthesis failure no longer invalidates a complete run#588
Conversation
…rade gracefully on synthesis failure
Two defects observed on a real run (2026-07-19):
1. PartialSummary rejected map partials whose list fields came back as
newline-bulleted strings ("\n- A\n- B") — drift the single-pass
synthesis path already tolerates. The model now coerces a string
into a list via a before-mode field validator (bullet/enumeration
markers stripped), and synthesize_panel_mapreduce writes the coerced
lists back onto the map result before reduce.
2. A synthesis-stage failure marked an otherwise-complete run invalid
(run_invalid=true, message="PANEL RUN INVALID", exit 2) even though
every panelist response completed and was present in the output.
Synthesis failure now degrades gracefully across CLI/SDK/MCP:
synthesis stays null (or the fallback payload), the structured
synthesis_error is kept, a synthesis_failed warning is surfaced, and
the run exits 0 with run_invalid=false. Panelist-level failures
(failure rate, total failure, cost gate, SIGINT, missing-input) still
invalidate. `panel synthesize` keeps exit 2 — synthesis is that
command's sole deliverable.
Regression tests: string-valued list fields parse through the map
boundary end-to-end; synthesis exception / pre-flight overflow / chunk
overflow / judge fallback no longer invalidate a complete run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live-run divergence warning (2026-07-19) flagged provider-reported cost ~33% above the local estimate. The OR feed confirms the blended rate moved from $0.10/$0.32 (2026-05-10 snapshot) to $0.13/$0.40. gpt-4o-mini was flagged in the same log (-26%) but its table entry exactly matches both the OR feed and OpenAI's published rates today — that divergence reflected downstream provider routing on the run, not a stale entry, so it is left unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 23c3cbf459
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # structured output) at the top level so MCP consumers don't have to | ||
| # walk into the nested synthesis dict to find them. | ||
| synthesis_warnings = list(synthesis_dict.get("warnings") or []) if isinstance(synthesis_dict, dict) else [] | ||
| if isinstance(synthesis_error, dict): |
There was a problem hiding this comment.
When run_panel_sync returns a normal SynthesisResult(is_fallback=True) after exhausting schema retries, its dictionary contains is_fallback and error but no nested synthesis_error. Consequently this new warning block never runs and the SDK returns neither a top-level synthesis_error nor synthesis_failed warning; _run_panel_async in the MCP server uses the same nested-key-only test. CLI explicitly converts this case into a structured synthesis_fallback error, so derive the same payload from is_fallback here and in MCP to avoid presenting an incomplete synthesis as healthy to programmatic callers.
Useful? React with 👍 / 👎.
| # succeeded, hiding the failure behind mostly-empty fields. Surface a | ||
| # loud warning + a machine-detectable error payload and mark the run | ||
| # invalid, mirroring the sp-avmm hard-fail on synthesis API errors. | ||
| # loud warning + a machine-detectable error payload. |
There was a problem hiding this comment.
When the judge returns is_fallback=True in the default text-output path, this branch now permits exit 0 but leaves synthesis_dict truthy. The run summary later at lines 3172-3173 therefore prints Synthesis: completed even though this branch just reported it as incomplete; with the invalid-run banner and nonzero exit removed, the final status is especially likely to mislead users. Select the text status using synthesis_error_payload or is_fallback before treating a synthesis dictionary as completed.
Useful? React with 👍 / 👎.
Resolutions: took main's graceful-degradation semantics for tests/test_synthesis_failure.py with renamed module paths; renamed merge-introduced synth_panel/synthpanel references in the #588 test, docs hint, and CLI hint; noqa'd the intentional en-dash bullet regex (RUF001 — newer ruff flags it; heals main's currently-red lint). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 participants
Summary
Fixes two defects observed on a real run (2026-07-19), plus a pricing refresh flagged in the same log.
1. Map-reduce typed validation rejected newline-bulleted strings
The map model sometimes emits
themes/agreements/disagreements/surprisesas one newline-bulleted string ("\n- A\n- B") instead of a JSON array — drift the single-pass synthesis path already tolerates.PartialSummarynow coerces a string into a list via a before-mode field validator (bullet/enumeration markers stripped), andsynthesize_panel_mapreducewrites the coerced lists back onto each map result before reduce.2. Synthesis-stage failure invalidated an otherwise-complete run
A synthesis failure set
run_invalid: true+message="PANEL RUN INVALID"+ exit 2 even though every panelist response completed and was present in the output. Synthesis failure now degrades gracefully across CLI/SDK/MCP:synthesis: null(or the fallback payload), structuredsynthesis_errorkept, asynthesis_failedwarning surfaced,run_invalid: false, exit 0. Panelist-level invalidity (failure rate, total failure, cost gate, SIGINT, missing-input refusals) still exits 2, andpanel synthesizekeeps exit 2 — synthesis is that command's sole deliverable. Docs updated (docs/production-operations.mdexit-code table).3. Pricing refresh (separate commit)
llama-3.3-70bwas stale vs the OpenRouter feed ($0.10/$0.32 → $0.13/$0.40, matching the +33% divergence warning).gpt-4o-miniwas flagged too (-26%) but its entry exactly matches the current feed and OpenAI published rates — that divergence was downstream provider routing, so it's unchanged.Tests
synthesize_panel_mapreduce); synthesis exception / pre-flight overflow / chunk overflow / judge fallback no longer invalidate a complete run.test_mcp_stdio_sampling,test_seed, onetest_structured_output) fail identically on clean HEAD in this environment (pre-existing).Labeled
semver:minorsince the exit-code/run_invalidenvelope semantics are a documented behavioral contract change.🤖 Generated with Claude Code