copilot provider: schema-shape failures bypass the parse-recovery loop and kill the workflow (hermes recovers from the identical response)

[franklixuefei]

Summary

When an agent returns syntactically valid JSON whose fields have the wrong shape, the copilot provider fails the entire workflow immediately with zero recovery attempts — even though the provider has a fully-built parse-recovery mechanism (max_parse_recovery_attempts, default 5) that is designed for exactly this class of "the model didn't follow the output contract" failure.

The cause is that the copilot provider never calls validate_output() inside its recovery loop. Schema validation happens one layer up, in executor/agent.py, after the provider has already returned — so the ValidationError is raised outside the loop that could have re-prompted the model.

The hermes provider does this correctly. copilot — the default provider — does not.

Impact

A ~3.5 hour, ~$2.30 multi-agent run died at the final agent, after all prior agents had succeeded and a git commit had already been made. The failure was a single field on a single response, recoverable by one re-prompt that the codebase already knows how to construct (_build_parse_recovery_prompt).

Workflow Failed
Output field 'decision' has wrong type: expected string, got dict
💡 Suggestion: Ensure agent returns correct type for 'decision'

The agent declared:

output:
  decision:
    type: string
    description: Review decision - APPROVE or REQUEST_CHANGES

…and the model returned decision as a nested object rather than the bare string. Everything else about the response was well-formed JSON.

This is materially worse than a parse failure, because it tends to surface with non-Anthropic models whose structured-output conventions differ, and it strands work that is otherwise complete.

Environment

Conductor	v0.1.26 (latest release; also verified against main)
Provider	copilot
Affected agent model	gpt-5.6-sol (reasoning: xhigh, context_tier: long_context)
OS	Windows 11, PowerShell 7

Reproduced with a workflow whose other five agents ran on claude-opus-5 / claude-sonnet-5 without incident — only the agent on a non-Anthropic model tripped it.

Root cause

Verified against main (paths below are current):

1. Validation lives outside the provider. src/conductor/executor/agent.py:318

# Validate output against schema (skip for partial output from interrupts)
if agent.output and not output.partial:
    validate_output(output.content, agent.output)

2. The copilot provider never validates in-session. src/conductor/providers/copilot.py contains zero occurrences of validate_output. Its recovery loop catches syntax errors only:

# providers/copilot.py:1202
except (json.JSONDecodeError, ValueError) as e:

So max_parse_recovery_attempts (default 5 for copilot) is unreachable for schema-shape failures.

3. validate_output is strict, with no coercion. src/conductor/executor/output.py:47

if not _check_type(value, expected_type):
    raise ValidationError(
        f"Output field '{field_name}' has wrong type: "
        f"expected {expected_type}, got {type(value).__name__}",
        ...
    )

The providers disagree with each other

Provider	Validates inside recovery loop?	Catches ValidationError?	Result on wrong-shaped field
hermes	✅ hermes.py:524	✅ hermes.py:526	Recovers (up to 3 attempts)
claude	✅ claude.py:1239	❌ claude.py:1271 — "Re-raise ValidationError without wrapping (non-retryable)"	Fails fast — an explicit, documented choice
copilot	❌ never calls it	❌ n/a	Fails, and cannot even try

hermes already contains the intended shape:

# providers/hermes.py:521-526
for attempt in range(_MAX_PARSE_RECOVERY_ATTEMPTS + 1):
    try:
        content = parse_json_output(response)
        validate_output(content, agent.output)
        return content
    except (json.JSONDecodeError, ValueError, ValidationError) as e:

The asymmetry looks unintentional rather than designed: copilot is the default provider and carries the largest recovery budget (5, versus hermes' 3), yet derives no benefit from it for this failure class.

Proposed fix

Mirror hermes in the copilot provider — validate inside the recovery loop and include ValidationError in the caught exceptions, so a wrong-shaped field is re-prompted like any other contract violation:

for recovery_attempt in range(max_recovery + 1):
    try:
        parsed_content = self._extract_json(response_content)
        if schema_for_prompt is not None:
            validate_output(parsed_content, schema_for_prompt)   # NEW
        return parsed_content, final_usage
    except (json.JSONDecodeError, ValueError, ValidationError) as e:   # ValidationError NEW
        ...

_build_parse_recovery_prompt(parse_error=..., original_response=..., schema=...) already takes the schema, so the recovery prompt would carry the exact type error and the expected shape without further changes.

Two smaller improvements worth considering alongside it:

• Unwrap the single-key case before failing. A very common model behaviour is emitting {"decision": {"decision": "APPROVE", "reasoning": "..."}} or {"decision": {"value": "APPROVE"}}. When a string is expected and a dict is received, unwrapping a sole string-valued key (or a value/matching-field-name key) would resolve most occurrences without a round-trip. This should be conservative and only applied where unambiguous.
• Surface the offending value in the error. The message names the field and the two types but not the received value, so diagnosing from logs alone requires re-running with verbose output.

Reproduction

1. Define an agent with output.decision.type: string, prompted with something like "Provide your decision: APPROVE or REQUEST_CHANGES".
2. Run it on provider: copilot with a non-Anthropic model (gpt-5.6-sol here).
3. When the model wraps the value in an object, the workflow terminates immediately — no entry in the log shows a recovery attempt, and max_parse_recovery_attempts has no effect.

Running the same workflow with provider: hermes recovers from the identical response.

Note

This is unrelated to #342 (Windows UnicodeEncodeError truncating the --silent result after a successful run). That one crashes on output serialization once the workflow has completed; this one fails the workflow itself, mid-run, and leaves the final agent without a result.

[jrob5756]

Thanks for this one. The diagnosis holds up: I checked every claim against main (ff7ed5a) and they all land. copilot.py has zero occurrences of validate_output, its recovery loop catches (json.JSONDecodeError, ValueError) only, and hermes.py:521-526 does exactly what you describe.

A few things came out of verifying it, two of which change the fix.

The proposed snippet won't work as written. validate_output(parsed_content, schema_for_prompt) would raise AttributeError on field_def.type, because schema_for_prompt is self._build_prompt_schema(output_schema) (copilot.py:955), a prompt-facing dict of type/description strings rather than dict[str, OutputField]. The argument you want is output_schema from agent.effective_output_schema(), which is already in scope right there in the loop.

The fix also has to live in the provider for a stronger reason than the one in the issue. copilot's finally: await session.disconnect() runs before the provider returns, so by the time executor/agent.py:318 raises, the session is already gone. The executor layer couldn't re-prompt even if you wanted it to.

One correction to the table: claude has the same gap. The except ValidationError: raise at claude.py:1271 is scoped to the network-retry loop and means "don't burn API retries on a deterministic error", which is sensible and a separate concern. But claude's actual parse-recovery loop is _execute_with_parse_recovery (claude.py:2035, default 2 attempts), and it returns the moment _extract_structured_output or _extract_json_fallback yields anything at all. It never checks shape either. Same blind spot as copilot, just with a smaller budget, so I'm fixing both in one PR.

One deliberate divergence from hermes. When the budget runs out I'm re-raising the original ValidationError instead of hermes' generic ProviderError("Failed to parse structured output after N attempts"). You get the recovery attempts and you still get Output field 'decision' has wrong type: expected string, got dict. Hermes throws that detail away today, and I'd rather not copy that part.

Taking both of your smaller suggestions too. The unwrap will sit provider-side as a normalization pass immediately before validation rather than inside validate_output, since that function also runs for set and script step output (engine/workflow.py:1077 and :1153) where silently reshaping a value would be a nasty surprise. Worth flagging that your own example, {"decision": {"decision": "APPROVE", "reasoning": "..."}}, has two keys, so a sole-key rule would miss it. Matching on the field name first is what catches that one.

Starting on it now.