Rubric Grader & Structured Output Validation for Local / Open-Weights Models

[nyarlathotep77]

Hi everyone,

I've been doing extensive testing of deepagents (using dcode) with local open-weights models via Ollama/llama.cpp, utilizing models like qwen3.8-coder and llama3.x.

I am seeing high crash and session deadlock rates when running rubric self-evaluation loops (RubricMiddleware / /goal grading) on local setups.

The Problem: Brittle Structured Output Handshakes
In deepagents, the rubric grader depends on strict Pydantic parsing (GraderResponse). On local hardware, even advanced models frequently fail this handshake due to formatting anomalies. When a validation failure occurs, the entire agent session crashes with a StructuredOutputValidationError.

Here are the three most common failure modes I've observed:

Schema Mismatches / Bad Parameter Types: Local models frequently output valid JSON but mismatch the nested schemas Pydantic expects. For example, returning a simple list of strings instead of a list of criterion dictionaries:

{
  "result": "needs_revision",
  "explanation": "Tests failed.",
  "criteria": ["Verify file exists"] // <-- Array of strings instead of List[dict]
}

Markdown Fences & Conversational Prose: Despite system instructions, local models often wrap their structured tool calls in markdown blocks or prefix them with conversational fillers (e.g. "Here is your JSON response..."), causing simple string parses to fail.

Missing Tool Trigger: The model responds with a clean JSON body in the text content but fails to trigger the response_format or native tool call metadata, causing structured_response to resolve to None.

Why This is Critical for Phase-Gate Middlewares?
If these grading loops are brittle, local model workflows become practically unusable because a single formatting hiccup crashes a long-running multi-turn agent run.
While trying to find if there's any best practice I've missed out to adopt in my local setup, I've implemented some local fallbacks which catch validation errors, extract JSON from raw message content and fall back to text-only mode calls if needed. At least I've resolved those crashes in my setup for the time being.

Other structured output libraries (like Pydantic AI and Instructor) handle this by implementing client-side "validation rescue" and text-fallback parsers rather than treating a validation error as a fatal crash.

Has anyone else encountered these rubric/grader validation crashes when running local models?
What is the preferred architectural pattern in deepagents to introduce this type of resilience (e.g., handling it inside RubricMiddleware, at the model resolution layer, or via self-correcting retry prompts)?
Id love to hear your thoughts and align on the best path forward!

[Hamed Rabah (hamedrabah)]

The recovery boundary should be inside RubricMiddleware (or a reusable structured-output wrapper it calls), not around the entire agent session. A grading-format failure is a recoverable model I/O failure; letting it escape the phase gate turns one bad serialization into loss of a long-running run.

A robust sequence is:

1. Validate the native tool/structured response.
2. On validation failure, retry with the validation errors and the exact schema, with a small bounded retry count.
3. If the model returned JSON as text, optionally extract and validate it as a compatibility fallback.
4. If it still fails, return an explicit unable_to_grade result and preserve the checkpoint instead of deadlocking/crashing the session.

The model adapter should still advertise the strategy it actually supports: use tool-based structured output for local models with reliable tool calling, and JSON/text parsing otherwise. But the rubric layer owns the semantic fallback because it knows whether failure should mean retry, needs_revision, or “grader unavailable.”

I would avoid silently coercing list[str] into list[Criterion]; that can hide a genuinely incorrect grade. Bounded repair plus a typed failure state is safer and observable.

[Nima Simyari (NimaSimyari)]

I think this is a real weak spot with local models rather than something wrong with your setup.

Looking at the current RubricMiddleware, a malformed or missing structured_response eventually becomes a grader_error, which is terminal. That seems a bit harsh for local models where the content can be correct but the formatting is slightly off.

I’d probably keep the recovery inside the grader/rubric layer, but make it bounded: on a validation error, retry once with the actual Pydantic error + schema. If the model returned JSON in plain text, try parsing and validating that before giving up.

I would avoid automatically converting things like list[str] into the expected criterion objects though. At that point it’s no longer just fixing serialization — it could change the meaning of the grade.

There’s also #5030 discussing reliability hooks/retries around RubricMiddleware, although that seems focused more on transport failures right now. Extending the same idea to structured-output validation would make local-model use much less brittle.

[Rafay M. (rafayaar)]

I’d separate recoverable formatting failures from actual schema/semantic failures.

For example, JSON returned in message.content, Markdown fences, or extra prose can probably be safely normalized and re-validated. But if the model returns the wrong structure, such as list[str] where List[Criterion] is expected, I’d treat that as a real grading failure rather than trying to coerce it.

That distinction could also make failures easier to observe/debug: format_recovered, validation_failed, grader_unavailable, etc.

So +1 to keeping recovery close to RubricMiddleware, but I think explicitly classifying the failure type would make the behavior safer and easier to reason about.

[Riconas Sun (RiconasSun)]

Because this Discussion predates the current RubricMiddleware, I tested current main at b643f6f34be0142fc3e33cd7402f171e62e69cc3 using deterministic scripted models through the real nested grader path. No model APIs or production-source changes were used.

The sync/async matrix showed:

Grader output	ToolStrategy	ProviderStrategy
Valid structured response	Completes	Completes
Permanently invalid schema	Validation feedback + retries	grader_error after one call
Valid JSON returned as ordinary text	Repeated missing-trigger behavior	Completes
Fenced/prefixed valid JSON	Same missing-trigger behavior	grader_error
First invalid, then valid	Recovers in two calls	Stops after first failure

The main finding is that current ToolStrategy already recovers from repairable schema-validation failures.

The remaining gap is different: when a model fails to use the expected structured-output mechanism, ToolStrategy can continue retrying for a long time. In an untouched-default probe I observed 64 grader responses before stopping the harness; this does not establish infinite execution. Measured failures preserved the parent answer and checkpoint, and I did not reproduce a caller crash.

I also prototyped an external recovery layer using the existing grader_middleware / model-call hooks. It:

• recovers unambiguous fenced or prefixed valid JSON;
• validates recovered data with the real GraderResponse;
• does not coerce schema-invalid values;
• bounds permanent failures to three attempts;
• preserves parent output/state on terminal grader_error.

So I would currently keep this opt-in and application-owned, implemented as grader middleware using existing public hooks. I did not find a need for a new RubricMiddleware hook in these cases.

Live model adapters and graders with ordinary tools still need validation.

If this boundary looks useful, I'm happy to turn the relevant cases into focused regression tests or a small public-API example before proposing any framework change.