Skip to content

fix(seeds): enforce minItems in structured-output schema to prevent short-count failures - #40

Merged
jakepresent merged 2 commits into
mainfrom
jakepresent/fix-seeds-minitems
May 12, 2026
Merged

fix(seeds): enforce minItems in structured-output schema to prevent short-count failures#40
jakepresent merged 2 commits into
mainfrom
jakepresent/fix-seeds-minitems

Conversation

@jakepresent

Copy link
Copy Markdown
Collaborator

Problem

The seeds stage aborts when the model returns fewer items than requested per-cell batch, even by one. With larger sample_size values, this is statistically inevitable and the pipeline cannot proceed past seeds.

Repro (clean main, fresh artifacts dir):

# eval config based on examples.travel_planner_langgraph
seeds:
  prompt:
    model: { name: azure/gpt-5.4-mini, temperature: 0.7 }
    sample_size: 500  # 18-cell design grid → ~27-28 seeds per cell

Three consecutive runs failed at the same stage with the same shape:

ValueError: prompt generation returned 26 seeds for a batch of 27
ValueError: prompt generation returned 22 seeds for a batch of 23
ValueError: prompt generation returned 25 seeds for a batch of 26

Bumping max_tokens 5× (3000 → 16000) did not help, confirming this is not output truncation. The model was simply emitting one fewer item than the prompt asked for.

Root cause

seeds_response_schema() was generating a schema with no minItems constraint on the seeds array:

"seeds": {"type": "array", "maxItems": 2000, "items": item_schema}

The prompt asked for N seeds in natural language, but the schema told the model "0 to 2000 is fine." When the structured-output layer enforced the schema, returning N-1 was a valid response. gpt-5.4-mini did this consistently at batch sizes above ~10-15 per cell.

The strict equality check at seeds.py:668 then aborted the entire stage on the first short response:

if len(returned_seeds) < job.count:
    raise ValueError(...)

Fix

Thread min_items through seeds_response_schema() and pin it to job.count per call. The schema now enforces the count the prompt asked for, eliminating the under-generation at the structured-output layer instead of catching it after the fact.

This mirrors the existing pattern in _labels_response_schema(), which already pins both minItems and maxItems to count - so the precedent in this codebase is to use schema-level enforcement for exact-count structured outputs.

Validation

  • Repro: confirmed broken on main. Three runs of sample_size=500 on examples.travel_planner_langgraph failed at seeds with the short-count error.
  • Repro: confirmed fixed on this branch. Same config now generates the full 502 test cases (500 prompts + 2 scenarios) and the pipeline proceeds to rollout. Run is still in progress at the time of this PR; will paste final perf numbers in a follow-up comment.
  • Unit tests: 4 new tests in SeedsResponseSchemaTest covering default behavior, min_items enforcement, and zero/negative-value handling. Full seed-related suite (105 tests) passes.

Why the strict check stays

I considered loosening the worker-layer check to accept-and-warn instead of raising on short count. Decided against it because:

  1. The schema fix removes the failure mode at its source rather than papering over it downstream.
  2. The strict check is still useful as a safety net - if the model violates the now-enforced minItems constraint somehow, we want to know rather than silently produce undersized batches.
  3. Other stages (notably _labels_response_schema) already use the same pin-via-schema pattern, so this keeps the codebase consistent.

If there's appetite for a separate retry-on-short-count or accept-and-warn fallback at the worker layer, happy to do that in a follow-up.

…hort-count failures

When seeds.sample_size produces per-cell batches larger than ~10-15,
gpt-5.4-mini frequently returns N-1 items because the JSON schema
allowed 0-2000 items regardless of the prompt's stated count. The
strict equality check at the worker layer then aborted the entire
seeds stage on the first short response, killing the pipeline.

Repro: travel-planner-langgraph eval config with sample_size=500
(18-cell design grid → 27-28 seeds per cell) fails at the seeds stage
on three consecutive runs with errors like:
  ValueError: prompt generation returned 26 seeds for a batch of 27
  ValueError: prompt generation returned 22 seeds for a batch of 23
  ValueError: prompt generation returned 25 seeds for a batch of 26

Bumping max_tokens 5x did not help, confirming this isn't output
truncation but model-side under-generation against an unconstrained
schema.

Fix: thread min_items through seeds_response_schema() and pin it to
job.count per call. Mirrors the existing pattern in
_labels_response_schema() which already pins both minItems and
maxItems to count.

Validated end-to-end with sample_size=500 on
examples.travel_planner_langgraph - seeds stage now generates the
full 502 test cases and pipeline proceeds to rollout cleanly.

Tests: 4 new schema-shape tests + full seeds suite (105 tests) passes.
@tangym

tangym commented May 11, 2026

Copy link
Copy Markdown
Collaborator

With minItems now pinned to job.count but maxItems still at 2000, could the model over-generate? Probably unlikely, but curious.

Address Yeming's review feedback on #40. The minItems pin alone left
the schema asymmetric (e.g. minItems=27, maxItems=2000), which
technically allows the model to over-generate. Over-generation isn't
the failure mode that motivated the PR, but pinning both bounds keeps
the schema's stated shape aligned with the prompt's stated count.

- seeds_response_schema now accepts an optional max_items override; the
  default behavior (maxItems=2000) is preserved when callers don't pass
  one, matching the existing seed-validation safety net.
- _generate_records now pins both bounds to job.count so each batch
  produces exactly the requested number of seeds.
- Two new unit tests cover the pinned-bounds case and the non-positive
  override fallback. 30/30 seed-schema tests pass.
@jakepresent

Copy link
Copy Markdown
Collaborator Author

Good catch @tangym - pushed a4169e4 that pins both bounds to job.count. Now the schema demands exactly N items per batch, matching the prompt's stated count.

Added two tests for the symmetric-bounds case and the non-positive override fallback. 30/30 seed-schema tests pass.

@jakepresent
jakepresent merged commit f431331 into main May 12, 2026
3 checks passed
changliu2 added a commit that referenced this pull request May 12, 2026
Absorb PR #40 (seeds.py minItems schema fix). Conflict in
p2m/stages/seeds.py at the per-job schema invocation: kept HEAD's
failure_mode rename, took main's new job_schema = seeds_response_schema(
tool_source, min_items=job.count, max_items=job.count) line.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
changliu2 added a commit that referenced this pull request May 12, 2026
Catch up the viewer-fix branch with main (17 commits behind, including
PR #32 artifact-cache + PR #34 exception handling + PR #40 seeds
minItems schema fix).

Conflicts:
  - p2m/viewer_read_model.py: kept HEAD's bumped SCHEMA_VERSION = 2
    + GENERATOR_VERSION = "viewer-read-model-v2", added main's
    log = logging.getLogger(__name__).
  - viewer/src/lib/server/artifacts.ts: same (kept v2 + added main's
    SUITE_ARTIFACTS_DIR = 'artifacts').

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AaronAspinwall123 added a commit that referenced this pull request May 14, 2026
Conflict in p2m/stages/seeds.py: combined main's per-job `seeds_response_schema(min_items=count, max_items=count)` (PR #40) with our per-batch try/except resilience wrapper. Each batch now (a) gets a schema that pins both bounds to the requested count, and (b) is wrapped in a per-batch error sentinel so a single bad payload no longer kills the stage.

Tests: 666 passed; 4 pre-existing Windows flakes unrelated to the merge.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants