fix(seeds): enforce minItems in structured-output schema to prevent short-count failures - #40
Merged
Merged
Conversation
…hort-count failures When seeds.sample_size produces per-cell batches larger than ~10-15, gpt-5.4-mini frequently returns N-1 items because the JSON schema allowed 0-2000 items regardless of the prompt's stated count. The strict equality check at the worker layer then aborted the entire seeds stage on the first short response, killing the pipeline. Repro: travel-planner-langgraph eval config with sample_size=500 (18-cell design grid → 27-28 seeds per cell) fails at the seeds stage on three consecutive runs with errors like: ValueError: prompt generation returned 26 seeds for a batch of 27 ValueError: prompt generation returned 22 seeds for a batch of 23 ValueError: prompt generation returned 25 seeds for a batch of 26 Bumping max_tokens 5x did not help, confirming this isn't output truncation but model-side under-generation against an unconstrained schema. Fix: thread min_items through seeds_response_schema() and pin it to job.count per call. Mirrors the existing pattern in _labels_response_schema() which already pins both minItems and maxItems to count. Validated end-to-end with sample_size=500 on examples.travel_planner_langgraph - seeds stage now generates the full 502 test cases and pipeline proceeds to rollout cleanly. Tests: 4 new schema-shape tests + full seeds suite (105 tests) passes.
Collaborator
|
With |
tangym
approved these changes
May 11, 2026
Address Yeming's review feedback on #40. The minItems pin alone left the schema asymmetric (e.g. minItems=27, maxItems=2000), which technically allows the model to over-generate. Over-generation isn't the failure mode that motivated the PR, but pinning both bounds keeps the schema's stated shape aligned with the prompt's stated count. - seeds_response_schema now accepts an optional max_items override; the default behavior (maxItems=2000) is preserved when callers don't pass one, matching the existing seed-validation safety net. - _generate_records now pins both bounds to job.count so each batch produces exactly the requested number of seeds. - Two new unit tests cover the pinned-bounds case and the non-positive override fallback. 30/30 seed-schema tests pass.
Collaborator
Author
tangym
approved these changes
May 12, 2026
changliu2
added a commit
that referenced
this pull request
May 12, 2026
Absorb PR #40 (seeds.py minItems schema fix). Conflict in p2m/stages/seeds.py at the per-job schema invocation: kept HEAD's failure_mode rename, took main's new job_schema = seeds_response_schema( tool_source, min_items=job.count, max_items=job.count) line. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
changliu2
added a commit
that referenced
this pull request
May 12, 2026
Catch up the viewer-fix branch with main (17 commits behind, including PR #32 artifact-cache + PR #34 exception handling + PR #40 seeds minItems schema fix). Conflicts: - p2m/viewer_read_model.py: kept HEAD's bumped SCHEMA_VERSION = 2 + GENERATOR_VERSION = "viewer-read-model-v2", added main's log = logging.getLogger(__name__). - viewer/src/lib/server/artifacts.ts: same (kept v2 + added main's SUITE_ARTIFACTS_DIR = 'artifacts'). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AaronAspinwall123
added a commit
that referenced
this pull request
May 14, 2026
Conflict in p2m/stages/seeds.py: combined main's per-job `seeds_response_schema(min_items=count, max_items=count)` (PR #40) with our per-batch try/except resilience wrapper. Each batch now (a) gets a schema that pins both bounds to the requested count, and (b) is wrapped in a per-batch error sentinel so a single bad payload no longer kills the stage. Tests: 666 passed; 4 pre-existing Windows flakes unrelated to the merge. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The seeds stage aborts when the model returns fewer items than requested per-cell batch, even by one. With larger
sample_sizevalues, this is statistically inevitable and the pipeline cannot proceed past seeds.Repro (clean main, fresh artifacts dir):
Three consecutive runs failed at the same stage with the same shape:
Bumping
max_tokens5× (3000 → 16000) did not help, confirming this is not output truncation. The model was simply emitting one fewer item than the prompt asked for.Root cause
seeds_response_schema()was generating a schema with nominItemsconstraint on the seeds array:The prompt asked for N seeds in natural language, but the schema told the model "0 to 2000 is fine." When the structured-output layer enforced the schema, returning N-1 was a valid response.
gpt-5.4-minidid this consistently at batch sizes above ~10-15 per cell.The strict equality check at
seeds.py:668then aborted the entire stage on the first short response:Fix
Thread
min_itemsthroughseeds_response_schema()and pin it tojob.countper call. The schema now enforces the count the prompt asked for, eliminating the under-generation at the structured-output layer instead of catching it after the fact.This mirrors the existing pattern in
_labels_response_schema(), which already pins bothminItemsandmaxItemsto count - so the precedent in this codebase is to use schema-level enforcement for exact-count structured outputs.Validation
sample_size=500onexamples.travel_planner_langgraphfailed at seeds with the short-count error.SeedsResponseSchemaTestcovering default behavior,min_itemsenforcement, and zero/negative-value handling. Full seed-related suite (105 tests) passes.Why the strict check stays
I considered loosening the worker-layer check to accept-and-warn instead of raising on short count. Decided against it because:
minItemsconstraint somehow, we want to know rather than silently produce undersized batches._labels_response_schema) already use the same pin-via-schema pattern, so this keeps the codebase consistent.If there's appetite for a separate retry-on-short-count or accept-and-warn fallback at the worker layer, happy to do that in a follow-up.