rollout/judge: per-seed error isolation with batch-level error-rate threshold - #44
rollout/judge: per-seed error isolation with batch-level error-rate threshold#44jakepresent wants to merge 5 commits into
Conversation
…ing batch A single target-side LLMInputError (e.g. Azure content filter rejecting an adversarial prompt) was killing the entire rollout stage. The overnight scale-500 run on May 11 hit this exactly: seed 501 of 502 was refused by Azure CF, and 500 already-completed transcripts were discarded because the worker re-raised LLMInputError and the gather loop fails the stage on the first exception. This is the wrong failure mode for an adversarial-evaluation pipeline: generating prompts the target refuses is exactly what we're supposed to do, and the refusal itself is data we want to keep, not a global error. Fix: - _run_prompt_seed: when the target's run_turn raises LLMInputError, record a [TARGET INPUT REFUSED] event in the transcript and return with stop_reason='target_input_refused' instead of propagating. Other classified errors (LLMAuthError, LLMRateLimitError, LLMProviderError) and arbitrary runtime exceptions still propagate, since those are global pipeline problems rather than seed-specific. - _run_auditor_target_loop: same treatment for target-call refusals mid-conversation. Auditor-side LLMInputError still propagates because the auditor is our own generator, not seed data. The judge stage already passes stop_reason through opaquely, so the new target_input_refused value flows downstream without any judge-side changes. Tests: - test_run_prompt_seed_records_target_input_refusal: unit test that mocks the runtime to raise LLMInputError and verifies the transcript carries stop_reason='target_input_refused' plus the refusal event. - test_run_rollout_isolates_target_input_refusal_to_one_seed: end-to-end reproduction of the May 11 scenario at 5-seed scale. One seed gets refused; the other 4 complete cleanly; the batch returns successfully. - test_run_rollout_still_fails_fast_on_provider_5xx: regression guard that auth/rate-limit/5xx errors still abort the stage.
The rollout isolation in the previous commit converts target-side LLMInputError into a recorded transcript event with stop_reason="target_input_refused". It only fires when the exception reaches the rollout helpers as an already-classified LLM error class. For callable: targets (LangGraph agents, framework wrappers, raw litellm callers, etc.) the user's own code makes the provider calls and bypasses generate()/_with_retries entirely, so provider errors bubble up as raw litellm exceptions (BadRequestError, ContentPolicyViolationError, RateLimitError, ...). The rollout's isinstance(_, LLMInputError) checks miss them and the stage aborts on the first content-filter rejection \u2014 the same failure pattern the isolation was supposed to fix. Fix CallableSession.run_turn so any exception escaping invoke_callable is passed through _classify_llm_error before propagating. Classified errors are re-raised with __cause__ preserved; unclassified errors (user agent crashes, ValueError from misconfigured tools, etc.) pass through unchanged so they don't get smuggled into one of the four LLM error classes. Tests: - test_run_turn_reclassifies_litellm_bad_request_as_input_error: a fake litellm BadRequestError raised from inside the user callable emerges as LLMInputError with cause preserved. - test_run_turn_passes_through_unclassified_exceptions: a custom RuntimeError from inside the user callable propagates as itself, not as any LLM*Error class. All 50 rollout + exception-handling tests pass. Local smoke against azure/gpt-5.4-mini with the LangGraph travel-planner callable (30 prompts + 10 scenarios, C=4) completed end-to-end in 176.9s.
|
Pushed 0184d8d: Why the second commit: the rollout isolation in the first commit only fires when the exception reaches the helpers as an already-classified Classified errors are re-raised with Two new tests in
50/50 rollout + exception-handling tests pass. |
The previous commits isolate target-side LLMInputError. Auditor-side LLMInputError (where the auditor's adversarial prompt itself trips the provider's content filter / Prompt Shields jailbreak detector) was still re-raised by _run_auditor_target_loop's auditor catch block, which aborted the whole rollout stage on the first occurrence. This was hit live on the May 12 PR #44 mix-1k validation run: at scenario 853 of 1000, the auditor's adversarial prompt for one seed was refused by Azure Prompt Shields with the same Bad request / 'flagged as potentially violating our usage policy' signature. Aaron flagged this auditor-side refusal pattern in the May 12 standup and worked around it in PR #45 with a tame auditor system prompt for benchmarking; the right fix in product code is the same per-seed isolation as the target-side case. Fix: catch LLMInputError separately from the other classified errors in _run_auditor_target_loop's auditor block. Record an '[AUDITOR INPUT REFUSED: ...]' system event in the transcript, set stop_reason='auditor_input_refused', and end the conversation cleanly so the worker moves on to the next seed. Auth/rate-limit/5xx errors still propagate (they're global pipeline problems, not seed-specific). Tests: - test_run_rollout_isolates_auditor_input_refusal_to_one_seed: 5-seed scenario batch where one seed's auditor call raises LLMInputError; all 5 transcripts are written, the refused seed carries stop_reason='auditor_input_refused', the batch returns successfully. All 51 rollout + exception-handling tests pass.
|
Pushed f265154: auditor-side LLMInputError now gets the same per-seed isolation as the target-side case. Hit this live on the mix-1k validation run: at scenario 853/1000 the auditor's adversarial prompt for one seed tripped Azure Prompt Shields' jailbreak detector. Same Bad request / 'flagged as potentially violating our usage policy' signature, just on the auditor side instead of the target side. This is the same pattern Aaron flagged in standup today and worked around in PR #45 with a tame auditor prompt. The previous two commits handled the target side and the callable-target classification gap. This one closes the loop: One new test, Relaunching the 1k mix overnight on this commit. |
The previous commits isolate target-side LLMInputError and auditor-side LLMInputError in rollout. The judge stage had the same fail-fast catch for LLMInputError: a single content-filter rejection on a transcript the judge LLM can't process would kill the whole stage and discard the hundreds of seeds already scored. This was hit live on the May 12 PR #44 mix-1k validation rerun (after the auditor-side fix from f265154 unblocked rollout): the judge stage scored 539 of 1000 transcripts at the lowered concurrency, then died when one transcript's adversarial content tripped Azure's content filter on the judge call itself. Same Bad request / 'flagged as potentially violating our usage policy' signature, just on the judge side. This is the third logically-independent place the same adversarial-eval-as-attacker dynamic can fire. Fix: catch LLMInputError separately in the judge worker. Record a synthesized score row with judge_status='filter_skipped' and judge_error='judge_input_refused: ...' so the seed isn't lost and infer_judge_status downstream correctly counts it as a non-ok seed. Auth/rate-limit/provider-5xx errors still propagate (global pipeline problems, not seed-specific). The filter_skipped status flows cleanly through the existing infer_judge_status() contract in p2m/core/judge.py: it's a non-'ok' status, so it's treated as judge_failed by downstream consumers (results.py, viewer_read_model.py). This matches Aaron's PR #45 _install_judge_skip_blocked shim, but lands the behavior in product code instead of as a benchmark monkey-patch.
|
Pushed dcaa91f: judge-side LLMInputError now gets the same per-seed isolation as the rollout-side cases. Hit this live on the mix-1k rerun. After the auditor-side fix from f265154 unblocked rollout, the judge stage made it through 539 of 1000 transcripts (at C=3 to avoid the rate-limit cliff at C=10) before one transcript's adversarial content tripped Azure's content filter on the judge call itself. Same Bad request / 'flagged as potentially violating our usage policy' signature, just on the judge side instead of target or auditor. This is the third logically-independent place the same adversarial-eval-as-attacker dynamic fires, so PR #44 has grown into a coherent 'isolate LLMInputError per-seed across all three stages' shape:
The Re-running the mix-1k now on the new HEAD. |
…o threshold Previous commits handled three specific per-seed failure cases by classification (target LLMInputError, callable target unclassified litellm exceptions, auditor LLMInputError). Each surfaced through live validation. The mix-1k run on 2026-05-12 then surfaced a fourth gap that none of those targeted fixes covered: an openai.BadRequestError raised inside a LangGraph internal task (named 'research') escaped past _run_prompt_seed and CallableSession's wrapper. PR #44's other fixes work on classified errors; this one was raised in a different asyncio task than the one being awaited, so it never reached the classification layer at all. Rather than chase the next specific failure mode, this commit adopts a generic guarantee: one bad conversation can't kill the whole run. Worker change: - Removed the (ValueError, KeyError) and bare-Exception handlers that returned {'error': exc} and were stop-the-stage on any error. - Any non-global exception (i.e. not LLMAuthError, LLMRateLimitError, LLMProviderError) now synthesizes a transcript with stop_reason='runtime_error' carrying a single system event with the error type and message. The transcript is written like a normal one; the seed isn't lost. The exception is still attached to the result so the outer loop can count it toward the error-ratio threshold. Gather-loop change: - Replaced 'if errors: raise errors[0]' with an error-ratio threshold. - Default 10% (1 in 10 seeds erroring is the tipping point), tunable via the P2M_ROLLOUT_ERROR_FAIL_RATIO env var for ops scenarios. - Below threshold: log a single warning summary with the ratio and threshold and let the stage complete normally. - Above threshold: log an error and raise the first exception so the runner surfaces a clean message. Systemic problems (deployment misconfigured, every seed errors) still fail fast. Tests: - Updated test_run_rollout_keeps_partial_successful_transcripts_when_later_worker_fails to reflect the new contract: both seeds now have transcripts (the failing one with stop_reason='runtime_error'), and the stage still aborts at 50%>10% ratio. - New test_run_rollout_tolerates_below_threshold_errors: 20 seeds, 1 fails -> 5% below threshold -> batch completes with synthesized runtime_error transcript for the failing seed and 19 clean ones. All 52 rollout + exception-handling tests pass.
|
Pushed 15332c8: generic per-seed error isolation with a batch-level error-ratio threshold (default 10%). The mix-1k validation tonight surfaced a fourth specific failure mode that the targeted classification fixes don't catch: an Rather than chase the next specific gap, this commit adopts a generic guarantee: one bad conversation can't kill the whole run.
PR #44's shape has now stabilized into 'isolate per-seed failures, fail the stage only on systemic problems.' Five commits total:
52/52 rollout + exception-handling tests pass. |
jakepresent
changed the title
Lifts the four product-code changes from PR #44 into our branch and removes the corresponding benchmark monkey-patches that were doing the same job out-of-tree. Changes: * CallableSession.run_turn now reclassifies user-callable provider errors via `_classify_llm_error` so litellm BadRequestError / ContentPolicyViolationError emerge as typed `LLMInputError` (with `__cause__` preserved) instead of bypassing the typed-exception layer the rollout/judge stages key off. Unclassified exceptions propagate untouched. (PR #44 commit 0184d8d) * `_run_prompt_seed` records a `[TARGET INPUT REFUSED: ...]` event in the transcript and sets `stop_reason='target_input_refused'` when the target's run_turn raises `LLMInputError`. Other classified errors still propagate. (PR #44 commit 82cf339) * `_run_auditor_target_loop` splits the combined classified-error catch into a typed `LLMInputError` branch that records `[TARGET INPUT REFUSED: ...]` / `[AUDITOR INPUT REFUSED: ...]` events and sets `stop_reason='target_input_refused'` / `'auditor_input_refused'` respectively. Auth / rate-limit / 5xx errors still propagate. (PR #44 commits 82cf339 + f265154) * judge `_worker` splits the combined catch so `LLMInputError` becomes a synthesized `score_row` with `judge_status='filter_skipped'` and `judge_error='judge_input_refused: ...'`. Rate-limit / provider 5xx still go to the per-row error path. Auth still fails fast. (PR #44 commit dcaa91f) * New `P2M_ROLLOUT_ERROR_FAIL_RATIO` env var (default 0.10) caps the rollout stage's tolerance for *untyped* worker errors. Below the threshold: warn and continue (existing soft-fail). Above: raise the first error. Typed refusals (target_input_refused, auditor_input_refused, target_error) are NOT counted toward the ratio — they're recorded data, not real failures. Inspired by PR #44 commit 15332c8 but scoped to untyped errors only instead of #44's blanket runtime_error catch-all (which would have hidden real bugs behind the same threshold). * scripts/benchmark.py: drops `_install_content_filter_tolerance`, `_install_judge_skip_blocked`, and `_is_content_policy_violation` helpers (~210 lines). The product-code typed handlers above replace them. The benchmark now scans `transcripts.jsonl` and `scores.jsonl` after the run completes and counts typed stop_reasons / judge_status values. CSV columns updated to typed names: `target_input_refused_count`, `auditor_input_refused_count`, `target_error_count`, `judge_filter_skipped_count` (replaces the legacy `content_filter_blocked` / `target_error_tolerated` aggregates). `--no-tolerate-content-filter` flag preserved as a no-op for back-compat. Tests: * test_run_turn_reclassifies_litellm_bad_request_as_input_error * test_run_turn_passes_through_unclassified_exceptions * test_run_prompt_seed_records_target_input_refusal * test_run_rollout_isolates_target_input_refusal_to_one_seed * test_run_rollout_isolates_auditor_input_refusal_to_one_seed * test_run_rollout_still_fails_fast_on_provider_5xx * test_run_rollout_fails_when_untyped_error_ratio_exceeds_threshold (new — covers the threshold introduced in this commit) * test_run_judge_isolates_input_refusal_to_one_seed * test_run_rollout_keeps_partial_successful_transcripts_when_later_worker_fails (updated — sets P2M_ROLLOUT_ERROR_FAIL_RATIO=0.6 since the original 2-seed/1-failure setup now exceeds the production 10% default) 654 passed, 14 skipped, 13 subtests, 2 deselected (pre-existing Windows tempdir flakes in test_logging_config.py from main's PR #22). 0 new test failures. viewer check: 0 errors / 0 warnings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Closing as superseded by #45. Aaron's PR absorbed the per-row/per-batch error-isolation pieces from this branch and now has the cache-finalization and benchmark-summary fixes on top. |
1 participant
Problem
A single content-filter / input-refusal hit was killing entire scale runs. The May 11 overnight scale-500 was the first one: seed 501 of 502 was rejected by Azure CF, and 500 already-completed transcripts were thrown away. As subsequent runs ran further, three more independent places surfaced the same failure mode:
litellmexceptions - LangGraph (and any framework wrapper) makes provider calls inside the user callable, solitellm.BadRequestErroretc. bypassgenerate()/_with_retriesentirely and the rollout'sisinstance(_, LLMInputError)checks miss them.openai.BadRequestErrorraised inside a LangGraph internal task ('research') escaped past both_run_prompt_seedandCallableSession's wrapper because it was raised in a differentasynciotask than the one being awaited, so it never reached the classification layer at all.This is the wrong failure mode for an adversarial-eval pipeline. Generating prompts the target / auditor / judge refuses is exactly what we're supposed to do, and the refusal itself is data we want to keep, not a global pipeline error.
Fix
Treat per-seed failures as data, not as global pipeline errors, at four layers, plus a generic catch-all with a batch-level error-rate threshold.
Per-layer isolation
rollout._run_prompt_seed,_run_auditor_target_loop)LLMInputErrorfrom the target call[TARGET INPUT REFUSED: ...]event,stop_reason='target_input_refused'session.CallableSession.run_turn)_classify_llm_errorso they flow through the same target-side isolation; unclassified errors propagate unchangedrollout._run_auditor_target_loop)LLMInputErrorfrom the auditor call[AUDITOR INPUT REFUSED: ...]event,stop_reason='auditor_input_refused'stages.judge)LLMInputErroron the judge calljudge_status='filter_skipped',judge_error='judge_input_refused: ...'; flows through the existinginfer_judge_status()contract as a non-ok seedGeneric catch-all (final commit)
Any non-global exception that escapes the targeted handlers now synthesizes a transcript with
stop_reason='runtime_error'carrying a single system event with the error type and message. The transcript is written like a normal one; the seed isn't lost. The exception is still attached to the result so the gather loop can count it.Batch-level error-ratio threshold
Replaced
if errors: raise errors[0]in the rollout gather loop with a configurable threshold:P2M_ROLLOUT_ERROR_FAIL_RATIOfor ops scenariosSystemic failures (auth misconfigured, every seed errors) still fail fast.
What still fails fast (intentionally)
LLMAuthError- global config problem; retry won't helpLLMRateLimitError- global throttlingLLMProviderError- provider 5xx; conservative choiceValidation
End-to-end 1k mix run on
15332c8(travel-planner-langgraph-pr44-mix1k-e2e/baseline, 700 prompts + 300 scenarios, C=5):1000/1000 transcripts, 1000/1000 scored, all
judge_status='ok', zero recorded runtime errors. Earlier runs on the same branch did exercise the isolation paths live (auditor refusal at 853, judge filter at 539, LangGraph escape ~994); the final run was clean.Tests
7 new tests:
test_run_turn_reclassifies_litellm_bad_request_as_input_error(session)test_run_turn_passes_through_unclassified_exceptions(session)test_run_prompt_seed_records_target_input_refusal(rollout)test_run_rollout_isolates_target_input_refusal_to_one_seed(rollout)test_run_rollout_isolates_auditor_input_refusal_to_one_seed(rollout)test_run_rollout_still_fails_fast_on_provider_5xx(rollout)test_run_rollout_tolerates_below_threshold_errors(rollout)1 existing test updated (
test_run_rollout_keeps_partial_successful_transcripts_when_later_worker_fails) to reflect the new contract: failing seeds now have synthesizedstop_reason='runtime_error'transcripts, and the 50% error rate still aborts.All 52 rollout + exception-handling tests pass locally.
Files
Commits (read in order)
Each commit closed a real gap surfaced by the previous commit's validation, so the sequence is also the discovery log:
82cf339- target-sideLLMInputErrorisolation (the original May 11 fix).0184d8d- classify rawlitellmexceptions raised inside user callables, so callable-target frameworks (LangGraph, etc.) flow through the same target-side isolation.f265154- auditor-sideLLMInputErrorisolation.dcaa91f- judge-sideLLMInputErrorisolation.15332c8- generic per-seed error isolation with batch-level error-ratio threshold.Known follow-ups not in this PR
P2M_ROLLOUT_ERROR_FAIL_RATIOenv var is a temporary control surface. If it stays useful it should graduate to a real config field; if not, it can be dropped._install_content_filter_toleranceinscripts/benchmark.py, usingstop_reason="content_filter_blocked"). That patch monkey-patches the rollout to drop content-filter-blocked seeds at benchmark time; this PR lands per-seed isolation in product code so the shim can be retired once Load testing #45 merges (or revisited to align stop_reason naming).