fix(run): report output guardrail results when a tripwire aborts the run - #4090
Merged
seratch merged 1 commit intoAug 1, 2026
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#4071 made input guardrails report the results that had already completed when a tripwire aborts the run. The output-guardrail sibling, directly below it in the same file, was left as it was — so the two exception types still report asymmetrically.
On an output tripwire,
run_output_guardrailsraises immediately and discards the localguardrail_resultsit had accumulated,run.pyhardcodesoutput_guardrail_results=[]when buildingRunErrorDetails, and the streamed path only assignsstreamed_result.output_guardrail_resultsafter the guardrail task returns — so a tripwire leaves it at its previous, empty value. Even the tripping result is absent fromrun_data; it is reachable only viaexc.guardrail_result.Reproduction with two output guardrails ordered by an
asyncio.Eventbarrier (passes, thentrips), onmain:With this change all four report
['passes', 'trips'].Fix
Mirrors #4071:
run_output_guardrailstakes aresults_sinkand records each result as it completes;run.pykeeps a run-level accumulator and passes it to the three call sites, using it forRunErrorDetails; the streamed path publishes the completed results ontostreamed_resultin itsexcept OutputGuardrailTripwireTriggeredbranch. Behaviour on the success path is unchanged.Test plan
Five tests appended to
tests/test_guardrails.py(the file #4071 used), reusing its_result_nameshelper and parameterised acrossrun/run_streamed/run_sync: results reported on tripwire for each entry point, results still reported on success, and completed results reported when a guardrail raises a non-tripwire exception.pytest tests/test_guardrails.py: 55 passed / 2 failed with the change; cleanmainbaseline is 50 passed / the same 2 failed (test_blocking_guardrail_cancels_remaining_on_triggerand its_streamingvariant — timing-sensitive, unrelated to this diff).pytest tests/models: 608 passed.ruff check/ruff format --checkclean;mypyreports no issues on the changed files.Verification-script note:
.agents/skills/code-change-verification/scripts/run.shcould not run here becausemakeis not installed on this machine, so I ran the underlying commands directly.Issue number
None — noticed while reading #4071.
Checks
.agents/skills/code-change-verification/scripts/run.sh— could not:makeunavailable locally (see Test plan); ran the equivalent commands directlymain)/reviewbefore submitting this PR — n/aAI assistance was used in preparing this change; I reviewed it, reproduced the behaviour and verified the fix locally.