fix(gate): don't read file content as a tool failure - #16
Closed
simonbbby wants to merge 1 commit into
Closed
Conversation
detect_failure() text-greps the tool response for failure words. For Bash that response is a command log, so the heuristic is right. For Edit/Write it is the file itself, so editing a doc containing 'Failed attempts', a changelog line 'fixed the failure', or a fixture with '3 errors' was reported as a failed tool call — and the agent was then told not to report completion. In one repo this fired on essentially every edit to a status doc that has a '## Failed attempts' section. Content tools signal failure structurally (success / ok / exit_code), never in prose, so they now consult only that signal: - new structural_success() reads explicit result fields only - exit_success() delegates to it, then falls back to the text heuristic - detect_failure() returns early for CONTENT_TOOLS Bash behaviour is unchanged, including textual-only failures with no exit code. A genuinely failed Edit is still caught, via the structural signal. tests/test_content_tool_failures.py covers it: 14 checks, and 7 of them fail against the unpatched parser.
Author
|
Closing as a duplicate of #15, which predates this by two weeks and fixes the bug more completely. Also tracked as #11 and #14. My version only exempted content tools ( The regression test from this PR has been adapted to #15 semantics and offered there: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Editing a file whose content contains a failure word gets reported as a failed tool call.
detect_failure()text-greps the tool response againstFAILURE_REregardless of tool type. ForBashthat response is a command log, so the heuristic is right. ForEdit/Writethe response is the file itself, so any of these trip it:## Failed attemptssectionfixed the failure in the parser3 errorsorerror: cannot find moduleThe agent then gets
fablize gate observed a tool failure. Do not report completion until it is fixed...for an edit that succeeded. In one repo this fired on essentially every edit to a status doc that has a## Failed attemptsheading — enough that the agent started routinely disclaiming the gate output as spurious, which is the opposite of what the gate is for.Fix
Content tools report failure structurally (
success/ok/exit_code), never in prose, so they now consult only that signal:structural_success()reads explicit result fields only, never textexit_success()delegates to it, then falls back to the text heuristic as beforedetect_failure()returns early forCONTENT_TOOLS(Edit,Write,NotebookEdit,MultiEdit,Read,NotebookRead)Bashbehaviour is unchanged, including textual-only failures with no exit code. A genuinely failedEditis still caught — via the structural signal rather than by accident.Tests
New
tests/test_content_tool_failures.py, following the existing standalone-script convention: 14 checks covering the false positives, structurally-failed content tools, and the unchanged Bash paths.Red-proofed — against the unpatched parser, 7 of the 14 fail:
With the fix, all 7 existing suites plus the new one exit 0.