Skip to content

Bounded recovery loop for hard verification failures#283

Merged
mmprotest merged 1 commit into
mmprotest/implement-shell-normalization-for-command-executionfrom
mmprotest/implement-bounded-recovery-loop-in-runner
Apr 11, 2026
Merged

Bounded recovery loop for hard verification failures#283
mmprotest merged 1 commit into
mmprotest/implement-shell-normalization-for-command-executionfrom
mmprotest/implement-bounded-recovery-loop-in-runner

Conversation

@mmprotest

Copy link
Copy Markdown
Owner

Motivation

  • Prevent runtime/syntax crashes from counting as verification success and stop delete-and-regenerate recovery that lost failing-file context.
  • Add a minimal, generic bounded recovery mode that forces a read→fix→rerun loop on the exact failing file and returns compact structured feedback when blocking disallowed actions.

Description

  • Added a compact recovery state on the runner (_recovery_mode, _failing_file, _failing_error_summary, _failing_command, _file_was_read_since_failure) and initialize/reset it at run start in Runner (villani_code/state.py).
  • Detect hard verification failures inside run_verification in villani_code/state_runtime.py via _detect_hard_failure(...) and when found: force VerificationStatus.FAIL, reduce confidence, capture failing_file/error_summary/failing_command, and activate recovery_mode; clear recovery only after a successful verification pass or repaired state.
  • Enforce bounded recovery behavior by extending small_model_tool_guard(...) in villani_code/state_runtime.py to: block edits that are not targeted at the active failing file, block edits to the failing file until it has been read since the failure, block delete/regenerate (patch delete or empty Write) and large rewrites of the failing file, and return compact JSON feedback via _recovery_blocked_feedback(...).
  • Apply the guard unconditionally in execute_tool_with_policy(...) (villani_code/state_tooling.py) so recovery checks run regardless of small-model/benchmark flags, and mark the failing-file read flag (_file_was_read_since_failure) on successful Read tool results in both state_tooling.execute_tool_with_lifecycle and the runner tool loop (villani_code/state.py).
  • Small, surgical changes only; reused existing verification/mission/session structures and existing tool lifecycle to avoid large refactors.
  • Files changed: villani_code/state_runtime.py (hard-failure detection, helpers, guard extension), villani_code/state.py (runner fields, tool-loop read/flag handling, reset), villani_code/state_tooling.py (apply guard earlier, mark read), and tests updated/added in tests/test_state_runtime.py and tests/test_state_tooling_mutation_policy.py.

Testing

  • Added unit tests: test_syntax_error_does_not_mark_verification_pass, test_name_error_does_not_mark_verification_pass, test_hard_failure_sets_recovery_mode_state, test_successful_rerun_clears_recovery_mode, test_recovery_mode_blocks_delete_of_active_failing_file, test_recovery_mode_blocks_edit_without_read_first, and test_read_failing_file_flips_recovery_read_flag in the two modified test files.
  • Ran: pytest -q tests/test_state_runtime.py tests/test_state_tooling_mutation_policy.py and observed all tests passing: 47 passed.

Codex Task

@mmprotest mmprotest merged commit c436588 into mmprotest/implement-shell-normalization-for-command-execution Apr 11, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant