Skip to content

Make no-progress detection failure-sensitive#384

Merged
mmprotest merged 1 commit into
mainfrom
mmprotest/improve-no-progress-detection-in-runner
Jun 8, 2026
Merged

Make no-progress detection failure-sensitive#384
mmprotest merged 1 commit into
mainfrom
mmprotest/improve-no-progress-detection-in-runner

Conversation

@mmprotest

Copy link
Copy Markdown
Owner

Motivation

  • Improve the runner’s no-progress detection so repeated failing validations are treated as progress when the failure cause changes, avoiding premature finalization.
  • Make evidence signatures more discriminating by incorporating a normalized failure fingerprint derived from command output.
  • Ensure file mutations, diagnostic actions, or successful write/patch operations reduce repeated-command pressure.

Description

  • Add a general failure_fingerprint(stdout, stderr, exit_code) that extracts and normalizes the most informative failure line (stripping ANSI, paths, temp paths, timestamps, UUIDs, addresses, line numbers) and falls back to a compact hashed tail. (see villani_code/execution_context.py)
  • Store the fingerprint on CommandRecord.failure_fingerprint and ValidationEvidence.failure_fingerprint, and compute it from command stdout/stderr when building command records.
  • Extend the repeated-evidence key to include fingerprint and track repetition with an epoch-aware counter (_evidence_repetitions and _progress_epoch) so that intervening workspace mutations or new diagnostic evidence reset repetition pressure.
  • Advance the progress epoch for workspace-affecting mutations and for diagnostic/tool uses (and for successful Write/Patch), so identical failures after meaningful changes are treated as progress.
  • Add unit tests covering: differing tracebacks treated as progress, identical traceback triggering no-progress after threshold, file mutation resetting matching failure pressure, and normalization of volatile traceback details.

Testing

  • Ran pytest -q tests/test_execution_context_isolation.py (targeted tests exercising the new logic) and all tests in that file passed.
  • Ran ruff check on the modified files and it passed.
  • Ran the full test suite pytest -q; the changes did not affect the targeted execution-context tests, but the full run reported 7 unrelated failures in hardening/interactive/mission/planning/TUI contract tests (these failures predate or are outside the scope of the execution-context change).

Codex Task

@mmprotest mmprotest merged commit 29a8361 into main Jun 8, 2026
8 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant