-
Notifications
You must be signed in to change notification settings - Fork 24
Open
Description
Problem
During the /test phase, the agent ran specific CLI unit tests and verified they passed, but failed to run the full E2E test suite (tests/e2e/test_critical_paths.py). This resulted in a broken test (test_assess_with_valid_config) being discovered only after pushing to CI.
Impact
- Delayed feedback loop (issues caught in CI rather than locally)
- Additional iteration cycles required to fix CI failures
- Wasted CI resources
Context
This issue was identified during the bugfix session for ambient-code/agentready#306 addressing ambient-code/agentready#302.
The agent's fix successfully started validating config files, which caught an invalid mock attribute (repomix_config) in an old test. Had the full test suite been run locally during /test, this would have been caught immediately.
Recommended Solution
The bugfix workflow's /test phase should:
- Explicitly instruct the agent to run the full test suite after implementing the fix and regression tests
- Include guidance to run both unit tests and E2E tests before moving to the
/documentphase - Add a checklist item: "Verify all existing tests still pass (not just new regression tests)"
Reference
From PR comment: #42 (comment)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels