fix: Use least-tested-first selection for regression testing#47
Closed
connor-tyndall wants to merge 1 commit intoAutoForgeAI:masterfrom
Closed
fix: Use least-tested-first selection for regression testing#47connor-tyndall wants to merge 1 commit intoAutoForgeAI:masterfrom
connor-tyndall wants to merge 1 commit intoAutoForgeAI:masterfrom
Conversation
Replaces random selection in feature_get_for_regression with a least-tested-first approach to ensure even distribution of regression testing across all features. Changes: - Add regression_count column to Feature model to track test frequency - Add database migration for existing databases - Update feature_get_for_regression to order by regression_count (ascending) - Increment regression_count when features are selected for testing This prevents the same features from being tested repeatedly while others are never tested, reducing wasted tokens and ensuring comprehensive regression coverage. Closes AutoForgeAI#20 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
I am using this PR on my server and it works. |
sundog75
added a commit
to sundog75/autocoder
that referenced
this pull request
Jan 24, 2026
Implements the regression_count column and feature_get_for_regression MCP tool to ensure even distribution of regression testing across all passing features. Changes: - Add regression_count column to Feature model with migration - Add feature_get_for_regression MCP tool that: - Returns passing features ordered by regression_count (ascending) - Increments count after selection for round-robin behavior - Prevents duplicate testing of same features - Remove unused RegressionInput class Based on PR AutoForgeAI#47 by connor-tyndall, cleanly reimplemented to avoid merge conflicts. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
rudiheydra
added a commit
to rudiheydra/AutoBuildr
that referenced
this pull request
Jan 27, 2026
…I#47) Support forbidden_tools blacklist for explicit blocking in addition to allowed_tools whitelist. Implementation: - Add ForbiddenToolBlocked exception class for forbidden tool violations - Add extract_forbidden_tools() to extract blacklist from tool_policy - Update ToolPolicyEnforcer to include forbidden_tools field - Update validate_tool_call() to check forbidden_tools after allowed_tools - Add create_forbidden_tools_violation() for PolicyViolation creation - Add record_forbidden_tools_violation() for event recording - Add get_forbidden_tool_error_message() for clear agent messages - Update VIOLATION_TYPES to include "forbidden_tools" - Export new functions from api/__init__.py All 5 feature steps verified: 1. Extract forbidden_tools from spec.tool_policy - PASS 2. After filtering by allowed_tools, also remove forbidden_tools - PASS 3. Block any tool call to forbidden tool - PASS 4. Record policy violation event - PASS 5. Return clear error message to agent - PASS Test results: - tests/test_feature_47_forbidden_tools.py: 47/47 tests PASS - tests/verify_feature_47.py: 21/21 verification checks PASS Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
rudiheydra
added a commit
to rudiheydra/AutoBuildr
that referenced
this pull request
Jan 27, 2026
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
getworken
pushed a commit
to getworken/autocoder
that referenced
this pull request
Jan 29, 2026
Implements the regression_count column and feature_get_for_regression MCP tool to ensure even distribution of regression testing across all passing features. Changes: - Add regression_count column to Feature model with migration - Add feature_get_for_regression MCP tool that: - Returns passing features ordered by regression_count (ascending) - Increments count after selection for round-robin behavior - Prevents duplicate testing of same features - Remove unused RegressionInput class Based on PR AutoForgeAI#47 by connor-tyndall, cleanly reimplemented to avoid merge conflicts. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
sundog75
added a commit
to sundog75/autocoder
that referenced
this pull request
Jan 30, 2026
Implements the regression_count column and feature_get_for_regression MCP tool to ensure even distribution of regression testing across all passing features. Changes: - Add regression_count column to Feature model with migration - Add feature_get_for_regression MCP tool that: - Returns passing features ordered by regression_count (ascending) - Increments count after selection for round-robin behavior - Prevents duplicate testing of same features - Remove unused RegressionInput class Based on PR AutoForgeAI#47 by connor-tyndall, cleanly reimplemented to avoid merge conflicts. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #20 - Random testing results in duplicate testing (wasting tokens / time)
regression_countcolumn to track how many times each feature has been regression testedProblem
The current
feature_get_for_regressionusesfunc.random()to select features:This causes:
Solution
Track regression test frequency and prioritize least-tested features:
regression_countcolumn - Tracks how many times each feature has been regression testedorder_by(regression_count.asc(), id.asc())Before
After
Test plan
regression_countcolumnfeature_get_for_regressionmultiple times - verify different features returned each timeregression_countincrements after each call🤖 Generated with Claude Code