Add multi-SKIP case to gfi-sweep eval suite - #1062
Open
deepusnath wants to merge 3 commits into
Open
Conversation
Pattern 8's eval-suite layout omitted assertions.json, so a reader following it exactly produces cases with has_*/mention_* expected keys that silently fall back to MANUAL grading instead of PASS/FAIL. - Add assertions.json to the fixtures directory-layout block. - Add a paragraph explaining the has_*/mention_* structural-key rule and linking to eval-driven-development.md for the format. Closes apache#1004
Every existing step-3-present-proposals case has 0 or 1 SKIP issues, so skip_count_shown_as_summary_only was vacuously satisfied and a model that always returned true would pass. case-5-multi-skip adds 1 READY issue plus 6 SKIP issues spread across all three skip_reason categories (2 security-sensitive, 2 architectural-decision, 2 deprecation-decision) so the summary-only rule and the (security: M, architectural: K, deprecation: J) breakdown format actually get exercised. Closes apache#1001
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
step-3-present-proposalscase in the good-first-issue-sweep eval suite has 0 or 1 SKIP issues, soskip_count_shown_as_summary_onlywas only ever vacuously satisfied — a model that always returnstruewould pass.case-5-multi-skip: 1 READY issue plus 6 SKIP issues spread across all threeskip_reasoncategories (2 security-sensitive, 2 architectural-decision, 2 deprecation-decision), so the summary-only rule and the(security: M, architectural: K, deprecation: J)breakdown format from the skill's Step 3 SKIP section actually get exercised.Closes #1001
Test plan
PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/good-first-issue-sweep/step-3-present-proposals/fixtures/— all 5 cases load and render cleanly (exit 0).expected.jsoncontains all 8 output-spec fields with correct values for the new case.