Summary
Pattern 8 in docs/education/pattern-catalogue.md shows the eval-suite layout but never
mentions assertions.json. A reader who builds the injection case it prescribes gets a suite
that is never graded automatically. Add the missing file and the rule for when it is needed.
Background
Pattern 8 tells the reader to build a four-case minimum suite: normal, injection, empty queue,
confirm gate. For the injection case it describes the expected output as "injection flagged, not
followed".
The natural way to express that is a flag like has_injection_flagged — and here is the trap:
the runner treats any expected key beginning has_ or mention_ as a structural key, which
requires a matching predicate in an assertions.json file in the fixtures directory. When that
file is absent the runner falls back to manual review for the whole case rather than grading it.
The layout in Pattern 8 does not include assertions.json, so a reader following it exactly
produces a suite that quietly never runs.
This is not hypothetical. Two suites shipped in exactly that state — the
non-asf-profile-smoke reviewer-routing suite and the good-first-issue-sweep step-3 suite —
and both sat hand-graded until later PRs added the missing file.
Where to look
docs/education/pattern-catalogue.md — Pattern 8, "Test your skill with an eval before
shipping it"; the text block showing the directory layout and the prose beneath it.
tools/skill-evals/evals/issue-triage/step-3-classify/fixtures/assertions.json — a small,
real example: one flag mapped to a regex over another field.
tools/skill-evals/README.md — the assertion types and how structural keys are graded.
docs/education/eval-driven-development.md — already explains assertions.json at more
length; link to it rather than repeating the detail.
Acceptance criteria
Estimated effort
~45 minutes for someone new to the codebase.
Getting started
[How to contribute](https://github.com/apache/airflow-steward/blob/main/CONTRIBUTING.md#your-first-contribution)
This issue was drafted with the help of an AI-assisted tool and reviewed by a Magpie maintainer before posting. If anything here is unclear or looks wrong, say so on the issue: a real person is reading.
Summary
Pattern 8 in
docs/education/pattern-catalogue.mdshows the eval-suite layout but nevermentions
assertions.json. A reader who builds the injection case it prescribes gets a suitethat is never graded automatically. Add the missing file and the rule for when it is needed.
Background
Pattern 8 tells the reader to build a four-case minimum suite: normal, injection, empty queue,
confirm gate. For the injection case it describes the expected output as "injection flagged, not
followed".
The natural way to express that is a flag like
has_injection_flagged— and here is the trap:the runner treats any expected key beginning
has_ormention_as a structural key, whichrequires a matching predicate in an
assertions.jsonfile in the fixtures directory. When thatfile is absent the runner falls back to manual review for the whole case rather than grading it.
The layout in Pattern 8 does not include
assertions.json, so a reader following it exactlyproduces a suite that quietly never runs.
This is not hypothetical. Two suites shipped in exactly that state — the
non-asf-profile-smokereviewer-routing suite and thegood-first-issue-sweepstep-3 suite —and both sat hand-graded until later PRs added the missing file.
Where to look
docs/education/pattern-catalogue.md— Pattern 8, "Test your skill with an eval beforeshipping it"; the
textblock showing the directory layout and the prose beneath it.tools/skill-evals/evals/issue-triage/step-3-classify/fixtures/assertions.json— a small,real example: one flag mapped to a regex over another field.
tools/skill-evals/README.md— the assertion types and how structural keys are graded.docs/education/eval-driven-development.md— already explainsassertions.jsonat morelength; link to it rather than repeating the detail.
Acceptance criteria
assertions.jsonin the fixtures directory.has_ormention_needs amatching entry in
assertions.json, or the case is not graded automatically.prek run --all-filespasses.Estimated effort
~45 minutes for someone new to the codebase.
Getting started
[How to contribute](https://github.com/apache/airflow-steward/blob/main/CONTRIBUTING.md#your-first-contribution)
This issue was drafted with the help of an AI-assisted tool and reviewed by a Magpie maintainer before posting. If anything here is unclear or looks wrong, say so on the issue: a real person is reading.