Skip to content

Document assertions.json in the pattern catalogue's eval-suite pattern #1004

Description

@justinmclean

Summary

Pattern 8 in docs/education/pattern-catalogue.md shows the eval-suite layout but never
mentions assertions.json. A reader who builds the injection case it prescribes gets a suite
that is never graded automatically. Add the missing file and the rule for when it is needed.

Background

Pattern 8 tells the reader to build a four-case minimum suite: normal, injection, empty queue,
confirm gate. For the injection case it describes the expected output as "injection flagged, not
followed".

The natural way to express that is a flag like has_injection_flagged — and here is the trap:
the runner treats any expected key beginning has_ or mention_ as a structural key, which
requires a matching predicate in an assertions.json file in the fixtures directory. When that
file is absent the runner falls back to manual review for the whole case rather than grading it.
The layout in Pattern 8 does not include assertions.json, so a reader following it exactly
produces a suite that quietly never runs.

This is not hypothetical. Two suites shipped in exactly that state — the
non-asf-profile-smoke reviewer-routing suite and the good-first-issue-sweep step-3 suite —
and both sat hand-graded until later PRs added the missing file.

Where to look

  • docs/education/pattern-catalogue.md — Pattern 8, "Test your skill with an eval before
    shipping it"; the text block showing the directory layout and the prose beneath it.
  • tools/skill-evals/evals/issue-triage/step-3-classify/fixtures/assertions.json — a small,
    real example: one flag mapped to a regex over another field.
  • tools/skill-evals/README.md — the assertion types and how structural keys are graded.
  • docs/education/eval-driven-development.md — already explains assertions.json at more
    length; link to it rather than repeating the detail.

Acceptance criteria

  • The layout block shows assertions.json in the fixtures directory.
  • The prose states the rule: an expected key starting with has_ or mention_ needs a
    matching entry in assertions.json, or the case is not graded automatically.
  • The four-case minimum guidance is unchanged.
  • prek run --all-files passes.

Estimated effort

~45 minutes for someone new to the codebase.

Getting started

[How to contribute](https://github.com/apache/airflow-steward/blob/main/CONTRIBUTING.md#your-first-contribution)

This issue was drafted with the help of an AI-assisted tool and reviewed by a Magpie maintainer before posting. If anything here is unclear or looks wrong, say so on the issue: a real person is reading.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions