Summary
The eval case case-4-all-required-missing exists to prove the skill reports all nine
required-field schema violations, but nothing in it checks the count. A model that reports one
violation passes. Pin the expected list so the case tests what it claims to.
Background
The case's expected.json contains no literal schema_violations value. The only thing
checking that field is a judge assertion whose rubric asks two questions: is
schema_violations non-empty when at least one required field is MISSING, and does each entry
name a genuinely-missing required field?
Both are true for this response:
{ "schema_violations": ["rc_label — required field is MISSING"] }One entry out of nine, and the case passes. The rubric checks that nothing wrong is in the list,
but never that nothing is missing from it.
The same file already does this correctly for the older fields_missing field: all nine names
are listed literally and compared exactly. The new field just needs the same treatment.
Where to look
tools/skill-evals/evals/release-audit-report/step-2-assemble-record/fixtures/case-4-all-required-missing/expected.json
— add the literal schema_violations list here. The fields_missing entry directly above
is the model to follow.skills/release-audit-report/SKILL.md — the Step 2 section states the exact string form each
violation takes ("<field> — required field is MISSING"). Use that form.tools/skill-evals/evals/release-audit-report/step-2-assemble-record/fixtures/assertions.json
— the has_schema_violations_consistent rubric, for context. You do not have to change it,
but if you would rather fix the rubric to require completeness instead, say so on the issue
first.
Acceptance criteria
Estimated effort
~1 hour for someone new to the codebase.
Getting started
[How to contribute](https://github.com/apache/airflow-steward/blob/main/CONTRIBUTING.md#your-first-contribution)
This issue was drafted with the help of an AI-assisted tool and reviewed by a Magpie maintainer before posting. If anything here is unclear or looks wrong, say so on the issue: a real person is reading.
Summary
The eval case
case-4-all-required-missingexists to prove the skill reports all ninerequired-field schema violations, but nothing in it checks the count. A model that reports one
violation passes. Pin the expected list so the case tests what it claims to.
Background
The case's
expected.jsoncontains no literalschema_violationsvalue. The only thingchecking that field is a judge assertion whose rubric asks two questions: is
schema_violationsnon-empty when at least one required field is MISSING, and does each entryname a genuinely-missing required field?
Both are true for this response:
{ "schema_violations": ["rc_label — required field is MISSING"] }One entry out of nine, and the case passes. The rubric checks that nothing wrong is in the list,
but never that nothing is missing from it.
The same file already does this correctly for the older
fields_missingfield: all nine namesare listed literally and compared exactly. The new field just needs the same treatment.
Where to look
tools/skill-evals/evals/release-audit-report/step-2-assemble-record/fixtures/case-4-all-required-missing/expected.json— add the literal
schema_violationslist here. Thefields_missingentry directly aboveis the model to follow.
skills/release-audit-report/SKILL.md— the Step 2 section states the exact string form eachviolation takes (
"<field> — required field is MISSING"). Use that form.tools/skill-evals/evals/release-audit-report/step-2-assemble-record/fixtures/assertions.json— the
has_schema_violations_consistentrubric, for context. You do not have to change it,but if you would rather fix the rubric to require completeness instead, say so on the issue
first.
Acceptance criteria
case-4'sexpected.jsonlists all nine required-field violations literally, in thestring form the skill specifies.
fields_missingin the same file.PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/release-audit-report/runs without error.Estimated effort
~1 hour for someone new to the codebase.
Getting started
[How to contribute](https://github.com/apache/airflow-steward/blob/main/CONTRIBUTING.md#your-first-contribution)
This issue was drafted with the help of an AI-assisted tool and reviewed by a Magpie maintainer before posting. If anything here is unclear or looks wrong, say so on the issue: a real person is reading.