Summary
Every assertion in TestOrganizationNonASFSmoke checks that a list of violations is empty. That
means the tests cannot tell "the checks ran and correctly found nothing" from "the checks did not
run at all". Add one case per surface where a violation is expected.
Background
The class covers three workflow surfaces for a non-ASF project — security intake, release
housekeeping, contributor governance — and asserts for each that organization: independent is
accepted and that the fixture body produces no ASF-coupling warnings. All seven assertions are of
the form assert x == [].
These tests do have value: the realistic regression is an ASF-coupling heuristic drifting too
broad and firing on a GHSA body or a GitHub-Releases workflow, and these fixtures would catch
that. What they cannot catch is the check going silent — delete the coupling scan tomorrow and
all four tests still pass.
A negative control fixes that: one fixture per surface that should raise a violation, asserting
it does. Then an empty result in the positive cases means something.
Where to look
tools/skill-and-tool-validator/tests/test_validator.py — the TestOrganizationNonASFSmoke
class; each test already builds its fixture under tmp_path, so the scaffolding you need is
written.- Other classes in the same file — most pair a "no violations" test with one that asserts a
specific violation is produced; TestOrganizationStructure is a compact example.
tools/skill-and-tool-validator/src/skill_and_tool_validator/__init__.py — the organization
membership check and the ASF-coupling scan, to see what input actually triggers each.
Acceptance criteria
Estimated effort
~1-2 hours for someone new to the codebase.
Getting started
[How to contribute](https://github.com/apache/airflow-steward/blob/main/CONTRIBUTING.md#your-first-contribution)
This issue was drafted with the help of an AI-assisted tool and reviewed by a Magpie maintainer before posting. If anything here is unclear or looks wrong, say so on the issue: a real person is reading.
Summary
Every assertion in
TestOrganizationNonASFSmokechecks that a list of violations is empty. Thatmeans the tests cannot tell "the checks ran and correctly found nothing" from "the checks did not
run at all". Add one case per surface where a violation is expected.
Background
The class covers three workflow surfaces for a non-ASF project — security intake, release
housekeeping, contributor governance — and asserts for each that
organization: independentisaccepted and that the fixture body produces no ASF-coupling warnings. All seven assertions are of
the form
assert x == [].These tests do have value: the realistic regression is an ASF-coupling heuristic drifting too
broad and firing on a GHSA body or a GitHub-Releases workflow, and these fixtures would catch
that. What they cannot catch is the check going silent — delete the coupling scan tomorrow and
all four tests still pass.
A negative control fixes that: one fixture per surface that should raise a violation, asserting
it does. Then an empty result in the positive cases means something.
Where to look
tools/skill-and-tool-validator/tests/test_validator.py— theTestOrganizationNonASFSmokeclass; each test already builds its fixture under
tmp_path, so the scaffolding you need iswritten.
specific violation is produced;
TestOrganizationStructureis a compact example.tools/skill-and-tool-validator/src/skill_and_tool_validator/__init__.py— the organizationmembership check and the ASF-coupling scan, to see what input actually triggers each.
Acceptance criteria
organization:value produces aviolation.
organization: independentproduces a coupling violation.list.
Estimated effort
~1-2 hours for someone new to the codebase.
Getting started
[How to contribute](https://github.com/apache/airflow-steward/blob/main/CONTRIBUTING.md#your-first-contribution)
This issue was drafted with the help of an AI-assisted tool and reviewed by a Magpie maintainer before posting. If anything here is unclear or looks wrong, say so on the issue: a real person is reading.