ci(harness): decide relevance in a gate job, not with paths: filters (#116 item 5)

[mmcky]

Prerequisite for making Action harness: all checks a required check on main. Closes #116 item 5 — though not the way that item proposed.

Why the filters have to go

A paths: filter suppresses creation of the workflow run, not just its jobs. No run means no check run is ever published for that commit, so a required check sits in "Expected — waiting for status to be reported" forever. GitHub documents this directly, in Troubleshooting required status checks → Handling skipped but required checks: a workflow skipped by path filtering blocks merging, whereas a job skipped by a conditional reports Success.

Two live demonstrations from this week: release PR #119 touched only CHANGELOG.md and GitHub reported "no checks reported on the 'release/v0.10.0' branch"; docs PR #121 ran nothing either. Under branch protection both would have been unmergeable.

Why not the companion workflow #116 suggested

#116 item 5 proposed "a companion workflow with the negated paths that emits a passing job of the same name". That does not work. paths-ignore is not the complement of paths — both triggers are existential over the changed-file set, so a PR touching both a covered and an uncovered path satisfies both filters, fires both workflows, and produces two check runs with the same name on one SHA. That case is the norm in this repo, not an edge case.

The design

Both filters go; the workflow always runs; a gate job decides relevance. Only the four root jobs take needs: gate — everything else already chains off them, and a job whose needs were skipped is itself skipped.

The decision rule is an ignore list, not a cover list, and that direction is the whole point. For a required check the expensive mistake is a green earned by running nothing. So anything unrecognised — a new action directory, a new fixture, a new helper — runs the whole harness. A cover list inverts the failure direction: forgetting to extend it would silently rubber-stamp untested action changes, whereas today the same omission blocks the merge. Forgetting to extend the ignore list only costs CI minutes, which are free on a public repo.

The gate self-tests. IGNORED is a hand-written regex evaluated only against live input, so a dropped | or a mis-escaped . could answer "irrelevant" for a real setup-environment/action.yml change — and harness-summary cannot catch that, because all-skipped is a legitimate shape by construction. So the gate derives its must-always-run set from the uses: ./<action> lines in the workflow itself and fails closed if IGNORED ever matches one. Deriving it from the workflow means future coverage (e.g. publish-gh-pages) is protected with no second edit.

Permissions are job-scoped. On a fork pull_request the other 15 jobs execute PR-authored composite-action code via uses: ./, and none of them needs a pull-request scope.

harness-summary is rewritten to certify both shapes, reject a vacuous empty job set, and fail when the gate and the fan-out disagree.

Verification

Both scripts were extracted from the YAML and run against mocked inputs before pushing.

Gate — decisions:

input	result
PR #119's real file list (CHANGELOG.md)	relevant=false
PR #121's real file list (.gitignore, PLAN.md, TESTING.md, tests/README.md)	relevant=false
setup-environment/action.yml	relevant=true
.github/fixtures/mini-lectures/lectures/lecture1.md	relevant=true
publish-gh-pages/action.yml (not yet covered)	relevant=true — fails open
workflow_dispatch	relevant=true

Gate — failure directions, which are what make the check trustworthy:

input	result
IGNORED widened to swallow setup-environment/	rc=1, gate self-test: IGNORED matches [setup-environment/action.yml], no decision emitted
gh api error on a PR	rc=1, fails closed, no decision emitted
compare 404 (force-push)	rc=0, relevant=true — fails open
push with no before SHA	rc=0, relevant=true — fails open

Summary — all ten reachable shapes: relevant+all-success → green; relevant+any failure or skip → red; not-relevant+all-skipped → green; not-relevant+any job ran → red; gate failed / skipped / missing → red; no jobs besides the gate → red; gate succeeded with no decision → red.

What this does not do

It does not enable branch protection — that is a separate, deliberate step once this is on main and observed working.

And a green required check gates the hosted-runner unit surface only. bjc-smoke still passes create-issue-on-failure: 'false', there is still no container job, and build-jupyter-cache's internal calls still run @v0. The chain that actually broke in lecture-dp is covered by the canary, not by this check.

Refs #116

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR updates the GitHub Actions “action harness” workflow so it always creates a workflow run (avoiding paths:-suppressed required checks), and moves “does this harness apply?” logic into a new gate job that conditionally skips the expensive jobs while still publishing a successful required check when appropriate.

Changes:

• Remove paths: filters from .github/workflows/test-actions.yml and add a gate job that determines whether the harness is relevant for the current event’s changed files.
• Make the four root harness jobs depend on the gate output (skip-on-irrelevant), and update harness-summary to validate both “all-ran” and “all-skipped” shapes and reject vacuous success.
• Update docs/metadata (TESTING/CONTRIBUTING/tests README) and CHANGELOG to reflect the new gate-based behavior and the required-check rationale.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/README.md	Updates the test-asset map and documents the new gate-based harness triggering model.
TESTING.md	Updates harness description to explain always-run workflow + gate-controlled job fan-out.
CONTRIBUTING.md	Adds contributor guidance to avoid reintroducing paths: filters and to extend the gate ignore list instead.
CHANGELOG.md	Records the CI behavior change and rationale under Unreleased.
.github/workflows/test-actions.yml	Removes paths: filters, adds relevance gate, wires root jobs to gate output, and hardens harness-summary certification logic.