Skip to content

Refuse a required context and a check name that disagree (#71) - #131

Merged
iderex merged 2 commits into
mainfrom
parity/a-required-context-and-a-check-name-agree
Aug 12, 2026
Merged

Refuse a required context and a check name that disagree (#71)#131
iderex merged 2 commits into
mainfrom
parity/a-required-context-and-a-check-name-agree

Conversation

@iderex

@iderex iderex commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Closes #71

What this changes

Adds the comparison between the two lists that have to agree for the merge gate
to mean what a reader thinks it means: the contexts the ruleset requires on the
default branch, and the check names the workflows in this tree declare.

internal/contexts is the judgement. It is a function over two lists and reads
nothing, so every rule is proved against a fixture written out in full rather
than against whatever the machine running the suite happened to contain.
cmd/contexts is the entry point and takes the required set on standard input.
.github/workflows/contexts.yml is where the ruleset is fetched, with a
read-only token granted on that one job, because half of this is a live setting
rather than a file in the checkout and lab opens no network connection.

Five refusals, each with a fixture that trips exactly it and no other:

  • a context the ruleset requires that no workflow here declares
  • a check name this tree declares that is neither required nor written down as a
    deliberate absence
  • a deliberate absence naming a check name no workflow declares
  • a deliberate absence naming a context the ruleset requires
  • a declared name still carrying a workflow expression the reader could not
    resolve, so a file in a shape this reader was not built for reddens the check
    instead of quietly declaring the wrong string

The deliberate absences are a list beside the comparison in
internal/contexts/contexts.go rather than a paragraph somewhere else, with a
reason on every entry and the issue that retires it where the absence is not
permanent. One entry is permanent: the supply-chain self-audit publishes from
the default branch and has no pull-request trigger, so requiring it would
require a context that never arrives on the thing being gated.

The rest of the list is one fact repeated. The ruleset requires no status check
at all today, so every name this tree declares is outside the set, and each one
is written down rather than the comparison switching itself off while the set is
empty. That is what catches a rename now: an absence names a literal string, so
a job renamed while the set is still empty leaves its entry pointing at nothing.
Issue #26 retires those entries by moving the names into the ruleset.

Two of the refusals also run in the ordinary suite, with no network, because the
answer is knowable without asking while the required set is empty.

Not every check run comes from a job, and the first commit here missed that. A
code-scanning upload creates a check run named after the analysis rather than
after the job that uploaded it, so it is a name a ruleset can require and no
reader of the workflow files can ever find. Measured rather than supposed:

gh api "repos/Flowfin/lab/commits/$(git rev-parse HEAD)/check-runs" --paginate --jq '.check_runs[] | .name + " " + .app.slug' | sort -u

On this branch's head that printed twenty-four names, of which twenty-two come
from github-actions and two, CodeQL and zizmor, come from
github-advanced-security. Those two are written down beside the absences with
the command that found them. Without that the comparison would refuse either of
them as a required context nothing reports, which is a false red on the change
that assembles the required set.

What failure it prevents

A check name is what the required set refers to and nothing on the platform ties
the two together. A job renamed in passing removes itself from the required set
while the tab still looks green, which is the quieter of the two failures and
survives indefinitely. A context required under a name nothing reports blocks
every merge and gives no reason on the pull request, so the first response is to
wait and the second is to drop the context, which repairs a blocked board by
making the gate smaller.

What was run

At the commit being pushed, on windows/amd64.

gofmt -l cmd internal
(no output)

go build ./cmd/... ./internal/...
go vet ./cmd/... ./internal/...
(no output from either)

go test -count=1 -v ./cmd/... ./internal/... | grep -cE '^--- (PASS|FAIL): '
109
go test -count=1 -v ./cmd/... ./internal/... | grep -cE '^--- SKIP: '
0
go test -count=1 -v ./cmd/... ./internal/... | grep -cE '^FAIL'
0

go run ./cmd/lab check .
examined .
1 experiment directory walked, 1 record read
18 decision records read
the time this run read is 2026-08-12T10:49:34Z
0 refused

Each of the five refusals was deleted in turn and the suite re-run. All five
leave it red:

name-carries-an-expression: RED
absence-names-nothing-declared: RED
absence-is-required: RED
required-context-nothing-reports: RED
declared-name-outside-the-required-set: RED

The rename, against this tree rather than a fixture. Renaming the vet job in
build.yml to go vet and leaving the required set alone:

--- FAIL: TestNoDeliberateAbsenceNamesSomethingThisTreeDoesNotDeclare
    the absence list names "vet" and no workflow in this tree declares it, so
    either a job was renamed and the entry was left behind or the entry never
    matched
--- FAIL: TestEveryCheckNameThisTreeDeclaresIsWrittenDown
    build.yml declares the check name "go vet" and nothing writes it down, so
    it is a check that runs and holds no merge

The other direction, through the command, with three contexts fed in and the
last of them naming nothing:

compared the required contexts against the check names this tree declares
  required by the ruleset: 3
  declared by the workflows: 25
  written down as deliberately absent: 25
note: dependency-review.yml: the check name "dependency-review" comes from the job id, because the job carries no name of its own
note: codeql.yml: the check name "CodeQL" is written in no workflow file and is expected here rather than read, so a change to the string it reports under is not caught by this comparison
note: zizmor.yml: the check name "zizmor" is written in no workflow file and is expected here rather than read, so a change to the string it reports under is not caught by this comparison
refused: vet: the ruleset requires this context and this list says it is deliberately outside the required set, so the two disagree (absence-is-required)
refused: format: the ruleset requires this context and this list says it is deliberately outside the required set, so the two disagree (absence-is-required)
refused: a context nothing reports: the ruleset on the default branch requires this context and no workflow in this tree declares a check name matching it, so every merge waits for a tick that never arrives (required-context-nothing-reports)
3 refusal(s), 3 note(s)
exit=1

The whole thing against the live ruleset, which is what the workflow does:

gh api "repos/Flowfin/lab/rules/branches/main" --jq '.[] | select(.type=="required_status_checks") | .parameters.required_status_checks[].context' > required.txt
wc -l < required.txt
0

go run ./cmd/contexts < required.txt
compared the required contexts against the check names this tree declares
  required by the ruleset: 0
  declared by the workflows: 25
  written down as deliberately absent: 25
note: dependency-review.yml: the check name "dependency-review" comes from the job id, because the job carries no name of its own
note: codeql.yml: the check name "CodeQL" is written in no workflow file and is expected here rather than read, so a change to the string it reports under is not caught by this comparison
note: zizmor.yml: the check name "zizmor" is written in no workflow file and is expected here rather than read, so a change to the string it reports under is not caught by this comparison
0 refusal(s), 3 note(s)
exit=0

The two refusal messages above are elided at the point where they repeat the
reason the absence list carries, which the run prints in full.

The means is Go, which is what every other check here is written in and what
record 0001 chose. It carries the three rules this tree holds itself to, it is
testable by the suite that already exists, and it adds no language, runtime or
dependency the tree does not already have. The one thing that is not Go is the
ruleset fetch, which is forced by the answer living on the platform rather than
in the checkout, and it is held to a single gh api call in the workflow.

Nobody else has read this change. The approving review count on this board is
zero, so nothing refuses that, and the evidence above is what stands in place of
a reader rather than something added beside one.

What this does not do

It cannot judge behaviour. A job that reports the right name having verified
nothing passes this comparison exactly like one that did its work, and the proof
that a check bites is the fixture its own suite carries rather than the string it
reports under.

It reads the ruleset as the API answered on the day it ran, so a context added
or removed by hand is caught on the next pull request rather than at the moment
it changes. Between those two moments the gate and this comparison disagree and
nothing says so.

Both limits are written in the package comment where the comparison is defined,
because a green result here is otherwise read as the gate being intact right
now.

It does not make anything required. Nothing in this tree holds a merge today,
which is issue #26, and this check goes into the required set with the rest when
that lands.

For the two names a code-scanning upload reports under, this is a statement that
the name is expected rather than a statement about what the tree says. The
string those two report under is decided by the analysis rather than by a line
anybody can change here, so the rename this whole check is about is not caught
for them. That is written at the list, and the report says which of the two
claims a reader is holding.

The reader of the workflow files understands the shape the workflows here are
written in and refuses anything else rather than guessing. That bound is stated
at the reader, and a workflow written differently reddens this check instead of
silently declaring the wrong string.

iderex added 2 commits August 12, 2026 12:44
The required set holds strings and the workflows declare strings, and nothing
on the platform ties the two together. A job renamed in passing removes itself
from the required set while the tab still looks green, and a context required
under a name nothing reports blocks every merge without saying why on the pull
request. Five issues in the plan carry that warning in different words and
nothing acted on it.

internal/contexts is the comparison. It is a function over two lists and reads
nothing, so every rule is proved against a fixture written out in full.
cmd/contexts is the entry point, and .github/workflows/contexts.yml is where
the ruleset is fetched, because half of this is a live setting rather than a
file and lab opens no network connection.

The deliberate absences are a list beside the comparison rather than a
paragraph in a document. Most of it is one fact repeated: the ruleset requires
no status check at all today, so every name this tree declares is outside the
set, and each is written down rather than the comparison switching itself off
while the set is empty. That is what catches the rename now: an absence names a
literal string, so a job renamed while the set is still empty leaves its entry
pointing at nothing and the check goes red.

Two of the refusals also run without a network, in the ordinary suite, so a
rename reddens the machine of whoever made it rather than waiting for the
pull request.

What a green result here does not say is written where the comparison is
defined: it cannot judge behaviour, and it reads the ruleset as the API
answered on the day it ran.

The means is Go, which is what every other check in this tree is written in and
what record 0001 chose. The fetch is the one thing that is not Go, and it is
held to a single gh call in the workflow because a ruleset is not in the
checkout.

Signed-off-by: Nils Lehnen <30603423+iderex@users.noreply.github.com>
The first commit read check names out of the workflow files and nothing else,
which is short by two. A code-scanning upload creates a check run of its own,
named after the analysis rather than after the job that uploaded it, and it is a
name a ruleset can require exactly like any other. Measured on the head commit
of this branch rather than supposed:

    gh api "repos/Flowfin/lab/commits/$(git rev-parse HEAD)/check-runs" --paginate \
      --jq '.check_runs[] | "\(.name)\t\(.app.slug)"' | sort -u

Twenty-four names, of which twenty-two come from github-actions and two, CodeQL
and zizmor, come from github-advanced-security.

Without this the comparison would refuse either of those as a required context
nothing reports, which is a false red on the change that assembles the required
set, and a false red is the failure this check exists to avoid rather than to
cause. The two names are written down beside the absences with the command that
found them, and what that costs is stated there: the string they report under is
decided by the analysis rather than by a line in this tree, so a change to it is
not caught here. The report says which of the two claims a reader is holding.

Signed-off-by: Nils Lehnen <30603423+iderex@users.noreply.github.com>
@iderex
iderex merged commit a5c5d9c into main Aug 12, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refuse a required check name that nothing reports, and the reverse

1 participant