Repository navigation
v0.12.0
Moving to v0.12.0
v0.12.0 adds two rules that can turn a repository red at its bump (job-timeout and runtime-install), one report-only rule (catalog-check), and widens pytest-workers, a rule released in v0.11.0. It adds an optional input to image.yml and changes the closing words of a verdict line. This bump is not pins only: a repository moves its pins, runs quality check, and then fixes what the debt and workflows rows show.
Pin form
uses: overnightworks/quality/.github/workflows/gate.yml@be32abeb120d29f75ee7ea456d6e510ae001de04 # v0.12.0codeql.ymltakes the same pin.- The comment is load-bearing: it must name the tag of the commit before it.
- Release workflows (
release-gate.yml,image.yml,release-tag.yml) pinned below v0.12.0 move to the same commit, with the comment. - A repository that has not yet moved to v0.11.0 or v0.11.1 moves straight to v0.12.0 and reads v0.11.0's page for the three rules it meets there (
agents-md,pr-concurrency,pytest-workers).
What changed
job-timeout: every job declares a limit of at most 15 minutes
New rule, area debt. No configuration switches it off. It is described in docs/rules.md.
-
Judged jobs. Every job of a workflow that runs on a pull request or on a push to main, and of the reusable workflows those call through a local
uses:(./or$/and a tracked.github/workflows/<file>.yml). A workflow is a root whenon:namespull_request,pull_request_targetorpull_request_review, or apushthat can reachmain(apushwith onlytagsis no root). A job that hasuses:is not judged itself; a matrix job is one job. -
Outside. A workflow no root reaches (release and image workflows stay outside), remote reusable workflows, composite actions. A step's
timeout-minutesis not read, no sum is taken and no matrix factor is applied. -
The three findings, at the file that defines the job:
Event Fact Fix missingjob <id> has no timeout-minutesset timeout-minutes to a number of at most 15invalidjob <id> has a timeout-minutes that is not a number above zeroset timeout-minutes to a number of at most 15new debtjob <id> has timeout-minutes <value> · limit 15lower timeout-minutes to 15 or lessWhere the baseline has no
[job_minutes]table, thenew debtfix readslower timeout-minutes to 15 or less, or run quality check --fix to record it. Only a plain unquoted decimal number above zero is a limit; an expression, a quoted number,0, a negative or a null isinvalid.15.5is above 15. -
Cannot judge (exit 2, naming the file): a workflow or called file that cannot be read, an alias or merge key where it can hide a judged key, a local call that names no tracked workflow. The other rules still judge.
-
The way out for a job that needs more than 15 minutes.
quality check --fixrecords the[job_minutes]table once inquality/baseline.toml, where git never held one: exactly the judged jobs above 15, each at its limit rounded up (17.5as18), keyed by the workflow file that defines the job. The debt row readsrecorded K job limits · quality/baseline.toml. Commit the file. -
After the table exists, a job at its entry is green; above its entry it is
grew(fix:bring it back to <entry> or below); below it,can fall(fix:quality check --fix). A job above 15 with no entry staysnew debt:--fixnever adds an entry and never raises one. -
--fixalso lowers an entry to its job's rounded-up limit, drops an entry that is no debt or whose job is gone, and moves a workflow's entries to its new path when git reads the workflow as renamed. A workflow that cannot be read refuses the whole fix while an entry is held (✗ not fixed, exit 2, nothing written). A ceiling file that lost a table git holds is refused with the restore step.--dry-runwords the same rows as what the fix would do. -
Loosenings in the verdict. A pull request that raises or adds a job limit is a loosening, named under the
looseningsrow, green only with a declared loosening or the operator's door:job limit baseline raises .github/workflows/ci.yml e2e 40 -> 45job limit baseline adds .github/workflows/ci.yml e2e 40job limit baseline removed
A
[job_minutes]table the base lacks is an adoption and compares nothing. -
An older quality refuses the table as unknown. A caller that moves to v0.12.0 and records the table cannot run an older
qualityon that checkout. -
The rule is judged with or without a ceiling file. Without one,
new debtis still a finding. -
What a caller changes: every judged job gets
timeout-minutesof at most 15, or the table is recorded once withquality check --fixfor jobs that must stay above 15. See the checklist.
runtime-install: no operating-system installs, no download piped into a shell
New rule, area workflows. No configuration switches it off. It judges every run: step of every tracked workflow, whether or not a pull request triggers it. Scripts a step calls, composite actions, uses: steps (a with: command: included) and Dockerfiles are not read.
- A finding is a call to an operating-system package manager that installs, or a download piped into a shell.
- Managers read:
apt-get,apt,dpkg -i,dnf,yum,apk,brew,snap,playwright install --with-deps(orinstall-deps), andpacman,zypper,choco,winget,port, each with its install words listed indocs/rules.md.updateofapt-get,apt,apkandbrewinstalls nothing and is no finding;remove,purgeandcleanare none either. - A download into a shell:
curl url | sh,bash <(curl url),sh -c "$(curl url)", withcurlorwgetas downloader andsh,bash,dash,zshorkshas shell.
- Managers read:
- Row. Event
installs, fact the command and the matched word (apt-get install,dpkg -i,playwright install --with-deps) ora download runs as a shell script. A call the reader cannot read isinvalid(command cannot be read to its end,command is built at run time, an install cannot be read), only when the line holds a manager and one of its install words. - Fix line:
take the tool from the lockfile, the runner image or an image pinned by digest, or explain on the line above: # runtime-install: <reason> - The routes: the repository's lockfile, the runner image, or a container image pinned by digest.
- The marker.
# runtime-install: <reason>(a reason with a letter) is for a step that proves an install on a fresh system, and for nothing else. Any other use is a deviation from the fleet standard, which only the operator rules. Quality cannot see a step's purpose; the review of the dependent's diff judges it. Written on the line directly above a step's first line, with its#in the column of the step's-, it clears every finding of that step. Written on a call's line or directly above it, it clears that call. Apytest-workersmarker clears noruntime-installfinding and the other way round. - Named gaps (read as no call): a download and its run in two steps,
eval "$(curl url)", a pipe intopythonorperl, wrappers such asmake,tox,docker run, and unlocked tool installs (pip install,npm install -g,cargo install,go install). - Known false alarms, not cleared by the marker: a heredoc that writes a Dockerfile with a chained
RUN apt-get update && apt-get install -y x, and a dry run (apt-get -s install,--dry-run,--print-uris), which read as real calls.docs/rules.mdlists them whole. - What a caller changes: take each tool from the lockfile, the runner image or a digest-pinned image; add the marker only to a step that proves an install.
pytest-workers reads more
The rule released in v0.11.0 now reads these spellings of a pytest call. A call there without -n was no call before and is a finding now (serial, or invalid where -n has no usable value).
- Wrappers:
timeout(with-k,-sand a duration),nice(with-n) andxargs(with its flags) are read through.timeout 600 pytest -qis a call. - A shell's
-cstring (sh,bash,dash,zsh,ksh), read whole as a script of its own:bash -c 'pytest -q'is a call. uvx tool@versionreads asuvx tool:uvx pytest@8 -qis a call.- Runners:
npx,bunx,pnpm exec,pnpm dlx,npm execandyarnstart a pytest call too, and a trailing@versionof the tool word is dropped.yarn dlx,yarn exec,pnpm <bin>andbun xstay named gaps. - Findings, facts, fix line and the
# pytest-workers: <reason>marker are unchanged. - What a caller changes:
-n autoon each newly read call, or the marker with a reason.
catalog-check: a note for a repository that declares the design catalogs
New rule, area workflows, report-only. It never turns a run red.
-
A repository is covered when its root
pyproject.tomldeclaresovernightworks-designwith the extracatalogs(in[project] dependencies, an optional-dependencies group or a dependency group). Any other repository sees nothing. -
A covered repository whose pull-request workflows have no
run:step that callsdesign-catalogs --checkdirectly gets one note row and exit 0. Theworkflowsrow stays clean. -
The note, as printed:
· .github/workflows not called no workflow step runs design-catalogs --check directly on a pull request; a call through make is not read yet -
A call through
make,tox,nox, a script, a composite action or a reusable workflow is not read yet and gets the note. A step withcontinue-on-error: trueor aworking-directoryother than.does not count. -
A later release reads a call through
makeand then enforces: the note becomes a finding in that release. -
What a caller changes: nothing. A repository whose pull-request workflow runs the check through
makekeeps it there: the release that enforces reads that call.
image.yml takes reproducible-layers
New optional input, default false. A caller that does not set it builds exactly as before.
- With
true, the build setsSOURCE_DATE_EPOCHto the constant0and the image output totype=image,rewrite-timestamp=true. Every file in the layers gets one fixed time, so a layer whose inputs did not change keeps its digest from one release to the next. - What a caller does in its Dockerfile. The workflow fixes times, not bytes. A layer repeats only if the Dockerfile writes no logs, caches or wall-clock data into it. A Dockerfile that declares
ARG SOURCE_DATE_EPOCHreads0. ASOURCE_DATE_EPOCHin the caller'sbuild-argswins over the constant. - The date. The image's
created, the history entries of the build and the annotationorg.opencontainers.image.createdread1970-01-01. A filter or a sort by image age (a registry pruning rule, for example) treats the image as old. - Not pinned: BuildKit. A newer BuildKit can assemble a layer differently once, and its digest then changes.
- Not yet proven: the push path on a large image (provenance, SBOM, attestation, build time, runner disk) is proven by the first caller's release with the input on.
- What a caller changes: nothing unless it wants the input. Set
reproducible-layers: truein theimage.ymlcall to opt in.
quality check --fix and the closing words
--fixrecords the job table where git never held one; seejob-timeoutabove. It also lowers, drops and moves job entries. A ceiling or table that git holds and the tree lacks is refused with the restore step.- Closing words changed. The verdict line says
debt ceiling measured, not enforcedwhere it saiddebt measured, not enforced, anddebt ceiling partly measured, not enforcedwhere it saiddebt partly measured, not enforced. A caller, script or tool that matches the old words changes its match. - Without a ceiling file the size and clone ceilings are a grey note with exit 0.
job-timeoutis judged with or without a ceiling file, so the verdict can showdebt ceiling measured, not enforcedwhilejob-timeoutfinds. - What a caller changes: a match on the closing words, if it has one.
Internal
- New modules, no behaviour change. What a caller could trip over:
quality/baseline.tomlmay now hold a[job_minutes]table, which an older quality refuses (seejob-timeout).
What a caller changes at its bump
- Move the pin in the
gate.ymlcaller and in thecodeql.ymlcaller to the v0.12.0 commit, each with the# v0.12.0comment. - Move the pins of
release-gate.yml,image.ymlandrelease-tag.ymlto the same commit, with the comment. - Run
quality checkat the new pin and read thedebtandworkflowsrows. -
job-timeout: give every judged jobtimeout-minutesof at most 15 (a plain number, not an expression or a quoted value). Check the local reusable workflows your pull-request and push-to-main workflows call. -
quality check --fixonce, if a job must stay above 15 minutes. Read therecorded K job limitsrow and commitquality/baseline.toml. A later job above 15 without an entry isnew debt; raising an entry needs a declared loosening. -
runtime-install: move each OS package install and each download piped into a shell to the lockfile, the runner image or an image pinned by digest. -
runtime-installmarker: add# runtime-install: <reason>only to a step that proves an install on a fresh system. Any other use is a deviation only the operator rules. -
pytest-workers: pass-n autoin each pytest call behindtimeout,nice,xargs, a shell's-cstring,uvx pytest@version,npx,bunx,pnpm exec,pnpm dlx,npm execoryarn, or write# pytest-workers: <reason>on the line or directly above it. - If a script or tool matches
debt measured, not enforced, change it todebt ceiling measured, not enforcedanddebt ceiling partly measured, not enforced. -
catalog-check: nothing to change. A repository that declaresovernightworks-design[catalogs]may see the note; nothing turns red. -
image.yml: the inputreproducible-layersis optional. Set it only with the Dockerfile and the image-age filter in view.
What is not in it
catalog-checkdoes not read a call throughmakeyet, and it does not enforce; a later release does both.- Unlocked tool installs in CI (
pip install,npm install -g,cargo install,go install) are not judged yet (quality#805). - The job limits of remote reusable workflows that only another repository calls are judged by no rule yet (quality#800).
literal-textandrender-passare not in this release.
Older pins
A gate pinned below v0.12.0 has none of this. It does not carry job-timeout, runtime-install or catalog-check; its pytest-workers does not read the wrappers, -c strings, uvx tool@version or the runners above; it refuses a [job_minutes] table as unknown; its verdict line says debt measured, not enforced; and its image.yml has no reproducible-layers input.
What's Changed
- Read through timeout, nice, xargs and a shell's -c string in pytest-workers by @FlexOr2 in #796
- Add the reproducible-layers input to the image workflow by @FlexOr2 in #799
- Judge every job's timeout-minutes (job-timeout, first half) by @FlexOr2 in #798
- Read the job_minutes table in job-timeout and the verdict by @FlexOr2 in #801
- Follow a renamed workflow's job entries in quality check --fix by @FlexOr2 in #804
- Judge installs and downloads run as a shell in run steps (runtime-install) by @FlexOr2 in #802
- Lower and drop held job entries in quality check --fix by @FlexOr2 in #808
- Report covered repositories whose pull-request workflows never call design-catalogs --check by @FlexOr2 in #809
- Record the job_minutes table in quality check --fix where git never held one by @FlexOr2 in #811
- Name v0.12.0 as the first release of three rules and pin gate.yml to it by @FlexOr2 in #814
Full Changelog: v0.11.1...v0.12.0