Skip to content

v0.12.0

Choose a tag to compare

@github-actions github-actions released this 12 Nov 16:01
· 31 commits to main since this release
Immutable release. Only release title and notes can be modified.
be32abe

Moving to v0.12.0

v0.12.0 adds two rules that can turn a repository red at its bump (job-timeout and runtime-install), one report-only rule (catalog-check), and widens pytest-workers, a rule released in v0.11.0. It adds an optional input to image.yml and changes the closing words of a verdict line. This bump is not pins only: a repository moves its pins, runs quality check, and then fixes what the debt and workflows rows show.

Pin form

uses: overnightworks/quality/.github/workflows/gate.yml@be32abeb120d29f75ee7ea456d6e510ae001de04 # v0.12.0
  • codeql.yml takes the same pin.
  • The comment is load-bearing: it must name the tag of the commit before it.
  • Release workflows (release-gate.yml, image.yml, release-tag.yml) pinned below v0.12.0 move to the same commit, with the comment.
  • A repository that has not yet moved to v0.11.0 or v0.11.1 moves straight to v0.12.0 and reads v0.11.0's page for the three rules it meets there (agents-md, pr-concurrency, pytest-workers).

What changed

job-timeout: every job declares a limit of at most 15 minutes

New rule, area debt. No configuration switches it off. It is described in docs/rules.md.

  • Judged jobs. Every job of a workflow that runs on a pull request or on a push to main, and of the reusable workflows those call through a local uses: (./ or $/ and a tracked .github/workflows/<file>.yml). A workflow is a root when on: names pull_request, pull_request_target or pull_request_review, or a push that can reach main (a push with only tags is no root). A job that has uses: is not judged itself; a matrix job is one job.

  • Outside. A workflow no root reaches (release and image workflows stay outside), remote reusable workflows, composite actions. A step's timeout-minutes is not read, no sum is taken and no matrix factor is applied.

  • The three findings, at the file that defines the job:

    Event Fact Fix
    missing job <id> has no timeout-minutes set timeout-minutes to a number of at most 15
    invalid job <id> has a timeout-minutes that is not a number above zero set timeout-minutes to a number of at most 15
    new debt job <id> has timeout-minutes <value> · limit 15 lower timeout-minutes to 15 or less

    Where the baseline has no [job_minutes] table, the new debt fix reads lower timeout-minutes to 15 or less, or run quality check --fix to record it. Only a plain unquoted decimal number above zero is a limit; an expression, a quoted number, 0, a negative or a null is invalid. 15.5 is above 15.

  • Cannot judge (exit 2, naming the file): a workflow or called file that cannot be read, an alias or merge key where it can hide a judged key, a local call that names no tracked workflow. The other rules still judge.

  • The way out for a job that needs more than 15 minutes. quality check --fix records the [job_minutes] table once in quality/baseline.toml, where git never held one: exactly the judged jobs above 15, each at its limit rounded up (17.5 as 18), keyed by the workflow file that defines the job. The debt row reads recorded K job limits · quality/baseline.toml. Commit the file.

  • After the table exists, a job at its entry is green; above its entry it is grew (fix: bring it back to <entry> or below); below it, can fall (fix: quality check --fix). A job above 15 with no entry stays new debt: --fix never adds an entry and never raises one.

  • --fix also lowers an entry to its job's rounded-up limit, drops an entry that is no debt or whose job is gone, and moves a workflow's entries to its new path when git reads the workflow as renamed. A workflow that cannot be read refuses the whole fix while an entry is held (✗ not fixed, exit 2, nothing written). A ceiling file that lost a table git holds is refused with the restore step. --dry-run words the same rows as what the fix would do.

  • Loosenings in the verdict. A pull request that raises or adds a job limit is a loosening, named under the loosenings row, green only with a declared loosening or the operator's door:

    • job limit baseline raises .github/workflows/ci.yml e2e 40 -> 45
    • job limit baseline adds .github/workflows/ci.yml e2e 40
    • job limit baseline removed

    A [job_minutes] table the base lacks is an adoption and compares nothing.

  • An older quality refuses the table as unknown. A caller that moves to v0.12.0 and records the table cannot run an older quality on that checkout.

  • The rule is judged with or without a ceiling file. Without one, new debt is still a finding.

  • What a caller changes: every judged job gets timeout-minutes of at most 15, or the table is recorded once with quality check --fix for jobs that must stay above 15. See the checklist.

runtime-install: no operating-system installs, no download piped into a shell

New rule, area workflows. No configuration switches it off. It judges every run: step of every tracked workflow, whether or not a pull request triggers it. Scripts a step calls, composite actions, uses: steps (a with: command: included) and Dockerfiles are not read.

  • A finding is a call to an operating-system package manager that installs, or a download piped into a shell.
    • Managers read: apt-get, apt, dpkg -i, dnf, yum, apk, brew, snap, playwright install --with-deps (or install-deps), and pacman, zypper, choco, winget, port, each with its install words listed in docs/rules.md. update of apt-get, apt, apk and brew installs nothing and is no finding; remove, purge and clean are none either.
    • A download into a shell: curl url | sh, bash <(curl url), sh -c "$(curl url)", with curl or wget as downloader and sh, bash, dash, zsh or ksh as shell.
  • Row. Event installs, fact the command and the matched word (apt-get install, dpkg -i, playwright install --with-deps) or a download runs as a shell script. A call the reader cannot read is invalid (command cannot be read to its end, command is built at run time, an install cannot be read), only when the line holds a manager and one of its install words.
  • Fix line: take the tool from the lockfile, the runner image or an image pinned by digest, or explain on the line above: # runtime-install: <reason>
  • The routes: the repository's lockfile, the runner image, or a container image pinned by digest.
  • The marker. # runtime-install: <reason> (a reason with a letter) is for a step that proves an install on a fresh system, and for nothing else. Any other use is a deviation from the fleet standard, which only the operator rules. Quality cannot see a step's purpose; the review of the dependent's diff judges it. Written on the line directly above a step's first line, with its # in the column of the step's -, it clears every finding of that step. Written on a call's line or directly above it, it clears that call. A pytest-workers marker clears no runtime-install finding and the other way round.
  • Named gaps (read as no call): a download and its run in two steps, eval "$(curl url)", a pipe into python or perl, wrappers such as make, tox, docker run, and unlocked tool installs (pip install, npm install -g, cargo install, go install).
  • Known false alarms, not cleared by the marker: a heredoc that writes a Dockerfile with a chained RUN apt-get update && apt-get install -y x, and a dry run (apt-get -s install, --dry-run, --print-uris), which read as real calls. docs/rules.md lists them whole.
  • What a caller changes: take each tool from the lockfile, the runner image or a digest-pinned image; add the marker only to a step that proves an install.

pytest-workers reads more

The rule released in v0.11.0 now reads these spellings of a pytest call. A call there without -n was no call before and is a finding now (serial, or invalid where -n has no usable value).

  • Wrappers: timeout (with -k, -s and a duration), nice (with -n) and xargs (with its flags) are read through. timeout 600 pytest -q is a call.
  • A shell's -c string (sh, bash, dash, zsh, ksh), read whole as a script of its own: bash -c 'pytest -q' is a call.
  • uvx tool@version reads as uvx tool: uvx pytest@8 -q is a call.
  • Runners: npx, bunx, pnpm exec, pnpm dlx, npm exec and yarn start a pytest call too, and a trailing @version of the tool word is dropped. yarn dlx, yarn exec, pnpm <bin> and bun x stay named gaps.
  • Findings, facts, fix line and the # pytest-workers: <reason> marker are unchanged.
  • What a caller changes: -n auto on each newly read call, or the marker with a reason.

catalog-check: a note for a repository that declares the design catalogs

New rule, area workflows, report-only. It never turns a run red.

  • A repository is covered when its root pyproject.toml declares overnightworks-design with the extra catalogs (in [project] dependencies, an optional-dependencies group or a dependency group). Any other repository sees nothing.

  • A covered repository whose pull-request workflows have no run: step that calls design-catalogs --check directly gets one note row and exit 0. The workflows row stays clean.

  • The note, as printed:

    · .github/workflows  not called  no workflow step runs design-catalogs --check directly on a pull request; a call through make is not read yet
    
  • A call through make, tox, nox, a script, a composite action or a reusable workflow is not read yet and gets the note. A step with continue-on-error: true or a working-directory other than . does not count.

  • A later release reads a call through make and then enforces: the note becomes a finding in that release.

  • What a caller changes: nothing. A repository whose pull-request workflow runs the check through make keeps it there: the release that enforces reads that call.

image.yml takes reproducible-layers

New optional input, default false. A caller that does not set it builds exactly as before.

  • With true, the build sets SOURCE_DATE_EPOCH to the constant 0 and the image output to type=image,rewrite-timestamp=true. Every file in the layers gets one fixed time, so a layer whose inputs did not change keeps its digest from one release to the next.
  • What a caller does in its Dockerfile. The workflow fixes times, not bytes. A layer repeats only if the Dockerfile writes no logs, caches or wall-clock data into it. A Dockerfile that declares ARG SOURCE_DATE_EPOCH reads 0. A SOURCE_DATE_EPOCH in the caller's build-args wins over the constant.
  • The date. The image's created, the history entries of the build and the annotation org.opencontainers.image.created read 1970-01-01. A filter or a sort by image age (a registry pruning rule, for example) treats the image as old.
  • Not pinned: BuildKit. A newer BuildKit can assemble a layer differently once, and its digest then changes.
  • Not yet proven: the push path on a large image (provenance, SBOM, attestation, build time, runner disk) is proven by the first caller's release with the input on.
  • What a caller changes: nothing unless it wants the input. Set reproducible-layers: true in the image.yml call to opt in.

quality check --fix and the closing words

  • --fix records the job table where git never held one; see job-timeout above. It also lowers, drops and moves job entries. A ceiling or table that git holds and the tree lacks is refused with the restore step.
  • Closing words changed. The verdict line says debt ceiling measured, not enforced where it said debt measured, not enforced, and debt ceiling partly measured, not enforced where it said debt partly measured, not enforced. A caller, script or tool that matches the old words changes its match.
  • Without a ceiling file the size and clone ceilings are a grey note with exit 0. job-timeout is judged with or without a ceiling file, so the verdict can show debt ceiling measured, not enforced while job-timeout finds.
  • What a caller changes: a match on the closing words, if it has one.

Internal

  • New modules, no behaviour change. What a caller could trip over: quality/baseline.toml may now hold a [job_minutes] table, which an older quality refuses (see job-timeout).

What a caller changes at its bump

  • Move the pin in the gate.yml caller and in the codeql.yml caller to the v0.12.0 commit, each with the # v0.12.0 comment.
  • Move the pins of release-gate.yml, image.yml and release-tag.yml to the same commit, with the comment.
  • Run quality check at the new pin and read the debt and workflows rows.
  • job-timeout: give every judged job timeout-minutes of at most 15 (a plain number, not an expression or a quoted value). Check the local reusable workflows your pull-request and push-to-main workflows call.
  • quality check --fix once, if a job must stay above 15 minutes. Read the recorded K job limits row and commit quality/baseline.toml. A later job above 15 without an entry is new debt; raising an entry needs a declared loosening.
  • runtime-install: move each OS package install and each download piped into a shell to the lockfile, the runner image or an image pinned by digest.
  • runtime-install marker: add # runtime-install: <reason> only to a step that proves an install on a fresh system. Any other use is a deviation only the operator rules.
  • pytest-workers: pass -n auto in each pytest call behind timeout, nice, xargs, a shell's -c string, uvx pytest@version, npx, bunx, pnpm exec, pnpm dlx, npm exec or yarn, or write # pytest-workers: <reason> on the line or directly above it.
  • If a script or tool matches debt measured, not enforced, change it to debt ceiling measured, not enforced and debt ceiling partly measured, not enforced.
  • catalog-check: nothing to change. A repository that declares overnightworks-design[catalogs] may see the note; nothing turns red.
  • image.yml: the input reproducible-layers is optional. Set it only with the Dockerfile and the image-age filter in view.

What is not in it

  • catalog-check does not read a call through make yet, and it does not enforce; a later release does both.
  • Unlocked tool installs in CI (pip install, npm install -g, cargo install, go install) are not judged yet (quality#805).
  • The job limits of remote reusable workflows that only another repository calls are judged by no rule yet (quality#800).
  • literal-text and render-pass are not in this release.

Older pins

A gate pinned below v0.12.0 has none of this. It does not carry job-timeout, runtime-install or catalog-check; its pytest-workers does not read the wrappers, -c strings, uvx tool@version or the runners above; it refuses a [job_minutes] table as unknown; its verdict line says debt measured, not enforced; and its image.yml has no reproducible-layers input.

What's Changed

  • Read through timeout, nice, xargs and a shell's -c string in pytest-workers by @FlexOr2 in #796
  • Add the reproducible-layers input to the image workflow by @FlexOr2 in #799
  • Judge every job's timeout-minutes (job-timeout, first half) by @FlexOr2 in #798
  • Read the job_minutes table in job-timeout and the verdict by @FlexOr2 in #801
  • Follow a renamed workflow's job entries in quality check --fix by @FlexOr2 in #804
  • Judge installs and downloads run as a shell in run steps (runtime-install) by @FlexOr2 in #802
  • Lower and drop held job entries in quality check --fix by @FlexOr2 in #808
  • Report covered repositories whose pull-request workflows never call design-catalogs --check by @FlexOr2 in #809
  • Record the job_minutes table in quality check --fix where git never held one by @FlexOr2 in #811
  • Name v0.12.0 as the first release of three rules and pin gate.yml to it by @FlexOr2 in #814

Full Changelog: v0.11.1...v0.12.0