Skip to content

Stop an npm registry outage from failing the whole pipeline - #212

Merged
VeryComplexAndLongName merged 1 commit into
mainfrom
ci/audit-own-job
Sep 4, 2026
Merged

Stop an npm registry outage from failing the whole pipeline#212
VeryComplexAndLongName merged 1 commit into
mainfrom
ci/audit-own-job

Conversation

@VeryComplexAndLongName

Copy link
Copy Markdown
Owner

The failure

npm audit was the last step of the quality job — and every other job
in this workflow declares needs: quality
. So an outage at npm's
advisory service skipped the merge gate, the extension integration suite
and the browser suite on pull requests that had nothing to do with
dependencies.

Observed twice in a row on #210:

04:59:17  npm run verify   ok, 263 tests passed
04:59:18  npm audit        started
05:06:19  npm warn audit 503 Service Unavailable - POST
          registry.npmjs.org/-/npm/v1/security/audits/quick
          npm error audit endpoint returned an error
          ##[error]The operation was canceled.

Seven minutes of retries against a failing endpoint, then the 10-minute
job limit, with the job's actual work already finished and green. The
third attempt, once the registry recovered, passed in 7m12s — still most
of the allowance inside one network call.

Two faults that were being conflated

npm audit exits non-zero for both of these, and they are not the same
claim:

Meaning Should it fail a check?
Found a high-severity advisory A real result Yes
Could not reach the service No result at all No

An audit that did not run says nothing about these dependencies — in
either direction.

What changed

npm audit moves into a dependency-audit job that declares no needs
and that nothing declares needs on, so its result decides only its own
check.

The move alone would not have been enough. Isolating the step without
separating the two faults would leave the same red check on an unrelated
pull request, just in a narrower place. So the report is read rather than
the exit status trusted:

  • a high or critical advisory fails the check;
  • an error response, unparseable output, a missing report, or absent
    vulnerability counts does not fail it, and emits a workflow warning
    saying the audit was not carried out;
  • timeout bounds the command itself, not just the job — a job timeout
    is a red check, the outcome this exists to prevent, so the command has
    to give up first for its result to be interpretable at all.

Absent counts are treated as unknown, never as zero — the same
distinction AuditEntry.usage draws between unreported and free. A check
that passes without having checked anything has to say so rather than
look clean.

Not weakened

A genuine high-severity advisory still fails its check. "Could not run"
being tolerated is not the same as a finding being tolerated, and
dependency-review continues to cover pull-request dependency changes
separately.

Verified, not reasoned about

The original failure was in the interpretation of an exit status, so
reasoning about the script without running it would have repeated it.

  • The interpreter was run against nine inputs — clean, high,
    moderate-only, a 503 error object, plain-text diagnostics, missing
    counts, an empty file, a missing file, and no argument — asserting each
    exit status. Only the "high" case exits 1.
  • The workflow was parsed to confirm quality no longer audits,
    dependency-audit declares no needs, and every other job's needs is
    byte-for-byte what it was.
  • The real audit ran end to end against this repository's lockfile:
    no high or critical advisories, interpreter exit 0.

Still to confirm by hand

tasks.md 3.6: on this pull request, that the checks list now shows
Dependency audit as its own entry, reporting independently of the
merge gate.

🤖 Generated with Claude Code

`npm audit` was the last step of the `quality` job, and every other job in
the workflow declares `needs: quality`. So an outage at npm's advisory
service skipped the merge gate, the extension integration suite and the
browser suite on pull requests that had nothing to do with dependencies.

Observed twice in a row on #210:

    04:59:17  npm run verify   ok, 263 tests passed
    04:59:18  npm audit        started
    05:06:19  npm warn audit 503 Service Unavailable - POST
              registry.npmjs.org/-/npm/v1/security/audits/quick
              ##[error]The operation was canceled.

Seven minutes of retries against a failing endpoint, then the 10-minute
job limit, with all of the job's actual work already finished and green.
The third attempt, once the registry recovered, passed the same job in
7m12s — still most of the allowance spent inside one network call.

The audit now runs in its own job that declares no `needs` and that
nothing declares `needs` on, so its result decides only its own check.

The move alone would not have been enough. `npm audit` exits non-zero
both when it finds a high-severity advisory and when it cannot reach the
service, and those are not the same claim: an audit that did not run says
nothing about these dependencies, in either direction. Isolating the step
without separating the two would have left the same red check on an
unrelated pull request, just in a narrower place. So the report is read
rather than the exit status trusted:

- a high or critical advisory fails the check;
- an error response, unparseable output, a missing report, or absent
  vulnerability counts does not fail it, and says in a workflow warning
  that the audit was not carried out;
- `timeout` bounds the command itself and not just the job, because a job
  timeout is a red check — the outcome this exists to prevent — so the
  command has to give up first for its result to be interpretable at all.

Absent counts are treated as unknown, never as zero, for the same reason
`AuditEntry.usage` distinguishes unreported from free: a check that passes
without having checked anything must say so rather than look clean.

Verified rather than reasoned about, since the original failure was in the
interpretation of an exit status: the interpreter was run against nine
inputs — clean, high, moderate-only, a 503 error object, plain-text
diagnostics, missing counts, an empty file, a missing file, and no
argument — asserting each exit status; the workflow was parsed to confirm
`quality` no longer audits, `dependency-audit` gates on nothing, and every
other job's `needs` is unchanged; and the real audit was run end to end
against this repository's lockfile.

No changeset: CI configuration, nothing published changes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@VeryComplexAndLongName
VeryComplexAndLongName merged commit ae94c5f into main Sep 4, 2026
8 checks passed
@VeryComplexAndLongName
VeryComplexAndLongName deleted the ci/audit-own-job branch September 4, 2026 05:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant