Stop an npm registry outage from failing the whole pipeline - #212
Merged
Conversation
`npm audit` was the last step of the `quality` job, and every other job in the workflow declares `needs: quality`. So an outage at npm's advisory service skipped the merge gate, the extension integration suite and the browser suite on pull requests that had nothing to do with dependencies. Observed twice in a row on #210: 04:59:17 npm run verify ok, 263 tests passed 04:59:18 npm audit started 05:06:19 npm warn audit 503 Service Unavailable - POST registry.npmjs.org/-/npm/v1/security/audits/quick ##[error]The operation was canceled. Seven minutes of retries against a failing endpoint, then the 10-minute job limit, with all of the job's actual work already finished and green. The third attempt, once the registry recovered, passed the same job in 7m12s — still most of the allowance spent inside one network call. The audit now runs in its own job that declares no `needs` and that nothing declares `needs` on, so its result decides only its own check. The move alone would not have been enough. `npm audit` exits non-zero both when it finds a high-severity advisory and when it cannot reach the service, and those are not the same claim: an audit that did not run says nothing about these dependencies, in either direction. Isolating the step without separating the two would have left the same red check on an unrelated pull request, just in a narrower place. So the report is read rather than the exit status trusted: - a high or critical advisory fails the check; - an error response, unparseable output, a missing report, or absent vulnerability counts does not fail it, and says in a workflow warning that the audit was not carried out; - `timeout` bounds the command itself and not just the job, because a job timeout is a red check — the outcome this exists to prevent — so the command has to give up first for its result to be interpretable at all. Absent counts are treated as unknown, never as zero, for the same reason `AuditEntry.usage` distinguishes unreported from free: a check that passes without having checked anything must say so rather than look clean. Verified rather than reasoned about, since the original failure was in the interpretation of an exit status: the interpreter was run against nine inputs — clean, high, moderate-only, a 503 error object, plain-text diagnostics, missing counts, an empty file, a missing file, and no argument — asserting each exit status; the workflow was parsed to confirm `quality` no longer audits, `dependency-audit` gates on nothing, and every other job's `needs` is unchanged; and the real audit was run end to end against this repository's lockfile. No changeset: CI configuration, nothing published changes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The failure
npm auditwas the last step of thequalityjob — and every other jobin this workflow declares
needs: quality. So an outage at npm'sadvisory service skipped the merge gate, the extension integration suite
and the browser suite on pull requests that had nothing to do with
dependencies.
Observed twice in a row on #210:
Seven minutes of retries against a failing endpoint, then the 10-minute
job limit, with the job's actual work already finished and green. The
third attempt, once the registry recovered, passed in 7m12s — still most
of the allowance inside one network call.
Two faults that were being conflated
npm auditexits non-zero for both of these, and they are not the sameclaim:
An audit that did not run says nothing about these dependencies — in
either direction.
What changed
npm auditmoves into adependency-auditjob that declares noneedsand that nothing declares
needson, so its result decides only its owncheck.
The move alone would not have been enough. Isolating the step without
separating the two faults would leave the same red check on an unrelated
pull request, just in a narrower place. So the report is read rather than
the exit status trusted:
vulnerability counts does not fail it, and emits a workflow warning
saying the audit was not carried out;
timeoutbounds the command itself, not just the job — a job timeoutis a red check, the outcome this exists to prevent, so the command has
to give up first for its result to be interpretable at all.
Absent counts are treated as unknown, never as zero — the same
distinction
AuditEntry.usagedraws between unreported and free. A checkthat passes without having checked anything has to say so rather than
look clean.
Not weakened
A genuine high-severity advisory still fails its check. "Could not run"
being tolerated is not the same as a finding being tolerated, and
dependency-reviewcontinues to cover pull-request dependency changesseparately.
Verified, not reasoned about
The original failure was in the interpretation of an exit status, so
reasoning about the script without running it would have repeated it.
moderate-only, a 503 error object, plain-text diagnostics, missing
counts, an empty file, a missing file, and no argument — asserting each
exit status. Only the "high" case exits 1.
qualityno longer audits,dependency-auditdeclares noneeds, and every other job'sneedsisbyte-for-byte what it was.
no high or critical advisories, interpreter exit 0.Still to confirm by hand
tasks.md3.6: on this pull request, that the checks list now showsDependency audit as its own entry, reporting independently of the
merge gate.
🤖 Generated with Claude Code