Improvement to flub check trustPolicy: report why a particular package version is flagged as a trust downgrade#27254
Open
TommyBrosman wants to merge 6 commits intomicrosoft:mainfrom
Conversation
Extends the `flub check trustPolicy` command so that each `ERR_PNPM_TRUST_POLICY_VIOLATION` reported by pnpm is enriched with publish metadata fetched directly from the registry, making it possible to triage violations without manually digging through `npm view` output.
For every violation the report now shows:
- **Offender**: publish timestamp, publishing account, and trust evidence kind (`none` / `provenance` / `trustedPublisher`) — matching pnpm's own `prettyPrintTrustEvidence` formatting.
- **Prior trusted**: the most recent earlier-published version of the same package that carried trust evidence (the version pnpm's policy is comparing against).
Why
`no-downgrade` violations only tell you that trust regressed; they don't tell you _from what_, _to what_, or _who published the offending version_. That information is what actually decides whether a violation is a publish-pipeline regression we can wave through, an upstream maintainer change, or a supply-chain concern worth blocking on.
How
- Registry access goes through `pacote.packument(name, { where: workspaceDir, fullMetadata: true })`. `where` makes pacote honor the audited workspace's `.npmrc` (scoped registries, `_authToken` / `_auth`), and `fullMetadata: true` is required because abbreviated packuments strip `_npmUser`, `dist.attestations`, and `time`.
- Trust-evidence ranking and the publish-time-ordered prior-version walk mirror pnpm's `getTrustEvidence` / `detectStrongestTrustEvidenceBeforeDate`, but retain the version identity that pnpm discards.
- Enrichment is best-effort: any per-package fetch/parse failure is recorded on `violation.enrichmentError` and the report falls back to pnpm's original `hint`, so one bad packument can't abort the batch.
- Sequential per-violation: violation lists are small, pacote's `packumentCache` already memoizes per-name, and registries can rate-limit.
Dependencies
Adds `pacote@^21` and `@types/pacote` to `build-tools/packages/build-cli`.
Validation
Run on the FluidFramework root surfaces 8 active violations (e.g. `langchain`, `semver@5.7.2`, etc.) with offender + prior-trusted lines populated. Behavior under registry/auth/network failure was exercised by forcing enrichment errors and confirming the fallback path.
Risk
Self-contained to a single `flub check` subcommand; no runtime/product code paths affected.
Contributor
|
Hi! Thank you for opening this PR. Want me to review it? Based on the diff (1333 lines, 4 files), I've queued these reviewers:
How this works
|
- Added root attribution for violations.
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Enhances flub check trustPolicy output by enriching pnpm trust-policy downgrade violations with registry publish metadata to make violations easier to triage.
Changes:
- Add registry packument enrichment (publisher, publish time, trust evidence, and “prior trusted” version) for each trust-policy violation.
- Track and report workspace “roots” that reach each violating
name@version. - Update pnpm execution to buffer output (no live streaming) and add
pacotedependency for registry access.
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| build-tools/packages/build-cli/src/commands/check/trustPolicy.ts | Adds violation enrichment via pacote, root attribution from pnpm list, and richer violation reporting. |
| build-tools/packages/build-cli/package.json | Adds pacote / @types/pacote to support registry metadata fetch. |
| build-tools/packages/build-cli/docs/build-perf.md | Updates CLI examples’ line continuation characters. |
Files not reviewed (1)
- build-tools/pnpm-lock.yaml: Language not supported
Comment on lines
+748
to
+771
| const offenderTime = Date.parse(offenderTimeStr); | ||
| const offenderIsPrerelease = semver.prerelease(v.version) !== null; | ||
| let bestTime = -Infinity; | ||
| // Walk every published version: keep ones predating the offender | ||
| // that carry trust evidence, pick the most recently published. | ||
| // Mirrors pnpm's `detectStrongestTrustEvidenceBeforeDate` but | ||
| // retains the specific version (pnpm only returns the evidence kind). | ||
| for (const [cVersion, cTimeStr] of Object.entries(pkg.time ?? {})) { | ||
| if (cVersion === "created" || cVersion === "modified") continue; | ||
| const cManifest = versions[cVersion]; | ||
| if (cManifest === undefined) continue; | ||
| if (!offenderIsPrerelease && semver.prerelease(cVersion) !== null) continue; | ||
| const cTime = Date.parse(cTimeStr); | ||
| if (Number.isNaN(cTime) || cTime >= offenderTime || cTime <= bestTime) continue; | ||
| const evidence = getTrustEvidence(cManifest); | ||
| if (evidence === "none") continue; | ||
| bestTime = cTime; | ||
| v.priorTrusted = { | ||
| version: cVersion, | ||
| publishedAt: cTimeStr, | ||
| publisher: cManifest._npmUser?.name, | ||
| evidence, | ||
| }; | ||
| } |
| v.publisher = offender._npmUser?.name; | ||
| v.evidence = getTrustEvidence(offender); | ||
|
|
||
| const offenderTime = Date.parse(offenderTimeStr); |
| const rootSet = rootsByKey.get(key); | ||
| const roots = | ||
| rootSet === undefined || rootSet.size === 0 ? undefined : [...rootSet].sort(); | ||
| violations.push({ name, version, ...reason, roots }); |
The 'What' commit picked up Windows-shell backtick line continuations from local oclif readme regeneration. Restore main's Linux-shell backslash continuations so CI's docs regeneration is clean.
When a trust-downgrade violation's offender or prior-trusted version carries provenance evidence, fetch the SLSA bundle from npm's attestations endpoint, verify it with sigstore.verify (Fulcio cert chain + Rekor inclusion + DSSE signature), and surface the source repo, commit, workflow, builder, run URL, and signer identity. Also flags any mismatch between the attested subject digest and the registry tarball's dist.integrity. sigstore is loaded via createRequire against locally-declared interfaces because sigstore 4.x's transitive .d.ts graph fails typecheck against this repo's @types/node baseline.
Move the npm-attestations fetch, sigstore.verify call, in-toto statement parsing, dist.integrity comparison, ProvenanceDetails interface, and renderProvenanceDetails out of trustPolicy.ts into a dedicated module. trustPolicy.ts now only knows about ProvenanceDetails as an opaque type and calls fetchAndVerifyProvenance / renderProvenanceDetails through the module boundary. No behavior change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends the
flub check trustPolicycommand so that eachERR_PNPM_TRUST_POLICY_VIOLATIONreported by pnpm is enriched with publish metadata fetched directly from the registry, making it possible to triage violations without manually digging throughnpm viewoutput.Description
For every violation the report now shows:
none/provenance/trustedPublisher) — matching pnpm's ownprettyPrintTrustEvidenceformatting.Why
no-downgradeviolations only tell you that trust regressed; they don't tell you from what, to what, or who published the offending version. That information is what actually decides whether a violation is a publish-pipeline regression we can wave through, an upstream maintainer change, or a supply-chain concern worth blocking on.How
pacote.packument(name, { where: workspaceDir, fullMetadata: true }).wheremakes pacote honor the audited workspace's.npmrc(scoped registries,_authToken/_auth), andfullMetadata: trueis required because abbreviated packuments strip_npmUser,dist.attestations, andtime.getTrustEvidence/detectStrongestTrustEvidenceBeforeDate, but retain the version identity that pnpm discards.violation.enrichmentErrorand the report falls back to pnpm's originalhint, so one bad packument can't abort the batch.packumentCachealready memoizes per-name, and registries can rate-limit.Dependencies
Adds
pacote@^21and@types/pacotetobuild-tools/packages/build-cli.Validation
Run on the FluidFramework root surfaces 8 active violations (e.g.
langchain,semver@5.7.2, etc.) with offender + prior-trusted lines populated. Behavior under registry/auth/network failure was exercised by forcing enrichment errors and confirming the fallback path.Risk
Self-contained to a single
flub checksubcommand; no runtime/product code paths affected.Reviewer Guidance