Publish reproducible evidence cases - #353
Merged
Merged
Conversation
Record pinned inputs, exact reproduction commands, evidence checksums, claim scope, and limitations for public assurance cases. Add a standard-library verifier and a Make target for checking the full catalog or inspecting one case.
Explain the pinned npm, pnpm, Yarn, Bun, Go, Python, and Maven graph cases, including detector fidelity and limitations. Add a deterministic fallback case that proves degraded coverage remains visible instead of being treated as equivalent native evidence.
Publish focused evidence for complex SPDX policy, vulnerability constraints, baselines, risky source changes, persisted findings, analyzer-tier reachability, and canonical read-only remediation guidance. State the limits of live advisory data and unknown evidence explicitly.
Add the public evidence guide and explain local, Git, container, and SBOM target cases. Record dated validator and portable workflow evidence, clarify that portable assurance runs unit tests and release builds, and document repeatable cold and warm performance measurements with their limits.
Contributor
📝 WalkthroughWalkthroughThis change adds a strict ChangesPublic evidence
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Makefile
participant publicevidence
participant casesJSON
participant EvidenceArtifacts
Makefile->>publicevidence: run evidence validator with optional case
publicevidence->>casesJSON: load and validate catalog
publicevidence->>EvidenceArtifacts: verify safe paths and SHA-256 hashes
EvidenceArtifacts-->>publicevidence: validated artifacts
publicevidence-->>Makefile: print verified cases or fatal error
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
bomly-guy
marked this pull request as ready for review
July 29, 2026 10:53
Contributor
Bomly Diff SummaryCompared Overview
Dependency Changes✅ No dependency changes. Vulnerabilities✅ No vulnerability changes. License Changes✅ No license changes. Project Posture✅ No project posture changes ( Policy Findings✅ No policy differences were identified. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/EVIDENCE.md`:
- Around line 39-41: Update the “Dependency graph evidence” entry in
docs/EVIDENCE.md to match the catalog: either link a dedicated dependency-graph
fallback case that explicitly accepts fallback output, or remove the claim that
the listed graph cases cover degraded fallback behavior and state that fallback
behavior is documented separately.
In `@docs/evidence/TARGETS_AND_OPERATIONS.md`:
- Around line 46-48: Update the manual workflow invocations near gh workflow run
sbom-interoperability.yml and the corresponding command near line 65 to pass an
immutable ref, using the recorded workflow commit or dedicated evidence tag from
docs/EVIDENCE.md, so both reruns use the same workflow definition.
In `@internal/tools/publicevidence/main.go`:
- Around line 250-265: Update the artifact validation flow around filepath.Clean
and os.Stat to resolve both the repository root and artifact path before
containment checking, then use filepath.Rel to reject any resolved path outside
the root, including symlinked components. Preserve regular-file validation and
error reporting, and add a test that rejects an artifact symlink targeting
outside the repository.
- Around line 230-234: Require immutable digest-based references in the
container validation branch of internal/tools/publicevidence/main.go for
pinned-input evidence, while continuing to allow tag-only references for
non-pinned evidence levels. Update test/evidence/cases.json at lines 56-59 to
use an immutable digest for the pinned alpine case, or downgrade its evidence
level if it intentionally tests mutable tag behavior.
- Around line 211-213: Update the validation around current.Proves and
current.Limitations to reject entries that are empty or contain only whitespace,
not merely empty slices. Trim or otherwise validate each entry before accepting
the case, while preserving the existing error behavior for missing or blank
proof and limitation content.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 1e309f2a-33b1-44b1-b565-8be2f4949447
📒 Files selected for processing (12)
MakefileREADME.mddocs/EVIDENCE.mddocs/README.mddocs/evidence/DEPENDENCY_GRAPHS.mddocs/evidence/POLICY_AND_GUIDANCE.mddocs/evidence/TARGETS_AND_OPERATIONS.mddocs/manifest.jsoninternal/tools/publicevidence/main.gointernal/tools/publicevidence/main_test.gotest/evidence/README.mdtest/evidence/cases.json
bomly-guy
commented
Jul 29, 2026
bomly-guy
left a comment
bomly-guy
left a comment
Collaborator
Author
There was a problem hiding this comment.
Reviewed the catalog, the three case-study pages, the tool, and all 24 cases, and specifically checked interaction with the two open docs PRs (#350 wave 1, #354 wave 2, stacked). Overall this is strong work and looks ready: the catalog enforces exactly the right contract (sorted unique IDs, mandatory proves and limitations, checksum-verified artifacts, full 40-char revisions), reproduce commands delegate to the pinned smoke cases with $-anchored -run patterns per the slice convention, the prose never overclaims (live-service results framed as dated observations, mutable container tags called out, "unreachable does not mean safe"), and there are no competitor mentions. The tool itself is careful: traversal-guarded artifact paths, size-limited decode, DisallowUnknownFields, trailing-data rejection.
Verified on a local three-way merge with the wave branches (docs/reference-wave1 + docs/reference-wave2):
- No textual conflicts — git auto-merges
README.md,docs/README.md,docs/manifest.jsoncleanly in either merge order, and this branch already contains main@0f4b09b (#352). - On the merged tree:
make evidencepasses (24 cases verified), the manifest sync test passes, and the offline link checker that #350 adds to CI passes over the new pages (744 links, 0 errors). So no coordination needed beyond one semantic point below. - One semantic note for whoever merges second: #350 rewrites
docs/README.mdso its sections mirror the manifest groups. The auto-merge lands the Reproducible Evidence bullet in the "Getting started" section, which stays consistent with this PR's"group": "start"— but see the inline comment on whetherreferenceis the better group; if the group changes, the bullet should move to the matching section in the same commit.
Two inline notes below (nav children flag, tool path nit). None of them block.
| "group": "start" | ||
| }, | ||
| { | ||
| "slug": "evidence", |
Collaborator
Author
There was a problem hiding this comment.
Two thoughts on this entry:
docs/evidence/has three child pages, but this entry doesn't set"hasChildren": true— the other parents with nested directories (detectors,matchers,schemas,plugins, andcommandsin docs: first wave of docs improvements (FAQ, network page, command reference, link CI) #350) all set it so the site nav exposes the children. Without it the case-study pages are reachable only through in-page links.- Is
startthe right group? This reads as assurance/reference material rather than a first-run step —reference("Generated specifications and supported matrices") or even a case forconceptsseems closer. Not blocking, but if you change it after docs: first wave of docs improvements (FAQ, network page, command reference, link CI) #350 lands, move thedocs/README.mdbullet to the matching section in the same commit (the index mirrors the groups from then on).
| if err != nil { | ||
| exitError(err) | ||
| } | ||
| loaded, err := loadCatalog(filepath.Join(root, filepath.FromSlash(*catalogPath))) |
Collaborator
Author
There was a problem hiding this comment.
Nit: an absolute -catalog path gets joined onto the repository root (filepath.Join(root, "/abs/path") → root/abs/path), which silently points at the wrong file. Worth honoring absolute paths:
resolved := filepath.FromSlash(*catalogPath)
if !filepath.IsAbs(resolved) {
resolved = filepath.Join(root, resolved)
}Internal tool, so fine to defer.
Harden evidence validation against blank claims, mutable pinned container tags, absolute catalog path mistakes, and symlink escapes. Pin manual workflow examples to the matching release definition and align evidence navigation and fallback wording with the catalog.
Keep advanced assurance material out of the everyday concepts section while retaining child navigation for the detailed evidence pages.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@internal/tools/publicevidence/main.go`:
- Around line 252-257: Update the container reference validation in
TestValidateInputRequiresDigestForPinnedContainer’s underlying validation flow
to reject refs that become empty after strings.TrimSpace, while preserving the
existing digest validation. Add a regression assertion covering a snapshot or
pinned container with Ref set to whitespace-only.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 1f2c4309-116e-45f4-9ce6-f133516aeddb
📒 Files selected for processing (8)
docs/EVIDENCE.mddocs/README.mddocs/evidence/TARGETS_AND_OPERATIONS.mddocs/manifest.jsoninternal/tools/publicevidence/main.gointernal/tools/publicevidence/main_test.gotest/evidence/README.mdtest/evidence/cases.json
🚧 Files skipped from review as they are similar to previous changes (2)
- docs/README.md
- docs/manifest.json
Comment on lines
+252
to
+257
| if current.Ref == "" { | ||
| return errors.New("container input requires an image reference") | ||
| } | ||
| if evidenceLevel == "pinned-input" && !containerDigestPattern.MatchString(current.Ref) { | ||
| return errors.New("pinned container input requires an immutable sha256 digest") | ||
| } |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Reject whitespace-only container references.
Line 252 only rejects "", so a snapshot case with Ref: " " validates despite not naming an image. Check strings.TrimSpace(current.Ref) == "" and add a regression assertion to TestValidateInputRequiresDigestForPinnedContainer.
Proposed fix
- if current.Ref == "" {
+ if strings.TrimSpace(current.Ref) == "" {
return errors.New("container input requires an image reference")
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if current.Ref == "" { | |
| return errors.New("container input requires an image reference") | |
| } | |
| if evidenceLevel == "pinned-input" && !containerDigestPattern.MatchString(current.Ref) { | |
| return errors.New("pinned container input requires an immutable sha256 digest") | |
| } | |
| if strings.TrimSpace(current.Ref) == "" { | |
| return errors.New("container input requires an image reference") | |
| } | |
| if evidenceLevel == "pinned-input" && !containerDigestPattern.MatchString(current.Ref) { | |
| return errors.New("pinned container input requires an immutable sha256 digest") | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@internal/tools/publicevidence/main.go` around lines 252 - 257, Update the
container reference validation in
TestValidateInputRequiresDigestForPinnedContainer’s underlying validation flow
to reject refs that become empty after strings.TrimSpace, while preserving the
existing digest validation. Add a regression assertion covering a snapshot or
pinned container with Ref set to whitespace-only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
make evidenceto validate the catalog or inspect one focused caseThe four commits keep the catalog, graph evidence, policy and guidance evidence, and target and operational evidence independently reviewable.
User impact
Readers can now trace important behavior claims to public inputs and repeatable commands without private infrastructure. Live advisory observations, detector fallbacks, unknown evidence, analyzer tiers, mutable container tags, and operational limits are called out explicitly.
This PR changes repository assurance and documentation only. It does not change CLI behavior or the CLI, SDK, MCP, or plugin schemas.
Validation
make evidencemake fmt-checkmake lintmake testmake buildgit diff --check origin/main...HEADmake generatewas not required because no generated contract source changed.Summary by CodeRabbit
New Features
make evidencereproducible evidence command, including optional case filtering.Documentation
Tests / Quality