fix: upload and download provenance bundles in S3 cache #292

leodido · 2025-11-21T01:21:07Z

Summary

This PR fixes the "no attestation bundle found" error when building packages with SLSA enabled. The issue occurs when a package downloaded from S3 remote cache is used as a dependency for a local build.

Problem

When building packages with SLSA enabled (Leeway v0.15.0-rc3), builds fail with:

[package:name] Reason: error accessing provenance bundle for /var/lib/leeway/cache/<hash>.tar: no attestation bundle found

Root Cause

After PR #283, provenance bundles were moved outside the .tar.gz file to maintain artifact determinism. However, the S3 cache implementation was not updated to handle these separate .provenance.jsonl files:

Package A is downloaded from S3 with SLSA verification
- Downloads: <hash>.tar.gz and <hash>.tar.gz.att ✅
- Does NOT download: <hash>.tar.gz.provenance.jsonl ❌
Package B is built locally and needs Package A as a dependency
- Calls getDependenciesProvenanceBundles() to collect provenance
- Looks for /var/lib/leeway/cache/<hash>.tar.provenance.jsonl
- File doesn't exist → Build fails ❌

Solution

Modified pkg/leeway/cache/remote/s3.go to:

1. Upload Provenance Bundles

After uploading artifact, check if .provenance.jsonl exists
Upload it alongside the artifact (non-blocking, won't fail if missing)
Uses same rate limiting and timeout handling as artifact upload

2. Download Provenance Bundles

After SLSA verification succeeds, attempt to download .provenance.jsonl
Best effort download (logs debug message if missing)
Gracefully handles missing provenance for backward compatibility

File Types

Three separate files with different purposes:

.tar.gz - The build artifact (deterministic)
.tar.gz.att - Sigstore Bundle v0.3 (SLSA attestation for verification)
.tar.gz.provenance.jsonl - In-toto v0.2 provenance bundle (for dependency provenance collection)

Backward Compatibility

✅ Fully backward compatible:

Provenance upload is optional (won't fail if file doesn't exist)
Provenance download is best effort (logs debug if missing)
Older artifacts without provenance continue to work
No cache invalidation needed

Testing

✅ Code compiles successfully
✅ Maintains existing S3 cache behavior
✅ Non-blocking for both upload and download
✅ Proper error handling and logging

Depends on PR fix: support SBOM generation with OCI layout export #291 (SBOM OCI layout support)
Depends on PR fix: support container extraction with OCI layout export #290 (SLSA L3 cache verification)
Fixes issue introduced in PR feat!: store in-toto provenance outside tar.gz #283 (provenance moved outside tar.gz)
Part of SLSA L3 compliance work

Checklist

Code compiles successfully
Follows conventional commits spec
Backward compatible (no breaking changes)
Non-blocking implementation (won't fail builds)
Proper error handling and logging
Graceful degradation for missing provenance

leodido · 2025-11-21T01:34:38Z

Integration Tests Added

Added comprehensive integration tests in s3_provenance_test.go to verify provenance bundle operations:

Test Coverage

TestS3Cache_ProvenanceUpload - Tests upload scenarios:

✅ Successful upload with valid provenance content
✅ Skip upload when provenance file is missing (expected behavior)
✅ Handle empty provenance files

TestS3Cache_ProvenanceDownload - Tests download scenarios:

✅ Successful download with content verification
✅ Backward compatibility (missing provenance doesn't fail)
✅ Empty file detection and rejection

TestS3Cache_ProvenanceRoundTrip - End-to-end test:

✅ Upload provenance to mock S3
✅ Download to different directory
✅ Verify content integrity matches

TestS3Cache_ProvenanceAtomicMove - Atomic operations:

✅ Atomic file moves (no race conditions)
✅ Temporary file cleanup
✅ No leftover .tmp files

Test Results

=== RUN   TestS3Cache_ProvenanceUpload
--- PASS: TestS3Cache_ProvenanceUpload (0.00s)
=== RUN   TestS3Cache_ProvenanceDownload
--- PASS: TestS3Cache_ProvenanceDownload (0.00s)
=== RUN   TestS3Cache_ProvenanceRoundTrip
--- PASS: TestS3Cache_ProvenanceRoundTrip (0.00s)
=== RUN   TestS3Cache_ProvenanceAtomicMove
--- PASS: TestS3Cache_ProvenanceAtomicMove (0.00s)
PASS

All existing remote cache tests continue to pass (59.7s total runtime).

leodido · 2025-11-21T11:56:51Z

Fixed: Backward Compatibility for Missing Provenance Bundles

✅ Added graceful degradation for artifacts built before v0.15.0-rc4.

Problem

Artifacts built before v0.15.0-rc4 don't have .provenance.jsonl files. When used as dependencies, builds fail with:

error accessing provenance bundle for /var/lib/leeway/cache/<hash>.tar: no attestation bundle found

Solution

Modified pkg/leeway/provenance.go:getDependenciesProvenanceBundles() to:

Detect missing provenance bundles (ErrNoAttestationBundle)
Log warning (not debug) for visibility
Continue build instead of failing
Allow gradual cache population

Key Features

✅ Security maintained: SLSA attestation verification still works
✅ Gradual rollout: Cache populates as packages rebuild
✅ Monitoring: Warnings visible in logs for tracking transition
✅ Temporary: Includes TODO with removal criteria and tracking issue

Tracking

Issue: Remove backward compatibility fallback for missing provenance bundles #293
Removal date: 2025-12-15 (4 weeks after deployment)
Criteria: Zero warnings for 2+ consecutive weeks

Testing

✅ Code compiles successfully
✅ Existing provenance tests pass
✅ Test skeleton added for missing provenance case

Commit

a4a961e fix: add backward compatibility for missing provenance bundles

This fix enables smooth transition to external provenance storage without breaking existing builds.

geropl

Tests and Code LGTM! ✔️

When building packages with SLSA enabled, provenance bundles are stored alongside artifacts as <artifact>.provenance.jsonl. These bundles are needed for dependency provenance collection during local builds. Previously, only the artifact (.tar.gz) and attestation (.tar.gz.att) were uploaded/downloaded from S3, causing builds to fail with: "error accessing provenance bundle: no attestation bundle found" This fix: - Uploads .provenance.jsonl files alongside artifacts (non-blocking) - Downloads .provenance.jsonl files after SLSA verification (best effort) - Gracefully handles missing provenance for backward compatibility The provenance download is non-critical and logs a debug message if missing, allowing older artifacts without provenance to work correctly. Fixes dependency provenance collection for SLSA L3 compliant builds. Co-authored-by: Ona <no-reply@ona.com>

Extract provenance upload/download logic into dedicated helper functions with improved error handling and verification: uploadProvenanceBundle(): - Checks if provenance file exists before attempting upload - Proper rate limiting and timeout handling - Non-blocking operation with clear logging - Returns early on any error to avoid cascading failures downloadProvenanceBundle(): - Rate limiting and timeout protection - Verifies downloaded file exists and has content - Atomic move to final location - Returns bool to indicate success/failure - Graceful handling of missing provenance (expected for older artifacts) Benefits: - Better separation of concerns - More robust error handling - Easier to test and maintain - Clear success/failure indicators - Improved logging at each step Co-authored-by: Ona <no-reply@ona.com>

Add integration tests for provenance bundle upload/download functionality: TestS3Cache_ProvenanceUpload: - Successful upload with valid provenance - Skip upload when provenance file missing - Handle empty provenance files TestS3Cache_ProvenanceDownload: - Successful download with content verification - Backward compatibility (missing provenance) - Empty file detection and rejection TestS3Cache_ProvenanceRoundTrip: - End-to-end upload and download - Content integrity verification TestS3Cache_ProvenanceAtomicMove: - Atomic file operations - Temporary file cleanup - No leftover .tmp files All tests pass and verify: - Non-blocking behavior - Proper error handling - File integrity checks - Backward compatibility - Atomic operations Co-authored-by: Ona <no-reply@ona.com>

When artifacts built before v0.15.0-rc4 are used as dependencies, they don't have .provenance.jsonl files, causing builds to fail with 'no attestation bundle found' error. This fix adds graceful degradation: - Detects missing provenance bundles (ErrNoAttestationBundle) - Logs warning instead of failing the build - Allows gradual cache population during transition period - Maintains SLSA attestation verification (security not compromised) The fallback is temporary and should be removed after all cached artifacts have provenance bundles (estimated 4 weeks). Tracking: #293 Co-authored-by: Ona <no-reply@ona.com>

Implements the test for TestGetDependenciesProvenanceBundles_MissingProvenance that was previously skipped with a TODO. The test verifies backward compatibility when dependency provenance bundles are missing (artifacts built before v0.15.0-rc5): 1. Tests that missing provenance returns ErrNoAttestationBundle 2. Tests that existing provenance is read correctly 3. Documents the backward compatibility mechanism This validates the error detection that enables getDependenciesProvenanceBundles() to gracefully handle missing provenance with a warning instead of failing builds during the transition period. Co-authored-by: Ona <no-reply@ona.com>

leodido self-assigned this Nov 21, 2025

leodido force-pushed the ldd/sbom-oci-layout branch from 0c347c7 to 1843ce1 Compare November 21, 2025 08:58

leodido force-pushed the leo-missing-provenance-bundle branch from 62f47a2 to 4245a86 Compare November 21, 2025 08:59

leodido requested review from corneliusludmann and kylos101 November 21, 2025 09:00

leodido force-pushed the ldd/sbom-oci-layout branch from 1843ce1 to 372a050 Compare November 21, 2025 09:19

leodido force-pushed the leo-missing-provenance-bundle branch from 4245a86 to bb687dd Compare November 21, 2025 09:21

leodido requested a review from geropl November 21, 2025 10:31

leodido mentioned this pull request Nov 21, 2025

Remove backward compatibility fallback for missing provenance bundles #293

Open

4 tasks

leodido force-pushed the leo-missing-provenance-bundle branch 2 times, most recently from c594a2c to 7b2010c Compare November 21, 2025 12:02

geropl approved these changes Nov 21, 2025

View reviewed changes

leodido and others added 4 commits November 23, 2025 11:49

leodido force-pushed the leo-missing-provenance-bundle branch from 48e8cdf to 72c405f Compare November 23, 2025 11:50

leodido changed the base branch from ldd/sbom-oci-layout to main November 23, 2025 11:50

leodido merged commit 27734d1 into main Nov 23, 2025
8 checks passed

leodido mentioned this pull request Nov 24, 2025

fix: correct build summary counts for packages built after verification failure #295

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: upload and download provenance bundles in S3 cache #292

fix: upload and download provenance bundles in S3 cache #292

Uh oh!

leodido commented Nov 21, 2025

Uh oh!

leodido commented Nov 21, 2025

Uh oh!

leodido commented Nov 21, 2025

Uh oh!

geropl left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fix: upload and download provenance bundles in S3 cache #292

fix: upload and download provenance bundles in S3 cache #292

Uh oh!

Conversation

leodido commented Nov 21, 2025

Summary

Problem

Root Cause

Solution

1. Upload Provenance Bundles

2. Download Provenance Bundles

File Types

Backward Compatibility

Testing

Related

Checklist

Uh oh!

leodido commented Nov 21, 2025

Integration Tests Added

Test Coverage

Test Results

Uh oh!

leodido commented Nov 21, 2025

Fixed: Backward Compatibility for Missing Provenance Bundles

Problem

Solution

Key Features

Tracking

Testing

Commit

Uh oh!

geropl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants