Build Artifact Caching System with Smart Invalidation

[github-actions]

Strategic Improvement

Cache build artifacts (installed dependencies, compiled code) between pipeline runs to eliminate redundant setup work. Smart invalidation detects when cache is stale based on dependency file changes. Faster setup means more iterations for actual development work, improving both cost and success rate.

Acceptance Criteria

• Cache npm/pip/go dependencies in ~/.shipwright/cache/{repo-hash}/{dependency-hash}/
• Detect dependency file changes (package.json, requirements.txt, go.mod) and invalidate cache
• Restore cache at build loop start if valid (add --use-cache flag to sw-loop.sh)
• Track cache hit rate in pipeline metrics
• Reduce dependency installation time by >60% on cache hits

Context

• Priority: P3
• Complexity: fast
• Generated by: Strategic Intelligence Agent
• Strategy alignment: P3: Cost Efficiency

[github-actions]

⚓ Shipwright Pipeline Starting

Field	Value
Template	standard
Complexity	standard
Max Iterations	20
Triage	Improves build loop performance and cost efficiency through intelligent dependency caching with smart invalidation, directly supporting Shipwright's core pipeline execution and cost intelligence goals.
Runner	ubuntu-latest
Risk Level	unknown
Workflow	24532045302

The autonomous pipeline is processing this issue. Updates will be posted as stages complete.

[github-actions]

[github-actions]

[github-actions]

📋 Implementation Plan

Click to expand full plan (28 lines)

The plan is ready. Here's a summary:

Plan: Build Artifact Caching System with Smart Invalidation

4 files to create/modify:

1. New: scripts/lib/artifact-cache.sh — Core cache library with 7 functions:

   • cache_compute_repo_hash (git URL → 12-char hash)
   • cache_compute_dep_hash (package.json + lock files → 16-char hash)
   • cache_is_valid (content-based, no TTL)
   • cache_restore / cache_save (tar archive, atomic writes)
   • cache_evict_stale (LRU eviction, 30-day/2GB limits)
   • cache_metrics_report (reads events.jsonl for hit rate)

2. Modify: scripts/sw-loop.sh — Add --use-cache flag + restore/save hooks at the right iteration boundaries

3. New: scripts/sw-artifact-cache-test.sh — 12 unit tests (determinism, restore, eviction, failure recovery)

4. Modify: package.json — Register test suite

Cache layout: ~/.shipwright/artifact-cache/{repo-hash}/{dep-hash}/node_modules.tar.gz

Key design decisions:

• Defaults false — zero blast radius for existing pipelines
• Archive integrity checked with tar tzf before restore (no corrupt-cache failures)
• PID-namespaced tmp files prevent concurrent-write corruption
• Lock files (package-lock.json, go.sum) included in hash — prevents same-name/different-version collisions
• All Bash 3.2 compatible

---

Generated by shipwright pipeline at 2026-04-16T20:31:26Z

[github-actions]

[github-actions]

[github-actions]

📐 Architecture Decision Record

Click to expand ADR (145 lines)

Design: Build Artifact Caching System with Smart Invalidation

Context

Pipeline builds repeatedly reinstall node_modules (and equivalents) across iterations, worktrees, and daemon-driven runs. Each install costs seconds-to-minutes per iteration; across sw-loop.sh's iteration loop and parallel worktrees (see scripts/sw-worktree.sh, scripts/sw-fleet.sh), this compounds into significant wall-clock and token cost.

Constraints from the codebase:

• Bash 3.2 compatibility (.claude/CLAUDE.md — no declare -A, no ${var,,}, no readarray).
• set -euo pipefail and atomic writes (tmp + mv) are required conventions.
• Runtime state root is ~/.shipwright/ (see runtime-state AUTO section); cache must live there.
• No new runtime deps — only tar, sha256sum/shasum, jq, already assumed in doctor.
• Test runner is the shipwright bash harness (PASS/FAIL counters, mock binaries), registered in package.json.
• Prior failure (.claude/memory/failures.json, 2026-04-12): sw-cleanup.sh stale-detection logic broke in dry-run mode — state-tracking and cleanup code is a known fragile area. Invalidation logic must be observable and dry-run safe.
• Blast radius: feature must default off; existing pipelines must be byte-identical when the flag is absent.

Decision

Ship a content-addressed, file-based artifact cache as a standalone library (scripts/lib/artifact-cache.sh) with a narrow integration point in scripts/sw-loop.sh behind --use-cache (default false).

Component Diagram

┌─────────────────────────┐        ┌──────────────────────────┐
│   sw-loop.sh            │        │  events.jsonl            │
│   (orchestrator)        │───────▶│  (observability sink)    │
│   --use-cache flag      │        └──────────────────────────┘
└──────────┬──────────────┘                    ▲
           │ sources                           │ emit_event
           ▼                                   │
┌─────────────────────────────────────────────┴────────┐
│  scripts/lib/artifact-cache.sh  (pure library)       │
│  ┌────────────┐ ┌──────────────┐ ┌─────────────────┐ │
│  │ hashing    │ │ store        │ │ eviction        │ │
│  │ repo/dep   │ │ restore/save │ │ LRU 30d / 2GB   │ │
│  └────────────┘ └──────┬───────┘ └─────────────────┘ │
└────────────────────────┼─────────────────────────────┘
                         ▼
           ~/.shipwright/artifact-cache/
             {repo-hash}/{dep-hash}/node_modules.tar.gz
             {repo-hash}/{dep-hash}/.meta.json   (atime, size, sha)

Dependencies point inward: sw-loop.sh → artifact-cache.sh → filesystem. The library knows nothing about loops, pipelines, or events beyond an injectable CACHE_EVENT_SINK callback (default: emit_event).

Interface Contracts

All functions return 0 on success, non-zero on recoverable failure (never exit). Errors go to stderr; structured events go through the sink.

# Hashing — deterministic, pure
cache_compute_repo_hash <repo_root> -> stdout: <12-hex>    # git remote URL + primary branch
cache_compute_dep_hash  <repo_root> -> stdout: <16-hex>    # sha256 of sorted(package.json, *lock*, go.sum, Cargo.lock)

# Validation
cache_is_valid <repo_hash> <dep_hash> -> exit 0 if archive exists AND `tar tzf` succeeds
                                         exit 1 otherwise (missing/corrupt)

# I/O — atomic, PID-namespaced
cache_restore <repo_hash> <dep_hash> <target_dir>
    # pre: target_dir exists, is writable
    # post: target_dir/node_modules populated OR exit 2 (miss) / exit 3 (corrupt)

</details>

---
_Generated by `shipwright pipeline` design stage at 2026-04-16T20:34:21Z_

[github-actions]

[github-actions]

🔨 Build started — shipwright loop with 20 max iterations, 1 agent(s), model: opus

[github-actions]

[github-actions]