Skip to content

Pipeline Design 232

Jump to bottom
ezigus edited this page Mar 27, 2026 · 3 revisions

ADR written to .claude/pipeline-artifacts/design.md.

Summary of the design:

  • Root cause: pipeline-tasks.md is never cleared on initialize and never validated on resume or before injection — three independent gaps in the state lifecycle
  • Fix: Defense-in-depth with 3 guards:
    1. initialize_state() — rm -f TASKS_FILE (clean slate on new pipeline)
    2. resume_state() — parse Issue: metadata, remove if mismatched
    3. stage_build() — validate Issue: before injection, skip if stale
  • Scope: ~18 production lines across 2 files, ~95 test lines, no new files or dependencies
  • Key risk mitigations: all grep calls use || true for pipefail safety, GITHUB_ISSUE handled as potentially unset, malformed files treated as stale dates that pipeline-tasks.md belongs to the same issue.
  1. stage_build() (scripts/lib/pipeline-stages-build.sh:172) checks only -s "$TASKS_FILE" (file exists and is non-empty) — no metadata validation before injection.

The task file contains an Issue: metadata field (written at scripts/lib/pipeline-stages-intake.sh:460) that uniquely identifies which pipeline run created it, making validation straightforward.

Constraints

  • Bash 3.2 compatibility required (no associative arrays, no ${var,,})
  • All scripts use set -euo pipefail — safe patterns (|| true) needed for optional checks
  • TASKS_FILE is in _GIT_BOOKKEEPING_FILES (never committed) — it's a transient build artifact
  • The fix must not break the happy path: same-issue resume must preserve valid tasks

Decision

Defense-in-depth with three independent validation points — clear on initialize, validate on resume, guard on inject.

Component Diagram

┌──────────────────────────────────────────────────────────────────┐
│                        Pipeline Lifecycle                        │
│                                                                  │
│  ┌─────────────────┐    ┌─────────────────┐    ┌──────────────┐ │
│  │  pipeline-state  │    │ stages-intake    │    │ stages-build │ │
│  │                  │    │                  │    │              │ │
│  │ initialize_state │    │ writes           │    │ reads        │ │
│  │   ► rm TASKS     │    │ pipeline-tasks   │    │ TASKS_FILE   │ │
│  │                  │    │ with Issue: meta │    │ validates    │ │
│  │ resume_state     │    │                  │    │ Issue: meta  │ │
│  │   ► validate     │    │                  │    │ before inject│ │
│  │     Issue: meta  │    │                  │    │              │ │
│  └────────┬─────────┘    └────────┬─────────┘    └──────┬───────┘ │
│           │                       │                      │        │
│           └───────────┬───────────┘                      │        │
│                       ▼                                  │        │
│              .claude/pipeline-tasks.md ──────────────────┘        │
│              (transient build artifact)                            │
└──────────────────────────────────────────────────────────────────┘

Data Flow

NEW PIPELINE:
  initialize_state() ── rm -f TASKS_FILE ──► clean slate
  intake stage ──────── writes TASKS_FILE with Issue: #NNN metadata
  build stage ────────── validates Issue: matches GITHUB_ISSUE ──► injects into goal

RESUME (same issue):
  resume_state() ──── reads Issue: from TASKS_FILE ── matches GITHUB_ISSUE ──► keeps file
  build stage ─────── validates Issue: matches ──► injects into goal ✓

RESUME (different issue / stale file):
  resume_state() ──── reads Issue: from TASKS_FILE ── mismatch ──► rm + warn
  build stage ─────── TASKS_FILE gone ──► no injection ✓

Interface Contracts

// initialize_state() — scripts/lib/pipeline-state.sh
// Precondition: ARTIFACTS_DIR and TASKS_FILE are set
// Postcondition: pipeline-tasks.md removed, state variables reset
// Side effects: rm -f TASKS_FILE
function initialize_state(): void

// resume_state() — scripts/lib/pipeline-state.sh
// Precondition: STATE_FILE exists, TASKS_FILE may or may not exist
// Postcondition: if TASKS_FILE exists with mismatched Issue, file is removed + warning logged
// Error contract: missing/malformed Issue metadata → treat as stale (safe default), remove
function resume_state(): void  // exits on fatal errors (missing state file, completed pipeline)

// stage_build() — scripts/lib/pipeline-stages-build.sh
// Precondition: TASKS_FILE and GITHUB_ISSUE may or may not be set
// Postcondition: TASKS_FILE content injected into enriched_goal ONLY if Issue matches
// Error contract: missing/malformed metadata → skip injection (defensive, no crash)
function stage_build(): void

Error Boundaries

Component Error Handling
initialize_state() TASKS_FILE doesn't exist rm -f is idempotent — no error
resume_state() TASKS_FILE missing Issue metadata Treat as stale, remove with warn
resume_state() TASKS_FILE doesn't exist No-op — [[ -f ]] guard
stage_build() Issue extraction fails (grep returns 1) `
stage_build() GITHUB_ISSUE unset Skip validation, inject as before (backwards compat for goal-only pipelines)

Validation Pattern (used in both resume and build)

# Extract "Issue: #232" → "#232" from Context section
local tasks_issue
tasks_issue=$(grep '^- Issue:' "$TASKS_FILE" 2>/dev/null | head -1 | sed 's/^- Issue: *//' || true)

# Compare against current pipeline's issue
if [[ -n "$tasks_issue" && "$tasks_issue" \!= "${GITHUB_ISSUE:-}" ]]; then
    warn "Stale pipeline-tasks.md (issue $tasks_issue ≠ ${GITHUB_ISSUE:-none}) — removing"
    rm -f "$TASKS_FILE"
fi

Alternatives Considered

  1. Metadata-Scoped Task Validation (commit hash + tree hash) — Pros: intelligent per-task filtering, future-proof for multi-issue pipelines / Cons: ~50+ lines of complexity, overkill since pipelines are single-issue, introduces git dependency in state management, hard to test
  2. Centralized validate_tasks_file() helper — Pros: single source of truth, DRY / Cons: ~30 lines of overhead for a localized 3-point bug, adds indirection that obscures simplicity. Defense-in-depth across lifecycle points is more robust than a single validation call because each guard catches different failure modes (stale from prior run vs. stale from issue switch vs. stale at injection time)

Implementation Plan

  • Files to create: none
  • Files to modify:
    • scripts/lib/pipeline-state.sh — initialize_state() (+1 line: rm -f), resume_state() (+11 lines: parse Issue metadata + validate + remove if stale)
    • scripts/lib/pipeline-stages-build.sh — stage_build() (+6 lines: validate Issue metadata before injection at line 172)
    • scripts/sw-lib-pipeline-stages-test.sh — 4 new test cases (~95 lines: initialize clears tasks, resume validates issue match, resume removes on mismatch, build skips stale injection)
  • Dependencies: none — all changes use existing variables (TASKS_FILE, GITHUB_ISSUE) and utilities (warn, info, grep)
  • Risk areas:
    • grep under set -euo pipefail exits 1 on no match — all grep calls must use || true
    • GITHUB_ISSUE may be unset/empty in goal-only pipelines (no --issue flag) — guards must handle ${GITHUB_ISSUE:-}
    • Task file may lack a Context section entirely (malformed) — treat as stale and remove

Root Cause Hypothesis

  1. [99% — CONFIRMED] pipeline-tasks.md not cleared/validated on resume — three independent gaps in the state lifecycle. Evidence: direct code inspection of initialize_state() (line 461 clears 2 files, not tasks), resume_state() (lines 534-610, no TASKS_FILE reference), and stage_build() (line 172, -s check only).
  2. [85%] TEST_CMD environment leak from test harness — export TEST_CMD="echo FAIL; exit 1" at sw-lib-pipeline-stages-test.sh:253 could persist across test boundaries. Secondary to stale tasks; investigate during build.
  3. [70%] Incomplete transient artifact cleanup pattern — initialize_state() only clears 2 of N transient files, suggesting this class of bug could recur for other artifacts.

Evidence Gathered

  • scripts/lib/pipeline-state.sh:452-463 — initialize_state() clears model-routing.log and .plan-failure-sig.txt only
  • scripts/lib/pipeline-state.sh:534-610 — resume_state() restores GITHUB_ISSUE at line 553 but never touches TASKS_FILE
  • scripts/lib/pipeline-stages-build.sh:172-177 — injection guarded only by [[ -s "$TASKS_FILE" ]]
  • scripts/lib/pipeline-stages-intake.sh:460 — metadata line - Issue: ${GITHUB_ISSUE:-none} is parseable

Fix Strategy

Defense-in-depth at three lifecycle points rather than a single validation call. Each guard is independent — if one is bypassed (e.g., direct resume without initialize), the others catch stale tasks. This matches the existing pattern where initialize_state() already clears per-run artifacts.

Verification Plan

  • Unit test: create TASKS_FILE, call initialize_state(), assert file removed
  • Unit test: create TASKS_FILE with Issue: #154, set GITHUB_ISSUE="#232", call resume_state(), assert file removed
  • Unit test: create TASKS_FILE with Issue: #232, set GITHUB_ISSUE="#232", call resume_state(), assert file preserved
  • Unit test: create TASKS_FILE with wrong issue, call stage_build(), assert not injected into goal
  • Integration: run ./scripts/sw-pipeline-test.sh — all 50+ existing tests pass (no regressions)

Validation Criteria

  • initialize_state() removes pipeline-tasks.md on every new pipeline start
  • resume_state() removes task file when Issue metadata doesn't match current GITHUB_ISSUE
  • resume_state() preserves task file when Issue metadata matches current GITHUB_ISSUE
  • stage_build() skips injection when Issue metadata doesn't match
  • Malformed/missing metadata in task file doesn't crash pipeline (graceful degradation)
  • Missing task file on resume is a no-op (no crash)
  • All existing pipeline tests pass with no regressions
  • All code is Bash 3.2 compatible

Clone this wiki locally