Skip to content

Pipeline Plan 313

Jump to bottom
ezigus edited this page Apr 18, 2026 · 2 revisions

The implementation plan has been written to .claude/pipeline-artifacts/plan.md. Here's a summary:

Scope: 3 files modified, 4 new functions, 2 new workflow steps

Files to modify:

  1. scripts/lib/ruflo-adapter.sh — 4 new functions + wiring into existing import/export
  2. .github/workflows/shipwright-pipeline.yml — 2 new workflow steps
  3. scripts/sw-ruflo-adapter-test.sh — 8 new tests

Key design decisions:

  • Orphan git branch chosen over actions/cache (unreliable eviction) and artifacts (expire, cross-workflow issues) — matches existing shipwright-data pattern already in the workflow
  • Merge strategy: jq -s '.[0] * .[1]' (local overwrites remote, since local is freshly exported = newer wins)
  • Push conflicts: 3-retry with exponential jitter, same pattern as shipwright-data branch
  • All functions return 0 — pipeline never fails on memory persistence errors
  • CI guard ([[ "${CI:-}" == "true" ]]) prevents local dev side effects
  • HNSW/Q-weight limitation: Documented inline — ruflo memory export may only capture KV store

11 tasks with explicit dependency chain, 8 unit tests planned, estimated <30s CI overhead. fra, simple, matches existing shipwright-data pattern already in the workflow (lines 803-852). Disadvantages: push conflicts with concurrent jobs, git operations add ~5-10s.

Approach B: GitHub Actions cache only (current state, enhanced) — Improve existing actions/cache steps. Advantages: simpler, no git operations. Disadvantages: cache eviction is unpredictable (7-day LRU), no merge semantics, cache key collisions.

Approach C: Artifact-based persistence — Use actions/upload-artifact / download-artifact. Advantages: no push conflicts. Disadvantages: artifacts expire (90 days default), no cross-workflow access, complex to query latest.

Chosen: Approach A — Matches the existing shipwright-data pattern in the workflow, provides reliable persistence (git branches don't expire), and the retry-merge strategy handles concurrent writes acceptably. The blast radius is minimal: two new functions in ruflo-adapter.sh, two new workflow steps.

Risk Assessment

  1. Push conflicts — Two concurrent pipelines pushing simultaneously. Mitigated by fetch-rebase-retry loop (3 attempts with jitter), matching the existing shipwright-data pattern.
  2. Large memory exports — If memory grows unbounded, git operations slow down. Mitigated by 90-day pruning before each push.
  3. ruflo memory export may not capture HNSW/Q-weights — Critical unknown. Document as limitation if confirmed rather than building workarounds.
  4. Orphan branch pollution — Only stores memory-export.json, single file, minimal footprint.

Dependency Analysis

  • Depends on: ruflo-adapter.sh (existing functions ruflo_available, ruflo_with_timeout, _ruflo_run_quiet, emit_event)
  • Depends on: shipwright-pipeline.yml (existing workflow structure, GITHUB_TOKEN permission contents: write already granted)
  • Callers: ruflo_import_memory() (line 500) already called from ruflo_init(), ruflo_export_memory() (line 518) already called from ruflo_cleanup()
  • No circular dependency risks

Simplicity Check

The minimum change is 2 new functions + 2 helper functions in ruflo-adapter.sh, wiring into existing import/export, and 2 new workflow steps. This touches exactly 2 files (plus test file). The existing actions/cache steps remain as a fallback layer.

Alternatives Considered

Approach Complexity Performance Maintainability Blast Radius
A: Orphan git branch Medium (git ops, merge logic) ~5-10s overhead High (matches shipwright-data pattern) 2 files
B: Enhanced actions/cache Low ~1s Medium (cache eviction unpredictable) 1 file
C: Artifact-based Medium (API queries) ~3s Low (artifacts expire, cross-workflow issues) 2 files

Chose A: Reliable, no expiration, matches existing project conventions.

Architecture

Component Diagram

+-----------------------------------------------------+
|                  shipwright-pipeline.yml              |
|                                                       |
|  +--------------+    +---------------------------+   |
|  | Restore step |--->|  ruflo_ci_memory_pull()   |   |
|  +--------------+    |  (fetch orphan -> import)  |   |
|                       +---------------------------+   |
|         ...pipeline runs...                           |
|  +--------------+    +---------------------------+   |
|  | Save step    |--->|  ruflo_ci_memory_push()   |   |
|  | (if: always) |    |  (export -> prune -> merge |   |
|  +--------------+    |   -> push orphan)          |   |
|                       +---------------------------+   |
+-----------------------------------------------------+

+-----------------------------------------------------+
|              ruflo-adapter.sh (new functions)         |
|                                                       |
|  ruflo_ci_memory_pull()                               |
|    +-- CI guard: [[ "${CI:-}" == "true" ]]           |
|    +-- git fetch origin refs/ruflo-memory            |
|    +-- git show -> memory-export.json                |
|    +-- ruflo memory import                           |
|                                                       |
|  ruflo_ci_memory_push()                               |
|    +-- CI guard                                       |
|    +-- ruflo memory export                            |
|    +-- ruflo_prune_memory_export() (90-day)          |
|    +-- Fetch remote -> ruflo_merge_memory_exports()  |
|    +-- git push (retry 3x with jitter)               |
|                                                       |
|  ruflo_prune_memory_export()                          |
|    +-- jq filter: remove entries > N days old        |
|                                                       |
|  ruflo_merge_memory_exports()                         |
|    +-- jq: union keys, newer timestamp wins          |
+-----------------------------------------------------+

+-----------------------------------------------------+
|              refs/ruflo-memory (orphan branch)        |
|                                                       |
|  memory-export.json   <- latest merged snapshot      |
+-----------------------------------------------------+

Interface Contracts

# ruflo_ci_memory_pull(): void
# Preconditions: CI=true, ruflo available
# Postconditions: .claude-flow/data/memory-export.json written (or no-op)
# Errors: all swallowed (return 0 always)
# Side effects: git fetch, file write, ruflo memory import

# ruflo_ci_memory_push(): void
# Preconditions: CI=true, ruflo available
# Postconditions: memory-export.json pushed to refs/ruflo-memory (or no-op)
# Errors: all swallowed (return 0 always), warning emitted on 3x failure
# Side effects: git fetch, git push, file write

# ruflo_prune_memory_export(file, max_age_days): void
# Preconditions: file exists, is valid JSON
# Postconditions: entries older than max_age_days removed in-place
# Errors: no-op on invalid JSON (return 0)

# ruflo_merge_memory_exports(local, remote, output): void
# Preconditions: both files exist, valid JSON
# Postconditions: output written with union of keys, newer timestamp wins
# Errors: no-op on invalid JSON, local used as-is (return 0)

Data Flow

CI Job Start -> ruflo_ci_memory_pull()
  -> git fetch refs/ruflo-memory
  -> git show memory-export.json -> local file
  -> ruflo memory import --input <file>
  -> ruflo HNSW index populated with prior learning

CI Job End -> ruflo_ci_memory_push()
  -> ruflo memory export --output <file>
  -> prune entries > 90 days
  -> git fetch latest remote state
  -> merge local + remote (union, newer timestamp wins)
  -> git push to refs/ruflo-memory (retry 3x)

Error Boundaries

All errors are swallowed at the function boundary -- every function returns 0. Warnings are emitted via warn() and emit_event() for observability. The pipeline never fails due to memory persistence.

Critical Unknown -- Verification Plan

What does ruflo memory export capture?

The issue asks us to verify whether HNSW index and Q-learning weights are included in the export. Since we cannot run ruflo memory export in this planning context, the implementation will:

  1. Add a diagnostic log line in ruflo_ci_memory_push() that logs the export file size and top-level JSON keys
  2. Document in the code that if only KV store is exported, HNSW/Q-learning improvements don't persist -- this is a known limitation to be addressed in issue 8b

Files to Modify

  1. scripts/lib/ruflo-adapter.sh -- Add 4 new functions: ruflo_ci_memory_pull(), ruflo_ci_memory_push(), ruflo_prune_memory_export(), ruflo_merge_memory_exports(). Wire into existing ruflo_import_memory() and ruflo_export_memory().
  2. .github/workflows/shipwright-pipeline.yml -- Add 2 new workflow steps: "Restore ruflo CI memory from git branch" and "Save ruflo CI memory to git branch"
  3. scripts/sw-ruflo-adapter-test.sh -- Add tests for the 4 new functions

Implementation Steps

Step 1: Add ruflo_prune_memory_export() to ruflo-adapter.sh

Add after ruflo_index_adr_artifacts() (after line 1337). Prunes entries older than N days from export JSON using jq. Uses dual-syntax date for Linux/macOS compat. Modifies file in-place via atomic tmp+mv. No-op on missing/invalid file.

Step 2: Add ruflo_merge_memory_exports() to ruflo-adapter.sh

Add after prune function. Merges two JSON files using jq -s '.[0] * .[1]' (local overwrites remote = newer wins since local is freshly exported). Fallback: if jq fails, copy local to output.

Step 3: Add ruflo_ci_memory_pull() to ruflo-adapter.sh

CI-only function. Fetches refs/ruflo-memory orphan branch, extracts memory-export.json, imports into ruflo. All operations guarded with || return 0.

Step 4: Add ruflo_ci_memory_push() to ruflo-adapter.sh

CI-only function. Exports memory, prunes, fetches remote, merges, pushes via temp git init + orphan commit. Retries 3x with exponential jitter. Never returns non-zero.

Step 5: Wire into existing import/export functions

Add ruflo_ci_memory_pull call at the beginning of ruflo_import_memory(). Add ruflo_ci_memory_push call at the end of ruflo_export_memory().

Step 6: Add workflow steps to shipwright-pipeline.yml

Add "Restore ruflo CI memory from git branch" step after the existing cache restore step. Add "Save ruflo CI memory to git branch" step before the existing cache save step, with if: always().

Step 7: Add tests to scripts/sw-ruflo-adapter-test.sh

8 tests covering prune, merge, CI guards, and error paths.

Step 8: Run test suite

Run ./scripts/sw-ruflo-adapter-test.sh and npm test to verify no regressions.

Task Checklist

  • Task 1: Add ruflo_prune_memory_export() to scripts/lib/ruflo-adapter.sh
  • Task 2: Add ruflo_merge_memory_exports() to scripts/lib/ruflo-adapter.sh
  • Task 3: Add ruflo_ci_memory_pull() to scripts/lib/ruflo-adapter.sh
  • Task 4: Add ruflo_ci_memory_push() to scripts/lib/ruflo-adapter.sh
  • Task 5: Wire ruflo_ci_memory_pull() into ruflo_import_memory()
  • Task 6: Wire ruflo_ci_memory_push() into ruflo_export_memory()
  • Task 7: Add "Restore ruflo CI memory" step to shipwright-pipeline.yml
  • Task 8: Add "Save ruflo CI memory" step to shipwright-pipeline.yml
  • Task 9: Add unit tests for prune, merge, pull, and push functions
  • Task 10: Run existing test suite to verify no regressions
  • Task 11: Document HNSW/Q-weight export limitation inline

Task Decomposition (with explicit dependencies)

  1. Task 1: Add ruflo_prune_memory_export() -- no dependencies
  2. Task 2: Add ruflo_merge_memory_exports() -- no dependencies
  3. Task 3: Add ruflo_ci_memory_pull() -- depends on none (pull doesn't prune/merge)
  4. Task 4: Add ruflo_ci_memory_push() -- depends on Task 1, 2 (uses prune before push, merge on conflict)
  5. Task 5: Wire pull into ruflo_import_memory() -- depends on Task 3
  6. Task 6: Wire push into ruflo_export_memory() -- depends on Task 4
  7. Task 7: Add workflow restore step -- depends on Task 3
  8. Task 8: Add workflow save step -- depends on Task 4
  9. Task 9: Add tests -- depends on Tasks 1-4
  10. Task 10: Run full test suite -- depends on all code changes
  11. Task 11: Document HNSW limitation -- no dependencies

Risk Analysis

Risk What Could Break Mitigation
Push conflicts (concurrent pipelines) Memory from one job lost 3-retry with jitter, same pattern as shipwright-data branch
Large memory file Slow git operations, timeout 90-day pruning caps growth; ruflo_with_timeout bounds operations
Invalid JSON from ruflo export jq crashes in prune/merge All jq operations wrapped in fallbacks, no-op on error
GITHUB_TOKEN insufficient Push fails Already has contents: write in workflow permissions
git show on non-existent branch Error output `2>/dev/null
Orphan branch visible in git branch User confusion Using refs/ruflo-memory ref (not refs/heads/), won't show
date -d not available on macOS CI Pruning fails silently Dual date syntax: date -d (GNU) fallback to date -v (BSD)
Temp dir push pollutes caller state Working dir changes All git ops in subshell ( cd ... )

Testing Approach

Test Pyramid Breakdown

  • 8 unit tests covering prune, merge, CI guard, and error paths
  • 0 integration tests (would require actual git remote -- covered by CI run itself)
  • 0 E2E tests (the first CI run after merge IS the E2E test)

Coverage Targets

  • 100% of new functions have at least one positive and one negative test
  • All error paths verified to return 0

Critical Paths to Test

  • Happy path: Prune removes old entries; merge produces union; CI guard skips when CI!=true
  • Error case 1: Missing file -> no-op, return 0
  • Error case 2: Invalid JSON -> no-op, return 0
  • Edge case 1: Empty export file -> skip push
  • Edge case 2: CI=false -> all CI functions are no-op

Definition of Done

  • ruflo_ci_memory_pull() restores memory from orphan branch in CI
  • ruflo_ci_memory_push() persists memory to orphan branch in CI
  • 90-day pruning runs before every push
  • Merge handles concurrent writes (union, newer timestamp wins)
  • All error paths return 0 -- pipeline never fails on memory errors
  • Tests pass: ./scripts/sw-ruflo-adapter-test.sh
  • Full test suite passes: npm test
  • GITHUB_TOKEN with contents: write is sufficient (verified by workflow permissions)
  • HNSW/Q-weight export limitation documented in code comments
  • New functions follow existing patterns: ruflo_available guard, emit_event, ruflo_with_timeout

Baseline Metrics

  • Current CI pipeline duration: ~8332s build, ~771s test (from baselines)
  • Expected overhead: ~5-10s for git fetch/push operations
  • Memory export size: Unknown until first run -- logged for observation

Optimization Targets

  • Keep total memory persistence overhead under 30s
  • Memory export file under 1MB after pruning

Profiling Strategy

  • Log export file size in ruflo_ci_memory_push() via emit_event
  • Log elapsed time for git operations via existing event system
  • Monitor orphan branch growth via git diagnostic events

Benchmark Plan

  • Before: CI run without memory persistence (current baseline)
  • After: CI run with memory persistence enabled
  • Success criteria: Total overhead < 30s, no pipeline failures attributed to memory ops

Clone this wiki locally