Skip to content

Pipeline Design 371

Jump to bottom
ezigus edited this page Apr 21, 2026 · 1 revision

Architecture Decision Record: feat(ruflo): inject ruflo_recall_similar_outcomes into stage_build

Context

The build stage (stage_build()) launches a hive-mind agent team with a goal string containing plan and design context, but no historical signal about how similar issues were built in the past. Ruflo memory stores vector-indexed build outcomes from prior executions; recalling similar outcomes before hive dispatch would seed agents with probabilistic guidance based on past patterns.

Constraints:

  • Must not modify ruflo_execute_build_hive() signature (issue #371 explicitly forbids this)
  • Must be safe when ruflo unavailable (guard with declare -f + ruflo_available)
  • Must never block the pipeline (|| true semantics)
  • Must follow existing injection patterns from stage_plan and stage_design in stage_intake.sh
  • Scope: single file, minimal code footprint (~15–45 lines for 3 injection points)

Decision

Inject ruflo_recall_similar_outcomes before hive dispatch by appending its output to the enriched_goal string.

The recall result is appended under a clear ## Historical Build Context header, making it visible in the goal string passed to the hive. This approach:

  1. Requires no changes to ruflo_execute_build_hive() or other function signatures
  2. Leverages the existing goal-string enrichment pattern already used in prior stages
  3. Fails gracefully when ruflo is unavailable or function missing
  4. Bounds output to 2000 chars to prevent argv overflow
  5. Works across all three build paths: hive dispatch, single-agent fallback, and native sw loop

Component Decomposition

┌─────────────────────────────────────────────────────────────────┐
│  stage_build() — Build Stage Main Entry Point                   │
├─────────────────────────────────────────────────────────────────┤
│                                                                   │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │ 1. Initialization                                        │   │
│  │    (Unchanged: load context, construct base goal)        │   │
│  └──────────────────────────────────────────────────────────┘   │
│                     ↓                                            │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │ 2. Historical Context Injection (NEW)                    │   │
│  │    • Guard: function exists + ruflo available            │   │
│  │    • Recall: ruflo_recall_similar_outcomes(...)          │   │
│  │    • Bound: head -c 2000 (prevent overflow)              │   │
│  │    • Append: enriched_goal += context header + result    │   │
│  └──────────────────────────────────────────────────────────┘   │
│                     ↓                                            │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │ 3. Execution Dispatch                                    │   │
│  │    Path A: Hive (if hive available)                      │   │
│  │            → ruflo_execute_build_hive "$enriched_goal"   │   │
│  │    Path B: Single-agent fallback                         │   │
│  │            → ruflo_execute_build "$enriched_goal"        │   │
│  │    Path C: Native sw loop fallback                       │   │
│  │            → shipwright loop (enriched goal in prompt)   │   │
│  └──────────────────────────────────────────────────────────┘   │
│                     ↓                                            │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │ 4. Result & Outcome Recording (Unchanged)                │   │
│  └──────────────────────────────────────────────────────────┘   │
│                                                                   │
└─────────────────────────────────────────────────────────────────┘

Components:

  1. Guard Layer — Availability checks

    • Validates ruflo_recall_similar_outcomes function exists
    • Validates ruflo_available function exists and returns true
    • Responsibility: Prevent failures when ruflo unavailable

  2. Recall Layer — Historical outcome retrieval

    • Calls ruflo_recall_similar_outcomes(issue_type, labels)
    • Captures output with 2>/dev/null || true (never fails)
    • Responsibility: Fetch similar past outcomes from memory

  3. Integration Layer — Goal enrichment

    • Bounds output to 2000 chars with head -c 2000
    • Appends under ## Historical Build Context header
    • Updates enriched_goal variable
    • Logs injection via info message
    • Responsibility: Safely integrate context into goal string

  4. Dispatch Layer — Unchanged, consumes enriched goal

    • Passes updated enriched_goal to hive/fallback paths
    • Responsibility: Execute build with historical context

Interface Contracts

Function: ruflo_recall_similar_outcomes 

# Input
issue_type: string       # e.g., "backend", "frontend", "devops"
labels: string          # comma-separated: "bug,urgent,regression"

# Output
stdout: string          # Plain text (NOT JSON) of similar past outcomes
                        # Empty string if no matches found
                        # May span multiple lines

# Error handling
# If function missing or ruflo unavailable:
#   Guard prevents call → enriched_goal unchanged
# If recall fails:
#   2>/dev/null captures stderr, || true continues
#   enriched_goal unchanged

# Example return value (plain text):
# "Similar build failed due to missing dependency lodash@4.17.20 in stage_intake.
#  Previous fix: updated npm lockfile and re-ran bootstrap.
#  Another match: build succeeded with plan-first approach on retry."

Function: ruflo_available 

# Input: (none)

# Output
# Returns 0 (success) if ruflo memory is available
# Returns non-zero if unavailable or uninitialized

# Side effects: None (pure availability check)

Data: enriched_goal variable

# Before injection:
enriched_goal="## Plan Context\n..."

# After injection:
enriched_goal="## Plan Context\n...
## Design Context\n...
## Historical Build Context\n<recall_output>"

# Invariant: Goal string is passed as-is to hive/fallback executors
#            No parsing or transformation; treated as opaque string

Data Flow

╔════════════════════════════════════════════════════════════════╗
║ Upstream: stage_plan → stage_design                           ║
╠════════════════════════════════════════════════════════════════╣
║ enriched_goal contains plan + design context                  ║
║ INTELLIGENCE_ISSUE_TYPE = intake detection (e.g., "backend")  ║
║ ISSUE_LABELS = comma-sep intake extraction                    ║
╚════════════════════════════════════════════════════════════════╝
                           ↓
╔════════════════════════════════════════════════════════════════╗
║ [NEW] stage_build: Historical Context Injection                ║
╠════════════════════════════════════════════════════════════════╣
║ IF declare -f ruflo_recall_similar_outcomes ✓                 ║
║    AND declare -f ruflo_available ✓                           ║
║    AND ruflo_available ✓:                                     ║
║                                                                ║
║    _ruflo_build_ctx = ruflo_recall_similar_outcomes(          ║
║        INTELLIGENCE_ISSUE_TYPE, ISSUE_LABELS)                 ║
║                                                                ║
║    IF _ruflo_build_ctx not empty:                             ║
║        Bound to 2000 chars                                    ║
║        enriched_goal += "\n\n## Historical Build Context\n"   ║
║                         + _ruflo_build_ctx                    ║
║        Log: "Ruflo build context injected (N bytes)"          ║
║ ELSE:                                                          ║
║    enriched_goal unchanged (silent pass-through)              ║
╚════════════════════════════════════════════════════════════════╝
                           ↓
╔════════════════════════════════════════════════════════════════╗
║ Dispatch Paths (all receive updated enriched_goal)             ║
╠════════════════════════════════════════════════════════════════╣
║ Path A: ruflo_execute_build_hive "$enriched_goal"             ║
║ Path B: ruflo_execute_build "$enriched_goal"                  ║
║ Path C: shipwright loop <goal embedded in prompt>             ║
╚════════════════════════════════════════════════════════════════╝
                           ↓
╔════════════════════════════════════════════════════════════════╗
║ Hive Agents / Single Agent / Loop Prompt                       ║
║ (sees full context: plan + design + history)                  ║
╚════════════════════════════════════════════════════════════════╝

Error Boundaries

Layer Error Case Handling Consequence
Guard ruflo_recall_similar_outcomes missing Skip entire block, enriched_goal unchanged Safe: builds without history
Guard ruflo_available missing Skip entire block, enriched_goal unchanged Safe: builds without history
Guard ruflo_available() returns false Skip entire block, enriched_goal unchanged Safe: builds without history
Recall ruflo_recall_similar_outcomes() fails/hangs 2>/dev/null || true captures error, empty string Safe: enriched_goal unchanged
Recall Returns empty string [[ -n "$_ruflo_build_ctx" ]] check skips append Safe: no empty headers in goal
Bound Output > 2000 chars head -c 2000 truncates silently Safe: prevents argv overflow
Append Variable expansion in heredoc Info log message uses printf, not unquoted Safe: no shell injection
Dispatch Hive receives invalid goal string Hive processes as-is; malformed context ≠ blocker Degraded: agents may ignore context

Alternatives Considered

Alternative Rationale Trade-offs Decision
Separate parameter to ruflo_execute_build_hive() Cleaner separation, structured data ❌ Modifies function signature, breaks existing callers, issue forbids it REJECTED
Pre-process goal in enriched_goal construction Earlier in pipeline, before branching ❌ Spreads logic across stages, harder to test in isolation, implicit intent REJECTED
Store context in temp file, pass path Avoids goal string length concerns ❌ Over-engineering for bash, goal string already works, adds I/O REJECTED
Inject context as prompt prefix, not goal Hive prompt more flexible than goal ❌ Requires modifying hive invocation, out of scope REJECTED
Append to goal (CHOSEN) Minimal code, follows existing pattern, no signature changes ✓ Goal string grows (mitigated: 2000-char cap), accepted trade-off ACCEPTED

Implementation Plan

Files to Modify

File Lines Changes
scripts/lib/pipeline-stages-build.sh ~412 (hive path), ~439 (single-agent), ~538 (native loop) Add 15–16 line recall injection block before each dispatch; 3 copies with local variable scoping

Files to Create

None. Changes are isolated to one existing file.

New Dependencies

None. Leverages existing ruflo-adapter functions and bash builtins.

Risk Areas

  1. Argument Length Overflow

    • Risk: enriched_goal grows unbounded, exceeds shell ARG_MAX limit (~131KB on Linux)
    • Mitigation: Bound recall output to 2000 chars with head -c 2000
    • Probability: Low (2000 chars safe for bash/C arg limits)

  2. Namespace Mismatch

    • Risk: ruflo_recall_similar_outcomes searches wrong namespace, returns empty
    • Mitigation: Use namespace derived from issue type/labels (implementation detail of ruflo adapter)
    • Probability: Low (ruflo-adapter defines contract)

  3. Silent Failures

    • Risk: Recall fails silently, goal unchanged but no visibility
    • Mitigation: Add info log when context injected; errors captured by 2>/dev/null
    • Probability: Medium (network/memory failures possible)

  4. Hive Confusion

    • Risk: Historical context confuses hive agents, degrades decision quality
    • Mitigation: Clear header ## Historical Build Context makes source explicit; hive can ignore if irrelevant
    • Probability: Low (agents are trained to handle long context)

  5. Performance Regression

    • Risk: Recall adds latency to build stage startup
    • Mitigation: Non-blocking call, pipeline continues regardless; issue #313 blocks real benefit anyway
    • Probability: Low (semantic search is <100ms)

Validation Criteria

  • Code Review: Diff shows only expected recall injection blocks in pipeline-stages-build.sh; no unrelated changes
  • Guard Logic: Both declare -f checks pass; function signatures match ruflo-adapter
  • Output Bounds: head -c 2000 truncates outputs > 2000 chars; verify with test input
  • Variable Isolation: Local _ruflo_build_ctx variable scoped to block; doesn't leak to outer scope
  • Enriched Goal: Appended text follows format \n\n## Historical Build Context\n<result>; no extra headers
  • Info Logging: info message logs bytes appended when context injected
  • Error Handling: Enriched goal unchanged when:
    • ruflo_recall_similar_outcomes function missing
    • ruflo_available function missing
    • ruflo_available() returns false
    • Recall function fails (stderr captured, stdout empty)
  • All Paths Covered: Recall injection implemented at all 3 dispatch points (hive ~412, single-agent ~439, native loop ~538)
  • Test Suite: ./scripts/sw-pipeline-test.sh passes with zero new failures
  • General Tests: npm test passes; all vitest suites green
  • Build Success: npm run build completes without errors
  • No Regressions: Hive behavior unchanged when ruflo unavailable; fallback paths behave identically to pre-change
  • Commit Message: References issue #371; explains "why" not just "what"

Success Metrics

  1. Acceptance Criteria from Issue #371: All 6 criteria met

    • ✓ ruflo_recall_similar_outcomes called before hive dispatch
    • ✓ Result appended to enriched_goal with ## Historical Build Context header
    • ✓ Guard uses declare -f + ruflo_available pattern
    • ✓ Never blocks pipeline (|| true)
    • ✓ Works when ruflo unavailable (goal unchanged)
    • ✓ Single-agent + native loop fallbacks also covered

  2. Code Quality: Single file, ~45 lines total (3 copies × 15 lines); follows existing patterns

  3. Safety: Zero breaking changes; backward-compatible when ruflo missing

  4. Test Coverage: Pipeline tests + npm test suite pass; no regressions

Implementation Checklist

  • Read scripts/lib/pipeline-stages-build.sh to locate hive (~412), single-agent (~439), native loop (~538) dispatch blocks
  • Read scripts/lib/pipeline-stages-intake.sh lines 374–400 to copy guard/append pattern
  • Verify scripts/lib/ruflo-adapter.sh for ruflo_recall_similar_outcomes(issue_type, labels) signature
  • Implement recall injection block before hive dispatch
  • Implement recall injection block before single-agent fallback
  • Implement recall injection block before native loop fallback
  • Add 2000-char bound with head -c 2000
  • Add info log message when context injected
  • Run ./scripts/sw-pipeline-test.sh — verify no new failures
  • Run npm test — verify all tests pass
  • Run npm run build — verify build succeeds
  • Stage changes: git add scripts/lib/pipeline-stages-build.sh
  • Commit with message: feat(ruflo): inject ruflo_recall_similar_outcomes into stage_build
  • Verify final git diff shows ~45 lines of additions only (no deletions/modifications)

Next Steps After Merge

  1. Monitor: Track how often recall context is used vs. ignored in hive decisions
  2. Unblock #313: Implement CI memory persistence to realize full performance benefit
  3. Iterate: Gather telemetry on hive accuracy with/without historical context

Clone this wiki locally