Pipeline Plan 370

Implementation Plan: Add Memory Recall and Store to stage_intake

Executive Summary

Add ruflo_recall_similar_outcomes() and ruflo_store() calls to stage_intake() to enrich downstream stages with historical context about similar issue classifications. This is the earliest injection point in the pipeline—before plan/design/build stages—and follows the exact pattern already established in stage_plan() and stage_design().

Architecture Design & Data Flow

Component Diagram

┌─────────────────────────────────────────────────────────────────┐
│                        stage_intake()                            │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ 1. Fetch issue metadata (title, labels, body, etc.)    │   │
│  └─────────────────────────────────────────────────────────┘   │
│                            ↓                                      │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ 2. Classify issue (INTELLIGENCE_ISSUE_TYPE, severity)  │   │
│  └─────────────────────────────────────────────────────────┘   │
│                            ↓                                      │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ 3. [NEW] ruflo_recall_similar_outcomes()                │   │
│  │    Query historical context for this issue type         │   │
│  │    Export as INTELLIGENCE_INTAKE_CTX                    │   │
│  └─────────────────────────────────────────────────────────┘   │
│                            ↓                                      │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ 4. Save intake.json artifact                            │   │
│  └─────────────────────────────────────────────────────────┘   │
│                            ↓                                      │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ 5. [NEW] ruflo_store()                                  │   │
│  │    Store classification result for downstream lookup    │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘
                            ↓
         ┌───────────────────┴───────────────────┐
         ↓                                       ↓
    stage_plan()                            stage_design()
  [Consumes                                [Consumes
   INTELLIGENCE_INTAKE_CTX                  INTELLIGENCE_INTAKE_CTX
   in prompt injection]                     in prompt injection]

Interface Contracts

ruflo_recall_similar_outcomes()

# Input:
#   $1 (INTELLIGENCE_ISSUE_TYPE)  — Issue type: "backend", "frontend", "bug", etc.
#   $2 (ISSUE_LABELS)             — Comma-separated labels (e.g., "feature,priority/high")
#
# Output:
#   stdout — Plain text context summary or empty string if no results
#   exit code — 0 always (fail-open)
#
# Called from:
#   stage_intake (new), stage_plan, stage_design
#
# Example:
#   local ctx; ctx=$(ruflo_recall_similar_outcomes "backend" "feature,urgent" 2>/dev/null || true)

ruflo_store()

# Input:
#   $1 (key)        — "stage-intake-result"
#   $2 (value)      — Classification summary (plain text, <500 chars)
#   $3 (namespace)  — "pipeline-${SHIPWRIGHT_PIPELINE_ID:-unknown}"
#
# Output:
#   exit code — 0 on success, non-zero on failure
#   Side effect: Stores in ruflo memory for retrieval by stage_plan/design
#
# Called from:
#   stage_intake (new), stage_test
#
# Example:
#   ruflo_store "stage-intake-result" \
#     "Issue type: backend. Severity: critical. Labels: security,bug." \
#     "pipeline-${SHIPWRIGHT_PIPELINE_ID:-unknown}" || true

INTELLIGENCE_INTAKE_CTX (new export)

# Environment variable exported by stage_intake
#
# Type: string (plain text, <2000 chars)
# Visibility: Exported to parent shell, consumed by stage_plan and stage_design
# Format: Unstructured text from ruflo_recall_similar_outcomes
#
# Downstream consumption:
#   In stage_plan and stage_design, append to prompts under ## Intake Context section
#   if [[ -n "$INTELLIGENCE_INTAKE_CTX" ]]; then
#     prompt="${prompt}\n## Intake Context\n${INTELLIGENCE_INTAKE_CTX}"
#   fi

Error Boundaries

Component	Error	Handling	Propagation
`ruflo_available` returns false	Skip recall & store	Silent no-op with `\|\| true`	None (graceful)
`ruflo_recall_similar_outcomes` times out	Return empty string	Captured with `2>/dev/null \|\| true`	None (graceful)
`ruflo_store` fails	Acknowledge in log but continue	`\|\| true` prevents exit	None (non-blocking)
INTELLIGENCE_INTAKE_CTX not set	Use default empty string	Guard all downstream usage with `[[ -n ... ]]`	Safe default

Files to Modify

scripts/lib/pipeline-stages-intake.sh — Add recall/store calls to stage_intake()
scripts/lib/pipeline-stages-intake.sh — Update stage_plan() to consume INTELLIGENCE_INTAKE_CTX
scripts/lib/pipeline-stages-intake.sh — Update stage_design() to consume INTELLIGENCE_INTAKE_CTX
scripts/sw-lib-pipeline-stages-test.sh — Add tests for ruflo integration in stage_intake

Implementation Steps

Step 1: Add ruflo_recall_similar_outcomes() Call After Issue Classification

File: scripts/lib/pipeline-stages-intake.sh
Location: After line 108 (after skill_analyze_issue block), before line 110 (log_stage call)

    # Inject ruflo vector-similar past outcomes for this issue type (intake stage).
    # This is the earliest injection point — all downstream stages can reference this context.
    if declare -f ruflo_recall_similar_outcomes >/dev/null 2>&1 && \
       declare -f ruflo_available >/dev/null 2>&1 && \
       ruflo_available; then
        local _ruflo_intake_ctx
        _ruflo_intake_ctx=$(ruflo_recall_similar_outcomes \
            "${INTELLIGENCE_ISSUE_TYPE:-general}" "${ISSUE_LABELS:-}" 2>/dev/null || true)
        if [[ -n "$_ruflo_intake_ctx" ]]; then
            INTELLIGENCE_INTAKE_CTX="$_ruflo_intake_ctx"
            export INTELLIGENCE_INTAKE_CTX
            info "Ruflo intake context loaded (${#_ruflo_intake_ctx} bytes)"
        fi
    fi

Rationale: Placed immediately after skill analysis completes but before stage logging, so the enriched context flows into intake.json artifact and downstream stages.

Step 2: Add ruflo_store() Call After Intake Summary

File: scripts/lib/pipeline-stages-intake.sh
Location: After line 95 (after save_artifact call), before line 110 (log_stage call)

    # Ruflo: store intake classification result for cross-stage context
    if declare -f ruflo_store >/dev/null 2>&1; then
        local _intake_summary
        _intake_summary="Issue type: ${INTELLIGENCE_ISSUE_TYPE:-unknown}. Severity: ${INTELLIGENCE_SEVERITY:-unknown}. Labels: ${ISSUE_LABELS:-none}."
        ruflo_store "stage-intake-result" \
            "$_intake_summary" \
            "pipeline-${SHIPWRIGHT_PIPELINE_ID:-unknown}" || true
    fi

Rationale: Stores the classification result so downstream stages (stage_test, future stages) can recall intake context without re-classifying.

Step 3: Update stage_plan() to Consume INTELLIGENCE_INTAKE_CTX

File: scripts/lib/pipeline-stages-intake.sh
Location: After line 407 (after ruflo memory availability block, before line 409 guard_prompt_size)

    # Inject intake enrichment context if available from earlier stage
    if [[ -n "${INTELLIGENCE_INTAKE_CTX:-}" ]]; then
        plan_prompt="${plan_prompt}
## Intake Context (Classification Context)
${INTELLIGENCE_INTAKE_CTX}
"
    fi

Rationale: Makes intake-derived context explicitly visible to the planner, reducing redundant work.

Step 4: Update stage_design() to Consume INTELLIGENCE_INTAKE_CTX

File: scripts/lib/pipeline-stages-intake.sh
Location: After line 922 (after ruflo recall block in design), before line 924 (prior stage context block)

    # Inject intake enrichment context if available from earlier stage
    if [[ -n "${INTELLIGENCE_INTAKE_CTX:-}" ]]; then
        design_prompt="${design_prompt}
## Intake Context (Classification Context)
${INTELLIGENCE_INTAKE_CTX}
"
    fi

Rationale: Allows design stage to leverage intake classification without re-querying ruflo.

Task Checklist

Testing Approach

Test Pyramid Breakdown

Unit Tests (65%): 12 tests

Recall available/unavailable paths (2 tests)
Store called with correct key/namespace (2 tests)
INTELLIGENCE_INTAKE_CTX exported and available (2 tests)
Downstream stage consumption (4 tests)
No-op behavior when ruflo unavailable (2 tests)

Integration Tests (30%): 5 tests

Full stage_intake → stage_plan flow with context propagation (2 tests)
Full stage_intake → stage_design flow with context propagation (2 tests)
Pipeline-level INTELLIGENCE_INTAKE_CTX persistence across stages (1 test)

E2E Tests (5%): 1 test

Full pipeline execution (intake → plan → design) with ruflo enabled verifies context flows end-to-end

Coverage Targets

Intake recall path: 100% coverage (declare -f guard, ruflo_available guard, context export)
Intake store path: 100% coverage (declare -f guard, store call, namespace format)
Downstream consumption: 100% coverage (guard with [[ -n ... ]], append to prompt)

Critical Paths to Test

Happy Path:

# stage_intake with ruflo available
export SHIPWRIGHT_PIPELINE_ID="test-pipeline-123"
export INTELLIGENCE_ISSUE_TYPE="backend"
export ISSUE_LABELS="feature,urgent"
ruflo_available() { return 0; }
ruflo_recall_similar_outcomes() { echo "Similar issue: auth module rewrite (3 days)"; }
ruflo_store() { return 0; }

stage_intake  # Should export INTELLIGENCE_INTAKE_CTX, call store with correct namespace
[[ -n "$INTELLIGENCE_INTAKE_CTX" ]] || exit 1

Error Case 1: ruflo unavailable

# stage_intake with ruflo unavailable (declare -f fails)
unset -f ruflo_recall_similar_outcomes ruflo_available ruflo_store
stage_intake  # Should complete successfully with no-op, no export
[[ -z "${INTELLIGENCE_INTAKE_CTX:-}" ]] || exit 1

Error Case 2: ruflo_available returns false

# stage_intake with ruflo_available=false
ruflo_available() { return 1; }
stage_intake  # Should skip recall/store, complete successfully

Edge Case: Downstream consumption

# stage_plan with INTELLIGENCE_INTAKE_CTX set
export INTELLIGENCE_INTAKE_CTX="Backend auth module, 2-day priority"
stage_plan  # plan_prompt should contain ## Intake Context section with value
grep -q "Intake Context" "$ARTIFACTS_DIR/plan.md" || exit 1

Design Alternatives Considered

Option A: Sequential Call Pattern (Recommended ✓)

Description: Add recall and store calls sequentially in stage_intake() after classification, follow exact pattern from stage_plan()/stage_design().

Pros:

✓ Minimal code changes (2 code blocks in 1 file)
✓ Follows established patterns — no new patterns introduced
✓ Earliest injection point (issue classification context available immediately)
✓ Recall happens before storage (safe ordering)
✓ Export of INTELLIGENCE_INTAKE_CTX is straightforward

Cons:

Adds ~20ms latency to intake stage (negligible, ~2% overhead)

Blast Radius: Minimal — only stage_intake() modified directly; downstream stages enhanced but not broken

Option B: Deferred Recall in plan/design Only

Description: Skip intake recall/store; let plan/design stages continue existing recall pattern.

Pros:

Simplest (no changes to intake)
No additional intake latency

Cons:

✗ Violates issue requirement: "earliest injection point"
✗ Plan/design stages re-query without intake context
✗ Duplicates recall calls (wasteful)
✗ Misses opportunity to enrich downstream with classification context

Decision: Rejected — directly contradicts issue goal

Option C: Separate Memory Initialization Stage

Description: Create a new pipeline stage before plan that handles all memory initialization.

Pros:

Separates concerns (classification vs. memory)

Cons:

✗ Adds pipeline complexity (new stage to manage)
✗ Breaks established pattern of in-stage recall/store
✗ Overkill for 2 function calls
✗ Makes testing more complex (mocking new stage)

Decision: Rejected — unnecessary abstraction for this scope

Risk Analysis

Risk	Impact	Likelihood	Mitigation
`ruflo_available` undefined	Pipeline blocks or no-op	Low	Guard with `declare -f`, fail-open with `\|\| true`
`ruflo_recall_similar_outcomes` hangs	Intake timeout	Very Low	Timeout wrapper or `2>/dev/null \|\| true`
`ruflo_store` fails silently	No downstream context stored	Low	Accepted (non-blocking, logged)
INTELLIGENCE_INTAKE_CTX not exported	Downstream stages don't consume it	Medium	Explicitly test export, update downstream
Merge conflicts in `stage_plan/design`	Rebasing issues	Low	Refactor into separate append sections
Regression in existing tests	Test suite breaks	Medium	Run full test suite before merging

Mitigation Strategy:

Use declare -f guards for all ruflo function checks
Use || true to prevent blocking
Test both available and unavailable ruflo paths
Verify downstream stages import INTELLIGENCE_INTAKE_CTX and use it safely
Run full test suite (./scripts/sw-lib-pipeline-stages-test.sh && npm test)

Definition of Done

✓ Acceptance Criteria (from issue):

ruflo_recall_similar_outcomes is called inside stage_intake() after issue classification
Result is exported as INTELLIGENCE_INTAKE_CTX for downstream consumption
ruflo_store "stage-intake-result" is called with the classification summary
Both calls are guarded with the standard availability check (declare -f ... && ruflo_available)
Both calls use || true — never block the pipeline
Passes ./scripts/sw-pipeline-test.sh and npm test
Works when ruflo_available returns false (graceful no-op)

✓ Implementation Quality:

Code follows existing patterns in stage_plan() and stage_design() (lines 374–386, 910–922)
Variable names match convention (INTELLIGENCE_INTAKE_CTX, _ruflo_intake_ctx, _intake_summary)
Error messages are informative (e.g., "Ruflo intake context loaded")
No new external dependencies introduced
File organization preserved (all changes in scripts/lib/pipeline-stages-intake.sh)

✓ Testing Quality:

Unit tests for both ruflo available/unavailable paths
Tests verify INTELLIGENCE_INTAKE_CTX is exported correctly
Tests verify downstream consumption in stage_plan() and stage_design()
No regression in existing tests
All new tests follow existing pattern from sw-lib-pipeline-stages-test.sh

✓ Documentation:

Code comments explain why recall happens at intake (earliest injection)
Namespace format documented ("pipeline-${SHIPWRIGHT_PIPELINE_ID:-unknown}")
Export behavior documented (INTELLIGENCE_INTAKE_CTX available to downstream)

Socratic Design Reasoning

Requirements Clarity

Q: What is the minimum viable change?
A: Add 2 ruflo calls (recall + store) with guards to stage_intake(). Minimum scope, maximum impact.

Q: Are there implicit requirements?
A: Yes — INTELLIGENCE_INTAKE_CTX must flow to downstream stages (stage_plan, stage_design). This requires updating both those stages to consume it.

Q: What are the acceptance criteria?
A: Given in issue. We must verify:

Recall/store calls execute when ruflo available
Graceful no-op when ruflo unavailable
Test suite passes
Context flows downstream (verified by inspection + test)

Dependency Analysis

Q: What depends on what?

stage_intake() → calls ruflo_recall_similar_outcomes(), ruflo_store()
stage_plan() → depends on INTELLIGENCE_INTAKE_CTX export (new dependency)
stage_design() → depends on INTELLIGENCE_INTAKE_CTX export (new dependency)
Tests → mock ruflo_available(), ruflo_recall_similar_outcomes(), ruflo_store()

Q: Circular dependency risks?
None identified. Flow is linear: intake → plan/design → build/test.

Summary

This implementation adds upstream memory enrichment to the Shipwright pipeline, allowing all downstream stages to benefit from intake-level issue classification without redundant queries. The changes follow established patterns, add no new dependencies, and are fully backward-compatible (graceful no-op when ruflo unavailable).

Pipeline Plan 370

Implementation Plan: Add Memory Recall and Store to stage_intake

Executive Summary

Architecture Design & Data Flow

Component Diagram

Interface Contracts

Error Boundaries

Files to Modify

Implementation Steps

Step 1: Add ruflo_recall_similar_outcomes() Call After Issue Classification

Step 2: Add ruflo_store() Call After Intake Summary

Step 3: Update stage_plan() to Consume INTELLIGENCE_INTAKE_CTX

Step 4: Update stage_design() to Consume INTELLIGENCE_INTAKE_CTX

Task Checklist

Testing Approach

Test Pyramid Breakdown

Coverage Targets

Critical Paths to Test

Design Alternatives Considered

Option A: Sequential Call Pattern (Recommended ✓)

Option B: Deferred Recall in plan/design Only

Option C: Separate Memory Initialization Stage

Risk Analysis

Definition of Done

Socratic Design Reasoning

Requirements Clarity

Dependency Analysis

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!