Pre-Flight Budget Forecast with In-Flight Cost Alerting

[github-actions]

Strategic Improvement

Pipelines often hit budget limits mid-execution, wasting partial progress. Build a forecast engine that predicts total cost before pipeline starts (based on issue complexity and historical data) and provides real-time cost alerts during execution. Prevents wasted partial runs and enables budget-aware scheduling.

Acceptance Criteria

• Predict pipeline cost before start using issue complexity + historical stage costs
• Display forecast in pipeline intake stage output
• Track actual vs forecasted cost during execution
• Alert when 80% of forecasted budget consumed
• Add forecast accuracy metric to cost dashboard
• Reject pipeline start if forecast exceeds remaining budget (when strict mode enabled)

Context

• Priority: P3
• Complexity: fast
• Generated by: Strategic Intelligence Agent
• Strategy alignment: P3: Cost Efficiency

[github-actions]

⚓ Shipwright Pipeline Starting

Field	Value
Template	standard
Complexity	standard
Max Iterations	20
Triage	Cost forecast with real-time alerting aligns with Shipwright's cost intelligence strategy; requires integrating cost tracking, complexity scoring, pipeline intake, and dashboard systems using existing infrastructure.
Runner	ubuntu-latest
Risk Level	unknown
Workflow	24541122175

The autonomous pipeline is processing this issue. Updates will be posted as stages complete.

[github-actions]

[github-actions]

[github-actions]

📋 Implementation Plan

Click to expand full plan (622 lines)

Based on my analysis of the Shipwright codebase, I'll create a comprehensive implementation plan for the Pre-Flight Budget Forecast feature.

IMPLEMENTATION PLAN: Pre-Flight Budget Forecast with In-Flight Cost Alerting

---

Analysis & Design Decisions

Requirements Clarity

Minimum viable change: Core features that satisfy all acceptance criteria:

1. Pre-pipeline cost estimation using historical data + complexity
2. Forecast display in intake stage output
3. Real-time actual vs. forecast tracking during pipeline
4. 80% budget alert mechanism
5. Forecast accuracy metric in cost dashboard

Implicit requirements identified:

• Forecast must gracefully degrade if insufficient historical data exists
• Strict mode (reject if forecast exceeds budget) is optional, not default
• Must not block pipeline start if forecast fails
• Should track per-issue accuracy for learning

Alternatives Considered

Option A: Simple Historical Average (Recommended)

• Group historical costs by complexity level (low/medium/high/very_high)
• Calculate average stage costs per complexity level
• For new pipeline: detect complexity → apply stage averages → sum for total
• Pros: Simple, predictable, deterministic, fast, easy to debug
• Cons: Ignores recent improvements/regressions
• Blast radius: Minimal—new module, no changes to existing cost tracking
• Complexity: ~200 LOC

Option B: Complexity-Weighted Regression

• Use linear regression on historical (complexity_score, total_cost) pairs
• More accurate but requires ML library or custom implementation
• Pros: More accurate forecasts
• Cons: Complex, harder to debug when forecasts are wrong, slower
• Blast radius: Moderate—requires new dependencies or ~400 LOC of math
• Complexity: ~400 LOC

Option C: Per-Stage Model Prediction

• For each stage, predict which model (haiku/sonnet/opus) based on complexity
• Multiply by known average tokens per model per stage
• Pros: Transparent, shows model routing decisions
• Cons: Requires stage-level historical data, fragile if token estimates change
• Blast radius: Large—affects multiple stages

Choice: Option A (Simple Historical Average) because it satisfies MVP, has minimal blast radius, and can be upgraded to Option B later without breaking changes.

✅ Task Checklist

• Task 1: Create cost-forecast.sh with forecast_init, detect_issue_complexity, calculate_forecast
• Task 2: Integrate forecast display into intake stage output
• Task 3: Add real-time actual vs. forecast tracking to cost_record
• Task 4: Implement 80% alert check in loop iteration
• Task 5: Add forecast accuracy calculation post-pipeline
• Task 6: Extend sw-cost show with forecast metrics dashboard
• Task 7: Implement --strict-budget flag with override option
• Task 8: Create comprehensive test suite (sw-cost-forecast-test.sh)
• Task 9: Integration test: full pipeline with forecast tracking
• Task 10: E2E test: multiple pipelines improving forecast accuracy
• Task 11: Documentation: Add cost forecasting guide to CLAUDE.md
• Task 12: Verify all existing tests still pass
• Task 13: Manual test: verify alert fires at correct threshold
• Task 14: Audit trail test: verify events emitted for cost alerts
• Task 15: Rollback test: verify system works with COST_FORECAST_DISABLED=true

---

Generated by shipwright pipeline at 2026-04-17T00:32:20Z

[github-actions]

[github-actions]

📐 Architecture Decision Record

Click to expand ADR (200 lines)

Design: Pre-Flight Budget Forecast with In-Flight Cost Alerting

Context

Shipwright pipelines consume variable amounts of Claude API tokens (often $1-$50+ per run). Users currently have no visibility into expected cost before a pipeline starts, and no in-flight warning when a run is trending over budget. The cost module (scripts/sw-cost.sh, 1056 lines) already tracks actuals in ~/.shipwright/costs.json and enforces daily budgets via ~/.shipwright/budget.json, but budget checks fire only after spending occurs.

Constraints from the codebase:

• Bash 3.2 compatible — no associative arrays, no readarray. All logic lives in shell scripts (except the Bun/TypeScript dashboard).
• Atomic writes required — prior failures (failures.json) show race conditions on JSON state; must use tmp-file + mv.
• Graceful degradation — features must not block pipeline start (memory: "pipeline works without forecast.json").
• Intake stage integration point — scripts/lib/pipeline-stages-intake.sh already detects TASK_TYPE and suggests a template; forecast fits naturally after template selection.
• Loop-level cost tracking — scripts/sw-loop.sh already calls cost_record() per iteration; alert check belongs here.
• Historical cost store — ~/.shipwright/costs.json is capped at 1000 entries; sufficient for forecasting.

Decision

Introduce a standalone forecasting library (scripts/lib/cost-forecast.sh) that is consumed by existing modules via thin integration hooks. The library owns all forecast logic and state; intake, loop, and cost dashboard call into it but contain no forecast math.

Model choice: complexity-bucketed historical median (MVP). Group historical pipelines into 4 complexity buckets (low/medium/high/very_high), compute median total cost per bucket from the last 90 days. For a new pipeline, detect complexity from TASK_TYPE + goal keywords + issue labels, then emit {total_usd, confidence, breakdown}. Chosen over linear regression because (a) medians are robust to outliers already flagged in the cost store, (b) explainable to users, (c) works with as few as 3 samples per bucket. Linear regression is deferred as a v2 upgrade behind the same interface.

Data flow:

intake stage → forecast_compute(task_type, goal, labels, template)
            → writes ARTIFACTS_DIR/forecast.json (pipeline-scoped)
            → displays "Est. cost: $X.XX (±Y% confidence)"

loop iteration → forecast_check_threshold(issue, actual_cost_so_far)
              → compares against forecast.json
              → fires one-shot alert event at 80%

pipeline completion → forecast_finalize(issue, actual_total)
                   → appends to ~/.shipwright/optimization/forecast-accuracy.jsonl
                   → updates rolling MAPE/bias in forecast-accuracy.json

cost dashboard → forecast_accuracy_summary()
              → reads forecast-accuracy.json for display

Error handling: every forecast call returns success even on failure; missing/corrupt data yields {total_usd: 0, confidence: 0} and logs a warning. COST_FORECAST_DISABLED=true disables all integration points. Strict mode (--strict-budget) is the only path that can block pipeline start, and only when forecast confidence ≥ 0.5 AND forecast > remaining budget.

Alert deduplication: loop writes ARTIFACTS_DIR/.forecast-alert-fired sentinel on first trigger; subsequent iterations skip alert emission.

Component Diagram

┌─────────────────────────────────────────────────────────────┐
│                   pipeline-stages-intake.sh                 │
│                   (intake stage integration)                │
└──────────────────────────┬──────────────────────────────────┘
                           │ forecast_compute()
                           ▼
┌─────────────────────────────────────────────────────────────┐
│                  scripts/lib/cost-forecast.sh               │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  Public API                                           │  │
│  │  - forecast_compute()                                 │  │
│  │  - forecast_check_threshold()                         │  │
│  │  - forecast_finalize()                                │  │
│  │  - forecast_accuracy_summary()                        │  │
│  │  - forecast_detect_complexity()                       │  │

</details>

---
_Generated by `shipwright pipeline` design stage at 2026-04-17T00:35:30Z_

[github-actions]

[github-actions]

[github-actions]

🔨 Build started — shipwright loop with 20 max iterations, 1 agent(s), model: opus

[github-actions]

[github-actions]

❌ Pipeline failed at stage build after 14m 43s.

No log available