Pipeline Health Monitor with Anomaly Detection & Auto-Alert System

[github-actions]

Strategic Improvement

Build real-time health monitoring that tracks success rate, duration, and cost metrics across all pipelines. Detects anomalies (sudden drop to 0%, 3x duration spikes, cost overruns) and alerts via tmux notifications, dashboard banners, and file markers. The current 0% success rate should have triggered an immediate alert—this system ensures we never go blind again.

Acceptance Criteria

• Tracks success rate, avg duration, and cost per pipeline in rolling 7-day window
• Detects anomalies: success rate drop >20%, duration spike >2x baseline, cost spike >2x baseline
• Alerts via: tmux notification, dashboard banner, ~/.shipwright/health-alert.json marker
• Triggers automatic diagnostic scan on severe degradation (success rate <50%)
• Integrates with existing events.jsonl and DORA metrics
• Test suite validates anomaly detection and alerting logic

Context

• Priority: P0
• Complexity: standard
• Generated by: Strategic Intelligence Agent
• Strategy alignment: P0: Reliability & Success Rate

[github-actions]

⚓ Shipwright Pipeline Starting

Field	Value
Template	standard
Complexity	standard
Max Iterations	20
Triage	P0 reliability feature—adds real-time health monitoring with anomaly detection to prevent pipeline blindness, integrating cleanly with existing events and DORA systems.
Runner	ubuntu-latest
Risk Level	unknown
Workflow	30093737594

The autonomous pipeline is processing this issue. Updates will be posted as stages complete.

[github-actions]

[github-actions]

[github-actions]

📋 Implementation Plan

Click to expand full plan (578 lines)

Implementation Plan: Pipeline Health Monitor with Anomaly Detection & Auto-Alert System

Requirements Analysis

Minimum Viable Change: A background service that reads pipeline execution metrics (success rate, duration, cost), establishes 7-day baselines, detects anomalies against thresholds, and alerts through multiple channels.

Implicit Requirements Identified:

• Must run continuously in daemon without blocking other tasks
• Must handle missing/incomplete event data gracefully
• Must be testable in isolation (unit + integration)
• Must not require external dependencies beyond existing stack
• Alerts should be non-intrusive (async, don't fail daemon on alert failure)

Acceptance Criteria (from issue + clarification):

• Health metrics (success rate, avg duration, avg cost) calculated for rolling 7-day window per pipeline
• Anomaly detection triggers on: success drop >20%, duration >2x baseline, cost >2x baseline
• Three alert channels working: tmux popup, dashboard banner, ~/.shipwright/health-alert.json
• Automatic diagnostic scan triggered when success rate < 50%
• Health monitor integrated into daemon main loop (runs every N iterations)
• Test suite validates metric calculation, anomaly logic, and alerting

---

Alternatives Considered

Option 1: Centralized Health Monitor Script (CHOSEN)

• Single dedicated script reads events.json, calculates metrics, detects anomalies, sends alerts
• Integrated into daemon main loop as periodic check
• Architecture: sw-health-monitor.sh + integration into sw-daemon.sh

Pros:

• Modular, single responsibility, testable in isolation
• Reusable from daemon, pipeline stages, or CLI
• Alerts are fire-and-forget (don't block pipeline execution)

Cons:

• One more script to maintain
• Must parse events.jsonl efficiently (larger files with time)

Blast Radius: Low — adds new script, minimal changes to daemon loop (single function call)

Option 2: Distributed Per-Stage Detection

• Each stage calculates its own health during execution
• Aggregates at daemon level during heartbeat pulse

Pros: Real-time per-stage granularity, tighter feedback loop

Cons:

• Duplicated logic across stages
• Harder to coordinate alerts (which stage fails first?)

✅ Task Checklist

• Health metrics (success rate, avg duration, avg cost) calculated for rolling 7-day window per pipeline
• Anomaly detection triggers on: success drop >20%, duration >2x baseline, cost >2x baseline
• Three alert channels working: tmux popup, dashboard banner, ~/.shipwright/health-alert.json
• Automatic diagnostic scan triggered when success rate < 50%
• Health monitor integrated into daemon main loop (runs every N iterations)
• Test suite validates metric calculation, anomaly logic, and alerting
• sw-health-monitor.sh implemented, passes all unit tests
• _read_events(), _calculate_baseline(), _detect_anomaly() functions verified
• Alert channels (tmux, file, dashboard) working end-to-end
• Auto-diagnostic triggered correctly on CRITICAL severity
• Integrated into daemon main loop without blocking
• Test suite (sw-health-monitor-test.sh) passes all 35+ tests
• Configuration schema added to daemon-config.json
• Documentation updated in CLAUDE.md (AUTO sections synced)
• Alert file format stable and documented
• Tested with real events.jsonl data from 7-day production run
• Dashboard shows health status and active alerts
• No breaking changes to existing daemon functionality
• VERSION variable in sw-health-monitor.sh matches package.json
• Task 1: Create sw-health-monitor.sh with _read_events(), _calculate_baseline(), _detect_anomaly(), _format_alert()

---

Generated by shipwright pipeline at 2026-07-24T12:43:54Z

[github-actions]

[github-actions]

📐 Architecture Decision Record

Click to expand ADR (141 lines)

I have enough grounding. My codebase check surfaced three facts that materially change the design, which I'll fold into the ADR:

1. The real completion event type is pipeline.completed (dotted), not pipeline_complete as the plan assumes — sw-replay.sh, sw-otel.sh, daemon-poll-health.sh all key off it.
2. The event schema is {ts, type, issue, result, duration_s, input_tokens, output_tokens, iterations} — there is no pipeline name field (keyed by issue), no status/timestamp fields, and no cost_usd (cost must be derived from tokens). emit_event in bootstrap.sh also serializes all values as strings.
3. scripts/lib/daemon-poll-health.sh already implements daemon_check_degradation() — rolling-window success-rate/CFR alerting with thresholds — plus events-file-size checks. The plan's new script substantially duplicates it.

---

Design: Pipeline Health Monitor with Anomaly Detection & Auto-Alert System

Context

Issue #808 requires a background monitor that establishes rolling baselines for pipeline health (success rate, duration, cost), detects anomalies, and alerts through multiple channels so the team "never goes blind" to a sustained failure (the motivating incident: a 0% success rate that should have triggered an immediate alert).

Constraints imposed by the existing codebase (verified, not assumed):

• Event source is fixed. Completion events are pipeline.completed written one-JSON-per-line to ${HOME}/.shipwright/events.jsonl (daemon reads via EVENTS_FILE="${DAEMON_DIR}/events.jsonl}", same path). Schema: {ts, type:"pipeline.completed", issue, result:"success"|"failure", duration_s, input_tokens, output_tokens, iterations, self_heal_count}. There is no pipeline name, no status/timestamp, and no cost_usd field — the plan's schema is wrong on all four.
• Existing health infra overlaps. scripts/lib/daemon-poll-health.sh:daemon_check_degradation() already computes rolling success/CFR percentages against DEGRADATION_SUCCESS_THRESHOLD (default 50) and DEGRADATION_CFR_THRESHOLD (default 30) and emits daemon.health. A parallel sw-health-monitor.sh that recomputes the same metrics is duplicated logic (an anti-pattern flagged below).
• Bash 3.2 + set -euo pipefail — no assoc arrays, atomic writes via tmp+mv, grep -c | ... || true pitfalls, jq --arg for JSON. emit_event stringifies values, so numeric jq ops need tonumber? guards.
• Offline/local mode must not break (NO_GITHUB), and alert failures must never crash the daemon.

Decision

Build a single-responsibility engine scripts/sw-health-monitor.sh that detects anomalies, decoupled from an alert dispatcher that notifies — but scope it as an extension of the existing daemon health layer, not a parallel silo. Concretely:

Data model — baseline keyed by repo, not "pipeline name." Since events carry no pipeline name and this is single-repo-per-daemon, baselines are computed per-repo (one aggregate) with optional grouping by issue. Cost is derived from tokens via the existing cost model (input_tokens/output_tokens), not read from a nonexistent field.

Component boundaries (each has one reason to change):

        ┌────────────────────────────────────────────────┐
        │              sw-daemon.sh (main loop)            │
        │   after poll_loop cycle → hm_check() [gated]     │
        └───────────────────────┬──────────────────────────┘
                                 │ source + call (non-fatal)
        ┌───────────────────────▼──────────────────────────┐
        │           sw-health-monitor.sh (ENGINE)           │
        │  _read_events → _calculate_baseline →             │
        │  _detect_anomaly → _format_alert  (pure, no I/O   │
        │                     side effects except read)     │
        └───────┬───────────────────────────────┬──────────┘
                │ anomaly JSON                    │ CRITICAL
        ┌───────▼──────────┐            ┌─────────▼──────────┐
        │ _send_alerts     │            │ _trigger_diagnostic│
        │  ├ _alert_file   │(primary)   │  → sw-doctor subset│
        │  ├ _alert_tmux   │(best-effort)│  (background)      │
        │  └ _alert_dash   │(file→server)└────────────────────┘
        └───────┬──────────┘
                │ writes
   ~/.shipwright/health-alert.json  +  emit_event "health.alert"
                                         └→ events.jsonl = source of truth
                                            └→ dashboard/server.ts /api/health

Data flow: daemon cycle → hm_check() reads last-N lines of events.jsonl (bounded tail, never full-file load) → filter type=="pipeline.completed" in trailing baseline_days window → compute baseline (guard count>=2) → compare current window vs baseline → on anomaly, _send_alerts() fans out (file first, then best-effort tmux/dashboard) and always emit_event "health.alert"; on CRITICAL, fire _trigger_diagnostic() in background.

Error handling: engine returns structured JSON; every alert channel is individually wrapped so one failure logs (emit_event "health.alert_channel" "channel=… status=failed") and continues. The file marker + event log are the guaranteed channels; tmux is explicitly best-effort (the motivating failure mode). hm_check is called as hm_check || warn "…non-fatal" so it can never abort the daemon.

Reuse over duplication: the engine's rolling-success calculation is extracted so daemon_check_degradation() and the health monitor call one shared function (in sw-health-monitor.sh, sourced by the daemon), rather than two thresholds drifting apart.

---

Generated by shipwright pipeline design stage at 2026-07-24T12:47:49Z

[github-actions]

[github-actions]

[github-actions]

🔨 Build started — shipwright loop with 20 max iterations, 1 agent(s), model: opus

[github-actions]

[github-actions]