An active incident-commander agent that watches a live production incident, writes every alert, metric anomaly, and human action into a typed event log as it happens, and streams a complete postmortem the moment you click Resolve — generated from the log, never reconstructed from chat scrollback.
Built for the Harness Engineering Hack (June 2026).
- The Idea
- What Makes It Different
- Architecture
- The Incident State Machine
- Sponsor Highlights
- Measured Numbers (Claim Integrity)
- Repository Layout
- Prerequisites
- Environment Variables
- Recreate It From Scratch
- Running the Demo
- API Reference
- Testing & CI
- Deployment (Render)
- Design Principles
- Repo Map
Every incident tool — PagerDuty, FireHydrant, Incident.io — is passive. It waits for humans to update it, then reconstructs the postmortem afterward from unstructured Slack history: a journalist piecing together a story from tweets, two days late, with the causal chain already lost.
IncidentSherpa is active. A persistent agent watches the incident live and writes every event into a typed, append-only log the moment it happens — simultaneously to a real-time analytics store (ClickHouse) and a governance-grade audit trail (Guild). When the on-call engineer clicks Resolve, the postmortem doesn't get "written" — it gets emitted from the log the agent already kept.
If an action isn't in the log, it didn't happen — and the postmortem cannot mention it. That is the stenographer principle, and it is enforced in code.
| Incident.io / FireHydrant | IncidentSherpa | |
|---|---|---|
| Source of postmortem | Unstructured Slack scrollback, reconstructed later | Typed event log, captured live |
| Causal analysis | Human eyeballs a dashboard | ClickHouse LAG/LEAD window SQL finds the chain |
| Severity classification | Manual, or a slow frontier LLM | GLiNER2 (300M-param) in ~120ms, before any LLM |
| Runbook / owner suggestions | Static wiki links | Senso-cited, grounded, refuses uncited knowledge |
| Audit trail | The tool's own DB | Dual-sink: ClickHouse + Guild governance log |
| Safety on outbound text | None | GLiGuard screens every Slack/Jira message |
Three Render services from one render.yaml Blueprint, and nine load-bearing sponsor integrations. Remove any one and a demo beat breaks.
graph TD
subgraph Sources["📡 Monitoring / Alert Sources"]
TRIGGER["scripts/trigger.py<br/>or any webhook POST"]
LOADGEN["scripts/load_generator.py<br/>continuous live metrics"]
end
subgraph Render["☁️ Render — one render.yaml Blueprint (3 services + cron)"]
API["🔵 webhook-api (FastAPI / web)<br/>POST /trigger · POST /resolve<br/>POST /confirm-owner<br/>GET /events (SSE) · GET /health<br/>Bearer auth · per-IP rate limit · idempotency"]
WORKER["⚙️ agent-worker (background)<br/>IncidentAgent state machine"]
FE["🟣 frontend (Next.js + OpenUI / web)<br/>timeline · causal graph · stepper<br/>streaming postmortem panel"]
CRON["⏰ health-ping (cron, every 5m)<br/>warm-start mitigation"]
end
subgraph Agent["🤖 IncidentAgent — INVESTIGATING → MITIGATING → RESOLVED"]
EMIT["every action = ONE typed event"]
end
subgraph Sinks["📝 Dual-Sink Event Log (the product)"]
CH[("🟡 ClickHouse `events`<br/>real-time + causal SQL")]
GUILD["⬛ Guild session<br/>append-only audit trail"]
end
subgraph Sponsors["🧩 Sponsor Integrations (each load-bearing)"]
PIONEER["🟪 Pioneer · GLiNER2<br/>severity + blast radius<br/>~120ms, before any LLM"]
GUARD["🟪 Pioneer · GLiGuard<br/>screens outbound text"]
SENSO["🔵 Senso<br/>cited runbooks + owners"]
AIRBYTE["🟦 Airbyte Context Store<br/>related tickets/PRs + history"]
COMPOSIO["🔷 Composio<br/>Slack + Jira via managed OAuth"]
CLAUDE["🟧 Claude Fable 5<br/>postmortem drafting ONLY"]
LF["⬛ Langfuse<br/>traces EVERY LLM/tool/SQL call"]
end
TRIGGER --> API
LOADGEN --> CH
API <-->|SSE typed events + postmortem tokens| FE
API --> WORKER
WORKER --> EMIT
EMIT --> CH
EMIT --> GUILD
EMIT --> PIONEER
EMIT --> SENSO
EMIT --> AIRBYTE
EMIT --> COMPOSIO
EMIT -.->|on Resolve, reads full log| CLAUDE
CLAUDE --> GUARD
GUARD -->|screened tokens| API
PIONEER -.-> LF
SENSO -.-> LF
CLAUDE -.-> LF
CH -.-> LF
Flow in one breath: an alert hits POST /trigger → GLiNER2 extracts severity/blast-radius (small model, first) → ClickHouse runs the causal LAG/LEAD query → Senso returns a cited runbook + owner → every step is written to both ClickHouse and a Guild audit session and streamed to the UI over SSE → Composio posts a GLiGuard-screened Slack update + Jira ticket → on Resolve, Claude Fable 5 drafts a postmortem strictly from the event log, GLiGuard screens it, and it streams token-by-token to the panel. Langfuse traces all of it.
The core domain model (apps/worker/agent.py). Every transition and action emits a typed event to both sinks.
stateDiagram-v2
[*] --> INVESTIGATING: POST /trigger (alert ingested)
INVESTIGATING --> MITIGATING: runbook step selected
MITIGATING --> RESOLVED: POST /resolve (human)
RESOLVED --> [*]: postmortem emitted from the log
note right of INVESTIGATING
1. GLiNER2 severity + blast radius (before any LLM)
2. ClickHouse causal LAG/LEAD query
3. Senso cited runbook + ownership
4. Airbyte Context Store (related tickets/PRs)
end note
note right of MITIGATING
Composio: GLiGuard-screened Slack update
+ Jira follow-up ("Suggested owner — awaiting confirmation")
end note
note right of RESOLVED
Claude Fable 5 drafts from the FULL event log
GLiGuard screens → stream token-by-token over SSE
end note
| State | Entered when | What the agent does | Sponsors |
|---|---|---|---|
| INVESTIGATING | Alert POST hits the webhook | GLiNER2 severity + blast radius (before any LLM) → ClickHouse causal SQL → Senso cited runbook + owner → Airbyte Context Store query | Pioneer, ClickHouse, Senso, Airbyte |
| MITIGATING | Runbook step selected | Composio posts a GLiGuard-screened Slack update + Jira ticket; owner is suggested, never auto-assigned | Composio, Pioneer (GLiGuard), Guild |
| RESOLVED | Human clicks Resolve | Claude Fable 5 drafts the postmortem from the log; GLiGuard screens; streams token-by-token | Claude, OpenUI, Langfuse |
Illegal transitions raise; the dual-sink writes to ClickHouse and Guild on every event. If a single sink degrades, an explicit DEGRADED event is logged and the agent continues — only the loss of both sinks is fatal, because the event log is the product.
Every integration is live and load-bearing — verified by real API calls, no mocks. Latencies are measured (see Measured Numbers).
The real-time event store and the causal engine. The signature query uses LAG/LEAD window functions with rolling z-score onset detection to find that payments-db-primary exhaustion preceded the payments-service latency breach by 135 seconds — a chain no human computes under pressure. Runs in ~260ms on the live cluster. metrics, events, and airbyte_history tables; replay + a continuous load generator keep real data flowing.
Traces every LLM call, sponsor API call, and ClickHouse query via a @traced decorator (libs/tracing.py). Runs on ClickHouse. Nothing in the pipeline is un-observable — a call without a span is treated as a bug.
Two distinct small-model roles (never swapped — this distinction is enforced in code and tests):
- GLiNER2 — schema-conditioned extraction. Classifies severity (
P0–P3) and extracts affected-service spans in ~120–180ms, before any frontier LLM is touched. The "small models first" economics. - GLiGuard — safety moderation. Screens every outbound string (Slack, Jira, postmortem) at the single choke point — it is structurally impossible to send unscreened text.
The governance-grade second sink. One Guild session per incident; every typed event is appended to an append-only audit trail via the control-plane API (app.guild.ai/api, authenticated by guild auth token). Verified live: a real session with 50+ audit entries. This is what makes "the stenographer kept the notes" literally true.
The cited knowledge base. Runbooks, past postmortems, and an ownership map are seeded into the Senso KB (two-step S3 upload, apiv2.senso.ai/api/v1). The agent retrieves grounded, cited answers via /org/search — and refuses uncited knowledge (UncitedResponseError). No hallucinated runbooks, ever.
The Context Store. The airbyte-agent-sdk connects GitHub + Jira connectors so the agent can semantically query related tickets/PRs during the incident and pull 90-day history into ClickHouse for the ownership baseline ("N of last M incidents on this service resolved by X").
Managed OAuth for real action. Posts a structured Slack incident update and creates a Jira follow-up ticket through the single GLiGuard-screened choke point. Idempotent per incident+state — replaying a webhook never double-posts. Owner wording is always "Suggested owner — awaiting confirmation," never "assigned."
The one frontier-LLM call in the whole pipeline — postmortem drafting only. Prompted strictly from the verbatim event log with hard claim-integrity rules. Concise (~350 words) so the full Resolve→stream completes in ~22s, under the 30s demo gate.
The generative UI: a scrolling typed-event timeline, a Guild state stepper, a causal dependency graph with a "precedes by Xm Ys" edge and a popover showing the real SQL, a suggested-owner confirm button, and the streaming postmortem panel. Latency badges render measured values only — never a fabricated number.
One render.yaml Blueprint deploys all three services (webhook-api, agent-worker, frontend) plus a health-ping cron for warm-start mitigation.
Every number that appears in the UI or the demo is measured, with the command that produced it. No estimates.
| Metric | Value | How measured |
|---|---|---|
| GLiNER2 severity extraction (server inference) | ~120–180 ms | extract_severity() live, result.data.latency_ms |
| GLiGuard outbound screen | ~426 ms | live screen() call |
| ClickHouse causal-chain query (960-row window) | ~263 ms | find_causal_chains(window_minutes=20) |
| Causal lag: DB → payments (the demo number) | 135 s (2m15s) | live lagInFrame onset pairing |
| Causal lag: payments → checkout (cascade) | 55 s | same query |
| Senso cited runbook retrieval | ~3.7 s | live get_runbook() |
| Claude Fable 5 postmortem (concise ~350w) | ~21 s | stream_anthropic_completion() |
| Full Resolve → postmortem streamed | ~22 s (≤30 s gate ✅) | generate_postmortem() end-to-end |
ClickHouse SELECT 1 round-trip (cold) |
~2.1 s | clickhouse-connect |
| Full ingest (dual-sink incl. Guild) | ~25 s | IncidentAgent.ingest_alert() |
⚠️ Claim-integrity note: GLiNER2 honestly classifies the demo incident as P3 (not a fabricated "P1"), and the demo causal number is 2m15s (the detected onset-to-onset lag), not the 4m10s CSV climb-to-breach. The UI shows what the models actually return.
harnes/
├── render.yaml # 3-service Render Blueprint + health cron
├── .github/workflows/ci.yml # ruff + pytest, and Next.js build, on push
├── pyproject.toml # ruff + pytest config (live tests marked & excluded)
│
├── apps/
│ ├── api/main.py # FastAPI: /trigger /resolve /confirm-owner /events(SSE) /health
│ │ # Bearer auth, per-IP rate limit, idempotency, EventBus
│ ├── worker/
│ │ ├── agent.py # IncidentAgent: state machine, TypedEvent, dual-sink emit
│ │ └── postmortem.py # stenographer prompt → Claude → GLiGuard → SSE stream
│ └── frontend/ # Next.js 16 + OpenUI
│ └── src/
│ ├── app/page.tsx
│ ├── components/ # timeline, causal-graph, guild-stepper, postmortem-panel,
│ │ # owner-confirm, latency-badges, fallback-postmortem
│ └── hooks/use-event-stream.ts # SSE w/ exponential-backoff reconnect
│
├── libs/
│ ├── tracing.py # Langfuse @traced decorator — import everywhere
│ ├── resilience.py # with_retries + circuit breaker → DegradedError
│ ├── errors.py # NotConfiguredError, UncitedResponseError, ...
│ ├── logging_config.py # structured JSON logging
│ ├── clickhouse/
│ │ ├── schema.py # metrics, events, airbyte_history DDL
│ │ └── causal.py # LAG/LEAD z-score onset causal SQL
│ ├── pioneer/
│ │ ├── gliner2.py # severity + affected-services extraction
│ │ └── gliguard.py # outbound-text safety screen
│ ├── senso/retrieve.py # cited runbook/ownership search (refuses uncited)
│ ├── guild/session.py # create session / append audit event / read trail
│ ├── airbyte/__init__.py # Context Store client
│ └── composio_actions/send.py # single GLiGuard-screened Slack/Jira choke point
│
├── scripts/
│ ├── replay.py # recorded incident CSV → ClickHouse at N×
│ ├── load_generator.py # continuous metrics + --inject db_pool_exhaustion
│ ├── make_incident_csv.py # reproducible incident dataset generator
│ ├── trigger.py # fire the demo alert at the webhook
│ ├── seed_senso.py # seed runbooks/postmortems/ownership (+ --verify)
│ ├── seed_jira_history.py # seed Jira incident history
│ ├── seed_slack_history.py # seed Slack history
│ ├── composio_link.py # Composio Slack/Jira OAuth link helper
│ └── demo_preflight.py # check every credential before a demo
│
├── demo_assets/
│ ├── incident_metrics.csv # 960-row recorded incident (pool exhaustion → breach)
│ └── incident_payload.json # the demo alert
│
└── docs/
├── ARCHITECTURE.md # system architecture for judges
├── SECURITY-AUDIT.md # dependency + secrets audit
└── NO-MOCK-AUDIT.md # adversarial no-mock sweep
Project rules & the full build ledger live in
CLAUDE.mdandBUILD-STATE.md. The war-room decision record is infinal-plan.md,demo-scripts.md,ideas.md,debate-log.md, andsponsors.md.
| Tool | Version | Notes |
|---|---|---|
| Python | 3.12+ | one venv at repo root for api + worker + libs |
| Node.js | 20+ | for the Next.js frontend |
guild CLI |
latest | npm i -g @guildai/cli — needed for Guild auth |
render CLI |
latest | optional, for deploy |
| Git | any |
Sponsor accounts (all have free tiers): ClickHouse Cloud, Langfuse Cloud, Pioneer (Fastino), Guild.ai, Senso.ai, Airbyte Cloud, Composio, Anthropic, Render.
Copy .env.example → .env (gitignored — never commit it) and fill what you have. Empty vars make each integration raise an honest NotConfiguredError (visible as SKIPPED_NOT_CONFIGURED events and /health "blocked" entries) — no mocks, no fake "ok."
# ── ClickHouse Cloud ──────────────────────────────────────────────
CLICKHOUSE_HOST=<your-cluster>.clickhouse.cloud
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=...
# ── Langfuse (runs on ClickHouse) ─────────────────────────────────
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_HOST=https://us.cloud.langfuse.com
# ── Pioneer / Fastino (GLiNER2 + GLiGuard) ────────────────────────
PIONEER_API_KEY=pio_sk_...
# ── Guild.ai (control-plane audit log) ────────────────────────────
GUILD_WORKSPACE=<workspace-uuid> # from `guild workspace list`
GUILD_TOKEN= # optional; else `guild auth token` is used
# (legacy, unused by the live client) GUILD_PAT / GUILD_API_BASE
# ── Senso.ai (cited knowledge base) ───────────────────────────────
SENSO_API_KEY=tgr_... # dashboard API key
# SENSO_BASE_URL defaults to https://apiv2.senso.ai/api/v1
# ── Airbyte Agent Engine ──────────────────────────────────────────
AIRBYTE_CLIENT_ID=...
AIRBYTE_CLIENT_SECRET=...
# ── Composio (Slack + Jira) ───────────────────────────────────────
COMPOSIO_API_KEY=ak_...
COMPOSIO_USER_ID=...
SLACK_INCIDENT_CHANNEL=#incidents
JIRA_PROJECT_KEY=INC
# ── Anthropic (postmortem drafting only) ──────────────────────────
ANTHROPIC_API_KEY=sk-ant-...
# ── Frontend + hardening ──────────────────────────────────────────
NEXT_PUBLIC_API_BASE=http://localhost:8000 # goes in apps/frontend/.env.local
WEBHOOK_AUTH_TOKEN= # set in prod → Bearer required on POST
RATE_LIMIT_PER_MINUTE=60 # per-IP token bucketgit clone https://github.com/nihalnihalani/harnesshack.git && cd harnesshack
python3 -m venv .venv && source .venv/bin/activate
pip install -r apps/api/requirements.txt -r apps/worker/requirements.txt -r requirements-dev.txt
cd apps/frontend && npm install && cd ../..
cp .env.example .env # then fill in credentialsguild auth login # browser OAuth → enables `guild auth token`
guild workspace list # copy a workspace UUID → GUILD_WORKSPACE in .env
python scripts/composio_link.py # Slack + Jira OAuth via Composio
render login # for deployEach command hits the real service and prints a measured result. Run after filling .env:
set -a && source .env && set +a
# ClickHouse — SELECT 1 + create tables
python -c "from libs.clickhouse import get_client; print(get_client().query('SELECT version()').result_rows)"
python -c "from libs.clickhouse.schema import ALL_TABLE_DDL; from libs.clickhouse import get_client; c=get_client(); [c.command(d) for d in ALL_TABLE_DDL.values()]; print('tables ready')"
# Langfuse — emit a real span
python -c "from libs.tracing import traced; traced('verify')(lambda: 'ok')(); print('span sent')"
# Pioneer — GLiNER2 severity + GLiGuard screen
python -c "from libs.pioneer.gliner2 import extract_severity; print(extract_severity('payments-service p99_ms breached 2466ms'))"
# Senso — seed the KB, then a cited query
python scripts/seed_senso.py --verify
python -c "from libs.senso.retrieve import get_runbook; print(get_runbook('payments p99 latency breach').citation)"
# Guild — create a session + read the audit trail
python -c "from libs.guild.session import create_session, read_audit_events; sid=create_session('inc-verify'); print(sid, len(read_audit_events(sid)))"Or run the all-in-one preflight:
python scripts/demo_preflight.py # checks every credential, prints PASS/blocked per dependencypython scripts/replay.py --truncate-first --speed 100 # 960 real rows → real cluster
# optional: keep data flowing continuously
python scripts/load_generator.py --inject db_pool_exhaustionIn three terminals (all with set -a && source .env && set +a):
# 1) API + SSE
uvicorn apps.api.main:app --reload --port 8000
# 2) Frontend (timeline on http://localhost:3000)
cd apps/frontend && npm run dev
# 3) Fire the incident, then resolve it
python scripts/trigger.py --payload demo_assets/incident_payload.json
# → watch the timeline populate live: GLiNER2 severity, the causal edge,
# the cited runbook, the suggested owner, the Slack/Jira actions
# → click "Resolve" in the UI (or: curl -X POST localhost:8000/incidents/<id>/resolve)
# → the postmortem streams token-by-token in ~22sThe wow moment: open the timeline already mid-incident, click Resolve, and a complete postmortem — root cause, the 135s causal chain, the suggested owner, action items — streams onto the screen in ~20 seconds, written entirely from the log the agent kept.
apps/api/main.py (FastAPI). When WEBHOOK_AUTH_TOKEN is set, POST routes require Authorization: Bearer <token>; a per-IP token bucket (RATE_LIMIT_PER_MINUTE) returns 429 with Retry-After.
| Method | Path | Purpose |
|---|---|---|
POST |
/trigger |
Ingest an alert (pydantic-validated). Idempotent via Idempotency-Key header or payload hash — replays return {"duplicate": true}. Spawns the agent pipeline. |
POST |
/incidents/{id}/resolve |
Transition to RESOLVED and stream the postmortem over SSE. |
POST |
/incidents/{id}/confirm-owner |
Confirm the suggested owner (emits owner_confirmed). |
GET |
/events |
SSE stream (text/event-stream, no-buffering headers) of typed events + postmortem tokens. |
GET |
/fallback/postmortem |
The cached F2 fallback artifact (404 until a real run produced it). |
GET |
/health |
Per-dependency status — each sponsor reported configured/blocked from real env presence (never a fake "ok"). |
Alert payload (demo_assets/incident_payload.json):
{ "service": "payments-service", "metric": "p99_ms", "value": 2466.1,
"timestamp": "2026-06-12T14:15:00Z", "incident_id": "inc-2026-0612-payments-p99" }ruff check . # lint — clean
pytest # 229 in-process tests against the real app code
# (1 live-marked test hits real ClickHouse; excluded by default)
pytest -m live # run the live-cluster test (needs creds)
cd apps/frontend && npm run build # Next.js production build — cleanTests use dependency-injection of the real pipeline (FastAPI TestClient, in-process fakes) — that's test isolation of real code, not runtime mocks. GitHub Actions (.github/workflows/ci.yml) runs ruff + pytest and the frontend build on every push. Audits: docs/SECURITY-AUDIT.md, docs/NO-MOCK-AUDIT.md.
render login
render blueprint launch # deploys webhook-api + agent-worker + frontend + health-ping cron
render env # set every credential via the dashboard/CLI — NEVER commit
curl https://<app>.onrender.com/health # expect all-dependencies-OK JSONRender notes: a server cannot run guild auth login, so set GUILD_TOKEN in the Render environment (a token captured from guild auth token). SSE works through Render's proxy with the correct no-buffering headers (already set in apps/api/main.py).
- The event log is the product. Every action becomes a typed event in ClickHouse and the Guild audit log. The postmortem is emitted from that log, never reconstructed.
- Small models first. GLiNER2 classifies before Claude reasons; GLiGuard screens before anything sends. The frontier LLM is the exception, not the path.
- Langfuse everything. A call without a trace is a bug.
- No mocks, ever. Missing credentials raise honest
NotConfiguredError(visibleSKIPPED_NOT_CONFIGUREDevents). Where demo variables are controlled (a replay CSV), it's disclosed on screen. - Claim integrity. Every on-screen number is measured. GLiNER2 = extraction; GLiGuard = moderation — never swapped.
- Degrade, don't die. A single sponsor failing logs an explicit
DEGRADEDevent and the agent continues. Only losing both event sinks is fatal.
| File | What it is |
|---|---|
CLAUDE.md |
Project law — architecture, claim-integrity rules, build commands |
BUILD-STATE.md |
Build ledger — phase status, credential blockers, measured numbers |
final-plan.md |
War-room output: build plan, prize mapping, risks |
demo-scripts.md |
Beat-by-beat 3-minute demo script + fallbacks |
ideas.md · debate-log.md · sponsors.md |
The war-room decision record |
docs/ARCHITECTURE.md |
System architecture for judges |
docs/SECURITY-AUDIT.md · docs/NO-MOCK-AUDIT.md |
Dependency/secrets + no-mock audits |
IncidentSherpa — the incident commander that writes the postmortem itself, because it was in the room.
Built with ClickHouse · Langfuse · Pioneer · Guild · Senso · Airbyte · Composio · OpenUI · Anthropic · Render