id	supportops-readme
type	artifact
version	1.2
created	2026-07-10
updated	2026-07-10
status	complete
links	CLAUDE guardrails-runtime changelog
summary	SupportOps-AI case study — all five phases (P0-P5) complete. AI-governed support triage pipeline with audit trail, HITL queue, hallucination-gated auto-resolution, RCA reporting, and KPI dashboard. Full run cost $0.75.

SupportOps-AI — An AI-Governed Support Operations Pipeline

A working demonstration of how a modern support organization triages tickets with AI under governance — confidence thresholds, human-in-the-loop escalation, a hallucination gate on customer-facing answers, a complete audit trail, and cost telemetry. Built by Jacob Alvarez Del Castillo the way a Director of Technical Support would design it: the interesting part isn't that AI classifies tickets — it's everything wrapped around the AI so you can trust it with customers.

Every number in this README comes from a real end-to-end run (300 synthetic tickets, July 2026). Total API cost: $0.75 — $2.51 per 1,000 tickets.

▶ Explore the live dashboard — the actual artifact from this run (hover the charts; every chart has a table view; follows your light/dark preference).

Architecture

flowchart LR
    G["synthetic ticket<br/>generator"] --> I["1 · Ingest"]
    I --> T["2 · AI Triage<br/>claude-haiku-4-5<br/>category · severity ·<br/>sentiment · confidence"]
    T --> R{"3 · Governed Routing<br/>config/governance.yaml<br/>(deterministic — no AI<br/>in the decision layer)"}
    R -->|"conf ≥ 0.90 AND how-to"| K["KB answer<br/>claude-sonnet-4-6"]
    K -->|"citation exists"| A["auto-resolved"]
    K -->|"uncited → withheld"| H
    R -->|"conf < 0.60 OR S1/S2 OR<br/>enterprise + negative"| H["human-review queue<br/>+ review CLI"]
    R -->|"otherwise"| Q["agent queue"]
    T & A & Q & H --> RCA["4 · RCA Summarizer<br/>deterministic clustering +<br/>one sonnet call"]
    RCA --> REP["systemic-issues<br/>report .md"]
    T & R & K & RCA -.every decision.-> AUD[("audit_log.jsonl")]
    AUD & A & Q & H & REP --> D["5 · KPI Dashboard<br/>single-file HTML"]

Loading

Model routing is a cost feature, not an implementation detail: high-volume classification runs on the cheapest model, and the mid-tier model is spent only where output quality is customer- or executive-facing. No Opus at runtime.

Stage	Model	Calls	Cost (this run)
Triage	claude-haiku-4-5	300	$0.36
KB answers	claude-sonnet-4-6	64	$0.36
RCA report	claude-sonnet-4-6	1	$0.04
Routing, clustering, dashboard	none (deterministic code)	0	$0.00

Results (measured, single run, n=300)

Metric	Value	Notes
Deflection rate	21.3%	64 tickets auto-resolved, all 64 with a verified KB citation
Escalation rate	63.3%	187 of 190 triggered by S1/S2 severity — see honest limits
Triage accuracy — category	95.7%	vs generator ground truth the model never saw
Triage accuracy — severity	43.0% exact, 90.3% ±1 level	model systematically over-assigns S2
Triage accuracy — sentiment	88.0%
Audit coverage	100%	364 lines, schema enforced from config
Cost	$2.51 / 1,000 tickets	measured tokens × published pricing

The governance layer (what makes this different)

Thresholds are config, not code. Every routing rule lives in config/governance.yaml. Changing policy is a reviewable config diff:

auto_resolve:
  min_confidence: 0.90
  allowed_categories: [how-to]
  require_kb_citation: true      # uncited answer -> human queue, never sent
escalate_to_human:
  max_confidence: 0.60
  always_escalate_severities: [S1, S2]
  tier_sentiment_rules:
    - customer_tier: enterprise
      sentiments: [negative]

Every AI decision writes an audit line — input hash, model, confidence, route, reasons, tokens, timestamp. The schema is enforced from config: a write missing a required field raises instead of logging a partial line.

{"ticket_id": "TCK-10003", "input_hash": "6484a38f29f8…", "stage": "routing",
 "model": "claude-haiku-4-5", "confidence": 0.92, "route": "human_review",
 "reasons": ["severity S2 always escalates"], "tokens_in": 809,
 "tokens_out": 79, "timestamp": "2026-07-10T15:16:26.634710+00:00"}

The hallucination gate. An auto-resolve candidate only ships if the drafted answer cites a KB article id that exists in the local index — verified in code, not trusted from the model. The model may also decline (kb_article_id: null) when no article applies; either way the ticket goes to a human with the draft attached. This run: 64 attempted, 64 cited, 0 withheld — on a KB purpose-built for the corpus (see honest limits).

The human queue is a product, not an afterthought. Escalations land in a reviewable queue with the triggering reasons attached, and scripts/review_queue.py records every human decision (action, note, reviewer, timestamp) to its own log — the human trail is as auditable as the AI one.

python scripts/review_queue.py list
python scripts/review_queue.py show TCK-10105
python scripts/review_queue.py decide TCK-10105 --action approve --note "called client"

The support→product bridge. Tickets are clustered deterministically (Jaccard similarity on normalized subjects — zero API cost), then one Sonnet call writes the top-5 systemic issues report with root-cause hypotheses, per-issue owners, and evidence restricted to real cluster counts. In this run the model correctly connected three separate outage clusters to a shared maintenance-window trigger.

The parent project's 14 conversational guardrails are mapped to enforced runtime behavior in guardrails_runtime.md.

Quickstart

pip install -r requirements.txt
echo ANTHROPIC_API_KEY=sk-ant-... > .env

python scripts/generate_tickets.py --count 300 --seed 42   # stdlib-only, deterministic
python run_pipeline.py --tickets data/synthetic.json       # full pipeline, ~$0.75
python scripts/evaluate_triage.py                          # accuracy vs ground truth
python scripts/review_queue.py list                        # work the human queue

Outputs land in outputs/: the audit log, three route queues, the RCA report, the cost report, and dashboard.html (open in a browser; charts load chart.js from a CDN). A budget guard warns at 80% of the $5-per-run cap set in governance.yaml.

Honest limits

• Synthetic data flatters the pipeline. Tickets come from templates, so clustering and KB retrieval are far easier than production text; the KB was written to cover the corpus, so the 100% citation rate is a ceiling, not a forecast. The generator's subject/body mismatches (deliberate noise) are the main realism check.
• Confidence is poorly calibrated — measured, not assumed. Mean self-reported confidence was 0.914 when the category was right and 0.903 when it was wrong. That's why no rule here trusts confidence alone: auto-resolve also requires the category gate and a verified citation. The <0.60 escalation rule never fired (minimum observed: 0.72).
• Severity skews high. 43% exact agreement; the model over-assigns S2, and since S1/S2 always escalate, the 63% escalation rate is a direct consequence. Right failure direction for a support desk, wrong operating point — the fix is threshold/prompt tuning against labeled data, and the audit trail is exactly the dataset you'd tune from.
• Citation existence ≠ citation faithfulness. The gate proves the cited article is real, not that the answer represents it accurately. A verifier pass would close that gap.
• FCR, MTTR, and CSAT are synthetic models (deterministic, seeded, labeled on the dashboard tiles) — the pipeline has no real resolution or survey data.
• Single run, no baseline. One 300-ticket pass; no A/B against human-only triage, no variance estimates.

What I'd do differently at 100× scale

• Batch API for triage — 50% off token pricing for non-latency-sensitive classification; streaming ingestion with a real queue in front.
• Replace self-reported confidence with calibrated thresholds — hold out a labeled set, fit per-category calibration, and review thresholds weekly from audit-log outcomes. The current audit schema was designed to be that training data.
• Embeddings for retrieval and clustering — keyword overlap and Jaccard are honest baselines; production needs semantic retrieval, and answer drafting needs a faithfulness verifier before send.
• Eval harness in CI — golden ticket set, accuracy/calibration/deflection tracked per prompt or model change; drift monitoring on category mix and confidence distribution in production.
• PII redaction before the model, retention policy after — synthetic data made this a non-issue here; real tickets make it issue #1.
• Close the human feedback loop — review-CLI decisions (approve / send-answer / reassign) are labeled outcomes; feed them back into threshold tuning and KB gap analysis instead of leaving them in a log.
• Per-team queues, SLA-aware prioritization, per-tenant cost budgets — the single human queue and global budget are demo simplifications.

Repo layout

config/governance.yaml       routing thresholds, model assignments, audit + cost policy
supportops/                  pipeline package (ingest, triage, routing, kb, audit, rca, dashboard, costlog)
run_pipeline.py              entry point: ingest -> triage -> routing -> RCA -> dashboard
scripts/generate_tickets.py  synthetic data generator (stdlib-only, deterministic)
scripts/evaluate_triage.py   accuracy + confidence calibration vs ground truth
scripts/review_queue.py      human-review queue CLI (list / show / decide)
data/                        generated tickets, ground-truth labels, KB articles
docs/                        dashboard screenshots, sample RCA report
outputs/                     run artifacts (audit log, queues, reports) — gitignored
guardrails_runtime.md        14 governance guardrails as enforced runtime behavior
changelog.md                 audit trail of the build itself

Provenance

Built with Claude Code in one day, inside a private governed workspace whose constitution requires every change to be logged — see changelog.md for this build's audit trail, the same audit-everything pattern the pipeline enforces at runtime. No real customer data exists anywhere in this repo; the synthetic generator is part of the codebase and is the proof.