Repository navigation
Capstone · Switchboard: an LOB-pluggable incident triage desk #39
akash-coded
started this conversation in
Hands-on Labs
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Switchboard
Capstone Project: An LOB-pluggable incident triage desk
Contents
Part 1: Orientation
1. Situation
A large enterprise runs one IT service desk across several Lines of Business. Four are in scope here: iCargo, iStay, iLogistics and iFlyLoyalty. Thousands of tickets arrive daily through email, a self-service portal, and automated monitoring alerts.
Nothing can be worked on until it has been triaged, and triage is four separate judgements rather than one.
All four are done by hand today. The work is slow, it is inconsistent between agents and between shifts, and it collapses during incident storms, which is precisely when accuracy matters most. A misrouted P1 during a storm costs more than a hundred correctly routed P4s.
The people the system serves want different things from it, and the fourth row below is the one that shapes the architecture.
2. Mental model
Four movements, in strict order. Everything in this document is an elaboration of one of them.
flowchart LR TXT[Ticket text] --> PROP[Proposal] CFG[LOB configuration] --> DEC[Decision] PROP --> DEC DEC --> ACT[Action] ACT --> REC[Record]Two consequences follow directly, and both are graded.
3. Concept recap
The agentic concepts this project draws on, and where each one lands.
Part 2: The system
4. Objectives and scope
Switchboard is a triage decision service. It reads a ticket, produces an auditable decision, and executes only the actions its configuration allows. It sits beside the ITSM platform rather than replacing it. The ITSM platform remains the system of record for tickets; Switchboard is the system of record for triage decisions.
5. Acceptance criteria
6. System context
flowchart TB REQ[Requester] L1[L1 Service Desk] L2[L2 and L3 Specialists] MGR[Service Desk Manager] OWN[LOB Owner] SW[Switchboard Triage Desk] ITSM[(ITSM Platform)] IDP[(Identity and Entitlement)] MON[(Monitoring and Alerting)] KB[(Knowledge Base and Known Errors)] NOTIF[(Notification Service)] REQ --> ITSM MON --> ITSM ITSM --> SW SW --> ITSM SW --> IDP SW --> KB SW --> NOTIF SW --> L1 SW --> L2 MGR --> SW OWN --> SWThe two arrows that are easy to overlook run from the manager and the LOB owner into the system. The manager operates the kill switch and reviews sampled decisions. The LOB owner ships configuration. Neither writes code and neither raises a ticket, so both need an interface designed for them from the start.
7. Components
flowchart LR INTAKE[Intake Adapter] NORM[Normalizer and Redactor] ORCH[Triage Orchestrator] POL[Policy Engine] GW[Tool Gateway] HITL[Approval Queue] OBS[Telemetry and Cost Meter] EVAL[Eval Harness] PACKS[(LOB Pack Registry)] RET[(Retrieval Store)] LEDGER[(Decision Ledger)] MOCK[(ITSM Mock)] INTAKE --> NORM NORM --> ORCH PACKS --> ORCH PACKS --> POL RET --> ORCH ORCH --> POL POL --> GW GW --> MOCK ORCH --> HITL HITL --> ORCH ORCH --> LEDGER POL --> LEDGER GW --> LEDGER ORCH --> OBS EVAL --> ORCH LEDGER --> EVAL8. The triage path
Nine stages. Each stage uses the cheapest mechanism that can answer its question: deterministic code first, retrieval second, a model only where the question genuinely requires language understanding.
flowchart TB S0[S0 Ingest and Normalize] S1[S1 Resolve LOB] S2[S2 Classify Category] S3[S3 Score Impact and Urgency] P{Priority Matrix} S4[S4 Enrich and Retrieve] S5[S5 Decide Disposition] G{Policy Gate} S6[S6 Execute Actions] S7[S7 Verify] S8[S8 Escalate to Human] S9[S9 Record and Emit] S0 --> S1 S1 --> S2 S2 --> S3 S3 --> P P --> S4 S4 --> S5 S5 --> G G -->|cleared| S6 G -->|blocked or low confidence| S8 S6 --> S7 S7 -->|verified| S9 S7 -->|failed| S8 S8 --> S99. Two journeys
Automated resolution
sequenceDiagram participant ITSM as ITSM Mock participant SW as Orchestrator participant PACK as LOB Pack participant M as Model participant POL as Policy Engine participant GW as Tool Gateway participant LED as Decision Ledger ITSM->>SW: new ticket SW->>SW: normalize and redact SW->>PACK: load pack for resolved LOB PACK-->>SW: taxonomy, matrix, autonomy rules SW->>M: classify against closed taxonomy M-->>SW: category plus confidence SW->>M: extract impact and urgency evidence M-->>SW: evidence fields SW->>SW: apply priority matrix SW->>M: propose disposition M-->>SW: proposed action plus confidence SW->>POL: submit proposal with pack version POL->>POL: check allowlist, priority ceiling, preconditions POL-->>SW: granted at level A2 SW->>GW: execute with idempotency key GW->>ITSM: apply resolution and close ITSM-->>GW: ok GW-->>SW: result SW->>SW: verify final ticket state SW->>LED: write decision record SW->>ITSM: notify requester with appeal pathThe policy engine receives the proposal, the category, the computed priority and the pack version. It does not receive the ticket body.
Escalation and resume
sequenceDiagram participant SW as Orchestrator participant POL as Policy Engine participant Q as Approval Queue participant H as L1 Agent participant GW as Tool Gateway participant LED as Decision Ledger SW->>POL: submit proposal at A2 POL-->>SW: downgraded to A1, reason entitlement_unverified SW->>Q: park decision with evidence bundle Q-->>H: item appears with proposal and evidence H->>Q: approve with an edit to the category Q-->>SW: resume with verdict and edit SW->>GW: execute the approved action GW-->>SW: result SW->>LED: record proposal, downgrade reason, verdict and editTwo requirements come out of this journey.
Part 3: The three pillars
10. Configuration model: the LOB pack
Zero-code onboarding holds only if every LOB-specific fact lives in configuration and nothing LOB-specific lives in source. One pack per Line of Business, loaded and validated at runtime.
Worked pack: iCargo
Validation
A pack failing any rule below is rejected at load, and that Line of Business falls back to fully manual triage while the others continue unaffected.
schema_versionmatches a version the loader knowsneverrule carries a reason stringThe test that proves it
No source file may contain the string
icargo,istay,ilogisticsoriflyloyalty. Enforce this withtests/test_no_tenant_strings.py. Write the test in the first milestone and let it fail until the design makes it pass.11. Decision model
Priority is a rule, not a judgement
A model reads a ticket well and answers how many people are affected and how fast the situation is degrading. It is far less consistent about what those two facts together should mean, and it cannot justify that step to a reviewer in a way that survives a second run. So the work splits.
flowchart LR T[Ticket text] --> M[Model extracts evidence] M --> I[impact: single_station] M --> U[urgency: blocking] I --> MX{Priority matrix from pack} U --> MX MX --> P[P2] P --> OV{Overrides} OV --> F[Final priority with cited cell]The decision record carries the matrix cell that produced the answer, so a reviewer can dispute the evidence without having to dispute the model. Routing splits the same way: the model chooses a category, and the pack maps categories to teams.
The autonomy ladder
Autonomy is a setting per category, declared in configuration.
The gate that grants a level:
flowchart TB PROP[Proposed action at requested level] C1{Category in allowlist} C2{Priority at or below ceiling} C3{Confidence at or above threshold} C4{All preconditions satisfied} C5{No never rule matches} C6{Rate limit not exceeded} GRANT[Grant requested level] DOWN[Downgrade to A1 and park] BLOCK[Block and route to human at A0] PROP --> C1 C1 -->|no| BLOCK C1 -->|yes| C2 C2 -->|no| BLOCK C2 -->|yes| C3 C3 -->|no| DOWN C3 -->|yes| C4 C4 -->|no| DOWN C4 -->|yes| C5 C5 -->|matches| BLOCK C5 -->|clear| C6 C6 -->|exceeded| DOWN C6 -->|within| GRANTEvery downgrade and every block writes its reason into the decision record.
12. Orchestration model
The dividing line to apply:
Almost all of Switchboard is a graph, because almost all of it is a fixed sequence with hard gates. Categorisation always precedes prioritisation. The policy gate always precedes execution. An agent choosing to skip one of those is a defect rather than initiative.
One place resists prediction. A ticket reporting slow portal response, failed EDI acknowledgements and a partial report might be a network problem, an authentication problem, an integration problem or a data problem, and which specialist should look next depends entirely on what the previous one found.
flowchart TB subgraph CONTROL[Control graph] A[Classify] B[Prioritise] C{Ambiguous multi-symptom} D[Policy gate] E[Execute] F[Record] end subgraph SWARM[Diagnostic swarm inside one node] N[Network specialist] AU[Auth specialist] IN[Integration specialist] DA[Data specialist] end A --> B B --> C C -->|no| D C -->|yes| SWARM SWARM --> D D --> E E --> F N --- AU AU --- IN IN --- DAConstraints on the swarm:
Framework decision
The current implementation of this flow uses no agent framework. Produce an architecture decision record comparing three options: no framework with a hand-written loop, Strands, and LangGraph. Judge them on state persistence across an approval pause, testability of the policy path, observability, and the volume of code that would have to change to switch. State the decision, the consequences accepted, and what would trigger a revisit.
The framework is reversible in one file and owned by the team. The platform, meaning identity, networking, observability and cost, is owned by architecture and is far harder to change.
Part 4: Controls
13. Tool gateway
Every side effect passes through one gateway. No stage calls a tool directly.
itsm.classifyitsm.set_priorityitsm.assignitsm.commentitsm.resolveidentity.reset_portal_passwordreport.reissuenotify.requesterThree properties the gateway enforces:
14. Retrieval
Retrieval answers one question: has something like this been seen before, and is there a known error or runbook for it.
Record which documents were retrieved and their scores. When a wrong disposition is investigated, the first question is what the system was looking at.
15. Guardrails
neverruleInjection through the ticket body
In a triage system the attacker writes the input. Anyone who can raise a ticket can attempt this, so the gold set carries adversarial cases:
The defence is architectural. A stronger system prompt is a mitigation; the control is that the ticket body cannot reach the code path that authorises an action.
flowchart LR TXT[Ticket text, untrusted] --> ORCH[Orchestrator] ORCH --> PROP[Structured proposal only] PROP --> POL[Policy Engine] PACK[LOB pack, trusted] --> POL POL --> GW[Tool Gateway] TXT -.blocked.-> POL TXT -.blocked.-> GWThe policy engine reads a validated structured proposal and a trusted pack, never free text. An injection can at best corrupt a category, which the gate then evaluates on its merits. It cannot grant itself an autonomy level. The input scan that flags instruction-like spans is a telemetry signal, recorded in the decision record, and not the thing standing between an attacker and a closed P1.
16. Human oversight
A category earns its autonomy level by demonstrating precision at each rung.
flowchart LR R0[Shadow] --> R1[Suggest] R1 --> R2[Assist] R2 --> R3[Auto for one category] R3 --> R4[Auto for the category set] R4 -.regression.-> R2Shadow mode also produces the labelled data that the precision target requires, which is the answer when a stakeholder asks how a brand new Line of Business could ever meet it.
Three controls run alongside the ladder:
Part 5: Evidence
17. Traceability
One record per ticket, written once, immutable after the run. A reviewer reconstructs the decision from this record alone, without reading application logs.
{ "schema_version": "1.0", "decision_id": "dec_01J9X...", "correlation_id": "cor_01J9X...", "ticket_id": "TCK-448120", "lob": "icargo", "pack_version": 4, "stages": [ { "stage": "classify", "mechanism": "model", "model_id": "us.anthropic.claude-haiku-4-5-20251001-v1:0", "prompt_version": "classify-v3", "input_hash": "sha256:9ac1...", "output": { "category": "edi_partner_feed_failure" }, "confidence": 0.82, "input_tokens": 812, "output_tokens": 47, "cost_usd": 0.00091, "latency_ms": 640, "retries": 0 }, { "stage": "prioritise", "mechanism": "rule", "evidence": { "impact": "single_station", "urgency": "blocking" }, "matrix_cell": "single_station/blocking", "output": { "priority": "P2" }, "override_applied": null } ], "retrieval": [ { "doc_id": "icargo-ke-0042", "score": 0.71, "title": "Partner EDI acknowledgement backlog" } ], "proposal": { "disposition": "route", "team": "cargo-integration-l3", "requested_level": "A1" }, "policy": { "granted_level": "A0", "decision": "blocked", "reason": "never_rule:priority in [P1, P2]", "rules_evaluated": 6 }, "guardrails": [ { "check": "injection_scan", "verdict": "flagged", "evidence_span": "line 14" }, { "check": "output_forbidden", "verdict": "pass" } ], "actions": [], "human": { "required": true, "queue": "cargo-integration-l3", "verdict": null }, "outcome": "escalated", "totals": { "cost_usd": 0.00214, "latency_ms": 1980, "model_calls": 3 } }Emit the same information as spans, one per stage, with the correlation id as the trace id. The ledger serves audit and evaluation; the spans serve operations.
18. Cost
Declare the baseline first: every stage running on the most capable available model. Every saving is measured against that number and attributed to a specific decision.
top_k, with retrieved token counts recordedA verify-then-escalate cascade is cheaper than always running the expensive model only when the cheap model resolves a large enough share of traffic alone. The break-even resolve rate is:
Below $r^$, both models run on most tickets and the cascade costs more than the simple approach. Compute $r^$ from the current published prices of the two models in use, measure the actual resolve rate, and state whether the cascade earns its place. A conclusion that it does not, supported by numbers, is a stronger result than a cascade shipped because it looked sophisticated.
Report cost per triaged ticket, cost per automatically resolved ticket, and the cost of an escalation, all against the declared baseline.
19. Evaluation
The corpus
The corpus is synthetic. Production service desk data carries personal and customer-identifying content and is not used.
Label every gold ticket with the correct LOB, category, impact, urgency, priority, team, and whether automated resolution would have been correct.
Two public datasets are useful for calibrating how messy real ticket text is.
Tobi-Bueck/customer-support-ticketson Hugging Face carries roughly 61,700 rows with queue, priority, type, language and tag columns, and its CC BY-NC 4.0 licence restricts it to non-commercial use. The Kaggle IT service ticket classification dataset carries roughly 47,800 rows with ticket text and a topic group only. Neither is used for scoring.Metrics
The optimisation target
Optimise auto-resolve precision subject to a coverage floor, rather than overall accuracy. The asymmetry is the reason. A ticket wrongly left for a human costs a few minutes; a P1 wrongly closed costs an outage nobody is working on. Any change that raises accuracy while lowering auto-resolve precision has made the system worse.
Error analysis and the regression gate
Take the twenty worst errors. For each, state the stage that failed, its mechanism, the root cause and the fix, distinguishing a taxonomy problem, meaning two categories that overlap so far that no model could separate them, from a model problem.
Ship a single command that runs the full evaluation offline and fails when any metric drops below its threshold or either hard gate is breached.
Part 6: Delivery
20. Milestones
Effort is per learner, working in a team of three or four.
21. Scope levels
Every team attempts Core. Stretch and Advanced are open to any team that completes Core.
22. Repository and environment
us-east-1us.anthropic.claude-haiku-4-5-20251001-v1:0for the capable tier, an Amazon Nova model for the cheap tierus.prefix is mandatory. A bare model id raises a validation exceptionRun instructions cover a local virtual environment and a hosted notebook environment, and the README states which produced the graded run.
23. Assessment
Rubric
Hard gates
Assessed as pass or fail, separately from the rubric. A submission failing any gate does not pass regardless of its total.
Model-assisted work disclosure
Select one artifact generated with a language model: a taxonomy, the priority matrix, or a batch of fifty synthetic tickets. Submit the prompt used, the raw unedited output, a written critique naming at least three concrete defects, and the corrected version with the changes identified. Defects worth looking for include overlapping categories that no classifier could separate, synthetic tickets where the label is guessable from a single keyword, unrealistically even distribution across priorities, and generated text containing the category name.
Demonstration
A fifth Line of Business, iPortside, is released shortly before the demonstration, as a pack with its gold set. Onboard it and run its evaluation live, with no change to any source file and no restart of the reasoning code. A malformed variant is released alongside it, and rejecting that variant cleanly is part of the test.
Viva
A short oral examination drawn from the submitted work. Expect questions on why a stage sits where it does, what happens when a named component fails, and a defence of one decision you would now make differently.
Executive memo
One page, written for a service desk director. State what was built, what it measurably does, what it deliberately does not do, what it cost, and what would have to be true before it could run against production traffic. Figures in the memo match the evaluation output.
24. Known limitations
All reactions