Incident Response Runbooks

Incident Response Runbooks

During an incident, protect agent workloads and evidence first. CCAM monitoring must remain fail-safe; do not make provider execution depend on restoring the dashboard.

Universal first five minutes

1. State the symptom, affected users/sources, and first known time.
2. Record CCAM version and deployment topology.
3. Preserve server/edge logs and current health output.
4. Check disk/volume capacity and database availability.
5. Avoid cleanup, reinstallation, or restore until the failure layer is known.

Dashboard unavailable

• Check process/pod state and /health.
• Check edge routing separately from direct service access.
• Verify database volume attachment and write permissions.
• Inspect restarts and the first startup error.
• If rollback is needed, back up first and retain the failed image/logs.

Provider hooks may fail harmlessly during the outage. Recover missed history later with import/rescan.

New sessions missing

• Determine whether all providers/sources or only one are affected.
• Run a controlled short session.
• Inspect hook installation and remote-source status.
• Compare raw provider transcript freshness.
• Check server persistence and only then the browser live connection.

Use Hook Health and Ingestion.

Sessions remain active or waiting

• Compare last event and transcript activity.
• Check child agents and long-running tools.
• Distinguish local, forwarded-hook, container, and remote-source liveness rules.
• Never directly rewrite status in SQLite.

Use Recipe: Investigate a Stuck Session.

Cost or token anomaly

• Freeze the relevant filter/time range.
• Identify contributing sessions and model identities.
• Check usage completeness and pricing overrides.
• Preserve the export before changing pricing.

Use Recipe: Audit an Expensive Session.

Suspected credential exposure

• Restrict external access.
• Rotate only the affected dashboard, hook, or MCP secret first.
• Restart consumers and verify the old credential fails.
• Inspect access logs and assess transcript exposure.

Closeout

Document impact, timeline, root cause, recovery, evidence, and follow-ups. Add a regression test or monitoring signal when the incident revealed a detectable gap.