DEVOP-571: add SECURITY-RUNBOOK.md (Shai-Hulud incident response)

[srt0422]

Summary

Org-wide incident response runbook for Shai-Hulud-class supply-chain compromise. Lives in the .github org repo so it surfaces on every repo's Security tab.

What landed

SECURITY-RUNBOOK.md at the repo root, covering all the required scenarios:

• Detection sources — Falco, IOC sweep, Dependabot, secret scanning, manual report; channel + owner per source.
• Triage decision tree in plain text (works on phone, in PDF, pasted into Slack).
• Scenario A — dev workstation compromise (disconnect → revoke → wipe → rebuild, with the credential-by-credential checklist).
• Scenario B — CI runner compromise (disable workflow, audit blast radius, rotate, audit recent publishes).
• Scenario C — compromised package we published (yank/deprecate, advisory, clean rebuild + downstream notify).
• Scenario D — cluster pod compromise (cordon, capture-before-delete forensics, rotate SA-scoped secrets, uncordon).
• Token rotation cadence — quarterly default, with "rotate immediately if" triggers.
• Tabletop exercise schedule — annual, Q1, with explicit format and skip-approval rule.
• Appendix — gh search incantations, cosign verify cookbook, sweep trigger, node drain.

Style is operational, not compliance-flavored: short imperative steps, explicit owner per step, each scenario sectioned as Stop the bleed / Audit blast radius / Restore service / Close-out so the on-call can skim during an actual incident.

Linear

https://linear.app/alloralabs/issue/DEVOP-571

Test plan

• Walk a DevOps engineer through Scenario A start-to-finish and check whether the credential-by-credential list misses anything they actually use.
• Confirm the runbook renders correctly on the org Security tab after merge.
• First annual tabletop (DEVOP-573) will be the real test — runbook should self-update based on what was slow/ambiguous.

🤖 Generated with Claude Code

---

Summary by cubic

Adds an org-wide incident response runbook for Shai-Hulud–class supply‑chain compromises. Lives in allora-network/.github as SECURITY-RUNBOOK.md so it shows on every repo’s Security tab and satisfies DEVOP-571.

• New Features

  • Detection sources with channel and owner.
  • Plain‑text triage decision tree.
  • Four scenarios with step‑by‑step actions: developer workstation, CI runner, compromised publish, and cluster pod (Stop the bleed → Audit → Restore → Close‑out), including deriving the ServiceAccount from saved pod YAML during Scenario D audits to avoid races after pod deletion.
  • Token rotation cadence with immediate‑rotate triggers.
  • Annual tabletop schedule and close‑out rules.
  • Appendix with quick commands for gh, cosign, and kubectl.

• Bug Fixes

  • Scenario D SA grep fallback now uses POSIX [[:space:]] to work on BSD/macOS grep.

^{Written for commit 6574da0. Summary will update on new commits.}

[cubic-dev-ai]

cubic analysis

1 issue found across 1 file

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="SECURITY-RUNBOOK.md">

<violation number="1" location="SECURITY-RUNBOOK.md:349">
P2: Deriving the ServiceAccount from `$POD` after the pod is deleted can fail and break blast-radius auditing. Capture SA from the previously saved pod YAML (or before deletion) so audit steps remain executable.</violation>
</file>

Linked issue analysis

Linked issue: DEVOP-571: Write SECURITY-RUNBOOK.md in .github org repo

Status	Acceptance criteria	Notes
✅	Detection sources: where alerts fire (Falco → Slack, IOC sweep → GitHub Issue, Dependabot, secret scanning push protection)	SECURITY-RUNBOOK.md contains a '1. Detection sources' table listing Falco → #security-alerts, IOC sweep → GitHub issue workflow, Dependabot, and secret scanning push protection.
✅	Triage decision tree: confirm vs. false-positive; who pages whom	The runbook includes a Triage decision tree flowchart with branching for false positives and instructions on who to page (on-call, publisher).
✅	Dev machine suspected infected: disconnect, revoke tokens, wipe, rebuild	Scenario A lists immediate disconnect, revoke token steps (detailed per-credential), preserve evidence guidance, and wipe+reinstall + reissue credentials.
✅	CI runner suspected infected: disable workflow, rotate every secret in the env, audit recent publishes/pushes, yank+republish if needed	Scenario B instructs disabling workflows or scaling runner pool to 0, enumerates rotating every credential in scope, and auditing recent publishes (with escalation to Scenario C if suspect).
✅	Compromised package published: yank from npm/PyPI/Harbor, publish corrected version from clean environment, notify downstream	Scenario C details yanking/deprecating/unpublishing behavior per registry, steps to publish corrected version from a clean environment, and notifying downstream consumers plus updating IOC lists.
✅	Cluster pod suspected compromised: cordon node, capture forensic data via Falco/audit logs, delete pod, rotate secrets the pod could read, post-mortem	Scenario D prescribes cordoning the node, captures-forensics commands (kubectl describe/logs/exec), deleting or scaling to 0, listing/rotating secrets the pod could read, and mandatory post-mortem.
✅	Token rotation cadence: quarterly for any long-lived credential not on OIDC	Section 7 contains a token rotation table specifying quarterly rotation for PATs/npm/PyPI/Harbor and notes about migrating to OIDC.
✅	Tabletop exercise schedule: annual (see DEVOP-573)	Section 8 defines an annual Q1 tabletop exercise, format, attendees, and skip rules referencing DEVOP-573.
❌	PR merged	Merge is an acceptance condition but cannot be satisfied by the patch diff itself; the PR is adding the runbook but is not merged yet.

Architecture diagram

sequenceDiagram
    participant Dev as Developer
    participant GH as GitHub
    participant Slack as Slack #security-alerts
    participant Falco as Falco (Cluster Runtime)
    participant IOC as Daily IOC Sweep
    participant Oncall as DevOps On-Call
    participant Package as Package Registry (npm/PyPI)
    participant Cluster as Kubernetes Cluster
    
    Note over Dev,Cluster: Detection Sources
    
    Falco->>Oncall: Alert: container behavior violation
    Falco->>Slack: Cross-post alert via Falcosidekick
    IOC->>GH: Auto-file issue in incident-response repo
    IOC->>Slack: Cross-post alert
    GH->>Slack: Dependabot alert
    GH->>Dev: Secret scanning push blocked
    Dev->>Slack: Manual report: suspicious activity
    
    Note over Oncall,Cluster: Triage Decision Tree (5-10 min)
    
    Oncall->>Oncall: Acknowledge in Slack within 5 min
    alt Falco alert & known false-positive
        Oncall->>Slack: Ack, tune rule in flux-*/falco/rules.yaml
    else IOC match: package@version
        alt We published that package?
            Oncall->>Package: Scenario C: yank/deprecate package
            Oncall->>Oncall: Page publisher + on-call
        else Not our publish
            Oncall->>GH: Pin known-good version, open PR
        end
    else Secret detected
        Oncall->>Oncall: Rotate secret immediately (see §7)
        Oncall->>GH: Audit usage for last 90 days
    else Weird workstation behavior
        Oncall->>Dev: Scenario A
    else Weird CI runner behavior
        Oncall->>GH: Scenario B
    else Weird pod behavior
        Oncall->>Cluster: Scenario D
    else Unknown alert
        Oncall->>Slack: Dig in, file follow-up ticket
    end
    
    Note over Dev,Cluster: Scenario A - Workstation Compromise
    
    Dev->>Dev: Disconnect machine from network
    Dev->>Slack: Post alert from phone
    Dev->>GH: Revoke all PATs and SSH keys
    Dev->>Package: Revoke npm/PyPI tokens
    Dev->>Dev: Revoke AWS access keys
    Dev->>Dev: Wipe + reinstall OS
    Dev->>Dev: Reissue fine-grained credentials only
    
    Note over GH,Cluster: Scenario B - CI Runner Compromise
    
    Oncall->>GH: gh workflow disable <name>
    Oncall->>Cluster: Scale Arc runner replicas to 0
    Oncall->>Oncall: List runner access scope
    Oncall->>Oncall: Check GitHub Actions audit log
    
    Note over Dev,Cluster: Scenario C - Compromised Package
    
    Oncall->>Package: Yank/deprecate package version
    Oncall->>GH: Publish GHSA advisory
    Oncall->>Dev: Notify downstream consumers
    
    Note over GH,Cluster: Scenario D - Cluster Pod Compromise
    
    Oncall->>Cluster: kubectl cordon node
    Oncall->>Cluster: kubectl describe pod > capture.txt
    Oncall->>Cluster: kubectl delete pod
    Oncall->>Cluster: kubectl uncordon node
    
    Note over Dev,Oncall: Token Rotation Cadence (Quarterly)
    Note over GH,Cluster: Tabletop Exercise (Annual Q1)

Loading

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review, or fix all with cubic.}

[cubic-dev-ai]

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="SECURITY-RUNBOOK.md">

<violation number="1" location="SECURITY-RUNBOOK.md:356">
P1: Replace the non-portable `\s` with `[[:space:]]` in the grep fallback so the command works correctly on both GNU and BSD/macOS systems.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review, or fix all with cubic.}