sentryctl

Linux incident debugger CLI. Simulate production faults and diagnose root causes — on bare metal or in containers.

sentryctl diagnose --fault=cpu-spike --duration=30

---

Architecture

┌─────────────────────────────────────────────────────────────┐
│                        sentryctl CLI                         │
│  cobra commands: inject | diagnose | incident               │
└───────────┬─────────────┬───────────────────────────────────┘
            │             │
   ┌────────▼──────┐  ┌───▼────────────────────────────────┐
   │  inject/      │  │  diagnose/                          │
   │  cpu.go       │  │  vmstat.go  ──► CPUUser/Wait/Mem   │
   │  mem.go       │  │  iostat.go  ──► Util/Await/IOPS    │
   │  net.go       │  │  strace.go  ──► syscall counts     │
   │  disk.go      │  │  tcpdump.go ──► packets/retransmit │
   │               │  │  perf.go    ──► IPC/PageFaults      │
   │  stress-ng    │  │  ebpf.go    ──► kprobe trace (opt) │
   │  tc netem     │  └───────────────┬────────────────────┘
   └───────────────┘                  │
                                 ┌────▼──────────┐
                                 │  report/       │
                                 │  Anomaly detect│
                                 │  JSON + summary│
                                 └────┬──────────┘
                                      │
                                 ┌────▼──────────────────────┐
                                 │  incident/                 │
                                 │  ~/.sentryctl/incidents/   │
                                 │  <id>/incident.json        │
                                 └───────────────────────────┘

---

Installation

# build
git clone https://github.com/ruthvikg/sentryctl
cd sentryctl
go build -o sentryctl .

# dependencies (Linux)
apt install stress-ng iproute2 sysstat strace tcpdump linux-perf
# or
yum install stress-ng iproute sysstat strace tcpdump perf

For eBPF support (optional):

apt install libbpf-dev clang llvm
go build -tags ebpf -o sentryctl .

---

Commands

sentryctl inject

Inject faults without capturing diagnostics.

Subcommand	Tool	Key flags
cpu-spike	stress-ng --cpu	--cpu N --timeout N
mem-leak	stress-ng --vm	--workers N --size MB --timeout N
net-latency	tc netem	--iface eth0 --latency ms --jitter ms
disk-io-saturation	stress-ng --io --hdd	--workers N --timeout N

sentryctl inject cpu-spike --cpu 4 --timeout 30
sentryctl inject mem-leak --size 512 --timeout 20
sentryctl inject net-latency --iface eth0 --latency 200 --jitter 50 --timeout 30
sentryctl inject disk-io-saturation --workers 8 --timeout 30

sentryctl diagnose

Inject fault and run all collectors concurrently. Saves incident + outputs JSON report.

# cpu spike: full diagnosis
sentryctl diagnose --fault=cpu-spike --duration=30 --cpu=4

# memory: diagnosis with mem-leak
sentryctl diagnose --fault=mem-leak --duration=30 --mem=512

# disk I/O saturation
sentryctl diagnose --fault=disk-io-saturation --duration=30

# network latency (requires root for tcpdump + tc)
sentryctl diagnose --fault=net-latency --iface=eth0 --duration=30

# disable specific collectors
sentryctl diagnose --fault=cpu-spike --no-strace --no-tcpdump --duration=30

Flags:

Flag	Default	Description
--fault	cpu-spike	Fault type
--duration	30	Seconds
--cpu	0	CPU workers (0=all)
--mem	256	MB per VM worker
--iface	any	Network interface
--disk-workers	4	Disk I/O workers
--no-strace	false	Skip strace
--no-tcpdump	false	Skip tcpdump
--no-perf	false	Skip perf stat
--no-ebpf	false	Skip eBPF tracing

sentryctl incident

# list all captured incidents
sentryctl incident list

# replay captured incident (re-prints raw data + report)
sentryctl incident replay cpu-spike-20240101-120000

---

Report output

{
  "incident_id": "cpu-spike-20240615-143022",
  "fault_type": "cpu-spike",
  "generated_at": "2024-06-15T14:30:52Z",
  "duration": "29s",
  "vmstat": {
    "cpu_user_pct": 87.4,
    "cpu_sys_pct": 4.1,
    "cpu_wait_pct": 1.2,
    "cpu_idle_pct": 7.3,
    "mem_free_kb": 512400
  },
  "disk": { "max_util_pct": 12.0, "max_await_ms": 8.0 },
  "perf": {
    "cycles": 48000000000,
    "instructions": 36000000000,
    "ipc": 0.75,
    "page_faults": 1024
  },
  "syscalls": {
    "source": "strace",
    "top_calls": [
      { "name": "read", "count": 48000 },
      { "name": "write", "count": 12000 }
    ]
  },
  "anomalies": [
    {
      "name": "cpu_saturation",
      "description": "avg CPU user=87.4% > 70%",
      "severity": "high"
    }
  ],
  "summary": "1 anomaly(s): [high]cpu_saturation"
}

---

Anomaly thresholds

Anomaly	Condition	Severity
cpu_saturation	avg CPU user > 70%	high
high_kernel_cpu	avg CPU sys > 30%	medium
high_iowait	avg CPU wait > 20%	high
low_memory	avg free mem < 50MB	high
disk_saturation	peak disk util > 80%	high
high_disk_latency	peak await > 100ms	medium
packet_retransmits	retransmit rate > 1%	high
tcp_resets	RST count > 10	medium
low_ipc	IPC < 0.5	medium
high_page_faults	page faults > 100K	high
high_context_switches	ctx switches > 500K	medium

---

eBPF (optional)

When compiled with -tags ebpf, sentryctl diagnose attaches a BPF kprobe at sys_enter and counts per-syscall invocations for the fault process.

Requires:

• Linux kernel ≥ 5.8 (CAP_PERFMON) or kernel < 5.8 (CAP_SYS_ADMIN)
• libbpf-dev installed
• BPF object at /usr/share/sentryctl/syscall_tracer.bpf.o

Falls back to strace when not available.

---

Incident store

All incidents saved to ~/.sentryctl/incidents/<id>/incident.json. Contains:

• Raw vmstat sample array
• Full root-cause report JSON
• Fault type and timestamp

Use sentryctl incident replay <id> to re-examine any past incident.

---

Demo

$ sentryctl diagnose --fault=cpu-spike --duration=10 --cpu=2 --no-tcpdump

Incident ID : cpu-spike-20240615-143022
Fault       : cpu-spike
Duration    : 10s
Starting fault injection + diagnostics...
Fault PID   : 18432

Collected: vmstat=9 samples  iostat=8 samples
Incident saved: ~/.sentryctl/incidents/cpu-spike-20240615-143022/

--- Root Cause Report ---
{
  "incident_id": "cpu-spike-20240615-143022",
  ...
  "anomalies": [{"name":"cpu_saturation","severity":"high",...}],
  "summary": "1 anomaly(s): [high]cpu_saturation"
}

==> 1 anomaly(s): [high]cpu_saturation