Summary
The st2 ding sidecar appears to spin or poll aggressively while waiting for inbox activity. In a deployment with no sustained message traffic, one sidecar averaged about 60% of one CPU core for more than 72 minutes.
This report intentionally anonymizes the host, account, catalog paths, agent identities, and project names. The numeric process evidence is retained.
Related issue
Version
st2 0.1.0- build commit:
46ff6ee
- Linux, x86_64
Reproduction
-
Start a persistent PTY for an agent and ensure its inbox is empty.
-
Start the sidecar:
st2 ding \
--identity example-agent \
--root /tmp/example/catalog
-
Do not send messages and leave the agent status unchanged.
-
Measure the sidecar:
pid=$(pgrep -n -f 'st2 ding')
pidstat -p "$pid" 1 60
ps -Lp "$pid" -o pid,lwp,etimes,stat,pcpu,comm,wchan
cat "/proc/$pid/io"
grep -E 'Threads|voluntary_ctxt|nonvoluntary_ctxt' "/proc/$pid/status"
Observed in a long-running deployment
After 4,346 seconds:
STAT %CPU RSS threads voluntary switches involuntary switches
Rsl 59.6 5 MiB 2 2,276,158 39,686,551
/proc/<pid>/io:
rchar: 3,209,458,431
wchar: 8,620
syscr: 434,210,543
syscw: 62
read_bytes: 602,112
write_bytes: 65,536
The very high syscall and involuntary-context-switch counts are disproportionate to an idle inbox watcher.
Expected
With no inbox changes, the sidecar should block on a filesystem notification or a low-frequency timer and consume near-zero CPU.
Suggested investigation
- Check the watch loop for nonblocking reads or immediate retry paths.
- Check whether presence/status reads are performed continuously while idle.
- Add metrics for watcher wakeups, directory scans, deferred-message checks, and PTY probes.
- Add an idle-load test that asserts bounded syscalls and CPU time over 30–60 seconds.
Summary
The
st2 dingsidecar appears to spin or poll aggressively while waiting for inbox activity. In a deployment with no sustained message traffic, one sidecar averaged about 60% of one CPU core for more than 72 minutes.This report intentionally anonymizes the host, account, catalog paths, agent identities, and project names. The numeric process evidence is retained.
Related issue
st2 upsupervisor. This issue covers the per-agent inbox sidecar: it has a different command, process, syscall profile, and reproduction.Version
st2 0.1.046ff6eeReproduction
Start a persistent PTY for an agent and ensure its inbox is empty.
Start the sidecar:
Do not send messages and leave the agent status unchanged.
Measure the sidecar:
Observed in a long-running deployment
After 4,346 seconds:
/proc/<pid>/io:The very high syscall and involuntary-context-switch counts are disproportionate to an idle inbox watcher.
Expected
With no inbox changes, the sidecar should block on a filesystem notification or a low-frequency timer and consume near-zero CPU.
Suggested investigation