Summary
st2 up appears to poll the catalog at a high frequency even when the supervised fleet is stable. In a long-running deployment it averaged about 40% of one CPU core for more than 11 hours.
This report intentionally anonymizes the host, account, catalog paths, agent identities, and project names. The numeric process evidence is retained.
Related issue
Version
st2 0.1.0- build commit:
46ff6ee
- Linux, x86_64
Reproduction
-
Create or use a catalog containing at least one stable agent whose task is already running.
-
Start the supervisor:
st2 up --catalog /tmp/example/catalog --host example-host
-
Leave the catalog and inbox unchanged for several minutes.
-
Measure the supervisor:
pid=$(pgrep -n -f 'st2 up')
pidstat -p "$pid" 1 60
ps -Lp "$pid" -o pid,lwp,etimes,stat,pcpu,comm,wchan
cat "/proc/$pid/io"
grep -E 'Threads|voluntary_ctxt|nonvoluntary_ctxt' "/proc/$pid/status"
Observed in a long-running deployment
After 39,706 seconds:
STAT %CPU threads voluntary switches involuntary switches
Ssl 41.5 2 4,762,962 1,305,248
/proc/<pid>/io:
rchar: 861,670,861,903
wchar: 614,311,139
syscr: 19,090,736,150
syscw: 10,066,129
read_bytes: 68,096,000
write_bytes: 536,576
The process was sleeping in poll_schedule_timeout, but its cumulative CPU and syscall volume continued to grow while the catalog was stable.
Expected
An idle supervisor should primarily block on filesystem/timer events and consume near-zero CPU. A periodic reconciliation timer should have a bounded, configurable cadence and should not generate billions of read syscalls.
Suggested investigation
- Measure reconciliation frequency and work performed per pass.
- Check whether filesystem events produced by ST2 itself retrigger reconciliation.
- Add counters or debug logs for watch wakeups, timer wakeups, and reconciliations.
- Add an idle-load regression test that runs a stable catalog for 30–60 seconds and asserts bounded reconciliation count and CPU time.
Summary
st2 upappears to poll the catalog at a high frequency even when the supervised fleet is stable. In a long-running deployment it averaged about 40% of one CPU core for more than 11 hours.This report intentionally anonymizes the host, account, catalog paths, agent identities, and project names. The numeric process evidence is retained.
Related issue
st2 dingsidecar. This issue covers the supervisor process itself: it has a different command, process, wait state, syscall profile, and reproduction.Version
st2 0.1.046ff6eeReproduction
Create or use a catalog containing at least one stable agent whose task is already running.
Start the supervisor:
st2 up --catalog /tmp/example/catalog --host example-hostLeave the catalog and inbox unchanged for several minutes.
Measure the supervisor:
Observed in a long-running deployment
After 39,706 seconds:
/proc/<pid>/io:The process was sleeping in
poll_schedule_timeout, but its cumulative CPU and syscall volume continued to grow while the catalog was stable.Expected
An idle supervisor should primarily block on filesystem/timer events and consume near-zero CPU. A periodic reconciliation timer should have a bounded, configurable cadence and should not generate billions of read syscalls.
Suggested investigation