Repository navigation
[Bug] dsh web RSS grows with session-store volume: boot-time session-history loading is off-heap and gets OOM-killed at a 1 GiB cgroup limit (0.1.7-rc.2) #8639
Dreamobi01
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Summary
In a production deployment of DSH web, the Node process RSS grows in proportion to the total volume of the session store on disk, is killed by the kernel at the container's cgroup memory limit, and recovers completely when the store is pruned. Evidence points to the growth living outside the V8 heap and accruing during (or right after) boot-time loading/replay of stored session history.
This looks adjacent to #6511, #7414, #7356 and #8277 (session-log decode/retention and O(session size) reads), but those describe heap-side retention tied to session opens / live sessions. What we measured is: boot-time, store-volume-scale, off-heap — a different mechanism in the same symptom family.
Reproduction
Three DSH web instances (0.1.7-rc.2), built from the same image and same bundle with identical
NODE_OPTIONS, each in its own container with a 1 GiB cgroup memory limit, Linux x86_64. The only meaningful variable between them is session-store volume:dmesg)Causal check: after pruning instance A's session store, RSS settles at ~183 MB and stays flat. No other process-level difference was present.
Current behavior
Two measurements isolate where the growth lives:
V8 heap is not the driver. Heap snapshots taken at RSS 178 MB and 529 MB (instance A, same boot window) diff to: total heap size net decreased, largest single-constructor delta +0.134 MB. No heap-side allocation account for hundreds of MB.
The growth is off-heap.
/proc/<pid>/smaps_rollupover the same window shows theAnonymous/Private_Dirtyregion growing by ~556 MB — a single off-heap region absorbing essentially all of the RSS growth.V8 high-water ratchet. Once touched, heap pages are not returned to the OS after usage drops, so RSS is monotonic across growth episodes.
CPU companion. For the first ~10 minutes after boot, the process burns ~96% user-space CPU, then stays elevated — consistent with a compact loop doing large off-heap allocations while loading/replaying stored session history.
Net effect: a deployment with a large accumulated session store cannot survive a 1 GiB (or likely any modest) container limit, and the store only ever grows in normal use.
Expected behavior / questions for maintainers
We would appreciate any of the following from the team:
dsh webwith tight memory limits?Environment
dsh web, version 0.1.7-rc.2NODE_OPTIONSon all instances)dmesgoom-kill records, per-process RSS sampling, V8 heap snapshot diffing,/proc/<pid>/smaps_rollupAll reactions