bug(subprocess-local): an active-empty Linux scope unit is never released without a termination request — leaks one unit plus a 5s systemctl polling loop per completed command

[grantyuan]

Problem

SystemdScopeOwner.emptyRange() requires terminationRequested before it releases a leftover active-empty scope unit. That flag is only set by signal()/terminate(). A command that completes on its own never issues a termination request, so:

1. The transient systemd scope unit stays active with TasksCurrent=0 forever — systemd ends a scope only on the populated→empty transition, and a payload that never populated the cgroup leaves the unit active indefinitely (the class doc comment already documents this).
2. The waitForExit() observation loop keeps spawning systemctl --user show for that unit every 5 s — forever.
3. The handle stays in the subprocess service's live set, retaining memory.

Under sustained multi-agent usage (hundreds of tool calls) this accumulates. Observed on a long-running dsh web host: 900+ leaked dsh-subprocess-* scope units, ~500 concurrent systemctl children at peak (spawn storm), 60% of one core burned inside child_process.spawn, and heap growth from the retained handles. Clearing the stale units collapsed the storm instantly and CPU returned to idle.

Fix

Drop the terminationRequested gate. The remaining predicate already proves the range is conclusively finished — the class's own doc comment argues exactly this: "a departed client cannot add another payload; a consumed request proves the payload already entered the scope." Either the direct client has departed (it forked every process it will ever fork) or the launch request was consumed (the payload entered the scope), and with TasksCurrent=0 the managed range holds nothing. Termination signaling is orthogonal to settlement.

aaa02a3970 introduced emptyRange/releaseEmptyRange but gated them on terminationRequested, which covers only the kill/terminate path; normal completion and launch failures leak.

Verification

• New regression test: releases an active empty scope once its client has completed without a termination request — completes a scope with no signal and asserts the leftover unit is released (systemctl stop issued) and the observation settles.
• Renamed keeps waiting for an active empty scope no termination has requested to ... while its client is still alive — the wait invariant is client liveness, not absence of termination.
• Full subprocess-local suite: 382 passed, 0 failed (also covers terminal scopes, spawn, windows-job suites).

---

Posted per the contributing guide (bug reports go to Discussions; external PRs are not accepted). Context: observed on a long-running dsh web host driving an external agent team (0.2.1-rc.3 through current master ce7073d5 — the gate is still present there). A fix is the one-line gate removal in emptyRange() plus dropping the now write-only field; I can share the exact patch in this thread if useful.

[dgesteves]

Your reading of the gate is right, and it's already reported: #9220 describes the same thing on 0.2.0-rc.2 (246 active scopes at TasksCurrent=0, same terminationRequested analysis), and a comment there from 10-25 attaches essentially this patch. Nobody from the team has replied yet, so cross-linking the two threads should help.

I checked against master ce7073d. The file is identical in rc.3, rc.5 and 0.2.2-alpha.2, last changed 10-15:

• emptyRange() still requires terminationRequested, which only signal() sets
• the poll interval doubles up to SYSTEMCTL_TIMEOUT_MS, so every leaked owner polls every 5 s
• the handle leaves live only once waitForExit() resolves

One nuance: a command that exits on its own normally doesn't hit this. systemd-run --scope puts itself into the scope and the bootstrap execves the payload, so the payload's exit is a populated-to-empty transition, and systemd stops and collects the unit. The leak needs systemd to miss that transition, and neither report has pinned down why it does. One thing worth checking: systemd only learns a scope is empty through an inotify watch on its cgroup.events, and v255 ignores a failed watch and only logs Failed to add control inotify watch descriptor. So journalctl --user -b | grep 'inotify watch descriptor' would show whether the user manager ran out of watches. Whatever the trigger, settling shouldn't depend on a termination request, and dropping the gate looks safe to me: a launch still in flight keeps its request file and a live client, and background descendants keep TasksCurrent above 0.

Until a fix lands, stopping the leftover units is enough, and you don't need a restart:

systemctl --user list-units --type=scope --state=active --plain --no-legend 'dsh-subprocess-*' 'dsh-terminal-*' \
  | awk '{print $1}' \
  | while read -r unit; do
      if [ "$(systemctl --user show --property=TasksCurrent --value "$unit")" = 0 ]; then
        systemctl --user stop "$unit"
      fi
    done

After the stop, the next poll finds the unit gone or inactive, the loop ends and the handle is released. That matches the instant recovery you saw. Running it from a user timer every few minutes keeps the count down.

I've since run this under systemd 255 (Ubuntu 24.04) with a user manager: a scope whose command exits on its own is collected, a scope whose cgroup watch couldn't be added stays active at TasksCurrent=0, and the script stops that one and leaves a live scope alone. Details in the reply below.

[dgesteves]

I've now reproduced this under systemd 255.4 (Ubuntu 24.04) with a user manager, using systemd-run --user --scope and a payload that exits after a second:

• Normally the scope is collected as soon as the payload exits (LoadState=not-found right after).
• With fs.inotify.max_user_watches set to what uid 1000 was already using, the user manager logged dsh-subprocess-leak1.scope: Failed to add control inotify watch descriptor for control group …/app.slice/dsh-subprocess-leak1.scope: No space left on device. After the payload exited, the unit stayed active running with TasksCurrent=0, which is the state in your report.
• The script above stopped that unit and left a live dsh-terminal-* scope running.

So on an affected host, journalctl --user -b | grep 'inotify watch descriptor' should tell you whether this is the trigger. If those lines are there, the user's inotify watches are used up, and raising fs.inotify.max_user_watches (or finding what holds them) should stop new leaks, independently of dropping the terminationRequested gate.