bug(diagnostics): check_systemd_timing_alignment false-positive — systemctl show returns default 90s for nonexistent --user unit

[afred-pennyworht]

Symptom

Running hermes-gateway from a system-level systemd unit (operator deploys via NixOS / Ansible / etc., not via hermes gateway install) produces a permanent startup warning even when the actual system unit's TimeoutStopSec is correctly sized:

```
WARNING gateway.run: Stale systemd unit detected: hermes-gateway.service has
TimeoutStopSec=90s but drain_timeout=180s (expected >=210s). systemd may
SIGKILL the gateway mid-drain. Run `hermes gateway service install --replace`
to regenerate the unit, or shorten agent.restart_drain_timeout.
```

…but systemctl cat hermes-gateway.service confirms TimeoutStopSec=240s, which satisfies the documented drain_timeout + 30s headroom = 210s threshold. No mismatch in reality.

(Side note: the warning text also references the old CLI surface — should be hermes gateway install --replace, not hermes gateway service install --replace. The service sub-noun was removed.)

Root cause

gateway/shutdown_forensics.py:367-389 in 0.15.2:

```python
for flag in (["--user"], []):
try:
result = subprocess.run(
["systemctl", *flag, "show", unit_name, "--property=TimeoutStopUSec"],
capture_output=True, text=True, timeout=2.0,
)
except (FileNotFoundError, subprocess.TimeoutExpired, OSError):
continue
if result.returncode != 0:
continue
# … parses TimeoutStopUSec, sets timeout_us, breaks out
```

systemctl show is permissive: for a unit that does not exist in the targeted manager, it returns rc=0 and emits the systemd compiled-in defaults — TimeoutStopUSec=1min 30s (90s). It does not error out the way systemctl status <nonexistent> would.

When the gateway runs as User=hermes with XDG_RUNTIME_DIR=/run/user/<uid> set (mandatory for rootless podman in the same service), the user-bus is reachable. So systemctl --user show hermes-gateway.service --property=TimeoutStopUSec is the first iteration of the loop, succeeds with rc=0, returns the 90s default, parses, breaks — and the system unit (with the correct 240s) is never consulted.

Reproducer (on a host where the hermes user has no user-level units installed):

```bash
$ sudo -u hermes -H \
env XDG_RUNTIME_DIR=/run/user/1002 \
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1002/bus \
systemctl --user list-unit-files | grep -i hermes
UNIT FILE STATE PRESET
57 unit files listed.

$ sudo -u hermes -H \
env XDG_RUNTIME_DIR=/run/user/1002 \
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1002/bus \
systemctl --user show hermes-gateway.service \
--property=LoadState --property=TimeoutStopUSec
LoadState=not-found
TimeoutStopUSec=1min 30s # ← bogus default, since LoadState=not-found
```

Hermes blindly trusts the 90s value.

Proposed fix

Add LoadState to the same query and skip iterations where the unit isn't loaded:

```diff
for flag in (["--user"], []):
try:
result = subprocess.run(

•        [\"systemctl\", *flag, \"show\", unit_name, \"--property=TimeoutStopUSec\"],
  

•        [\"systemctl\", *flag, \"show\", unit_name,
  

•         \"--property=LoadState\",
  

•         \"--property=TimeoutStopUSec\"],
         capture_output=True, text=True, timeout=2.0,
     )
  

  except (FileNotFoundError, subprocess.TimeoutExpired, OSError):
  continue
  if result.returncode != 0:
  continue

• Output: "TimeoutStopUSec=1min 30s" or "TimeoutStopUSec=90000000"

• Skip if the unit isn't actually loaded in this manager — `systemctl

• show` returns rc=0 and emits compiled-in defaults for nonexistent

• units (notably TimeoutStopUSec=1min 30s), which would produce a

• false-positive 'stale unit' warning when the gateway is deployed

• via a system-level unit but the user manager is reachable.

• load_state: Optional[str] = None
• for line in result.stdout.splitlines():

•    if line.startswith(\"LoadState=\"):
  

•        load_state = line.split(\"=\", 1)[1].strip()
  

•        break
  

• if load_state and load_state != "loaded":

•    continue
  

  for line in result.stdout.splitlines():
  if line.startswith("TimeoutStopUSec="):
  ...
  ```

Same selectivity, one extra property in the query, no other behaviour change.

Bonus: while touching the file, the warning text on gateway/run.py:3904 could be updated to reference hermes gateway install --replace (the service sub-noun was removed in a recent CLI refactor — the current text gives invalid choice: 'service').

Happy to send a PR.

Environment

• hermes-agent 0.15.2 (v2026.5.29.2)
• NixOS x86_64, gateway deployed via systemd.services.hermes-gateway (system unit, NOT via hermes gateway install)
• gateway user: hermes (linger=true, system service with XDG_RUNTIME_DIR=/run/user/<uid> set for rootless podman)
• No user-level hermes-gateway.service installed (verified via systemctl --user list-unit-files)

[dsameer0-code]

Unsubscribe

…

On Mon, Jun 1, 2026 at 7:35 AM Alfred Pennyworth ***@***.***> wrote: *afred-pennyworht* created an issue (NousResearch/hermes-agent#36755) <#36755> Symptom Running hermes-gateway from a *system-level* systemd unit (operator deploys via NixOS / Ansible / etc., not via hermes gateway install) produces a permanent startup warning even when the actual system unit's TimeoutStopSec is correctly sized: ``` WARNING gateway.run: Stale systemd unit detected: hermes-gateway.service has TimeoutStopSec=90s but drain_timeout=180s (expected >=210s). systemd may SIGKILL the gateway mid-drain. Run `hermes gateway service install --replace` to regenerate the unit, or shorten agent.restart_drain_timeout. ``` …but systemctl cat hermes-gateway.service confirms TimeoutStopSec=240s, which satisfies the documented drain_timeout + 30s headroom = 210s threshold. No mismatch in reality. (Side note: the warning text also references the old CLI surface — should be hermes gateway install --replace, not hermes gateway service install --replace. The service sub-noun was removed.) Root cause gateway/shutdown_forensics.py:367-389 in 0.15.2: ```python for flag in (["--user"], []): try: result = subprocess.run( ["systemctl", *flag, "show", unit_name, "--property=TimeoutStopUSec"], capture_output=True, text=True, timeout=2.0, ) except (FileNotFoundError, subprocess.TimeoutExpired, OSError): continue if result.returncode != 0: continue # … parses TimeoutStopUSec, sets timeout_us, breaks out ``` systemctl show is *permissive*: for a unit that does not exist in the targeted manager, it returns rc=0 and emits the systemd compiled-in defaults — TimeoutStopUSec=1min 30s (90s). It does *not* error out the way systemctl status <nonexistent> would. When the gateway runs as User=hermes with XDG_RUNTIME_DIR=/run/user/<uid> set (mandatory for rootless podman in the same service), the user-bus is reachable. So systemctl --user show hermes-gateway.service --property=TimeoutStopUSec is the first iteration of the loop, succeeds with rc=0, returns the 90s default, parses, breaks — and the system unit (with the correct 240s) is never consulted. Reproducer (on a host where the hermes user has no user-level units installed): ```bash $ sudo -u hermes -H \ env XDG_RUNTIME_DIR=/run/user/1002 \ DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1002/bus \ systemctl --user list-unit-files | grep -i hermes UNIT FILE STATE PRESET 57 unit files listed. $ sudo -u hermes -H \ env XDG_RUNTIME_DIR=/run/user/1002 \ DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1002/bus \ systemctl --user show hermes-gateway.service \ --property=LoadState --property=TimeoutStopUSec LoadState=not-found TimeoutStopUSec=1min 30s # ← bogus default, since LoadState=not-found ``` Hermes blindly trusts the 90s value. Proposed fix Add LoadState to the same query and skip iterations where the unit isn't loaded: ```diff for flag in (["--user"], []): try: result = subprocess.run( - [\"systemctl\", *flag, \"show\", unit_name, \"--property=TimeoutStopUSec\"], - [\"systemctl\", *flag, \"show\", unit_name, - \"--property=LoadState\", - \"--property=TimeoutStopUSec\"], capture_output=True, text=True, timeout=2.0, ) except (FileNotFoundError, subprocess.TimeoutExpired, OSError): continue if result.returncode != 0: continue - Output: "TimeoutStopUSec=1min 30s" or "TimeoutStopUSec=90000000" - Skip if the unit isn't actually loaded in this manager — `systemctl - show` returns rc=0 and emits compiled-in defaults for nonexistent - units (notably TimeoutStopUSec=1min 30s), which would produce a - false-positive 'stale unit' warning when the gateway is deployed - via a system-level unit but the user manager is reachable. - load_state: Optional[str] = None - for line in result.stdout.splitlines(): - if line.startswith(\"LoadState=\"): - load_state = line.split(\"=\", 1)[1].strip() - break - if load_state and load_state != "loaded": - continue for line in result.stdout.splitlines(): if line.startswith("TimeoutStopUSec="): ... ``` Same selectivity, one extra property in the query, no other behaviour change. Bonus: while touching the file, the warning text on gateway/run.py:3904 could be updated to reference hermes gateway install --replace (the service sub-noun was removed in a recent CLI refactor — the current text gives invalid choice: 'service'). Happy to send a PR. Environment - hermes-agent 0.15.2 (v2026.5.29.2) - NixOS x86_64, gateway deployed via systemd.services.hermes-gateway (system unit, NOT via hermes gateway install) - gateway user: hermes (linger=true, system service with XDG_RUNTIME_DIR=/run/user/<uid> set for rootless podman) - No user-level hermes-gateway.service installed (verified via systemctl --user list-unit-files) — Reply to this email directly, view it on GitHub <#36755?email_source=notifications&email_token=BVAK7EOPLOPM5EA3UBR34MD45VTA7A5CNFSL4Z3JMQ5C6L3HNF2C22DVMIXUS43TOVSS6NBVGYZDOMRUGIZDHJTSMVQXG33OVJZXKYTTMNZGSYTFMSSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BVAK7EOOTUAAO7DSG7VAITD45VTA7AVCNFSM6AAAAACZVN3BV2VHI2DSMVQWIX3LMV43ASLTON2WKOZUGU3DENZSGQZDEMY> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/BVAK7ENIUPMJRBY6FCJEQF345VTA7A5CNFSL4Z3JMQ5C6L3HNF2C22DVMIXUS43TOVSS6NBVGYZDOMRUGIZDHJTSMVQXG33OVJZXKYTTMNZGSYTFMSSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android <https://github.com/notifications/mobile/android/BVAK7EKTWL3MIDXZ5OLXUF345VTA7A5CNFSL4Z3JMQ5C6L3HNF2C22DVMIXUS43TOVSS6NBVGYZDOMRUGIZDHJTSMVQXG33OVJZXKYTTMNZGSYTFMSSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[liuhao1024]

This is fixed by #36766 — fix(gateway): skip systemd timing check for non-loaded units.

[kristian-dott-42]

Independent confirmation: still present in v0.18.0 (2026.7.1) and on current main as of 2026-07-03 — gateway/shutdown_forensics.py is unchanged since the check landed in cede612, and #36766 hasn't merged.

A second, lower-friction trigger: root system unit + root linger. The original report needed XDG_RUNTIME_DIR set in the unit (for rootless podman). That isn't required: on a stock hermes gateway install --system --run-as-user root deployment the unit sets no XDG_RUNTIME_DIR at all, but hermes sets it itself at runtime — _ensure_user_systemd_env() in hermes_cli/gateway.py does os.environ["XDG_RUNTIME_DIR"] = "/run/user/<uid>" whenever that directory exists. With Linger=yes for root (user@0.service running), the user bus becomes reachable from inside the service, and the first loop iteration trusts the user manager's compiled-in default:

# from inside the service's environment
$ systemctl --user show hermes-gateway.service --property=LoadState --property=TimeoutStopUSec
LoadState=not-found
TimeoutStopUSec=1min 30s        # bogus default — this manager doesn't own the unit

$ systemctl show hermes-gateway.service --property=LoadState --property=TimeoutStopUSec
LoadState=loaded
TimeoutStopUSec=3min 30s        # the real value, never consulted

Result: every gateway start logs

WARNING gateway.run: Stale systemd unit detected: hermes-gateway.service has TimeoutStopSec=90s but drain_timeout=180s (expected >=210s). systemd may SIGKILL the gateway mid-drain. Run `hermes gateway install --force` to regenerate the unit, or shorten agent.restart_drain_timeout.

even though the real system unit already has TimeoutStopSec=210 — exactly the max(60, drain_timeout) + 30 the current template writes. So any root-run system install on a host with root linger hits this out of the box; no custom unit or env needed.

The remedy in the warning text doesn't clear it. We ran hermes gateway install --force (unit regenerated from the 0.18.0 template, TimeoutStopSec=210, daemon-reload, restart) — the warning fired again on the next start, because the unit was never the problem.

Fix verified in production. Applied the LoadState-skip fix (equivalent to #36766) locally on the affected host (Ubuntu 24.04, systemd 255, hermes 0.18.0, root system unit, linger enabled): the --user iteration is now skipped on LoadState=not-found, the system manager's 210s is used, no warning at startup, and the gateway stays active with adapters connecting normally. Would be great to see #36766 rebased and merged.

[rodion-m]

Linked fix PR: #57901

[itpartypattaya]

Still reproducible on current main d33becd87 (v2026.7.1 / v0.18.0): for a unit managed by the system manager, systemctl --user show answers rc=0 with defaults (TimeoutStopSec=90s, LoadState=not-found), and the check trusts it — false 'stale systemd unit timing' warning even though the system unit is configured correctly.

Workaround we run in production: skip any manager whose reply has LoadState=not-found before comparing timeouts — the false positive disappears while real mismatches are still caught. Happy to submit it as a PR if useful.

[alt-glitch]

This was generated by AI during triage.

Still reproduces on main as of today: gateway/shutdown_forensics.py still queries systemctl --user show ... --property=TimeoutStopUSec first with no LoadState guard, so a nonexistent user unit returns the compiled-in 90s default (rc=0) and fires the spurious stale-unit warning. The 'fixed by #36766' claim is premature — #36766 is still open/unmerged and its LoadState-check mechanism hasn't landed. Open fix cluster with this mechanism: #34734 (earliest), #36766, #37324. Diagnostic-only false warning (no functional impact).