Summary
When a new gateway starts (even with --replace) and a different already-running gateway holds the Telegram bot token, the new gateway detects the conflict (Telegram bot token already in use (PID X)) but does not terminate the holder. It retries forever, blocking until a human manually kills the stale process.
Root cause
There are two independent duplicate-gateway guards that are not connected:
- PID-file guard —
gateway/run.py:20689 (start_gateway), which calls get_running_pid() + terminate_pid(). This only kills a target it finds in the PID-file/lock record for HERMES_HOME. If that record is stale/cleared at startup, --replace kills nothing.
- Scoped token lock —
gateway/platforms/base.py:2760 (_acquire_platform_lock), which uses acquire_scoped_lock (gateway/status.py:986). Its identity is the bot token, so on conflict it knows the live holder PID (existing['pid']), but only logs a fatal + retryable=True (base.py:2777) and loops.
There is no recovery path from guard #2 back to a terminate_pid() of the named holder. The conflict handler treats "token held by another live gateway" as a permanent retry condition rather than a takeover the new gateway should win.
Why --replace missed it
In the observed case, the new gateway (PID 67230, build 0.18.2, macOS) started with --replace but the PID-file record did not point at the real token holder (stale June-7 gateway PID 83560). So --replace found nothing to kill; 67230 launched and only hit the token conflict ~minutes later at platform-connect time. The holder was then only removed by a manual SIGKILL 83560, after which 67230 reconnected successfully (log: ✓ telegram reconnected successfully).
Platform-specific aggravator
On macOS/Windows, _get_process_start_time() returns None (gateway/status.py:1259), so the PID-reuse guard in get_running_pid() falls back to PID-only equality. This makes stale-PID-file misreads more likely on exactly the platforms without /proc, weakening guard #1 further.
Expected behavior
When the scoped token lock reports a live holder PID that is a genuine gateway process, the new gateway should terminate that holder (mirroring what --replace does at startup) rather than retry indefinitely. terminate_pid() (gateway/status.py:98) already exists and supports force-kill.
Suggested fix
In _acquire_platform_lock (base.py:2777), on a non-stale conflict where existing['pid'] is a live gateway (verify via _looks_like_gateway_process), call terminate_pid(existing_pid, force=True), then re-attempt acquire_scoped_lock once the holder exits. Guard against flapping (same TTL/marker discipline as the --replace takeover marker in status.py:1165).
Repro
- Start a gateway with a Telegram token (old process, PID A).
- Corrupt/clear the PID-file record so
get_running_pid() returns None (or start A such that its PID isn't in the new process's PID-file view).
- Start a second gateway for the same token with
--replace.
- Observe: new gateway logs
Telegram bot token already in use (PID A) and never connects until A is killed externally.
Environment
- Hermes Agent 0.18.2
- macOS (26.4)
- python-telegram-bot 22.6
Summary
When a new gateway starts (even with
--replace) and a different already-running gateway holds the Telegram bot token, the new gateway detects the conflict (Telegram bot token already in use (PID X)) but does not terminate the holder. It retries forever, blocking until a human manually kills the stale process.Root cause
There are two independent duplicate-gateway guards that are not connected:
gateway/run.py:20689(start_gateway), which callsget_running_pid()+terminate_pid(). This only kills a target it finds in the PID-file/lock record forHERMES_HOME. If that record is stale/cleared at startup,--replacekills nothing.gateway/platforms/base.py:2760(_acquire_platform_lock), which usesacquire_scoped_lock(gateway/status.py:986). Itsidentityis the bot token, so on conflict it knows the live holder PID (existing['pid']), but only logs a fatal +retryable=True(base.py:2777) and loops.There is no recovery path from guard #2 back to a
terminate_pid()of the named holder. The conflict handler treats "token held by another live gateway" as a permanent retry condition rather than a takeover the new gateway should win.Why
--replacemissed itIn the observed case, the new gateway (PID 67230, build 0.18.2, macOS) started with
--replacebut the PID-file record did not point at the real token holder (stale June-7 gateway PID 83560). So--replacefound nothing to kill; 67230 launched and only hit the token conflict ~minutes later at platform-connect time. The holder was then only removed by a manualSIGKILL 83560, after which 67230 reconnected successfully (log:✓ telegram reconnected successfully).Platform-specific aggravator
On macOS/Windows,
_get_process_start_time()returnsNone(gateway/status.py:1259), so the PID-reuse guard inget_running_pid()falls back to PID-only equality. This makes stale-PID-file misreads more likely on exactly the platforms without/proc, weakening guard #1 further.Expected behavior
When the scoped token lock reports a live holder PID that is a genuine gateway process, the new gateway should terminate that holder (mirroring what
--replacedoes at startup) rather than retry indefinitely.terminate_pid()(gateway/status.py:98) already exists and supports force-kill.Suggested fix
In
_acquire_platform_lock(base.py:2777), on a non-stale conflict whereexisting['pid']is a live gateway (verify via_looks_like_gateway_process), callterminate_pid(existing_pid, force=True), then re-attemptacquire_scoped_lockonce the holder exits. Guard against flapping (same TTL/marker discipline as the--replacetakeover marker instatus.py:1165).Repro
get_running_pid()returnsNone(or start A such that its PID isn't in the new process's PID-file view).--replace.Telegram bot token already in use (PID A)and never connects until A is killed externally.Environment