Skip to content

Gateway doesn't auto-resolve Telegram token conflict at platform connect — waits for manual SIGKILL #65176

Description

@AllardQuek

Summary

When a new gateway starts (even with --replace) and a different already-running gateway holds the Telegram bot token, the new gateway detects the conflict (Telegram bot token already in use (PID X)) but does not terminate the holder. It retries forever, blocking until a human manually kills the stale process.

Root cause

There are two independent duplicate-gateway guards that are not connected:

  1. PID-file guardgateway/run.py:20689 (start_gateway), which calls get_running_pid() + terminate_pid(). This only kills a target it finds in the PID-file/lock record for HERMES_HOME. If that record is stale/cleared at startup, --replace kills nothing.
  2. Scoped token lockgateway/platforms/base.py:2760 (_acquire_platform_lock), which uses acquire_scoped_lock (gateway/status.py:986). Its identity is the bot token, so on conflict it knows the live holder PID (existing['pid']), but only logs a fatal + retryable=True (base.py:2777) and loops.

There is no recovery path from guard #2 back to a terminate_pid() of the named holder. The conflict handler treats "token held by another live gateway" as a permanent retry condition rather than a takeover the new gateway should win.

Why --replace missed it

In the observed case, the new gateway (PID 67230, build 0.18.2, macOS) started with --replace but the PID-file record did not point at the real token holder (stale June-7 gateway PID 83560). So --replace found nothing to kill; 67230 launched and only hit the token conflict ~minutes later at platform-connect time. The holder was then only removed by a manual SIGKILL 83560, after which 67230 reconnected successfully (log: ✓ telegram reconnected successfully).

Platform-specific aggravator

On macOS/Windows, _get_process_start_time() returns None (gateway/status.py:1259), so the PID-reuse guard in get_running_pid() falls back to PID-only equality. This makes stale-PID-file misreads more likely on exactly the platforms without /proc, weakening guard #1 further.

Expected behavior

When the scoped token lock reports a live holder PID that is a genuine gateway process, the new gateway should terminate that holder (mirroring what --replace does at startup) rather than retry indefinitely. terminate_pid() (gateway/status.py:98) already exists and supports force-kill.

Suggested fix

In _acquire_platform_lock (base.py:2777), on a non-stale conflict where existing['pid'] is a live gateway (verify via _looks_like_gateway_process), call terminate_pid(existing_pid, force=True), then re-attempt acquire_scoped_lock once the holder exits. Guard against flapping (same TTL/marker discipline as the --replace takeover marker in status.py:1165).

Repro

  1. Start a gateway with a Telegram token (old process, PID A).
  2. Corrupt/clear the PID-file record so get_running_pid() returns None (or start A such that its PID isn't in the new process's PID-file view).
  3. Start a second gateway for the same token with --replace.
  4. Observe: new gateway logs Telegram bot token already in use (PID A) and never connects until A is killed externally.

Environment

  • Hermes Agent 0.18.2
  • macOS (26.4)
  • python-telegram-bot 22.6

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium — degraded but workaround existscomp/gatewayGateway runner, session dispatch, deliveryplatform/telegramTelegram bot adaptersweeper:risk-message-deliverySweeper risk: may drop, duplicate, misroute, or suppress messagestype/bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions