Skip to content

Stale-feed watchdog treats 30s of user-hub silence as dead and force-reconnects a healthy suite #129

Description

@TexasCoding

Summary

HealthMonitoringMixin._stale_feed_watchdog_loop uses one threshold (stale_feed_seconds = 30) for both hubs. The user hub only emits order/position/account events. While flat (or overnight with no fills), 30s of user-hub silence is normal.

The watchdog then logs User feed stale; forcing reconnect even when health_score is 100, calls force_health_reconnect()disconnect() + connect(), and can crash the process:

RuntimeError: Cannot connect while not disconnected

Related to the watchdog added for #97 (market tick freeze). Market-hub silence is a real signal. User-hub silence is not.

Evidence (project-x-py 4.1.1, MNQ+MES, no position)

User feed stale; forcing reconnect  hub=user seconds_since_message=30.00 threshold_seconds=30.0
Forcing reconnection due to poor health: 100.0
Health-based reconnection failed
...
RuntimeError: Cannot connect while not disconnected

Process then died (SIGTERM) ~8s later as the suite disconnected.

Suggested fix

  • Do not treat user hub silence as stale, or use a much larger timeout (minutes, not 30s).
  • Keep market hub stale detection (quotes/depth/ticks).
  • force_health_reconnect should not call connect() while the transport is still connecting / not fully disconnected (Cannot connect while not disconnected).
  • Do not force-reconnect when health_score is 100.

Workaround

After TradingSuite.create(), wrap realtime._hub_silence_age so hub == "user" returns None.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions