xsconsole never reconnects to xapi after a single connection timeout (console stuck on "Pool master is unreachable.")

[olivierlambert]

Summary

Once a long-lived xsconsole process experiences one XenAPI call that hits the 15-second socket timeout, it never talks to xapi again for the rest of its life. The physical-console instance (spawned by agetty/systemd) then shows Pool master is unreachable. forever — even after xapi has fully recovered — while newly launched xsconsole instances on the same host work fine. The only workaround is killing the stuck process.

Originally reported by a user after an XCP-ng HA pool-master failover test, where xapi on the surviving host was unresponsive for a while during the master transition: https://xcp-ng.org/forum/post/106689 — the console xsconsole stayed "unable to reach the pool master" for an hour (until restarted), while an SSH-launched xsconsole displayed pool status correctly.

Root cause

Auth.__init__ sets a process-wide 15s socket timeout and initialises a masterConnectionBroken flag:

xsconsole/XSConsoleAuth.py

Lines 45 to 46 in 5fa04f1

	self.masterConnectionBroken = False
	socket.setdefaulttimeout(15)

Auth.OpenSession() sets that flag on any socket.timeout:

xsconsole/XSConsoleAuth.py

Lines 172 to 175 in 5fa04f1

	except socket.timeout:
	session = None
	self.masterConnectionBroken = True
	self.error = 'The master connection has timed out.'

…and is gated on it, so once the flag is set no further connection attempt is ever made:

xsconsole/XSConsoleAuth.py

Line 164 in 5fa04f1

if not self.masterConnectionBroken:

The flag is never reset anywhere in the codebase. It is a one-way latch: a single 15s hang of xapi (HA master takeover, xapi restart under load, toolstack restart on a busy host…) permanently disables the running xsconsole's XenAPI access.

Two aggravating factors:

• The "XenAPI connection has timed out" dialogue is only displayed by a ready handler at startup (plugins-base/XSFeatureXAPIConnection.py#L40-L44), so an instance that latches mid-life gives no hint of what happened; nothing is logged either.
• Data refreshes are keypress-driven (the periodic 4s Data.Update() in XSConsoleTerm.MainLoop only runs while the host has no management IP), so the latch typically triggers exactly when a user is interacting with the console during the outage — and then blames the pool master.

Reproduction (deterministic, single host, no HA needed)

On XCP-ng 8.3 (xsconsole 11.0.9.1, code unchanged on current master):

1. Leave the physical-console xsconsole running and displaying data (interact with it once so it isn't in inactivity sleep).
2. Freeze xapi so XenAPI calls hang rather than fail fast (a hang is required — connection-refused takes the generic Exception path which does not latch):

   kill -STOP $(pgrep -x xapi)
   

3. While frozen, press F5 (Refresh) on the console twice, ~20s apart. The first press blocks ~30s (two 15s timeouts: the existing session's call, then the reconnect attempt inside OpenSession(), which latches the flag). The second press returns instantly — OpenSession() no longer even tries.
4. Resume xapi after ~60s: kill -CONT $(pgrep -x xapi) and verify it is healthy (xe host-list works, a freshly launched xsconsole shows correct data).
5. The console xsconsole displays Pool master is unreachable. and never recovers, no matter how long you wait or how often you refresh. Only killing/restarting it helps.

Verified with temporary logging added at the latch and at the OpenSession() gate: after step 4 the process skips the connection attempt on every update cycle (~every 4s), indefinitely, while XenAPI.xapi_local() + login from a new process on the same host succeeds immediately.

Suggested fix

Turn the latch into a cooldown: after a timeout, skip connection attempts for N seconds (so an actually-down xapi doesn't block the UI for 15s on every 4s update cycle — presumably the reason the latch exists), then retry, and clear the flag on the next successful login. Log the transitions. PR follows.

With that change, in the repro above the console recovers by itself within ~a minute of xapi returning, and survives a real HA master failover.