Server disappears from Servers/Dashboard list when offline, instead of showing "offline" status

[snet]

Hello everybody,

according to the docs (docs/servers.md), a server should be marked "offline" and remain visible with an offline badge when its xySat WebSocket disconnects:

"A server is online while its xySat WebSocket is connected. If the socket drops, the server is immediately marked offline. The UI updates in real time."

"Status: Online/offline badge, label/hostname, IP, OS/arch, CPU details, memory, virtualization, agent version, uptime, and groups"

In practice, when I shut down a server (a FreePBX VPS I decommissioned), it did not show up as "offline" — it disappeared entirely from the Servers list / Dashboard. Historical data (snapshots, monitor history) is still there and queryable, so the server record itself is clearly not deleted from persistent storage.

Expected behavior
Per the documentation, I expected the server to stay listed with an "offline" badge, so it's immediately visible that a host is down.

Actual behavior
The server is just gone from the list, as if it never existed. Nothing indicates that a previously known server is currently unreachable — you'd only notice by trying to recall its name.

My hypothesis
Looking at the API docs, get_active_servers is described as: "Fetch all active servers (connected to the current conductor server)." If the main Servers/Dashboard view is populated from this "active" in-memory registry (rather than the persisted server records used by get_server / search_servers), that would explain why offline servers vanish from the UI instead of showing an offline badge as documented.

Why this matters
For fleet monitoring this is a real operational risk: a server going down silently drops off the dashboard instead of raising visibility. I know there's a "notify on server offline" system hook recipe in the changelog, which is a good workaround, but it shouldn't be the only way to notice — the primary UI list should reflect what the docs describe.

Questions
Is this the intended behavior (dashboard = active/connected servers only), and is the docs wording just outdated/misleading?
If so, is there a way to toggle the Servers view to include known-offline servers (with a badge), rather than only currently connected ones?
If this is unintended, happy to help narrow it down further — I can share xyOps deployment details, etc.
Conductor v1.0.92 - xySat Version v1.0.41

[jhuckaby]

Hi @snet, thanks for the detailed report. You are correct that the documentation was unclear here.

This is currently intended behavior, but there is an important distinction between the main Servers list and Server History. The main list contains all currently connected servers, plus a small in-memory cache of recently disconnected servers. When a server disconnects, it is immediately marked offline and temporarily remains visible with the offline styling.

That temporary cache currently holds the 10 most recently disconnected servers and is cleared at midnight in the conductor server's local timezone. There is also a maintenance check that removes entries older than 24 hours. Consequently, an offline server can disappear from the main list sooner than 24 hours, either at midnight or when newer disconnected servers displace it.

Nothing is deleted from persistent storage when this happens. Click Search History at the bottom of the Servers page to view historical servers, including those no longer present in the main list. There is currently no toggle to merge the entire server history into the main list.

After reviewing this, I agree that clearing the cache at midnight is not the best behavior. In a future release, I plan to make the 24-hour age limit the primary expiration mechanism, and make both the retention time and the 10-server cache limit configurable.

I'll also clarify the current behavior in the server docs. Thanks again for bringing this to our attention!

[jhuckaby]

xyOps v1.0.93 was just released, which has two new configuration properties you can adjust for this:

---

server_cache_max_time

This is the maximum amount of time servers are kept in the offline memory cache (default: 24 hours).

Note that all offline servers are available on the "Server History" page (subject to database retention settings). The offline cache just keeps them in memory for a short period after they go offline.

server_cache_max_count

This is the maximum number of server entries allowed in the offline cache, before older ones are removed (default: 100).

Note that all offline servers are available on the "Server History" page (subject to database retention settings). The offline cache just keeps the most recent ones in memory for a short period.

---

Hope this helps!

[snet]

That's a fast turnaround, thank you!
The new config options address this cleanly — glad the 24h TTL isn't tied to midnight anymore.
Appreciate you looking into it.