Integration does not auto-recover from a transient HTTP 502 — stays unavailable indefinitely after one panel-side blip

[rancur]

Summary

The integration enters a permanent unavailable state after a single transient HTTP 502 (or other 5xx) from the panel's local API, and never auto-retries on its own. The panel itself remains reachable and healthy; only a manual config-entry reload (or HA core restart) recovers the integration. This is reminiscent of the long-closed #29 but is happening on the current 2.0.x line and at the connection-handler level rather than at entity state.

Environment

• SPAN Panel running spanos2 firmware (v3 series)
• Integration: span_panel v2.0.6
• Home Assistant: recent stable
• Network: panel is on the local LAN; sub-2ms ping; HTTP 200 from /api/v1/status once the integration is reloaded

Observed behaviour

• Integration was healthy for weeks, then a brief panel-side blip produced a single HTTP 502 from /api/v1/status.
• All entities transitioned to unavailable and stayed that way for ~4 days continuously.
• During the outage the panel itself remained reachable from the HA host — manual curl http://<panel>/api/v1/status returned 200 the whole time; ping was sub-2ms.
• Recovery was instantaneous after issuing a config-entry reload via POST /api/config/config_entries/entry/<entry_id>/reload — no re-auth, no zeroconf rediscovery, no HA restart. Every entity came back within seconds.

Expected behaviour

A transient 5xx or timeout should not be terminal. The integration should retry on a backoff cadence (or re-arm the coordinator) so it recovers without operator intervention once the panel is healthy again.

Why this is distinct from existing fixes

The v2.0.7 release notes call out unavailable fixes for the door, grid-islandable, and BESS-connected binary sensors — those address entity-state mapping when the panel reports UNKNOWN. That is the right fix for the entity layer, but it does not address the connection-handler layer where the coordinator gives up after a single 5xx. The pattern reported here looks like the same failure-mode family as #29 but on the 2.0.x rewrite — once the coordinator decides the panel is gone, nothing brings it back short of a reload.

Suggested direction

Two non-exclusive options:

1. Don't treat a single 5xx as fatal in the coordinator — the existing retry / catch-up logic in _async_fetch_with_retry should bubble back into the coordinator's normal polling cadence, not flip the integration into a permanent unavailable state. The next polling interval should re-attempt.
2. Add backoff-based reconnect — if N consecutive polls fail with 5xx / timeouts, schedule an exponential-backoff retry (with a reasonable cap, e.g. 5–10 min) using async_schedule_update_ha_state(force_refresh=True) or by re-invoking the equivalent of async_setup_entry for the live entry. Either path is strictly better than waiting for the operator to notice.

A 4-day silent outage from a single blip is the kind of thing operators only catch via the energy dashboard going flat, which is too late.

Workaround

For anyone hitting this in the meantime: a periodic config-entry reload via the HA REST API restores service immediately. It's not a fix, but it's reliable as a stopgap and doesn't disturb the panel.

Happy to help

If a maintainer can point at the right place in the 2.0.x coordinator / API client layer, happy to test a patch in production on the same panel that produced this report.

[cayossarian]

Thanks for the report. There was a recent firmware update that I'm wondering may have precipitated this issue due to CA private cert rotation at that time. You mentioned a 'single blip' with 502, do you have any logs from that first error? The fix is likely in the span-panel-api package with a watchdog but I'm especially interested in the exact errors from the around the time the panel went offline. The log may have been rotated but would presumably still exist in one of the other logs if you can't find it in the main log. There should be no secrets in your logs if you want to just upload them from that time period.

[rancur]

thanks for digging in. unfortunately the original 502 was ~4 days before i filed and my HA log retention only goes back ~2 days — the rotated logs from the blip window are gone (checked /config/home-assistant.log.*, supervisor ha core logs, and the log_backups/ dir, all empty/rotated past that window). so i don't have the exact stack from the first error to share right now.

happy to grab clean logs the next time it recurs. a few asks to make sure i capture what you need:

1. is there a debug logger config you'd like me to drop in configuration.yaml ahead of time? i'm running default: warning today — happy to add something like:

   logger:
     logs:
       custom_components.span_panel: debug
       span_panel_api: debug

   or whatever module names you'd prefer to see at debug.

2. anything specific you want from the panel side at the moment of failure (curl against /api/v1/status, cert inspection via openssl s_client, etc.) — i can wire a small watcher that triggers on the next coordinator error and snapshots all of that automatically.

re: CA cert rotation theory — that tracks with the timing for me; the firmware updated on its own a few days before the blip. if it'd help, i can also pull the current cert chain off the panel and confirm whether it's the rotated one now.

[cayossarian]

Ok no worries about the log, you would have seen a reconnect warning or SSL error and the after a few attempts it would stop. Any future failure logs would helpful.

I did not see this issue when the firmware updated so there was something unusual and the cert theory is one potential gap.

Debug logging would be much too noisy for production. My guess is you might not see the same issue for some time.

I will address the potential cert rotation issue and put a watchdog in place in the next release.

[cayossarian]

Fixed in v2.0.8, let us know if you have any more issues. If not, we can close this.

The underlying package will retry with an exponential backoff capped at 60s max, then retry every 60s with a full reconnect.