Cluster showing degraded - but not clear why?

[emsofo]

Describe the bug
Since latest version - my cluster is showing degraded status - but I can't figure out why?

To Reproduce
Steps to reproduce the behavior:

1. Go to Settings
2. Look at cluster and see Yellow status of degraded

Expected behavior
Green status or more detail as to why it's degraded

Environment:

• Pulse Version: 4.26.4
• Installation Type: Proxmox LXC

Additional context

💡 Tip: If you're experiencing connection issues, API errors, or missing data, you can attach diagnostics from Settings → Diagnostics tab → Export for GitHub (sanitized version)

[rcourtman]

"Degraded" in Pulse means the monitoring server can reach some of the cluster endpoints listed in Settings but cannot reach all of them. The cluster itself may be healthy from Proxmox's perspective; the yellow badge is specifically about Pulse's network connectivity to each node URL. A change is queued for the next release so the Settings page shows, per endpoint, whether Proxmox reports it online and whether Pulse can contact it, including the last connection error.

[luckman212]

Is there a way to adjust the "Client.Timeout or "context deadline"?

I had a host that was taking a loooong time (>30s) to return the data from the /storage API. But manual test showed it eventually worked. Would like to extend the default timeout in Pulse to e.g. 60s.

[rcourtman]

The CONNECTION_TIMEOUT environment variable exists (default 45s, max 300s), but storage API calls use a separate hardcoded 30-second timeout that doesn't respect this setting. That's why increasing CONNECTION_TIMEOUT wouldn't help.

Before looking at increasing the timeout, it would help to understand the situation better.

A few questions:

1. What type of storage is this? NFS, iSCSI, local ZFS, Ceph?
2. Is this one specific storage or all storages on that node?
3. Has this storage always been slow, or did it recently become slow?
4. Is the >30s response time consistent, or does it vary?

Related to issue #418, where a similar timeout error was caused by unavailable storage mounts. In that case, disabling the unavailable storage in Proxmox resolved the issue. Worth checking if any storage is in a degraded or partially available state.

If the storage genuinely needs >30s to respond under normal operation, making that timeout configurable is reasonable. However, storage queries typically complete in under 5 seconds, so this might indicate an underlying issue worth investigating first.

[luckman212]

@rcourtman you're 100% right, there's something wrong with my storage that's causing the storage API endpoint to take this long. It's related to iSCSI + IPv6 I think (I only recently enabled IPv6 on my NAS)

Trying to debug that.

[luckman212]

@rcourtman I fixed my iSCSI storage that was causing timeouts. Everything is "green" again 👍

[luckman212]

@rcourtman So after more testing, it seems there is actually a bug somewhere.

I corrected my storage issue and direct calls to the API are working fine and returning results in 2-4 seconds for all nodes (tested manually).

But, Pulese is still showing 2 nodes "degraded" with Timeout warnings. One node also has an error VM 303 is not running. These are old/stale warnings, not sure where it is reading them from or caching them but it seems to be a cache issue.

Stopping the Pulse container and restarting it (temporarily) fixes the issue...

I'm running 4.33.1 now

related: #754

screenshot that shows what I am talking about...

[rcourtman]

Fixed in commit 6db4ee7.

The issue was that error messages stored in the cluster client's lastError map were never cleared after successful operations. This caused stale timeout warnings to persist in the UI even after the underlying issue was resolved.

The fix clears these cached errors whenever an endpoint successfully completes a connectivity test or API operation.

This will be in the next release.

[luckman212]

Just adding:

I'm on 4.34.1 now and my nodes are all still showing yellow "Degraded" but no indication of why. Calls to /storage API are fine (tested) and no issues I can see.

[rcourtman]

@luckman212 Thanks for the follow-up. You're right - the fix in 6db4ee7 was incomplete.

The commit cleared stale error messages (lastError map) but didn't mark endpoints as healthy again in the nodeHealth map. Once a cluster endpoint was marked unhealthy, it only recovered on restart or when ALL endpoints became unhealthy simultaneously.

Fixed in commit 4e05285. This will be in the next release.

Workaround until then: Restart Pulse to reset the health tracking.