Storage API error 500 in Pulse on large Proxmox cluster with Ceph + iSCSI

[kevinpetit-tech]

I’m using Pulse to monitor two Proxmox clusters:

A 3-node cluster with Ceph storage → works fine, no issues.

An 8-node cluster with Ceph + iSCSI storage → nodes appear correctly in Pulse, but I get repeated errors when Pulse tries to access the storage API:

Error: Get "https://XX.XX.XX.XX:8006/api2/json/nodes/pve0X-n0X/storage": net/http: timeout awaiting response headers

On the host, I have this error in access.log :
::ffff:XX.XX.XX.XX - pulse-monitor@pam!pulse-XX-XX-XX-XX-1764660431 [02/12/2025:09:34:28 +0100] "GET /api2/json/nodes/pve0X-n0X/storage HTTP/1.1" 500 -

When I manually query the same API endpoint, it responds normally with valid data.
So it seems the API is reachable, but Pulse fails to retrieve the storage information on this specific cluster.

I’m looking for help understanding why Pulse times out on the storage API only on the 8-node cluster (Ceph + iSCSI).

[rcourtman]

The 500 error is coming from Proxmox itself, not Pulse - the access.log shows Proxmox returning HTTP 500 to the storage API request. The timeout error in Pulse is a consequence of waiting for a response that never comes because Proxmox is failing internally.

This pattern often occurs when:

1. iSCSI targets are slow or unresponsive - the storage API enumerates all storage backends and iSCSI can hang if targets aren't reachable
2. Ceph cluster load - on 8-node clusters with heavy Ceph usage, storage enumeration can take longer than Proxmox's internal timeout
3. Storage not mounted on that specific node - when querying /nodes/{node}/storage, Proxmox tries to check all storages configured for that node

To confirm, on the Proxmox host showing the 500:

# Check if the issue is specific to one storage type
pvesm status
# Try querying the API directly with a longer timeout
curl -sk "https://localhost:8006/api2/json/nodes/$(hostname)/storage" \
  -H "Authorization: PVEAPIToken=..." --max-time 120

If the direct curl also times out or returns 500, the issue is in Proxmox/storage configuration, not Pulse.

As a workaround, you can exclude problematic storage from Pulse monitoring by going to Settings → Nodes → [Node] → Storage and deselecting specific storage IDs that cause issues. Pulse will skip polling those.

Could you share the output of pvesm status from one of the affected nodes?