context deadline exceeded

[kieric82]

Describe the bug
Datacenter with 16 nodes connected, but i get the error: 'Context deadline Exceeded' and the logs goes like crazy non-stop.

To Reproduce
Not sure how to reproduce, may be environment specific. It was working fine up to the version 4.30. All the following ones have been tested but no luck.

Environment:

• Pulse Version: 4.30 works, the ones after no.
• Installation Type: Docker running as non-root

Additional context

pulse-diagnostics-sanitized-2025-11-25.json

[luckman212]

Related to #659

I am also seeing this, even after clearing the issue, Pulse shows the node(s) as degraded. It's like the error message is being cached and never cleared, even though querying the node's API directly works fine.

[rcourtman]

This should also be addressed by commit 6db4ee7 (related to #659).

The stale error caching issue could cause "context deadline exceeded" messages to persist in the UI even after connectivity is restored. The fix clears these cached errors whenever an endpoint successfully completes a connectivity test or API operation.

Let us know if you continue to see issues after updating to a version with this fix.

[luckman212]

@rcourtman Sorry to keep bugging on this. But I'm running 4.35.0 and all my nodes are still showing yellow "Degraded" status.

I'm also seeing the memory bar graph bug (#788) not sure if that is the cause?

[rcourtman]

Looking at your screenshot - the nodes (pve01, pve02, pve03) all show green status dots indicating they're Online. The yellow/orange color in the Memory column is intentional: it indicates balloon memory in use, not a degraded status.

The StackedMemoryBar component shows:

• Green = Active memory
• Yellow = Balloon memory (reclaimed by hypervisor)
• Purple stripe = Swap usage

If your VMs use memory ballooning (common in Proxmox), you'll see yellow in the memory bars - this is expected behavior.

The "Degraded" entry visible in your guest list appears to be the name of a VM, not a status indicator. The refresh/question-mark icon beside it suggests that particular VM may have an issue.

The memory bar width fix (commit 32ffc10) is in v4.35.0. If bars still look incorrect, could you clarify what specifically looks wrong? The screenshot shows bars displaying percentages correctly (12%, 23%, 50%, 30% etc).

[luckman212]

@rcourtman But the yellow dots here seems to indicate an error condition, right?

What I am saying is, there is no actual error condition here. As an example, take the memory bar for the pbx VM:

It is showing a value of 23%. Could you tell me why the bar is "full" then? And why it is painted in Yellow?

Seems there's a bug here.

[rcourtman]

You're right about two things:

1. Memory bar width bug: The bar showing 23% but filling 100% is a rendering bug. Fixed in commit 32ffc10, but that went in after v4.35.0 was tagged. Will be in v4.36.0.

2. Yellow status dots: Your second screenshot does show yellow/amber dots (different from the green ones in the first screenshot). Yellow means the node's connectionHealth is in a degraded state - not the same as the memory bar color.

The yellow dots suggest something is causing intermittent degraded connection status. When you see the yellow dots, could you check Settings → Nodes to see what status is reported? Or export diagnostics at that moment - the connectionHealth values would help identify what's triggering it.

[luckman212]

Ah, here is the screenshot from Settings → Nodes. But, as to WHY it shows that context deadline exceeded (Client.Timeout exceeded while awaiting headers) error- I am at a loss. The node itself reports no issues, and hitting its API returns results quickly.

[rcourtman]

Thanks for the Settings → Nodes screenshot. That confirms the issue - pve01 and pve02 show persistent timeout errors while pve03 works fine.

Since you can hit the APIs directly from your workstation, the issue is specific to network connectivity from where Pulse runs (Docker container) to those nodes.

Could you check from the Pulse container:

# Test direct connectivity to each node
docker exec -it <pulse-container> sh -c 'wget -qO- --timeout=5 https://192.168.20.51:8006/api2/json/version 2>&1'
docker exec -it <pulse-container> sh -c 'wget -qO- --timeout=5 https://192.168.20.52:8006/api2/json/version 2>&1'
docker exec -it <pulse-container> sh -c 'wget -qO- --timeout=5 https://192.168.20.53:8006/api2/json/version 2>&1'

Also check if there's a routing or DNS issue:

docker exec -it <pulse-container> sh -c 'ping -c 2 192.168.20.51'
docker exec -it <pulse-container> sh -c 'ping -c 2 192.168.20.52'

If those work, the issue might be TLS handshake delays. Try with --no-check-certificate to rule out cert validation slowness:

docker exec -it <pulse-container> sh -c 'wget -qO- --no-check-certificate --timeout=5 https://192.168.20.51:8006/api2/json/version 2>&1'

This will help identify whether it's a Docker networking issue, DNS issue, or TLS issue.

[luckman212]

Here's some output from those commands run directly from the Pulse container:

/app # wget -qO- --timeout=5 https://192.168.20.51:8006/api2/json/version
28AB1CFE3F7F0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:2103:
ssl_client: SSL_connect
wget: error getting response: Connection reset by peer

/app # wget -qO- --timeout=5 https://192.168.20.52:8006/api2/json/version
284B4DDD957F0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:2103:
ssl_client: SSL_connect
wget: error getting response: Connection reset by peer

/app # wget -qO- --timeout=5 https://192.168.20.53:8006/api2/json/version
286B029E0E7F0000:error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:2103:
ssl_client: SSL_connect
wget: error getting response: Connection reset by peer

/app # ping -c 2 192.168.20.51
PING 192.168.20.51 (192.168.20.51): 56 data bytes
64 bytes from 192.168.20.51: seq=0 ttl=62 time=0.823 ms
64 bytes from 192.168.20.51: seq=1 ttl=62 time=0.964 ms

--- 192.168.20.51 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.823/0.893/0.964 ms

/app # ping -c 2 192.168.20.52
PING 192.168.20.52 (192.168.20.52): 56 data bytes
64 bytes from 192.168.20.52: seq=0 ttl=62 time=0.574 ms
64 bytes from 192.168.20.52: seq=1 ttl=62 time=0.496 ms

--- 192.168.20.52 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.496/0.535/0.574 ms

/app # ping -c 2 192.168.20.53
PING 192.168.20.53 (192.168.20.53): 56 data bytes
64 bytes from 192.168.20.53: seq=0 ttl=62 time=0.506 ms
64 bytes from 192.168.20.53: seq=1 ttl=62 time=0.549 ms

--- 192.168.20.53 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.506/0.527/0.549 ms

/app # wget -qO- --no-check-certificate --timeout=5 https://192.168.20.51:8006/api2/json/version
wget: server returned error: HTTP/1.1 401 No ticket

/app # wget -qO- --no-check-certificate --timeout=5 https://192.168.20.52:8006/api2/json/version
wget: server returned error: HTTP/1.1 401 No ticket

/app # wget -qO- --no-check-certificate --timeout=5 https://192.168.20.53:8006/api2/json/version
wget: server returned error: HTTP/1.1 401 No ticket

[rcourtman]

Your wget tests prove network connectivity is fine - the 401 No ticket responses confirm HTTP works from the container to all nodes.

Found the bug: the recovery mechanism for unhealthy nodes only triggered when ALL nodes were down. Since pve03 was healthy, pve01 and pve02 stayed stuck in degraded state forever because the recovery path was never invoked.

Fixed in commit e3b4012 - recovery now runs whenever any unhealthy nodes exist, not just when the entire cluster is down.

This will be in the next release. You can test now by pulling the latest main and rebuilding, or wait for v4.36.0.

[luckman212]

Awesome! Looking forward to testing this. Thanks for helping to debug it.