3 Proxmox 8 nodes in a cluster, only 1 node shows temperatures (v. 4.32.7)

[Fabbio2]

Dears,
as another user highlighted on this board, on a clean Proxmox 8 cluster made of 3 nodes, only temperatures coming from node "A" where the Pulse LXC resides (created automatically via script run on Proxmox host) are visualized in the GUI.
I spent hours trying to make it work from the other nodes B and C but no joy (AI tools I've used proved useless in this case!!), even tried the HTTP version of the script but nothing.
All other metrics are properly seen in the GUI, from the 3 nodes.

Adding: "sensors" command works in all nodes. There might be some remnants of the previous HTTP version script in the nodes B and C.

GUI diagnostic says:

Temperature proxy
Proxy socketDetected
DaemonResponding
Socket path: /mnt/pulse-proxy/pulse-sensor-proxy.sock
Permissions: 0666
Owner: nobody / nogroup
Proxy version: v4.32.0
SSH directory: /var/lib/pulse-sensor-proxy/ssh
(with DETECTED and RESPONDING green icons)

I'm therefore lost. Any good soul that can support?

Thanks in advance.

[rcourtman]

The diagnostic shows your proxy version is v4.32.0 but Pulse is v4.32.7. There were critical cluster temperature fixes after v4.32.0 that explain why only the local node works.

Re-run the installer on the Proxmox host where Pulse runs to update the proxy:

curl -fsSL https://github.com/rcourtman/Pulse/releases/latest/download/install-sensor-proxy.sh | \
  sudo bash -s -- --ctid <YOUR_CONTAINER_ID> --pulse-server http://<PULSE_IP>:7655

Replace <YOUR_CONTAINER_ID> with your Pulse LXC container ID and <PULSE_IP> with the address Pulse is reachable at.

After that, check the proxy logs for SSH issues:

journalctl -u pulse-sensor-proxy -f

You should see entries for each node. If nodes B and C show SSH errors, the installer needs to distribute SSH keys to those nodes. Re-running the installer on a clustered Proxmox host handles that automatically via pvecm.

[Fabbio2]

Thanks, but unfortunately I got "The requested URL returned error: 404"
It seems https://github.com/rcourtman/Pulse/releases/latest/download/install-sensor-proxy.sh is NOT available.

Looking forward to hearing from you soon.

[ampls]

Correct URL is: https://raw.githubusercontent.com/rcourtman/Pulse/main/scripts/install-sensor-proxy.sh

Documentation LINK

[rcourtman]

Sorry about that. The install-sensor-proxy.sh script was added to the release assets as part of v4.33.0, but that version has not been published yet. The current latest release (v4.32.7) does not include it.

Ironically, you have hit the exact scenario that prompted this change. Previously, install scripts were pulled directly from the main branch, which meant users could end up running code that referenced features or files that did not exist in their version. By switching to release-based assets, everything stays in sync with the actual released version.

v4.33.0 should be out in the next couple of hours. Hold tight.

[Fabbio2]

Thanks @rcourtman and @ampls for your kind and quick support.
Pulse is really fantastic and @rcourtman kudos for that !

So FYI:

1. I've entered the script command above, BEFORE applying any updates and this fixed temperatures on server B (server A, the LXC one, was already OK), still one to go.

2. Then I've applied the v4.33.0 via the GUI (banner/button in the upper part of the GUI) and rebooted the LXC: now temperatures on nodes A and B are displayed "alternately" (as in node A, then node B, then a couple of seconds together, but mostly either A or B).

As for the node C, still MIA ;-)

Logs on node A say
journalctl -u pulse-sensor-proxy -f

" Failed to get temperatures error="failed to fetch temperatures: exit status 127 (stderr: bash: line 1: /opt/pulse/sensor-proxy/bin/pulse-sensor-wrapper.sh: No such file or directory)" corr_id=e0345ac0-ac0d-4a74-aee2-36886dbf99e4 method=get_temperature node=192.168.xx.yy pid=260962 uid=100999
Nov 26 17:58:55 NodeA pulse-sensor-proxy[3928735]: 5:58PM WRN Handler failed error="failed to get temperatures: failed to fetch temperatures: exit status 127 (stderr: bash: line 1: /opt/pulse/sensor-proxy/bin/pulse-sensor-wrapper.sh: No such file or directory)" corr_id=e0345ac0-ac0d-4a74-aee2-36886dbf99e4 method=get_temperature pid=260962 uid=100999
Nov 26 17:59:05 NodeA pulse-sensor-proxy[3928735]: 5:59PM WRN Failed to load host key from Proxmox; falling back to ssh-keyscan error="open /etc/pve/priv/known_hosts: permission denied" node=192.168.XX.YY ".

(where 192.168.XX.YY is the node C, still temperatures missing)

Any suggestions?
Cheers
Fabbio

[rcourtman]

The node C error is clear: /opt/pulse/sensor-proxy/bin/pulse-sensor-wrapper.sh doesn't exist on that node. The installer didn't deploy the wrapper there.

The permission denied warning for /etc/pve/priv/known_hosts is expected in LXC containers - it falls back to ssh-keyscan, so that's not the blocker.

Re-run the installer from the Proxmox host (not inside the LXC):

curl -fsSL https://github.com/rcourtman/Pulse/releases/latest/download/install-sensor-proxy.sh | \
  sudo bash -s -- --ctid <YOUR_CONTAINER_ID> --pulse-server http://<PULSE_IP>:7655

Watch the output for any errors related to node C. If it fails again, paste the relevant output.

For the alternating A/B display - is that still happening? If so, can you check browser devtools network tab to see if the API response contains temps for both nodes simultaneously, or if only one appears per request?

[Fabbio2]

Hi rcourtman, thanks for your reply.
I deployed the script again but nothing changed.

This is still the proxy errors from NodeA:

Nov 27 08:51:59 NodeA pulse-sensor-proxy[1755688]: 8:51AM WRN Failed to load host key from Proxmox; falling back to ssh-keyscan error="open /etc/pve/priv/known_hosts: permission denied" node=192.168.xx.4
Nov 27 08:51:59 NodeA pulse-sensor-proxy[1755688]: 8:51AM WRN Failed to load host key from Proxmox; falling back to ssh-keyscan error="open /etc/pve/priv/known_hosts: permission denied" node=192.168.xx13
Nov 27 08:51:59 NodeA pulse-sensor-proxy[1755688]: 8:51AM WRN Rate limit exceeded pid=260962 reason=peer_concurrency uid=100999
Nov 27 08:51:59 NodeA pulse-sensor-proxy[1755688]: 8:51AM INF Temperature data fetched successfully corr_id=6d463967-844f-4a54-92fb-c4067fbc484e method=get_temperature node=192.168.xx.4 pid=260962 uid=100999
Nov 27 08:51:59 NodeA pulse-sensor-proxy[1755688]: 8:51AM INF Request completed corr_id=6d463967-844f-4a54-92fb-c4067fbc484e method=get_temperature pid=260962 uid=100999
Nov 27 08:51:59 NodeA pulse-sensor-proxy[1755688]: 8:51AM WRN Failed to get temperatures error="failed to fetch temperatures: exit status 127 (stderr: bash: line 1: /opt/pulse/sensor-proxy/bin/pulse-sensor-wrapper.sh: No such file or directory)" corr_id=55565967-4915-46c3-b1e8-d6eca295f15b method=get_temperature node=192.168.xx.13 pid=260962 uid=100999
Nov 27 08:51:59 NodeA pulse-sensor-proxy[1755688]: 8:51AM WRN Handler failed error="failed to get temperatures: failed to fetch temperatures: exit status 127 (stderr: bash: line 1: /opt/pulse/sensor-proxy/bin/pulse-sensor-wrapper.sh: No such file or directory)" corr_id=55565967-4915-46c3-b1e8-d6eca295f15b method=get_temperature pid=260962 uid=100999
….. Rate limit exceeded pid=260962 reason=peer_concurrency uid=100999

Node 192.168.xx.13 is the one with no temperatures (but all the other data is working OK).
Could it be possible that some previous scripts' remnants are still "alive" in this node (due to my previous messing around)? Is there a way to remove everything related to this proxy on that specific node and force a re-install?

FYI: I can SSH from NodeA to this node and from Pulse LXC (they are all on the same VLAN)

The alternating value thing is still there, but it's not a big deal, it can stay as it is. Worth to be noted, though, is that the physical disks' temps don't show this behavior.

Cheers
Fabbio