Heavynode permanently grays out in Grid UI due to telegraf redisqueue metric parse error — 2.4.211

[JonBut-kk]

Version

2.4.211

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

32

RAM

62GB

Storage for /

65G

Storage for /nsm

324G

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Distributed deployment: 1 manager, 1 heavynode, 2 search nodes.

While I understand we should be using a forward node instead of the heavynode, we wanted to use the extra storage of our heavynode for cold storage, but I only figured out later, that heavynode is hard-coded to not allow that. But the issues are still very real and still affect the variuos docker services mentioned in this report.

This is also my first time using SO in a distributed environment, so apologies for any lack of knowledge on my part.

1st Issue: Heavynode permanently grays out in Grid UI

The heavynode grays out in the SOC Grid interface repeatedly. All services are healthy — so-status is all green, Salt connectivity works (test.ping returns True), sensoroni heartbeats return HTTP 200 every 10 seconds, alerts generate normally, and Zeek/Suricata operate correctly. The node is fully functional operationally but the Grid never shows it as active consistently.

The Root cause i think i identified: The telegraf redisqueue input plugin is deployed to the heavynode via Salt state, but the heavynode does not run so-redis (the main pipeline Redis on port 6379). The heavynode only runs so-strelka-coordinator (port 6380) and so-strelka-gatekeeper (port 6381). The redisqueue script queries port 6379, receives empty output, and produces unparseable metrics: metric parse error: expected field at 1:21: "redisqueue unparsed=,parsed=,mem_used="

This error fires every 30 seconds continuously and has been present since node deployment (confirmed in rotated logs back to June 2nd). Because telegraf cannot parse any metrics, nothing is shipped to InfluxDB on the manager. To my knowledge, the Grid UI relies on InfluxDB for node health data — with no metrics present for the heavynode, it renders the node as gray/stale. This was confirmed via:

sudo tail -50 /opt/so/log/telegraf/telegraf.log > continuous parsing errors

sudo docker exec so-influxdb influx query \ 'from(bucket:"so_short_term") |> range(start:-1h) |> filter(fn:(r) => r.host == "soc-heavynode") |> limit(n:5)' > no heavynode metrics in InfluxDB at all

The workround for this is running so-checkin > this restores the node in the Grid interface albeit temporarily, untill the next heartbeat fails to receive any meaningful data.

2nd issue: Some PCAP retrieval jobs fail with do to a date range conflict:

Some PCAP jobs dispatched to the heavynode are skipped immediately. The sensoroni log shows the stenoquery module reporting a permanently inverted availability window:
{ "availableDataBeginDate": "2026-06-15T11:18:38.251225795Z", "availableDataEndDate": "2026-06-15T11:16:38.251256576Z", "jobBeginDate": "2026-06-12T05:59:32.385Z", "jobEndDate": "2026-06-12T07:03:32.385Z", "message": "Skipping steno processor due to date range conflict" }

This is not for all PCAPs but i have noticed a few fail to retrieve data even though they are quite fresh and still in our PCAP folder. I understand since we're getting alot of network data, our current storage constraints only allow us of 6days of PCAPs to be stored, but this particular example i picked from a PCAP that still exists by confirming that Stenographer is healthy >

stenoread > queries execute successfully from CLI and return data
/nsm/pcap > contains active data spanning June 9–15 2026 (~102TB)
so-steno > shows as running, Up 5 days
NTP sync confirmed healthy (3 microseconds offset via chrony)
Container time matches host time exactly

The suriquery module is also configured in sensoroni.json but /nsm/suripcap is empty > Suricata PCAP is not in use. This may be a contributing factor to the date range calculation issue, but i am just guessing here.

The manager's telegraf log also shows a similar empty metric error > metric parse error: expected field at 1:19: "influxsize kbytes="

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Are the two servers date/time synchronized?

[JonBut-kk]

As of now, we've changed our heavynode into a dedicated sensor node, and at least for now, all is well, if any of the issues come back, i will create a new discussion, thank you for the response.