Sensor nodes stopped sending events/alerts

[ejgh-oe]

Version

2.4.211

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

10

RAM

32

Storage for /

500

Storage for /nsm

500

Network Traffic Collection

span port

Network Traffic Speeds

more than 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Hi,
Just found out that S.O. hasn't been generating alerts for around two weeks.

Symptoms:

• Grid shows 0 (zero) EPS for all nodes - sensor, search, management
• even for high traffic sensor nodes traffic in/out displayed in Grid shows only a few MBit/s. Indeed the LAN-interface of the sensor nodes has very little traffic
• Hunt>connections: last traffic shown dates back to late April

What I already checked/tried:

• reboot all nodes -> didn't change anything
• so-status on all nodes involved -> OK
• suricata on the sensor nodes is definitely active -> top shows decent CPU-usage for Suricata process. Normal activity in /nsm/suricap

Any idea what's going on here, i.e. why all sensor nodes stop sending alerts - and how I could get them working again?

Thanks much for any clue...

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

Troubleshoot your ingest pipeline, you can start at the sensor. Check your raw logs for Zeek and Suricata in /nsm. If the are writing current logs, then check your Elastic Agent to make sure it is running and has the correct agent policy.

Logstash would be next, on your Manager node, check the Logstash log /opt/so/log/logstash/logstash.log for any issues. If that is fine and your Redis queue is working, the next place to check is Logstash on your search node, then Elasticsearch on the search node.

Do you see any issues in the pipeline?

[ejgh-oe]

Troubleshoot your ingest pipeline, you can start at the sensor.
Check your raw logs for Zeek and Suricata in /nsm.

/nsm/suricata: Plenty of "eve"-Files and data is written to them
`/nsm/zeek/logs: one directory for every day. For today I've got almost 600 files

If the are writing current logs, then check your Elastic Agent to make sure it is running and has the correct agent policy.

How should I go about that? Are they stored on the sensor node? In a file? Where?

Logstash would be next, on your Manager node, check the Logstash log /opt/so/log/logstash/logstash.log for any issues.

Oops - /opt/so/log/logstash/logstash.log has tons of messages like these

[2026-05-11T09:23:49,545][INFO ][org.logstash.beats.BeatsHandler] [local: 172.17.1.29:5055, remote: 192.168.162.166:47950] Handling exception: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed (caused by: java.security.cert.CertificateExpiredException: NotAfter: Sat Apr 18 08:58:04 GMT 2026)
[2026-05-11T09:23:49,546][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:500) ~[netty-codec-4.1.126.Final.jar:4.1.126.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.126.Final.jar:4.1.126.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.126.Final.jar:4.1.126.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.126.Final.jar:4.1.126.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.126.Final.jar:4.1.126.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) ~[netty-transport-4.1.126.Final.jar:4.1.126.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.126.Final.jar:4.1.126.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.126.Final.jar:4.1.126.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) ~[netty-transport-4.1.126.Final.jar:4.1.126.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.126.Final.jar:4.1.126.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) ~[netty-transport-4.1.126.Final.jar:4.1.126.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732) ~[netty-transport-4.1.126.Final.jar:4.1.126.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658) ~[netty-transport-4.1.126.Final.jar:4.1.126.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.126.Final.jar:4.1.126.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) ~[netty-common-4.1.126.Final.jar:4.1.126.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.126.Final.jar:4.1.126.Final]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]

As it turned out I've got the exact same error message for every S.O. node in the grid (!)

To me that java.security.cert.CertificateExpiredException: NotAfter: Sat Apr 18 08:58:04 GMT 2026 from the error messages sounds like some certificate expired.

The question is which one and equally important - how could I fix this in order to get the setup working agin?

If that is fine and your Redis queue is working, the next place to check is Logstash on your search node, then Elasticsearch on the search node.
Do you see any issues in the pipeline?

Didn't check these so far, because of the above error message.

[cm-ops]

Check this certificate first - openssl x509 -in /etc/pki/elasticfleet-logstash.crt -noout -enddate if that certificate is valid then check the one in Fleet.

Fleet > Settings > edit grid-logstash output (click the pencil on the right)

If the one in Fleet is expired, replace the client certificate and key in Fleet with /etc/pki/elasticfleet-logstash.crt and elasticfleet-logstash.key

[ejgh-oe]

Hi,
TL;DR;
Awesame - problem solved. All onions on the grid are fine again :-)

Following your receipe openssl x509 -in /etc/pki/elasticfleet-logstash.crt -noout -enddate showed an end date of notAfter=Mar 14 15:50:40 2027 GMT so I proceeded with the next step you described, i.e. in Fleet > Settings change certificate and key to the contents of /etc/pki/elasticfleet-logstash.crt and /etc/pki/elasticfleet-logstash.key respectively

Saved the changes...

and after some minutes events started to come in again.

On the manager node logstash is currently consuming a fair share of CPU available, but that's a good sign ;-). Likewise the search nodes are doing their job as expected again.

So thank you so much for your help - you definitely brought all "all our little onions" ;-) back to life again.

One question remaining though: The certificate in question expires Mar 14 2027, so less than one year from now.
But - what happens after Mar 14 2027, the day even this certificate expires?

How do I make sure not to run into certificate lifetime issues in the future? Any other certs I should keep an eye on?

In that context: When upgrading from S.O. 2.4.111 to 3.x will all certificates involved in the entire grid setup be replaced by more recent ones?