Alerts not loading

[cerberus10748]

Version

3.0.0

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

22

RAM

100 gb

Storage for /

1000

Storage for /nsm

18000

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

I recently had to readd a search node to my cluster. Now when i go to alerts i get the following error The search query encountered a failure within the Elasticsearch cluster. Check SOC logs for details.

Please advise next steps to try I have tried rebooting everything.

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[cm-ops]

What does the SOC log say? /opt/so/log/soc/sensoroni-server.log

Also, on your manager's CLI, what does this output? sudo so-elasticsearch-troubleshoot

[mbaki]

Good morning

I have the same issue, brand new 3.0 installation, and my alerts are not showing. I went to a server that I am monitoring and made sure sid 2100498 is enabled. The alert does not show.
[admin@securityonion ~]$ sudo tail -f /opt/so/log/soc/sensoroni-server.log
{"fields":{"contentLength":0,"elapsedMs":0,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:54146","requestId":"69da243c-0806-44b8-b6f0-2e8ceb395e8a","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"192.168.1.163","statusCode":200},"level":"info","timestamp":"2026-05-21T12:38:49.364586047Z","message":"Handled request"}
{"fields":{"contentLength":0,"elapsedMs":0,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:54150","requestId":"ab03162d-f6b8-4e89-8dd7-dce46eee5819","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"192.168.1.163","statusCode":200},"level":"info","timestamp":"2026-05-21T12:38:50.369270456Z","message":"Handled request"}
{"fields":{"contentLength":0,"elapsedMs":0,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:54156","requestId":"fde2b78f-5680-473f-bd0c-f4fad1decfc1","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"192.168.1.163","statusCode":200},"level":"info","timestamp":"2026-05-21T12:38:51.374288896Z","message":"Handled request"}
{"fields":{"contentLength":0,"elapsedMs":482,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:54160","requestId":"9edd7a37-7bf8-4ba4-91d8-aa1015e2e023","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"192.168.1.163","statusCode":200},"level":"info","timestamp":"2026-05-21T12:38:52.860065496Z","message":"Handled request"}
{"fields":{"contentLength":0,"elapsedMs":0,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:37034","requestId":"83281fb3-bafc-4858-912b-e822a8cc4e42","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"192.168.1.163","statusCode":200},"level":"info","timestamp":"2026-05-21T12:38:53.863607653Z","message":"Handled request"}
{"fields":{"contentLength":0,"elapsedMs":0,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:37044","requestId":"b5e47471-ea76-48ba-a52b-035070ece181","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"192.168.1.163","statusCode":200},"level":"info","timestamp":"2026-05-21T12:38:54.866883506Z","message":"Handled request"} [admin@securityonion ~]
$ sudo so-elasticsearch-troubleshoot

================ Elasticsearch Status ================

All health indicators are green

================ Disk Usage Check ================

LOW:80% HIGH:85% FLOOD:90%

securityonion disk usage: 5.71%

================ Unassigned Shards Check ================

All shards are assigned
[admin@securityonion ~]$

On my server that I am monitoring if I run curl testmynids.org/uid/index.html, and on my security onion if I run [admin@securityonion ~]$ sudo tcpdump -i enp7s0 port 80
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp7s0, link-type EN10MB (Ethernet), snapshot length 262144 bytes

12:43:58.017494 IP 192.168.1.5.30540 > server-18-165-83-29.iad55.r.cloudfront.net.http: Flags [S], seq 3484714600, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2632407066 ecr 0], length 0
12:43:58.023808 IP server-18-165-83-29.iad55.r.cloudfront.net.http > 192.168.1.5.30540: Flags [S.], seq 3514675262, ack 3484714601, win 65535, options [mss 1440,sackOK,TS val 1083151211 ecr 2632407066,nop,wscale 9], length 0
12:43:58.023930 IP 192.168.1.5.30540 > server-18-165-83-29.iad55.r.cloudfront.net.http: Flags [.], ack 1, win 1035, options [nop,nop,TS val 2632407077 ecr 1083151211], length 0
12:43:58.033847 IP 192.168.1.5.30540 > server-18-165-83-29.iad55.r.cloudfront.net.http: Flags [P.], seq 1:93, ack 1, win 1035, options [nop,nop,TS val 2632407077 ecr 1083151211], length 92: HTTP: GET /uid/index.html HTTP/1.1
12:43:58.033847 IP server-18-165-83-29.iad55.r.cloudfront.net.http > 192.168.1.5.30540: Flags [.], ack 93, win 128, options [nop,nop,TS val 1083151221 ecr 2632407077], length 0
12:43:58.037615 IP server-18-165-83-29.iad55.r.cloudfront.net.http > 192.168.1.5.30540: Flags [P.], seq 1:539, ack 93, win 128, options [nop,nop,TS val 1083151223 ecr 2632407077], length 538: HTTP: HTTP/1.1 200 OK
12:43:58.037985 IP 192.168.1.5.30540 > server-18-165-83-29.iad55.r.cloudfront.net.http: Flags [F.], seq 93, ack 539, win 1035, options [nop,nop,TS val 2632407087 ecr 1083151223], length 0

[cerberus10748]

It says 268 unavailable primary shards.

[cm-ops]

@mbaki please start another post with your issue.

[cerberus10748]

{"fields":{"reason":"N/A","reasonType":"no_shard_available_action_exception"},"level":"warn","timestamp":"2026-05-21T17:47:55.274005641Z","message":"Shard failure"}
{"fields":{"reason":"N/A","reasonType":"no_shard_available_action_exception"},"level":"warn","timestamp":"2026-05-21T17:47:55.274030791Z","message":"Shard failure"}
{"fields":{"reason":"N/A","reasonType":"no_shard_available_action_exception"},"level":"warn","timestamp":"2026-05-21T17:47:55.274053825Z","message":"Shard failure"}
{"fields":{"reason":"N/A","reasonType":"no_shard_available_action_exception"},"level":"warn","timestamp":"2026-05-21T17:47:55.274077198Z","message":"Shard failure"}
{"fields":{"reason":"N/A","reasonType":"no_shard_available_action_exception"},"level":"warn","timestamp":"2026-05-21T17:47:55.274100565Z","message":"Shard failure"}
{"fields":{"reason":"N/A","reasonType":"no_shard_available_action_exception"},"level":"warn","timestamp":"2026-05-21T17:47:55.274124395Z","message":"Shard failure"}
{"fields":{"reason":"N/A","reasonType":"no_shard_available_action_exception"},"level":"warn","timestamp":"2026-05-21T17:47:55.274157488Z","message":"Shard failure"}
{"fields":{"reason":"N/A","reasonType":"no_shard_available_action_exception"},"level":"warn","timestamp":"2026-05-21T17:47:55.274181228Z","message":"Shard failure"}
{"fields":{"reason":"N/A","reasonType":"no_shard_available_action_exception"},"level":"warn","timestamp":"2026-05-21T17:47:55.274208555Z","message":"Shard failure"}
{"fields":{"reason":"N/A","reasonType":"no_shard_available_action_exception"},"level":"warn","timestamp":"2026-05-21T17:47:55.274232295Z","message":"Shard failure"}
{"fields":{"reason":"N/A","reasonType":"no_shard_available_action_exception"},"level":"warn","timestamp":"2026-05-21T17:47:55.274260177Z","message":"Shard failure"}
{"fields":{"error":"ERROR_QUERY_FAILED_ELASTICSEARCH","requestId":"bd90425f-9288-40b2-9aa1-36cf549c4c9c","requestorId":"90aff02d-8b9f-48e8-baba-49de80b66618"},"level":"warn","timestamp":"2026-05-21T17:47:55.274298017Z","message":"Request did not complete successfully"}
{"fields":{"contentLength":32,"elapsedMs":104,"impl":"/build/server/eventhandler.go:89:github.com/security-onion-solutions/securityonion-soc/server.(EventHandler).getEvent","remoteAddr":"172.17.1.1:51940","requestId":"bd90425f-9288-40b2-9aa1-36cf549c4c9c","requestMethod":"GET","requestPath":"/api/events/","requestQuery":{"eventLimit":["500"],"format":["2006/01/02 3:04:05 PM"],"metricLimit":["500"],"query":["() AND tags:alert AND NOT event.acknowledged:true AND NOT event.escalated:true | groupby rule.name event.module* event.severity_label rule.uuid"],"range":["2026/05/20 01:47:55 PM - 2026/05/21 01:47:55 PM"],"zone":["America/New_York"]},"requestorId":"90aff02d-8b9f-48e8-baba-49de80b66618","sourceIp":"10.184.1.128","statusCode":500},"level":"info","timestamp":"2026-05-21T17:47:55.274415864Z","message":"Handled request"}
{"fields":{"contentLength":0,"elapsedMs":0,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:51942","requestId":"aca992e7-d2ab-4262-8379-743aa854a073","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"192.168.1.250","statusCode":200},"level":"info","timestamp":"2026-05-21T17:47:55.495209609Z","message":"Handled request"}
{"fields":{"contentLength":0,"elapsedMs":0,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:51948","requestId":"0a557ca2-5060-40b5-a17b-7d692ad0cffa","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"192.168.2.250","statusCode":200},"level":"info","timestamp":"2026-05-21T17:47:55.497507299Z","message":"Handled request"}
{"fields":{"contentLength":0,"elapsedMs":0,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:51964","requestId":"e7a550de-752e-4ff5-84a1-f05228544785","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"192.168.2.249","statusCode":200},"level":"info","timestamp":"2026-05-21T17:47:55.498405132Z","message":"Handled request"}
{"fields":{"contentLength":0,"elapsedMs":0,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:51972","requestId":"0df637de-0086-4985-9bc7-c9228484e1ae","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"192.168.2.252","statusCode":200},"level":"info","timestamp":"2026-05-21T17:47:55.498827306Z","message":"Handled request"}
{"fields":{"contentLength":0,"elapsedMs":0,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:51974","requestId":"a0194479-7d5c-4626-8d24-676c02c45773","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"10.184.1.250","statusCode":200},"level":"info","timestamp":"2026-05-21T17:47:55.51772482Z","message":"Handled request"}
{"fields":{"messageContent":"{ "Kind": "Ping" }","messagePath":"/ws","messageRemoteAddr":"172.17.1.1:41562","messageSourceIp":"10.184.1.128","messageType":1,"requestId":"6bde6987-401b-48ea-8a22-ed455cdeba37"},"level":"info","timestamp":"2026-05-21T17:48:01.179878206Z","message":"WebSocket message received"}
{"fields":{"contentLength":0,"elapsedMs":0,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:39106","requestId":"878948d9-6755-4a9a-ac31-5460844b8565","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"192.168.2.250","statusCode":200},"level":"info","timestamp":"2026-05-21T17:48:05.511451305Z","message":"Handled request"}
{"fields":{"contentLength":0,"elapsedMs":0,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:39098","requestId":"57a285dc-fe4c-487f-aaf9-d74107eb5bf0","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"192.168.1.250","statusCode":200},"level":"info","timestamp":"2026-05-21T17:48:05.511941205Z","message":"Handled request"}
{"fields":{"contentLength":0,"elapsedMs":0,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:39114","requestId":"8fab0ef1-bf37-4283-9875-f86b5c774b03","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"192.168.2.249","statusCode":200},"level":"info","timestamp":"2026-05-21T17:48:05.512873619Z","message":"Handled request"}
{"fields":{"contentLength":0,"elapsedMs":0,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:39116","requestId":"2a0dd684-bbaa-45b2-8197-40f6b05269d5","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"192.168.2.252","statusCode":200},"level":"info","timestamp":"2026-05-21T17:48:05.513111109Z","message":"Handled request"}
{"fields":{"contentLength":0,"elapsedMs":0,"impl":"/build/server/nodehandler.go:67:github.com/security-onion-solutions/securityonion-soc/server.(*NodeHandler).postNode","remoteAddr":"172.17.1.1:39118","requestId":"9b773108-6ee9-45fe-8970-c4de25637750","requestMethod":"POST","requestPath":"/api/node","requestQuery":{},"requestorId":"00000000-0000-0000-0000-000000000000","sourceIp":"10.184.1.250","statusCode":200},"level":"info","timestamp":"2026-05-21T17:48:05.541469657Z","message":"Handled request"}

[cm-ops]

That is the issue, Elasticsearch knows about those shards but can't find a node to assign them. Did you follow https://docs.securityonion.net/en/3/main/removing-a-node/?h= If not, the data went with the removed search node, you can delete the unassigned shards from Elasticsearch to bring you back to good.

[cerberus10748]

I did not follow that no. How do i go about deleting the unassigned shards?

[cm-ops]

There are a couple of ways to do it. If you log into Kibana > Stack Management > Index Management you can delete from there. Second, from the CLI - sudo so-elasticsearch-query $index -XDELETE (replace $index with the actual index name, for example, .ds-logs-zeek-so-2026.03.11-000026)

[cerberus10748]

I found that in Kibana , but which Index do I delete?

[cm-ops]

The indices that are health Red. If you were to check your indices with sudo so-index-list | grep -v green that will list which ones are not started.

You can click the check box, click Manage Index, and Delete Index. If you get an error that says the index is the write index, rollover the data stream (https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover)