Elasticsearch RED status - 600 unassigned shards due to disk watermark

[gil0x]

Version

2.4.180

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

other (please provide detail below)

Location

airgap

Hardware Specs

Meets minimum requirements

CPU

Intel Xeon Gold 6330 @ 2.00GHz 16 cores

RAM

62

Storage for /

3.9 T

Storage for /nsm

3.8 T

Network Traffic Collection

span port

Network Traffic Speeds

more than 10Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Hi,
Got an issue with SO 2.4.180 where Elasticsearch went RED and stopped indexing new logs.
What happened:
The /nsm partition filled up to 95.4% (3.6TB out of 4TB, mostly old Zeek logs). ES won't allocate shards anymore because it's above the 95% watermark.
Cluster shows 600 unassigned shards and status is RED. Already tried /_cluster/reroute?retry_failed=true but that didn't help.
My setup:

Manager: 16 cores, 62GB RAM, 4TB storage
Network collection via SPAN port
Grid shows so-elastalert and so-elastic-fleet as "missing"

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[yudin-s]

Do not expect /_cluster/reroute?retry_failed=true to help while the node is still above the flood-stage/high disk watermark. Elasticsearch is refusing allocation because the disk condition that caused the failure is still present.

I would handle it in this order:

1. Free space on /nsm first, or add storage. Get comfortably below the watermark, not just barely below 95%.
2. Use Security Onion-supported retention/cleanup paths where possible. Avoid manually deleting Elasticsearch shard directories from disk.
3. After space is available, retry allocation and watch the unassigned shard reasons:

curl -s localhost:9200/_cluster/allocation/explain?pretty
curl -s localhost:9200/_cat/shards?v | grep UNASSIGNED

4. Only then use a reroute/retry if shards are still marked failed after the disk problem is fixed.
5. Recheck service health after Elasticsearch recovers; so-elastalert and so-elastic-fleet being missing may be a downstream symptom of Elasticsearch being unhealthy, not the root cause.

The bigger issue is capacity/retention. With more than 10Gbps SPAN traffic, 4TB can fill quickly depending on what is being logged. After recovery, reduce retention/ingest volume or increase /nsm capacity so Elasticsearch does not immediately cross the watermark again.

If this solves it, please mark this comment as the answer so other people can find it faster.