Elasticsearch Status: Pending , Unassigned elastalert_status

[geistchevalier]

Version

3.1.0

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

6

RAM

64

Storage for /

500GB

Storage for /nsm

2TB

Network Traffic Collection

tap

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

I'm getting Elasticsearch Status: Pending on my standalone node status page, which usually means that there's an index that needs to be turned over and deleted.

But this time when I ran sudo so-elasticsearch-query _cat/shards | grep UNASSIGNED
I get this instead

elastalert_status    0   r UNASSIGNED

Calling highstate I get one failure which is the following

          ID: so_index_template_dir
    Function: file.directory
        Name: /opt/so/conf/elasticsearch/templates/index
      Result: False
     Comment: The following requisites were not found:
                                 require:
                                     file: so_index_template_so-logs-fortinet.firewall
     Started: 06:01:53.516920
    Duration: 0.005 ms
     Changes:

Only found [WARN ] in /opt/so/log/elasticsearch/securityonion.log, can't find any [ERR]/[ERROR]

[2026-07-08T04:13:44,086][INFO ][org.elasticsearch.xpack.transform.transforms.TransformFailureHandler] [logs-extrahop.latest_investigation-default-0.1.0] Transform encountered an exception: [org.elasticsearch.ElasticsearchSecurityException: Cannot start transform [logs-extrahop.latest_investigation-default-0.1.0] because user lacks required permissions, see privileges_check_failed issue for more details]; Will automatically retry [1/-1]
org.elasticsearch.ElasticsearchSecurityException: Cannot start transform [logs-extrahop.latest_investigation-default-0.1.0] because user lacks required permissions, see privileges_check_failed issue for more details
        at org.elasticsearch.xpack.transform.transforms.TransformIndexer.onStart(TransformIndexer.java:290) ~[?:?]
        at org.elasticsearch.xpack.core.indexing.AsyncTwoPhaseIndexer.lambda$maybeTriggerAsyncJob$3(AsyncTwoPhaseIndexer.java:235) ~[?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) ~[elasticsearch-9.3.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[?:?]
        at java.lang.Thread.run(Thread.java:1474) ~[?:?]

[2026-07-08T04:18:37,772][WARN ][org.elasticsearch.index.engine.Engine] failed engine [merge failed]
org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=91e8335b actual=f654b17a (resource=BufferedChecksumIndexInput(MemorySegmentIndexInputpath="/usr/share/elasticsearch/data/indices/N2GTcAi_Th-m0GoUYhNPyQ/0/index/_19d.cfs") [slice=_19d_Lucene90_0.dvd]))

        at org.elasticsearch.index.engine.InternalEngine$5.doRun(InternalEngine.java:3085) ~[elasticsearch-9.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1113) ~[elasticsearch-9.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) ~[elasticsearch-9.3.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) ~[?:?]
        at java.lang.Thread.run(Thread.java:1474) ~[?:?]
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=91e8335b actual=f654b17a (resource=BufferedChecksumIndexInput(MemorySegmentIndexInput(path="/usr/share/elasticsearch/data/indices/N2GTcAi_Th-m0GoUYhNPyQ/0/index/_19d.cfs") [slice=_19d_Lucene90_0.dvd]))
        at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:441) ~[lucene-core-10.3.2.jar:?]
        at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:620) ~[lucene-core-10.3.2.jar:?]
        at org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer.checkIntegrity(Lucene90DocValuesProducer.java:1790) ~[lucene-core-10.3.2.jar:?]
        at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.checkIntegrity(PerFieldDocValuesFormat.java:362) ~[lucene-core-10.3.2.jar:?]
        at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:135) ~[lucene-core-10.3.2.jar:?]
        at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.merge(PerFieldDocValuesFormat.java:150) ~[lucene-core-10.3.2.jar:?]
        at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:188) ~[lucene-core-10.3.2.jar:?]
        at org.apache.lucene.index.SegmentMerger.mergeWithLogging(SegmentMerger.java:314) ~[lucene-core-10.3.2.jar:?]
        at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:149) ~[lucene-core-10.3.2.jar:?]
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5317) ~[lucene-core-10.3.2.jar:?]
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4778) ~[lucene-core-10.3.2.jar:?]
        at org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:6601) ~[lucene-core-10.3.2.jar:?]
        at org.elasticsearch.index.engine.ThreadPoolMergeScheduler.doMerge(ThreadPoolMergeScheduler.java:479) ~[elasticsearch-9.3.3.jar:?]
        at org.elasticsearch.index.engine.ThreadPoolMergeScheduler$MergeTask.run(ThreadPoolMergeScheduler.java:598) ~[elasticsearch-9.3.3.jar:?]
        at org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.runMergeTask(ThreadPoolMergeExecutorService.java:370) ~[elasticsearch-9.3.3.jar:?]
        at org.elasticsearch.index.engine.ThreadPoolMergeExecutorService.lambda$enqueueMergeTaskExecution$11(ThreadPoolMergeExecutorService.java:338) ~[elasticsearch-9.3.3.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:1046) ~[elasticsearch-9.3.3.jar:?]
        ... 3 more

Other details:
_cluster/health?pretty result (max cluster size is 1500)

{
  "cluster_name" : "securityonion",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 1015,
  "active_shards" : 1015,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 1,
  "unassigned_primary_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 99.9015748031496
}

GET _cluster/settings

{
  "persistent": {
    "cluster": {
      "remote": {
        "securityonion-a-00": {
          "seeds": [
            "127.0.0.1:9300"
          ]
        }
      },
      "max_shards_per_node": "1500"
    },
    "logsdb": {
      "prior_logs_usage": "true"
    },
    "search": {
      "max_open_scroll_context": "1000"
    }
  },
  "transient": {}
}

Suricata sync failed

{"fields":{"detectionEngine":"suricata","error":"integrity check failed; discrepancies found"},"level":"error","timestamp":"2026-07-08T06:56:06.650608819Z","message":"integrity check first failure, running force sync"}

{"fields":{"detectionEngine":"suricata","error":"failed to get existing detections: Elasticsearch response is not a valid JSON search result","syncId":"15e9874e-c560-4407-9d86-3fb42a73a0e3"},"level":"error","timestamp":"2026-07-08T06:56:50.125600282Z","message":"failed to apply user state"}

{"fields":{"detectionEngine":"suricata","error":"failed to sync community rules","syncId":"15e9874e-c560-4407-9d86-3fb42a73a0e3"},"level":"error","timestamp":"2026-07-08T06:56:50.125614475Z","message":"failed to sync community rules"}

Unsure on what else to look for.

edit: the suricata sync fixed itself I guess? still have the Elasticsearch status: Pending though

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[reyesj2]

You can set the number of replicas for elastalert_status to 0 (since you only have 1 elasticsearch node it can't create a replica you have configured)

sudo so-elasticsearch-query elastalert_status/_settings -d '{"number_of_replicas":0}' -XPUT

As for the error you are seeing in your highstate

          ID: so_index_template_dir
    Function: file.directory
        Name: /opt/so/conf/elasticsearch/templates/index
      Result: False
     Comment: The following requisites were not found:
                                 require:
                                     file: so_index_template_so-logs-fortinet.firewall
     Started: 06:01:53.516920
    Duration: 0.005 ms
     Changes:

that is related to #15959 and fixed in the next release

[geistchevalier]

That fixed it, thank you.