Initialize wazuh-indexer-setup plugin

[AlexRuiz7]

Description

Create the basic folder structure and source code for the Wazuh Indexer initialization plugin.

The plugin must be able to use OpenSearch's lifecycle to initialize a set of index templates (see wazuh/wazuh-indexer#270).

[AlexRuiz7]

I made a simple plugin that create an index on startup, using custom mappings and settings.

[2024-06-20T11:34:16,662][INFO ][o.w.s.WazuhIndexerSetupPlugin] [integTest-0] wazuh-indexer-setup-plugin index created

[AlexRuiz7]

Installed the plugin in a production environment

[vagrant@rhel9 ~]$ curl -X GET https://localhost:9200 -u 'admin:SecretPassword.1234' --insecure
{
  "name" : "rhel9.localdomain",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "3k7EopDeSr-rZ_USXvqpiQ",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.14.0",
    "build_type" : "rpm",
    "build_hash" : "aaa555453f4713d652b52436874e11ba258d8f03",
    "build_date" : "2024-05-09T18:51:01.558090944Z",
    "build_snapshot" : false,
    "lucene_version" : "9.10.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}
[vagrant@rhel9 ~]$ curl -X GET https://localhost:9200/_cat/indices -u 'admin:SecretPassword.1234' --insecure
green open .opensearch-observability  mBHtGkJ0RL6gF3RkagEIpg 1 0  0 0   208b   208b
green open .plugins-ml-config         ALvIY4F8S3WvXwTKlhZL-Q 1 0  1 0  3.8kb  3.8kb
green open wazuh-indexer-setup-plugin aaRDd_yYRZuAsyS2gsvixA 1 0  0 0   208b   208b
green open .opendistro_security       _SifTF-KStiFfJL5w222Qw 1 0 10 0 77.6kb 77.6kb
[vagrant@rhel9 ~]$ 

[AlexRuiz7]

Test scenario

During the testing of the rollover + alias feature that was planned for 4.8.0, we saw that it was possible to index data before the initialization process was completed. As a result, indices were created without mappings or settings, completely messing up Wazuh's indexing workflow. See wazuh/wazuh#18999.

We through we could solve this race condition using plugins. The plugin will hook into the Indexer's initialization process to create all the index templates required by Wazuh, naturally blocking any indexing request during the process.

Test results

Using a test environment with OpenSearch 2.14.0, I built and installed the PoC plugin from #4. The test consists of trying to index anything to the wazuh-indexer-setup-plugin index before the index initializes the index. As we time window is very short to made it manually, I created a bash script to automate the process.

#!/bin/bash

USERNAME="admin"
PASSWORD="SecretPassword.1234"

# If OpenSearch is running, remove the wazuh-indexer-setup-plugin index and restart the service
curl -X DELETE https://192.168.56.10:9200/wazuh-indexer-setup-plugin -k -u $USERNAME:$PASSWORD

systemctl restart opensearch

while true; do
    echo "Injecting..."
    curl -X POST https://192.168.56.10:9200/wazuh-indexer-setup-plugin/_doc -k -u $USERNAME:$PASSWORD -H 'Content-Type: application/json' -d '{"type":"test"}'
    sleep 50e-3 # 50 ms
done

The test has been executed several times, returning the same results:

[vagrant@rhel9 ~]$ sudo bash injector.sh 
{"acknowledged":true}Injecting...
OpenSearch Security not initialized.Injecting...
OpenSearch Security not initialized.Injecting...
OpenSearch Security not initialized.Injecting...
OpenSearch Security not initialized.Injecting...
OpenSearch Security not initialized.Injecting...
OpenSearch Security not initialized.Injecting...
OpenSearch Security not initialized.Injecting...
OpenSearch Security not initialized.Injecting...
OpenSearch Security not initialized.Injecting...
OpenSearch Security not initialized.Injecting...
OpenSearch Security not initialized.Injecting...
OpenSearch Security not initialized.Injecting...
OpenSearch Security not initialized.Injecting...
OpenSearch Security not initialized.Injecting...
OpenSearch Security not initialized.Injecting...
{"_index":"wazuh-indexer-setup-plugin","_id":"PD6OdJABKFzsAk5STmRx","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}Injecting...
{"_index":"wazuh-indexer-setup-plugin","_id":"PT6OdJABKFzsAk5STmT-","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1}Injecting...
{"_index":"wazuh-indexer-setup-plugin","_id":"Pj6OdJABKFzsAk5ST2SV","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":2,"_primary_term":1}Injecting...
{"_index":"wazuh-indexer-setup-plugin","_id":"Pz6OdJABKFzsAk5SUGQn","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":3,"_primary_term":1}Injecting...
{"_index":"wazuh-indexer-setup-plugin","_id":"QD6OdJABKFzsAk5SUGS7","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":4,"_primary_term":1}Injecting...
{"_index":"wazuh-indexer-setup-plugin","_id":"QT6OdJABKFzsAk5SUWRP","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":5,"_primary_term":1}Injecting...
{"_index":"wazuh-indexer-setup-plugin","_id":"Qj6OdJABKFzsAk5SUWTY","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":6,"_primary_term":1}Injecting...
{"_index":"wazuh-indexer-setup-plugin","_id":"Qz6OdJABKFzsAk5SUmRl","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":7,"_primary_term":1}Injecting...
{"_index":"wazuh-indexer-setup-plugin","_id":"RD6OdJABKFzsAk5SUmT8","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":8,"_primary_term":1}Injecting...
{"_index":"wazuh-indexer-setup-plugin","_id":"RT6OdJABKFzsAk5SU2SS","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":9,"_primary_term":1}Injecting...
{"_index":"wazuh-indexer-setup-plugin","_id":"Rj6OdJABKFzsAk5SVGQr","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"_seq_no":10,"_primary_term":1}Injecting...

As we can see, the script first prepares the test scenario by removing the target index and restart OpenSearch. On start, the plugin will create the index template. At the same time, the script tries to index a document every 50ms.

All the indexing requests are ignored until the initialization process is completed. The index is created correctly, using the mappings and settings defined by the plugin.