allow customizing indexes for logs written to OpenSearch/Elasticsearch

[mmguero]

Some users want to be able to define their own different indices for Zeek and Suricata and not just lump them in with the Arkime indices. If this is done, the user will have to understand that without creating something like an Elasticsearch data view those logs wouldn't be visible in the UI/dashboards.

• primary opensearch output
• secondary opensearch output

We will also allow for the configuring of Arkime's rotateIndex value via environment variable.

Applicable configuration via environment variables:

• arkime.env

# How often to create a new index in OpenSearch/Elasticsearch
#   https://arkime.com/settings#rotateIndex
ARKIME_ROTATE_INDEX=daily

• opensearch.env

# OpenSearch index patterns and timestamp fields
# Index pattern for network traffic logs written via Logstash (e.g., Zeek logs, Suricata alerts)
MALCOLM_NETWORK_INDEX_PATTERN=arkime_sessions3-*
# Default time field to use for network traffic logs in Logstash and Dashboards
MALCOLM_NETWORK_INDEX_TIME_FIELD=firstPacket
# Suffix used to create index to which network traffic logs are written (supports Ruby strftime strings in %{})
MALCOLM_NETWORK_INDEX_SUFFIX=%{%y%m%d}
# Index pattern for other logs written via Logstash (e.g., nginx, beats, fluent-bit, etc.)
MALCOLM_OTHER_INDEX_PATTERN=malcolm_beats_*
# Default time field to use for other logs in Logstash and Dashboards
MALCOLM_OTHER_INDEX_TIME_FIELD=@timestamp
# Suffix used to create index to which other logs are written (supports Ruby strftime strings in %{})
MALCOLM_OTHER_INDEX_SUFFIX=%{%y%m%d}
# Index pattern used specifically by Arkime (will probably match MALCOLM_NETWORK_INDEX_PATTERN, should probably be arkime_sessions3-*)
ARKIME_NETWORK_INDEX_PATTERN=arkime_sessions3-*
# Default time field used by for sessions in Arkime viewer
ARKIME_NETWORK_INDEX_TIME_FIELD=firstPacket

[mmguero]

For example, in order to use weekly time bucketing for the indices:

• arkime.env

...
# How often to create a new index in OpenSearch/Elasticsearch
#   https://arkime.com/settings#rotateIndex
ARKIME_ROTATE_INDEX=weekly
...

• opensearch.env

...
MALCOLM_NETWORK_INDEX_SUFFIX=%{%yw%U}
...
MALCOLM_OTHER_INDEX_SUFFIX=%{%yw%U}
...

resulting in:

dc exec -u $(id -u) opensearch curl -sS -XGET http://localhost:9200/_cat/indices|grep 24w
green  open malcolm_beats_nginx_24w02         -TvMFHXrSoCF5gkL_IRLjQ 1 0    9   2  74.2kb  74.2kb
green  open malcolm_beats_disk_24w02          5Q64CQkqRGi7zNsoUOA-MQ 1 0   21   0  39.1kb  39.1kb
green  open malcolm_beats_thermal_24w02       Z4hFCNqoTEOvls7Bvowwaw 1 0   10   0  79.8kb  79.8kb
green  open arkime_sessions3-24w02            WiPND_cDRVyVN1U-Tmsw6w 1 0  897   0   1.9mb   1.9mb
green  open malcolm_beats_mem_24w02           73TwZodrRQSP8Sy7DrC2Cg 1 0    6   0  41.4kb  41.4kb
green  open malcolm_beats_network_24w02       yE-Nh0U5R8ivusN6-DrD0w 1 0    6   0  57.7kb  57.7kb
green  open malcolm_beats_cpu_24w02           gAICFOJoR46APHXuvu0NhQ 1 0    2   0  51.1kb  51.1kb
green  open malcolm_beats_systemd_24w02       DxBV4MhWSniRBNeblMHm1A 1 0  122  11 313.6kb 313.6kb

[mmguero]

Or hourly:

• arkime.env

...
# How often to create a new index in OpenSearch/Elasticsearch
#   https://arkime.com/settings#rotateIndex
ARKIME_ROTATE_INDEX=hourly
...

• opensearch.env

...
MALCOLM_NETWORK_INDEX_SUFFIX=%{%y%m%dh%H}
...
MALCOLM_OTHER_INDEX_SUFFIX=%{%y%m%dh%H}
...

resulting in:

dc exec -u $(id -u) opensearch curl -sS -XGET http://localhost:9200/_cat/indices|grep 15h
green open malcolm_beats_thermal_240115h20 h0_uCC0_QleHz4CA0Uu8tw 1 0    9   0  60.6kb  60.6kb
green open malcolm_beats_cpu_240115h20     jYRE9N7CRwCHwI0CWzWMjQ 1 0    3   0  76.6kb  76.6kb
green open malcolm_beats_mem_240115h20     oo9rfLNGSweLXH9q_JeX0g 1 0    6   0  33.4kb  33.4kb
green open malcolm_beats_systemd_240115h20 jIi_E1CoTZCuflzLt1GoCw 1 0   90   7 247.3kb 247.3kb
green open malcolm_beats_disk_240115h20    BUY76o1xRPCS_yPfgg-MuQ 1 0   21   0  35.7kb  35.7kb
green open malcolm_beats_network_240115h20 uS7aCi2MSku-su287dmVHQ 1 0    6   0    40kb    40kb
green open malcolm_beats_nginx_240115h20   N0MEgqtxSRm-qdB7667TQA 1 0   11   2  32.3kb  32.3kb
green open arkime_sessions3-240115h20      HZpVSHOLScmO5cd8-T4M7A 1 0  512   0 898.5kb 898.5kb