Logstash fails to start after upgrade to 2.4.211 - "http.host" setting removed in Logstash 9.x

[Kyle-44]

Version

2.4.201

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Distributed

Location

airgap

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

64

Storage for /

5 TB

Storage for /nsm

4 TB

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

Less than 1Gbps

Status

No, one or more services are failed (please provide detail below)

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Category: 2.4
I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.
Security Onion Version: 2.4.211
Deployment Type: Distributed (Manager, 2 Search Nodes, Receiver, Sensor)
OS: Oracle Linux 9.7
Description:
After upgrading from 2.4.201 to 2.4.211 via soup, so-logstash fails to start on the manager node. All other containers come up healthy. so-status shows so-logstash as "missing."
Docker logs show:
Using bundled JDK: /usr/share/logstash/jdk
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Your settings are invalid. Reason: Setting "http.host" doesn't exist. Please check if you haven't made a typo.
Root Cause Analysis:
2.4.211 upgrades Elasticsearch to 9.0.8, which appears to also bring in Logstash 9.x. Logstash 9.x fully removed the deprecated http.host setting (see elastic/logstash#16552). It must now be api.http.host.
The defaults.yaml at /opt/so/saltstack/default/salt/logstash/defaults.yaml correctly defines the config key as api_x_http_x_host: 0.0.0.0. The logstash.yml template at /opt/so/saltstack/default/salt/logstash/etc/logstash.yml renders config using:
{{ LOGSTASH_MERGED.config | yaml(False) | replace("x", ".") }}
This should produce api.http.host: 0.0.0.0, but the rendered file at /opt/so/conf/logstash/etc/logstash.yml contains http.host: 0.0.0.0 instead. The pillar only contains pipeline_x_workers as a local override, so the default api_x_http_x_host key should be used, but something in the merge or SOC config database appears to be substituting the old http_x_host key.
Steps to Reproduce:
Start with a distributed deployment running 2.4.201
Run soup to upgrade to 2.4.211
After upgrade completes, run so-status
so-logstash shows as missing
sudo docker logs so-logstash shows the "http.host" doesn't exist error
Troubleshooting Performed:
Manually edited /opt/so/conf/logstash/etc/logstash.yml to use api.http.host — Salt overwrites on next highstate
Attempted docker cp back into container — volume is mounted read-only
Searched entire Salt tree (grep -r "http.host" /opt/so/saltstack/) — no matches in logstash files
SOC Configuration UI (Administration > Configuration > logstash > config) shows api_x_http_x_host correctly but it is read-only
sudo salt-call pillar.get logstash:config --output=json only returns pipeline_x_workers: 12
Attempted creating local Salt override at /opt/so/saltstack/local/salt/logstash/etc/logstash.yml — not yet confirmed if this survives highstate
Expected Behavior:
The rendered logstash.yml should contain api.http.host: 0.0.0.0 to be compatible with Logstash 9.x.
Actual Behavior:
The rendered logstash.yml contains http.host: 0.0.0.0, which causes Logstash 9.x to refuse to start.
Note: This will likely also affect upgrades from 2.4.210 to 3.0.0 since 3.0 also ships ES 9.0.8.

[cm-ops]

Do you have a local mod at /opt/so/saltstack/local/salt/logstash/? Anything in local would override the default path and setting.

[Kyle-44]

No local mods at /opt/so/saltstack/local/salt/logstash/. We traced the full chain and the correct key exists everywhere it should:
SOC Administration UI (Configuration > logstash > config) shows api_x_http_x_host: 0.0.0.0 (read-only)
defaults.yaml at /opt/so/saltstack/default/salt/logstash/defaults.yaml has api_x_http_x_host: 0.0.0.0
Pillar (salt-call pillar.get logstash:config) only returns pipeline_x_workers: 12, no http_x_host override
The Jinja template at /opt/so/saltstack/default/salt/logstash/etc/logstash.yml uses {{ LOGSTASH_MERGED.config | yaml(False) | replace("x", ".") }}
But the rendered output at /opt/so/conf/logstash/etc/logstash.yml always comes out as http.host: 0.0.0.0 instead of api.http.host: 0.0.0.0. We edited the rendered file multiple times to api.http.host and Salt overwrote it back to http.host on every highstate. Something in LOGSTASH_MERGED is injecting the old key during the merge but we couldn't identify the source.

[cm-ops]

The Logstash defaults.yaml and what is in the Logstash pillar make up the LOGSTASH_MERGED. Look at the pillar, cat /opt/so/saltstack/local/pillar/logstash/soc_logstash.sls, check the minion pillar file for any logstash config, You already confirmed there is no local defaults.yaml.

Did you custom volume bind any file? docker inspect --format='{{json .Mounts}}' so-logstash | jq`

[Kyle-44]

Logstash 9.x fails to start after 2.4.x upgrade: stale http_x_host pillar key renders deprecated http.host: into logstash.yml
Summary
After upgrading a distributed cluster from 2.4.201 → 2.4.210, Logstash 9.x crashes on startup across all nodes with:
Your settings are invalid. Reason: Setting "http.host" doesn't exist.
Please check if you haven't made a typo.
The rendered /opt/so/conf/logstash/etc/logstash.yml contains both the new key (api.http.host: 0.0.0.0) and the deprecated key (http.host: None or http.host: 0.0.0.0). Logstash 9.x rejects the presence of the deprecated key regardless of value.
Environment

SO version: 2.4.210 (upgraded from 2.4.201)
Install type: Distributed (1 manager + 2 search nodes + 1 receiver)
OS: Oracle Linux 9
Hypervisor: VMware vSphere
Mode: Airgapped

Steps to Reproduce

Run a distributed SO cluster on 2.4.201 (or earlier) with http_x_host keyed into the Logstash pillar at any time during its lifetime (e.g. via SOC web UI before that key was deprecated)
Run sudo soup to upgrade to 2.4.210
Observe Logstash container fails to start on all nodes after upgrade completes
Inspect /opt/so/conf/logstash/etc/logstash.yml and observe deprecated http.host: line present

Expected Behavior
Logstash 9.x starts cleanly. Rendered logstash.yml contains only api.http.host: (the new key), not the deprecated http.host:.
Actual Behavior
Rendered logstash.yml contains both keys. Logstash 9.x refuses to start. The entire downstream Fleet integration update chain fails because Logstash is unavailable.
Investigation
The Logstash YAML template at /opt/so/saltstack/default/salt/logstash/etc/logstash.yml is a single Jinja line:
jinja{{ LOGSTASH_MERGED.config | yaml(False) | replace("x", ".") }}
This renders the merged pillar dict and substitutes x → . to produce dotted-key Logstash syntax. The deprecated key appears because http_x_host is somehow present in LOGSTASH_MERGED.config at render time, even though:

/opt/so/saltstack/default/salt/logstash/defaults.yaml defines only api_x_http_x_host (verified via grep -n http /opt/so/saltstack/default/salt/logstash/defaults.yaml)
/opt/so/saltstack/local/pillar/logstash/soc_logstash.sls sets only api_x_http_x_host: 0.0.0.0
sudo salt '' pillar.get logstash --output=yaml returns only valid keys under config: (no http_x_host)
map.jinja does not mutate LOGSTASH_MERGED.config with the legacy key

The http_x_host key appears to enter LOGSTASH_MERGED between pillar merge and template render. Suspected source: legacy SOC web-UI configuration entries from before the http_x_host → api_x_http_x_host rename. SOC job-history JSON files under /nsm/soc/jobs//100[1-6].json contain references to http_x_host and may be the persistence layer carrying the deprecated key across upgrades. Definitive source not yet confirmed.
Workaround
Patch the rendering template to filter http_x_host before YAML serialization. Replace the content of /opt/so/saltstack/default/salt/logstash/etc/logstash.yml with:
jinja{%- set FILTERED = {} -%}
{%- for k, v in LOGSTASH_MERGED.config.items() -%}
{%- if k != 'http_x_host' -%}
{%- do FILTERED.update({k: v}) -%}
{%- endif -%}
{%- endfor -%}
{{ FILTERED | yaml(False) | replace("x", ".") }}
Then push: sudo salt '*' state.apply logstash.
This restores Logstash. Caveat: the patched file is under /opt/so/saltstack/default/, which SO owns and which subsequent soup runs may overwrite. Operators must re-apply after each upgrade.
A dict-comprehension version of this filter does NOT parse under Salt's Jinja (expected token ',', got 'for'); the {% set %} / {% for %} / {% do %} form is the working one.

[Kyle-44]

#15932

Appears they have fix this issue.