2.4.10 to 2.4.20 Upgrade error

[Sahale]

Version

2.4.20

Installation Method

Security Onion ISO image

Description

upgrading

Installation Type

Distributed

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

16

RAM

32GB

Storage for /

100

Storage for /nsm

100

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

Yes, there are salt failures (please provide detail below)

Logs

No, there are no additional clues

Detail

Hello!
Upgrade distributed setup from 2.4.10 to 2.4.20 (Manager, 1x Fleet and 1x Heavy).
Got error:

[INFO    ] Executing command 'yum' in directory '/root'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'influxdb/enabled.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'influxdb/config.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'soc/merged.map.jinja'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'soc/defaults.map.jinja'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'soc/defaults.yaml'
[ERROR   ] Rendering exception occurred
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 476, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 6, in top-level template code
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1405, in make_module
    return TemplateModule(self, ctx)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/environment.py", line 1535, in __init__
    body_stream = list(template.root_render_func(context))  # type: ignore
  File "/var/cache/salt/minion/files/base/soc/merged.map.jinja", line 10, in top-level template code
    {% do DOCKER_EXTRA_HOSTS.append({GLOBALS.influxdb_host:pillar.node_data[GLOBALS.influxdb_host].ip}) %}
  File "/opt/saltstack/salt/lib/python3.10/site-packages/jinja2/sandbox.py", line 326, in getattr
    value = getattr(obj, attribute)
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'so-24-test'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 218, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/templates.py", line 482, in render_jinja_tmpl
    raise SaltRenderError("Jinja variable {}{}".format(exc, out), line, tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'dict object' has no attribute 'so-24-test'
/var/cache/salt/minion/files/base/soc/merged.map.jinja(10):
---
[...]

{% from 'vars/globals.map.jinja' import GLOBALS %}
{% from 'soc/defaults.map.jinja' import SOCDEFAULTS with context %}
{% from 'logstash/map.jinja' import LOGSTASH_NODES %}
{% set DOCKER_EXTRA_HOSTS = LOGSTASH_NODES %}
{% do DOCKER_EXTRA_HOSTS.append({GLOBALS.influxdb_host:pillar.node_data[GLOBALS.influxdb_host].ip}) %}    <======================

{% set SOCMERGED = salt['pillar.get']('soc', SOCDEFAULTS, merge=true) %}

{# if SOCMERGED.config.server.modules.cases == httpcase details come from the soc pillar #}
{% if SOCMERGED.config.server.modules.cases != 'soc' %}
[...]
---
[CRITICAL] Rendering SLS 'base:soc' failed: Jinja variable 'dict object' has no attribute 'so-24-test'
/var/cache/salt/minion/files/base/soc/merged.map.jinja(10):
---
[...]

{% from 'vars/globals.map.jinja' import GLOBALS %}
{% from 'soc/defaults.map.jinja' import SOCDEFAULTS with context %}
{% from 'logstash/map.jinja' import LOGSTASH_NODES %}
{% set DOCKER_EXTRA_HOSTS = LOGSTASH_NODES %}
{% do DOCKER_EXTRA_HOSTS.append({GLOBALS.influxdb_host:pillar.node_data[GLOBALS.influxdb_host].ip}) %}    <======================

{% set SOCMERGED = salt['pillar.get']('soc', SOCDEFAULTS, merge=true) %}

{# if SOCMERGED.config.server.modules.cases == httpcase details come from the soc pillar #}
{% if SOCMERGED.config.server.modules.cases != 'soc' %}
[...]
---
[INFO    ] Fetching file from saltenv 'base', ** done ** 'manager/init.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'sensoroni/defaults.yaml'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'telegraf/defaults.yaml'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'telegraf/enabled.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'idstools/enabled.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'idstools/sync_files.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'mysql/config.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'logstash/enabled.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'redis/enabled.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'redis/config.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'playbook/enabled.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'playbook/config.sls'
[INFO    ] Executing command 'ls' in directory '/root'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'elasticfleet/enabled.sls'
[INFO    ] Fetching file from saltenv 'base', ** done ** 'elasticfleet/config.sls'
local:
    Data failed to compile:
----------
    Rendering SLS 'base:soc' failed: Jinja variable 'dict object' has no attribute 'so-24-test'
/var/cache/salt/minion/files/base/soc/merged.map.jinja(10):
---
[...]

{% from 'vars/globals.map.jinja' import GLOBALS %}
{% from 'soc/defaults.map.jinja' import SOCDEFAULTS with context %}
{% from 'logstash/map.jinja' import LOGSTASH_NODES %}
{% set DOCKER_EXTRA_HOSTS = LOGSTASH_NODES %}
{% do DOCKER_EXTRA_HOSTS.append({GLOBALS.influxdb_host:pillar.node_data[GLOBALS.influxdb_host].ip}) %}    <======================

{% set SOCMERGED = salt['pillar.get']('soc', SOCDEFAULTS, merge=true) %}

{# if SOCMERGED.config.server.modules.cases == httpcase details come from the soc pillar #}
{% if SOCMERGED.config.server.modules.cases != 'soc' %}
[...]
---

Starting crond service at 09:44:29.245856
Successfully started crond.

Same error when run # salt-call state.highstate

Guidelines

• I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[TOoSmOotH]

Did you modify extra hosts under docker for the influx container?

[Sahale]

No.
But I modified web address from hostname to fqdn. Can it be the reason?

[m0duspwnens]

This issue occurs when data can't be retrieved from the Salt mine during a soup. The mine should be populated by now, are you able to successfully run sudo salt-call state.highstate on the manager? If not, can you run sudo salt \* mine.update first then perform the highstate?

An issue has been opened to track this: #11502

[Sahale]

sudo salt-call state.highstate worked successfully. Can I rerun soup? Or it's better to wait for fix?

[m0duspwnens]

Since the highstate was successful, then the node should be running properly. In the SOC UI, under Grid, are the versions of your nodes 2.4.20 and are they in OK status?

[Sahale]

Oh, yes, it's 2.4.20 and status is OK!
Thanks!