FIX: Logstash container missing on distributed receiver

[m0duspwnens]

Discussed in #11092

^{Originally posted by rosswakelin August 17, 2023}
I have just built my first distributed 2.4 setup in the lab, and the receiver node won't bring up logstash, with a so-status error of "missing"
Running so-logstash-restart give 58 succeeded tasks and 7 Failed tasks, all to do with missing entries for ssl keys.
Examples are:

          ID: rediskeyperms
    Function: file.managed
        Name: /etc/pki/redis.key
      Result: True
     Comment: File /etc/pki/redis.key exists with proper permissions. No changes made.
     Started: 00:05:55.617048
    Duration: 1.256 ms
     Changes:   
----------
          ID: etc_elasticfleet_logstash_key
    Function: x509.private_key_managed
        Name: /etc/pki/elasticfleet-logstash.key
      Result: False
     Comment: The following requisites were not found:
                                 prereq:
                                     x509: etc_elasticfleet_crt
     Started: 00:05:55.618931
    Duration: 0.003 ms
     Changes:   

and

          ID: etc_elasticfleet_logstash_crt
    Function: file.managed
        Name: /etc/pki/elasticfleet-logstash.crt
      Result: True
     Comment: File /etc/pki/elasticfleet-logstash.crt not updated
     Started: 00:05:55.976855
    Duration: 0.002 ms
     Changes:   
----------
          ID: etc_elasticfleet_logstash_crt
    Function: cmd.run
        Name: /usr/bin/openssl pkcs8 -in /etc/pki/elasticfleet-logstash.key -topk8 -out /etc/pki/elasticfleet-logstash.p8 -nocrypt
      Result: False
     Comment: One or more requisite failed: ssl.etc_elasticfleet_logstash_key
     Started: 00:05:55.978349
    Duration: 0.004 ms
     Changes:   

Any hints?

[rosswakelin]

Requested information
SO version 2.4.10
On-prem - with internet access
Installed from ISO
1 x Manager node, 2 x search nodes
Virtual machines, with CPU/MEM/Disk all more than adequate
Since installing, not a single event has appeared on any of the dashboards.
The fourth node installed was the receiver node, and the node status shows Fault, with the so-logstash container "missing"

This is the tail of the sosetup.log from the receiver node:

[INFO    ] Loading fresh modules for state activity
[INFO    ] Fetching file from saltenv 'base', ** done ** 'setup/highstate_cron.sls'
[INFO    ] Running state [PATH=$PATH:/usr/sbin salt-call state.highstate] at time 23:37:53.609575
[INFO    ] Executing state cron.present for [PATH=$PATH:/usr/sbin salt-call state.highstate]
[INFO    ] Executing command 'crontab' in directory '/root'
[INFO    ] Executing command 'crontab' in directory '/root'
[INFO    ] {'root': 'PATH=$PATH:/usr/sbin salt-call state.highstate'}
[INFO    ] Completed state [PATH=$PATH:/usr/sbin salt-call state.highstate] at time 23:37:53.624580 (duration_in_ms=15.006)
local:
----------
          ID: post_setup_cron
    Function: cron.present
        Name: PATH=$PATH:/usr/sbin salt-call state.highstate
      Result: True
     Comment: Cron PATH=$PATH:/usr/sbin salt-call state.highstate added to root's crontab
     Started: 23:37:53.609574
    Duration: 15.006 ms
     Changes:
              ----------
              root:
                  PATH=$PATH:/usr/sbin salt-call state.highstate

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  15.006 ms
2023-08-17T23:37:53Z | INFO | Verifying setup
Successfully completed setup!

There are no logfiles in the /opt/so/log/logstash directory.

A salt-call state.highstate on the manager gives two errors about not finding elastic-fleet urls.
A salt-call state.highstate on the receiver gives:

local:
    Data failed to compile:
----------
    Pillar failed to render with the following messages:
----------
    Specified SLS 'minions.chl-osor-v02_receiver' in environment 'base' is not available on the salt master
----------
    Specified SLS 'minions.adv_chl-osor-v02_receiver' in environment 'base' is not available on the salt master

[defensivedepth]

@rosswakelin *Edit

I see the other issue, thanks for the extra info!