[BUG] s3fs + masterless does not work without adding roots to fileserver_backend

[block-erichter]

Description
Using only s3fs for a fileserver_backend does not work properly (cannot find top.sls) without the addition of roots to fileserver_backend and then pointing file_roots to the s3cache in /var/cache/salt/minion/s3cache/$S3_BUCKET.

Setup
Tested on a AWS t2.micro EC2 instance (IMDSv2) running AL2022 with 5.15.57-30.131.amzn2022.x86_64 kernel

NAME="Amazon Linux"
VERSION="2022"
ID="amzn"
ID_LIKE="fedora"
VERSION_ID="2022"
PLATFORM_ID="platform:al2022"
PRETTY_NAME="Amazon Linux 2022"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2022"
HOME_URL="https://amazonlinux.com/"

Linux ip-10-0-0-233.ec2.internal 5.15.57-30.131.amzn2022.x86_64 #1 SMP Wed Aug 17 23:50:15 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Full non-working minion configuration (see error below)

top_file_merging_strategy: same

#####     File Directory Settings    #####
##########################################
# The Salt Minion can redirect all file server operations to a local directory,
# this allows for the same state tree that is on the master to be used if
# copied completely onto the minion. This is a literal copy of the settings on
# the master but used to reference a local directory on the minion.

# Set the file client. The client defaults to looking on the master server for
# files, but can be directed to look at the local file directory setting
# defined below by setting it to "local". Setting a local file_client runs the
# minion in masterless mode.
file_client: local

fileserver_backend:
  - s3fs

s3.keyid: REDACTED
s3.key: REDACTED
s3.location: us-east-1
s3.service_url: s3.us-east-1.amazonaws.com

s3.buckets:
  - salt-config-676XXXXXXXX
s3.https_enable: true

s3.verify_ssl: True


# The file directory works on environments passed to the minion, each environment
# can have multiple root directories, the subdirectories in the multiple file
# roots cannot match, otherwise the downloaded files will not be able to be
# reliably ensured. A base environment is required to house the top file.
# Example:
#file_roots:
#  base:
#    - /srv/salt

Workaround required to get s3fs working

top_file_merging_strategy: same

#####     File Directory Settings    #####
##########################################
# The Salt Minion can redirect all file server operations to a local directory,
# this allows for the same state tree that is on the master to be used if
# copied completely onto the minion. This is a literal copy of the settings on
# the master but used to reference a local directory on the minion.

# Set the file client. The client defaults to looking on the master server for
# files, but can be directed to look at the local file directory setting
# defined below by setting it to "local". Setting a local file_client runs the
# minion in masterless mode.
file_client: local

fileserver_backend:
  - s3fs
  - roots

s3.keyid: REDACTED
s3.key: REDACTED
s3.location: us-east-1
s3.service_url: s3.us-east-1.amazonaws.com

s3.buckets:
  - salt-config-676XXXXXXXX
s3.https_enable: true

s3.verify_ssl: True


# The file directory works on environments passed to the minion, each environment
# can have multiple root directories, the subdirectories in the multiple file
# roots cannot match, otherwise the downloaded files will not be able to be
# reliably ensured. A base environment is required to house the top file.
# Example:
file_roots:
  base:
    - /var/cache/salt/minion/s3cache/salt-config-676XXXXXXXX

Steps to Reproduce the behavior

1. Launch an EC2 instance and create an S3 bucket to serve salt files
2. Configure minion using the above configs and point to the proper bucket
3. Attempt to apply state with salt-call state.apply or salt-call --local state.apply
   Both of these produce the following error (debug logs below for greater detail):

[ERROR   ] The 'base' saltenv has no top file, and the fallback saltenv specified by default_top (base) also has no top file
local:
----------
          ID: states
    Function: no.None
      Result: False
     Comment: No Top file or master_tops data matches found. Please see master log for details.
     Changes:

Full debug using salt-call state.apply -l debug

[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load log4mongo
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load raven
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: ip-10-0-0-233.ec2.internal
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load log4mongo
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load raven
[DEBUG   ] Configuration file path: /etc/salt/minion
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pywintypes
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load dns
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load tldextract
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load tldextract
[DEBUG   ] The functions from module 'core' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'disks' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'extra' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'lvm' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'mdadm' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'minion_process' are being loaded by dir() on the loaded module
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load napalm
[DEBUG   ] The functions from module 'opts' are being loaded by dir() on the loaded module
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pythoncom
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load win32com
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load win32api
[DEBUG   ] Override  __utils__: <module 'salt.loaded.int.grains.zfs' from '/opt/saltstack/salt/run/salt/grains/zfs.py'>
[DEBUG   ] The functions from module 'zfs' are being loaded by dir() on the loaded module
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Elapsed time getting FQDNs: 0.024873018264770508 seconds
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load _bootlocale
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load lsb_release
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] The functions from module 'zfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded zfs.is_supported
[DEBUG   ] Determining pillar cache
[DEBUG   ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] The functions from module 'state' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded state.apply
[DEBUG   ] The functions from module 'direct_call' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded direct_call.execute
[DEBUG   ] Override  __grains__: <module 'salt.loaded.int.module.grains' from '/opt/saltstack/salt/run/salt/modules/grains.py'>
[DEBUG   ] The functions from module 'grains' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded grains.get
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pywintypes
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load immutables
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load singledispatch
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load singledispatch_helpers
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load backports_abc
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load backports
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load saltwinshell
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load esky
[DEBUG   ] The functions from module 'saltutil' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded saltutil.is_running
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load smbprotocol
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pypsexec
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load winrm
[DEBUG   ] The functions from module 'config' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded config.get
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load git
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pygit2
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Updating s3fs fileserver cache
[DEBUG   ] Refreshing buckets cache file
[DEBUG   ] The functions from module 's3' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3.query
[DEBUG   ] S3 Request: https://salt-config-67XXXXXXXXXXX.s3.us-east-1.amazonaws.com/?marker=
[DEBUG   ] S3 Headers::
[DEBUG   ]     Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=26bcff00305dad8ac2b1f7ab2e1f3bb8120d36f376ea6d2475d112c0ce96c158
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load netrc
[DEBUG   ] Starting new HTTPS connection (1): salt-config-676XXXXXXXX .s3.us-east-1.amazonaws.com:443
[DEBUG   ] https://salt-config-676XXXXXXXXXXX.s3.us-east-1.amazonaws.com:443 "GET /?marker= HTTP/1.1" 200 None
[DEBUG   ] S3 Response Status Code: 200
[DEBUG   ] Writing buckets cache file
[INFO    ] Syncing local cache from S3...
[INFO    ] salt-config-676XXXXXXXX -  : base.sls
[INFO    ] salt-config-676XXXXXXXX -  : top.sls
[INFO    ] Sync local cache from S3 completed.
[DEBUG   ] Gathering pillar data for state run
[DEBUG   ] Determining pillar cache
[DEBUG   ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] Finished gathering pillar data for state run
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Could not find file 'salt://top.sls' in saltenv 'base'
[DEBUG   ] No contents loaded for saltenv 'base'
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Reading buckets cache file
[DEBUG   ] In saltenv '', looking at rel_path 'top.sls' to resolve 'salt://top.sls'
[DEBUG   ] In saltenv '', ** considering ** path '/var/cache/salt/minion/files/top.sls' to resolve 'salt://top.sls'
[DEBUG   ] Fetching file from saltenv '', ** attempting ** 'salt://top.sls'
[DEBUG   ] No dest file found
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Reading buckets cache file
[INFO    ] Fetching file from saltenv '', ** done ** 'top.sls'
[DEBUG   ] compile template: /var/cache/salt/minion/files/op.sls
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/op.sls' using 'jinja' renderer: 0.003148794174194336
[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/op.sls:
base:
  '*':
    - base

[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/op.sls' using 'yaml' renderer: 0.0002684593200683594
[ERROR   ] The 'base' saltenv has no top file, and the fallback saltenv specified by default_top (base) also has no top file
[DEBUG   ] Section for saltenv 'base' in the '' saltenv's top file will be ignored, as the top_file_merging_strategy is set to 'same' and the saltenvs do not match
[DEBUG   ] The functions from module 'state' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded state.check_result
[DEBUG   ] The functions from module 'highstate' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded highstate.output
local:
----------
          ID: states
    Function: no.None
      Result: False
     Comment: No Top file or master_tops data matches found. Please see master log for details.
     Changes:

Summary for local
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   0.000 ms
[DEBUG   ] Restoring sys.argv to: ['/opt/saltstack/salt/run/run', 'state.apply', '-l', 'debug']

Note that even though the above fails, the s3cache is still created and populated:

┌──(root@ip-10-0-0-233)-[/var/cache/salt/minion/s3cache/salt-config-67XXXXXXX]
└─$ ls -la
total 8
drwx------. 2 root root  37 Sep 15 16:49 .
drwx------. 3 root root  65 Sep 15 16:49 ..
-rw-------. 1 root root 101 Sep 15 16:49 base.sls
-rw-------. 1 root root  24 Sep 15 16:49 top.sls

┌──(root@ip-10-0-0-233)-[/var/cache/salt/minion/s3cache/salt-config-676252252670]
└─$ tail -n +1 *
==> base.sls <==
/etc/host:
  file.managed:
    - replace: False
    - user: root
    - group: root
    - mode: '0644'
==> top.sls <==
base:
  '*':
    - base

┌──(root@ip-10-0-0-233)-[/var/cache/salt/minion/files]
└─$ ls -la
total 4
drwx------. 2 root root 20 Sep 15 16:49 .
drwx------. 5 root root 46 Sep 15 16:49 ..
-rw-------. 1 root root 24 Sep 15 16:49 op.sls

Debug output of workaround using fileserver_backend roots workaround

[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load log4mongo
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load raven
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: ip-10-0-0-233.ec2.internal
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load log4mongo
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load raven
[DEBUG   ] Configuration file path: /etc/salt/minion
[DEBUG   ] Grains refresh requested. Refreshing grains.
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pywintypes
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load dns
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load tldextract
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load tldextract
[DEBUG   ] The functions from module 'core' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'disks' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'extra' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'lvm' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'mdadm' are being loaded by dir() on the loaded module
[DEBUG   ] The functions from module 'minion_process' are being loaded by dir() on the loaded module
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load napalm
[DEBUG   ] The functions from module 'opts' are being loaded by dir() on the loaded module
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pythoncom
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load win32com
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load win32api
[DEBUG   ] Override  __utils__: <module 'salt.loaded.int.grains.zfs' from '/opt/saltstack/salt/run/salt/grains/zfs.py'>
[DEBUG   ] The functions from module 'zfs' are being loaded by dir() on the loaded module
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Elapsed time getting FQDNs: 0.029502391815185547 seconds
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load _bootlocale
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load lsb_release
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG   ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG   ] The functions from module 'zfs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded zfs.is_supported
[DEBUG   ] Determining pillar cache
[DEBUG   ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] The functions from module 'state' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded state.apply
[DEBUG   ] The functions from module 'direct_call' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded direct_call.execute
[DEBUG   ] Override  __grains__: <module 'salt.loaded.int.module.grains' from '/opt/saltstack/salt/run/salt/modules/grains.py'>
[DEBUG   ] The functions from module 'grains' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded grains.get
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pywintypes
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load immutables
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load singledispatch
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load singledispatch_helpers
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load backports_abc
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load backports
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load saltwinshell
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load esky
[DEBUG   ] The functions from module 'saltutil' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded saltutil.is_running
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load smbprotocol
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pypsexec
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load winrm
[DEBUG   ] The functions from module 'config' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded config.get
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load git
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pygit2
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG   ] Updating s3fs fileserver cache
[DEBUG   ] Refreshing buckets cache file
[DEBUG   ] The functions from module 's3' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3.query
[DEBUG   ] S3 Request: https://salt-config-676XXXXXXXX.s3.us-east-1.amazonaws.com/?marker=
[DEBUG   ] S3 Headers::
[DEBUG   ]     Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3a2b68ef747aacfda018c43cf2713fb1d5b244b0e3b0012427fb3d272deb2846
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load netrc
[DEBUG   ] Starting new HTTPS connection (1): salt-config-676XXXXXXXX .s3.us-east-1.amazonaws.com:443
[DEBUG   ] https://salt-config-676XXXXXXXXXXX.s3.us-east-1.amazonaws.com:443 "GET /?marker= HTTP/1.1" 200 None
[DEBUG   ] S3 Response Status Code: 200
[DEBUG   ] Writing buckets cache file
[INFO    ] Syncing local cache from S3...
[INFO    ] salt-config-676XXXXXXXX -  : base.sls
[INFO    ] salt-config-676XXXXXXXX -  : top.sls
[INFO    ] Sync local cache from S3 completed.
[DEBUG   ] Updating roots fileserver cache
[DEBUG   ] Gathering pillar data for state run
[DEBUG   ] Determining pillar cache
[DEBUG   ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] Finished gathering pillar data for state run
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Reading buckets cache file
[DEBUG   ] In saltenv 'base', looking at rel_path 'top.sls' to resolve 'salt://top.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/top.sls' to resolve 'salt://top.sls'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://top.sls'
[DEBUG   ] No dest file found
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Reading buckets cache file
[INFO    ] Fetching file from saltenv 'base', ** done ** 'top.sls'
[DEBUG   ] compile template: /var/cache/salt/minion/files/base/top.sls
[DEBUG   ] Jinja search path: ['/var/cache/salt/minion/files/base']
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/top.sls' using 'jinja' renderer: 0.05261063575744629
[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/base/top.sls:
base:
  '*':
    - base

[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/top.sls' using 'yaml' renderer: 0.00021910667419433594
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Reading buckets cache file
[DEBUG   ] In saltenv '', looking at rel_path 'top.sls' to resolve 'salt://top.sls'
[DEBUG   ] In saltenv '', ** considering ** path '/var/cache/salt/minion/files/top.sls' to resolve 'salt://top.sls'
[DEBUG   ] Fetching file from saltenv '', ** attempting ** 'salt://top.sls'
[DEBUG   ] No dest file found
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Reading buckets cache file
[INFO    ] Fetching file from saltenv '', ** done ** 'top.sls'
[DEBUG   ] compile template: /var/cache/salt/minion/files/op.sls
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/op.sls' using 'jinja' renderer: 0.000751495361328125
[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/op.sls:
base:
  '*':
    - base

[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/op.sls' using 'yaml' renderer: 0.00015544891357421875
[DEBUG   ] Section for saltenv 'base' in the '' saltenv's top file will be ignored, as the top_file_merging_strategy is set to 'same' and the saltenvs do not match
[DEBUG   ] The functions from module 'confirm_top' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded confirm_top.confirm_top
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load seco
[DEBUG   ] The functions from module 'compound_match' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded compound_match.match
[DEBUG   ] compound_match: ip-10-0-0-233.ec2.internal ? *
[DEBUG   ] The functions from module 'glob_match' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded glob_match.match
[DEBUG   ] compound_match ip-10-0-0-233.ec2.internal ? "*" => "True"
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load esky
[DEBUG   ] The functions from module 'saltutil' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded saltutil.sync_all
[DEBUG   ] Syncing all
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/clouds'
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO    ] Syncing clouds for environment 'base'
[INFO    ] Loading cache from salt://_clouds, for base
[INFO    ] Caching directory '_clouds/' for environment 'base'
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_clouds'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/beacons'
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO    ] Syncing beacons for environment 'base'
[INFO    ] Loading cache from salt://_beacons, for base
[INFO    ] Caching directory '_beacons/' for environment 'base'
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_beacons'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/modules'
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO    ] Syncing modules for environment 'base'
[INFO    ] Loading cache from salt://_modules, for base
[INFO    ] Caching directory '_modules/' for environment 'base'
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_modules'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/states'
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO    ] Syncing states for environment 'base'
[INFO    ] Loading cache from salt://_states, for base
[INFO    ] Caching directory '_states/' for environment 'base'
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_states'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/sdb'
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO    ] Syncing sdb for environment 'base'
[INFO    ] Loading cache from salt://_sdb, for base
[INFO    ] Caching directory '_sdb/' for environment 'base'
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_sdb'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/grains'
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO    ] Syncing grains for environment 'base'
[INFO    ] Loading cache from salt://_grains, for base
[INFO    ] Caching directory '_grains/' for environment 'base'
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_grains'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/renderers'
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO    ] Syncing renderers for environment 'base'
[INFO    ] Loading cache from salt://_renderers, for base
[INFO    ] Caching directory '_renderers/' for environment 'base'
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_renderers'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/returners'
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO    ] Syncing returners for environment 'base'
[INFO    ] Loading cache from salt://_returners, for base
[INFO    ] Caching directory '_returners/' for environment 'base'
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_returners'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/output'
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO    ] Syncing output for environment 'base'
[INFO    ] Loading cache from salt://_output, for base
[INFO    ] Caching directory '_output/' for environment 'base'
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_output'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/utils'
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO    ] Syncing utils for environment 'base'
[INFO    ] Loading cache from salt://_utils, for base
[INFO    ] Caching directory '_utils/' for environment 'base'
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_utils'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/log_handlers'
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO    ] Syncing log_handlers for environment 'base'
[INFO    ] Loading cache from salt://_log_handlers, for base
[INFO    ] Caching directory '_log_handlers/' for environment 'base'
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_log_handlers'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/executors'
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO    ] Syncing executors for environment 'base'
[INFO    ] Loading cache from salt://_executors, for base
[INFO    ] Caching directory '_executors/' for environment 'base'
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_executors'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/proxy'
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO    ] Syncing proxy for environment 'base'
[INFO    ] Loading cache from salt://_proxy, for base
[INFO    ] Caching directory '_proxy/' for environment 'base'
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_proxy'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/engines'
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO    ] Syncing engines for environment 'base'
[INFO    ] Loading cache from salt://_engines, for base
[INFO    ] Caching directory '_engines/' for environment 'base'
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_engines'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/thorium'
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO    ] Syncing thorium for environment 'base'
[INFO    ] Loading cache from salt://_thorium, for base
[INFO    ] Caching directory '_thorium/' for environment 'base'
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_thorium'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/serializers'
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO    ] Syncing serializers for environment 'base'
[INFO    ] Loading cache from salt://_serializers, for base
[INFO    ] Caching directory '_serializers/' for environment 'base'
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_serializers'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/matchers'
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO    ] Syncing matchers for environment 'base'
[INFO    ] Loading cache from salt://_matchers, for base
[INFO    ] Caching directory '_matchers/' for environment 'base'
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_matchers'
[INFO    ] Creating module dir '/var/cache/salt/minion/extmods/pillar'
[DEBUG   ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded s3fs.envs
[DEBUG   ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded roots.envs
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG   ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG   ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO    ] Syncing pillar for environment 'base'
[INFO    ] Loading cache from salt://_pillar, for base
[INFO    ] Caching directory '_pillar/' for environment 'base'
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_pillar'
[DEBUG   ] Refreshing modules...
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load site
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load esky
[DEBUG   ] The functions from module 'saltutil' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded saltutil.refresh_modules
[DEBUG   ] The functions from module 'event' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded event.fire
[DEBUG   ] SaltEvent PUB socket URI: /var/run/salt/minion/minion_event_01eb525e9d_pub.ipc
[DEBUG   ] SaltEvent PULL socket URI: /var/run/salt/minion/minion_event_01eb525e9d_pull.ipc
[ERROR   ] Unable to connect pusher: Stream is closed
Traceback (most recent call last):
  File "salt/utils/event.py", line 427, in connect_pull
    self.pusher.connect(timeout=timeout)
  File "salt/utils/asynchronous.py", line 125, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "salt/utils/asynchronous.py", line 131, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "salt/ext/tornado/ioloop.py", line 459, in run_sync
    return future_cell[0].result()
  File "salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "salt/transport/ipc.py", line 345, in _connect
    yield self.stream.connect(sock_addr)
  File "salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
salt.ext.tornado.iostream.StreamClosedError: Stream is closed
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Returning file list from cache: age=2 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Reading buckets cache file
[DEBUG   ] In saltenv 'base', looking at rel_path 'base.sls' to resolve 'salt://base.sls'
[DEBUG   ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/base.sls' to resolve 'salt://base.sls'
[DEBUG   ] Fetching file from saltenv 'base', ** attempting ** 'salt://base.sls'
[DEBUG   ] No dest file found
[DEBUG   ] Reading buckets cache file
[DEBUG   ] Closing IPCMessageClient instance
[DEBUG   ] Reading buckets cache file
[INFO    ] Fetching file from saltenv 'base', ** done ** 'base.sls'
[DEBUG   ] compile template: /var/cache/salt/minion/files/base/base.sls
[DEBUG   ] Jinja search path: ['/var/cache/salt/minion/files/base']
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/base.sls' using 'jinja' renderer: 0.0013020038604736328
[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/base/base.sls:
/etc/host:
  file.managed:
    - replace: False
    - user: root
    - group: root
    - mode: '0644'
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/base.sls' using 'yaml' renderer: 0.00031065940856933594
[DEBUG   ] The functions from module 'config' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded config.option
[DEBUG   ] The functions from module 'file' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded file.managed
[INFO    ] Running state [/etc/host] at time 16:51:27.596731
[INFO    ] Executing state file.managed for [/etc/host]
[DEBUG   ] TiamatPipPathFinder.find_spec: First time trying to load mmap
[DEBUG   ] The functions from module 'file' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded file.user_to_uid
[INFO    ] File /etc/host exists with proper permissions. No changes made.
[INFO    ] Completed state [/etc/host] at time 16:51:27.637698 (duration_in_ms=40.968)
[DEBUG   ] File /var/cache/salt/minion/accumulator/133644863202016 does not exist, no need to cleanup
[DEBUG   ] The functions from module 'state' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded state.check_result
[DEBUG   ] The functions from module 'highstate' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded highstate.output
local:
----------
          ID: /etc/host
    Function: file.managed
      Result: True
     Comment: File /etc/host exists with proper permissions. No changes made.
     Started: 16:51:27.596730
    Duration: 40.968 ms
     Changes:

Summary for local
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1
Total run time:  40.968 ms
[DEBUG   ] Restoring sys.argv to: ['/opt/saltstack/salt/run/run', 'state.apply', '-l', 'debug']

Expected behavior
Using only s3fs should be sufficient and not require any roots configured on the host. Using an s3fs only configuration would ensure that all configurations applied come directly from S3 instead using the workaround of pulling states/files from S3 to disk and then applying configurations that are read from disk.

Versions Report

<details><summary>
Salt Version:
          Salt: 3005

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.0
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
        Python: 3.9.13 (main, Aug 23 2022, 18:33:16)
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: amzn 2022
        locale: utf-8
       machine: x86_64
       release: 5.15.57-30.131.amzn2022.x86_64
        system: Linux
       version: Amazon Linux 2022
</summary>
</details>

Additional context
Workaround appears to work but this does not feel like the expected configuration. Also broached in the Salt Slack channel here:
https://saltstackcommunity.slack.com/archives/C7K04SEJC/p1663115884126479

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!