[Neon] Tracebacks in pkg.installed

[max-arnold]

The state succeeds, but there are a couple of tracebacks in the debug log triggered by the loader (this probably means that these unrelated modules should be fixed):

sudo salt-call state.single pkg.installed nginx -l debug

...
[DEBUG   ] Failed to import utils psutil_compat:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1589, in _load_module
    mod = spec.loader.load_module()
  File "<frozen importlib._bootstrap_external>", line 399, in _check_name_wrapper
  File "<frozen importlib._bootstrap_external>", line 823, in load_module
  File "<frozen importlib._bootstrap_external>", line 682, in load_module
  File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
  File "<frozen importlib._bootstrap>", line 684, in _load
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/lib/python3/dist-packages/salt/utils/psutil_compat.py", line 20, in <module>
    import psutil  # pylint: disable=3rd-party-module-not-gated
ModuleNotFoundError: No module named 'psutil'
[DEBUG   ] Could not LazyLoad boto3.assign_funcs: 'boto3.assign_funcs' is not available.
[DEBUG   ] Error loading module.boto3_elasticsearch: __init__ failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1653, in _load_module
    module_init(self.opts)
  File "/usr/lib/python3/dist-packages/salt/modules/boto3_elasticsearch.py", line 92, in __init__
    __utils__['boto3.assign_funcs'](__name__, 'es')
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1237, in __getitem__
    func = super(LazyLoader, self).__getitem__(item)
  File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
    raise KeyError(key)
KeyError: 'boto3.assign_funcs'
[DEBUG   ] key: ifttt.secret_key, ret: _|-
[DEBUG   ] key: ifttt:secret_key, ret: _|-
[DEBUG   ] key: pushbullet.api_key, ret: _|-
[DEBUG   ] key: pushbullet:api_key, ret: _|-
[DEBUG   ] Error loading module.swarm: __init__ failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/loader.py", line 1653, in _load_module
    module_init(self.opts)
  File "/usr/lib/python3/dist-packages/salt/modules/swarm.py", line 53, in __init__
    __context__['client'] = docker.from_env()
AttributeError: module 'docker' has no attribute 'from_env'
[DEBUG   ] key: victorops.api_key, ret: _|-
[DEBUG   ] key: victorops:api_key, ret: _|-
[DEBUG   ] DSC: Only available on Windows systems
[DEBUG   ] Module PSGet: Only available on Windows systems
[DEBUG   ] Could not LazyLoad pkg.ex_mod_init: 'pkg.ex_mod_init' is not available.
[INFO    ] Running state [nginx] at time 02:58:07.836108
[INFO    ] Executing state pkg.installed for [nginx]
[DEBUG   ] Could not LazyLoad pkg.resolve_capabilities: 'pkg.resolve_capabilities' is not available.
[INFO    ] Executing command ['dpkg-query', '--showformat', '${Status} ${Package} ${Version} ${Architecture}', '-W'] in directory '/home/vagrant'
[DEBUG   ] Could not LazyLoad pkg.normalize_name: 'pkg.normalize_name' is not available.
[INFO    ] All specified packages are already installed
[INFO    ] Completed state [nginx] at time 02:58:07.883557 (duration_in_ms=47.45)
...

Versions Report

Master branch:

salt-call --versions-report

Salt Version:
           Salt: 2019.2.0-n/a-b637401

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.9 (default, Nov  7 2019, 10:44:02)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-72-generic
         system: Linux
        version: Ubuntu 18.04 bionic

[Ch3LL]

im replicating the boto and docker tracebacks but not the psutil one and i ensured i did not have it installed. When i looked at the code it looks like its intentionally not catching the importerror because of this comment https://github.com/saltstack/salt/blame/master/salt/utils/psutil_compat.py#L19
but i did not think we required psutil. Can you confirm @s0undt3ch that we do not indeed require psutil? if so we can just catch that importerror exception

[s0undt3ch]

psutils is not a hard requirement.
I guess we started having this when we started supporting __utils__, and in this context, we start catching the exception.

[max-arnold]

I wonder what might be different in your setup in regard to psutil. I just tried to reproduce it on a clean Vagrant box (bento/ubuntu-18.04) and it happens reliably. Below is the bootstrap command I used:

bootstrap-salt.sh -P -x python3 -f -g https://github.com/saltstack/salt.git -F -c /tmp git master

[oeuftete]

The swarm module traceback looks to be fixed in #57489.

I've opened #58554 for the boto3_elasticsearch traceback.

@max-arnold I can still reproduce the psutil utils problem using your instructions in #55848 (comment) with the latest master (931580a).

[lnxslck]

I have this issue too. Is there a fix for this?