_utils modules have no access to __salt__ or __utils__

[jtraub91]

Description of Issue

Putting utility functions that aren't intended to be run via the cli in _utils is a good way to organize code, however, its utility is severely limited by the fact that you cannot cross-call execution nor utility modules (via __salt__ / __utils__)

Setup

Normal salt installation. Create some example modules to demonstrate.

# /srv/salt/_utils/general.py

def utility_ping():
    return __salt__['test.ping']()


def utility_call():
    return __utils__['math.add'](2, 2)

# /srv/salt/_utils/math.py

def add(num1, num2):
    return num1 + num2

# /srv/salt/_modules/example.py

def execute_utility_ping():
    return __utils__['general.utility_ping']()

def execute_utility_call():
    return __utils__['general.utility_call']()

Be sure to sync:

# salt \* saltutil.sync_all

Steps to Reproduce Issue

Demonstrate the problem:

# salt \* example.execute_utility_ping
ubuntu-salt-test:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1664, in _thread_return
        return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
      File "/usr/lib/python2.7/dist-packages/salt/executors/direct_call.py", line 12, in execute
        return func(*args, **kwargs)
      File "/var/cache/salt/minion/extmods/modules/example.py", line 2, in execute_utility_ping
        return __utils__['general.utility_ping']()
      File "/var/cache/salt/minion/extmods/utils/general.py", line 2, in utility_ping
        return __salt__['test.ping']()
    NameError: global name '__salt__' is not defined
ERROR: Minions returned with non-zero exit code

# salt \* example.execute_utility_call
ubuntu-salt-test:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1664, in _thread_return
        return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
      File "/usr/lib/python2.7/dist-packages/salt/executors/direct_call.py", line 12, in execute
        return func(*args, **kwargs)
      File "/var/cache/salt/minion/extmods/modules/example.py", line 5, in execute_utility_call
        return __utils__['general.utility_call']()
      File "/var/cache/salt/minion/extmods/utils/general.py", line 6, in utility_call
        return __utils__['math.add'](2, 2)
    NameError: global name '__utils__' is not defined
ERROR: Minions returned with non-zero exit code

Versions Report

# salt --versions-report
Salt Version:
           Salt: 2019.2.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15+ (default, Oct  7 2019, 17:39:04)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 5.0.0-32-generic
         system: Linux
        version: Ubuntu 18.04 bionic

[Ch3LL]

utils need to be loaded up before __salt__ so therefore you wont have access to that dunder dict unless you load up the minion_mods yourself. you can take a look at the botomod util module as an example here: https://github.com/saltstack/salt/blob/v2019.2.2/salt/utils/botomod.py#L77-L80

looking at the loader code for utils it does not look like it packs __utils__ when loading utils so you would not be able to call __utils__ from a util module.

[jtraub91]

Can we add packing __utils__ as a feature request? It would follow the paradigm used with __salt__ cross calling and would allow for segregating modules by functionality yet allow them to be interdependent.

[max-arnold]

It is possible to cross-call core utils from custom ones just by importing them:

% grep -r 'import salt.utils' salt/utils

...
salt/utils/fsutils.py:import salt.utils.files
salt/utils/preseed.py:import salt.utils.files
salt/utils/preseed.py:import salt.utils.stringutils
salt/utils/preseed.py:import salt.utils.yaml
salt/utils/openstack/swift.py:import salt.utils.files
salt/utils/openstack/nova.py:import salt.utils.cloud
salt/utils/openstack/nova.py:import salt.utils.files
...

[jtraub91]

True, I suppose the same logic cannot be applied to custom _utils ?

[max-arnold]

I haven't tried importing custom utils, but I saw a couple of PRs that enable this use case: #52001 #53167

[jtraub91]

#52001 is cool and allows for modules in _utils to be imported in the normal python way. It would be nice to do that the same for modules; maybe that is what #53167 is about?

[Ch3LL]

Can we add packing utils as a feature request? It would follow the paradigm used with salt cross calling and would allow for segregating modules by functionality yet allow them to be interdependent.

I believe this is doable and should be able to be done similar to how we are doing it with __salt__. will approve as a feature request.