_utils modules have no access to __salt__ or __utils__

[jtraub91]

Description of Issue

Putting utility functions that aren't intended to be run via the cli in _utils is a good way to organize code, however, its utility is severely limited by the fact that you cannot cross-call execution nor utility modules (via __salt__ / __utils__)

Setup

Normal salt installation. Create some example modules to demonstrate.

# /srv/salt/_utils/general.py

def utility_ping():
    return __salt__['test.ping']()


def utility_call():
    return __utils__['math.add'](2, 2)

# /srv/salt/_utils/math.py

def add(num1, num2):
    return num1 + num2

# /srv/salt/_modules/example.py

def execute_utility_ping():
    return __utils__['general.utility_ping']()

def execute_utility_call():
    return __utils__['general.utility_call']()

Be sure to sync:

# salt \* saltutil.sync_all

Steps to Reproduce Issue

Demonstrate the problem:

# salt \* example.execute_utility_ping
ubuntu-salt-test:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1664, in _thread_return
        return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
      File "/usr/lib/python2.7/dist-packages/salt/executors/direct_call.py", line 12, in execute
        return func(*args, **kwargs)
      File "/var/cache/salt/minion/extmods/modules/example.py", line 2, in execute_utility_ping
        return __utils__['general.utility_ping']()
      File "/var/cache/salt/minion/extmods/utils/general.py", line 2, in utility_ping
        return __salt__['test.ping']()
    NameError: global name '__salt__' is not defined
ERROR: Minions returned with non-zero exit code

# salt \* example.execute_utility_call
ubuntu-salt-test:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1664, in _thread_return
        return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
      File "/usr/lib/python2.7/dist-packages/salt/executors/direct_call.py", line 12, in execute
        return func(*args, **kwargs)
      File "/var/cache/salt/minion/extmods/modules/example.py", line 5, in execute_utility_call
        return __utils__['general.utility_call']()
      File "/var/cache/salt/minion/extmods/utils/general.py", line 6, in utility_call
        return __utils__['math.add'](2, 2)
    NameError: global name '__utils__' is not defined
ERROR: Minions returned with non-zero exit code

Versions Report

# salt --versions-report
Salt Version:
           Salt: 2019.2.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15+ (default, Oct  7 2019, 17:39:04)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 5.0.0-32-generic
         system: Linux
        version: Ubuntu 18.04 bionic

[Ch3LL]

utils need to be loaded up before __salt__ so therefore you wont have access to that dunder dict unless you load up the minion_mods yourself. you can take a look at the botomod util module as an example here: https://github.com/saltstack/salt/blob/v2019.2.2/salt/utils/botomod.py#L77-L80

looking at the loader code for utils it does not look like it packs __utils__ when loading utils so you would not be able to call __utils__ from a util module.

[jtraub91]

Can we add packing __utils__ as a feature request? It would follow the paradigm used with __salt__ cross calling and would allow for segregating modules by functionality yet allow them to be interdependent.

[max-arnold]

It is possible to cross-call core utils from custom ones just by importing them:

% grep -r 'import salt.utils' salt/utils

...
salt/utils/fsutils.py:import salt.utils.files
salt/utils/preseed.py:import salt.utils.files
salt/utils/preseed.py:import salt.utils.stringutils
salt/utils/preseed.py:import salt.utils.yaml
salt/utils/openstack/swift.py:import salt.utils.files
salt/utils/openstack/nova.py:import salt.utils.cloud
salt/utils/openstack/nova.py:import salt.utils.files
...

[jtraub91]

True, I suppose the same logic cannot be applied to custom _utils ?

[max-arnold]

I haven't tried importing custom utils, but I saw a couple of PRs that enable this use case: #52001 #53167

[jtraub91]

#52001 is cool and allows for modules in _utils to be imported in the normal python way. It would be nice to do that the same for modules; maybe that is what #53167 is about?

[Ch3LL]

Can we add packing utils as a feature request? It would follow the paradigm used with salt cross calling and would allow for segregating modules by functionality yet allow them to be interdependent.

I believe this is doable and should be able to be done similar to how we are doing it with __salt__. will approve as a feature request.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.