Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

_utils modules have no access to __salt__ or __utils__ #55232

Closed
jtraub91 opened this issue Nov 7, 2019 · 8 comments
Closed

_utils modules have no access to __salt__ or __utils__ #55232

jtraub91 opened this issue Nov 7, 2019 · 8 comments
Assignees
Milestone

Comments

@jtraub91
Copy link
Contributor

@jtraub91 jtraub91 commented Nov 7, 2019

Description of Issue

Putting utility functions that aren't intended to be run via the cli in _utils is a good way to organize code, however, its utility is severely limited by the fact that you cannot cross-call execution nor utility modules (via __salt__ / __utils__)

Setup

Normal salt installation. Create some example modules to demonstrate.

# /srv/salt/_utils/general.py

def utility_ping():
    return __salt__['test.ping']()


def utility_call():
    return __utils__['math.add'](2, 2)

# /srv/salt/_utils/math.py

def add(num1, num2):
    return num1 + num2

# /srv/salt/_modules/example.py

def execute_utility_ping():
    return __utils__['general.utility_ping']()

def execute_utility_call():
    return __utils__['general.utility_call']()

Be sure to sync:

# salt \* saltutil.sync_all

Steps to Reproduce Issue

Demonstrate the problem:

# salt \* example.execute_utility_ping
ubuntu-salt-test:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1664, in _thread_return
        return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
      File "/usr/lib/python2.7/dist-packages/salt/executors/direct_call.py", line 12, in execute
        return func(*args, **kwargs)
      File "/var/cache/salt/minion/extmods/modules/example.py", line 2, in execute_utility_ping
        return __utils__['general.utility_ping']()
      File "/var/cache/salt/minion/extmods/utils/general.py", line 2, in utility_ping
        return __salt__['test.ping']()
    NameError: global name '__salt__' is not defined
ERROR: Minions returned with non-zero exit code
# salt \* example.execute_utility_call
ubuntu-salt-test:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1664, in _thread_return
        return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
      File "/usr/lib/python2.7/dist-packages/salt/executors/direct_call.py", line 12, in execute
        return func(*args, **kwargs)
      File "/var/cache/salt/minion/extmods/modules/example.py", line 5, in execute_utility_call
        return __utils__['general.utility_call']()
      File "/var/cache/salt/minion/extmods/utils/general.py", line 6, in utility_call
        return __utils__['math.add'](2, 2)
    NameError: global name '__utils__' is not defined
ERROR: Minions returned with non-zero exit code

Versions Report

# salt --versions-report
Salt Version:
           Salt: 2019.2.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15+ (default, Oct  7 2019, 17:39:04)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 5.0.0-32-generic
         system: Linux
        version: Ubuntu 18.04 bionic

@Ch3LL

This comment has been minimized.

Copy link
Contributor

@Ch3LL Ch3LL commented Nov 7, 2019

utils need to be loaded up before __salt__ so therefore you wont have access to that dunder dict unless you load up the minion_mods yourself. you can take a look at the botomod util module as an example here: https://github.com/saltstack/salt/blob/v2019.2.2/salt/utils/botomod.py#L77-L80

looking at the loader code for utils it does not look like it packs __utils__ when loading utils so you would not be able to call __utils__ from a util module.

@Ch3LL Ch3LL added this to the Blocked milestone Nov 7, 2019
@jtraub91

This comment has been minimized.

Copy link
Contributor Author

@jtraub91 jtraub91 commented Nov 7, 2019

Can we add packing __utils__ as a feature request? It would follow the paradigm used with __salt__ cross calling and would allow for segregating modules by functionality yet allow them to be interdependent.

@max-arnold

This comment has been minimized.

Copy link
Contributor

@max-arnold max-arnold commented Nov 8, 2019

It is possible to cross-call core utils from custom ones just by importing them:

% grep -r 'import salt.utils' salt/utils

...
salt/utils/fsutils.py:import salt.utils.files
salt/utils/preseed.py:import salt.utils.files
salt/utils/preseed.py:import salt.utils.stringutils
salt/utils/preseed.py:import salt.utils.yaml
salt/utils/openstack/swift.py:import salt.utils.files
salt/utils/openstack/nova.py:import salt.utils.cloud
salt/utils/openstack/nova.py:import salt.utils.files
...
@jtraub91

This comment has been minimized.

Copy link
Contributor Author

@jtraub91 jtraub91 commented Nov 8, 2019

True, I suppose the same logic cannot be applied to custom _utils ?

@max-arnold

This comment has been minimized.

Copy link
Contributor

@max-arnold max-arnold commented Nov 8, 2019

I haven't tried importing custom utils, but I saw a couple of PRs that enable this use case: #52001 #53167

@jtraub91

This comment has been minimized.

Copy link
Contributor Author

@jtraub91 jtraub91 commented Nov 10, 2019

#52001 is cool and allows for modules in _utils to be imported in the normal python way. It would be nice to do that the same for modules; maybe that is what #53167 is about?

@Ch3LL

This comment has been minimized.

Copy link
Contributor

@Ch3LL Ch3LL commented Dec 11, 2019

Can we add packing utils as a feature request? It would follow the paradigm used with salt cross calling and would allow for segregating modules by functionality yet allow them to be interdependent.

I believe this is doable and should be able to be done similar to how we are doing it with __salt__. will approve as a feature request.

@Ch3LL Ch3LL modified the milestones: Blocked, Approved Dec 11, 2019
@Ch3LL Ch3LL self-assigned this Dec 11, 2019
@stale

This comment has been minimized.

Copy link

@stale stale bot commented Jan 10, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

@stale stale bot added the stale label Jan 10, 2020
@stale stale bot closed this Jan 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.