custom execution modules not working

[syntax-terr0r]

Describe the bug
Custom execution modules placed in the salt://_modules directory can not be executed.

Steps To Reproduce

1. create custom module:

root@salt-gbone:/srv/salt# cat _modules/example.py
def first():
    return True

2. try to run the module:

root@salt-gbone:/srv/salt# salt-sproxy cr-testing* example.first
cr-testing01.lab1:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/local/lib/python3.7/dist-packages/salt_sproxy/_runners/proxy.py", line 641, in salt_call
        ret = sa_proxy.functions[salt_function](*args, **kwargs)
      File "/usr/lib/python3/dist-packages/salt/loader.py", line 1278, in __getitem__
        func = super(LazyLoader, self).__getitem__(item)
      File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
        raise KeyError(key)
    KeyError: 'example.first'
ERROR: Minions returned with non-zero exit code

Expected behavior
Expect the custom module to execute and return "True"

Versions Report

root@salt-gbone:/srv/salt# salt-sproxy -V
Salt Version:
           Salt: 3001.1
    Salt SProxy: 2020.7.0

Dependency Versions:
        Ansible: Not Installed
           cffi: 1.14.2
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: 2.0.5
      gitpython: 2.1.11
         Jinja2: 2.10
     junos-eznc: 2.5.3
       jxmlease: Not Installed
        libgit2: 0.27.7
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
         NAPALM: 3.1.0
       ncclient: 0.6.9
        Netmiko: 3.2.0
       paramiko: 2.7.2
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: 3.6.1
         pyeapi: 0.8.3
         pygit2: 0.27.4
       PyNetBox: Not Installed
          PyNSO: Not Installed
         Python: 3.7.3 (default, Jul 25 2020, 13:03:44)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 17.1.2
            scp: 0.13.2
          smmap: 2.0.5
        textfsm: 1.1.0
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1

System Versions:
           dist: debian 10 buster
         locale: utf-8
        machine: x86_64
        release: 4.19.0-10-amd64
         system: Linux
        version: Debian GNU/Linux 10 buster

Additional context
I followed the documentation for creating custom execution modules here:
https://docs.saltstack.com/en/latest/ref/modules/
as well as your blog:
https://mirceaulinic.net/2019-04-24-extending-napalm-salt/
https://mirceaulinic.net/2019-06-17-minionless-salt-automation/

This is my file_roots config:

file_roots:
  base:
    - /srv/salt/
    - /srv/salt/ext

Running standard modules on the minion works fine:

root@salt-gbone:/srv/salt# salt-sproxy cr-testing* test.ping
cr-testing01.lab1:
    True

Running saltutil.sync_all, saltutil.sync_modules, sys.reload_modules or saltutil.refresh_modules does not change the outcome. As I understand, running these commands should also not be necessary with salt-sproxy because the modules are loaded on runtime.

The only way I have found to make this work is to place the module either in
/usr/lib/python3/dist-packages/salt/modules/ or /usr/local/lib/python3.7/dist-packages/salt_sproxy/_modules/:

root@salt-gbone:/srv/salt# mv _modules/example.py /usr/local/lib/python3.7/dist-packages/salt_sproxy/_modules/
root@salt-gbone:/srv/salt# salt-sproxy cr-testing* example.first
cr-testing01.lab1:
    True

I'm relatively new to salt and salt-sproxy so I can't rule out that I'm missing something simple here.
However I'm unable to figure out what it might be.

[mirceaulinic]

Hi. Can you try with salt-sproxy cr-testing* example.first --sync-modules? Alternatively, you can put sync_modules: true on your Master, or best, have a (scheduled?) salt-run saltutil.sync_modules.

[syntax-terr0r]

Of course:

root@salt-gbone:/srv/salt# salt-sproxy cr-testing* example.first --sync-modules
cr-testing01.lab1:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/local/lib/python3.7/dist-packages/salt_sproxy/_runners/proxy.py", line 641, in salt_call
        ret = sa_proxy.functions[salt_function](*args, **kwargs)
      File "/usr/lib/python3/dist-packages/salt/loader.py", line 1278, in __getitem__
        func = super(LazyLoader, self).__getitem__(item)
      File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
        raise KeyError(key)
    KeyError: 'example.first'
ERROR: Minions returned with non-zero exit code

root@salt-gbone:/srv/salt# salt-run saltutil.sync_modules
- modules.example

root@salt-gbone:/srv/salt# salt-sproxy cr-testing* example.first
cr-testing01.lab1:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/local/lib/python3.7/dist-packages/salt_sproxy/_runners/proxy.py", line 641, in salt_call
        ret = sa_proxy.functions[salt_function](*args, **kwargs)
      File "/usr/lib/python3/dist-packages/salt/loader.py", line 1278, in __getitem__
        func = super(LazyLoader, self).__getitem__(item)
      File "/usr/lib/python3/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
        raise KeyError(key)
    KeyError: 'example.first'
ERROR: Minions returned with non-zero exit code

Result is the same though.
As I said in the initial statement, none of the other methods for updating or syncing modules that I could find worked either.

[mirceaulinic]

If it doesn't show up in salt-run saltutil.sync_modules, it's likely a misconfiguration - you should be able to at least see it in there.

Nevermind, it is in there... 🙄

[mirceaulinic]

@syntax-terr0r could you try #177 to confirm it fixes this? If you're unable to apply the patch, just setting kwargs['sync_modules'] = sync_modules on line 199 in salt_sproxy/cli.py should normally do it.

[syntax-terr0r]

I can confirm that setting kwargs['sync_modules'] = sync_modules fixed it:

root@salt-gbone:~# salt-sproxy cr-testing* example.first
cr-testing01.mydomain.com:
    True

Thanks for the quick fix!