Python3 can't override core module

[jvdbeemt]

We were investigating if we could take the step to python3. Our first tests were fine.
But we found a blocking issue. If we use a internal module, let's say network. We override one function, sync this to a minion. It wont be picked up. If we use python2.7 this works:

Python2.7

Example module:
/srv/salt/states/_modules
jurgen@jurgen-HP-ZBook-15u-G3:/srv/salt/states/_modules$
cat test.py

import salt
import logging
import json

try:
    import requests
    HAS_LIBS = True
except ImportError:
    HAS_LIBS = False

__virtualname__ = 'test'

def __virtual__():
    '''
    Only load if requests is installed
    '''
    if HAS_LIBS:
        return __virtualname__
    else:
        return False, 'The \'{0}\' module could not be loaded: \'requests\' is not installed.'.format(__virtualname__)


def spam(eggs):
    '''
    A function to make some spam with eggs!

    CLI Example::

        salt '*' test.spam eggs
    '''
    return eggs

sudo salt-call saltutil.sync_all -l

local:
    ----------
    beacons:
    clouds:
    engines:
    grains:
    log_handlers:
    modules:
        - modules.test
    output:
    pillar:
    proxymodules:
    renderers:
    returners:
    sdb:
    states:
    thorium:
    utils:

sudo salt-call test.spam eggs 
local:
    eggs

python3.5
Same as above with the following end result:

sudo salt-call test.spam eggs 
'test.spam' is not available.

Salt Version:
Salt: 2018.3.0

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.4.2
docker-py: Not Installed
gitdb: 0.6.4
gitpython: 1.0.1
ioflo: Not Installed
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.3
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.12 (default, Dec 4 2017, 14:50:18) pyton3.5 if we use the py3 repo
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: 0.9.0
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: Ubuntu 16.04 xenial
locale: UTF-8
machine: x86_64
release: 4.13.0-38-generic
system: Linux
version: Ubuntu 16.04 xenial

[Ch3LL]

this is working fine for me on the version report:

Salt Version:                                                                                                                                                                  
           Salt: 2018.3.0                                                                                                                                                      
                                                                                                                                                                               
Dependency Versions:                                                                                                                                                           
           cffi: 1.11.2                                                                                                                                                        
       cherrypy: Not Installed                                                                                                                                                 
       dateutil: 2.6.1                                                                                                                                                         
      docker-py: 2.7.0                                                                                                                                                         
          gitdb: Not Installed                                                                                                                                                 
      gitpython: Not Installed                                                                                                                                                 
          ioflo: Not Installed                                                                                                                                                 
         Jinja2: 2.10                                                                                                                                                          
        libgit2: Not Installed                                                                                                                                                 
        libnacl: Not Installed                                                                                                                                                 
       M2Crypto: Not Installed                                                                                                                                                 
           Mako: 1.0.7                                                                                                                                                         
   msgpack-pure: Not Installed                                                                                                                                                 
 msgpack-python: 0.4.8                                                                                                                                                         
   mysql-python: 1.3.12                                                                                                                                                        
      pycparser: 2.18                                                                                                                                                          
       pycrypto: 2.6.1                                                                                                                                                         
   pycryptodome: Not Installed                                                                                                                                                 
         pygit2: Not Installed                                                                                                                                                 
         Python: 3.6.4 (default, Dec 23 2017, 19:07:07)                                                                                                                        
   python-gnupg: Not Installed                                                                                                                                                 
         PyYAML: 3.12                                                                                                                                                          
          PyZMQ: 16.0.3                                                                                                                                                        
           RAET: Not Installed                                                                                                                                                 
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.2
            ZMQ: 4.1.6

System Versions:
           dist: arch Manjaro Linux
         locale: UTF-8
        machine: x86_64
        release: 4.11.12-1-MANJARO
         system: Linux
        version: arch Manjaro Linux

➜  _modules sudo salt-call test.spam eggs -lwarning
local:
    eggs

How are you installing python3? Maybe there is some conflicting pycache when installing between the two versions

[jvdbeemt]

After some investigation the above indeed works. Thx:) Except if you try to override an existing function of test, it won't work with python3.5. Python2.7 will allow overriding. Can you confirm?

[jvdbeemt]

Can you please respond to my question? We would really like to use python3. Thanks in advance.

[Ch3LL]

this also worked for me:

Here is the test.py module i used:

import salt
import logging
import json

try:
    import requests
    HAS_LIBS = True
except ImportError:
    HAS_LIBS = False

__virtualname__ = 'test'

def __virtual__():
    '''
    Only load if requests is installed
    '''
    if HAS_LIBS:
        return __virtualname__
    else:
        return False, 'The \'{0}\' module could not be loaded: \'requests\' is not installed.'.format(__virtualname__)


def spam(eggs):
    '''
    A function to make some spam with eggs!

    CLI Example::

        salt '*' test.spam eggs
    '''
    return eggs

def ping():
    '''
    A function to make some spam with eggs!

    CLI Example::

        salt '*' test.spam eggs
    '''
    return 'itworked'

and when I run it it returns itworked:

➜  salt git:(e745639763) sudo salt '*cake*' test.ping
the_cake_is_a_lie:
    itworked
➜  salt git:(e745639763) 

but i bet you are running into this issue: #46924

as a workaround if you clear your pycache after syncing the module does it work?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.