New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stale custom roster from __pycache__ executed instead of changed .py file #46924

Closed
chschmitt opened this Issue Apr 6, 2018 · 3 comments

Comments

Projects
None yet
4 participants
@chschmitt

chschmitt commented Apr 6, 2018

Description of Issue

I implemented a custom roster module. After changing the module, salt-ssh executes the stale .pyc file from the __pycache__ directory, instead of the the changed .py file.

Setup

Dummy roster implementation stale.py:

from __future__ import print_function

def targets(tgt, tgt_type='glob', **kwargs):
    print(__file__)
    return {}

Steps to Reproduce Issue

  1. Create custom roster /tmp/ext/roster/stale.py and add /tmp/ext to module_dirs in master configuration.

  2. Execute salt-ssh '*' test.ping --roster=stale, output:

    /tmp/ext/roster/stale.py
    [ERROR   ] No matching targets found in roster.
    
  3. Change roster module, e.g. insert print('file was changed')

  4. Execute salt-ssh '*' test.ping --roster=stale, change in module is not refelected. Output:

    /tmp/ext/roster/__pycache__/stale.cpython-35.pyc
    [ERROR   ] No matching targets found in roster.
    
  5. Remove __pycache__: rm -rfv /tmp/ext/roster/__pycache__

  6. Execute salt-ssh '*' test.ping --roster=stale, output:

    file was changed
    /tmp/ext/roster/stale.py
    [ERROR   ] No matching targets found in roster.
    

Workaround

Delete the __pycache__ directory and create a __pycache__ file (!) in the roster directory instead. Generally speaking, create a condition that prevents the executing Python process from creating the __pycache__ directory permanently.

Versions Report

Salt Version:
           Salt: 2018.3.0
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.5.2 (default, Nov 23 2017, 16:37:01)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 17.0.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.6
 
System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-116-generic
         system: Linux
        version: Ubuntu 16.04 xenial
@Ch3LL

This comment has been minimized.

Contributor

Ch3LL commented Apr 6, 2018

I was able to bisect this to 4951b9a which is PR: #45095

Here was the script i was using to replicate the issue:

cd ~/git/salt/
git reset --hard HEAD; git clean -dfx; sudo pip3 install -e .
sudo salt-call --local test.ping
read  -n 1 -p "Input Selection:" blah
sudo salt-call --local test.ping

during the read waiting for input i change the test.py file to return False instead of True.

@Ch3LL Ch3LL added this to the Approved milestone Apr 6, 2018

@Ch3LL

This comment has been minimized.

Contributor

Ch3LL commented Apr 6, 2018

ping @terminalmage any chance you can take a look here?

@terminalmage terminalmage self-assigned this Apr 6, 2018

pritambaral added a commit to pritambaral/salt that referenced this issue Jul 15, 2018

loader: Fix suffix order when importlib is used
On Python versions 2 and 3.0-3.4, imp.get_suffixes() is used to get the suffixes
in an ordered list, which contains '.py' before '.pyc'. On Python versions 3.5
and above, our code to construct such a list using importlib.machinery
inadvertently places '.pyc' before '.py'. This, combined with the effects of
4951b9a means that things like saltstack#46924 happen.

This fixes saltstack#46924 by constructing `SUFFIXES` in the same order
imp.get_suffixes() constructs its result.

pritambaral added a commit to pritambaral/salt that referenced this issue Jul 15, 2018

loader: Fix suffix order when importlib is used
On Python versions 2 and 3.0-3.4, imp.get_suffixes() is used to get the suffixes
in an ordered list, which contains '.py' before '.pyc'. On Python versions 3.5
and above, our code to construct such a list using importlib.machinery
inadvertently places '.pyc' before '.py'. This, combined with the effects of
4951b9a means that things like saltstack#46924 happen.

This fixes saltstack#46924 by constructing `SUFFIXES` in the same order
imp.get_suffixes() constructs its result.

pritambaral added a commit to pritambaral/salt that referenced this issue Jul 15, 2018

loader: Fix suffix order when importlib is used
On Python versions 2 and 3.0-3.4, imp.get_suffixes() is used to get the suffixes
in an ordered list, which contains '.py' before '.pyc'. On Python versions 3.5
and above, our code to construct such a list using importlib.machinery
inadvertently places '.pyc' before '.py'. This, combined with the effects of
4951b9a means that things like saltstack#46924 happen.

This fixes the issue by constructing `SUFFIXES` in the same order
imp.get_suffixes() constructs its result.
@terminalmage

This comment has been minimized.

Member

terminalmage commented Jul 18, 2018

This was fixed via #48593.

cro added a commit to cro/salt that referenced this issue Jul 26, 2018

loader: Fix suffix order when importlib is used
On Python versions 2 and 3.0-3.4, imp.get_suffixes() is used to get the suffixes
in an ordered list, which contains '.py' before '.pyc'. On Python versions 3.5
and above, our code to construct such a list using importlib.machinery
inadvertently places '.pyc' before '.py'. This, combined with the effects of
4951b9a means that things like saltstack#46924 happen.

This fixes saltstack#46924 by constructing `SUFFIXES` in the same order
imp.get_suffixes() constructs its result.

pritambaral added a commit to pritambaral/salt that referenced this issue Aug 4, 2018

loader: Fix suffix order when importlib is used
On Python versions 2 and 3.0-3.4, imp.get_suffixes() is used to get the suffixes
in an ordered list, which contains '.py' before '.pyc'. On Python versions 3.5
and above, our code to construct such a list using importlib.machinery
inadvertently places '.pyc' before '.py'. This, combined with the effects of
4951b9a means that things like saltstack#46924 happen.

This fixes the issue by constructing `SUFFIXES` in the same order
imp.get_suffixes() constructs its result.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment