New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Salt minion 2017.7.6 disregards "bin_env" in pip state #48122

Closed
pasmon opened this Issue Jun 14, 2018 · 7 comments

Comments

@pasmon

pasmon commented Jun 14, 2018

Description of Issue/Question

Python modules fail to install for Python 3 since pip state doesn't seem to make use of "bin_env" defined in the state file.

Setup

flake8.sls:

install-python2-flake8:
  pip.installed:
    - name: flake8

install-python3-flake8:
  pip.installed:
    - name: flake8
    - bin_env: '/bin/pip3'

Steps to Reproduce Issue

Install Salt minion version 2017.7.6 and run the state, Salt fails to install the Python 3 version of the flake8:

2018-06-14 04:19:51,353 [salt.state       :1770][INFO    ][1738] Running state [flake8] at time 04:19:51.353488
2018-06-14 04:19:51,356 [salt.state       :1803][INFO    ][1738] Executing state pip.installed for [flake8]
2018-06-14 04:19:51,359 [salt.loaded.int.module.cmdmod:395 ][INFO    ][1738] Executing command ['/usr/bin/python', '-m', 'pip', 'freeze', '--all'] in directory '/root'
2018-06-14 04:19:51,739 [salt.loaded.int.module.cmdmod:395 ][INFO    ][1738] Executing command ['/usr/bin/python', '-m', 'pip', 'install', 'flake8'] in directory '/root'
2018-06-14 04:19:55,238 [salt.loaded.int.module.cmdmod:395 ][INFO    ][1738] Executing command ['/usr/bin/python', '-m', 'pip', '--version'] in directory '/root'
2018-06-14 04:19:55,488 [salt.loaded.int.module.cmdmod:395 ][INFO    ][1738] Executing command ['/usr/bin/python', '-m', 'pip', 'freeze', '--all'] in directory '/root'
2018-06-14 04:19:55,925 [salt.state       :290 ][INFO    ][1738] {'flake8==3.5.0': 'Installed'}
2018-06-14 04:19:55,926 [salt.state       :1941][INFO    ][1738] Completed state [flake8] at time 04:19:55.926176 duration_in_ms=4572.688
2018-06-14 04:19:55,927 [salt.state       :1770][INFO    ][1738] Running state [flake8] at time 04:19:55.927579
2018-06-14 04:19:55,928 [salt.state       :1803][INFO    ][1738] Executing state pip.installed for [flake8]
2018-06-14 04:19:55,930 [salt.loaded.int.module.cmdmod:395 ][INFO    ][1738] Executing command ['/bin/python', '-m', 'pip', 'freeze', '--all'] in directory '/root'
2018-06-14 04:19:56,330 [salt.state       :290 ][INFO    ][1738] Python package flake8 was already installed
All specified packages are already installed
2018-06-14 04:19:56,331 [salt.state       :1941][INFO    ][1738] Completed state [flake8] at time 04:19:56.330461 duration_in_ms=402.88

Salt minion 2017.7.5 uses pip directly instead of "python -m pip" and with that version the installation works:

2018-06-14 11:00:01,843 [salt.state       :1761][INFO    ][19930] Running state [flake8] at time 11:00:01.843667
2018-06-14 11:00:01,843 [salt.state       :1794][INFO    ][19930] Executing state pip.installed for [flake8]
2018-06-14 11:00:01,845 [salt.loaded.int.module.cmdmod:394 ][INFO    ][19930] Executing command '/usr/bin/pip2.7 --version' in directory '/root'
2018-06-14 11:00:02,314 [salt.loaded.int.module.cmdmod:394 ][INFO    ][19930] Executing command ['/usr/bin/pip2.7', 'freeze', '--all'] in directory '/root'
2018-06-14 11:00:02,610 [requests.packages.urllib3.connectionpool:214 ][INFO    ][21228] Starting new HTTP connection (1): 169.254.169.254
2018-06-14 11:00:02,620 [requests.packages.urllib3.connectionpool:214 ][INFO    ][21228] Starting new HTTP connection (1): 169.254.169.254
2018-06-14 11:00:02,629 [requests.packages.urllib3.connectionpool:214 ][INFO    ][21228] Starting new HTTP connection (1): 169.254.169.254
2018-06-14 11:00:02,995 [salt.loaded.int.module.cmdmod:394 ][INFO    ][19930] Executing command ['/usr/bin/pip2.7', 'install'] in directory '/root'
2018-06-14 11:00:03,511 [salt.state       :290 ][INFO    ][19930] Python package flake8 was already installed
All packages were successfully installed
2018-06-14 11:00:03,512 [salt.state       :1934][INFO    ][19930] Completed state [flake8] at time 11:00:03.512029 duration_in_ms=1668.362
2018-06-14 11:00:03,512 [salt.state       :1761][INFO    ][19930] Running state [flake8] at time 11:00:03.512364
2018-06-14 11:00:03,512 [salt.state       :1794][INFO    ][19930] Executing state pip.installed for [flake8]
2018-06-14 11:00:03,514 [salt.loaded.int.module.cmdmod:394 ][INFO    ][19930] Executing command '/bin/pip3 --version' in directory '/root'
2018-06-14 11:00:03,963 [salt.loaded.int.module.cmdmod:394 ][INFO    ][19930] Executing command ['/bin/pip3', 'freeze', '--all'] in directory '/root'
2018-06-14 11:00:04,573 [salt.loaded.int.module.cmdmod:394 ][INFO    ][19930] Executing command ['/bin/pip3', 'install', 'flake8'] in directory '/root'
2018-06-14 11:00:06,392 [salt.loaded.int.module.cmdmod:394 ][INFO    ][19930] Executing command '/bin/pip3 --version' in directory '/root'
2018-06-14 11:00:06,804 [salt.loaded.int.module.cmdmod:394 ][INFO    ][19930] Executing command ['/bin/pip3', 'freeze', '--all'] in directory '/root'
2018-06-14 11:00:07,278 [salt.state       :290 ][INFO    ][19930] {'flake8==3.5.0': 'Installed'}
2018-06-14 11:00:07,278 [salt.state       :1934][INFO    ][19930] Completed state [flake8] at time 11:00:07.278668 duration_in_ms=3766.303

Versions Report

Salt Version:
           Salt: 2017.7.6
 
Dependency Versions:
           cffi: 1.5.2
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8.1
        libgit2: 0.27.0
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.1
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.27.0
         Python: 2.7.5 (default, Feb 20 2018, 09:19:12)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: redhat 7.5 Maipo
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-862.el7.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 7.5 Maipo


@gtmanfred

This comment has been minimized.

Contributor

gtmanfred commented Jun 14, 2018

This is only broken when using the system pip3, if you use a virtualenv to install, it will use the correct python from the virtualenv, and -m pip, so it will work there.

This change was made to allow the pip module to be able to upgrade itself.

            if 'pip' in os.path.basename(bin_env):
                # Look in the same directory as the pip binary, and also its
                # parent directories.
                pip_dirname = os.path.dirname(bin_env)
                pip_parent_dir = os.path.dirname(pip_dirname)
                for bin_path in _search_paths(pip_dirname, pip_parent_dir):
                    if os.path.isfile(bin_path):
                        logger.debug('pip: Found python binary: %s', bin_path)
                        return [os.path.normpath(bin_path), '-m', 'pip']

I am marking this as a blocker for 2018.3.3, to fix this behavior.

(normally it would be 2018.3.2, but that is going to be a very narrow release to fix something in the core of salt with syndics.)

@teran-mckinney

This comment has been minimized.

teran-mckinney commented Jun 15, 2018

This is also a blocker for me. I hope 2018.3.3 is ready soon, will stick with 2018.3.0 for now.

@gtmanfred

This comment has been minimized.

Contributor

gtmanfred commented Jun 16, 2018

@rallytime

This comment has been minimized.

Contributor

rallytime commented Jun 18, 2018

@gtmanfred Can you get this fixed up for the 2018.3.3 release?

@gtmanfred

This comment has been minimized.

Contributor

gtmanfred commented Jun 18, 2018

Yeah.

gtmanfred added a commit to gtmanfred/salt that referenced this issue Jun 18, 2018

If pip binary is passed to bin_env, use that pip binary
We don't need to do a lookup for python to then run `python -m pip`
because we already know where pip is.

Fixes using /bin/pip3

Fixes saltstack#48122

gtmanfred added a commit to gtmanfred/salt that referenced this issue Jun 18, 2018

If pip binary is passed to bin_env, use that pip binary
We don't need to do a lookup for python to then run `python -m pip`
because we already know where pip is.

Fixes using /bin/pip3

Fixes saltstack#48122
@gtmanfred

This comment has been minimized.

Contributor

gtmanfred commented Jun 18, 2018

This will be fixed by #48189

Just waiting on verifying that this doesn't break anything on windows.

gtmanfred added a commit to gtmanfred/salt that referenced this issue Jun 18, 2018

If pip binary is passed to bin_env, use that pip binary
We don't need to do a lookup for python to then run `python -m pip`
because we already know where pip is.

Fixes using /bin/pip3

Fixes saltstack#48122

rallytime added a commit to rallytime/salt that referenced this issue Jun 19, 2018

If pip binary is passed to bin_env, use that pip binary
We don't need to do a lookup for python to then run `python -m pip`
because we already know where pip is.

Fixes using /bin/pip3

Fixes saltstack#48122

@rallytime rallytime closed this Jun 27, 2018

@teran-mckinney

This comment has been minimized.

teran-mckinney commented Jun 27, 2018

Fantastic. Thank you, everyone!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment