pip state cannot install packages on Windows minions

[otrempe]

With python/mako.sls file being:

mako:
    pip.installed

salt -G 'os:Windows' state.sls python.mako

gives

----------
    State: - pip
    Name:      mako
    Function:  install
        Result:    False
        Comment:   State pip.installed found in sls python.mako is unavailable

        Changes:
                   ----------

Also,

salt -G 'os:Windows' pip.list

gives

    ----------
    pip:
        1.4.1
    pywin32:
        218

I tested all of this on a linux minion and it all works well.

[otrempe]

Oops, forgot to specify versions:

Master:

           Salt: 0.17.0
         Python: 2.7.4 (default, Sep 26 2013, 03:20:26)
         Jinja2: 2.6
       M2Crypto: 0.21.1
 msgpack-python: 0.2.0
   msgpack-pure: Not Installed
       pycrypto: 2.6
         PyYAML: 3.10
          PyZMQ: 13.0.0
            ZMQ: 3.2.2

Minion:

           Salt: 0.17.1
         Python: 2.7.5 (default, May 15 2013, 22:43:36) [MSC v.1500 32 bit (Intel)]
         Jinja2: 2.7.1
       M2Crypto: 0.21.1
 msgpack-python: 0.3.0
   msgpack-pure: Not Installed
       pycrypto: 2.6
         PyYAML: 3.10
          PyZMQ: 13.1.0
            ZMQ: 3.2.2

[basepi]

@UtahDave is the pip state even Windows compatible? I'm unsure.

Thanks for the report, we'll look into it.

[UtahDave]

Pip itself works great on Windows. The only issue is that we freeze python and all Salt's deps for the Windows deployment. We could definitely add pip to the freeze list, but I'm not sure if pip will work with the frozen python install. I'll have to do some testing.

[UtahDave]

I'm assigning this ticket to me and assigning it to Hydrogen. It should be pretty quick to tell if it's going to work or not.

[otrempe]

This ticked contains 2 unrelated issues so I moved information about the pip module issue in a new ticket: #9592

[otrempe]

I did some tests to realize it is not possible to import a regularly installed python modules (in python's site packages) into a salt module freezed in the installer so installing pip on a Windows minion doesn't fix the issue... unfortunately.

[whardier]

Since the minion uses it's frozen python interpreter how would I go about adding to it or creating modules that can use external dependencies (especially if there's any c extensions)

[UtahDave]

You can drop an egg in along with the other eggs that come with the frozen Salt install.

[nickva]

Would there be any use to provide a pip.bootstrap (especially for Windows) like there is a chocolatey.bootstrap? Basically something that would make it simple to go from 'install Python' step to 'start installing packages using pip' step?

[UtahDave]

@nickva sure!

[blaffoy]

Is there any movement on this issue? I'm still seeing the same problem on 2014.1.13 and 2014.7.0

foo.sls

devpi:
  pip.install

PS C:\> salt-call state.sls foo

----------
          ID: devpi
    Function: pip.install
      Result: False
     Comment: State 'pip.install' found in SLS 'foo' is unavailable
     Started:
    Duration:
     Changes:

[blaffoy]

I tried implementing the workaround described above, copying the egg distribution from another python installation into the salt python distribution, but it seems not to have worked.

With my main python installed at C:\Python27\, and a pip egg at C:\Python27\Lib\site-packages\pip-6.0.6-py2.7.egg, and salt installed to C:\salt\, I tried:

PS C:\> cp -Recurse C:\Python27\Lib\site-packages\pip-6.0.6-py2.7.egg\ .\salt-2014.7.0.win32\
PS C:\> Get-Service salt-minion | Restart-Service

But I continue to see the same error.

I also tried copying the modified pip.py provided by Ollie Walsh here to C:\salt\salt-2014.7.0.win32\salt-2014.7.0-py2.7.egg\salt\modules, which also didn't help.

Am I misunderstanding how these workarounds should work?

[blaffoy]

After reviewing the code in salt/states/pip_state.py, it seems that the only calls to the imported python pip module are to the method pip.req.InstallRequirement.from_line in the helper function _check_pkg_version_format.

Other than that, pip_state.py relies only[1] on calls to the pip execution module. It seems to me, that if we could implement the InstallRequirement check without using pip, then that would completely break the dependency, without the need to import pip from Salt's own Windows deployment.

Does that make sense?

In the meantime, for my own purposes, I have made that function return simply the name and the version as specified in the calling sls, eliminating the calls to pip.req.InstallRequirement.from_line altogether, and also removing the check for import pip, pip.req at the start.

[1] excepting a single call to pip.__version__ that could be replaced with a __salt__['pip.version']

[blaffoy]

Another note on this, just in case anybody is trying to do the same thing that I am. The pip execution module does some text processing that leads to the command line array looking like:

"pip", "install", "--use-wheel", "--index-url='http://pypi/root/prod'", "--upgrade", "'Mako'"

Note the double-then-single quotation marks around the PyPI index and the package name. This is causing errors in Windows that aren't apparent when running the same thing in Linux.

(foobar) C:\env\foobar\Scripts>pip.exe install --use-wheel --index-url='http://pypi/root/prod' --upgrade 'Mako'
Exception:
Traceback (most recent call last):
  File "C:\env\foobar\lib\site-packages\pip\basecommand.py", line 122, in main
    status = self.run(options, args)
  File "C:\env\foobar\lib\site-packages\pip\commands\install.py", line 257, in run
    InstallRequirement.from_line(name, None))
  File "C:\env\foobar\lib\site-packages\pip\req.py", line 172, in from_line
    return cls(req, comes_from, url=url, prereleases=prereleases)
  File "C:\env\foobar\lib\site-packages\pip\req.py", line 70, in __init__
    req = pkg_resources.Requirement.parse(req)
  File "C:\env\foobar\lib\site-packages\pip\_vendor\pkg_resources.py", line 2667, in parse
    reqs = list(parse_requirements(s))
  File "C:\env\foobar\lib\site-packages\pip\_vendor\pkg_resources.py", line 2593, in parse_requirements
    raise ValueError("Missing distribution spec", line)
ValueError: ('Missing distribution spec', "'Mako'")

and

(foobar) C:\env\foobar\Scripts>pip.exe install --use-wheel --index-url='http://pypi/root/prod' --upgrade Mako
Cleaning up...
Exception:
Traceback (most recent call last):
  File "C:\env\foobar\lib\site-packages\pip\basecommand.py", line 122, in main
    status = self.run(options, args)
  File "C:\env\foobar\lib\site-packages\pip\commands\install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "C:\env\foobar\lib\site-packages\pip\req.py", line 1096, in prepare_files
    req_to_install, self.upgrade)
  File "C:\env\foobar\lib\site-packages\pip\index.py", line 194, in find_requirement
    page = self._get_page(main_index_url, req)
  File "C:\env\foobar\lib\site-packages\pip\index.py", line 568, in _get_page
    session=self.session,
  File "C:\env\foobar\lib\site-packages\pip\index.py", line 670, in get_page
    resp = session.get(url, headers={"Accept": "text/html"})
  File "C:\env\foobar\lib\site-packages\pip\_vendor\requests\sessions.py", line 468, in get
    return self.request('GET', url, **kwargs)
  File "C:\env\foobar\lib\site-packages\pip\download.py", line 237, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "C:\env\foobar\lib\site-packages\pip\_vendor\requests\sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\env\foobar\lib\site-packages\pip\_vendor\requests\sessions.py", line 553, in send
    adapter = self.get_adapter(url=request.url)
  File "C:\env\foobar\lib\site-packages\pip\_vendor\requests\sessions.py", line 608, in get_adapter
    raise InvalidSchema("No connection adapters were found for '%s'" % url)
InvalidSchema: No connection adapters were found for ''http://pypi/root/prod'/Mako/'

Where a call like this will work:

(foobar) C:\env\foobar\Scripts>pip.exe install --use-wheel --index-url=http://pypi/root/prod --upgrade Mako
http://pypi/root/prod/Mako/ uses an insecure transport scheme (http). Consider using https if pypi has it available
Requirement already up-to-date: Mako in c:\python27\lib\site-packages
http://pypi/root/prod/MarkupSafe/ uses an insecure transport scheme (http). Consider using https if pypi has it available
Requirement already up-to-date: MarkupSafe>=0.9.2 in c:\python27\lib\site-packages (from Mako)
Cleaning up...

The pip execution module does some text processing with the arguments to the install call like

  cmd.append('--index-url={0!r}'.format(index_url))

Using {0!r} results in a string formatted as "'http://pypi/root/prod'" (note double-then-single quotes) by calling repr. I've worked around this problem by doing this string formatting using a simpler {0}.format()