New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Removal of escaping in shebang in pip install causes "failed to create process" on Windows #398

Closed
bb-migration opened this Issue Jun 19, 2015 · 12 comments

Comments

Projects
None yet
4 participants
@bb-migration

bb-migration commented Jun 19, 2015

Originally reported by: joeyespo (Bitbucket: joeyespo, GitHub: joeyespo)


I have Python installed in C:\Program Files (x86)\Python (for legacy reasons--it's kind of late to move it now), and any console_script I install from PyPI breaks now. Running the script causes a "failed to create process" error.

I noticed that older console_scripts were working, and turns out it's because they were quoted. Ex: #!"C:\Program Files (x86)\Python\python.exe"

Manually adding quotes around the newly installed projectname-script.py fixes it.

It looks like the culprit may have been the fix in #188. From that issue:

If it is possible to detect being on a posix platform (probably not that easy?), we could just skip the escaping, as it is never beneficial. If this is not possible, maybe just skip the escaping all together? The benefit seems very platform specific.

Turns out skipping escaping all together just threw the problem back over the fence to Windows. Perhaps it'd be best to detect the platform? Escaping is indeed a platform-specific thing.


@bb-migration

This comment has been minimized.

bb-migration commented Jun 19, 2015

Original comment by joeyespo (Bitbucket: joeyespo, GitHub: joeyespo):


I'm not sure if this is a good idea or not, but one way to fix this without involving escaping would be to use ctypes.windll.kernel32.GetShortPathNameW when it's available. This gives you the "short" filename that will never contain spaces. It might be appropriate if the complexity of escaping is high and if the filename here is isolated (as in not public / not used anywhere else).

Details: http://stackoverflow.com/questions/23598289/how-to-get-windows-short-file-name-in-python

@bb-migration

This comment has been minimized.

bb-migration commented Jun 25, 2015

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):


I'm -1 for using the short name. I'd like to devise a solution that supports all legal file paths with their canonical names.

@bb-migration

This comment has been minimized.

bb-migration commented Jun 25, 2015

Original comment by joeyespo (Bitbucket: joeyespo, GitHub: joeyespo):


@jaraco Agreed. +1 to that.

@bb-migration

This comment has been minimized.

bb-migration commented Dec 26, 2015

Original comment by johnthagen (Bitbucket: johnthagen, GitHub: johnthagen):


Just wanted to add that this problem manifests itself in a slightly different way when Python 3.5 is installed by default on Windows. If the user has a space in their name (e.g. John Doe), then this problem arises and is very difficult to debug.

Example path:

C:\Users>where.exe python
C:\Users\John Hagen\AppData\Local\Programs\Python\Python35\python.exe

pypa/pip#2783 (comment)

@bb-migration

This comment has been minimized.

bb-migration commented Dec 27, 2015

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):


Before I switched to OS X in May, I ran Windows almost exclusively, but I didn't encounter this issue because I used SETUPTOOLS_LAUNCHER=natural.

Still, I would not expect this issue to arise even without that setting as the script generation is specifically special-cased for Windows.

I'll take a quick look to see if I can replicate the issue.

@bb-migration

This comment has been minimized.

bb-migration commented Dec 27, 2015

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):


I'm not able to replicate the issue simply. Here's my install of setuptools on Python 3.5 in a directory with spaces and evidence that the script shebang includes quotes:

C:\Users\jaraco> & 'C:\Program Files\Python 3.5\python.exe' -m easy_install setuptools
Searching for setuptools
Best match: setuptools 18.5
Adding setuptools 18.5 to easy-install.pth file
Installing easy_install-script.py script to C:\Program Files\Python 3.5\Scripts
Installing easy_install.exe script to C:\Program Files\Python 3.5\Scripts
Installing easy_install-3.5-script.py script to C:\Program Files\Python 3.5\Scripts
Installing easy_install-3.5.exe script to C:\Program Files\Python 3.5\Scripts

Using c:\python\lib\site-packages
Processing dependencies for setuptools
Finished processing dependencies for setuptools
C:\Users\jaraco> cat C:\python\scripts\easy_install-3.5-script.py
#!"C:\Program Files\Python 3.5\python.exe"
# EASY-INSTALL-ENTRY-SCRIPT: 'setuptools==18.5','console_scripts','easy_install-3.5'
__requires__ = 'setuptools==18.5'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('setuptools==18.5', 'console_scripts', 'easy_install-3.5')()
    )
C:\Users\jaraco> echo $env:SETUPTOOLS_LAUNCHER
executable

Can someone whose environment has this issue what shebang appears when installing setuptools itself? If it doesn't include the quotes, can you trace the code to determine what in your environment might differ from mine? Or alternately, how can I configure my environment to replicate the behavior you see?

@bb-migration

This comment has been minimized.

bb-migration commented Dec 27, 2015

Original comment by johnthagen (Bitbucket: johnthagen, GitHub: johnthagen):


@jaraco I have been able to reproduce the error with the following steps on two Windows 10 x64 machines and a Windows 10 x64 VM.

  1. Install Windows 10 x64 (though I suspect that this problem exists on at least Windows 8 and 7 as well).
  2. Create a user with a space in their name (e.g. John Doe)
  3. Log into John Doe.
  4. Install Python 3.5.1 (64-bit) and select the option to add python to PATH.
  5. > pip install cpplint
  6. > cpplint

failed to create process.

Note that what is actually failing here is the console_script .exe file (cpplint.exe) that pip creates when it installs the package.

If you easy_install cpplint instead of using pip, it works. So it seems to be a bug in how pip creates the .exe files, related to having a space in some path.

@bb-migration

This comment has been minimized.

bb-migration commented Dec 27, 2015

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):


Aha. In that case, the issue lies with pip and distlib. I'm not sure whether to file the issue with the former or the latter, but I'd start with the former, as you're using pip, so the issue applies to it, and if an upstream bug needs to be filed with distlib, the pip team can advise.

@tomduck

This comment has been minimized.

tomduck commented Apr 5, 2016

See pip Issue #2783 for a discussion of the same problem, including a workaround that can be used in setup.py.

@fkrull

This comment has been minimized.

Contributor

fkrull commented Jun 12, 2016

I did some digging into this issue; it's either a setuptools or distutils problem. I'll stick to cpplint, but any package using setuptools and entry points should show this behaviour. Things I have observed:

  1. pip install cpplint installs the wheel and works fine.
  2. pip install --no-binary cpplint cpplint doesn't quote the shebang properly. It installs from sdist by running setup.py install --single-version-externally-managed locally.
  3. setup.py install from the source installs fine.
  4. setup.py install --single-version-externally-managed from the source doesn't quote the shebang line properly.

I don't know what the egg installation code does different, but in setuptools/command/install_scripts.py line 33 exec_param is retrieved from the build_scripts command which in turn gets it from the build command (distutils/command/build_scripts.py line 38) which, if unspecified, sets it to this (`distutils/command/build.py line 120):

self.executable = os.path.normpath(sys.executable)

Then, exec_param -- which is just an unquoted path, possibly containing spaces -- gets passed to CommandSpec.from_param (setuptools/command/install_scripts.py line 42) which parses it as a command line by splitting it along spaces (setuptools/command/easy_install.py line 1954). As a consequence, the special quoting code gets bypassed.

Wrapping exec_param in a one-element list solved the problem (not quite a one-line fix because it shouldn't be wrapped if it's None). I can roll a PR, but I don't know the setuptools codebase well enough to determine if this is the best place to fix this.

@anthrotype

This comment has been minimized.

Contributor

anthrotype commented Jun 12, 2016

You're right, it's a setuptools issue, not pip or distlib.

When installing from a binary wheel (or when installing from sdist package), pip uses distlib to create the console/gui scripts, and distlib does take care of "enquoting" the executable in the shebang:

https://github.com/pypa/pip/blob/master/pip/_vendor/distlib/scripts.py#L161-L164
https://github.com/pypa/pip/blob/master/pip/_vendor/distlib/scripts.py#L66-L79

I think it's in easy_install.ScriptWriter.get_args where the executable enquoting should go.

@anthrotype

This comment has been minimized.

Contributor

anthrotype commented Jun 12, 2016

I confirm what @fkrull just said.

The executable string from distutils build_scripts command is being split using shlex inside easy_install's CommandSpec.from_string.

Passing [exec_param] as a one-element list (when not None) solves the issue, as CommandSpec._render method will then concatenate the strings with subprocess.list2cmdline, which in turn correctly escapes spaces with double quotes following MS rules.

Thanks for finding this out! 👍

fkrull added a commit to fkrull/setuptools that referenced this issue Jun 22, 2016

Ensure shebang lines in scripts on Windows are properly quoted.
This is important to make the exe wrappers work if the path to the
Python binary contains spaces. Fixes issue pypa#398.

fkrull added a commit to fkrull/setuptools that referenced this issue Jun 22, 2016

Add test to check for unquoted shebang lines on Windows.
See issue pypa#398; if the path to the Python executable contains spaces, the
shebang lines in generated scripts aren't quoted which breaks the exe
wrappers.

fkrull added a commit to fkrull/setuptools that referenced this issue Jun 22, 2016

Ensure shebang lines are correctly quoted on Windows. Fixes issue pyp…
…a#398.

The executable path was passed as a single string which, if it contained
spaces, caused it to be parsed as a command with a series of arguments,
rather than as a single command containing spaces. As a consequence, the
path in the generated shebang line wasn't quoted which caused the
generated exe wrapper to trip over itself. By passing the executable path
as a single-argument list, paths with spaces aren't split and are quoted
properly.

fkrull added a commit to fkrull/setuptools that referenced this issue Jun 25, 2016

Test quoting of shebang lines.
See issues pypa#188 and pypa#398; on Windows, the launcher executables support shebangs
that use quotes to escape spaces, so these should be used when necessary. On
the other hand, Unix systems don't support shebangs with quotes so they should
never have quotes.

fkrull added a commit to fkrull/setuptools that referenced this issue Jun 25, 2016

Ensure shebang lines are correctly quoted if sys.executable contains …
…spaces.

Fixes issue pypa#398. This only special-cases sys.executable; if the --executable
parameter is used, paths with spaces have to be quoted there explicitly. While
this change also applies to Unix platforms, if sys.executable contains spaces
on Unix, any shebang lines created with it aren't going to work either way,
whether they are quoted or not.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment