pip dependency installation uses unicode __file__ in Python 2

[pabigot]

I recently received a bug report against PyXB that pip would not install pyxb as a dependency in another package. pip has no problems installing pyxb directly. Though blame can be distributed to three places, I'm going to try to convince you pip is the primary place that should be fixed. Here's what's going on:

pip.req.InstallRequirement.run_egg_info builds a script that wraps a package's setup.py. Because pip now understands unicode and the requirement project_name happens to be unicode, it turns out the string stored as setup_py is also unicode, and the generated wrapper includes something like:

__file__ = u'/tmp/pip_build_pab/tester/setup.py'

Now, PyXB's setup.py uses __file__ as a starting point to search for subcomponents that need to be installed. Python os.listdir() will, when given a unicode path, return file names in unicode. Subsequent string manipulations in PyXB preserve this, resulting in unicode package name and paths.

This causes setuptools in Python 2 to reject the installation with:

error in PyXB setup command: package_data must be a dictionary mapping package names to lists of wildcard patterns 

because it has a check_extras function that requires package_data keys and values to be str not basestring instances.

The problem doesn't arise outside of pip, or when pip installs PyXB directly, because then the __file__ that PyXB's setup.py sees is an 8-bit string not a unicode string.

A case could be made that setuptools shouldn't reject unicode package names and paths, but I'm not going to fight that battle. I could also add code to PyXB's setup.py to convert unicode strings, but honestly I don't think it's my problem.

I believe pip shouldn't make __file__ contain a unicode string unless the underlying file system encoding type requires it and you're in Python 3.

[Natim]

Thx @pabigot
@pypa could you have a look at this PR please ?

[dstufft]

This will be fixed with #1538