PIP creates invalid .pth when namespace package is unicode #449

Closed
jaraco opened this Issue Feb 6, 2012 · 4 comments

Projects

None yet

3 participants

@jaraco
Python Packaging Authority member

When using PIP to install a namespace package which has unicode literals for the package name, it fails with a TypeError.

jaraco@hideaki:~$ virtualenv env
New python executable in env/bin/python2.7
Also creating executable in env/bin/python
Installing setuptools............done.
Installing pip...............done.
jaraco@hideaki:~$ env/bin/pip install jaraco.util==5.0.1
Downloading/unpacking jaraco.util==5.0.1
  Downloading jaraco.util-5.0.1.zip (76Kb): 76Kb downloaded
  Running setup.py egg_info for package jaraco.util
    zip_safe flag not set; analyzing archive contents...

    Installed /home/jaraco/env/build/jaraco.util/hgtools-1.0.1-py2.7.egg
    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'use_2to3'
      warnings.warn(msg)
    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'use_2to3_exclude_fixers'
      warnings.warn(msg)

Installing collected packages: jaraco.util
  Running setup.py install for jaraco.util
    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'use_2to3'
      warnings.warn(msg)
    /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'use_2to3_exclude_fixers'
      warnings.warn(msg)
    Skipping installation of /home/jaraco/env/lib/python2.7/site-packages/jaraco/__init__.py (namespace package)

    Installing /home/jaraco/env/lib/python2.7/site-packages/jaraco.util-5.0.1-py2.7-nspkg.pth
    Installing roll-dice script to /home/jaraco/env/bin
    Installing calc-prorate script to /home/jaraco/env/bin
Successfully installed jaraco.util
Cleaning up...
jaraco@hideaki:~$ env/bin/python
Traceback (most recent call last):
  File "/home/jaraco/env/lib/python2.7/site.py", line 677, in <module>
    main()
  File "/home/jaraco/env/lib/python2.7/site.py", line 657, in main
    paths_in_sys = addsitepackages(paths_in_sys)
  File "/home/jaraco/env/lib/python2.7/site.py", line 273, in addsitepackages
    addsitedir(sitedir, known_paths)
  File "/home/jaraco/env/lib/python2.7/site.py", line 203, in addsitedir
    addpackage(sitedir, name, known_paths)
  File "/home/jaraco/env/lib/python2.7/site.py", line 172, in addpackage
    exec(line)
  File "<string>", line 1, in <module>
TypeError: module.__init__() argument 1 must be string, not unicode

By troubleshooting site.py, it seems the error occurs when it tries to process the .pth file for jaraco.util:

jaraco@hideaki:~$ cat env/lib/python2.7/site-packages/jaraco.util-5.0.1-py2.7-nspkg.pth
import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *(u'jaraco',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault(u'jaraco',new.module(u'jaraco')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p)

As you can see, the generated .pth file is using unicode literals, which causes the failure when importing site.py. These errors do not occur with setuptools or distribute for the same package.

Replacing namespace_packages=['jaraco'] with namespace_packages=[bytes('jaraco')] in the setup.py works around the issue.

PIP should probably try to generate .pth files that don't corrupt the environment even if the namespace packages are specified using unicode strings.

@carljm

Thanks for the report.

Pip doesn't generate the pth file, setuptools/distribute does. This is a bug in the implementation of the --single-version-externally-managed option in setuptools/distribute, and would have to be fixed there. There's nothing pip can do about it, short of reimplementing setuptools/distribute or monkeypatching.

The only reason the bug doesn't show up with setuptools/distribute by default is that they install into eggs, which don't require a pth file at all for namespace package support (rather, they can use a function call in the namespace package's __init__.py, since each package installed in the namespace gets its own egg and thus its own namespace package directory and __init__.py).

But pip installs "flat" by using the --single-version-externally-managed flag to setup.py install, which causes setuptools/distribute to switch to the pth-file style of namespace packages (to avoid the need for a namespace package __init__.py file, which would conflict since each package in the namespace would try to provide it).

You'll see the same bug if you run python setup.py install --single-version-externally-managed on the package in question, with no involvement from pip.

@carljm carljm closed this Feb 6, 2012
@tseaver

Seems like the problem is with pip's choice to use that installation method, which is not sane for namespace packages. Blaming setuptools/distribute doesn't actually move the ball forward: the setuptools docs describe that option as a 'convenience [for] creating "system packages' of setuptools-based projects," which is hardly pip's job.

@jaraco
Python Packaging Authority member

First, thanks to carljm for highlighting the underlying cause. It led me to file https://bitbucket.org/tarek/distribute/issue/272/unicode-string-in-namespace_package-name in distribute.

I tend to agree with tseaver, but since PIP now depends on that feature, and the bug does in fact exist in that feature, it's probably best to fix the upstream bug.

Regardless of the ultimate resolution, closing the bug is not appropriate, as the issue remains, even if it is caused by an upstream library. If PIP should continue to depend on the --single-version-externally-managed flag, it suggest PIP should consider filing bug reports with the upstream libraries and leave this ticket open until both are resolved.

Distribute 0.6.25 was released with a fix for this issue, so half of the upstream libraries are fixed. I hope that helps.

@carljm

Thanks @jaraco for filing and fixing the upstream bug. I am comfortable leaving this issue closed if it is fixed in distribute.

Abandoning the use of --single-version-externally-managed (for namespace packages only) is something I've considered and would likely accept a pull request for, but that's a separate issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment