pip doesn't recreate symlinks in a package

[lebedov]

When I attempt to install a package that contains symlinks, pip appears to have difficulty recreating them. In the example below, the symlink from version.py to scikits/cuda/version.py is not recreated and causes installation to fail because setup.py attempts to import version:

$ cd /tmp
$ virtualenv TEST      
New python executable in TEST/bin/python
Installing setuptools............done.
Installing pip...............done.
$ wget --quiet http://pypi.python.org/packages/source/s/scikits.cuda/scikits.cuda-0.04.tar.gz
$ tar zftv scikits.cuda-0.04.tar.gz |egrep '\->'
lrwxrwxrwx lev/lev           0 2011-05-11 10:40 scikits.cuda-0.04/doc/source/install.rst -> ../../INSTALL
lrwxrwxrwx lev/lev           0 2011-05-11 10:40 scikits.cuda-0.04/doc/source/changes.rst -> ../../CHANGES
lrwxrwxrwx lev/lev           0 2011-05-11 10:40 scikits.cuda-0.04/doc/source/authors.rst -> ../../AUTHORS
lrwxrwxrwx lev/lev           0 2011-05-11 10:40 scikits.cuda-0.04/doc/source/license.rst -> ../../LICENSE
lrwxrwxrwx lev/lev           0 2011-05-11 10:40 scikits.cuda-0.04/version.py -> scikits/cuda/version.py
TEST/bin/pip install --upgrade scikits.cuda-0.04.tar.gz                           
Unpacking ./scikits.cuda-0.04.tar.gz
  In the tar file /tmp/scikits.cuda-0.04.tar.gz the member scikits.cuda-0.04/doc/source/install.rst is invalid: 'NoneType' object has no attribute 'isreg'
  In the tar file /tmp/scikits.cuda-0.04.tar.gz the member scikits.cuda-0.04/doc/source/changes.rst is invalid: 'NoneType' object has no attribute 'isreg'
  In the tar file /tmp/scikits.cuda-0.04.tar.gz the member scikits.cuda-0.04/doc/source/authors.rst is invalid: 'NoneType' object has no attribute 'isreg'
  In the tar file /tmp/scikits.cuda-0.04.tar.gz the member scikits.cuda-0.04/doc/source/license.rst is invalid: 'NoneType' object has no attribute 'isreg'
  In the tar file /tmp/scikits.cuda-0.04.tar.gz the member scikits.cuda-0.04/version.py is invalid: 'NoneType' object has no attribute 'isreg'
  Running setup.py egg_info for package from file:///tmp/scikits.cuda-0.04.tar.gz
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "/tmp/pip-9HAXcy-build/setup.py", line 10, in <module>
        from version import __version__
    ImportError: No module named version
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
  File "<string>", line 14, in <module>
  File "/tmp/pip-9HAXcy-build/setup.py", line 10, in <module>
    from version import __version__
ImportError: No module named version

[carljm]

Based on the relevant source code in pip (https://github.com/pypa/pip/blob/develop/pip/util.py#L429), it appears that if this is a bug anywhere, it's in the "tarfile" module of Python itself. Pip isn't doing anything unusual here, just calling extractfile() on TarInfo objects returned from getmembers(), as documented. In this case, that extractfile() call is raising an AttributeError. It might have something to do with the way you're packing up your tar file?

If you're able to track this bug down and propose a reasonable workaround that pip could use, I'd be open to that.

[lebedov]

The extractfile() method attempts to extract the target of a symlink rather than the symlink itself; if the target does not yet exist (e.g., because a symlink is extracted before its target), an exception occurs. Regardless of whether this behavior was intended by the Python developers, here is a way to circumvent it (diff obtained against the latest revision in git):

diff -r 12f87be77714 pip/util.py
--- a/pip/util.py   Wed May 18 10:34:58 2011 -0500
+++ b/pip/util.py   Mon May 23 15:09:02 2011 -0400
@@ -425,6 +425,17 @@
             if member.isdir():
                 if not os.path.exists(path):
                     os.makedirs(path)
+            elif member.issym():
+                try:
+                    tar._extract_member(member, path)
+                except:
+                    e = sys.exc_info()[1]
+                    # Some corrupt tar files seem to produce this
+                    # (specifically bad symlinks)
+                    logger.warn(
+                        'In the tar file %s the member %s is invalid: %s'
+                        % (filename, member.name, e))
+                    continue
             else:
                 try:
                     fp = tar.extractfile(member)

[pnasrat]

Thanks for the patch, could you follow the instructions here to create a clone and a pull request with this commited on a branch.

https://github.com/pypa/pip/blob/develop/docs/how-to-contribute.txt

[carljm]

Pull request here is #293

[lebedov]

Yes - #288 is the issue number.

[tlynn]

It's not just symlinks, it loses permissions and other TarInfo fields on regular files as well. It's affecting installing gevent on Linux, which uses a ./configure script that should be executable.

[duncf]

I've filed the "loses permissions" issue as issue #317 to ensure it doesn't get lost.

[carljm]

Thanks @lebedov for the patch! Sorry it took so long to merge.