New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip doesn't recreate symlinks in a package #288

Closed
lebedov opened this Issue May 22, 2011 · 8 comments

Comments

Projects
None yet
5 participants
@lebedov
Contributor

lebedov commented May 22, 2011

When I attempt to install a package that contains symlinks, pip appears to have difficulty recreating them. In the example below, the symlink from version.py to scikits/cuda/version.py is not recreated and causes installation to fail because setup.py attempts to import version:

$ cd /tmp
$ virtualenv TEST      
New python executable in TEST/bin/python
Installing setuptools............done.
Installing pip...............done.
$ wget --quiet http://pypi.python.org/packages/source/s/scikits.cuda/scikits.cuda-0.04.tar.gz
$ tar zftv scikits.cuda-0.04.tar.gz |egrep '\->'
lrwxrwxrwx lev/lev           0 2011-05-11 10:40 scikits.cuda-0.04/doc/source/install.rst -> ../../INSTALL
lrwxrwxrwx lev/lev           0 2011-05-11 10:40 scikits.cuda-0.04/doc/source/changes.rst -> ../../CHANGES
lrwxrwxrwx lev/lev           0 2011-05-11 10:40 scikits.cuda-0.04/doc/source/authors.rst -> ../../AUTHORS
lrwxrwxrwx lev/lev           0 2011-05-11 10:40 scikits.cuda-0.04/doc/source/license.rst -> ../../LICENSE
lrwxrwxrwx lev/lev           0 2011-05-11 10:40 scikits.cuda-0.04/version.py -> scikits/cuda/version.py
TEST/bin/pip install --upgrade scikits.cuda-0.04.tar.gz                           
Unpacking ./scikits.cuda-0.04.tar.gz
  In the tar file /tmp/scikits.cuda-0.04.tar.gz the member scikits.cuda-0.04/doc/source/install.rst is invalid: 'NoneType' object has no attribute 'isreg'
  In the tar file /tmp/scikits.cuda-0.04.tar.gz the member scikits.cuda-0.04/doc/source/changes.rst is invalid: 'NoneType' object has no attribute 'isreg'
  In the tar file /tmp/scikits.cuda-0.04.tar.gz the member scikits.cuda-0.04/doc/source/authors.rst is invalid: 'NoneType' object has no attribute 'isreg'
  In the tar file /tmp/scikits.cuda-0.04.tar.gz the member scikits.cuda-0.04/doc/source/license.rst is invalid: 'NoneType' object has no attribute 'isreg'
  In the tar file /tmp/scikits.cuda-0.04.tar.gz the member scikits.cuda-0.04/version.py is invalid: 'NoneType' object has no attribute 'isreg'
  Running setup.py egg_info for package from file:///tmp/scikits.cuda-0.04.tar.gz
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "/tmp/pip-9HAXcy-build/setup.py", line 10, in <module>
        from version import __version__
    ImportError: No module named version
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
  File "<string>", line 14, in <module>
  File "/tmp/pip-9HAXcy-build/setup.py", line 10, in <module>
    from version import __version__
ImportError: No module named version
@carljm

This comment has been minimized.

Show comment
Hide comment
@carljm

carljm May 22, 2011

Contributor

Based on the relevant source code in pip (https://github.com/pypa/pip/blob/develop/pip/util.py#L429), it appears that if this is a bug anywhere, it's in the "tarfile" module of Python itself. Pip isn't doing anything unusual here, just calling extractfile() on TarInfo objects returned from getmembers(), as documented. In this case, that extractfile() call is raising an AttributeError. It might have something to do with the way you're packing up your tar file?

If you're able to track this bug down and propose a reasonable workaround that pip could use, I'd be open to that.

Contributor

carljm commented May 22, 2011

Based on the relevant source code in pip (https://github.com/pypa/pip/blob/develop/pip/util.py#L429), it appears that if this is a bug anywhere, it's in the "tarfile" module of Python itself. Pip isn't doing anything unusual here, just calling extractfile() on TarInfo objects returned from getmembers(), as documented. In this case, that extractfile() call is raising an AttributeError. It might have something to do with the way you're packing up your tar file?

If you're able to track this bug down and propose a reasonable workaround that pip could use, I'd be open to that.

@lebedov

This comment has been minimized.

Show comment
Hide comment
@lebedov

lebedov May 23, 2011

Contributor

The extractfile() method attempts to extract the target of a symlink rather than the symlink itself; if the target does not yet exist (e.g., because a symlink is extracted before its target), an exception occurs. Regardless of whether this behavior was intended by the Python developers, here is a way to circumvent it (diff obtained against the latest revision in git):

diff -r 12f87be77714 pip/util.py
--- a/pip/util.py   Wed May 18 10:34:58 2011 -0500
+++ b/pip/util.py   Mon May 23 15:09:02 2011 -0400
@@ -425,6 +425,17 @@
             if member.isdir():
                 if not os.path.exists(path):
                     os.makedirs(path)
+            elif member.issym():
+                try:
+                    tar._extract_member(member, path)
+                except:
+                    e = sys.exc_info()[1]
+                    # Some corrupt tar files seem to produce this
+                    # (specifically bad symlinks)
+                    logger.warn(
+                        'In the tar file %s the member %s is invalid: %s'
+                        % (filename, member.name, e))
+                    continue
             else:
                 try:
                     fp = tar.extractfile(member)
Contributor

lebedov commented May 23, 2011

The extractfile() method attempts to extract the target of a symlink rather than the symlink itself; if the target does not yet exist (e.g., because a symlink is extracted before its target), an exception occurs. Regardless of whether this behavior was intended by the Python developers, here is a way to circumvent it (diff obtained against the latest revision in git):

diff -r 12f87be77714 pip/util.py
--- a/pip/util.py   Wed May 18 10:34:58 2011 -0500
+++ b/pip/util.py   Mon May 23 15:09:02 2011 -0400
@@ -425,6 +425,17 @@
             if member.isdir():
                 if not os.path.exists(path):
                     os.makedirs(path)
+            elif member.issym():
+                try:
+                    tar._extract_member(member, path)
+                except:
+                    e = sys.exc_info()[1]
+                    # Some corrupt tar files seem to produce this
+                    # (specifically bad symlinks)
+                    logger.warn(
+                        'In the tar file %s the member %s is invalid: %s'
+                        % (filename, member.name, e))
+                    continue
             else:
                 try:
                     fp = tar.extractfile(member)
@pnasrat

This comment has been minimized.

Show comment
Hide comment
@pnasrat

pnasrat May 29, 2011

Contributor

Thanks for the patch, could you follow the instructions here to create a clone and a pull request with this commited on a branch.

https://github.com/pypa/pip/blob/develop/docs/how-to-contribute.txt

Contributor

pnasrat commented May 29, 2011

Thanks for the patch, could you follow the instructions here to create a clone and a pull request with this commited on a branch.

https://github.com/pypa/pip/blob/develop/docs/how-to-contribute.txt

@carljm

This comment has been minimized.

Show comment
Hide comment
@carljm

carljm Jun 5, 2011

Contributor

Pull request here is #293

Contributor

carljm commented Jun 5, 2011

Pull request here is #293

@lebedov

This comment has been minimized.

Show comment
Hide comment
@lebedov

lebedov Jun 5, 2011

Contributor

Yes - #288 is the issue number.

Contributor

lebedov commented Jun 5, 2011

Yes - #288 is the issue number.

@tlynn

This comment has been minimized.

Show comment
Hide comment
@tlynn

tlynn Jun 11, 2011

It's not just symlinks, it loses permissions and other TarInfo fields on regular files as well. It's affecting installing gevent on Linux, which uses a ./configure script that should be executable.

tlynn commented Jun 11, 2011

It's not just symlinks, it loses permissions and other TarInfo fields on regular files as well. It's affecting installing gevent on Linux, which uses a ./configure script that should be executable.

@duncf

This comment has been minimized.

Show comment
Hide comment
@duncf

duncf Jul 11, 2011

I've filed the "loses permissions" issue as issue #317 to ensure it doesn't get lost.

duncf commented Jul 11, 2011

I've filed the "loses permissions" issue as issue #317 to ensure it doesn't get lost.

carljm added a commit that referenced this issue Oct 23, 2011

Merge pull request #293 from lebedov/develop
Directly extract symbolic links rather than their targets (#288).

carljm added a commit that referenced this issue Oct 23, 2011

@carljm carljm closed this Oct 23, 2011

@carljm

This comment has been minimized.

Show comment
Hide comment
@carljm

carljm Oct 23, 2011

Contributor

Thanks @lebedov for the patch! Sorry it took so long to merge.

Contributor

carljm commented Oct 23, 2011

Thanks @lebedov for the patch! Sorry it took so long to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment