New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support .tar.xz packages #722

Closed
wummel opened this Issue Nov 13, 2012 · 1 comment

Comments

Projects
None yet
3 participants
@wummel

wummel commented Nov 13, 2012

Hi,

the following patch enables detection and unpacking of .tar.xz archives. Since xz (ie. the lzma compression method) is not supported by the tarfile module, the code falls back to system binary. Not pretty, but works for me - tested with "pip install linkchecker".

See also http://bugs.python.org/issue6715 for the Python bug adding .xz support (but only in Python >= 3.3).

There is apparently no way to attach diff files, so here goes:

--- download.py 2012-11-13 15:10:22.086133699 +0100
+++ /usr/lib/python2.7/dist-packages/pip/download.py    2012-11-13 15:10:34.189942352 +0100
@@ -279,7 +279,7 @@

 def is_archive_file(name):
     """Return True if `name` is a considered as an archive file."""
-    archives = ('.zip', '.tar.gz', '.tar.bz2', '.tgz', '.tar', '.pybundle')
+    archives = ('.zip', '.tar.gz', '.tar.bz2', '.tar.xz', '.tgz', '.tar', '.pybundle')
     ext = splitext(name)[1].lower()
     if ext in archives:
         return True
--- index.py    2012-11-13 15:08:10.936207032 +0100
+++ /usr/lib/python2.7/dist-packages/pip/index.py   2012-11-13 15:09:48.102670938 +0100
@@ -292,7 +292,7 @@
                 # Special double-extension case:
                 egg_info = egg_info[:-4]
                 ext = '.tar' + ext
-            if ext not in ('.tar.gz', '.tar.bz2', '.tar', '.tgz', '.zip'):
+            if ext not in ('.tar.gz', '.tar.bz2', '.tar.xz', '.tar', '.tgz', '.zip'):
                 if link not in self.logged_links:
                     logger.debug('Skipping link %s; unknown archive format: %s' % (link, ext))
                     self.logged_links.add(link)
@@ -428,7 +428,7 @@
                     if cache.is_archive(url):
                         return None
                 filename = link.filename
-                for bad_ext in ['.tar', '.tar.gz', '.tar.bz2', '.tgz', '.zip']:
+                for bad_ext in ['.tar', '.tar.gz', '.tar.bz2', '.tar.xz', '.tgz', '.zip']:
                     if filename.endswith(bad_ext):
                         content_type = cls._get_content_type(url)
                         if content_type.lower().startswith('text/html'):
--- util.py 2012-11-13 15:10:44.565778323 +0100
+++ /usr/lib/python2.7/dist-packages/pip/util.py    2012-11-13 15:44:27.159051300 +0100
@@ -401,15 +401,26 @@
     finally:
         zipfp.close()

+def decompress_xz(filename):
+    import subprocess
+    subprocess.check_call(["xz", "-dk", filename])
+    return filename[:-3]

 def untar_file(filename, location):
     """Untar the file (tar file located at filename) to the destination location"""
+    remove_file = False
     if not os.path.exists(location):
         os.makedirs(location)
     if filename.lower().endswith('.gz') or filename.lower().endswith('.tgz'):
         mode = 'r:gz'
     elif filename.lower().endswith('.bz2') or filename.lower().endswith('.tbz'):
         mode = 'r:bz2'
+    elif filename.lower().endswith('.xz'):
+        mode = 'r'
+        # extract to (temporary) .tar file since the tarfile module does not
+        # support xz compression
+        filename = decompress_xz(filename)
+        remove_file = True
     elif filename.lower().endswith('.tar'):
         mode = 'r'
     else:
@@ -464,6 +475,8 @@
                 fp.close()
     finally:
         tar.close()
+        if remove_file:
+            os.unlink(filename)


 def create_download_cache_folder(folder):
@@ -490,7 +503,7 @@
         unzip_file(filename, location, flatten=not filename.endswith('.pybundle'))
     elif (content_type == 'application/x-gzip'
           or tarfile.is_tarfile(filename)
-          or splitext(filename)[1].lower() in ('.tar', '.tar.gz', '.tar.bz2', '.tgz', '.tbz')):
+          or splitext(filename)[1].lower() in ('.tar', '.tar.gz', '.tar.bz2', '.tar.xz', '.tgz', '.tbz')):
         untar_file(filename, location)
     elif (content_type and content_type.startswith('text/html')
           and is_svn_page(file_contents(filename))):
@pnasrat

This comment has been minimized.

Show comment
Hide comment
@pnasrat

pnasrat Nov 13, 2012

Contributor

Can you submit a pull request from your branch on github.

Contributor

pnasrat commented Nov 13, 2012

Can you submit a pull request from your branch on github.

xavfernandez added a commit to xavfernandez/pip that referenced this issue Oct 23, 2015

xavfernandez added a commit to xavfernandez/pip that referenced this issue Oct 23, 2015

xavfernandez added a commit to xavfernandez/pip that referenced this issue Nov 13, 2015

xavfernandez added a commit to xavfernandez/pip that referenced this issue Nov 25, 2015

xavfernandez added a commit to xavfernandez/pip that referenced this issue Nov 27, 2015

xavfernandez added a commit to xavfernandez/pip that referenced this issue Nov 29, 2015

xavfernandez added a commit to xavfernandez/pip that referenced this issue Dec 3, 2015

xavfernandez added a commit to xavfernandez/pip that referenced this issue Dec 5, 2015

xavfernandez added a commit to xavfernandez/pip that referenced this issue Dec 13, 2015

xavfernandez added a commit to xavfernandez/pip that referenced this issue Dec 28, 2015

xavfernandez added a commit to xavfernandez/pip that referenced this issue Jan 2, 2016

xavfernandez added a commit to xavfernandez/pip that referenced this issue Jan 3, 2016

xavfernandez added a commit to xavfernandez/pip that referenced this issue Jan 13, 2016

xavfernandez added a commit to xavfernandez/pip that referenced this issue Jan 14, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment