-
-
Notifications
You must be signed in to change notification settings - Fork 31.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support xz compression in tarfile module #49939
Comments
GNU tar now supports lzma compression as a compression method. Please lzma extension at http://svn.fancycode.com/repos/python/pylzma/trunk/ lzma is used in many tools (7zip, dpkg, rpm), offers faster |
As for an lzma module - I would prefer one that isn't LGPL'ed. Instead, |
If we support LZMA, we should do so on all platforms; it kind of |
The LZMA implementation from 7-zip has been released as public domain |
That's good news. Now, if somebody could contribute a Python wrapper for
There are. The Linux version apparently originates from the same |
I'm the author of the pyliblzma module, and if desired, I'd be happy to help out adapting pyliblzma for inclusion with python.
I plan on doing these things sooner or later anyways, it's pretty much just a matter of motivation and priorities standing in the way, actual interest from others would certainly have a positive effect on this. ;) For other alternatives to the LGPL liblzma, you really don't have any, keep in mind that LZMA is "merely" the algorithm, while xz (and LZMA_alone, used for '.lzma', now obsolete, but still supported) are the actual format you want support for. The LZMA SDK does not provide any compatibility for this. |
ps: pylzma uses the LZMA SDK, which is not what you want. You'll find it available at http://launchpad.net/pyliblzma |
If that's really the case (which I don't believe it is), then this |
The XZ Utils website ( http://tukaani.org/xz/ ) states the following: "The most interesting parts of XZ Utils (e.g. liblzma) are in the public domain. You can do whatever you want with the public domain parts. Some parts of XZ Utils (e.g. build system and some utilities) are under different free software licenses such as GNU LGPLv2.1, GNU GPLv2, or GNU GPLv3." So, liblzma is not the problem. But the license of PylibLZMA is LGPL3. |
Martin, this is wrong, you don't have to bundle the source *in* the object code package. Making it available on some HTTP or FTP site is sufficient. |
Per, on 2010-03-17, I asked you via email: "I was looking at http://bugs.python.org/issue5689 and Martin's comments about the licensing of the bindings; is there a special reason for the lgpl3 license of the bindings, given that both python and xz-utils are not gpl'ed?" Does pyliblzma need to be licensed under the lgpl3? |
Am 26.05.2010 10:51, schrieb Antoine Pitrou:
That's why I said "along". I'm still not willing to do that: making the |
if you're already looking at bpo-6715, then I don't get why you're asking.. ;) quoting from msg106433: The reason why I picked LGPLv3 in the past was simply just because liblzma at the time was licensed under it, so I just picked the same for simplicity. Martin: For LGPL (or even GPL for that matter, disregarding linking restrictions) libraries you don't have to distribute the sources of those libraries at all (they're already made available by others, so that would be quite overly redundant, uh?;). LGPL actually doesn't even care at all about the license of your software as long as you only dynamically link against it. I don't really get what the issue would be even if liblzma were still LGPL, it doesn't prohibit you from distributing a dynamically linked library along with python either if necessary (which of course would be of convenience on win32..).. tsktsk, discussions about python module for xz compression should anyways be kept at bpo-6715 as this one is about the tarfile module ;p |
Ok, following up there. |
Attached is a patch with the current state of my work on lzma integration into tarfile (17 test errors). |
Python now has an lzma module. Lars, do you have the time to update your patch or should I do it? |
I will be happy to, but my spare time is limited right now, so this could take about a week. If this is a problem, please go ahead. |
There is plenty of time until 3.3. OTOH, if Eric wants to work on it now: you got a week :-) Do recognize that there is a patch to start from already. |
I’m perfectly happy to let Lars do it next week or next month, there is no rush. The existing patch may even require very little or no change, as Nadeem’s module (in the stdlib) provides the same classes than the other lzma module which was used by the patch. |
For those who want to test it first, I post the current state of the patch here. It is ready for commit, there are no failing tests. If nobody objects, I will apply it this weekend. |
Some comments about 2011-12-08-tarfile-lzma.diff:
Micro-optimization: you can use self.buf.startswith((b"\x5d\x00\x00\x80", b"\xfd7zXZ")) here.
Error messages usually don't end with a dot (or am I wrong?). It would be better to use a skip instead of just return here: def test_no_name_argument(self):
if self.mode.endswith("bz2") or self.mode.endswith("xz"):
# BZ2File and LZMAFile have no name attribute.
return In _Stream.__init__, for zlib:
Could you add a test for this change? |
Patch looks great. I did a review on Rietveld. |
New changeset 899a8c7b2310 by Lars Gustäbel in branch 'default': |
Thanks for the review, guys! I can't close this issue yet because it depends on bpo-6715. |
Great stuff! I'll close this issue along with bpo-6715 once the buildbot |
Lars, as part of a small doc patch I want to change this in tarfile.rst: The :mod:`tarfile` module makes it possible to read and write tar Any objection? |
Please, go ahead! |
There is failure on a XP buildbot. I don't know if it is a sporadic issue or not. http://www.python.org/dev/buildbot/all/builders/x86%20XP-5%203.x/builds/3921/steps/test/logs/stdio ====================================================================== Traceback (most recent call last):
File "D:\Buildslave\3.x.moore-windows\build\lib\test\test_tarfile.py", line 1539, in test_append_lzma
self._create_testtar("w:xz")
File "D:\Buildslave\3.x.moore-windows\build\lib\test\test_tarfile.py", line 1486, in _create_testtar
with tarfile.open(self.tarname, mode) as tar:
File "D:\Buildslave\3.x.moore-windows\build\lib\tarfile.py", line 1721, in open
return func(name, filemode, fileobj, **kwargs)
File "D:\Buildslave\3.x.moore-windows\build\lib\tarfile.py", line 1826, in xzopen
mode=mode, fileobj=fileobj, preset=preset)
File "D:\Buildslave\3.x.moore-windows\build\lib\lzma.py", line 117, in __init__
preset=preset, filters=filters)
MemoryError |
Perhaps Paul can try to reproduce and diagnose the issue directly on the buildbot? |
A simple rebuild and test run of that test in debug mode didn't fail... I'll run the full test suite as a check, but that could take some time - that buildslave isn't the fastest in the world... |
Not to worry - as I said in my previous message, I can reproduce the error I also noticed that running test_tarfile alone doesn't trigger the errors, I suspect that the problem is at least partially caused by the fact that |
Wouldn't it be better then to use a default compresslevel of 6 in tarfile? I used level 9 in my patch without a particular reason, just because I thought 9 must be better than 6 ;-) |
Yes, that's a good idea. I've been testing a similar change, and it seems You might want to use preset=None instead of explicitly saying preset=6, Something unrelated that I noticed in the surrounding code: gzopen and |
Yes, that's much better. Thanks for the tip. |
Patch looks good to me. |
Ping. Windows buildbots are still failing with MemoryError because of this preset=9. |
New changeset b86b54fcb5c2 by Lars Gustäbel in branch 'default': |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: