-
-
Notifications
You must be signed in to change notification settings - Fork 31.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
xz compressor support #50964
Comments
Python currently supports zlib, gzip and bzip2 compressors. What is missing is support |
Is zc really a C library? I could find a standalone program, but no |
What is xz compression and why is it important? Skip |
Yes, xz-utils contains a C library, though it still caries the name xz is the successor of lzma, which provides a better compression than |
Are xz and lzma formats compatible with each other? If not, which one is |
As I understand it from the http://tukaani.org/xz/ page, .lzma and .xz
|
.lzma is actually not a format. It is just the raw output of the LZMA1 |
Once Python gets native support for lzma/xz like it does for zlib, bzip2 it could switch to using it for bundles and remote transfers. See: http://mercurial.selenic.com/bts/issue1463 With lzma/xz being able to compress so much better, it'd be really appreciated by users on especially slow links(!!). |
Ugh, can't edit previous message. Meant to say: "Once Python gets native support for lzma/xz like it does for zlib and bzip2, Mercurial could switch to using it for bundles and remote transfers." For platforms with native support in-kernel (e.g. Linux) that could be used instead of the bundled version. (Since Python is officially switching to Mercurial, arguably this issue even more important.) |
About why xz is important. gnu.org, tug.org started publishing sources in xz format, quick grep: autoconf/autoconf.spec:Source0: http://ftp.gnu.org/gnu/autoconf/%{name}-%{version}.tar.xz xz is also supported by automake as dist target. |
We all agree that lzma/xz is important, what is needed is a patch. |
There's a Python binding for some lzma lib here: https://launchpad.net/pyliblzma |
Once xz is implemented, xz compatibility should be added to the tarfile library. |
Ooops, I kinda should've commented on this issue here in stead, rather than in bpo-5689, so I'll just copy-paste it here as well: I'm the author of the pyliblzma module, and if desired, I'd be happy to help out adapting pyliblzma for inclusion with python.
I plan on doing these things sooner or later anyways, it's pretty much just a matter of motivation and priorities standing in the way, actual interest from others would certainly have a positive effect on this. ;) For other alternatives to the LGPL liblzma, you really don't have any, keep in mind that LZMA is "merely" the algorithm, while xz (and LZMA_alone, used for '.lzma', now obsolete, but still supported) are the actual format you want support for. The LZMA SDK does not provide any compatibility for this. |
I will happily review any implementation, and I can help with inclusion into python trunk.
Can you check which licences cover the different parts of the module? I think that you will have to contribute your code under the Python Contributor Agreement; and I just grabbed some copy of the "xz-utils" source package, and it states that "liblzma is in the public domain". |
ah, you're right, I forgot that the license for the library had changed as well (motivated by attempt of pleasing BSD people IIRC;), in the past the library was LGPL while only the 'xz' util was public domain.. For my code, feel free to use your own/any other license you'd like or even public domain (if the license of bz2module.c that much of it's derived from permits of course)! I guess everyone should be happy now then. :) Btw. for review, I think the code already available should be pretty much good 'nuff for an initial review. Some feedback on things not derived from bz2module.c would be nice, especially the LZMAOptions class would be nice as it's where most of the remaining work required for adding additional filters support. Would kinda blow if I did the work using an approach that would be dismissed as utterly rubbish. ;) Oh well, it's out there available for anyone already, I probably won't(/shouldn't;) have time for it in a month at least, do as you please meanwhile. :) |
Hello,
Well, I wouldn't say bz2module is the best module out there, but as you Is pyliblzma compatible with Python 3.x? It's too late to incorporate
Hmm, then perhaps you should first fix the current API so that adding By the way, adding a new module to the stdlib probably requires writing Finally, when a module is in the stdlib, it is expected that maintenance |
Yeah, I guess I anyways can just break the current API right away to make it compatible with future changes, I've already figured since long ago how it should look like. It's not like I have to implement the actual functionality to ensure compatibility, no-op works like charm. ;) |
[Replying to msg106566]
Can you please submit a contributor form?
Of course you do. Quoting from the LGPL "You may convey a Combined Work ... if you also do each of the following:
Of course I can distribute a copy of an lzma DLL. However, I would have to provide ("convey") a copy of the source code of that DLL as well. |
Can you tell me where you are currently providing the source code for Oh, and by the way, you should probably shut down PyPI, since there are You seem to have no problem "conveying" copies for the source code of |
We don't, as they aren't included in the Windows distribution. The
This is off-topic for this bug tracker. Please remain objective and
Not sure what you are referring to. We don't provide the sources for the |
This whole subthread was already off-topic (since it was pointed out, Actually, I would argue that the whole idea of promoting a rigorous (of course, you will also have understood that I disagree with such a
This was not about providing the sources together with the installer |
I've ported pyliblzma to py3k now and also implemented the missing functionality I mentioned earlier, for anyone interested in my progress the branch is found at: I need to fix some memory leakages (side effect of the new PyUnicode/Pybytes change I'm not 100% with yet;) and some various memory errors reported by valgrind etc. though, but things are starting to look quite nice already. :) |
I've (finally) finalized the api and prepared pyliblzma to be ready for inclusion now. The code can be found in the 'py3k' branch referred to earlier. Someone else (don't remember who:p) volunteered for writing the PEP earlier, so I leave it up to that person to write the PEP, I won't be able to get around to do so myself in the near future.. |
As promised, here is a quick review of the module.
Also, I did not find any documentation. 3.2beta1 is scheduled for November 13, 2010, and no new feature will be accepted after. Do you think you can update it before the limit? otherwise the package could live in PyPI before it is shipped with Python. |
All fixed now. :) |
Does that refer to msg119743? Please provide this module Also, please provide Doc/library/lzma.rst. |
It seemed to me that Nadeem had rewritten everything from scratch. Is |
I've opened bpo-13507 to track adding liblzma to the OS X installer builds. |
Ah, I thought that he had reused most of the original C code in _lzmamodule.c not replaced by python code, but I see that not being the case now (only slight fragments;). Oh well, I thought that I'd still earned a note with some slight credit at least, but I guess I won't go postal or anything in lack of either.. :p |
I completely agree. Sometimes people get credit for simple bug fixes (count me among them) so the author of the first working implementation deserves some recognition IMO. |
Nadeem: Instead of duplicating the list of archiving/compression modules in each doc, what about only linking to the shutil doc for archives and the archiving.rst file? (I can make the patch, just wanted feedback first) |
That is correct. Based on my experience with the bz2 module, rewriting
Of course. How does this look?
|
Sure, go ahead. I actually hadn't realized that each section of the |
--- a/Misc/NEWS
The entry in Misc/ACKS, Doc/whatsnew/3.3.rst and the commit message should be enough. Lately I’ve noticed some attributions in NEWS, but it’s usually not done (as redundant). |
what about a mention in lzmamodule.c? |
As the docs for zlib, gzip, bz2, lzma, zipfile and tarfile are in the archiving subsection, there’s already a link to the subsection index, so I just removed the “See also zlib, etc.” lines (except for the link from zlib to gzip). I added a link from archiving.rst to shutil and more links and reST targets in shutil. |
I generally add attributions in NEWS since that's where most people |
New changeset 6cde416ef03b by Nadeem Vawda in branch 'default': |
Looks good to me.
According to Doc/whatsnew/3.3.rst: "The maintainer will go through Misc/NEWS periodically and add
Done and committed. |
I’ll commit my doc patch to all branches later. I checked the 4 red 3.3 builbots and they can’t build _lzma. |
OK, great.
For the record, the 3.x buildbots that currently aren't able to build the
Additionally, the AMD64 debian bigmem, PPC Tiger and PPC Leopard builders |
Le jeudi 08 décembre 2011 à 17:40 +0000, Nadeem Vawda a écrit :
Ok, I've e-mailed the owners. Will someone write an entry in the "what's new" file for 3.3? Regards Antoine. |
New changeset 66df5ace0eee by Nadeem Vawda in branch 'default': |
Thanks. I was just thinking I should send a reminder, in case they
Done |
Is there any reason why this issue is still open? |
Yes, almost half of the buildbots still don't have the xz-utils headers |
New changeset 11bd2d32b4e8 by Éric Araujo in branch '3.2': New changeset 1cb9b8126534 by Éric Araujo in branch 'default': |
BTW, any plans on a PyPI backport for fun and profit? |
At present, no. I'll look into it later in the year, but at the moment |
New changeset 7c22281e967c by Éric Araujo in branch '2.7': |
Why is this issue still open? |
I had intended to wait until the code had been tested on all of the |
Apologies for noise, but since a backport was discussed, I'm mentioning this here. I've started implementing a backport, currently working and tested on Mac OS X and Linux, back to Python 3.0 - supporting Python 2 would be nice but probably significantly more work (assistance welcome - please get in touch on github): Assuming Nadeem has no objections, I intend to publish this on PyPI (I have tried to email directly but perhaps I'm using an old email address or he has been busy, another reason for commenting here). Thanks! |
Apologies again for the noise, but I've just made the first public release of the lzma backport at http://pypi.python.org/pypi/backports.lzma/ with the repository as mentioned before at https://github.com/peterjc/backports.lzma I have tested this on Python 2.6, 2.7 and 3.0 through 3.3 under Linux and Mac OS. I've not tried building this on Windows yet, but it should be possible and I will be reviewing Amaury's notes on this thread. My thanks to Nadeem and Per for looking over this with helpful feedback, and agreeing to the use of the 3-clause BSD license for the backport. If anyone has any further questions about the backport, please get in touch via GitHub or email me. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: