Add support for lzma filter

[migueldvb]

In addition to the supported zlib compression type, it is useful to have support ifor other algorithms like liblzma and bzip2 for lossless data compression. The interface of these modules in the python standard library is similar and lzma has some advantages for some types of data compared to zlib. If these methods are not supported by the standard it would be useful to have user-defined filters to implement the compression.

[mdboom]

It's important to remember that ASDF aims to be an interchange language beyond just for Python. Anything that we put into the spec that makes implementations in other languages more difficult is something we want to avoid.

By that standard, the criteria for inclusion of a compression algorithm in the spec needs to be:

1. widely available (i.e. libraries available for all major programming languages)
2. patent unencumbered
3. has an implementation under a permissive license

bzip2 probably meets those criteria, though I could only find one Javascript implementation, where there are many for zlib (it's a much simpler algorithm).

lzma is less ideal. They don't explicitly claim that they are patent free, which can really matter for compression algorithms. If someone claims a patent on it, users could be obliged to pay royalties. It's also later to the Python game (only in the stdlib in 3.3 and later).

If these methods are not supported by the standard it would be useful to have user-defined filters to implement the compression.

The point of the standard is to ensure interoperability with other libraries and implementations, so I don't think we should knowingly veer from the standard in any implementation.

[migueldvb]

That sounds good. I was thinking about having user-defined compression filters that could be part of the standard like the registered compression filters in HDF5, apart from the supported zlib compression method. But I agree that zlib is a good choice of compression library.because of those requirements.

[mdboom]

I think bzip2 is probably fine to add, though. There may be an argument to only support only that perhaps, in the interest of not creating too many ways to do things. It's not too late to take zlib away, as we haven't made a release yet. @embray: Thoughts?

[embray]

I don't have strong opinions about it, though I agree with the guidelines @mdboom laid out. Interestingly the talk of optional compression filters relates pretty well to my comments elsewhere today about output filters...

[migueldvb]

bzip2 compresses more effectively than zlib but it is also generally slower. I think that it would be nice that the user can choose the compression method but I understand the guidelines and don't have a strong preference about any of them.

[embray]

I don't see too much of a problem with allowing one or the other (or possibly other schemes that could be added later so long as they meet the criteria).

[mdboom]

I'm going to close this. We now have bzip2 support and I think lzma support is a little problematic for right now.

[migueldvb]

That is fine. It is good to have bzip2 support.