Support LZMA on Python 2 via backports.lzma

[jakirkham]

Fixes https://github.com/alimanfoo/numcodecs/issues/11

Tries to use backports.lzma as a fallback. Adds testing for it with Travis CI.

[alimanfoo]

Thanks @jakirkham. I'm looking at the travis logs and I don't think backports.lzma is getting installed, which then means the tests for the LZMA codec are not being run. I don't know why, I'm guessing something to do with the "if" condition. Is there not a travis environment variable to check which is current major Python version?

[jakirkham]

Yeah, I had some ' that caused me some issues. Have corrected. Looks like that issue is fixed now.

Is there not a travis environment variable to check which is current major Python version?

I looked for one briefly in the docs, but didn't see one. 😞 If you find something like that, let me know and I'll happily change it.

[jakirkham]

Getting back to the matter at hand, it seems that check in the codec's config is represented as a long in Python 2, but the rest are int. In Python 3, I'm guessing these are all long. We might need to tweak some things with storing config values and encoding/decoding to JSON to support Python 2/3.

[alimanfoo]

Re detecting travis python version, I think there's a $TRAVIS_PYTHON_VERSION variable, see this example.

[alimanfoo]

Something I don't understand: The Python 2.7 build on travis is failing, because apparently the LZMA codec is not available, which suggests that backports.lzma was not installed successfully. However, if I look at the travis log, there is a section where it claims it is installed successfully. But then when I try to install backports.lzma on my local system, it cannot compile because lzma.h is not found, because I haven't installed XZ utils.

Could you try adding:

- if [[ $TRAVIS_PYTHON_VERSION == '2.7' ]]; then pip install -U backports.lzma; fi

...into the travis.yml file to see if gets installed successfully outside of tox?

[alimanfoo]

I think I see why the PY27 travis tests are failing. The name LZMA needs to get conditionally imported up into the root numcodecs namespace. Some modification is needed to this section of code.

[alimanfoo]

How does the int/long issue come up? The following works for me on PY27:

(spike-lzma) aliman@uluru:~$ python
Python 2.7.12+ (default, Sep 17 2016, 12:08:02) 
[GCC 6.2.0 20160914] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from numcodecs.lzma import LZMA
>>> repr(eval('LZMA(format=1, check=0, preset=1, filters=None)'))
'LZMA(format=1, check=0, preset=1, filters=None)'

[jakirkham]

Thanks for all the tips.

I did recall there was an environment variable for Python on Travis CI. Thank for figuring out what it was.

We can try moving the install back outside tox. However, when I had fixed the syntax errors I found that it wasn't able to import it unless it was installed in the environment that we were using for testing. 😕 Then again I don't use tox very often. So I may just be missing something.

Good point on the LZMA codec registration code. Was familiarizing myself with that the other day. Changing that should be straightforward.

I'll take a look at the long/int issues again. IIRC the issue came up on this line. I'm not sure why, but I think _lzma.CHECK_NONE was coming up as a long.

[alimanfoo]

On Tue, Feb 21, 2017 at 2:45 PM, jakirkham ***@***.***> wrote: Thanks for all the tips. I did recall there was an environment variable for Python on Travis CI. Thank for figuring out what it was. We can try moving the install back outside tox. However, when I had fixed the syntax errors I found that it wasn't able to import it unless it was installed in the environment that we were using for testing. 😕 Then again I don't use tox very often. So I may just be missing something.

No, you're probably right, it does need to be installed inside the tox environment. I think I just wanted to check it would install properly outside tox. But thrashing around a bit here, don't fully understand what's going on.

Good point on the LZMA codec registration code. Was familiarizing myself with that the other day. Changing that should be straightforward.

Thanks.

I'll take a look at the long/int issues again. IIRC the issue came up on this line <https://github.com/alimanfoo/numcodecs/blob/7a3a3b904c3a4e5baf8615cbec3b16d69f56c927/numcodecs/tests/test_lzma.py#L44>. I'm not sure why, but I think _lzma.CHECK_NONE was coming up as a long.

Ah, OK, thanks. I was looking at the changes you made to test_repr.

…

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> The Wellcome Trust Centre for Human Genetics Roosevelt Drive Oxford OX3 7BN United Kingdom Email: alimanfoo@googlemail.com Web: http://purl.org/net/aliman Twitter: https://twitter.com/alimanfoo Tel: +44 (0)1865 287721

[jakirkham]

Well, I think registering the LZMA codec on Python 2 was all that was still needed. This seems to be installing ok and passing the tests.

[alimanfoo]

Many thanks, one small comment, otherwise looks good to go.

[alimanfoo]

Is this conditional logic needed for PY2? On my local system check_repr('LZMA(format=1, check=0, preset=1, filters=None)') works ok for both PY2 and PY3.

[jakirkham]

Yep, you're right. Was running into this issue before updating the registration of LZMA. Now seems ok. So have reverted this change.

[alimanfoo]

Can possibly remove this, see comment on test_repr.

[jakirkham]

Have addressed your comments. Please let me know if there is anything else.

[alimanfoo]

Thank you, very nice compatibility add.

[jakirkham]

Of course.

[jakirkham]

Proposing a backport(?) of this to Zarr in PR ( https://github.com/alimanfoo/zarr/pull/123 ).