Failing Travis tests

[oogali]

@esc So, it looks we have some test failures in Travis, but they're not what I expected?

The errors seem to fall into two categories.

1. It looks like the expected compression ratio has been cut in half.

2. The newer version of blosc introduces the zstd compression library, in which the tests don't expect that in the output.

The latter is easily fixable by updating the tests. The former... not so much.

[esc]

Correct, initially I was very pedantic about maximum test coverage, maybe the ratio tests are too strict though and perhaps should be refactored to simply expect an output of a certain length, rather than a specific ratio. I will do it eventually, but don't mind being pre-empted. :) 👍

[esc]

https://twitter.com/hintjens/status/758606665553969152

[esc]

You can probably use regular expressions in the cram tests, I've done that already in a few places, feel free to use that as template.

[esc]

The zstd this is a real bug in the tests, btw...

[oogali]

Would this be an appropriate change for the ratio tests?

diff --git a/test_cmdline/test_append.cram b/test_cmdline/test_append.cram
index 67cc677..1a7bf29 100644
--- a/test_cmdline/test_append.cram
+++ b/test_cmdline/test_append.cram
@@ -12,12 +12,14 @@ Create a test datafile.

 Test basic append:

+  $ ls -lah  data.dat
+  .* 1 .* .* \d{3}M .* .* .* data.dat (re)
   $ blpk compress data.dat
   $ ls -lah  data.dat.blp
-  .* 1 .* .* 13M .* .* .* data.dat.blp (re)
+  .* 1 .* .* \d{2}M .* .* .* data.dat.blp (re)
   $ blpk append data.dat.blp data.dat
   $ ls -lah  data.dat.blp
-  .* 1 .* .* 26M .* .* .* data.dat.blp (re)
+  .* 1 .* .* \d{2}M .* .* .* data.dat.blp (re)

 Cleanup.

[esc]

Yeah, that would be fine.

[oogali]

I want to respect the pedantry, but I'm not quite sure on how to do that without making the tests a bit more complex.

If I use regular expressions, \d+ lets me match any digit... including zeroes. So if a 0B file is emitted, the tests will pass.

In order to guard against that, I only see two options:

• Change all the digit matching from \d+ to something along the lines of [1-9]\d*
• Pepper the tests with if statements, which then negates using cram for testing...

Halp?

[esc]

How about [1-9]\d* ? Does that work? 0B files would be OK too, I guess... Putting if statements is a no-go.

[esc]

BTW: I promise to make a release within 24 hours after this issues has been fixed. 😁

[oogali]

I have all of the tests fixed to use regular expressions to match digits (and match non-zero numbers where needed).

I had to then slice up some of the dictionaries to do a partial match (exclude compression ratio and sizes from the test), hope that's okay.

However, we seem to have a real test failure in test_headers.test_decode_blosc_header (see: https://travis-ci.org/oogali/bloscpack/builds/149805682). It looks like the flag change isn't picked up by the decode function.

[esc]

I guess the way the internal flags are stored/used may have changed. cc @FrancescAlted

[esc]

Actually, the problem lies elsewhere, my mistake. The no-shuffle flag is being picked up already (!). It seems like the problem lies with the test-data. This data is compressible when shuffled but--probably due to changes in blosc itself--are now suddenly incompressible when not shuffled. This is why the flag has the value two/second bit. You can try this for yourself with the following patch:

diff --git i/test/test_headers.py w/test/test_headers.py
index 211cf7e330..768175863d 100644
--- i/test/test_headers.py
+++ w/test/test_headers.py
@@ -140,6 +140,8 @@ def test_decode_blosc_header():
     header_slice = dict((k, header[k]) for k in expected.keys())
     nt.assert_equal(expected, header_slice)
     # deactivate shuffle
+
+    array_ = np.ones(16000, dtype=np.uint8)
     blosc_args.shuffle = False
     compressed = blosc.compress(array_, **blosc_args)
     header = decode_blosc_header(compressed)

This data is compressible for sure and the flags are neither 1 (shuffle) nor 2 (pure memcpy/incompressible) nor 3 (both).

[esc]

closed by #57