Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

blosc_extension.error: Error -1 while compressing data (Issue probably coming from Shuffle filter) #62

Closed
josephernest opened this issue Mar 15, 2014 · 12 comments
Closed

blosc_extension.error: Error -1 while compressing data (Issue probably coming from Shuffle filter) #62

josephernest opened this issue Mar 15, 2014 · 12 comments

Comments

@josephernest
Copy link

When using this code (with blosc-1.2.1.win-amd64-py2.7.exe, Python 2.7 64bits, Windows 7 x64) :

import numpy as np
import blosc

with open('blah.bin','rb') as f:
    w = np.fromstring(f.read(), dtype=np.int16)
print w    # w is a regular, standard, numpy array

z = blosc.pack_array(w, cname='lz4')   # here we have a bug

then we have this crash :
(Please download the blah.bin file here : https://dl.dropboxusercontent.com/u/83031018/blah.bin)

[   0    0    2 ..., -872 -258 -599]
Traceback (most recent call last):
  File "D:\Documents\projects\coding\python\vrac\compression\blosc_bug.py", line 7, in <module>
    z = blosc.pack_array(w, cname='lz4')
  File "C:\Python27-64\lib\site-packages\blosc\toplevel.py", line 579, in pack_array
    packed_array = compress(pickled_array, itemsize, clevel, shuffle, cname)
  File "C:\Python27-64\lib\site-packages\blosc\toplevel.py", line 309, in compress
    return _ext.compress(bytesobj, typesize, clevel, shuffle, cname)
blosc_extension.error: Error -1 while compressing data

Important note : When shuffle = False, there is no more error. So the issue probably comes from "Shuffle".
@esc
Copy link
Member

esc commented Mar 17, 2014

I can not reproduce this error. My machine details are as follows:

@Basjo what are your machine details and how did you install python-blosc?

@josephernest
Copy link
Author

@esc I used

@FrancescAlted
Copy link
Member

Hmm, I am not able to reproduce this in Mac OSX, Linux or even Windows (using Python 3.3 64bit). I aways get something similar to:

C:\Users\Francesc\software\python-blosc>python prova.py
[   0    0    2 ..., -872 -258 -599]

Here are my details:

C:\Users\Francesc\software\python-blosc>python blosc\test.py
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
python-blosc version: 1.2.2.dev
Blosc version: 1.3.3 ($Date:: 2014-02-15 #$)
Compressors available: ['blosclz', 'lz4', 'lz4hc']
Compressor library versions:
  BloscLZ: 1.0.1
  LZ4: 1.1.3
Python version: 3.3.1 (v3.3.1:d9893d13c628, Apr  6 2013, 20:30:21) [MSC v.1600
4 bit (AMD64)]
Platform: Windows-7-AMD64 (6.1.7601)
Processor: Intel64 Family 6 Model 58 Stepping 9, GenuineIntel
Byte-ordering: little
Detected cores: 4
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
test_all_compressors (__main__.TestCodec) ... ok
test_basic_codec (__main__.TestCodec) ... ok
test_compress_exceptions (__main__.TestCodec) ... ok
test_compress_ptr_exceptions (__main__.TestCodec) ... ok
test_decompress_exceptions (__main__.TestCodec) ... ok
test_decompress_ptr_exceptions (__main__.TestCodec) ... ok
test_get_clib (__main__.TestCodec) ... ok
test_pack_array_exceptions (__main__.TestCodec) ... ok
test_set_nthreads_exceptions (__main__.TestCodec) ... ok
test_unpack_array_exceptions (__main__.TestCodec) ... ok
compress (blosc.toplevel)
Doctest: blosc.toplevel.compress ... ok
compress_ptr (blosc.toplevel)
Doctest: blosc.toplevel.compress_ptr ... ok
decompress (blosc.toplevel)
Doctest: blosc.toplevel.decompress ... ok
decompress_ptr (blosc.toplevel)
Doctest: blosc.toplevel.decompress_ptr ... ok
free_resources (blosc.toplevel)
Doctest: blosc.toplevel.free_resources ... ok
pack_array (blosc.toplevel)
Doctest: blosc.toplevel.pack_array ... ok
set_nthreads (blosc.toplevel)
Doctest: blosc.toplevel.set_nthreads ... ok
unpack_array (blosc.toplevel)
Doctest: blosc.toplevel.unpack_array ... ok

----------------------------------------------------------------------
Ran 18 tests in 0.504s

OK

I would say there should be something wrong in the 2.7 build for Win.

@esc
Copy link
Member

esc commented Mar 18, 2014

@FrancescAlted can you test the the pre-built binaries? I tried accessing the page, but it seems down right now.

@FrancescAlted
Copy link
Member

@esc It is not exactly down, but very slow. @Basjo could you try with Python 3.3 in your computer?

@FrancescAlted
Copy link
Member

Checked with Python 2.7 and MinGW (installed via Anaconda):

C:\Users\Francesc\software\python-blosc>python prova.py
[   0    0    2 ..., -872 -258 -599]

C:\Users\Francesc\software\python-blosc>python blosc\test.py
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
python-blosc version: 1.2.2.dev
Blosc version: 1.3.3 ($Date:: 2014-02-15 #$)
Compressors available: ['blosclz', 'lz4', 'lz4hc']
Compressor library versions:
  BloscLZ: 1.0.1
  LZ4: 1.1.3
Python version: 2.7.4 |Anaconda 1.8.0 (32-bit)| (default, Apr  9 2013, 12:19:24)
 [MSC v.1500 32 bit (Intel)]
Platform: Windows-7-AMD64 (6.1.7601)
Processor: Intel64 Family 6 Model 58 Stepping 9, GenuineIntel
Byte-ordering: little
Detected cores: 4
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
test_all_compressors (__main__.TestCodec) ... ok
test_basic_codec (__main__.TestCodec) ... ok
test_compress_exceptions (__main__.TestCodec) ... ok
test_compress_ptr_exceptions (__main__.TestCodec) ... ok
test_decompress_exceptions (__main__.TestCodec) ... ok
test_decompress_ptr_exceptions (__main__.TestCodec) ... ok
test_get_clib (__main__.TestCodec) ... ok
test_pack_array_exceptions (__main__.TestCodec) ... ok
test_set_nthreads_exceptions (__main__.TestCodec) ... ok
test_unpack_array_exceptions (__main__.TestCodec) ... ok
compress (blosc.toplevel)
Doctest: blosc.toplevel.compress ... ok
compress_ptr (blosc.toplevel)
Doctest: blosc.toplevel.compress_ptr ... ok
decompress (blosc.toplevel)
Doctest: blosc.toplevel.decompress ... ok
decompress_ptr (blosc.toplevel)
Doctest: blosc.toplevel.decompress_ptr ... ok
free_resources (blosc.toplevel)
Doctest: blosc.toplevel.free_resources ... ok
pack_array (blosc.toplevel)
Doctest: blosc.toplevel.pack_array ... ok
set_nthreads (blosc.toplevel)
Doctest: blosc.toplevel.set_nthreads ... ok
unpack_array (blosc.toplevel)
Doctest: blosc.toplevel.unpack_array ... ok

----------------------------------------------------------------------
Ran 18 tests in 0.411s

OK

Everything seems fine here too.

@josephernest
Copy link
Author

Hi @FrancescAlted, could you send me your latest installer for that ? Then I could try with your latest version, maybe it would solve it.

Important info : I tried it with Python 2.7 32bits, and it works : no error.
The error only comes when using Python 2.7 64bits.

However the test.py works fine even with Python 2.7 64bits :

test_all_compressors (__main__.TestCodec) ... ok
test_basic_codec (__main__.TestCodec) ... ok
test_compress_exceptions (__main__.TestCodec) ... ok
test_compress_ptr_exceptions (__main__.TestCodec) ... ok
test_decompress_exceptions (__main__.TestCodec) ... ok
test_decompress_ptr_exceptions (__main__.TestCodec) ... ok
test_get_clib (__main__.TestCodec) ... ok
test_pack_array_exceptions (__main__.TestCodec) ... ok
test_set_nthreads_exceptions (__main__.TestCodec) ... ok
test_unpack_array_exceptions (__main__.TestCodec) ... ok
compress (blosc.toplevel)
Doctest: blosc.toplevel.compress ... ok
compress_ptr (blosc.toplevel)
Doctest: blosc.toplevel.compress_ptr ... ok
decompress (blosc.toplevel)
Doctest: blosc.toplevel.decompress ... ok
decompress_ptr (blosc.toplevel)
Doctest: blosc.toplevel.decompress_ptr ... ok
free_resources (blosc.toplevel)
Doctest: blosc.toplevel.free_resources ... ok
pack_array (blosc.toplevel)
Doctest: blosc.toplevel.pack_array ... ok
set_nthreads (blosc.toplevel)
Doctest: blosc.toplevel.set_nthreads ... ok
unpack_array (blosc.toplevel)
Doctest: blosc.toplevel.unpack_array ...         -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
python-blosc version: 1.2.1

Blosc version: 1.3.3 ($Date:: 2014-02-15 #$)
Compressors available: ['blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib']

Compressor library versions:
  BloscLZ: 1.0.1
  LZ4: 1.1.3
  Snappy: 1.1.1
  Zlib: 1.2.8
Python version: 2.7.6 (default, Nov 10 2013, 19:24:24) [MSC v.1500 64 bit (AMD64)]
Platform: Windows-7-AMD64 (6.1.7600)
Processor: Intel64 Family 6 Model 42 Stepping 7, GenuineIntel

Byte-ordering: little
Detected cores: 4
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
ok

----------------------------------------------------------------------
Ran 18 tests in 0.249s

OK

@FrancescAlted
Copy link
Member

Hi @Basjo, I tried the binaries at http://www.lfd.uci.edu/~gohlke/pythonlibs/#blosc, and I can reproduce your error (however, using other compressors instide Blosc like blosclz, snappy and zlib work perfectly fine).

Sorry, but I cannot compile Blosc for Python 2.7 because I only have Visual Studio 2010/2012, and for this I would need 2008. But for Python 3.3 64-bit and VS2010 everything is fine here, so I think the best would be to report this ticket to Christoph Gohlke.

@josephernest
Copy link
Author

Hi @FrancescAlted Yes it's the same for me : the error is only for lz4. When using blosclz, snappy, or zlib, no error.
I could report this ticket to C. Gohlke, but I don't think it would change anything : the binaries were properly compiled (when I run test.py, all is "OK"), so if he compiles again the same code, it will give the same result, don't you think so ?

What is the internal blosc reason for a blosc_extension.error: Error -1 while compressing data, what could it be ?

By the way, when you try with shuffle = False, do you get an error as well ?

@esc
Copy link
Member

esc commented Mar 20, 2014

-1 is the error that blosc returns in case anything went wrong. This can happen in a multiple of error conditions, not enough space, buffer overrun and others.

You need to grep for -1 in:

https://github.com/Blosc/c-blosc/blob/master/blosc/blosc.c

@FrancescAlted
Copy link
Member

Yes, I discovered what I think was the root of the problem (incidentally, this affected to BLZ too: ContinuumIO/blz#5). The fix was basically this one:

Blosc/c-blosc@ef311ed

Closing this. @Basjo in case you are still getting the error on Win, please reopen this ticket.
I have released new versions of c-blosc (1.3.4) and python-blosc (1.2.2) with this fix in.

@josephernest
Copy link
Author

Thanks @FrancescAlted, problem solved !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@esc @FrancescAlted @josephernest