Build failure on raspberry pi running arch

[mike-lawrence]

Log:

    [user@scene_pi python-blosc]$ sudo python setup.py install
    running install
    running bdist_egg
    running egg_info
    writing dependency_links to blosc.egg-info/dependency_links.txt
    writing top-level names to blosc.egg-info/top_level.txt
    writing blosc.egg-info/PKG-INFO
    reading manifest file 'blosc.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no files found matching '*.txt'
    warning: no files found matching '*.cpp' under directory 'c-blosc'
    warning: no files found matching '*.hpp' under directory 'c-blosc'
    writing manifest file 'blosc.egg-info/SOURCES.txt'
    installing library code to build/bdist.linux-armv7l/egg
    running install_lib
    running build_py
    creating build
    creating build/lib.linux-armv7l-3.4
    creating build/lib.linux-armv7l-3.4/blosc
    copying blosc/test.py -> build/lib.linux-armv7l-3.4/blosc
    copying blosc/__init__.py -> build/lib.linux-armv7l-3.4/blosc
    copying blosc/version.py -> build/lib.linux-armv7l-3.4/blosc
    copying blosc/toplevel.py -> build/lib.linux-armv7l-3.4/blosc
    running build_ext
    building 'blosc.blosc_extension' extension
    creating build/temp.linux-armv7l-3.4
    creating build/temp.linux-armv7l-3.4/blosc
    creating build/temp.linux-armv7l-3.4/c-blosc
    creating build/temp.linux-armv7l-3.4/c-blosc/blosc
    creating build/temp.linux-armv7l-3.4/c-blosc/internal-complibs
    creating build/temp.linux-armv7l-3.4/c-blosc/internal-complibs/lz4-1.7.0
    creating build/temp.linux-armv7l-3.4/c-blosc/internal-complibs/snappy-1.1.1
    creating build/temp.linux-armv7l-3.4/c-blosc/internal-complibs/zlib-1.2.8
    gcc -pthread -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16 -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 -fPIC -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -Ic-blosc/blosc -Ic-blosc/internal-complibs/lz4-1.7.0 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -I/usr/include/python3.4m -c blosc/blosc_extension.c -o build/temp.linux-armv7l-3.4/blosc/blosc_extension.o
    gcc -pthread -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16 -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 -fPIC -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -Ic-blosc/blosc -Ic-blosc/internal-complibs/lz4-1.7.0 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -I/usr/include/python3.4m -c c-blosc/blosc/blosc.c -o build/temp.linux-armv7l-3.4/c-blosc/blosc/blosc.o
    c-blosc/blosc/blosc.c: In function 'blosc_getitem':
    c-blosc/blosc/blosc.c:1275:7: warning: unused variable 'tmp_init' [-Wunused-variable]
       int tmp_init = 0;
           ^
    gcc -pthread -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16 -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 -fPIC -DHAVE_LZ4=1 -DHAVE_SNAPPY=1 -DHAVE_ZLIB=1 -Ic-blosc/blosc -Ic-blosc/internal-complibs/lz4-1.7.0 -Ic-blosc/internal-complibs/snappy-1.1.1 -Ic-blosc/internal-complibs/zlib-1.2.8 -I/usr/include/python3.4m -c c-blosc/blosc/shuffle-sse2.c -o build/temp.linux-armv7l-3.4/c-blosc/blosc/shuffle-sse2.o
    c-blosc/blosc/shuffle-sse2.c:14:4: error: #error SSE2 is not supported by the target architecture/platform and/or this compiler.
       #error SSE2 is not supported by the target architecture/platform and/or this compiler.
        ^
    c-blosc/blosc/shuffle-sse2.c:17:23: fatal error: emmintrin.h: No such file or directory
    compilation terminated.
    error: command 'gcc' failed with exit status 1

[FrancescAlted]

Hi,

Yes, python-blosc is being developed in Intel and normally only checked with it. To overcome this problem we need to recognize the architecture prior to call the C compiler: this would allow to set compiler options accordingly. PR welcome!

[mike-lawrence]

Ah, I thought that because c-blosc is supposedly compatible with arm/RPi (at least, according to the cblosc 1.3.6 to 1.4.0 change notes), that python-blosc would be as well.

[esc]

Any news on this? Perhaps a new version has fixed the issue? Or maybe it magically disappeared?

[itdaniher]

I had a surprising amount of luck with blosc on a recent ARM core. Eagerly looking forward to c-blosc2's NEON support, but figured I'd chime in and suggest that there no fundamental issues with c-blosc or python-blosc on ARM.

Some stats and comparisons below.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
python-blosc version: 1.4.4
Blosc version: 1.11.2 ($Date:: 2017-01-27 #$)
Compressors available: ['blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib', 'zstd']
Compressor library versions:
  BloscLZ: 1.0.5
  LZ4: 1.7.5
  Snappy: 1.1.1
  Zlib: 1.2.8
  Zstd: 1.1.2
Python version: 3.6.0 (default, Dec 31 2016, 21:20:16)
[GCC 4.9.2]
Platform: Linux-3.4.113-sun8i-armv7l (#50 SMP PREEMPT Mon Nov 14 08:41:55 CET 2016)
Linux dist: debian 9.0
Processor: not recognized
Byte-ordering: little
Detected cores: 4
Number of threads to use by default: 4
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
  *** ctypes.memmove() *** Time for memcpy():   0.015 s (93.57 MB/s)

Times for compressing/decompressing with clevel=5 and 4 threads

*** user input ***
  *** blosclz , noshuffle  ***  0.015 s (89.93 MB/s) / 0.010 s (138.32 MB/s)    Compr. ratio:   2.7x
  *** blosclz , shuffle    ***  0.023 s (60.25 MB/s) / 0.012 s (112.71 MB/s)    Compr. ratio:   2.3x
  *** blosclz , bitshuffle ***  0.018 s (77.63 MB/s) / 0.021 s (66.76 MB/s)     Compr. ratio:   7.3x
  *** lz4     , noshuffle  ***  0.008 s (177.14 MB/s) / 0.009 s (159.00 MB/s)   Compr. ratio:   3.6x
  *** lz4     , shuffle    ***  0.010 s (131.29 MB/s) / 0.012 s (117.69 MB/s)   Compr. ratio:   3.5x
  *** lz4     , bitshuffle ***  0.015 s (89.97 MB/s) / 0.022 s (63.62 MB/s)     Compr. ratio:   8.4x
  *** lz4hc   , noshuffle  ***  0.071 s (19.30 MB/s) / 0.007 s (186.64 MB/s)    Compr. ratio:   8.6x
  *** lz4hc   , shuffle    ***  0.079 s (17.30 MB/s) / 0.014 s (95.99 MB/s)     Compr. ratio:   6.2x
  *** lz4hc   , bitshuffle ***  0.062 s (22.23 MB/s) / 0.027 s (51.53 MB/s)     Compr. ratio:   9.7x
  *** snappy  , noshuffle  ***  0.008 s (173.87 MB/s) / 0.009 s (148.77 MB/s)   Compr. ratio:   4.4x
  *** snappy  , shuffle    ***  0.011 s (123.22 MB/s) / 0.016 s (85.16 MB/s)    Compr. ratio:   4.4x
  *** snappy  , bitshuffle ***  0.015 s (89.02 MB/s) / 0.021 s (64.87 MB/s)     Compr. ratio:   6.2x
  *** zlib    , noshuffle  ***  0.047 s (29.26 MB/s) / 0.011 s (121.83 MB/s)    Compr. ratio:  14.7x
  *** zlib    , shuffle    ***  0.080 s (17.20 MB/s) / 0.022 s (63.61 MB/s)     Compr. ratio:   9.4x
  *** zlib    , bitshuffle ***  0.059 s (23.50 MB/s) / 0.033 s (41.10 MB/s)     Compr. ratio:  10.5x
  *** zstd    , noshuffle  ***  0.113 s (12.21 MB/s) / 0.011 s (124.64 MB/s)    Compr. ratio:  15.6x
  *** zstd    , shuffle    ***  0.154 s (8.92 MB/s) / 0.026 s (52.56 MB/s)      Compr. ratio:   9.9x
  *** zstd    , bitshuffle ***  0.116 s (11.86 MB/s) / 0.036 s (38.40 MB/s)     Compr. ratio:  11.4x

Script is a slightly modified version of compress_ptr.py -
https://gist.github.com/itdaniher/544215e2ebd25495a95d9165633972c6

Tried a few different formats for my sampled analog signal compression - pysoundfile's ogg and flac encodings, python-lz4, and python's integrated zlib. Very happy with Blosc overall - stellar performance even without native SIMD (yet!)

[itdaniher]

@FrancescAlted I'd be glad to order your choice of single board computer to better enable you to test on ARM CPUs. Also have plenty of recommendations, the ability to offer native hosted ARM continuous integration, etc. Contact me @gmail.com if you're interested! Thanks for Blosc-and-friends!

[FrancescAlted]

@itdaniher Thanks for reporting. As you had no fundamental flaws with C-Blosc in ARM, I am going to close this one. I already contacted you about the ARM board and CI for ARM. Thanks again.