Problem while building cblosc2 on ARM64

[kif]

Hi,
I am testing blosc2 on ARM64, actually the Pine64. Not a super powerful board but it is 64bits and has neon SIMD and this is what I am interested in.
https://www.pine64.org/devices/single-board-computers/pine-a64/
The operating system is debian10 hence kernel 4.19 and gcc8.3

Here is the error message, it is likely the expectations for following the standard is slightly different from one architecture to another ... but I am not that fluent in C:

kieffer@pine64:/data/workspace/c-blosc2$ cd build/
kieffer@pine64:/data/workspace/c-blosc2/build$ rm -rf *    
kieffer@pine64:/data/workspace/c-blosc2/build$ cmake ..
-- The C compiler identification is GNU 8.3.0
-- The CXX compiler identification is GNU 8.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Configuring for Blosc version: 2.0.0-beta.4.dev
-- Using LZ4 internal sources.
-- Using LIZARD internal sources.
-- Using MINIZ internal sources for ZLIB support.
-- Using ZSTD internal sources.
-- No IPP libraries found.
-- Not using IPP accelerated compression.
-- No build type specified. Defaulting to 'Release'.
-- Building for system processor aarch64
-- Adding run-time support for NEON
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
Skipping test_shuffle_roundtrip_altivec on non-ALTIVEC builds
Skipping test_shuffle_roundtrip_avx2 on non-AVX2 builds
Skipping test_shuffle_roundtrip_sse2 on non-SSE2 builds
-- Configuring done
CMake Warning (dev) at blosc/CMakeLists.txt:258 (add_library):
  Policy CMP0063 is not set: Honor visibility properties for all target
  types.  Run "cmake --help-policy CMP0063" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  Target "blosc2_static" of type "STATIC_LIBRARY" has the following
  visibility properties set for C:

    C_VISIBILITY_PRESET

  For compatibility CMake is not honoring them for this target.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done
-- Build files have been written to: /data/workspace/c-blosc2/build
kieffer@pine64:/data/workspace/c-blosc2/build$ make
Scanning dependencies of target blosc_shared_testing
[  1%] Building C object blosc/CMakeFiles/blosc_shared_testing.dir/blosc2.c.o
In file included from /data/workspace/c-blosc2/blosc/blosc2.c:18:
/data/workspace/c-blosc2/blosc/blosc2.h:1162:34: warning: 'struct timespec' declared inside parameter list will not be visible outside of this definition or declaration
 #define blosc_timestamp_t struct timespec
                                  ^~~~~~~~
/data/workspace/c-blosc2/blosc/blosc2.h:1168:39: note: in expansion of macro 'blosc_timestamp_t'
 BLOSC_EXPORT void blosc_set_timestamp(blosc_timestamp_t* timestamp);
                                       ^~~~~~~~~~~~~~~~~
/data/workspace/c-blosc2/blosc/blosc2.h:1162:34: warning: 'struct timespec' declared inside parameter list will not be visible outside of this definition or declaration
 #define blosc_timestamp_t struct timespec
                                  ^~~~~~~~
/data/workspace/c-blosc2/blosc/blosc2.h:1173:41: note: in expansion of macro 'blosc_timestamp_t'
 BLOSC_EXPORT double blosc_elapsed_nsecs(blosc_timestamp_t start_time,
                                         ^~~~~~~~~~~~~~~~~
/data/workspace/c-blosc2/blosc/blosc2.h:1162:34: warning: 'struct timespec' declared inside parameter list will not be visible outside of this definition or declaration
 #define blosc_timestamp_t struct timespec
                                  ^~~~~~~~
/data/workspace/c-blosc2/blosc/blosc2.h:1179:40: note: in expansion of macro 'blosc_timestamp_t'
 BLOSC_EXPORT double blosc_elapsed_secs(blosc_timestamp_t start_time,
                                        ^~~~~~~~~~~~~~~~~
In file included from /data/workspace/c-blosc2/blosc/blosc2.c:51:
/data/workspace/c-blosc2/internal-complibs/miniz-2.0.8/miniz.c:3031:9: note: #pragma message: Using fopen, ftello, fseeko, stat() etc. path for file I/O - this path may not support large files.
 #pragma message("Using fopen, ftello, fseeko, stat() etc. path for file I/O - this path may not support large files.")
         ^~~~~~~
/data/workspace/c-blosc2/blosc/blosc2.c: In function 'blosc_compress_context':
/data/workspace/c-blosc2/blosc/blosc2.c:1592:23: warning: passing argument 1 of 'blosc_set_timestamp' from incompatible pointer type [-Wincompatible-pointer-types]
   blosc_set_timestamp(&last);
                       ^~~~~
In file included from /data/workspace/c-blosc2/blosc/blosc2.c:18:
/data/workspace/c-blosc2/blosc/blosc2.h:1168:58: note: expected 'struct timespec *' but argument is of type 'struct timespec *'
 BLOSC_EXPORT void blosc_set_timestamp(blosc_timestamp_t* timestamp);
                                                          ^
/data/workspace/c-blosc2/blosc/blosc2.c:1644:25: warning: passing argument 1 of 'blosc_set_timestamp' from incompatible pointer type [-Wincompatible-pointer-types]
     blosc_set_timestamp(&current);
                         ^~~~~~~~
In file included from /data/workspace/c-blosc2/blosc/blosc2.c:18:
/data/workspace/c-blosc2/blosc/blosc2.h:1168:58: note: expected 'struct timespec *' but argument is of type 'struct timespec *'
 BLOSC_EXPORT void blosc_set_timestamp(blosc_timestamp_t* timestamp);
                                                          ^
/data/workspace/c-blosc2/blosc/blosc2.c:1645:39: error: type of formal parameter 1 is incomplete
     double ctime = blosc_elapsed_secs(last, current);
                                       ^~~~
/data/workspace/c-blosc2/blosc/blosc2.c:1645:45: error: type of formal parameter 2 is incomplete
     double ctime = blosc_elapsed_secs(last, current);
                                             ^~~~~~~
/data/workspace/c-blosc2/blosc/blosc2.c: In function 'blosc_get_complib_info':
/data/workspace/c-blosc2/blosc/blosc2.c:2592:14: warning: implicit declaration of function 'strdup'; did you mean 'strcmp'? [-Wimplicit-function-declaration]
   *complib = strdup(clibname);
              ^~~~~~
              strcmp
/data/workspace/c-blosc2/blosc/blosc2.c:2592:12: warning: assignment to 'char *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
   *complib = strdup(clibname);
            ^
/data/workspace/c-blosc2/blosc/blosc2.c:2593:12: warning: assignment to 'char *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
   *version = strdup(clibversion);
            ^
make[2]: *** [blosc/CMakeFiles/blosc_shared_testing.dir/build.make:63: blosc/CMakeFiles/blosc_shared_testing.dir/blosc2.c.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:127: blosc/CMakeFiles/blosc_shared_testing.dir/all] Error 2
make: *** [Makefile:163: all] Error 2

[kif]

Apparently it has to be C11, not C99 ... it compiles a bit further

[kif]

The auto-guess of the version of the glibc apparently is the root of the bug ...

[FrancescAlted]

So are you finally able to compile blosc2 in your pine64 box?

[kif]

On Thu, 12 Sep 2019 11:40:24 -0700 Francesc Alted ***@***.***> wrote: So are you finally able to compile blosc2 in your pine64 box?

I had to modify quite a bit the source code... enforcing the C11 spec was not enough. Instead, using a "#define __USE_POSIX199309" before including time.h helped. In time.h there is: #if defined __USE_POSIX199309 || defined __USE_ISOC11 # include <bits/types/struct_timespec.h> #endif I have no idea on how to fix this "properly" as all computers were running the same operating system. Oddities of the arm64 ? I did some benchmarking of filters, focusing of the "shuffle" and "bitshuffle", on different architectures in order to check the effectiveness of SIMD implementations. x86_64 (SSE2 & AVX2) and arm64 (NEON) for now and hopefully ppc64le (ALTIVEC) soon, correlated with the bloc size and the last level cache size of the processor. Quick conclusion are that SSE2 helps a lot, AVX2 is not so much faster if at all. For NEON it is even more complicated: the results are extremely dependent on the cache size. I can share them with you if you are interested. Just a comment on the API which is not exposed to the public: While the API looks consistent for shuffle, I think it is less consistent for bitshuffle. For example neon implementation is called: bitshuffle_neon(...) which is rather natural, but the generic, sse2 and avx2 implementations are called: bshuf_trans_bit_elem_scal(...) _sse2 and _avx2. Is there are reason for not keeping the "_generic" and the same base for the function name ? It would be perfect if the signatures could be similar but the bitshuffle takes an extra buffer argument. So maybe it is not worth changing this. Cheers, Jerome

[FrancescAlted]

Ha, it is curious because I don't remeber having this problems with struct_timespec.h in Ubuntu 16.04/18.04 for Raspberry. Maybe the issue is just in Debian? Very strange...

Regarding bshuf_trans_bit_elem_scal() name inconsistencies, yeah, currently that is quite a mess of naming, but fortunately, this is not part of the public API, so if you want to put some order in this naming jungle, you are welcome.

[kif]

Maybe it is debian ... I just got access (again) to the power9 with debian 10 and it is the same issue. I'll check twice on amd64 and arm32 but surprisingly it works there without complain.

About the unconsistency of the private API, I noticed it only because I benchmarked those functions.

[kif]

Coming back on this "failed to build from scratch", I tested on the power9 and arm64:

#include <time.h>
#include <stdio.h>
int main()
{
   printf("Posix is %d",__USE_POSIX199309);
   return 0;
}

And guess what? both are properly defined. May the error be in the cmake default options ?

[kif]

I tested if "features.h" was properly included or not and apparently it was. The error is somewhere else.
@picca this issue looks to be specific to debian10, any idea ?

[picca]

Hello, maybe the best would be contact the maintainer of c-blosc in Debian, and request an upload of the beta version of the currently beta version into experimental. This wait it will be build on all our architectures, and we could have a nice visibility of the portability of the code. https://tracker.debian.org/pkg/c-blosc Cheers Fred

[kif]

I just "debootstrapped" a ubuntu 18.04 system (gcc7) on ppc64le and the issue is apparently the same.

It should not be a debian issue. I will git-bisect.

[kif]

It compiles with clang7 but neither with gcc7 nor gcc8. The problem is apparently in the expand of the macros. It looks like std=c99 is not having effect, or at least not as much as on amd64

[kif]

I got it:
https://stackoverflow.com/questions/3875197/gcc-with-std-c99-complains-about-not-knowing-struct-timespec

PR under creation