Building with native Apple Clang fails if NO_AVX512 not set

[zbeekman]

It seems that a build with GFortran 9 and Apple clang version 11.0.0 (clang-1100.0.33.17) fails unless NO_AVX512 is set in the environment.

This was discovered while updating the Homebrew formula. I suspect something similar may happen when using GCC 9 for C and Fortran too.

Here is what the build & environment look like:

DYNAMIC_ARCH=1
USE_OPENMP=1
make CC=clang FC=gfortran libs netlib shared
make PREFIX=/usr/local/Cellar/openblas/0.3.8 install

And here are the errors we see during CI:

15:22:04 ==> make CC=clang FC=gfortran libs netlib shared
15:22:04 make[1]: warning: -jN forced in submake: disabling jobserver mode.
15:22:04 clang -O2 -DMAX_STACK_ALLOC=2048 -DEXPRECISION -fopenmp -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=6 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=_saxpy -DASMFNAME=_saxpy_ -DNAME=saxpy_ -DCNAME=saxpy -DCHAR_NAME=\"saxpy_\" -DCHAR_CNAME=\"saxpy\" -DNO_AFFINITY -I.. -I. -UDOUBLE  -UCOMPLEX -c axpy.c -o saxpy.o
15:22:04 clang -O2 -DMAX_STACK_ALLOC=2048 -DEXPRECISION -fopenmp -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=6 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=_sswap -DASMFNAME=_sswap_ -DNAME=sswap_ -DCNAME=sswap -DCHAR_NAME=\"sswap_\" -DCHAR_CNAME=\"sswap\" -DNO_AFFINITY -I.. -I. -UDOUBLE  -UCOMPLEX -c swap.c -o sswap.o

...

[snip]

...

15:53:55 clang -c -O2 -DMAX_STACK_ALLOC=2048 -DEXPRECISION -fopenmp -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=6 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=_sgemv_t_SKYLAKEX -DASMFNAME=_sgemv_t_SKYLAKEX_ -DNAME=sgemv_t_SKYLAKEX_ -DCNAME=sgemv_t_SKYLAKEX -DCHAR_NAME=\"sgemv_t_SKYLAKEX_\" -DCHAR_CNAME=\"sgemv_t_SKYLAKEX\" -DNO_AFFINITY -DTS=_SKYLAKEX -I.. -DBUILD_KERNEL -DTABLE_NAME=gotoblas_SKYLAKEX -march=skylake-avx512 -UDOUBLE  -UCOMPLEX -UDOUBLE -UCOMPLEX  -DTRANS  ../kernel/x86_64/sgemv_t_4.c -o sgemv_t_SKYLAKEX.o
15:53:55 In file included from ../kernel/x86_64/saxpy.c:37:
15:53:55 ../kernel/x86_64/saxpy_microk_skylakex-2.c:41:13: error: always_inline function '_mm256_broadcastss_ps' requires target feature 'avx2', but would be inlined into function 'saxpy_kernel_16' that is compiled without support for 'avx2'
15:53:55         __alpha =  _mm256_broadcastss_ps(_mm_load_ss(alpha));
15:53:55                    ^
15:53:55 ../kernel/x86_64/saxpy_microk_skylakex-2.c:60:3: error: always_inline function '_mm256_storeu_ps' requires target feature 'avx', but would be inlined into function 'saxpy_kernel_16' that is compiled without support for 'avx'
15:53:55                 _mm256_storeu_ps(&y[i +  0], _mm256_loadu_ps(&y[i +  0]) + __alpha * _mm256_loadu_ps(&x[i +  0]));
15:53:55                 ^
15:53:55 ../kernel/x86_64/saxpy_microk_skylakex-2.c:60:32: error: always_inline function '_mm256_loadu_ps' requires target feature 'avx', but would be inlined into function 'saxpy_kernel_16' that is compiled without support for 'avx'
15:53:55                 _mm256_storeu_ps(&y[i +  0], _mm256_loadu_ps(&y[i +  0]) + __alpha * _mm256_loadu_ps(&x[i +  0]));

[isuruf]

Can you run clang --version?

[martin-frbg]

That looks more like an AVX2 problem, though perhaps NO_AVX512 would mask it somehow.
For gcc (and gfortran, if used), Makefile.x86_64 already adds either -march=skylake-avx512 or -mavx2 automatically as required, I did not observe that this is also required for clang (now ?)
EDIT: the -march=skylake-avx512 gets added for any compiler, only the -mavx2 on Haswell is limited to gcc

[zbeekman]

@isuruf

Can you run clang --version?

Included in my original post:

Apple clang version 11.0.0 (clang-1100.0.33.17)

[isuruf]

@zbeekman, does Apple clang support -fopenmp now?

[zbeekman]

Yes, looks like it:

$ clang --help

# ...

  -fobjc-weak             Enable ARC-style weak references in Objective-C
  -fopenmp-simd           Emit OpenMP code only for SIMD-based constructs.
  -fopenmp-targets=<value>
                          Specify comma-separated list of triples OpenMP offloading targets to be supported
  -fopenmp                Parse OpenMP pragmas and generate parallel code.
  -foptimization-record-file=<value>
                          Specify the file name of any generated YAML optimization record
  -foptimization-record-passes=<value>
                          Only include passes which match a specified regular expression in the generated optimization record (by default, include all passes)
  -fpack-struct=<value>   Specify the default maximum struct packing alignment
  -fpascal-strings        Recognize and construct Pascal-style string literals

[isuruf]

Weird. Here's what I get,

osx:~ isuru$ clang --version
Apple clang version 11.0.0 (clang-1100.0.33.17)
Target: x86_64-apple-darwin19.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
osx:~ isuru$ clang -fopenmp test.c
clang: error: unsupported option '-fopenmp'
clang: error: unsupported option '-fopenmp'

[zbeekman]

I think you need to do something like what is described here: https://iscinumpy.gitlab.io/post/omp-on-high-sierra/

[zbeekman]

Also, didn't realize you wanted the full output of --version:

Apple clang version 11.0.0 (clang-1100.0.33.17)
Target: x86_64-apple-darwin18.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

[isuruf]

I think you need to do something like what is described here: https://iscinumpy.gitlab.io/post/omp-on-high-sierra/

Yes, but the logs you gave doesn't use the -Xcompiler trick which made me think that you are using a different clang.

[zbeekman]

The following homebrew packages have a dependency on libomp which indicates that they use OpenMP with Apple Clang:

• cryfs
• faiss
• imagemagick
• libraw
• libtorch
• lightgbm
• ngt
• rpm
• siril

These are good candidates to look at to see how other build systems do this.

Looking at cryfs it appears that they are explicitly setting these compiler flags:

-Xpreprocessor -fopenmp -I/usr/local/opt/libomp/include

and explicitly telling their build system to link against /usr/local/opt/libomp/lib/libomp.dylib.

@isuruf

Yes, but the logs you gave doesn't use the -Xcompiler trick which made me think that you are using a different clang.

Forgive me, but I am a bit confused here. We are definitely building with AppleClang. The flags passed to the compiler are determined by and encoded in OpenBLAS' build system, unless we need to patch/override them. The only "special" things homebrew is doing in the OpenBLAS with Clang formula to enable OpenMP are:

1. Creating a dependency on libomp so that it is present
2. Setting USE_OPENMP=1 in the environment before calling OpenBLAS' make

I just double checked, and I'm a bit surprised that the OpenBLAS I just built with clang does not appear to link libomp at all:

$  brew linkage openblas
System libraries:
  /usr/lib/libSystem.B.dylib
Homebrew libraries:
  /usr/local/lib/gcc/9/libgcc_s.1.dylib (gcc)
  /usr/local/opt/gcc/lib/gcc/9/libgfortran.5.dylib (gcc)
  /usr/local/opt/gcc/lib/gcc/9/libgomp.1.dylib (gcc)
  /usr/local/opt/gcc/lib/gcc/9/libquadmath.0.dylib (gcc)
Dependencies with no linkage:
  libomp

So while this discussion of OpenMP seems orthogonal to the AVX512/ARM issue, it may still be important since I see linkage against libgomp and not libomp.

Should I open a new issue to track this? Or is there an environment variable we should be setting to get Make to pass the required OpenMP flags to clang?

[isuruf]

Can you run,

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -c -O2 -DMAX_STACK_ALLOC=2048 -DEXPRECISION -fopenmp -Wall -m64 -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=6 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=_saxpy_k_SKYLAKEX -DASMFNAME=_saxpy_k_SKYLAKEX_ -DNAME=saxpy_k_SKYLAKEX_ -DCNAME=saxpy_k_SKYLAKEX -DCHAR_NAME=\"saxpy_k_SKYLAKEX_\" -DCHAR_CNAME=\"saxpy_k_SKYLAKEX\" -DNO_AFFINITY -DTS=_SKYLAKEX -I. -DBUILD_KERNEL -DTABLE_NAME=gotoblas_SKYLAKEX -march=skylake-avx512 -UDOUBLE  -UCOMPLEX -UCOMPLEX -UDOUBLE kernel/x86_64/saxpy.c -o saxpy_k_SKYLAKEX.o -v

and send the output?

[zbeekman]

@isuruf no, because the skylake machine is our CI bot/infra in MacStadium. My local machine is Haswell, and I could remove the NO_AVX512 variable and build locally, but I won't have time for a bit to get to this. I've spent more of my day on this already than I should. But if it would be useful to you I can rebuild locally to trigger the same failure, send you the logs and you can then give me the command you want me to test drive.

[isuruf]

But if it would be useful to you I can rebuild locally to trigger the same failure, send you the logs and you can then give me the command you want me to test drive.

Sure. Let me know if you can reproduce locally and send me the logs.

[zbeekman]

I have reproduced locally before, but isn't the issue reported here unrelated to OpenMP? I want to verify OpenMP is working in the clang based build for sure, but I also want to make sure the ARM/AVX512 problem is addressed, if possible.

Honestly, the OpenMP issue is probably a bigger problem if the code is not compiling with threading properly enabled, so I'll try to improve the Homebrew formula test to verify OpenMP functionality too, and if that's broken I'll check in here about whether you want a new separate issue for OpenMP with clang in addition to this issue about ARM instructions being generated unless the NO_AVX512 variable is set.

Is there a small example code you could point me to, or copy/paste that exercises multithreaded OpenBLAS?

[isuruf]

I'll check in here about whether you want a new separate issue for OpenMP with clang in addition to this issue about ARM instructions being generated unless the NO_AVX512 variable is set.

This issue is not about ARM instructions. This is about a compiler issue with clang as @martin-frbg mentioned. I'd like to see the error again with verbose to see what the issue of the compiler is. I can't reproduce this issue without homebrew, so if you can run the failing command line with -v, then that'll be helpful.