DYNAMIC_ARCH seems to fail for the CMake build (at least on MacOS)

[cesss]

Trying to build with

cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DTARGET=SKYLAKEX -DNUM_CORES=12 -DBUILD_WITHOUT_LAPACK=OFF -DDYNAMIC_ARCH=ON -DDYNAMIC_OLDER=ON -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_INSTALL_PREFIX="$HOME"/OpenBLAS/0.3.7 ../../../../src/OpenBLAS/0.3.7

I'm getting a lot of linker errors like these:

[...many more like these...]
"_xtrmm_kernel_RR_SANDYBRIDGE", referenced from:
      _gotoblas_SANDYBRIDGE in libopenblas.a(setparam_SANDYBRIDGE.c.o)
"_xtrmm_kernel_RR_SKYLAKEX", referenced from:
      _gotoblas_SKYLAKEX in libopenblas.a(setparam_SKYLAKEX.c.o)
[...many more like those...]  

I believe the linker line doesn't include all the required objects... in fact I searched the code and found that the suffix names in those files seems to be implemented for the Makefile system only, and not for CMake. I tried both with 0.3.7 and with the dev branch, and they both fail the same, at least on MacOS.

If I build with the provided Makefile instead of CMake, it builds fine.

[martin-frbg]

Strange - the cmake mechanism on OSX should be the same as elsewhere, and at least the current develop branch builds fine for me with DYNAMIC_ARCH on Linux. Could you upload the full build log here please, or at least check if there are any earlier errors from trying to build the missing functions ?

[cesss]

Here you have the full build log. My guess is that some source files might not be compiled at all (but then they are not even in the link line, because if they were, the linker would trigger a "file not found" error before the unresolved symbols errors. So it is like if a list of files was not correctly constructed, either by a shell script or by CMake.

BTW, I'm using CMake 3.15.5, if that matters.

Also note that I successfully built it with the standard Makefile procedure, so I can use OpenBLAS, this failure with CMake is not a showstopper for me.

log.txt.zip

[brada4]

Clang 7.0 spotted in the logs - it is archaic and certainly does not support AVX512 required by SkylakeX
Is your CPU capable of executing AVX-512 instructions?
What is your OS version? Xcode does not seem to match.

[brada4]

Reading the docs further - there is only 12 core SKYLAKE Xeon in apple's offering, which means what you are building will result in SIGILL as shared code is compiled in more advanced ISA.

[martin-frbg]

Normally you will want to set TARGET to the oldest cpu you expect to support (so that the shared code in OpenBLAS - thread setup, lapack and so on - will not depend on instructions that are available only on more modern systems.) This should have nothing to do with the linker errors however.

(Also if the compiler happens to be too old to support SkylakeX - I believe the minimum clang version for this was actually clang 6, not 7, this should be caught by the build system, you would see -DNO_AVX512 automatically added to the compiler commands and SkylakeX would simply behave like Haswell.)

[cesss]

Yes, the log I posted corresponds to my first attempt. When using the same settings for the standard Makefile procedure, I got SIGILL in one of the tests. Then I went to the homebrew recipe and built my copy with the same settings they do, with OpenMP -which I wasn't using- and disabling AVX512 (and using make instead of CMake) and then all tests passed. I'm now looking at the MacPorts way of installing OpenBLAS, and they also use the Makefile instead of CMake. It is interesting that they apply some patches of their own. I don't have time for looking at what these patches do at this moment.

I didn't know that TARGET should be set to the minimum CPU you plan to support (in fact I thought TARGET was superfluous and redundant once you build with DYNAMIC_ARCH). So, I built it incorrectly last night, because in my final build I choose Nehalem, but now I'm going to rebuild it again with Banias (Actually Yonah is the oldest to be supported, but Makefile.system doesn't seem to consider Yonah, and it does consider Banias). I'll set DYNAMIC_ARCH, but without DYNAMIC_OLD, because Banias doesn't seem to belong to DYNAMIC_OLD.

I'll first try with CMake (I have the feeling it's going to fail anyway at the link line, like yesterday), and then if it fails I'll proceed with the standard Makefile.

[martin-frbg]

CMake build problem seems to be related to DYNAMIC_OLDER, I have now reproduced it locally. And I pushed an update to README.md last night that documents DYNAMIC_ARCH (and its use of TARGET).

[martin-frbg]

Banias and Yonah are both 32 bit cpus, you will not get them in a libopenblas built for 64bit. Not sure why Yonah is not in the list for x86 - perhaps it was too similar to Banias. (The list of x86 cores is stil the same as it was in the GotoBLAS of 2010)

[cesss]

It also failed for me with CMake (and without DYNAMIC_OLDER). I'm setting now TARGET=CORE2 to see if BANIAS being out of the list was the reason for the failure.

[cesss]

Oh, how strange, now with TARGET=CORE2 and DYNAMIC_ARCH it fails much sooner with an error that didn't happen before:

/Users/cesss/build/MacOSX10.12-llvm7.0.0-x86_64/OpenBLAS/0.3.7/kernel/CMakeFiles/sgemm_incopy_NEHALEM.S:-186:2: error: expected relocatable expression
 PREFETCH RPREFETCHSIZE * 4(%r11)
 ^
/Users/cesss/build/MacOSX10.12-llvm7.0.0-x86_64/OpenBLAS/0.3.7/kernel/CMakeFiles/sgemm_incopy_NEHALEM.S:-185:2: error: expected relocatable expression
 PREFETCH RPREFETCHSIZE * 4(%r12)
 ^
/Users/cesss/build/MacOSX10.12-llvm7.0.0-x86_64/OpenBLAS/0.3.7/kernel/CMakeFiles/sgemm_incopy_NEHALEM.S:-184:2: error: expected relocatable expression
 PREFETCH RPREFETCHSIZE * 4(%r13)
 ^
/Users/cesss/build/MacOSX10.12-llvm7.0.0-x86_64/OpenBLAS/0.3.7/kernel/CMakeFiles/sgemm_incopy_NEHALEM.S:-183:2: error: expected relocatable expression
 PREFETCH RPREFETCHSIZE * 4(%rax)
 ^
/Users/cesss/build/MacOSX10.12-llvm7.0.0-x86_64/OpenBLAS/0.3.7/kernel/CMakeFiles/sgemm_incopy_NEHALEM.S:-181:2: error: expected relocatable expression
 PREFETCHW WPREFETCHSIZE * 4(%r8)
 ^
/Users/cesss/build/MacOSX10.12-llvm7.0.0-x86_64/OpenBLAS/0.3.7/kernel/CMakeFiles/sgemm_incopy_NEHALEM.S:-95:2: error: expected relocatable expression
 PREFETCH RPREFETCHSIZE * 4(%r11)
 ^
/Users/cesss/build/MacOSX10.12-llvm7.0.0-x86_64/OpenBLAS/0.3.7/kernel/CMakeFiles/sgemm_incopy_NEHALEM.S:-94:2: error: expected relocatable expression
 PREFETCH RPREFETCHSIZE * 4(%r12)
 ^
/Users/cesss/build/MacOSX10.12-llvm7.0.0-x86_64/OpenBLAS/0.3.7/kernel/CMakeFiles/sgemm_incopy_NEHALEM.S:-92:2: error: expected relocatable expression
 PREFETCHW WPREFETCHSIZE * 4(%r8)
 ^

Maybe this is the reason why the homebrew formula for OpenBLAS blacklists clang and uses gcc instead... (well, the comment line says gcc is needed because of gfortran, but they blacklist clang anyway in the next line --or that's what it seems, because I don't have good knowledge of the homebrew formulas syntax)

[martin-frbg]

The latest does look like a clang bug.
The DYNAMIC_ARCH problem appears to be a logic bug in the rules for creating the TRMM kernel names for those cpus that do not have a separate TRMM kernel. (Possibly caused by #1436) Not sure why this has not come up in the regular CI builds - but I had to cut down the list of cpus to keep the build times within the allowed limits.

[cesss]

Ok. Regarding the possible clang bug, I built it with the Makefile and it built fine, even the tests pass, so I searched in the build logs to see why sgemm_incopy_NEHALEM.S fails with CMake but not with make, but however sgemm_incopy_NEHALEM.S doesn't appear in the Makefile build log... why?

[cesss]

That error was not a clang bug, but that CMake is not honoring my CPPFLAGS, CFLAGS, nor CXXFLAGS, so it's not adding the -isysroot flag to the Apple SDK. How can I pass flags like this to the compiler (I'm used to use CFLAGS, CPPFLAGS and CXXFLAGS with other CMake projects, and they are usually honored)

BTW, I set the -DCMAKE_OSX_SYSROOT variable, and it's honored in some files, but not in others, so because of this some files fail to build (and this is why this compiler error happens with CMake but not with the Makefile)

[martin-frbg]

TARGET=SKYLAKEX appears to be the culprit - have not had time to debug exactly how it manages to break the build. (Edit: with HASWELL it fails as well, so must be a spillover of the USE_TRMM=1 as I suspected yesterday. Will commit a fix later today)