Homebrew build of 0.3.8 fails with GCC 9, passes with native Apple clang when OpenMP enabled

[zbeekman]

EDIT: Possible dupe of #2416

I was upgrading Homebrew's OpenBLAS to 0.3.8 and the build failed with GCC 9.2.

The build errors are are all error: invalid instruction mnemonic 'dmb' in level3_thread.c.

Fortunately using native Apple Clang (which Homebrew prefers to do) resolves this issue.

Also, I'm not sure whether or not you are aware, but Homebrew switched to using OpenBLAS as the default BLAS and LAPACK implementation instead of Apple's (outdated) Accelerate framework in the last 6 months/year.

Here is a more extended log incase the Jenkins CI log gets deleted (which it will after a while).

# ...

gcc-9 -O2 -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DNO_AVX512 -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=6 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=_sgemm_thread_nn -DASMFNAME=_sgemm_thread_nn_ -DNAME=sgemm_thread_nn_ -DCNAME=sgemm_thread_nn -DCHAR_NAME=\"sgemm_thread_nn_\" -DCHAR_CNAME=\"sgemm_thread_nn\" -DNO_AFFINITY -I../.. -UDOUBLE  -UCOMPLEX  -c -DTHREADED_LEVEL3 -UDOUBLE -UCOMPLEX -DNN gemm.c -o sgemm_thread_nn.o
gcc-9 -O2 -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DNO_AVX512 -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=6 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=_sgemm_thread_nt -DASMFNAME=_sgemm_thread_nt_ -DNAME=sgemm_thread_nt_ -DCNAME=sgemm_thread_nt -DCHAR_NAME=\"sgemm_thread_nt_\" -DCHAR_CNAME=\"sgemm_thread_nt\" -DNO_AFFINITY -I../.. -UDOUBLE  -UCOMPLEX  -c -DTHREADED_LEVEL3 -UDOUBLE -UCOMPLEX -DNT gemm.c -o sgemm_thread_nt.o
gcc-9 -O2 -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DNO_AVX512 -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=6 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=_sgemm_thread_tn -DASMFNAME=_sgemm_thread_tn_ -DNAME=sgemm_thread_tn_ -DCNAME=sgemm_thread_tn -DCHAR_NAME=\"sgemm_thread_tn_\" -DCHAR_CNAME=\"sgemm_thread_tn\" -DNO_AFFINITY -I../.. -UDOUBLE  -UCOMPLEX  -c -DTHREADED_LEVEL3 -UDOUBLE -UCOMPLEX -DTN gemm.c -o sgemm_thread_tn.o
gcc-9 -O2 -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DNO_AVX512 -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=6 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=_sgemm_thread_tt -DASMFNAME=_sgemm_thread_tt_ -DNAME=sgemm_thread_tt_ -DCNAME=sgemm_thread_tt -DCHAR_NAME=\"sgemm_thread_tt_\" -DCHAR_CNAME=\"sgemm_thread_tt\" -DNO_AFFINITY -I../.. -UDOUBLE  -UCOMPLEX  -c -DTHREADED_LEVEL3 -UDOUBLE -UCOMPLEX -DTT gemm.c -o sgemm_thread_tt.o
gcc-9 -c -DTHREADED_LEVEL3 -O2 -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DNO_AVX512 -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=6 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=_ssymm_thread_LU -DASMFNAME=_ssymm_thread_LU_ -DNAME=ssymm_thread_LU_ -DCNAME=ssymm_thread_LU -DCHAR_NAME=\"ssymm_thread_LU_\" -DCHAR_CNAME=\"ssymm_thread_LU\" -DNO_AFFINITY -I../.. -UDOUBLE  -UCOMPLEX -UDOUBLE -UCOMPLEX -ULOWER -URSIDE -DNN symm_k.c -o ssymm_thread_LU.o
gcc-9 -c -DTHREADED_LEVEL3 -O2 -DMAX_STACK_ALLOC=2048 -fopenmp -Wall -DF_INTERFACE_GFORT -fPIC -DDYNAMIC_ARCH -DNO_AVX512 -DSMP_SERVER -DUSE_OPENMP -DNO_WARMUP -DMAX_CPU_NUMBER=6 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.8\" -DASMNAME=_ssymm_thread_LL -DASMFNAME=_ssymm_thread_LL_ -DNAME=ssymm_thread_LL_ -DCNAME=ssymm_thread_LL -DCHAR_NAME=\"ssymm_thread_LL_\" -DCHAR_CNAME=\"ssymm_thread_LL\" -DNO_AFFINITY -I../.. -UDOUBLE  -UCOMPLEX -UDOUBLE -UCOMPLEX -DLOWER -URSIDE -DNN symm_k.c -o ssymm_thread_LL.o
level3_thread.c:354:2: error: invalid instruction mnemonic 'dmb'
        dmb  ish
        ^~level3_thread.c:354~
:2: error: invalid instruction mnemonic 'dmb'
        dmb  ish
        ^~~
level3_thread.c:401:2: error: invalid instruction mnemonic 'dmb'
        dmb  ishst
        ^~~
make[1]: *** [sgemm_thread_nt.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [sgemm_thread_nn.o] Error 1
level3_thread.c:354:2: error: invalid instruction mnemonic 'dmb'
        dmb  ish
        ^~~
level3_thread.c:401:2: error: invalid instruction mnemonic 'dmb'
        dmb  ishst
        ^~~
make[1]: *** [ssymm_thread_LU.o] Error 1
level3_thread.c:354:2: error: invalid instruction mnemonic 'dmb'
        dmb  ish
        ^~~
level3_thread.c:401:2: error: invalid instruction mnemonic 'dmb'
        dmb  ishst
        ^~~
make[1]: *** [ssymm_thread_LL.o] Error 1
level3_thread.c:354:2: error: invalid instruction mnemonic 'dmb'
        dmb  ish
        ^~~
level3_thread.c:401:2: error: invalid instruction mnemonic 'dmb'
        dmb  ishst
        ^~~
make[1]: *** [sgemm_thread_tn.o] Error 1
level3_thread.c:354:2: error: invalid instruction mnemonic 'dmb'
        dmb  ish
        ^~~
level3_thread.c:401:2: error: invalid instruction mnemonic 'dmb'
        dmb  ishst
        ^~~
make[1]: *** [sgemm_thread_tt.o] Error 1
make: *** [libs] Error 1

==> Formula
Path: /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/openblas.rb
==> Configuration
HOMEBREW_VERSION: 2.2.5-56-g23ab620
ORIGIN: https://github.com/Homebrew/brew
HEAD: 23ab620cd412a5b80d300055e18e5325a8d8edee
Last commit: 9 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core/
Core tap HEAD: e0100449735763a88accb342e0fcd6e3ecaf4da6
Core tap last commit: 4 minutes ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CACHE: /Users/brew/Library/Caches/Homebrew
HOMEBREW_LOGS: /Users/brew/Jenkins/workspace/core/logs
HOMEBREW_DEV_CMD_RUN: 1
HOMEBREW_FAIL_LOG_LINES: 150
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_GIT_EMAIL: homebrew-test-bot@lists.sfconservancy.org
HOMEBREW_GIT_NAME: BrewTestBot
HOMEBREW_HOME: /Users/brew/Jenkins/workspace/core/home
HOMEBREW_NO_ANALYTICS: 1
HOMEBREW_NO_AUTO_UPDATE: 1
HOMEBREW_NO_EMOJI: 1
CPU: hexa-core 64-bit penryn
Homebrew Ruby: 2.6.3 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.3/bin/ruby
Clang: 11.0 build 1100
Git: 2.21.1 => /Applications/Xcode.app/Contents/Developer/usr/bin/git
Curl: 7.54.0 => /usr/bin/curl
Java: 13.0.2, 1.8.0_242
macOS: 10.14.6-x86_64
CLT: 10.3.0.0.1.1562985497
Xcode: 11.3.1
Error: openblas 0.3.8 did not build
XQuartz: 2.7.11 => /opt/X11
==> ENV
HOMEBREW_CC: gcc-9
HOMEBREW_CXX: g++-9
MAKEFLAGS: -j6
CMAKE_PREFIX_PATH: /usr/local
CMAKE_INCLUDE_PATH: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/libxml2:/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers
CMAKE_LIBRARY_PATH: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Libraries
PKG_CONFIG_PATH: /usr/local/opt/isl/lib/pkgconfig:/usr/local/opt/mpfr/lib/pkgconfig
PKG_CONFIG_LIBDIR: /usr/lib/pkgconfig:/usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig/10.14
HOMEBREW_GIT: git
HOMEBREW_SDKROOT: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk
ACLOCAL_PATH: /usr/local/share/aclocal
PATH: /usr/local/Homebrew/Library/Homebrew/shims/mac/super:/usr/local/opt/gcc/bin:/usr/bin:/bin:/usr/sbin:/sbin

[martin-frbg]

No, definitely not related to #2416 - that is "only" about not linking to libopenmp, what you have here is a weird mixture of what appears to be an x86_64 build environment trying to understand an arm64 instruction. Were you trying to cross-compile for iOS by any chance ? Then this is the host compiler (or assembler) getting run instead of the arm64 cross-compiler, check your path settings.

[zbeekman]

This is building in Homebrew's sanitized environment, so it's built exactly the same way 0.3.7 was built. It is definitely NOT trying to cross compile, intentionally or accidentally: it is building x86_64 binaries for each of the 3 currently supported macOS releases. Homebrew strips the environment down and then explicitly sets all environment variables controlling the build.

Normally packages are built with Apple's native clang compiler (and gfortran). But for OpenBLAS 0.3.7 (and previous versions) were being built with the latest stable GCC instead of clang.

Here are the environment variables that were being explicitly set specifically for OpenBLAS:

DYNAMIC_ARCH=1
USE_OPENMP=1
NO_AVX512=1
CC=/path/to/gcc-9
FC=gfortran

The normal build environment for all packages is visible at the bottom of the error log above (except the formula had been marked as fails_with :clang so normally the C/C++ compilers would be clang and clang++):

HOMEBREW_CC: gcc-9
HOMEBREW_CXX: g++-9
MAKEFLAGS: -j6
CMAKE_PREFIX_PATH: /usr/local
CMAKE_INCLUDE_PATH: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/libxml2:/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers
CMAKE_LIBRARY_PATH: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Libraries
PKG_CONFIG_PATH: /usr/local/opt/isl/lib/pkgconfig:/usr/local/opt/mpfr/lib/pkgconfig
PKG_CONFIG_LIBDIR: /usr/lib/pkgconfig:/usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig/10.14
HOMEBREW_GIT: git
HOMEBREW_SDKROOT: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk
ACLOCAL_PATH: /usr/local/share/aclocal
PATH: /usr/local/Homebrew/Library/Homebrew/shims/mac/super:/usr/local/opt/gcc/bin:/usr/bin:/bin:/usr/sbin:/sbin

The commands to build and install OpenBLAS are:

system "make", "CC=gcc-9", "FC=gfortran", "libs", "netlib", "shared"
system "make", "PREFIX=/usr/local/Cellar/openblas/0.3.8", "install"

It's possible these need updating.

But simply switching to clang or removing USE_OPENMP=1 resolves the issue.

Building packages with GCC is admittedly less well-tested than with clang, but, nevertheless, gets used fairly regularly. I'll ping some of the other Homebrew maintainers and see if they have any thoughts on the matter. In the meantime I had linked the full Jenkins build logs in my original post, and if you're able to take a quick look at them to at least point me in the right direction it would be much appreciated.

[isuruf]

Can you print the contents of config.h after make fails?

[isuruf]

https://github.com/xianyi/OpenBLAS/pull/2279/files#diff-7cf3cca9d0a67dfeb94e7187f4adba11R263-R275 is the issue here.

[isuruf]

See #2424

[martin-frbg]

@isuruf thanks - the intention with #2279 was to facilitate cross-compiling for iOS, removing the need to hack various files as described in the wiki. To that end I cobbled together information from said wiki entry, success reports from earlier issues and bits and pieces from stackoverflow. The result appeared to work in the Travis enviroment but I admit I do not have local OSX nor iOS hardware for testing.

[isuruf]

I don't have iOS hardware for testing. I checked that I can cross-compile using #2424. After that PR is merged, I'll update the wiki.

[zbeekman]

The result appeared to work in the Travis enviroment but I admit I do not have local OSX nor iOS hardware for testing.

FWIW, I find Travis-CI somewhat unreliable for macOS. Github Actions work much more reliably in my experience.

Would you accept a PR to do nightly macOS builds of the development branch using the homebrew build formula? (With Github Actions.) That way you can see when something breaks either on your end or, possibly, on the Homebrew or macOS end.

@isuruf here is config.h:

#define OS_DARWIN	1
#define ARCH_ARM64	1
#define C_GCC	1
#define __64BIT__	1
#define FUNDERSCORE	_
#define PTHREAD_CREATE_FUNC	pthread_create
#define BUNDERSCORE	_
#define NEEDBUNDERSCORE	1
#define HASWELL
#define L1_CODE_SIZE 32768
#define L1_CODE_ASSOCIATIVE 8
#define L1_CODE_LINESIZE 64
#define L1_DATA_SIZE 32768
#define L1_DATA_ASSOCIATIVE 8
#define L1_DATA_LINESIZE 64
#define L2_SIZE 262144
#define L2_ASSOCIATIVE 8
#define L2_LINESIZE 64
#define ITB_SIZE 2097152
#define ITB_ASSOCIATIVE 0
#define ITB_ENTRIES 8
#define DTB_SIZE 4096
#define DTB_ASSOCIATIVE 4
#define DTB_DEFAULT_ENTRIES 64
#define HAVE_CMOV
#define HAVE_MMX
#define HAVE_SSE
#define HAVE_SSE2
#define HAVE_SSE3
#define HAVE_SSSE3
#define HAVE_SSE4_1
#define HAVE_SSE4_2
#define HAVE_AVX
#define HAVE_AVX2
#define HAVE_FMA3
#define HAVE_CFLUSH
#define NUM_SHAREDCACHE 2
#define NUM_CORES 8
#define CORE_HASWELL
#define CHAR_CORENAME "HASWELL"
#define GEMM_MULTITHREAD_THRESHOLD	4

[zbeekman]

yeah, if OS_DARWIN, ARCH_ARM64 is definitely not appropriate.

[isuruf]

@zbeekman, can you check #2424?

It's a hard to catch error. If CC=gcc or CC=clang, you wouldn't get this error. You'll get this if CC=gcc-9.