0.3.7 fails to compile on arm32, while 0.3.6 compiles just fine

[jfikar]

Hello,

I was compiling OpenBlas-0.3.7 on a Raspberry Pi4 using recent arm32 Raspbian. I did just make -j4 and got en error:

...
gemm_thread_tt -DCHAR_NAME=\"sgemm_thread_tt_\" -DCHAR_CNAME=\"sgemm_thread_tt\" -DNO_AFFINITY -I../.. -UDOUBLE  -UCOMPLEX  -c -DTHREADED_LEVEL3 -UDOUBLE -UCOMPLEX -DTT gemm.c -o sgemm_thread_tt.o
/tmp/ccVcvHlA.s: Assembler messages:
/tmp/ccvMTz5B.s: /tmp/ccVcvHlA.s:494: Error: selected processor does not support `dmb ish' in ARM mode
Assembler messages:
/tmp/ccvMTz5B.s:498: Error: selected processor does not support `dmb ish' in ARM mode
/tmp/ccvMTz5B.s:588: Error: selected processor does not support `dmb ishst' in ARM mode
/tmp/ccvMTz5B.s:650: Error: selected processor does not support `dmb ishst' in ARM mode
/tmp/ccvMTz5B.s:745: Error: selected processor does not support `dmb ishst' in ARM mode
/tmp/ccvMTz5B.s:824: Error: selected processor does not support `dmb ish' in ARM mode
/tmp/ccvMTz5B.s:843: Error: selected processor does not support `dmb ish' in ARM mode
/tmp/ccvMTz5B.s:885: Error: selected processor does not support `dmb ish' in ARM mode
/tmp/ccVcvHlA.s:582: Error: selected processor does not support `dmb ishst' in ARM mode
/tmp/ccVcvHlA.s:644: Error: selected processor does not support `dmb ishst' in ARM mode
/tmp/ccVcvHlA.s:739: Error: selected processor does not support `dmb ishst' in ARM mode
/tmp/ccVcvHlA.s:818: Error: selected processor does not support `dmb ish' in ARM mode
/tmp/ccVcvHlA.s:837: Error: selected processor does not support `dmb ish' in ARM mode
/tmp/ccVcvHlA.s:879: Error: selected processor does not support `dmb ish' in ARM mode
make[1]: *** [Makefile:483: sgemm_thread_nn.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [Makefile:486: sgemm_thread_nt.o] Error 1
/tmp/cc2taceD.s: Assembler messages:
/tmp/cc2taceD.s:499: Error: selected processor does not support `dmb ish' in ARM mode
/tmp/cc2taceD.s:589: Error: selected processor does not support `dmb ishst' in ARM mode
/tmp/cc2taceD.s:651: Error: selected processor does not support `dmb ishst' in ARM mode
/tmp/cc2taceD.s:746: Error: selected processor does not support `dmb ishst' in ARM mode
/tmp/cc2taceD.s:826: Error: selected processor does not support `dmb ish' in ARM mode
/tmp/cc2taceD.s:845: Error: selected processor does not support `dmb ish' in ARM mode
/tmp/cc2taceD.s:887: Error: selected processor does not support `dmb ish' in ARM mode
make[1]: *** [Makefile:489: sgemm_thread_tn.o] Error 1
/tmp/ccdtGaKI.s: Assembler messages:
/tmp/ccdtGaKI.s:499: Error: selected processor does not support `dmb ish' in ARM mode
/tmp/ccdtGaKI.s:587: Error: selected processor does not support `dmb ishst' in ARM mode
/tmp/ccdtGaKI.s:649: Error: selected processor does not support `dmb ishst' in ARM mode
/tmp/ccdtGaKI.s:744: Error: selected processor does not support `dmb ishst' in ARM mode
/tmp/ccdtGaKI.s:824: Error: selected processor does not support `dmb ish' in ARM mode
/tmp/ccdtGaKI.s:843: Error: selected processor does not support `dmb ish' in ARM mode
/tmp/ccdtGaKI.s:885: Error: selected processor does not support `dmb ish' in ARM mode
make[1]: *** [Makefile:492: sgemm_thread_tt.o] Error 1
make[1]: Leaving directory '/home/pi/OpenBLAS-0.3.7/driver/level3'
make: *** [Makefile:150: libs] Error 1

On the other hand, the version 0.3.6 compiles just fine and both versions compile fine on aarch64.

[martin-frbg]

Hmm.This looks like a return of issue #1961 (i.e., not automatically switchingTARGET to ARMV7 when BINARY=32 was given), but I am not aware that any recent change accidentally reverted #1987. Did you use the same make arguments and compiler for both 0.3.6 and 0.3.7 ?

[jfikar]

Yes, it is the same:

gcc --version
gcc (Raspbian 8.3.0-6+rpi1) 8.3.0

I'm doing just this:

cd OpenBLAS-0.3.x
make clean
make -j4

However, I can see the automatically used compiler flags are different in both cases. 0.3.6:

cc -O2 -DMAX_STACK_ALLOC=2048 -marm -Wall -DF_INTERFACE_GFORT -fPIC
-DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=4 -DMAX_PARALLEL_NUMBER=1
-DVERSION=\"0.3.6\" -mfpu=neon-fp-armv8 -march=armv8-a+crc+simd -DASMNAME=samin
-DASMFNAME=samin_ -DNAME=samin_ -DCNAME=samin -DCHAR_NAME=\"samin_\"
-DCHAR_CNAME=\"samin\" -DNO_AFFINITY -I.. -I. -UDOUBLE  -UCOMPLEX -c -DUSE_ABS
-DUSE_MIN

0.3.7:

cc -O2 -DMAX_STACK_ALLOC=2048 -Wall -DF_INTERFACE_GFORT -fPIC -DSMP_SERVER
-DNO_WARMUP -DMAX_CPU_NUMBER=4 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.7\"
-DASMNAME=cblas_sdsdot -DASMFNAME=cblas_sdsdot_ -DNAME=cblas_sdsdot_
-DCNAME=cblas_sdsdot -DCHAR_NAME=\"cblas_sdsdot_\" -DCHAR_CNAME=\"cblas_sdsdot\"
-DNO_AFFINITY -I.. -I. -UDOUBLE  -UCOMPLEX -DCBLAS```

[martin-frbg]

Are these from the 32bit builds? The 0.3.7 one seems to miss a -march hint for the compiler, so it could be that my #1987 was incomplete

[jfikar]

Yes, these are both arm32 builds. The 0.3.7 is also missing -marm. Should it be auto-detected?

[martin-frbg]

It should be, but it looks as if the auto-detection still sees it as an aarch64 system when it is running a 32bit os. (Not sure why this is, as that informations is derived from either uname or gcc -E output.)
Right now I am no longer sure if the added code from #1987 gets executed at all in your case, but if it does it should be equivalent to compiling as make TARGET=ARMV7.

[jfikar]

Now I see what is happening. I'm using aarch64 kernel and both arm32 and aarch64 userland. Then even in arm32 userland the uname -a still returns aarch64, which probably confuses the auto-detection:

Linux raspberrypi4 4.19.66-v8-fc5826fb999e-p4-bis+ #2 SMP PREEMPT Fri Aug 16 13:58:31 GMT 2019 aarch64 GNU/Linux

Strange that 0.3.6 gets it right.

So using arm32 kernel: 0.3.7 compiles just fine with simple make -j4

However using aarch64 kernel: 0.3.7 fails the same way as in my initial post even when I specify make -j4 TARGET=ARMV7

[martin-frbg]

So I got myself a Pi4B to debug this - cpuinfo in raspbian appears to be a complete mess (raspberrypi/linux#3022), and the default raspbian install appears to be a purely 32bit armeabihf environment despite the ARMV8 cpu. How does one get an aarch64 kernel on this thing, and optionally the aarch64 userland to match ? (Though for now I am primarily interested in reproducing your mixed arm32/arm64 setup)

[jfikar]

Great :) First you need Sakaki's pre-compiled aarch64 kernel. Better wait a bit, she promised to fix problem with WiFi later today. Untar it to -C / and I think the only change needed in /boot/config.txt is kernel=kernel8-p4.img and you boot aarch64 kernel into Raspbian arm32 userland.

But before you also need to run sudo rpi-update. It will update the arm32 kernel (which you don't need), but also firmware files, which are compatible with the new kernel.

Then for aarch64 userland the easiest is schroot with vanilla Debian Buster aarch64.

sudo apt-get install -y debootstrap schroot
sudo debootstrap --arch=arm64 buster /srv/chroot/pi64

Then you fill this into the file /etc/schroot/chroot.d/pi64:

[pi64]
description=VC4 arm64 testing
type=directory
directory=/srv/chroot/pi64
users=pi
root-groups=root
profile=desktop
personality=linux
preserve-environment=true

you may need to install e.g. sudo as well, otherwise you'd not be able to install anything once inside of the chroot:

sudo schroot -c pi64 -- apt-get install -y sudo

and you can chroot yourself into aarch64 userland:
schroot -c pi64

[jfikar]

The new Sakaki's kernel 4.19.67.20190827 is out.

[martin-frbg]

Bisected to #2110 (and we even had misgivings about using uname -m to define ARCH early in the build ☹️ ) I'll do the PR later

[jfikar]

Oh, that was fast :) Could you, please, look into #2230, once you already have RPi4? You would need a small heatsink and a small fan though, to avoid thermal throttling. Or alternatively you can test small problem sizes.

[martin-frbg]

No promises as I still have to try and schroot myself in the foot... also my Pi4 came with a whole collection of tiny heatsinks but no fan.