OpenBLAS tests fail `potrf:smoketest_trivial` with LLVM 20.1.8 on aarch64

[Thyre]

Heya 👋

I'm currently looking into building software packages for a potential LLVM toolchain in EasyBuild.

While trying to build OpenBLAS on a Grace Hopper system, I noticed that OpenBLAS fails to run a single test in this configuration:

$ make -j 16 shared  BINARY='64'  CC='clang'  FC='flang'  MAKE_NB_JOBS='-1'  USE_OPENMP='1'  USE_THREAD='1'  CFLAGS='-O2 -Wl,--undefined-version -Wno-unused-command-line-argument -Wno-error=int-conversion'
$ OMP_NUM_THREADS=16 OPENBLAS_NUM_THREADS=16   make tests  BINARY='64'  CC='clang'  FC='flang'  MAKE_NB_JOBS='-1'  USE_OPENMP='1'  USE_THREAD='1'

With any number of threads above 1, we see this failure (I've added the (%d) at the end to get the actual info value):

TEST 124/126 potrf:smoketest_trivial [FAIL]
  ERR: test_potrs.c:463  info != 0 (2)

With a single thread, this part passes, but we fail later on:

TEST 124/126 potrf:smoketest_trivial [FAIL]
  ERR: test_potrs.c:537  U s(1,1) difference: 0.0449998

Given that flang is not entirely mature, I wouldn't be surprised if there's still a compiler bug somewhere.
However, I failed to find a trace of this being reported in LLVM or here.

The same configuration worked just fine on Intel Alderlake and AMD Zen4, so I'm guessing that this is an aarch64 specific issue.

---

System info: Rocky Linux 9.6, NVIDIA Grace Hopper GH200, 480 GB RAM

Used software:

$ module list
Currently Loaded Modules:
  1) GCCcore/14.3.0   3) binutils/2.44   5) XZ/5.8.1         7) ncurses/6.5   9) Z3/4.15.1  11) lz4/1.10.0  13) LLVM/20.1.8            15) make/4.4.1   17) libreadline/8.2   19) Tcl/9.0.1      21) OpenSSL/3
  2) zlib/1.3.1       4) libffi/3.5.1    6) libxml2/2.14.3   8) GMP/6.3.0    10) gzip/1.14  12) zstd/1.5.7  14) llvm-compilers/20.1.8  16) bzip2/1.0.8  18) libtommath/1.3.0  20) SQLite/3.50.1  22) Python/3.13.5

---

If any more information is needed, I'm happy to help out. Unforunately, I don't have much of an idea where this issue could be coming from...

[martin-frbg]

potrf is reimplemented in OpenBLAS, so flang is unlikely to be the culprit, the problem is more likely to be related to the NeoverseV2 GEMM kernel - or the test code itself (ISTR there being optimization issues with test_potrs.c on other platforms in the past). Does it pass the level3 (C)BLAS tests in test and ctest ?

[martin-frbg]

not immediately reproducible with current develop branch (but my HPC system for the test was a Pixel8pro)
If anything, #5516 (which reverts an optimization that could create race conditions in parallel GEMM) might have something to do with it

[Thyre]

All of the other tests in make tests passed, I'm currently testing make lapack-test.
Is there some way to only run the Level 3 (C)BLAS tests? For CMake, I would probably need to spend a bit more time to build things, but I can try that out as well.

I can also try the PR out to check if this fixes things.

[Thyre]

Here's the full output of make tests with the current HEAD of develop:

• out.txt

Unfortunately, this still ends in the same test failure. I'll try a completely manual build outside of EasyBuild as well, just to ensure that nothing from the environment causes issues...
make lapack-test gave me some trouble as well, but this could well be my environment.

[martin-frbg]

Hmm, there's a ton of errors in the level2 blas tests (the weird "less than half accurate" wording is inherited from Reference-LAPACK) that I don't see locally with 20.1.8. Not sure what is going on there

[Thyre]

Testing from source:

CMake:

$ cmake -S .. -B . -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_Fortran_COMPILER=flang
$ make -j12
$ ctest --output-on-failure
[...]
112/116 Test #112: xccblat2 ....................................................................................   Passed    0.70 sec
        Start 113: xccblat3
113/116 Test #113: xccblat3 ....................................................................................   Passed    0.15 sec
        Start 114: xzcblat1
114/116 Test #114: xzcblat1 ....................................................................................   Passed    0.00 sec
        Start 115: xzcblat2
115/116 Test #115: xzcblat2 ....................................................................................   Passed    0.70 sec
        Start 116: xzcblat3
116/116 Test #116: xzcblat3 ....................................................................................   Passed    1.18 sec

100% tests passed, 0 tests failed out of 116

Total Test time (real) = 123.32 sec

• openblas-cmake.log

make:

$ make -j 16 shared  BINARY='64'  CC='clang'  FC='flang'  MAKE_NB_JOBS='-1'  USE_OPENMP='1'  USE_THREAD='1'
$ make tests  BINARY='64'  CC='clang'  FC='flang'  MAKE_NB_JOBS='-1'  USE_OPENMP='1'  USE_THREAD='1'
[...]
TEST 117/126 zgemv:0_nan_inf [OK]
TEST 118/126 zgemv:0_nan_inf_incy_2 [OK]
TEST 119/126 zgemv:0_2_nan_1_inf_1 [OK]
TEST 120/126 zgemv:0_2_nan_1_inf_1_incy_2 [OK]
TEST 121/126 zgemv:2_0_nan_1_inf_1 [OK]
TEST 122/126 zgemv:2_0_nan_1_inf_1_incy_2 [OK]
TEST 123/126 potrf:bug_695 [OK]
TEST 124/126 potrf:smoketest_trivial [FAIL]
  ERR: test_potrs.c:463  info != 0
TEST 125/126 kernel_regress:skx_avx [OK]
TEST 126/126 fork:safety_after_fork_in_parent [OK]
RESULTS: 126 tests (125 ok, 1 failed, 0 skipped) ran in 7 ms
make[1]: *** [Makefile:86: run_test] Error 1
make[1]: Leaving directory '/p/project1/cswmanage/reuter1/OpenBLAS/utest'
make: *** [Makefile:177: tests] Error 2

Just by quickly glancing over the CMake results, they look better I guess?

Well, looking at it more closely, one can see

Fortran STOP
IEEE arithmetic exceptions signaled: INEXACT

in quite a few tests. That's probably not ideal.

[martin-frbg]

Only difference should be optimization level - the gmake default is -O2 while CMake I guess uses no optimization flags when no build type is specified (and -O3 if CMAKE_BUILD_TYPE="Release").
Not sure about the INEXACT signalling, could be from LAPACK's internal checks that determine machine constants like smallest exactly representable number etc

[martin-frbg]

Reproduced the problem locally, but have no time to debug it until later this evening

[Thyre]

No problem. Thanks for looking into this at all 😄

[martin-frbg]

Apparently it all hinges on the misoptimized/misbehaving SDOT, but I have not found a better solution than adding #pragma clang optimize off at the top of kernel/arm64/dot_kernel_sve.c

[Thyre]

Apparently it all hinges on the misoptimized/misbehaving SDOT, but I have not found a better solution than adding #pragma clang optimize off at the top of kernel/arm64/dot_kernel_sve.c

If LLVM 21.1.x or even trunk fix the misoptimization/misbehaving, I think that would be acceptable for now, even if it's not ideal. This can be easily patched for people being affected...
I can try to check this.
While performance will likely be worse, that's still better than incorrect results 😄

[martin-frbg]

I haven't tried trunk but 21.4 doesn't fix it. And there might be a (somewhat) legitimate reason why the optimizer apparently assumes it can read beyond the array ( though I can't see it, and IIRC the code was contributed by someone at Arm Ltd)

[mayeut]

If I'm not mistaken, the n argument is not mapped correctly in the assembly, x0 is used instead of %[N_] which is consistent with an out-of-bound read. The clobber list is also missing some registers.
I'll open a PR if the tests pass CI on my end.