"FATAL ERROR" when testing "./xdcblat3 < din3"

[njutcz]

"FATAL ERROR" when testing "./xdcblat3 < din3"

Environment: ARMV8 CentOS 7.6, OpenBLAS-0.3.7
compile: make TARGET=ARMV8 -j

cd ctest
mycount=0; while (( $mycount < 100 )); do ./xdcblat3 < din3;sleep 1; ((mycount=$mycount+1)); done;

The error will occur about every 7 or 8 times! And when "export OMP_NUM_THREADS=1", the error doesn't occur! I'm not sure if this is a precision issue or a calculation error!

[njutcz]

Only cblas_dsymm has the error. Cblas_dgemm never occurs the same error!

[martin-frbg]

What kind of hardware is this, how many cores and which compiler version ? I cannot reproduce this problem so far.

[brada4]

There was something similar, solution that worked was to rpmbuild Fedora's SRPM/SPEC that likely has some compiler quirks included.
You can start here (or seek SRPM in rawhide repos)
https://src.fedoraproject.org/rpms/openblas/blob/HEAD/f/openblas.spec

[njutcz]

What kind of hardware is this, how many cores and which compiler version ? I cannot reproduce this problem so far.

kunpeng920 96cores gcc 4.8.5

[njutcz]

What kind of hardware is this, how many cores and which compiler version ? I cannot reproduce this problem so far.

When I build with “USE_SIMPLE_THREADED_LEVEL3=1”， the ERROR doesn't occur. But i don't know why.

[martin-frbg]

Thanks for the additional information. Building with USE_SIMPLE_THREADED_LEVEL3=1 disables some additional splitting of the workload that could still contain race conditions - though I do not get any error messages from running valgrind/helgrind on the xdcblat3 test.
Historically OpenBLAS - like GotoBLAS before it - has not been thread safe until well into the 0.2.x series - with the slow cpu speeds and low core counts available ten years ago the probability of an actual failure seems to have been very low. Even now that most bugs have hopefully been fixed, there is unfortunately still very limited experience with core counts beyond about 32.
I guess there is still some chance that it could be a compiler bug "working around" a lock or barrier though.

[martin-frbg]

I have now managed to obtain related helgrind warnings for this on a different system. Apparently the current set of memory barriers in level3_thread.c is insufficient to sequence accesses to the segments of the common gemm buffer.
The attached working version of level3_thread.c addresses these by riddling the code with mutex locks, hopefully (a) this actually solves the problem you encountered and (b) it can be stripped down to a less intrusive solution
level3_thread.txt

[martin-frbg]

@njutcz did you test if my proposed version of level3_thread.c fixes the problem for you ? With my limited range of hardware, I never saw the actual miscalculation.

[njutcz]

Yes, it can solve my problem but the performance drops a lot when too many threads.
Thanks a lot anyway!

[martin-frbg]

Well, it was just a first stab at the problem, with lots of overhead from individual lock/unlock calls. (Compiling with OpenMP would probably provide similar protection, not sure if you tried that as well ?)

[brada4]

Yes, it can solve my problem but the performance drops a lot when too many threads.

You should not try more threads than actual cores. What happens is cache spill that you actually access main memory and everything gets disproportionally 10-20x slower.

[njutcz]

Yes, it can solve my problem but the performance drops a lot when too many threads.

You should not try more threads than actual cores. What happens is cache spill that you actually access main memory and everything gets disproportionally 10-20x slower.

Well, I tested on Kunpeng920. It has 96 or 128 cores.

[brada4]

OpenBLAS might get per-cpu task sizing wrong too especially with that many CPUs....

[njutcz]

Well, it was just a first stab at the problem, with lots of overhead from individual lock/unlock calls. (Compiling with OpenMP would probably provide similar protection, not sure if you tried that as well ?)

I tried to build with "make USE_OPENMP=1 TARGET=ARMV8", but this can't solve the problem.