openblas issue with multithreading and make.

[MiguelSen]

hi, all,
i'm using a raspberry pi 3, with raspbian 32 bit (armv7).
I'm working on a code that basically do svd using lapack function dgesvdx_ (and do other minor functions like fft, hankel matrix), this code works on temporal windows, every window has an input matrix and it simulates a real-time application taking the input from a txt.file. I can compile the code using gcc (version 6.3.0), but the code stops working without any error displayed in the terminal after a few window's processes and the cpu continues to use about the 400% in top application. Using dbg debugger sometimes i receive a segmentation fault or a SIGILL in some random part of the code but the most of the time nothing.This issue doesn't appear if i choose export OMP_NUM_THREADS=2, but it appears with 3 and 4 threads. First i tried the same code with fedora 64 bit (ARMV8) and i don't have this issue (but the performances are worse than 2 threads of raspbian).
If i try to make the openblas from source, during the execution of make (make FC=gfortran TARGET=ARMV7) i receive a SEGMENTATION FAULT every time at a different point.
Are these issues correlated? Have i to give up at the idea that i can use only 2 threads (and 200% of CPU) in my code using raspbian?
Thank you for your answers.

[martin-frbg]

Which version of OpenBLAS are you using (or trying to build) ? There has been some work on fixing thread races in recent months. make segfaulting however makes it sound more like a hardware problem (is it trying to run jobs in parallel perhaps ?). With ARMV8 the speed issue may have been from using a default TARGET=ARMV8 build that is not making use of cpu-specific optimizations, you may be able to specify CORTEXA57 as the cpu target.

[MiguelSen]

i'm trying to build the OpenBLAS-0.2.20. Yes, it's trying to do it in parallel, using top i see 4 process cc1, so the problem is parallelization, right? is there a solution for this? i tried to build the library by source because my code doesn't work with more than 2 threads and i thought that was an issue of the pre-compiled library.

[martin-frbg]

Try setting MAKE_NB_JOBS=2 with make (see Makefile.rule for all available options), maybe your system is overheating when all cores are busy ?

[MiguelSen]

thank you, now i try, but the main problem is that i want that raspberry uses all the 4 cores, so if i can do it with this option and then i can't using all the 4 cores is not very useful for me.

i knew that raspberry pi 3 is using CORTEXA53, or not?

[martin-frbg]

See if make runs to completion when it uses only two cores. Any segmentation faults you saw there will not have much if anything to do with OpenBLAS (unless they happened in the tests at the very end of the build), but with how that particular raspi handles high cpu loads.

[MiguelSen]

Yes, make runs now, thank you!

OS ... Linux
Architecture ... arm
BINARY ... 32bit
C compiler ... GCC (command line : gcc)
Fortran compiler ... GFORTRAN (command line : gfortran)
Library Name ... libopenblas_armv7p-r0.2.20.a (Multi threaded; Max num-threads is 4)

i'd like to know if it's a issue of my raspberry or a general bug now.

[martin-frbg]

I do not think it is a general bug - as I wrote, make running four compilers in parallel has little or nothing to do with the program it compiles.
The program locking up with four threads may be a bug in OpenBLAS, we would need to know the function that hangs (and you could try compiling with USE_SIMPLE_THREADED_LEVEL3=1 if it happens in a blas3 function). Random faults with SIGILL on the other hand make it look more like hardware again, perhaps insufficient cooling or a problematic power supply ?
(CORTEXA53 is what the pi3 seems to be using according to various sources on the internet. OpenBLAS has optimized ARMV8 code for the A57 and I believe they both use the same instruction set. I do not have a raspberry pi or any other ARM hardware however.)

[MiguelSen]

i'm using the official power supply of the raspberry, i thought about cooling yesterday so i removed the case and i used an hairdryer (yeah, that's the best i had) to cool the raspberry but it didn't work, using /opt/vc/bin/vcgencmd measure_temp to check the temperature.
I have to use USE_SIMPLE_THREADED_LEVEL3=1 in "make" command, right?

[martin-frbg]

Yes, that is a setting for the make command.
Do you have a heat sink on the cpu ? Seems these are optional and not included in the standard kit. (If you checked the temperature you will already have an idea how much it changes between idle and maximum load)

[brada4]

What command generates segmentation fault? Is it a compiler or any of the tests?

[MiguelSen]

no, i don't have heat sink, with the case the cpu's temperature goes from 40°-45°to 70°-75° celsius, without case and using the hairdryer remains to a value of about 55°-60° celsius.

@brada4 well, every time changes
i have to uninstall the precompiled openblas before to compile the code or is it unnecessary?

[brada4]

You can Google for pi3 overheating solutions. It is not openblas issue. You can heat any processor with one nice yes >/dev/null & per core with same success.
Probably fedora got thermal throttling right, but this raspbian let's it heat to visible failure.

Presence of other BLAS libraries does not influence build process:
https://github.com/xianyi/OpenBLAS/wiki/faq#debianlts

[MiguelSen]

Probably you're right, it's not a openblas issue but i don't think that it's a temperature's issue. Without case the temperature with 2 cores and 3-4 cores are pretty similar and lower than the temperature with the case and raspberry with 2 core (and it works in that case), but i'll buy the heatsink anyway.
Thank you for the information :)

[brada4]

You can close this issue as unrelated to OpenBLAS.

[MiguelSen]

well, is there a section for raspbian/raspberry here?