Tester segfault due to libgfortran mixup

[ckoe-bccms]

Hello everybody,

sorry for the initially empty bug report. Anyway.

I am observing tester failures while using a self-built gcc/gfortran 9.2 on top of Centos 7. These failures are the result of the testers being linked against the systems /lib64/libgfortran.so.3 instead of the compilers required libgfortran.so.5.

The normal openblas (0.3.7) built process compiles for example sblat1 with

# gfortran -g -Wall -frecursive -fno-optimize-sibling-calls -m64 -fopenmp -mavx2 -g  -o sblat1 sblat1.o ../libopenblas_omp_haswellp-r0.3.7.a -lm -lpthread -lgfortran -lm -lpthread -lgfortran -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../..  -lgomp -lpthread -lc
# ldd ./sblat1
        linux-vdso.so.1 =>  (0x00007ffd57d16000)
        libgfortran.so.3 => /lib64/libgfortran.so.3 (0x00007f6bb3486000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f6bb3184000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6bb2f68000)
        libgomp.so.1 => /cluster/comp/gcc/9.2.0/lib64/libgomp.so.1 (0x00007f6bb2d33000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f6bb2966000)
        libgcc_s.so.1 => /cluster/comp/gcc/9.2.0/lib64/libgcc_s.so.1 (0x00007f6bb274e000)
        libquadmath.so.0 => /cluster/comp/gcc/9.2.0/lib64/libquadmath.so.0 (0x00007f6bb2508000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f6bb37a8000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f6bb2304000)
# ./sblat1

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x7F0D3F69D697
#1  0x7F0D3F69DCDE
#2  0x7F0D3EB9A27F
#3  0x7F0D3F75C4C0
#4  0x7F0D3F769E27
#5  0x403DEA in MAIN__ at sblat1.f:63
Segmentation fault

EDIT: Copied initially wrong ldd output, now hopefully the correct one !

Now, if I prune library path arguments passed to the compiler and compile with

gfortran -g -Wall -frecursive -fno-optimize-sibling-calls -m64 -fopenmp -mavx2 -g  -o sblat1 sblat1.o ../libopenblas_omp_haswellp-r0.3.7.a -lm -lpthread -lgfortran -lm -lpthread -lgfortran   -lgomp -lpthread -lc
# ldd ./sblat1
        linux-vdso.so.1 =>  (0x00007ffed6dc2000)
        libgfortran.so.5 => /cluster/comp/gcc/9.2.0/lib64/libgfortran.so.5 (0x00007f1a4351a000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f1a43218000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1a42ffc000)
        libgomp.so.1 => /cluster/comp/gcc/9.2.0/lib64/libgomp.so.1 (0x00007f1a42dc7000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f1a429fa000)
        libgcc_s.so.1 => /cluster/comp/gcc/9.2.0/lib64/libgcc_s.so.1 (0x00007f1a427e2000)
        libquadmath.so.0 => /cluster/comp/gcc/9.2.0/lib64/libquadmath.so.0 (0x00007f1a4259c000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1a439a8000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f1a42398000)
# ./sblat1|head
 Real BLAS Test Program Results


 Test of subprogram number  1             SDOT 
                                    ----- PASS -----

this now links the correct libgfortran.5.so from the compiler.

This leads to the question if the original compiler call should contain the /usr/lib/gcc/x86_64-redhat-linux/4.8.5 etc. library paths which appear to be problematic ? Also, is there a way I could influence this from Makefile.rules, i.e. without changing lower level files ?

Best Regards

Christof

[martin-frbg]

If you have several compiler versions installed, you should set up your PATH and LD_LIBRARY_PATH to point to the desired one, and/or use CC and FC to override what gets detected by the build system (something like CC=/cluster/comp/gcc/9.2.0/bin/gcc and likewise for FC/gfortran). Your first example looks like a combination of a 9.2.0 gcc and a 4.8.5 gfortran, which is guaranteed to end in tears.

[ckoe-bccms]

Hello,

sure ! We have a modules environment set up on the cluster, see below, but may be we are actually missing something there. To me currently the problem instead appears to be that the openblas build system passes the for the fortran testers superfluous (?)

CEXTRALIB=-L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../..  -lgomp -lpthread -lc

to the gfortran compiler.

Not changing anything else (e.g. LD_LIBRARY_PATH as set in the module, see below) but just removing the -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../.. from the gfortran call links the right libgfortran.so.5 instead of the libgfortran.so.3 from the system. I assume that libgfortran.so.3 is only included because it can be discovered after -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64 is set:

# ls /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libgfor*
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libgfortran.so.3
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libgfortran.so.3.0.0

I should have added in the initial post that this is a cluster using a proper lmod environment system. However, it might be that additional settings are required. The current module is this:

proc ModulesHelp { } {
   puts stderr "\tgcc 9.2.0; compiled by ckoe"
}

family "compiler" 

module-whatis   "gcc 9.2.0; compiled by ckoe"

set             root              /cluster/comp/gcc/9.2.0/
prepend-path     PATH              $root/bin
prepend-path     MANPATH           $root/share/man
prepend-path     LD_LIBRARY_PATH   $root/lib
prepend-path     LD_LIBRARY_PATH   $root/lib64
prepend-path     INCLUDE           $root/include

prepend-path MODULEPATH /cluster/modulefiles/comp/gcc-9

and exactly this module is loaded.

Best regards

Christof

[ckoe-bccms]

In addition in the environment with loaded module I can demonstrate that the setup is not obviously broken:

# echo $PATH
/cluster/comp/gcc/9.2.0/bin:/opt/xcat/bin:/opt/xcat/sbin:/opt/xcat/share/xcat/tools:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/maui/sbin:/usr/local/maui/bin:/root/bin
# echo $LD_LIBRARY_PATH 
/cluster/comp/gcc/9.2.0/lib64:/cluster/comp/gcc/9.2.0/lib
# echo $INCLUDE 
/cluster/comp/gcc/9.2.0/include
# gfortran -v
Using built-in specs.
COLLECT_GCC=gfortran
COLLECT_LTO_WRAPPER=/cluster/comp/gcc/9.2.0/libexec/gcc/x86_64-pc-linux-gnu/9.2.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: ./configure --disable-multilib --prefix=/cluster/comp/gcc/9.2.0
Thread model: posix
gcc version 9.2.0 (GCC) 
# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/cluster/comp/gcc/9.2.0/libexec/gcc/x86_64-pc-linux-gnu/9.2.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: ./configure --disable-multilib --prefix=/cluster/comp/gcc/9.2.0
Thread model: posix
gcc version 9.2.0 (GCC)

A trivial test of the compiler works

# cat peule.f90 
program peule
  write(*,*) 'peule'
end 
# gfortran peule.f90
# ldd ./a.out 
        linux-vdso.so.1 =>  (0x00007fffc727c000)
        libgfortran.so.5 => /cluster/comp/gcc/9.2.0/lib64/libgfortran.so.5 (0x00007f0adb4a1000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f0adb19f000)
        libgcc_s.so.1 => /cluster/comp/gcc/9.2.0/lib64/libgcc_s.so.1 (0x00007f0adaf87000)
        libquadmath.so.0 => /cluster/comp/gcc/9.2.0/lib64/libquadmath.so.0 (0x00007f0adad41000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f0ada974000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f0adb92f000)
# ./a.out 
 peule

[martin-frbg]

CEXTRALIB is deduced from the output of a test compile (see the c_check perl script around line 270, where it executes "cc -v ctest2.o -o ctest2) so something in your system setup seems to still point to the "old" libraries (and has them take precedence over the newer ones).

[ckoe-bccms]

Aha ! I assume cc is used for portability ? I will have to figure out how to best deal with this.

The Makefile.rules comment appears to suggest gcc is used throughout to me,

# C compiler including binary type(32bit / 64bit). Default is gcc.
# Don't use Intel Compiler or PGI, it won't generate right codes as I expect.
# CC = gcc

From it I also assume I cannot force the usage of gcc by uncommenting this ? How does it deal with, say, a intel or sun compiler ?

Thank you very much for your help !

# cc -v
Using built-in specs.
COLLECT_GCC=cc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/isl-install --with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/cloog-install --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC)

[martin-frbg]

"cc" seems to be the gmake default nowadays (https://www.gnu.org/software/make/manual/html_node/Implicit-Variables.html), setting CC=gcc or whatever overrides it (and /usr/bin/cc is usually a link to whatever system compiler gets installed by default).

[ckoe-bccms]

"cc" seems to be the gmake default nowadays, setting CC=gcc or whatever overrides it.

Thanks, I will try that ! I am not sure that the obvious dirty alternative of symlinking for each compiler cc to the right gcc in its install directory is a good idea, as you write something else might assume that cc must always point to the base systems C compiler. And I have a vague recollection that was one (of many ?) conventions a long time ago, at least on solaris or OSF1.

We had the same library mixup with gcc 6 and gcc 7 compilers. However, their libgfortran.so's are apparently compatible with the systems libgfortran.3.so, so I never suspected something was amiss although the openblas build process of course clearly reports cc in its output. Mea culpa !

Again, thank you very much for your help !