Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lapack > 3.8.0 fails to build with IBM XL compiler #606

Closed
1 task done
quellyn opened this issue Jul 26, 2021 · 14 comments
Closed
1 task done

lapack > 3.8.0 fails to build with IBM XL compiler #606

quellyn opened this issue Jul 26, 2021 · 14 comments

Comments

@quellyn
Copy link
Contributor

quellyn commented Jul 26, 2021

Beginning with lapack version 3.9.0, I am unable to build using the IBM XL compiler on a Power9 Linux platform.

To reproduce:

  1. Login to a Power9 Linux platform with a relatively recent version of the IBM XL compiler and CMake:
$ arch
ppc64le
$ xlf -qversion
IBM XL Fortran for Linux, V16.1.1 (5725-C75, 5765-J15)
Version: 16.01.0001.0007
$ cmake --version
cmake version 3.19.2

CMake suite maintained and supported by Kitware (kitware.com/cmake).
  1. Download lapack > 3.8.0:
$ wget https://github.com/Reference-LAPACK/lapack/archive/refs/tags/v3.10.0.tar.gz
$ tar xvzf https://github.com/Reference-LAPACK/lapack/archive/refs/tags/v3.10.0.tar.gz
  1. Attempt cmake build:
$ mkdir build; cd build
$ cmake -G 'Unix Makefiles' -DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo -DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=OFF -DBUILD_SHARED_LIBS:BOOL=OFF -DLAPACKE:BOOL=ON -DLAPACKE_WITH_TMG:BOOL=ON -DCBLAS=ON -DCMAKE_Fortran_COMPILER=xlf_r -DCMAKE_Fortran_FLAGS='-F/projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/etc/xlf.cfg.rhel.7.8.gcc.8.3.0.cuda.10.1 -Wl,-rpath,/projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/lib -L/projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/lib -Wl,-rpath,/projects/opt/ppc64le/ibm/xlf-16.1.1.7/lib -L/projects/opt/ppc64le/ibm/xlf-16.1.1.7/lib -O3 -qnohot' -DBUILD_DEPRECATED:BOOL=ON -DBUILD_TESTING:BOOL=OFF /home/quellyn/GIT/lapack/3.10.0/lapack-3.10.0

The above fails with:

-- Verifying Fortran/C Compiler Compatibility - Failed
CMake Error at /projects/opt/rhel7/ppc64le/cmake/3.19.2/share/cmake-3.19/Modules/FortranCInterface.cmake:383 (message):
  The Fortran compiler:

    /projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/bin/xlf_r

  and the C compiler:

    /projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/bin/xlc

  failed to compile a simple test project using both languages.  The output
  was:

    Change Dir: /vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC
    
    Run Build Command(s):/usr/bin/gmake VerifyFortranC && /projects/opt/rhel7/ppc64le/cmake/3.19.2/bin/cmake -S/projects/opt/rhel7/ppc64le/cmake/3.19.2/share/cmake-3.19/Modules/FortranCInterface/Verify -B/vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC --check-build-system CMakeFiles/Makefile.cmake 0
    /usr/bin/gmake  -f CMakeFiles/Makefile2 VerifyFortranC
    gmake[1]: Entering directory `/vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC'
    /projects/opt/rhel7/ppc64le/cmake/3.19.2/bin/cmake -S/projects/opt/rhel7/ppc64le/cmake/3.19.2/share/cmake-3.19/Modules/FortranCInterface/Verify -B/vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC --check-build-system CMakeFiles/Makefile.cmake 0
    /projects/opt/rhel7/ppc64le/cmake/3.19.2/bin/cmake -E cmake_progress_start /vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC/CMakeFiles 5
    /usr/bin/gmake  -f CMakeFiles/Makefile2 CMakeFiles/VerifyFortranC.dir/all
    gmake[2]: Entering directory `/vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC'
    /usr/bin/gmake  -f CMakeFiles/VerifyFortran.dir/build.make CMakeFiles/VerifyFortran.dir/depend
    gmake[3]: Entering directory `/vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC'
    cd /vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC && /projects/opt/rhel7/ppc64le/cmake/3.19.2/bin/cmake -E cmake_depends "Unix Makefiles" /projects/opt/rhel7/ppc64le/cmake/3.19.2/share/cmake-3.19/Modules/FortranCInterface/Verify /projects/opt/rhel7/ppc64le/cmake/3.19.2/share/cmake-3.19/Modules/FortranCInterface/Verify /vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC /vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC /vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC/CMakeFiles/VerifyFortran.dir/DependInfo.cmake
    Scanning dependencies of target VerifyFortran
    gmake[3]: Leaving directory `/vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC'
    /usr/bin/gmake  -f CMakeFiles/VerifyFortran.dir/build.make CMakeFiles/VerifyFortran.dir/build
    gmake[3]: Entering directory `/vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC'
    [ 20%] Building Fortran object CMakeFiles/VerifyFortran.dir/VerifyFortran.f.o
    /projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/bin/xlf_r  -I/vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC -F/projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/etc/xlf.cfg.rhel.7.8.gcc.8.3.0.cuda.10.1 -Wl,-rpath,/projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/lib -L/projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/lib -Wl,-rpath,/projects/opt/ppc64le/ibm/xlf-16.1.1.7/lib -L/projects/opt/ppc64le/ibm/xlf-16.1.1.7/lib -O3 -qnohot -recursive -qnosave -qstrict=none -O -c /projects/opt/rhel7/ppc64le/cmake/3.19.2/share/cmake-3.19/Modules/FortranCInterface/Verify/VerifyFortran.f -o CMakeFiles/VerifyFortran.dir/VerifyFortran.f.o
    ** verifyfortran   === End of Compilation 1 ===
    1501-510  Compilation successful for file VerifyFortran.f.
    [ 40%] Linking Fortran static library libVerifyFortran.a
    /projects/opt/rhel7/ppc64le/cmake/3.19.2/bin/cmake -P CMakeFiles/VerifyFortran.dir/cmake_clean_target.cmake
    /projects/opt/rhel7/ppc64le/cmake/3.19.2/bin/cmake -E cmake_link_script CMakeFiles/VerifyFortran.dir/link.txt --verbose=1
    /usr/bin/ar qc libVerifyFortran.a CMakeFiles/VerifyFortran.dir/VerifyFortran.f.o
    /usr/bin/ranlib libVerifyFortran.a
    gmake[3]: Leaving directory `/vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC'
    [ 40%] Built target VerifyFortran
    /usr/bin/gmake  -f CMakeFiles/VerifyFortranC.dir/build.make CMakeFiles/VerifyFortranC.dir/depend
    gmake[3]: Entering directory `/vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC'
    cd /vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC && /projects/opt/rhel7/ppc64le/cmake/3.19.2/bin/cmake -E cmake_depends "Unix Makefiles" /projects/opt/rhel7/ppc64le/cmake/3.19.2/share/cmake-3.19/Modules/FortranCInterface/Verify /projects/opt/rhel7/ppc64le/cmake/3.19.2/share/cmake-3.19/Modules/FortranCInterface/Verify /vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC /vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC /vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC/CMakeFiles/VerifyFortranC.dir/DependInfo.cmake
    Scanning dependencies of target VerifyFortranC
    gmake[3]: Leaving directory `/vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC'
    /usr/bin/gmake  -f CMakeFiles/VerifyFortranC.dir/build.make CMakeFiles/VerifyFortranC.dir/build
    gmake[3]: Entering directory `/vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC'
    [ 60%] Building C object CMakeFiles/VerifyFortranC.dir/main.c.o
    /projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/bin/xlc  -F/projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/etc/xlc.cfg.rhel.7.8.gcc.8.3.0.cuda.10.1  -I/vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC -O -o CMakeFiles/VerifyFortranC.dir/main.c.o -c /projects/opt/rhel7/ppc64le/cmake/3.19.2/share/cmake-3.19/Modules/FortranCInterface/Verify/main.c
    [ 80%] Building C object CMakeFiles/VerifyFortranC.dir/VerifyC.c.o
    /projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/bin/xlc  -F/projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/etc/xlc.cfg.rhel.7.8.gcc.8.3.0.cuda.10.1  -I/vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC -O -o CMakeFiles/VerifyFortranC.dir/VerifyC.c.o -c /projects/opt/rhel7/ppc64le/cmake/3.19.2/share/cmake-3.19/Modules/FortranCInterface/Verify/VerifyC.c
    [100%] Linking C executable VerifyFortranC
    /projects/opt/rhel7/ppc64le/cmake/3.19.2/bin/cmake -E cmake_link_script CMakeFiles/VerifyFortranC.dir/link.txt --verbose=1
    /projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/bin/xlc  -F/projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/etc/xlc.cfg.rhel.7.8.gcc.8.3.0.cuda.10.1 -O CMakeFiles/VerifyFortranC.dir/main.c.o CMakeFiles/VerifyFortranC.dir/VerifyC.c.o -o VerifyFortranC   -L/projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/lib  -L/projects/opt/ppc64le/ibm/xlf-16.1.1.7/lib  -L/projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlsmp/5.1.1/lib  -L/projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlmass/9.1.1/lib  libVerifyFortran.a -lxlf90_r -lxlomp_ser -lxlfmath -lrt 
    CMakeFiles/VerifyFortranC.dir/main.c.o: In function `main':
    /projects/opt/rhel7/ppc64le/cmake/3.19.2/share/cmake-3.19/Modules/FortranCInterface/Verify/main.c:(.text+0x1c): undefined reference to `VerifyFortran'
    gmake[3]: *** [VerifyFortranC] Error 1
    gmake[3]: Leaving directory `/vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC'
    gmake[2]: *** [CMakeFiles/VerifyFortranC.dir/all] Error 2
    gmake[2]: Leaving directory `/vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC'
    gmake[1]: *** [CMakeFiles/VerifyFortranC.dir/rule] Error 2
    gmake[1]: Leaving directory `/vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/FortranCInterface/VerifyC'
    gmake: *** [VerifyFortranC] Error 2
    
    
Call Stack (most recent call first):
  CBLAS/CMakeLists.txt:9 (FortranCInterface_VERIFY)


-- Configuring incomplete, errors occurred!
See also "/vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/CMakeOutput.log".
See also "/vast/home/quellyn/GIT/lapack/3.10.0/build/CMakeFiles/CMakeError.log".

I've included the CMake output and error logs:

CMakeError.log
CMakeOutput.log

I'm including @KineticTheory on this report, as he has an interest in this issue as well.

Thanks for looking!
Quellyn Snead

Checklist

  • I've included a minimal example to reproduce the issue
  • [] I'd be willing to make a PR to solve this issue
@martin-frbg
Copy link
Collaborator

This looks to me to be failing in generic CMAKE test code for verifying that the compiler can build anything at all, way before anything LAPACK-specific. Were you able to compile 3.8.0 on the exact same system ?

@quellyn
Copy link
Contributor Author

quellyn commented Jul 27, 2021

This looks to me to be failing in generic CMAKE test code for verifying that the compiler can build anything at all, way before anything LAPACK-specific. Were you able to compile 3.8.0 on the exact same system ?

Hi Martin,

Yes, 3.8.0 will work with a small patch. To see that, do:

$ wget https://github.com/Reference-LAPACK/lapack/archive/refs/tags/v3.8.0.tar.gz
$ tar xvzf https://github.com/Reference-LAPACK/lapack/archive/refs/tags/v3.8.0.tar.gz
$ cd lapack-3.8.0
$ patch -p1 < /path/to/ibm-xl.patch
$ cd ..

Then do a CMake build as above:

$ mkdir build; cd build

$ cmake -G 'Unix Makefiles' -DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo -DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=OFF -DBUILD_SHARED_LIBS:BOOL=OFF -DLAPACKE:BOOL=ON -DLAPACKE_WITH_TMG:BOOL=ON -DCBLAS=ON -DCMAKE_Fortran_COMPILER=xlf_r -DCMAKE_Fortran_FLAGS='-F/projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/etc/xlf.cfg.rhel.7.8.gcc.8.3.0.cuda.10.1 -Wl,-rpath,/projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/lib -L/projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/lib -Wl,-rpath,/projects/opt/ppc64le/ibm/xlf-16.1.1.7/lib -L/projects/opt/ppc64le/ibm/xlf-16.1.1.7/lib -O3 -qnohot' -DBUILD_DEPRECATED:BOOL=ON -DBUILD_TESTING:BOOL=OFF /home/quellyn/GIT/lapack/3.8.0/lapack-3.8.0

I've attached the output of that so you can see it was successful.

This patch originated from a PR that @KineticTheory put into lapack some time ago. It was adopted into Spack and is used there successfully for building lapack 3.8.0 as shown above. However, it does not apply to 3.9.0. Based on @KineticTheory 's PR, it appears that all or some of it was accepted upstream. Even so, a build of lapack 3.9.0 fails:

$ wget https://github.com/Reference-LAPACK/lapack/archive/refs/tags/v3.9.0.tar.gz
$ tar xvzf v3.9.0.tar.gz
$ mkdir build; cd build
$ cmake -G 'Unix Makefiles' -DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo -DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=OFF -DBUILD_SHARED_LIBS:BOOL=OFF -DLAPACKE:BOOL=ON -DLAPACKE_WITH_TMG:BOOL=ON -DCBLAS=ON -DCMAKE_Fortran_COMPILER=xlf_r -DCMAKE_Fortran_FLAGS='-F/projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/etc/xlf.cfg.rhel.7.8.gcc.8.3.0.cuda.10.1 -Wl,-rpath,/projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/lib -L/projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/lib -Wl,-rpath,/projects/opt/ppc64le/ibm/xlf-16.1.1.7/lib -L/projects/opt/ppc64le/ibm/xlf-16.1.1.7/lib -O3 -qnohot' -DBUILD_DEPRECATED:BOOL=ON -DBUILD_TESTING:BOOL=OFF /home/quellyn/GIT/lapack/3.9.0/lapack-3.9.0

The errors are the same as I see in 3.10.0:

-- The Fortran compiler identification is XL 16.1.1
-- The C compiler identification is XLClang 16.1.1.7
-- Check for working Fortran compiler: /projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/bin/xlf_r
-- Check for working Fortran compiler: /projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/bin/xlf_r - works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/bin/xlf_r supports Fortran 90
-- Checking whether /projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/bin/xlf_r supports Fortran 90 - yes
-- Check for working C compiler: /projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/bin/xlc
-- Check for working C compiler: /projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/bin/xlc - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Checking if build type is 'Coverage'
-- Checking if build type is 'Coverage': 0
-- Build tests: OFF
-- Enabling fixed format F90/F95 with -qfixed
-- Looking for Fortran NONE - found
-- Looking for Fortran INT_CPU_TIME - found
-- Looking for Fortran EXT_ETIME - not found
-- Looking for Fortran EXT_ETIME_ - found
-- Looking for Fortran INT_ETIME - not found
-- --> Will use second_EXT_ETIME_.f and dsecnd_EXT_ETIME_.f as timing function.
-- Build deprecated routines: ON
-- Build single precision real: ON
-- Build double precision real: ON
-- Build single precision complex: ON
-- Build double precision complex: ON
-- Using supplied NETLIB BLAS implementation
-- CBLAS enable
-- Detecting Fortran/C Interface
Failed to compile
-- Verifying Fortran/C Compiler Compatibility
Failed to compile
CMake Warning (dev) at /projects/opt/ppc64le/p9/cmake/3.17.3/share/cmake-3.17/Modules/FortranCInterface.cmake:309 (message):
  No FortranCInterface mangling known for VerifyFortran
Call Stack (most recent call first):
  /projects/opt/ppc64le/p9/cmake/3.17.3/share/cmake-3.17/Modules/FortranCInterface/Verify/CMakeLists.txt:16 (FortranCInterface_HEADER)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Verifying Fortran/C Compiler Compatibility - Failed
CMake Error at /projects/opt/ppc64le/p9/cmake/3.17.3/share/cmake-3.17/Modules/FortranCInterface.cmake:383 (message):
  The Fortran compiler:

    /projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/bin/xlf_r

  and the C compiler:

    /projects/opt/ppc64le/ibm/xlc-16.1.1.7/xlC/16.1.1/bin/xlc

  failed to compile a simple test project using both languages.  The output
  was:

    Change Dir: /home/quellyn/GIT/lapack/3.9.0/build/CMakeFiles/FortranCInterface/VerifyC
    
    Run Build Command(s):/usr/bin/gmake VerifyFortranC && /projects/opt/ppc64le/p9/cmake/3.17.3/bin/cmake -S/projects/opt/ppc64le/p9/cmake/3.17.3/share/cmake-3.17/Modules/FortranCInterface/Verify -B/home/quellyn/GIT/lapack/3.9.0/build/CMakeFiles/FortranCInterface/VerifyC --check-build-system CMakeFiles/Makefile.cmake 0
    /usr/bin/gmake  -f CMakeFiles/Makefile2 VerifyFortranC
    gmake[1]: Entering directory `/vast/home/quellyn/GIT/lapack/3.9.0/build/CMakeFiles/FortranCInterface/VerifyC'
    /projects/opt/ppc64le/p9/cmake/3.17.3/bin/cmake -S/projects/opt/ppc64le/p9/cmake/3.17.3/share/cmake-3.17/Modules/FortranCInterface/Verify -B/home/quellyn/GIT/lapack/3.9.0/build/CMakeFiles/FortranCInterface/VerifyC --check-build-system CMakeFiles/Makefile.cmake 0
    /projects/opt/ppc64le/p9/cmake/3.17.3/bin/cmake -E cmake_progress_start /home/quellyn/GIT/lapack/3.9.0/build/CMakeFiles/FortranCInterface/VerifyC/CMakeFiles 5
    /usr/bin/gmake  -f CMakeFiles/Makefile2 CMakeFiles/VerifyFortranC.dir/all
    gmake[2]: Entering directory `/vast/home/quellyn/GIT/lapack/3.9.0/build/CMakeFiles/FortranCInterface/VerifyC'
    /usr/bin/gmake  -f CMakeFiles/VerifyFortran.dir/build.make CMakeFiles/VerifyFortran.dir/depend
    gmake[3]: Entering directory `/vast/home/quellyn/GIT/lapack/3.9.0/build/CMakeFiles/FortranCInterface/VerifyC'
    cd /home/quellyn/GIT/lapack/3.9.0/build/CMakeFiles/FortranCInterface/VerifyC && /projects/opt/ppc64le/p9/cmake/3.17.3/bin/cmake -E cmake_depends "Unix Makefiles" /projects/opt/ppc64le/p9/cmake/3.17.3/share/cmake-3.17/Modules/FortranCInterface/Verify /projects/opt/ppc64le/p9/cmake/3.17.3/share/cmake-3.17/Modules/FortranCInterface/Verify /home/quellyn/GIT/lapack/3.9.0/build/CMakeFiles/FortranCInterface/VerifyC /home/quellyn/GIT/lapack/3.9.0/build/CMakeFiles/FortranCInterface/VerifyC /home/quellyn/GIT/lapack/3.9.0/build/CMakeFiles/FortranCInterface/VerifyC/CMakeFiles/VerifyFortran.dir/DependInfo.cmake
    Scanning dependencies of target VerifyFortran
    gmake[3]: Leaving directory `/vast/home/quellyn/GIT/lapack/3.9.0/build/CMakeFiles/FortranCInterface/VerifyC'
    /usr/bin/gmake  -f CMakeFiles/VerifyFortran.dir/build.make CMakeFiles/VerifyFortran.dir/build
    gmake[3]: Entering directory `/vast/home/quellyn/GIT/lapack/3.9.0/build/CMakeFiles/FortranCInterface/VerifyC'
    [ 20%] Building Fortran object CMakeFiles/VerifyFortran.dir/VerifyFortran.f.o
    /projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/bin/xlf_r  -I/home/quellyn/GIT/lapack/3.9.0/build/CMakeFiles/FortranCInterface/VerifyC  -F/projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/etc/xlf.cfg.rhel.7.8.gcc.8.3.0.cuda.10.1 -Wl,-rpath,/projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/lib -L/projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/lib -Wl,-rpath,/projects/opt/ppc64le/ibm/xlf-16.1.1.7/lib -L/projects/opt/ppc64le/ibm/xlf-16.1.1.7/lib -O3 -qnohot;-qnosave;-qstrict=none -qfixed -O   -c /projects/opt/ppc64le/p9/cmake/3.17.3/share/cmake-3.17/Modules/FortranCInterface/Verify/VerifyFortran.f -o CMakeFiles/VerifyFortran.dir/VerifyFortran.f.o
    /projects/opt/ppc64le/ibm/xlf-16.1.1.7/xlf/16.1.1/bin/.orig/xlf_r: 1501-294 (S) No input file specified. Please use -qhelp for more information.
    /bin/sh: -qnosave: command not found
    /bin/sh: -qstrict=none: command not found
    gmake[3]: *** [CMakeFiles/VerifyFortran.dir/VerifyFortran.f.o] Error 127
    gmake[3]: Leaving directory `/vast/home/quellyn/GIT/lapack/3.9.0/build/CMakeFiles/FortranCInterface/VerifyC'
    gmake[2]: *** [CMakeFiles/VerifyFortran.dir/all] Error 2
    gmake[2]: Leaving directory `/vast/home/quellyn/GIT/lapack/3.9.0/build/CMakeFiles/FortranCInterface/VerifyC'
    gmake[1]: *** [CMakeFiles/VerifyFortranC.dir/rule] Error 2
    gmake[1]: Leaving directory `/vast/home/quellyn/GIT/lapack/3.9.0/build/CMakeFiles/FortranCInterface/VerifyC'
    gmake: *** [VerifyFortranC] Error 2
    
    
Call Stack (most recent call first):
  CBLAS/CMakeLists.txt:9 (FortranCInterface_VERIFY)


-- Configuring incomplete, errors occurred!
See also "/home/quellyn/GIT/lapack/3.9.0/build/CMakeFiles/CMakeOutput.log".
See also "/home/quellyn/GIT/lapack/3.9.0/build/CMakeFiles/CMakeError.log".

Thanks for looking at this,
Quellyn

@martin-frbg
Copy link
Collaborator

Hmm. The effect is the same, but the source of the error appears to be different - in the 3.9.0 log, the compiler aborts because it does not understand the -qnosave -qstrict=none options from the toplevel CMakeLists.txt (which is part of what the patch fixed, and something that was not in the LAPACK PR). In your build log for 3.10.0, it looks to me as if the test failed because it tried to link against a library component of the GCC compiler (libgcc_s) that could not be found, suggesting that a library search path may be set up incorrectly
The failing test itself was added in 3.7.0, and has not been modified since then. Unfortunately I do not have easy access to power9 right now except through the OSUOSL Jenkins CI, so not sure if I can reproduce this myself

@quellyn
Copy link
Contributor Author

quellyn commented Jul 28, 2021

Hmm. The effect is the same, but the source of the error appears to be different - in the 3.9.0 log, the compiler aborts because it does not understand the -qnosave -qstrict=none options from the toplevel CMakeLists.txt (which is part of what the patch fixed, and something that was not in the LAPACK PR). In your build log for 3.10.0, it looks to me as if the test failed because it tried to link against a library component of the GCC compiler (libgcc_s) that could not be found, suggesting that a library search path may be set up incorrectly
The failing test itself was added in 3.7.0, and has not been modified since then. Unfortunately I do not have easy access to power9 right now except through the OSUOSL Jenkins CI, so not sure if I can reproduce this myself

Usually I perform all my builds within Spack, but I did these build attempts by hand in order to make the simplest possible reproducer. It's possible that I did not have something set correctly in that case. Let me repeat all the various build attempts (3.8.0, 3.9.0, and 3.10.0) and make sure that I am running all of them in the same way.

@quellyn
Copy link
Contributor Author

quellyn commented Jul 28, 2021

Okay, I have repeated all the builds, one after another, on the same Power9 node with the same modules (for cmake and IBM XL) loaded. Here are the complete build directories with the CMake output logs, etc.:

3.8.0.tar.gz
3.9.0.tar.gz
3.10.0.tar.gz

I apologize for uploading such large files but as you don't have ready access to a similar Power9 platform I wanted to give you as much as I could.

@quellyn
Copy link
Contributor Author

quellyn commented Jul 28, 2021

Hmm. The effect is the same, but the source of the error appears to be different - in the 3.9.0 log, the compiler aborts because it does not understand the -qnosave -qstrict=none options from the toplevel CMakeLists.txt (which is part of what the patch fixed, and something that was not in the LAPACK PR).

Yes, I can see that now. I've created a new patch to address the -qnosave -qstrict=none issue. When I apply that to 3.9.0 or 3.10.0, I get a little further:

3.9.0-patched.tar.gz
3.10.0-patched.tar.gz

@martin-frbg
Copy link
Collaborator

martin-frbg commented Jul 28, 2021

Actually in 3.9.0, "something" appears to have inserted semicolon characters into the xlf command line between the qnohot/qnosave/qstrict options, turning it into separate, nonsensical commands. (So first xlf complains about not seeing a file to compile, then the shell complains about an unrecognized command "qnohot" etc.) I think this is related to how the options string is concatenated in cmake (i.e. I made such a mistake in a different project in the past) but I need to find where it happens.
The patched 3.10.0 is still looking in vain for libgcc_s - that dependency does not appear to have been present in 3.7., but the only difference in options seems to be the -recursive (which I think is unlikely to cause it)

@quellyn
Copy link
Contributor Author

quellyn commented Jul 28, 2021

Actually in 3.9.0, "something" appears to have inserted semicolon characters into the xlf command line between the qnohot/qnosave/qstrict options, turning it into separate, nonsensical commands. (So first xlf complains about not seeing a file to compile, then the shell complains about an unrecognized command "qnohot" etc.) I think this is related to how the options string is concatenated in cmake (i.e. I made such a mistake in a different project in the past) but I need to find where it happens.
The patched 3.10.0 is still looking in vain for libgcc_s - that dependency does not appear to have been present in 3.7., but the only difference in options seems to be the -recursive (which I think is unlikely to cause it)

Thank you again for looking. Please let me know if there's anything I can try out on my Power9 platform here. I'm not much of a coder but I'm willing to try with some guidance :-)

@quellyn
Copy link
Contributor Author

quellyn commented Aug 16, 2021

Any progress on this issue? Anything you would like me to try out on my end?

@martin-frbg
Copy link
Collaborator

Sorry, no idea why the 3.10.0 build would not find libgcc_s , and no time to look further. (The library seems to have been found in the other builds, the test that needs it is part of cmake and I saw no obvious difference in compiler arguments.)
The stray semicolons in the 3.9.0 build must have come from using "LIST (APPEND ... " in your patch without converting the list back to a string before passing it to the compiler - maybe you used a different patch there, or maybe something changed in the
3.10 cmakefiles that happened to do the conversion for you

@quellyn
Copy link
Contributor Author

quellyn commented Sep 4, 2021

Hi Martin,

Sorry, no idea why the 3.10.0 build would not find libgcc_s , and no time to look further. (The library seems to have been found in the other builds, the test that needs it is part of cmake and I saw no obvious difference in compiler arguments.)

I've found the problem: Beginning with netlib-lapack 3.9.1, new logic was added in the top-level CMakeLists.txt that attempts to detect and add compiler-specific flags for recursive Fortran procedures. I needed to add in the flag for IBM XLF (-qrecur). Additionally, I found that CMake's CheckFortranCompilerFlag would also add -recursive to CMAKE_Fortran_FLAGS, which is incorrect as that is an Intel-specific flag. The resulting attempt by xlf to compile with this bogus flag fails.

You can see the incorrect compiler behavior with a simple "Hello World" program:

$ xlf_r -recursive -o hello-dumb hello.f90
** helloworld   === End of Compilation 1 ===
1501-510  Compilation successful for file hello.f90.
ld: cannot find -lgcc_s
ld: cannot find -lgcc_s

$ ls
hello.f90  hello.o

$ file hello.o
hello.o: ELF 64-bit LSB relocatable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), not stripped

The xlf compiler produces an intermediate object code that will not link -- but it doesn't outright fail. In the case of netlib-lapack's CMakeLists.txt, I believe that CMake CheckFortranCompilerFlag is interpreting that as success and so goes ahead and adds this bogus flag to CMAKE_Fortran_FLAGS, which results in the compile failure. I've submitted a patch to the Spack project which corrects the problem, and have attached it here in case you'd like to consider upstreaming it. If so, please let me know and I'll open a PR with the LAPACK project for this:

ibm-xl-3.9.1.patch.txt

The stray semicolons in the 3.9.0 build must have come from using "LIST (APPEND ... " in your patch without converting the list back to a string before passing it to the compiler - maybe you used a different patch there, or maybe something changed in the 3.10 cmakefiles that happened to do the conversion for you

Indeed, that was the issue. In netlib-lapack 3.8.0 the options were added via a simple string; beginning with netlib-lapack 3.9.0 that logic was changed to use lists but failed to convert the result back to a string. In netlib-lapack 3.10.0, they've reverted back to using simple strings.

@martin-frbg
Copy link
Collaborator

Ah, great. Yes please open a PR if you have the time. And sorry again for dropping out of this discussion - I do not have access to ppc systems with the xlf compiler, and I was busy with other tasks.

@martin-frbg
Copy link
Collaborator

(And looking at your PR for spack, perhaps the current set of tests for a supported "recursive" keyword could be retained as a catch-all in case the Fortran compiler is something else not on the list of known names ?)

mpbelhorn added a commit to mpbelhorn/olcf-spack-environments that referenced this issue Sep 9, 2021
Additional build variants of TAU with combinatorial support CUDA and OMP
profiling enabled-or-disabled are added to Summit. Module suffixes are
used to distinguish the different build variants.

Darshan-runtime without HDF5 support is added and made the default
build. Modules for this variant are suffixed `-lite` and re-added to the
default loaded DefApps modules.

Adios 1.13.1 is installed for a special case request by a user.

Imagemagick is installed at user request.

An attempt to install the latest netlib-lapack using the XL compilers is
added. These builds still fail due to a configure-time bug validating
xlf when building the static versions of the libraries. XLF attempts to
link GCC runtime libs into the build without providing the necessary
library directories or without static libraries available. The failure output
from a build attempt which forces the `-qstaticlink` flag is shown below, but
`-qstaticlink` is not set for a normal build and it's presence has no effect on
the error. A similar error is already reported upstream Reference-LAPACK/lapack#606:

```
Build flags: ;-O3;-qnohot;-qstaticlink
Id flags: -v

The output was:
1
exec: export(export,XL_CONFIG=/sw/summit/xl/16.1.1-10/xlf/16.1.1/etc/xlf.cfg.rhel.8.2.gcc.8.3.1.cuda.11.0:xlf_r,NULL)
exec: export(export,XL_DIS=/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/exe/dis -o "CMakeFortranCompilerId.s" "CMakeFortranCompilerId.o",NULL)
exec: /autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/exe/cpp(/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/exe/cpp,CMakeFortranCompilerId.F,/tmp/xlf.jWEE22/0hkYDH,-D__ibmxl__,-C,-q64,-I/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlmass/9.1.1/include,-I/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlsmp/5.1.1/include,-I/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/include,NULL)
exec: /autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/exe/xlfentry(/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/exe/xlfentry,/tmp/xlf.jWEE22/0hkYDH,/tmp/xlf.jWEE22/vMULQf,/tmp/xlf.jWEE22/vMULQfF.lst,xlfsmsg.cat,xlfmsg.cat,CMakeFortranCompilerId.F,OSVAR(rhel.8.2),THREADED,NOZEROSIZE,ALIAS(intptr),POSITION(appendold),XLF90(noautodealloc,nosignedzero,oldpad),XLF77(intarg,intxor,noleadzero,gedit77,noblankpad,oldboz,softeof),XLF2003(nopolymorphic,nobozlitargs,nostopexcept,novolatile,noautorealloc,oldnaninf),XLF2008(nocheckpresence),64,GNU_VERSION(8.3.1),SAVEOPTFILE(/tmp/xlf.jWEE22/optf.lUZ5eB),SAVEVERSIONFILE(/tmp/xlf.jWEE22/version.Ck8wr9),OPT(3),NOHOT,WSTREAMS(/tmp/xlf.jWEE22/h1,/tmp/xlf.jWEE22/b1,/tmp/xlf.jWEE22/s1),DEFMSG(/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/msg/en_US),XFLAG(CPP),-I/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlmass/9.1.1/include,-I/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlsmp/5.1.1/include,-I/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/include,NULL)
** cmakefortrancompilerid   === End of Compilation 1 ===
unlink: /tmp/xlf.jWEE22/0hkYDH
exec: export(export,XL_FRONTEND=/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/exe/xlfentry,NULL)
exec: export(export,XL_ASTI=/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/exe/xlfhot,NULL)
exec: export(export,XL_BACKEND=/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/exe/xlfcode,NULL)
exec: export(export,XL_LINKER=/usr/bin/ld,NULL)
exec: export(export,XL_BOLT=/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/exe/bolt,NULL)
exec: /autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/exe/xlfhot(/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/exe/xlfhot,/tmp/xlf.jWEE22/h1,/tmp/xlf.jWEE22/h2,/tmp/xlf.jWEE22/b1,/tmp/xlf.jWEE22/b2,/tmp/xlf.jWEE22/s1,/tmp/xlf.jWEE22/s2,/tmp/xlf.jWEE22/vMULQf,/tmp/xlf.jWEE22/vMULQfA.lst,NULL)
exec: export(export,XL_FRONTEND=/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/exe/xlfentry,NULL)
exec: export(export,XL_ASTI=/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/exe/xlfhot,NULL)
exec: export(export,XL_BACKEND=/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/exe/xlfcode,NULL)
exec: export(export,XL_LINKER=/usr/bin/ld,NULL)
exec: export(export,XL_BOLT=/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/exe/bolt,NULL)
exec: /autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/exe/xlfcode(/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/exe/xlfcode,/tmp/xlf.jWEE22/h2,/tmp/xlf.jWEE22/b2,CMakeFortranCompilerId.o,/tmp/xlf.jWEE22/vMULQfB.lst,/tmp/xlf.jWEE22/s2,NULL)
1501-510  Compilation successful for file CMakeFortranCompilerId.F.
unlink: /tmp/xlf.jWEE22/vMULQfA.lst
exec: /usr/bin/ld(ld,--eh-frame-hdr,-Qy,-melf64lppc,-static,/usr/lib/gcc/ppc64le-redhat-linux/8/../../../../lib64/crt1.o,/usr/lib/gcc/ppc64le-redhat-linux/8/../../../../lib64/crti.o,/usr/lib/gcc/ppc64le-redhat-linux/8/crtbeginT.o,-L/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlsmp/5.1.1/lib,-L/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlmass/9.1.1/lib,-L/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/xlf/16.1.1/lib,-R/autofs/nccs-svm1_sw/summit/xl/16.1.1-10/lib,-L/usr/lib/gcc/ppc64le-redhat-linux/8,-L/usr/lib/gcc/ppc64le-redhat-linux/8/../../../../lib64,-L/lib/../lib64,-L/usr/lib/../lib64,-L/sw/summit/spack-envs/base/opt/linux-rhel8-ppc64le/xl-16.1.1-10/spectrum-mpi-10.4.0.3-20210112-v7qymniwgi6mtxqsjd7p5jxinxzdkhn3/lib,-L/ccs/proj/ccsstaff/olcf4_acceptance/devel/nccs_test_harness/libraries,-L/usr/lib/gcc/ppc64le-redhat-linux/8/../../..,--no-toc-optimize,--no-toc-optimize,--no-toc-optimize,--no-toc-optimize,--no-toc-optimize,--no-toc-optimize,CMakeFortranCompilerId.o,-lxlf90_r,-lxlopt,-lxlomp_ser,-lxl,-lxlfmath,-ldl,-lrt,-lpthread,-lm,-lc,-lgcc_eh,-lc,-lgcc,/usr/lib/gcc/ppc64le-redhat-linux/8/crtend.o,/usr/lib/gcc/ppc64le-redhat-linux/8/../../../../lib64/crtn.o,NULL)  |
exec: /usr/bin/awk(/usr/bin/awk,BEGIN{a=0;b=0;s=""};{if(($0!~/.*-224.*@tp.*/)&&($0!~/.*-34[45].*loadmap.*/||b==1||a==0)){b=1;print $0;}else{a=1;if((s=="")&& $0 ~ /.*-34[45].*loadmap.*/){s=$0}}};END{if(b==1 && s!=""){print s}},NULL)
ld: cannot find -ldl
ld: cannot find -lrt
ld: cannot find -lpthread
ld: cannot find -lm
ld: cannot find -lc
ld: cannot find -lc
unlink: /tmp/xlf.jWEE22/optf.lUZ5eB
unlink: /tmp/xlf.jWEE22/h1
unlink: /tmp/xlf.jWEE22/h2
unlink: /tmp/xlf.jWEE22/b1
unlink: /tmp/xlf.jWEE22/b2
unlink: /tmp/xlf.jWEE22/s1
unlink: /tmp/xlf.jWEE22/s2
unlink: /tmp/xlf.jWEE22/version.Ck8wr9
unlink: /tmp/xlf.jWEE22
```
quellyn pushed a commit to quellyn/lapack that referenced this issue Sep 15, 2021
Be more explicit about selecting recursive fortran flags, as not
every compiler will correctly reject an incorrect option (e.g., XL).
@quellyn
Copy link
Contributor Author

quellyn commented Jan 27, 2022

I think this issue can be closed. My troubles have been resolved by the above-mentioned PR to lapack.

@quellyn quellyn closed this as completed Jan 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants