tests suddenly failing in LAPACKConfig.cmake

[glhenni]

I believe something in TriBITs has broken our code. Last night all our tests failed during the configure phase with errors from tribits shown below. We're using cmake 3.19.2

CMake Error at /scratch/charonops/jenkins/workspace/ASCIC_Build_Farm_CDE-GNU-cov/TEST.COV/cmake_packages/LAPACK/LAPACKConfig.cmake:17 (add_library):
  add_library cannot create imported target "LAPACK::dl" because another
  target with the same name already exists.
Call Stack (most recent call first):
  /scratch/charonops/jenkins/workspace/ASCIC_Build_Farm_CDE-GNU-cov/TriBITS/tribits/core/package_arch/TribitsTplFindIncludeDirsAndLibraries.cmake:686 (include)
  /scratch/charonops/jenkins/workspace/ASCIC_Build_Farm_CDE-GNU-cov/TriBITS/tribits/common_tpls/FindTPLLAPACK.cmake:107 (tribits_tpl_find_include_dirs_and_libraries)
  /scratch/charonops/jenkins/workspace/ASCIC_Build_Farm_CDE-GNU-cov/TriBITS/tribits/core/package_arch/TribitsProcessEnabledTpl.cmake:106 (include)
  /scratch/charonops/jenkins/workspace/ASCIC_Build_Farm_CDE-GNU-cov/TriBITS/tribits/core/package_arch/TribitsGlobalMacros.cmake:1565 (tribits_process_enabled_tpl)
  /scratch/charonops/jenkins/workspace/ASCIC_Build_Farm_CDE-GNU-cov/TriBITS/tribits/core/package_arch/TribitsProjectImpl.cmake:196 (tribits_process_enabled_tpls)
  /scratch/charonops/jenkins/workspace/ASCIC_Build_Farm_CDE-GNU-cov/TriBITS/tribits/core/package_arch/TribitsProject.cmake:93 (tribits_project_impl)
  CMakeLists.txt:61 (TRIBITS_PROJECT)


CMake Error at /scratch/charonops/jenkins/workspace/ASCIC_Build_Farm_CDE-GNU-cov/TEST.COV/cmake_packages/LAPACK/LAPACKConfig.cmake:23 (add_library):
  add_library cannot create imported target "LAPACK::m" because another
  target with the same name already exists.
Call Stack (most recent call first):
  /scratch/charonops/jenkins/workspace/ASCIC_Build_Farm_CDE-GNU-cov/TriBITS/tribits/core/package_arch/TribitsTplFindIncludeDirsAndLibraries.cmake:686 (include)
  /scratch/charonops/jenkins/workspace/ASCIC_Build_Farm_CDE-GNU-cov/TriBITS/tribits/common_tpls/FindTPLLAPACK.cmake:107 (tribits_tpl_find_include_dirs_and_libraries)
  /scratch/charonops/jenkins/workspace/ASCIC_Build_Farm_CDE-GNU-cov/TriBITS/tribits/core/package_arch/TribitsProcessEnabledTpl.cmake:106 (include)
  /scratch/charonops/jenkins/workspace/ASCIC_Build_Farm_CDE-GNU-cov/TriBITS/tribits/core/package_arch/TribitsGlobalMacros.cmake:1565 (tribits_process_enabled_tpl)
  /scratch/charonops/jenkins/workspace/ASCIC_Build_Farm_CDE-GNU-cov/TriBITS/tribits/core/package_arch/TribitsProjectImpl.cmake:196 (tribits_process_enabled_tpls)
  /scratch/charonops/jenkins/workspace/ASCIC_Build_Farm_CDE-GNU-cov/TriBITS/tribits/core/package_arch/TribitsProject.cmake:93 (tribits_project_impl)
  CMakeLists.txt:61 (TRIBITS_PROJECT)

[bartlettroscoe]

@glhenni, can you please attach the output from the configure of Trilinos cmake STDOUT? specifically, I want to see what is shown for the line -- TPL_LAPACK_LIBRARIES. My guess is that we are going to see duplicate entries for dl and m but I want to confirm that before I implement unit tests to reproduce and put in a fix.

FYI: This must have been triggered by the merge of PR #426 yesterday and Charon2 must be pulling in TriBITS from 'master'.

[glhenni]

We have some messiness of our own in our cmake so I tried to edit that out of the screen output. Hopefully this gives you what you need.
cmake-error.txt

[bartlettroscoe]

@glhenni, perfect. What we see is:

-- TPL_LAPACK_LIBRARIES='/projects/cde/v2/spack/opt/spack/linux-rhel7-x86_64/intel-19.1.2/intel-mkl-2020.3.279-4uk7qgxqeji6bsj4gflxhbspn3v2pm3m/mkl/lib/intel64_lin/libmkl_gf_lp64.so;/projects/cde/v2/spack/opt/spack/linux-rhel7-x86_64/intel-19.1.2/intel-mkl-2020.3.279-4uk7qgxqeji6bsj4gflxhbspn3v2pm3m/mkl/lib/intel64_lin/libmkl_gnu_thread.so;/projects/cde/v2/spack/opt/spack/linux-rhel7-x86_64/intel-19.1.2/intel-mkl-2020.3.279-4uk7qgxqeji6bsj4gflxhbspn3v2pm3m/mkl/lib/intel64_lin/libmkl_core.so;/projects/cde/v2/spack/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.2.0-cgjuig7mygdnho2c6pqgjagk6qfkz3w6/lib64/libgomp.so;-lm;-ldl;-lm;-ldl'

which shows the duplicate libs -lm;-ldl;-lm;-ldl

which is immediately followed by the CMake Error reported above.

I will create a unit test that exposes the problem and write code to fix this and post a new PR and merge it.

NOTE: Don't bother changing your configure process. Adding duplicate libs should be harmless (and I should have considered that possibility).

[bartlettroscoe]

@glhenni, I just merged PR #428 which should fix the error reported in this Issue. Please pull TriBITS 'master' and know me know if this solves the problem or not.

[glhenni]

Pretty sure we're up and running again @bartlettroscoe. I manually kicked off a couple of our nightly tests that failed last night and they've gotten through the configure and are now compiling. Well past where the failed last night. I won't know 100% until all our testing runs tonight, but I'm willing to be at a 99% level that the problem is gone. Thank you!

[glhenni]

I have ONE platform for ONE test that is still failing. The error is similar but not identical. I thought maybe I had a pointer to an old TriBITS clone but no, as far as I can tell it's a fresh clone. There's a whole sequence of errors like the one below. I'll also attach the full cmake output.

CMake Error at /scratch/charonops/jenkins/workspace/CEE_Build_Farm_CDE-GNU-opt/TEST.OPT/cmake_packages/HDF5/HDF5Config.cmake:7 (add_library):
  add_library cannot create imported target "HDF5::m" because another target
  with the same name already exists.
Call Stack (most recent call first):
  /scratch/charonops/jenkins/workspace/CEE_Build_Farm_CDE-GNU-opt/TriBITS/tribits/core/package_arch/TribitsTplFindIncludeDirsAndLibraries.cmake:686 (include)
  /scratch/charonops/jenkins/workspace/CEE_Build_Farm_CDE-GNU-opt/TriBITS/tribits/common_tpls/FindTPLHDF5.cmake:71 (tribits_tpl_find_include_dirs_and_libraries)
  /scratch/charonops/jenkins/workspace/CEE_Build_Farm_CDE-GNU-opt/TriBITS/tribits/core/package_arch/TribitsProcessEnabledTpl.cmake:106 (include)
  /scratch/charonops/jenkins/workspace/CEE_Build_Farm_CDE-GNU-opt/TriBITS/tribits/core/package_arch/TribitsGlobalMacros.cmake:1565 (tribits_process_enabled_tpl)
  /scratch/charonops/jenkins/workspace/CEE_Build_Farm_CDE-GNU-opt/TriBITS/tribits/core/package_arch/TribitsProjectImpl.cmake:196 (tribits_process_enabled_tpls)
  /scratch/charonops/jenkins/workspace/CEE_Build_Farm_CDE-GNU-opt/TriBITS/tribits/core/package_arch/TribitsProject.cmake:93 (tribits_project_impl)
  CMakeLists.txt:61 (TRIBITS_PROJECT)

cmake-trib-error.txt

[bartlettroscoe]

From the output cmake-trib-error.txt file above we see:

-- TPL_HDF5_LIBRARIES='/projects/cde/v2/spack/opt/spack/linux-rhel7-x86_64/gcc-7.2.0/hdf5-1.10.6-egjaen7nygsj5i5ovntklz4jykdgshy3/lib/libhdf5.so;/projects/cde/v2/spack/opt/spack/linux-rhel7-x86_64/gcc-7.2.0/anaconda3-4.6.14-nturlzzhxcigcxs37fp4lnv7daifalpa/lib/libz.so;/usr/lib64/libdl.so;/usr/lib64/libm.so'

there is no duplicate m library there.

Otherwise, I think I know what the problem must be. From the configure output we see:

Processing enabled TPL: HDF5 (enabled explicitly, disable with -DTPL_ENABLE_HDF5=OFF)
-- Using find_package(HDF5 ...) ...
-- Found HDF5: /projects/cde/v2/spack/opt/spack/linux-rhel7-x86_64/gcc-7.2.0/hdf5-1.10.6-egjaen7nygsj5i5ovntklz4jykdgshy3/lib/libhdf5.so;/projects/cde/v2/spack/opt/spack/linux-rhel7-x86_64/gcc-7.2.0/anaconda3-4.6.14-nturlzzhxcigcxs37fp4lnv7daifalpa/lib/libz.so;/usr/lib64/libdl.so;/usr/lib64/libm.so (found version "1.10.6") found components: C 

My guess is that this may be finding the HDF5Config.cmake module that was sitting there from a previous configure and this already defines the targets when find_package(HDF5 ...) is called. I think that is the case because we also see the error:

CMake Error at /scratch/charonops/jenkins/workspace/CEE_Build_Farm_CDE-GNU-opt/TEST.OPT/cmake_packages/HDF5/HDF5Config.cmake:29 (add_library):
  add_library cannot create imported target "HDF5::all_libs" because another
  target with the same name already exists.
Call Stack (most recent call first):
  /scratch/charonops/jenkins/workspace/CEE_Build_Farm_CDE-GNU-opt/TriBITS/tribits/core/package_arch/TribitsTplFindIncludeDirsAndLibraries.cmake:686 (include)
  /scratch/charonops/jenkins/workspace/CEE_Build_Farm_CDE-GNU-opt/TriBITS/tribits/common_tpls/FindTPLHDF5.cmake:71 (tribits_tpl_find_include_dirs_and_libraries)
  /scratch/charonops/jenkins/workspace/CEE_Build_Farm_CDE-GNU-opt/TriBITS/tribits/core/package_arch/TribitsProcessEnabledTpl.cmake:106 (include)
  /scratch/charonops/jenkins/workspace/CEE_Build_Farm_CDE-GNU-opt/TriBITS/tribits/core/package_arch/TribitsGlobalMacros.cmake:1565 (tribits_process_enabled_tpl)
  /scratch/charonops/jenkins/workspace/CEE_Build_Farm_CDE-GNU-opt/TriBITS/tribits/core/package_arch/TribitsProjectImpl.cmake:196 (tribits_process_enabled_tpls)
  /scratch/charonops/jenkins/workspace/CEE_Build_Farm_CDE-GNU-opt/TriBITS/tribits/core/package_arch/TribitsProject.cmake:93 (tribits_project_impl)
  CMakeLists.txt:61 (TRIBITS_PROJECT)

When a TriBITS FindTPL<tplName>.cmake module calls find_package(<tplName>), it can't be finding <tplName>Config.cmake under <buildDir>/cmake_packages/<tplName>/. That is not what you want.

Looking at the configure output, I can see this configure is running on a CEE build machine:

-- tcad-charon_HOSTNAME='cee-build031'

@glhenni, can you please point me to your configure script for this build offline so that I can reproduce this myself?

BTW, as a workaround, my guess is that if you delete the <buildDir>/cmake_packages/ directory before each reconfigure, then this problem will go away. (That will be fast and not slow down your rebuilds at all.) (But I want to reproduce this myself and understand exactly what is happening so that I can resolve this in a robust way going forward.)

[bartlettroscoe]

my guess is that if you delete the <buildDir>/cmake_packages/ directory before each reconfigure, then this problem will go away

@glhenni, let me go ahead and put in that workaround directly in TriBITS and see if that fixes your problem for now. Stay tuned.

[glhenni]

It's very possible that a tribits change exposed some problem with our config. It's SO odd that it's only happening on this platform for this test. There is another almost identical platform and I don't see this error.

As an FYI, I completely wiped out the build directory before starting the build above. There should have been no artifacts left over.

[bartlettroscoe]

As an FYI, I completely wiped out the build directory before starting the build above. There should have been no artifacts left over.

@glhenni, in that case, can you please provide your configure script for this build so I can reproduce this?

[glhenni]

I'm not sure what all you'll need Ross. I use a python script to generate a cmake invocation. Below is the cmake command it invokes. What else do you require?

/projects/cde/v2/spack/opt/spack/linux-rhel7-x86_64/gcc-7.2.0/cmake-3.19.2-ygawmgvyrwfhelyp3r6knajkdli3rxo4/bin/cmake -D tcad-charon_ENABLE_PyTrilinos:BOOL=OFF -D tcad-charon_TEST_CATEGORIES:STRING="HEAVY" -D tcad-charon_VERBOSE_CONFIGURE:BOOL=OFF -D tcad-charon_ENABLE_Kokkos:BOOL=ON -D tcad-charon_ENABLE_KokkosAlgorithms:BOOL=ON -D tcad-charon_ENABLE_KokkosCore:BOOL=ON -D tcad-charon_ENABLE_INSTALL_CMAKE_CONFIG_FILES:BOOL=ON -D tcad-charon_ENABLE_CHECKED_STL:BOOL=OFF -D tcad-charon_ENABLE_TEUCHOS_TIME_MONITOR:BOOL=ON -D tcad-charon_ENABLE_MueLu:BOOL=ON -D tcad-charon_ENABLE_Stokhos:BOOL=OFF -D tcad-charon_ENABLE_ROL:BOOL=OFF -D tcad-charon_ENABLE_SECONDARY_TESTED_CODE:BOOL=OFF -D tcad-charon_ENABLE_ALL_PACKAGES:BOOL=OFF -D tcad-charon_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=OFF -D tcad-charon_ENABLE_SEACASEx1ex2v2:BOOL=OFF -D tcad-charon_ENABLE_SEACASEx2ex1v2:BOOL=OFF -D tcad-charon_ENABLE_SEACASExomatlab:BOOL=OFF -D tcad-charon_ENABLE_Sacado:BOOL=ON -D tcad-charon_ENABLE_AztecOO:BOOL=ON -D tcad-charon_ENABLE_Ifpack:BOOL=ON -D tcad-charon_ENABLE_Amesos2:BOOL=ON -D tcad-charon_ENABLE_Ifpack2:BOOL=ON -D tcad-charon_ENABLE_Tempus:BOOL=ON -D tcad-charon_ENABLE_Zoltan2:BOOL=ON -D tcad-charon_ENABLE_Xpetra:BOOL=ON -D tcad-charon_ENABLE_EXPLICIT_INSTANTIATION:BOOL=ON -D tcad-charon_ENABLE_STKUnit_test_utils:BOOL=OFF -D tcad-charon_ENABLE_STKSearch:BOOL=OFF -D tcad-charon_ENABLE_STKSearchUtil:BOOL=OFF -D tcad-charon_ENABLE_STKUnit_tests:BOOL=OFF -D tcad-charon_ENABLE_STKDoc_tests:BOOL=OFF -D tcad-charon_ENABLE_STKExprEval:BOOL=OFF -D tcad-charon_ENABLE_TESTS:BOOL=OFF -D tcad-charon_ENABLE_EXAMPLES:BOOL=OFF -D tcad-charon_ENABLE_DAKOTA_DRIVERS:BOOL=ON -D tcad-charon_ENABLE_PYMESH:BOOL=ON -D AztecOO_ENABLE_TEUCHOS_TIME_MONITOR:BOOL=ON -D Intrepid2_ENABLE_DEBUG_INF_CHECK:BOOL=OFF -D SEACASExodus_ENABLE_MPI:BOOL=OFF -D TPL_ENABLE_MPI:BOOL=ON -D TPL_ENABLE_Matio:BOOL=OFF -D TPL_ENABLE_X11:BOOL=OFF -D tcad-charon_ENABLE_Charon:BOOL=ON -D Phalanx_INDEX_SIZE_TYPE:STRING="INT" -D Phalanx_SHOW_DEPRECATED_WARNINGS:BOOL=OFF -D Kokkos_ENABLE_SERIAL:BOOL=ON -D Kokkos_ENABLE_OPENMP:BOOL=OFF -D Kokkos_ENABLE_PTHREAD:BOOL=OFF -D Kokkos_ENABLE_CUDA:BOOL=OFF -D Charon_ENABLE_TESTS:BOOL=ON -D Charon_ENABLE_EXAMPLES:BOOL=ON -D Charon_ENABLE_DEBUG:BOOL=ON -D Charon_ENABLE_OUO_TESTS:BOOL=ON -D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF -D CMAKE_SKIP_RULE_DEPENDENCY:BOOL=ON -D CMAKE_CXX_STANDARD:STRING=14 -D TPL_ENABLE_HDF5:BOOL=OFF -D TPL_ENABLE_Boost:BOOL=ON -D TPL_ENABLE_BoostLib:BOOL=ON -D TPL_ENABLE_Netcdf:BOOL=ON -D Panzer_ENABLE_TESTS:BOOL=OFF -D EpetraExt_ENABLE_HDF5:BOOL=OFF -D GIT_EXECUTABLE:FILEPATH="$(which git)" -D BUILD_SHARED_LIBS:BOOL=ON -D TPL_ENABLE_HDF5:BOOL=ON -D CMAKE_INSTALL_PREFIX:PATH=/projects/charon/install/BOD/cee.cde.gnu.opt.2 -D CMAKE_VERBOSE_MAKEFILE:BOOL=OFF -D CMAKE_INSTALL_PREFIX:PATH=/projects/charon/install/BOD/cee.cde.gnu.opt.2 -D CMAKE_BUILD_TYPE:STRING=RELEASE  /scratch/charonops/jenkins/workspace/CEE_Build_Farm_CDE-GNU-opt/tcad-charon

[glhenni]

Here's the CMakeCache.txt if that helps...
CMakeCache.txt

[bartlettroscoe]

BTW, it is also possible that find_package(HDF5) is finding HDF5Config.cmake from some other install of Trilinos that happens to get found through some path set through CMAKE_PREFIX_PATH or some other place found by find_package():

• https://cmake.org/cmake/help/v3.17/command/find_package.html#search-procedure

The questions is, how was HDF5 found in this build before this update of TriBITS? Do you have the Jenkins log showing that?

[glhenni]

I certainly have the console output of it working before the tribits change, but I don't see any obvious difference. Of course I can always checkout a version of tribits used on that day and get more explicit output if that would help. Attached is the console log...
cmake-pre-trib-change.txt

[glhenni]

Here's a great data point that's not confusing at all...if I invoke cmake as above in an empty build directory it fails with the aforementioned errors about HDF5::*. If I rerun cmake in the now populated directory in exactly the same way, with exactly the same arguments, literally just using bash command line history to give me the same cmake command two times in a row, the second time the errors disappear and cmake successfully gives me the makefiles and I can build charon. 😠

[bartlettroscoe]

Here's a great data point that's not confusing at all

@glhenni, that makes no sense :-)

Let me try running a modified version of your configure command above and see happens for me.

This may also be impacted by your env. Can you capture your bash env and email it to me offline? (Don't want to put that in a GitHub Issue.)

[glhenni]

I agree, it makes no sense and yet I can give you output of cmake --trace --verbose ... the first time that bombs with the errors above and the second time, with no commands between, that works.

BTW, I don't see it picking up anything from old builds or weird locations in those traces.

If you have access to /projects/charon/ForRoss I put output from the cmake traces I references above and the environment in files within that directory. You should be able to access it from any general access system, cee-buildXXX, cee-computeXXX, ascicXXX, for example.

[glhenni]

Found it! You were right @bartlettroscoe. I looked more closely at the trace output and it was indeed finding a HDF5Config.cmake in the CMAKE_INSTALL_PREFIX directory whose contents were the result of a make install from a build a few months ago. Was this because HDF5Config.cmake changed?

Looks like I'll have to clean out that directory prior building in the future, which is probably not a bad idea anyway.

[glhenni]

I don't know why this just bit us. Seems like something in cmake or tribits changed that uses CMAKE_INSTALL_PREFIX to search for components. That's not ideal because now I have to clean out that directory BEFORE a build. Up until this point my sequence was:

1. Build the code via ctest/cmake/make
2. If successful clean out the installation directory
3. Perform a make install of the build in 1. into the clean installation directory

Now I have to clean the installation directory before a build and if the build fails then that directory will remain empty until the build is fixed, instead of containing an install from a previously successful build. It's not a huge issue for us because we already have a mechanism in place for other reasons to insure a build is available for analysts, but still, like I said, not ideal.

[glhenni]

-D CMAKE_FIND_NO_INSTALL_PREFIX=TRUE

I think this did fix our issue on all instances @bartlettroscoe. At least I don't see any failures due to this problem from our regression tests last night.

[bartlettroscoe]

@glhenni, FYI, I am going to post a PR that will fix address this issue without having to set -D CMAKE_FIND_NO_INSTALL_PREFIX=TRUE or make any other changes so this does not impact anyone else (i.e. when trilinos/Trilinos#9894 gets merged).

[bartlettroscoe]

@glhenni, FYI, PR #431 (which I just merged to 'master') should address this problem going forward without having to set -D CMAKE_FIND_NO_INSTALL_PREFIX=TRUE or change any configuration settings.

[bartlettroscoe]

With the merge of PR #431 and the tests associated with that, I fully expect the problems highlighted in this Issue should be resolved for all customers (without having to set -D CMAKE_FIND_NO_INSTALL_PREFIX=TRUE).

Therefore, I will close this as complete. If a new issue comes up, please post a new GitHub issue.