CUDA link issue with incremental build #852

esseivaju · 2023-07-12T23:54:17Z

#848 seems to break incremental builds. Building celeritas from scratch will compile and link everything without issues. Changing a cuda-related file - e.g. ActionLauncher.device.hh, even just adding a blank line - then trying to recompile and link will result in the linker error below. This wasn't an issue in 9e1d185.

@sethrj @drbenmorgan @pcanal

[36/43] Linking CXX executable bin/celer-sim
FAILED: bin/celer-sim 
: && /cvmfs/sft.cern.ch/lcg/releases/gcc/11.3.0-ad0f5/x86_64-centos7/bin/g++ -Wall -Wextra -Wno-psabi -pedantic -pedantic-errors -O3 -DNDEBUG -march=cascadelake -mtune=cascadelake  app/celer-sim/CMakeFiles/celer-sim.dir/celer-sim.cc.o app/celer-sim/CMakeFiles/celer-sim.dir/Runner.cc.o app/celer-sim/CMakeFiles/celer-sim.dir/RunnerOutput.cc.o app/celer-sim/CMakeFiles/celer-sim.dir/Transporter.cc.o app/celer-sim/CMakeFiles/celer-sim.dir/RunnerInputIO.json.cc.o app/celer-sim/CMakeFiles/celer-sim.dir/RootOutput.cc.o -o bin/celer-sim  -Wl,-rpath,/home/esseivaj/bld4/devel/spack/var/spack/environments/celeritas/.spack-env/view/lib/root:/home/esseivaj/bld4/devel/celeritas/build-ndebug/lib64:/home/esseivaj/bld4/devel/spack/var/spack/environments/celeritas/.spack-env/view/lib64::  /home/esseivaj/bld4/devel/spack/var/spack/environments/celeritas/.spack-env/view/lib/root/libTree.so.6.28.04  lib64/libceleritas_final.so  /cvmfs/sft.cern.ch/lcg/releases/gcc/11.3.0-ad0f5/x86_64-centos7/lib64/libgomp.so  -lpthread  /home/esseivaj/bld4/devel/spack/var/spack/environments/celeritas/.spack-env/view/lib/root/libNet.so.6.28.04  /home/esseivaj/bld4/devel/spack/var/spack/environments/celeritas/.spack-env/view/lib/root/libRIO.so.6.28.04  /home/esseivaj/bld4/devel/spack/var/spack/environments/celeritas/.spack-env/view/lib/root/libThread.so.6.28.04  -lpthread  /home/esseivaj/bld4/devel/spack/var/spack/environments/celeritas/.spack-env/view/lib/root/libMathCore.so.6.28.04  /home/esseivaj/bld4/devel/spack/var/spack/environments/celeritas/.spack-env/view/lib/root/libCore.so.6.28.04  lib64/libceleritas.so  /home/esseivaj/bld4/devel/spack/var/spack/environments/celeritas/.spack-env/view/lib64/libvecgeomcuda.so  lib64/libcorecel.so  /home/esseivaj/bld4/devel/spack/var/spack/environments/celeritas/.spack-env/view/lib64/libvecgeom.so  -lrt  -lpthread  -Wl,-rpath-link,/home/esseivaj/bld4/devel/spack/var/spack/environments/celeritas/.spack-env/view/lib64 && :
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_f9193967_17_BoundaryAction_cu_7847f3ed_23182'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_78cc259c_24_MuBremsstrahlungModel_cu_7847f3ed_23116'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_d62f876c_31_AlongStepRZMapFieldMscAction_cu_7847f3ed_28614'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_74d9648b_19_AlongStepKernels_cu_7847f3ed_28441'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_30326de6_20_CombinedBremModel_cu_7847f3ed_23003'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_7d3cad3b_28_ExtendFromPrimariesAction_cu_7847f3ed_28701'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_0b145eb6_16_RayleighModel_cu_7847f3ed_23125'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_853ff234_24_RelativisticBremModel_cu_7847f3ed_28562'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_c1895311_20_MollerBhabhaModel_cu_7847f3ed_28439'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_14eea905_30_ExtendFromSecondariesAction_cu_7847f3ed_28705'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_9ea30712_28_AlongStepUniformMscAction_cu_7847f3ed_23269'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_9e1fdb60_19_ActionDiagnostic_cu_7847f3ed_23457'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_b35ed0ed_17_StepDiagnostic_cu_7847f3ed_28727'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_85e8a02e_15_EPlusGGModel_cu_7847f3ed_23009'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_30326de6_20_CombinedBremModel_cu_7847f3ed_28443'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_78cc259c_24_MuBremsstrahlungModel_cu_7847f3ed_28483'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_facc871a_21_SeltzerBergerModel_cu_7847f3ed_28570'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_b42bb3d4_25_InitializeTracksAction_cu_7847f3ed_23442'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_d62f876c_31_AlongStepRZMapFieldMscAction_cu_7847f3ed_23297'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_f9193967_17_BoundaryAction_cu_7847f3ed_28587'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_f2107e2b_20_BetheHeitlerModel_cu_7847f3ed_22994'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_b35ed0ed_17_StepDiagnostic_cu_7847f3ed_23499'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_4e3221b4_19_StepGatherAction_cu_7847f3ed_28772'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_4e3221b4_19_StepGatherAction_cu_7847f3ed_23533'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_50663ced_25_AlongStepNeutralAction_cu_7847f3ed_23241'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_b7556077_23_DiscreteSelectAction_cu_7847f3ed_28617'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_0b145eb6_16_RayleighModel_cu_7847f3ed_28555'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_9ea30712_28_AlongStepUniformMscAction_cu_7847f3ed_28592'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_c1895311_20_MollerBhabhaModel_cu_7847f3ed_23063'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_6907ffe6_16_PreStepAction_cu_7847f3ed_23318'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_5bdc085a_20_KleinNishinaModel_cu_7847f3ed_23024'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_74d9648b_19_AlongStepKernels_cu_7847f3ed_22925'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_853ff234_24_RelativisticBremModel_cu_7847f3ed_23144'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_7d3cad3b_28_ExtendFromPrimariesAction_cu_7847f3ed_23409'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_14eea905_30_ExtendFromSecondariesAction_cu_7847f3ed_23437'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_5bdc085a_20_KleinNishinaModel_cu_7847f3ed_28440'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_6907ffe6_16_PreStepAction_cu_7847f3ed_28702'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_b7556077_23_DiscreteSelectAction_cu_7847f3ed_23298'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_9e1fdb60_19_ActionDiagnostic_cu_7847f3ed_28709'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_8aacd802_19_LivermorePEModel_cu_7847f3ed_23059'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_85e8a02e_15_EPlusGGModel_cu_7847f3ed_28437'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_f2107e2b_20_BetheHeitlerModel_cu_7847f3ed_28438'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_8aacd802_19_LivermorePEModel_cu_7847f3ed_28442'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_b42bb3d4_25_InitializeTracksAction_cu_7847f3ed_28710'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas_final.so: undefined reference to `__fatbinwrap_facc871a_21_SeltzerBergerModel_cu_7847f3ed_23181'
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.37-355ed/x86_64-centos7/bin/ld: lib64/libceleritas.so: undefined reference to `__cudaRegisterLinkedBinary_50663ced_25_AlongStepNeutralAction_cu_7847f3ed_28588'

The text was updated successfully, but these errors were encountered:

drbenmorgan · 2023-07-13T17:26:07Z

Hi @esseivaju, I'm not able to reproduce this with the default preset build using the current tip of develop (b427fde) and a local spack env with gcc 11.3 as:

Build from scratch cmake --preset=default ., run tests, everything fine
Add a new empty line in src/celeritas/global/ActionLauncher.device.hh
Run ninja again and the incremental build completes without error.

Which commit are you building, and which CUDA version (I'm on 11.7.1)?

sethrj · 2023-07-13T17:34:21Z

Could it be #847 introducing

celeritas/CMakeLists.txt

Line 177 in b427fde

celeritas_set_default(CMAKE_LINK_DEPENDS_NO_SHARED ON)

? If you run with -D CMAKE_LINK_DEPENDS_NO_SHARED=OFF is the problem resolved?

esseivaju · 2023-07-13T18:00:08Z

I am building b427fde with gcc11.3 / cuda 12.1. Setting -D CMAKE_LINK_DEPENDS_NO_SHARED=OFF does seem to fix the issue.

pcanal · 2023-07-13T18:07:19Z

I also see similar problem in an updated build. Let me investigate.

pcanal · 2023-07-13T18:24:28Z

: && /srv/software/gnu10/10.2.0/bin/g++ -Wall -Wextra -pedantic -g  test/CMakeFiles/testdetail_TestMacros.dir/TestMacros.test.cc.o -o test/testdetail_TestMacros  -Wl,-rpath,/nashome/p/pcanal/geant/gcc10/builds/scalar-cuda12/celeritas/lib64:/nashome/p/pcanal/geant/gcc10/install/opt/lib64  lib64/libtestcel_harness.so  lib64/libcorecel_final.so  /nashome/p/pcanal/geant/gcc10/install/opt/lib64/libgtest.so.1.10.0  -pthread  lib64/libcorecel.so && :
lib64/libcorecel.so: undefined reference to `__cudaRegisterLinkedBinary_4e0d1928_9_Filler_cu_314c4826_29355'
lib64/libcorecel_final.so: undefined reference to `__fatbinwrap_4e0d1928_9_Filler_cu_438f1dcd_109866'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
pcanal@wc:celeritas $ touch ~/geant/sources/celeritas/src/corecel/data/detail/Filler.cu
pcanal@wc:celeritas $ ninja -j 1
[4/365] Linking CXX executable test/testdetail_TestMacros
FAILED: test/testdetail_TestMacros
: && /srv/software/gnu10/10.2.0/bin/g++ -Wall -Wextra -pedantic -g  test/CMakeFiles/testdetail_TestMacros.dir/TestMacros.test.cc.o -o test/testdetail_TestMacros  -Wl,-rpath,/nashome/p/pcanal/geant/gcc10/builds/scalar-cuda12/celeritas/lib64:/nashome/p/pcanal/geant/gcc10/install/opt/lib64  lib64/libtestcel_harness.so  lib64/libcorecel_final.so  /nashome/p/pcanal/geant/gcc10/install/opt/lib64/libgtest.so.1.10.0  -pthread  lib64/libcorecel.so && :
lib64/libcorecel.so: undefined reference to `__cudaRegisterLinkedBinary_4e0d1928_9_Filler_cu_ce41cbe7_29634'
lib64/libcorecel_final.so: undefined reference to `__fatbinwrap_4e0d1928_9_Filler_cu_438f1dcd_109866'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

pcanal · 2023-07-13T18:26:02Z

I.e. for each compilation of a RDC object file, it generates a 'unique' identifier and thus requires the running of the nvlink that depends on that .cu.o file.

The symbols:

__cudaRegisterLinkedBinary_4e0d1928_9_Filler_cu_314c4826_29355
__cudaRegisterLinkedBinary_4e0d1928_9_Filler_cu_ce41cbe7_29634

pcanal · 2023-07-13T18:28:51Z

And indeed reverting #847 (completely for now) solves the problem (and of course doing ninja clean is a (lame) work-around)

sethrj · 2023-07-13T18:50:13Z

OK, maybe we need to just set CMAKE_LINK_DEPENDS_NO_SHARED conditionally on RDC (aka vecgeom + cuda). I'm surprised because VecGeom has this option set as well.

pcanal · 2023-07-13T18:51:59Z

I am building b427fde with gcc11.3 / cuda 12.1. Setting -D CMAKE_LINK_DEPENDS_NO_SHARED=OFF does seem to fix the issue.

That's odd. It works for me (with and without the space after the D).

pcanal · 2023-07-13T19:09:50Z

I'm surprised because VecGeom has this option set as well.

I see it in the source (and even I see that line executed) but I don't see it in the CMakeCache.txt (i.e. is it somehow not activated?)

sethrj · 2023-07-13T19:32:23Z

It's not being set as a cache variable, just as a directory scope one (so it will be "seen" by the targets but not saved from one configuration to the next).

pcanal · 2023-07-13T20:14:35Z

Ok. It is still 'uneffective' (it leads to the proper rebuilds ... I am checking why)

pcanal · 2023-07-13T21:53:43Z

And this is because there is a (hard) dependency from the result of nvlink and the static libraries. In Celeritas the dependency on the static library is only soft/order while the shared library dependency is 'hard' when it works and soft when it fails. This might be because we introduce the static library dependency as:

          target_link_options(${_target_final}
            PRIVATE
            $<DEVICE_LINK:$<TARGET_FILE:${_libstatic}>>
          )

esseivaju added the bug Something isn't working label Jul 12, 2023

pcanal mentioned this issue Jul 14, 2023

Fix linking errors with CUDA+VecGeom caused by #847 #856

Merged

pcanal self-assigned this Jul 14, 2023

sethrj linked a pull request Jul 15, 2023 that will close this issue

Fix linking errors with CUDA+VecGeom caused by #847 #856

Merged

sethrj closed this as completed in #856 Jul 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CUDA link issue with incremental build #852

CUDA link issue with incremental build #852

esseivaju commented Jul 12, 2023 •

edited

Loading

drbenmorgan commented Jul 13, 2023

sethrj commented Jul 13, 2023

esseivaju commented Jul 13, 2023

pcanal commented Jul 13, 2023

pcanal commented Jul 13, 2023

pcanal commented Jul 13, 2023

pcanal commented Jul 13, 2023 •

edited

Loading

sethrj commented Jul 13, 2023

pcanal commented Jul 13, 2023

pcanal commented Jul 13, 2023 •

edited

Loading

sethrj commented Jul 13, 2023

pcanal commented Jul 13, 2023

pcanal commented Jul 13, 2023

CUDA link issue with incremental build #852

CUDA link issue with incremental build #852

Comments

esseivaju commented Jul 12, 2023 • edited Loading

drbenmorgan commented Jul 13, 2023

sethrj commented Jul 13, 2023

esseivaju commented Jul 13, 2023

pcanal commented Jul 13, 2023

pcanal commented Jul 13, 2023

pcanal commented Jul 13, 2023

pcanal commented Jul 13, 2023 • edited Loading

sethrj commented Jul 13, 2023

pcanal commented Jul 13, 2023

pcanal commented Jul 13, 2023 • edited Loading

sethrj commented Jul 13, 2023

pcanal commented Jul 13, 2023

pcanal commented Jul 13, 2023

esseivaju commented Jul 12, 2023 •

edited

Loading

pcanal commented Jul 13, 2023 •

edited

Loading

pcanal commented Jul 13, 2023 •

edited

Loading