Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

k4simdelphes: standalone exes hang when building using the cvmfs installation #381

Closed
vvolkl opened this issue Jun 2, 2022 · 5 comments
Closed
Labels
bug Something isn't working

Comments

@vvolkl
Copy link
Contributor

vvolkl commented Jun 2, 2022

When using the /cvmfs/sw.hsf.org/key4hep/setup.sh or /cvmfs/sw-nightlies.hsf.org/key4hep/setup.sh releases to build k4simdelphes, the resulting standalone executables get stuck, i.e. they don't show any output and do not terminate.

@vvolkl vvolkl added enhancement New feature or request bug Something isn't working and removed enhancement New feature or request labels Jun 2, 2022
@andresailer
Copy link
Contributor

What does strace tell?

@vvolkl
Copy link
Contributor Author

vvolkl commented Jun 16, 2022

That it's hanging during a ROOT dictionary lookup:

strace ./standalone/DelphesPythia8_EDM4HEP /home/vavolkl/k4simdelphes/build/tests/test_delphes_card_IDEA.tcl /home/vavolkl/k4simdelphes/examples/edm4hep_output_config.tcl /home/vavolkl/k4simdelphes/build/tests/test_ee_Z_bbbar_ecm91GeV.cmd pythia_converter_output_ee_Z_bbbar.root
stat("/cvmfs/sw-nightlies.hsf.org/spackages5/garfieldpp/4.0/x86_64-centos7-gcc11.2.0-opt/b2msl/lib", 0x7ffd805793e0) = -1 ENOENT (No such file or directory)
stat("/cvmfs/sw-nightlies.hsf.org/spackages5/dd4hep/commit.c4e56a5eb7ac8b2406e2c3c22e438087beff268d/x86_64-centos7-gcc11.2.0-opt/2nqu5/lib64", 0x7ffd805793e0) = -1 ENOENT (No such file or directory)
stat("/cvmfs/sw-nightlies.hsf.org/spackages5/dd4hep/commit.c4e56a5eb7ac8b2406e2c3c22e438087beff268d/x86_64-centos7-gcc11.2.0-opt/2nqu5/lib", {st_mode=S_IFDIR|S_ISGID|0755, st_size=4096, ...}) = 0
openat(AT_FDCWD, "/cvmfs/sw-nightlies.hsf.org/spackages5/dd4hep/commit.c4e56a5eb7ac8b2406e2c3c22e438087beff268d/x86_64-centos7-gcc11.2.0-opt/2nqu5/lib", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
getdents(3, /* 85 entries */, 32768)    = 3632
getdents(3, /* 0 entries */, 32768)     = 0
close(3)                                = 0
stat("/cvmfs/sw-nightlies.hsf.org/spackages5/edm4hep/commit.97b66f9a8fe3ca1667c9b77717840f0c471ff96a/x86_64-centos7-gcc11.2.0-opt/o5oh7/lib64", {st_mode=S_IFDIR|S_ISGID|0755, st_size=243, ...}) = 0
openat(AT_FDCWD, "/cvmfs/sw-nightlies.hsf.org/spackages5/edm4hep/commit.97b66f9a8fe3ca1667c9b77717840f0c471ff96a/x86_64-centos7-gcc11.2.0-opt/o5oh7/lib64", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
getdents(3, /* 11 entries */, 32768)    = 440
access("/cvmfs/sw-nightlies.hsf.org/spackages5/edm4hep/commit.97b66f9a8fe3ca1667c9b77717840f0c471ff96a/x86_64-centos7-gcc11.2.0-opt/o5oh7/lib64/edm4hepDictDict.rootmap", R_OK) = 0
open("/cvmfs/sw-nightlies.hsf.org/spackages5/edm4hep/commit.97b66f9a8fe3ca1667c9b77717840f0c471ff96a/x86_64-centos7-gcc11.2.0-opt/o5oh7/lib64/edm4hepDictDict.rootmap", O_RDONLY) = 4
read(4, "{ decls }\nnamespace edm4hep {  }"..., 8191) = 2970
futex(0x10654d0, FUTEX_WAIT_PRIVATE, 2, NULL

GDB also hints at a possible deadlock related to the the error message warning about the duplicate classes in edm4hep and dual-readout:

/home/vavolkl/k4simdelphes/build (main)$ gdb ./standalone/DelphesPythia8_EDM4HEP /home/vavolkl/k4simdelphes/build/tests/test_delphes_card_IDEA.tcl /home/vavolkl/k4simdelphes/examples/edm4hep_output_config.tcl /home/vavolkl/k4simdelphes/build/tests/test_ee_Z_bbbar_ecm91GeV.cmd pythia_converter_output_ee_Z_bbbar.root
Excess command line arguments ignored. (/home/vavolkl/k4simdelphes/examples/edm4hep_output_config.tcl ...)
GNU gdb (GDB) 11.1
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./standalone/DelphesPythia8_EDM4HEP...
"/home/vavolkl/k4simdelphes/build/tests/test_delphes_card_IDEA.tcl" is not a core dump: file format not recognized
(gdb) run
Starting program: /home/vavolkl/k4simdelphes/build/standalone/DelphesPythia8_EDM4HEP 
warning: File "/cvmfs/sw-nightlies.hsf.org/spackages5/root/6.26.02/x86_64-centos7-gcc11.2.0-opt/62nsk/lib/libCore.so.6.26.02-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/cvmfs/sw.hsf.org/spackages5/python/3.9.10/x86_64-centos7-gcc11.2.0-opt/7j5vq/bin/python3.9-gdb.py".
To enable execution of this file add
	add-auto-load-safe-path /cvmfs/sw-nightlies.hsf.org/spackages5/root/6.26.02/x86_64-centos7-gcc11.2.0-opt/62nsk/lib/libCore.so.6.26.02-gdb.py
line to your configuration file "/afs/cern.ch/user/v/vavolkl/.config/gdb/gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/afs/cern.ch/user/v/vavolkl/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
warning: File "/cvmfs/sw.hsf.org/contrib/gcc/11.2.0/x86_64-centos7-gcc8.3.0-opt/d3epy/lib64/libstdc++.so.6.0.29-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/cvmfs/sw.hsf.org/spackages5/python/3.9.10/x86_64-centos7-gcc11.2.0-opt/7j5vq/bin/python3.9-gdb.py".
[Detaching after fork from child process 30998]
[Detaching after fork from child process 31000]
[Detaching after fork from child process 31002]
[Detaching after fork from child process 31004]
[Detaching after fork from child process 31018]
[Detaching after fork from child process 31020]
^C
Program received signal SIGINT, Interrupt.
0x00007ffff55f454d in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007ffff55f454d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007ffff55efe9b in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007ffff55efd68 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007ffff59c8ca4 in __gthread_mutex_lock (__mutex=0x50e4d0)
    at /cvmfs/sw.hsf.org/contrib/gcc/11.2.0/x86_64-centos7-gcc8.3.0-opt/d3epy/include/c++/11.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:749
#4  std::mutex::lock (this=0x50e4d0)
    at /cvmfs/sw.hsf.org/contrib/gcc/11.2.0/x86_64-centos7-gcc8.3.0-opt/d3epy/include/c++/11.2.0/bits/std_mutex.h:100
#5  std::lock_guard<std::mutex>::lock_guard (__m=..., this=<synthetic pointer>)
    at /cvmfs/sw.hsf.org/contrib/gcc/11.2.0/x86_64-centos7-gcc8.3.0-opt/d3epy/include/c++/11.2.0/bits/std_mutex.h:229
#6  DebugPrint (fmt=0x7ffff5b4b3ec "%s\n")
    at /tmp/gitlab-runner/spack-stage/spack-stage-root-6.26.02-62nskdfxpmmntylsmftzmyxezn5jgrw4/spack-src/core/base/src/TErrorDefaultHandler.cxx:86
#7  0x00007ffff59c8e92 in DefaultErrorHandler (level=<optimized out>, 
    abort_bool=<optimized out>, location=<optimized out>, msg=<optimized out>)
    at /cvmfs/sw.hsf.org/contrib/gcc/11.2.0/x86_64-centos7-gcc8.3.0-opt/d3epy/include/c++/11.2.0/bits/basic_string.h:194
#8  0x00007ffff5a75c2b in ErrorHandler(Int_t, const char *, const char *, typedef __va_list_tag __va_list_tag *) (level=2000, location=<optimized out>, 
    fmt=<optimized out>, ap=<optimized out>)
    at /cvmfs/sw.hsf.org/contrib/gcc/11.2.0/x86_64-centos7-gcc8.3.0-opt/d3epy/include/c++/11.2.0/bits/basic_string.h:194
#9  0x00007ffff59db54c in TObject::Warning (this=this@entry=0x5283b0, 
    location=location@entry=0x7ffff03edfdc "ReadRootmapFile", 
    fmt=fmt@entry=0x7ffff03ec890 "%s %s found in %s is already in %s")
    at /tmp/gitlab-runner/spack-stage/spack-stage-root-6.26.02-62nskdfxpmmntylsmftzmyxezn5jgrw4/spack-src/core/base/src/TObject.cxx:883
#10 0x00007fffed44f3b3 in TCling::ReadRootmapFile (this=0x5283b0, 
    rootmapfile=<optimized out>, uniqueString=0x7ffffffdb330)
    at /cvmfs/sw.hsf.org/contrib/gcc/11.2.0/x86_64-centos7-gcc8.3.0-opt/d3epy/include/c++/11.2.0/bits/basic_string.h:194
#11 0x00007fffed479ce3 in TCling::LoadLibraryMap (this=0x5283b0, 
    rootmapfile=0x0)
    at /tmp/gitlab-runner/spack-stage/spack-stage-root-6.26.02-62nskdfxpmmntylsmftzmyxezn5jgrw4/spack-src/core/metacling/src/TCling.cxx:5702
#12 0x00007fffed47abef in TCling::Initialize (this=0x5283b0)
    at /tmp/gitlab-runner/spack-stage/spack-stage-root-6.26.02-62nskdfxpmmntylsmftzmyxezn5jgrw4/spack-src/core/metacling/src/TCling.cxx:1669
#13 TCling::Initialize (this=0x5283b0)
    at /tmp/gitlab-runner/spack-stage/spack-stage-root-6.26.02-62nskdfxpmmntylsmftzmyxezn5jgrw4/spack-src/core/metacling/src/TCling.cxx:1654
#14 0x00007ffff599ce1d in TROOT::InitInterpreter (
    this=0x7ffff5c3d780 <ROOT::Internal::GetROOT1()::alloc>)
    at /tmp/gitlab-runner/spack-stage/spack-stage-root-6.26.02-62nskdfxpmmntylsmftzmyxezn5jgrw4/spack-src/core/base/src/TROOT.cxx:2102

Indeed, removing dualreadout from the LD_LIBRARY_PATH fixes this issue. This is possibly a ROOT bug; I'll see if I can write a reproducer and report it.

@vvolkl
Copy link
Contributor Author

vvolkl commented Jul 20, 2022

The same thing can occur when generating libDDEvePlugins.components during a DD4hep build.

@vvolkl
Copy link
Contributor Author

vvolkl commented Dec 2, 2022

Fixed by adressing the ROOT dict warning in dual-readout. Should still be checked with the latest ROOT version and reported to ROOT.

@jmcarcell
Copy link
Contributor

This doesn't seem to happen anymore, at least with the nightlies

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants