Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segfault in HGCalCassette::getShift() in workflow 25234.911 #41376

Closed
makortel opened this issue Apr 19, 2023 · 9 comments
Closed

Segfault in HGCalCassette::getShift() in workflow 25234.911 #41376

makortel opened this issue Apr 19, 2023 · 9 comments
Labels
geometry-pending pending-signatures simulation-pending upgrade-pending

Comments

@makortel
Copy link
Contributor

Workflow 25234.911 step 1 crashed in CMSSW_13_1_NONLTO_X_2023-04-19-1100 (el8_amd64_gcc11)

Thread 10 (Thread 0x2b0b19200700 (LWP 793) "cmsRun"):
#2  0x00002b0abe79344c in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  0x00002b0abe795dcb in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00002b0ae093ea48 in HGCalCassette::getShift(int, int, int) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/lib/el8_amd64_gcc11/libGeometryHGCalCommonData.so
#6  0x00002b0ae094d32f in HGCalDDDConstants::assignCellTrap(float, float, float, int, bool) const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/lib/el8_amd64_gcc11/libGeometryHGCalCommonData.so
#7  0x00002b0af114153b in HGCalNumberingScheme::getUnitID(int, int, int, int, CLHEP::Hep3Vector const&, double&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#8  0x00002b0af113e680 in HGCScintSD::setDetUnitId(int, int, int, int, CLHEP::Hep3Vector&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#9  0x00002b0af1140117 in HGCScintSD::setDetUnitId(G4Step const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#10 0x00002b0af110c3d6 in CaloSD::ProcessHits(G4Step*, G4TouchableHistory*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#11 0x00002b0af200a903 in G4SteppingManager::Stepping() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#12 0x00002b0af19c48eb in G4TrackingManager::ProcessOneTrack(G4Track*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#13 0x00002b0af164ee19 in G4EventManager::DoProcessing(G4Event*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#14 0x00002b0af12b7ced in RunManagerMTWorker::produce(edm::Event const&, edm::EventSetup const&, RunManagerMT&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#15 0x00002b0af12cea82 in omt::ThreadHandoff::Functor<OscarMTProducer::produce(edm::Event&, edm::EventSetup const&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#16 0x00002b0af12c3d6a in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so

Thread 9 (Thread 0x2b0b18200700 (LWP 792) "cmsRun"):
#0  __tls_get_addr () at ../sysdeps/x86_64/tls_get_addr.S:29
#1  0x00002b0af1b7fafd in G4VPhysicalVolume::GetTranslation() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#2  0x00002b0af1b9c671 in G4VoxelNavigation::LevelLocate(G4NavigationHistory&, G4VPhysicalVolume const*, int, CLHEP::Hep3Vector const&, CLHEP::Hep3Vector const*, bool, CLHEP::Hep3Vector&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#3  0x00002b0af17b0a50 in G4Navigator::LocateGlobalPointAndSetup(CLHEP::Hep3Vector const&, CLHEP::Hep3Vector const*, bool, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#4  0x00002b0af19ce67f in G4Transportation::PostStepDoIt(G4Track const&, G4Step const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#5  0x00002b0af200a395 in G4SteppingManager::InvokePSDIP(unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#6  0x00002b0af200a65b in G4SteppingManager::InvokePostStepDoItProcs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#7  0x00002b0af200ab81 in G4SteppingManager::Stepping() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#8  0x00002b0af19c48eb in G4TrackingManager::ProcessOneTrack(G4Track*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#9  0x00002b0af164ee19 in G4EventManager::DoProcessing(G4Event*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#10 0x00002b0af12b7ced in RunManagerMTWorker::produce(edm::Event const&, edm::EventSetup const&, RunManagerMT&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#11 0x00002b0af12cea82 in omt::ThreadHandoff::Functor<OscarMTProducer::produce(edm::Event&, edm::EventSetup const&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so

Thread 8 (Thread 0x2b0b17200700 (LWP 791) "cmsRun"):
#0  __tls_get_addr () at ../sysdeps/x86_64/tls_get_addr.S:29
#1  0x00002b0af19c27f0 in aTouchableHistoryAllocator() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#2  0x00002b0af19ce69e in G4Transportation::PostStepDoIt(G4Track const&, G4Step const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#3  0x00002b0af200a395 in G4SteppingManager::InvokePSDIP(unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#4  0x00002b0af200a65b in G4SteppingManager::InvokePostStepDoItProcs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#5  0x00002b0af200ab81 in G4SteppingManager::Stepping() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#6  0x00002b0af19c48eb in G4TrackingManager::ProcessOneTrack(G4Track*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#7  0x00002b0af164ee19 in G4EventManager::DoProcessing(G4Event*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#8  0x00002b0af12b7ced in RunManagerMTWorker::produce(edm::Event const&, edm::EventSetup const&, RunManagerMT&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#9  0x00002b0af12cea82 in omt::ThreadHandoff::Functor<OscarMTProducer::produce(edm::Event&, edm::EventSetup const&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#10 0x00002b0af12c3d6a in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so

Thread 7 (Thread 0x2b0b16280700 (LWP 790) "cmsRun"):
#0  __tls_get_addr () at ../sysdeps/x86_64/tls_get_addr.S:29
#1  0x00002b0af1b7fafd in G4VPhysicalVolume::GetTranslation() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#2  0x00002b0af1b9c671 in G4VoxelNavigation::LevelLocate(G4NavigationHistory&, G4VPhysicalVolume const*, int, CLHEP::Hep3Vector const&, CLHEP::Hep3Vector const*, bool, CLHEP::Hep3Vector&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#3  0x00002b0af17b0a50 in G4Navigator::LocateGlobalPointAndSetup(CLHEP::Hep3Vector const&, CLHEP::Hep3Vector const*, bool, bool) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#4  0x00002b0af19ce67f in G4Transportation::PostStepDoIt(G4Track const&, G4Step const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#5  0x00002b0af200a395 in G4SteppingManager::InvokePSDIP(unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#6  0x00002b0af200a65b in G4SteppingManager::InvokePostStepDoItProcs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#7  0x00002b0af200ab81 in G4SteppingManager::Stepping() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#8  0x00002b0af19c48eb in G4TrackingManager::ProcessOneTrack(G4Track*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#9  0x00002b0af164ee19 in G4EventManager::DoProcessing(G4Event*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#10 0x00002b0af12b7ced in RunManagerMTWorker::produce(edm::Event const&, edm::EventSetup const&, RunManagerMT&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#11 0x00002b0af12cea82 in omt::ThreadHandoff::Functor<OscarMTProducer::produce(edm::Event&, edm::EventSetup const&)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#12 0x00002b0af12c3d6a in omt::ThreadHandoff::threadLoop(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so

Thread 6 (Thread 0x2b0b151e6700 (LWP 783) "cmsRun"):
#0  0x00002b0ab878045c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00002b0ab8294e90 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865
#2  std::__condvar::wait (__m=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#3  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#4  0x00002b0af12abebb in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so

Thread 5 (Thread 0x2b0aeac00700 (LWP 32173) "cmsRun"):
#2  0x00002b0abe792610 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b0ab878045c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#5  0x00002b0ab8294e90 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865
#6  std::__condvar::wait (__m=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#7  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#8  0x00002b0af12d22db in OscarMTProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#9  0x00002b0ab6301bfd in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#10 0x00002b0ab62e5abf in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so

Thread 4 (Thread 0x2b0ae9df0700 (LWP 32171) "cmsRun"):
#2  0x00002b0abe792610 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b0ab878045c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#5  0x00002b0ab8294e90 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865
#6  std::__condvar::wait (__m=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#7  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#8  0x00002b0af12d22db in OscarMTProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#9  0x00002b0ab6301bfd in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#10 0x00002b0ab62e5abf in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so

Thread 3 (Thread 0x2b0ae93ef700 (LWP 32170) "cmsRun"):
#2  0x00002b0abe792610 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b0ab878045c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#5  0x00002b0ab8294e90 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865
#6  std::__condvar::wait (__m=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#7  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#8  0x00002b0af12d22db in OscarMTProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#9  0x00002b0ab6301bfd in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#10 0x00002b0ab62e5abf in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so

Thread 1 (Thread 0x2b0ab98000c0 (LWP 29696) "cmsRun"):
#2  0x00002b0abe792610 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00002b0ab878045c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#5  0x00002b0ab8294e90 in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/x86_64-redhat-linux-gnu/bits/gthr-default.h:865
#6  std::__condvar::wait (__m=..., this=<optimized out>) at /data/cmsbld/jenkins/workspace/jenkins-test-bootstrap/toolconf/BUILD/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/x86_64-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#7  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#8  0x00002b0af12d22db in OscarMTProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/biglib/el8_amd64_gcc11/pluginSimulation.so
#9  0x00002b0ab6301bfd in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so
#10 0x00002b0ab62e5abf in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02781/el8_amd64_gcc11/cms/cmssw/CMSSW_13_1_NONLTO_X_2023-04-17-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so

Current Modules:
Module: non-CMSSW (crashed)
Module: OscarMTProducer:g4SimHits
Module: OscarMTProducer:g4SimHits
Module: OscarMTProducer:g4SimHits
Module: OscarMTProducer:g4SimHits

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc11/CMSSW_13_1_NONLTO_X_2023-04-19-1100/pyRelValMatrixLogs/run/25234.911_TTbar_14TeV+2026D99_DD4hep/step1_TTbar_14TeV+2026D99_DD4hep.log#/
@cmsbuild
Copy link
Contributor

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@makortel
Copy link
Contributor Author

assign geometry,upgrade,simulation

@cmsbuild
Copy link
Contributor

New categories assigned: geometry,upgrade,simulation

@mdhildreth,@mdhildreth,@AdrianoDee,@Dr15Jones,@srimanob,@makortel,@bsunanda,@civanch,@civanch you have been requested to review this Pull request/Issue and eventually sign? Thanks

@makortel
Copy link
Contributor Author

The same workflow step 2 failed in CMSSW_13_1_X_2023-04-19-1100 (el8_amd64_gcc11) with

----- Begin Fatal Exception 19-Apr-2023 13:36:50 CEST-----------------------
An exception of category 'OutOfBound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 3
   [1] Running path 'FEVTDEBUGHLToutput_step'
   [2] Prefetching for module PoolOutputModule/'FEVTDEBUGHLToutput'
   [3] Prefetching for module L1TEGMultiMerger/'l1tLayer1EG'
   [4] Prefetching for module L1TCorrelatorLayer1Producer/'l1tLayer1HGCal'
   [5] Prefetching for module PFClusterProducerFromHGC3DClusters/'l1tPFClustersFromHGC3DClusters'
   [6] Calling method for module HGCalBackendLayer2Producer/'l1tHGCalBackEndLayer2Producer'
Exception Message:
TC X1 = inf out of the seeding histogram bounds 0.076 - 0.58
----- End Fatal Exception -------------------------------------------------
...
----- Begin Fatal Exception 19-Apr-2023 13:36:51 CEST-----------------------
An exception of category 'OutOfBound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 4 stream: 0
   [1] Running path 'FEVTDEBUGHLToutput_step'
   [2] Prefetching for module PoolOutputModule/'FEVTDEBUGHLToutput'
   [3] Prefetching for module L1TEGMultiMerger/'l1tLayer1EG'
   [4] Prefetching for module L1TCorrelatorLayer1Producer/'l1tLayer1HGCal'
   [5] Prefetching for module PFClusterProducerFromHGC3DClusters/'l1tPFClustersFromHGC3DClusters'
   [6] Calling method for module HGCalBackendLayer2Producer/'l1tHGCalBackEndLayer2Producer'
Exception Message:
TC X1 = inf out of the seeding histogram bounds 0.076 - 0.58
----- End Fatal Exception -------------------------------------------------

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc11/CMSSW_13_1_X_2023-04-19-1100/pyRelValMatrixLogs/run/25234.911_TTbar_14TeV+2026D99_DD4hep/step2_TTbar_14TeV+2026D99_DD4hep.log#/

that may be related

@srimanob
Copy link
Contributor

The error seems to be the same as D88 when the fix is implemented in #40404 (3c9543c)

@perrotta
Copy link
Contributor

@bsunanda FYI

@perrotta perrotta mentioned this issue Apr 20, 2023
Merged
@bsunanda
Copy link
Contributor

Please try with #41380

@makortel
Copy link
Contributor Author

The crash seems to be gone, but the exception #41376 (comment) is still happening. It is probably better to follow up in a separate issue.

@makortel makortel mentioned this issue Apr 27, 2023
Closed
@makortel makortel mentioned this issue Jun 12, 2023
Closed
@makortel
Copy link
Contributor Author

The segfault in HGCalCassette::getShift() is back, see issue #41927

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
geometry-pending pending-signatures simulation-pending upgrade-pending
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@makortel @perrotta @cmsbuild @srimanob @bsunanda