Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crash in ZdcHitReconstructor #18463

Closed
Dr15Jones opened this issue Apr 25, 2017 · 9 comments
Closed

Crash in ZdcHitReconstructor #18463

Dr15Jones opened this issue Apr 25, 2017 · 9 comments
Labels
fully-signed reconstruction-approved

Comments

@Dr15Jones
Copy link
Contributor

The CMSSW_9_1_X_2017-04-23-0000 IB crashed in workflow 134.0_MinimumBiasBS with the backtrace

#3  0x00007ff05e721b35 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_1_X_2017-04-23-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007ff0475cbcba in ZdcSimpleRecAlgo::reconstruct(ZDCDataFrame const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, HcalCoder const&, HcalCalibrations const&) const () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_1_X_2017-04-23-0000/lib/slc6_amd64_gcc630/libRecoLocalCaloHcalRecAlgos.so
#6  0x00007ff01006b336 in ZdcHitReconstructor::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_1_X_2017-04-23-0000/lib/slc6_amd64_gcc630/pluginRecoLocalCaloHcalRecProducers.so
#7  0x00007ff066c42263 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#8  0x00007ff066c19b32 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#9  0x00007ff066b18037 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#10 0x00007ff066b18223 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#11 0x00007ff066b185be in void edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#12 0x00007ff066b18b61 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/lib/slc6_amd64_gcc630/libFWCoreFramework.so

The relevant items running on the other threads of the job were

Thread 9 (Thread 0x7feffffbe700 (LWP 25500)):
#3  <signal handler called>
#4  0x00007ff02837cdfd in void SiStripMonitorTrack::RecHitInfo<SiStripRecHit2D>(SiStripRecHit2D const*, Vector3DBase<float, LocalTag>, edm::Event const&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_1_X_2017-04-23-0000/lib/slc6_amd64_gcc630/pluginDQMSiStripMonitorTrack.so
#5  0x00007ff02837324a in SiStripMonitorTrack::trajectoryStudy(reco::Track const&, edm::Event const&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_1_X_2017-04-23-0000/lib/slc6_amd64_gcc630/pluginDQMSiStripMonitorTrack.so
#6  0x00007ff0283736f0 in SiStripMonitorTrack::trackStudyFromTrajectory(edm::Handle<std::vector<reco::Track, std::allocator<reco::Track> > >, edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_1_X_2017-04-23-0000/lib/slc6_amd64_gcc630/pluginDQMSiStripMonitorTrack.so
#7  0x00007ff0283738df in SiStripMonitorTrack::trackStudy(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_1_X_2017-04-23-0000/lib/slc6_amd64_gcc630/pluginDQMSiStripMonitorTrack.so
#8  0x00007ff028375f89 in SiStripMonitorTrack::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_1_X_2017-04-23-0000/lib/slc6_amd64_gcc630/pluginDQMSiStripMonitorTrack.so
#9  0x00007ff066c45ada in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/lib/slc6_amd64_gcc630/libFWCoreFramework.so
...

Thread 8 (Thread 0x7ff0009bf700 (LWP 25499)):
#3  <signal handler called>
#4  je_tcache_bin_flush_small (tsd=0x7ff0009ba2c0, tcache=tcache@entry=0x7ff003cef000, tbin=tbin@entry=0x7ff003cef048, binind=<optimized out>, rem=100) at src/tcache.c:130
#5  0x00007ff065a0d9ba in je_tcache_dalloc_small (slow_path=false, binind=<optimized out>, ptr=0x7fef567b4c90, tcache=<optimized out>, tsd=<optimized out>) at include/jemalloc/internal/tcache.h:416
#6  je_arena_dalloc (slow_path=false, tcache=<optimized out>, ptr=0x7fef567b4c90, tsdn=<optimized out>) at include/jemalloc/internal/arena.h:1429
#7  je_idalloctm (slow_path=false, is_metadata=false, tcache=<optimized out>, ptr=0x7fef567b4c90, tsdn=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:1164
#8  je_iqalloc (slow_path=false, tcache=<optimized out>, ptr=0x7fef567b4c90, tsd=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:1181
#9  ifree (tsd=<optimized out>, slow_path=false, tcache=<optimized out>, ptr=0x7fef567b4c90) at src/jemalloc.c:1871
#10 free (ptr=0x7fef567b4c90) at src/jemalloc.c:1991
#11 0x00007ff02837cf9e in void SiStripMonitorTrack::RecHitInfo<SiStripRecHit2D>(SiStripRecHit2D const*, Vector3DBase<float, LocalTag>, edm::Event const&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_1_X_2017-04-23-0000/lib/slc6_amd64_gcc630/pluginDQMSiStripMonitorTrack.so
#12 0x00007ff02837324a in SiStripMonitorTrack::trajectoryStudy(reco::Track const&, edm::Event const&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_1_X_2017-04-23-0000/lib/slc6_amd64_gcc630/pluginDQMSiStripMonitorTrack.so
#13 0x00007ff0283736f0 in SiStripMonitorTrack::trackStudyFromTrajectory(edm::Handle<std::vector<reco::Track, std::allocator<reco::Track> > >, edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_1_X_2017-04-23-0000/lib/slc6_amd64_gcc630/pluginDQMSiStripMonitorTrack.so
#14 0x00007ff0283738df in SiStripMonitorTrack::trackStudy(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_1_X_2017-04-23-0000/lib/slc6_amd64_gcc630/pluginDQMSiStripMonitorTrack.so
#15 0x00007ff028375f89 in SiStripMonitorTrack::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_1_X_2017-04-23-0000/lib/slc6_amd64_gcc630/pluginDQMSiStripMonitorTrack.so
#16 0x00007ff066c45ada in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/lib/slc6_amd64_gcc630/libFWCoreFramework.so
...

Thread 7 (Thread 0x7ff041325700 (LWP 25478)):
#0  0x0000003af14e5409 in syscall () from /lib64/libc.so.6
#1  0x00007ff043ed286f in XrdSys::LinuxSemaphore::Wait (this=0x7ff044412070) at /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.6.1.RC2/xrootd-4.6.1.RC2/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
...

Thread 6 (Thread 0x7ff041d26700 (LWP 25477)):
#0  0x0000003af14e5409 in syscall () from /lib64/libc.so.6
#1  0x00007ff043ed286f in XrdSys::LinuxSemaphore::Wait (this=0x7ff044412070) at /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.6.1.RC2/xrootd-4.6.1.RC2/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
...

Thread 5 (Thread 0x7ff042727700 (LWP 25476)):
#0  0x0000003af14e5409 in syscall () from /lib64/libc.so.6
#1  0x00007ff043ed286f in XrdSys::LinuxSemaphore::Wait (this=0x7ff044412070) at /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.6.1.RC2/xrootd-4.6.1.RC2/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
...

Thread 4 (Thread 0x7ff043128700 (LWP 25475)):
#0  0x0000003af180f00d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007ff043d5e763 in XrdSysTimer::Wait (mills=<optimized out>) at /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.6.1.RC2/xrootd-4.6.1.RC2/src/XrdSys/XrdSysTimer.cc:239
...

Thread 3 (Thread 0x7ff043b29700 (LWP 25474)):
#0  0x0000003af14e91b3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007ff043d63814 in XrdSys::IOEvents::PollE::Begin (this=0x7ff0443c9620, syncsem=<optimized out>, retcode=<optimized out>, eTxt=<optimized out>) at /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.6.1.RC2/xrootd-4.6.1.RC2/src/./XrdSys/XrdSysIOEventsPollE.icc:213
#2  0x00007ff043d5fe85 in XrdSys::IOEvents::BootStrap::Start (parg=0x7ffd708d9420) at /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.6.1.RC2/xrootd-4.6.1.RC2/src/XrdSys/XrdSysIOEvents.cc:131
...

Thread 2 (Thread 0x7ff04b3ff700 (LWP 25455)):
#0  0x0000003af180f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007ff05e71f8a7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_1_X_2017-04-23-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007ff05e71ff85 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_1_X_2017-04-23-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
...

Thread 1 (Thread 0x7ff064d8ac20 (LWP 25254)):
#3  <signal handler called>
#4  0x00007ff06645feac in ROOT::Math::Cephes::igam(double, double) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/external/slc6_amd64_gcc630/lib/libMathCore.so
#5  0x00007ff06645fb26 in ROOT::Math::Cephes::igamc(double, double) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/external/slc6_amd64_gcc630/lib/libMathCore.so
#6  0x00007ff02a93f2e1 in TrackAnalyzer::analyze(edm::Event const&, edm::EventSetup const&, reco::Track const&) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_1_X_2017-04-23-0000/lib/slc6_amd64_gcc630/pluginDQMTrackingMonitor.so
#7  0x00007ff02a8da5d5 in TrackingMonitor::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_1_X_2017-04-23-0000/lib/slc6_amd64_gcc630/pluginDQMTrackingMonitor.so
#8  0x00007ff066c45ada in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02469/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_9_1_X_2017-04-24-1100/lib/slc6_amd64_gcc630/libFWCoreFramework.so

The link to the full file is
https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc630/CMSSW_9_1_X_2017-04-24-1100/pyRelValMatrixLogs/run/134.0_MinimumBiasBS+RunMinBias2011B+RECODR1+ALCAPROMPT+ALCAHARVD/step2_MinimumBiasBS+RunMinBias2011B+RECODR1+ALCAPROMPT+ALCAHARVD.log
@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 25, 2017 •
edited by smuzaffar

A new Issue was created by @Dr15Jones Chris Jones.

@davidlange6, @Dr15Jones, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@Dr15Jones
Copy link
Contributor Author

assign reconstruction

@cmsbuild
Copy link
Contributor

New categories assigned: reconstruction

@slava77,@perrotta you have been requested to review this Pull request/Issue and eventually sign? Thanks

@perrotta
Copy link
Contributor

perrotta commented Apr 26, 2017 •
edited

(Removed comment, and "+1": my mistake)

@cmsbuild
Copy link
Contributor

This issue is fully signed and ready to be closed.

@slava77
Copy link
Contributor

slava77 commented Apr 26, 2017

@perrotta
I'm missing how a fix in CTPPS can affect ZDC.
Was this issue signed by mistake?

@perrotta
Copy link
Contributor

Yes, you are right @slava77 : I found the notification of this issue together with the other message forwarded by you about the crash in CTPPS, and I did not read it carefully enough: sorry for that.

Removed the "+1" in my comment above: please @Dr15Jones reopen this issue

@Dr15Jones Dr15Jones reopened this Apr 26, 2017
@slava77
Copy link
Contributor

slava77 commented Oct 20, 2018

+1

this was duplicated in #21250 and then resolved with #22063

@cmsbuild
Copy link
Contributor

This issue is fully signed and ready to be closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
fully-signed reconstruction-approved
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Dr15Jones @perrotta @cmsbuild @slava77