DQMHcal crash in CMSSW_8_1_THREADED_X_2016-06-10-2300 #14898

bouril · 2016-06-15T18:48:34Z

Build CMSSW_8_1_THREADED_X_2016-06-10-2300

4.45 RunJet2012A

DQMHcal problem

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_THREADED_X_2016-06-10-2300/pyRelValMatrixLogs/run/4.45_RunJet2012A+RunJet2012A+HLTD+RECODR1reHLT+HARVESTDR1reHLT/step3_RunJet2012A+RunJet2012A+HLTD+RECODR1reHLT+HARVESTDR1reHLT.log

Thread 2 (Thread 0x7fb2c873f700 (LWP 32294)):
#0 0x0000003fccadf283 in poll () from /lib64/libc.so.6
#1 0x00007fb386dd9da4 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-10-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2 0x00007fb386dda00a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-10-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3 0x00007fb386dda18b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-10-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4
#5 0x00007fb34d040d9d in RawTask::process(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-10-2300/lib/slc6_amd64_gcc530/pluginDQMHcalTasksAuto.so
#6 0x00007fb34cf20d2a in hcaldqm::DQTask::analyze(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-10-2300/lib/slc6_amd64_gcc530/libDQMHcalCommon.so
#7 0x00007fb38c7f5a1a in edm::stream::EDAnalyzerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry, edm::ModuleCallingContext const_) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-10-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#8 0x00007fb38c7cf81f in edm::WorkerTedm::stream::EDAnalyzerAdaptorBase::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-10-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9 0x00007fb38c70972c in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-10-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007fb38c7098f2 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-10-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007fb38c725e2e in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const_)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-10-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007fb38c7260e2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-10-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007fb38c726505 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-10-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007fb38c7267fb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-10-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007fb38c726a6f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-10-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007fb38c78c612 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-10-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007fb38c78cd47 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-10-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007fb38c793524 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-10-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007fb38b4d89b5 in tbb::internal::custom_schedulertbb::internal::IntelSchedulerTraits::local_wait_for_all (this=0x7fb36196fe00, parent=..., child=) at ../../src/tbb/custom_scheduler.h:463
#20 0x00007fb38b4d21a4 in tbb::internal::arena::process (this=this@entry=0x7fb38a01f780, s=...) at ../../src/tbb/arena.cpp:156
#21 0x00007fb38b4d1264 in tbb::internal::market::process (this=0x7fb38a047c00, j=...) at ../../src/tbb/market.cpp:502
#22 0x00007fb38b4cd736 in tbb::internal::rml::private_worker::run (this=0x7fb389e66080) at ../../src/tbb/private_server.cpp:275
#23 0x00007fb38b4cd989 in tbb::internal::rml::private_worker::thread_routine (arg=) at ../../src/tbb/private_server.cpp:228
#24 0x0000003fcce07aa1 in start_thread () from /lib64/libpthread.so.0
#25 0x0000003fccae8aad in clone () from /lib64/libc.so.6

cmsbuild · 2016-06-15T18:48:51Z

A new Issue was created by @bouril Dimitri Bourilkov.

@davidlange6, @smuzaffar, @Degano, @davidlt, @Dr15Jones can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

Dr15Jones · 2016-06-22T14:50:39Z

assign core

cmsbuild · 2016-06-22T14:50:53Z

New categories assigned: core

@Dr15Jones,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

Dr15Jones · 2016-06-22T14:51:37Z

@dan131riley looks like this happened again
https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc530/CMSSW_8_1_THREADED_X_2016-06-21-2300/pyRelValMatrixLogs/run/134.703_RunDoubleMuon2015B+RunDoubleMuon2015B+HLTDR2_50ns+RECODR2_50nsreHLT+HARVESTDR2/step3_RunDoubleMuon2015B+RunDoubleMuon2015B+HLTDR2_50ns+RECODR2_50nsreHLT+HARVESTDR2.log

dan131riley · 2016-06-23T14:31:28Z

@Dr15Jones this looks like another memory overrun, we're getting these in memcheck:

==83065== Thread 3:
==83065== Invalid read of size 2
==83065== at 0x4E51FD9D: getBunchNumber (EventFilter/HcalRawToDigi/interface/HcalHTRData.h:116)
==83065== by 0x4E51FD9D: RawTask::_process(edm::Event const&, edm::EventSetup const&) (DQM/HcalTasks/plugins/RawTask.cc:276)
==83065== by 0x4E618529: hcaldqm::DQTask::analyze(edm::Event const&, edm::EventSetup const&) (in /cvmfs/cms-ib.cern.ch/2016-26/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_THREADED_X_2016-06-19-0000/lib/slc6_amd64_gcc530/libDQMHcalCommon.so)

In the log file there are lots of "unable to read", "could not get", "is not valid", etc. complaints indicating that analyzers aren't finding what they expected in the event, and it looks like the HcalTasks analyzer isn't properly protected against that--it is getting 0 length data packets, and here:

cmssw/EventFilter/HcalRawToDigi/src/HcalHTRData.cc

Line 47 in 0397259

void HcalHTRData::adoptData(const unsigned short* data, int length) {

where the HcalHTRData class "adopts" the data it notices that the length is 0 and sets a flag that it's invalid:

if (m_rawLength<5) {
m_formatVersion=-2; // invalid!
} else {

but that doesn't get used by the caller. I can put in a point fix, something like:

--- a/DQM/HcalTasks/plugins/RawTask.cc
+++ b/DQM/HcalTasks/plugins/RawTask.cc
@@ -263,7 +263,7 @@ RawTask::RawTask(edm::ParameterSet const& ps):
for (int is=0; is<HcalDCCHeader::SPIGOT_COUNT; is++)
{
int r = hdcc->getSpigotData(is, htr, raw.size());

```
                          if (r!=0)
```

                          if (r!=0 || !htr.check())
                                continue;
                        HcalElectronicsId eid = HcalElectronicsId(
                                constants::FIBERCH_MIN, constants::FIBER_VME_MIN,

That pattern is used in several other places, like

cmssw/DQM/HcalMonitorTasks/src/HcalBaseDQMonitor.cc

Line 354 in 0397259

if (!htr.check()) continue;

so it should be safe, and is pretty minimal, but I'm wondering if there ought to be more protection somewhere upstream. Any advice?

-dan

Dr15Jones · 2016-06-23T14:47:45Z

I think create a pull request which uses the check and in the comments of the pull request talk about having a check further up stream. That way the discussion can happen in the right place (with the pull request) and hopefully with the right people.

smuzaffar · 2019-04-27T23:21:05Z

+1

cmsbuild · 2019-04-27T23:21:33Z

This issue is fully signed and ready to be closed.

cmsbuild added the pending-assignment label Jun 15, 2016

cmsbuild added core-pending pending-signatures and removed pending-assignment labels Jun 22, 2016

dan131riley mentioned this issue Jun 23, 2016

protect RawTask::_process from processing bad HcalHTRData #14963

Merged

smuzaffar closed this as completed Apr 27, 2019

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels Apr 27, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DQMHcal crash in CMSSW_8_1_THREADED_X_2016-06-10-2300 #14898

DQMHcal crash in CMSSW_8_1_THREADED_X_2016-06-10-2300 #14898

bouril commented Jun 15, 2016

cmsbuild commented Jun 15, 2016 •

edited by smuzaffar

Dr15Jones commented Jun 22, 2016

cmsbuild commented Jun 22, 2016

Dr15Jones commented Jun 22, 2016

dan131riley commented Jun 23, 2016

Dr15Jones commented Jun 23, 2016

smuzaffar commented Apr 27, 2019

cmsbuild commented Apr 27, 2019

DQMHcal crash in CMSSW_8_1_THREADED_X_2016-06-10-2300 #14898

DQMHcal crash in CMSSW_8_1_THREADED_X_2016-06-10-2300 #14898

Comments

bouril commented Jun 15, 2016

cmsbuild commented Jun 15, 2016 • edited by smuzaffar

Dr15Jones commented Jun 22, 2016

cmsbuild commented Jun 22, 2016

Dr15Jones commented Jun 22, 2016

dan131riley commented Jun 23, 2016

Dr15Jones commented Jun 23, 2016

smuzaffar commented Apr 27, 2019

cmsbuild commented Apr 27, 2019

cmsbuild commented Jun 15, 2016 •

edited by smuzaffar