segmentation faults in EcalRawToDigi:ecalDigis

[dan131riley]

Two segmentation faults, in the last two devel builds, so far only seen in devel. EcalRawToDigi hasn't been touched recently. EcalDigiToRaw was recently updated in #18322

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc7_amd64_gcc630/CMSSW_9_3_DEVEL_X_2017-08-08-2300/pyRelValMatrixLogs/run/1.0_ProdMinBias+ProdMinBias+DIGIPROD1+RECOPROD1/step3_ProdMinBias+ProdMinBias+DIGIPROD1+RECOPROD1.log
https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc630/CMSSW_9_3_DEVEL_X_2017-08-09-2300/pyRelValMatrixLogs/run/1343.0_ZpMM_2250_13+ZpMM_2250_13INPUT+DIGIUP15+RECOUP15+HARVESTUP15/step3_ZpMM_2250_13+ZpMM_2250_13INPUT+DIGIUP15+RECOUP15+HARVESTUP15.log

Begin processing the 1st record. Run 1, Event 3, LumiSection 1 at 10-Aug-2017 05:51:35.683 CEST
Begin processing the 2nd record. Run 1, Event 6, LumiSection 1 at 10-Aug-2017 05:51:35.743 CEST
Begin processing the 3rd record. Run 1, Event 5, LumiSection 1 at 10-Aug-2017 05:51:35.748 CEST
Begin processing the 4th record. Run 1, Event 4, LumiSection 1 at 10-Aug-2017 05:51:35.754 CEST


A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7f3b78abf700 (LWP 20991)):
#0  0x00007f3bed916383 in poll () from /lib64/libc.so.6
#1  0x00007f3bdeb225b4 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007f3bdeb22d8a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007f3bdeb23de5 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f3b92092da0 in DCCEBTCCBlock::addTriggerPrimitivesToCollection() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libEventFilterEcalRawToDigi.so
#6  0x00007f3b920b0a6b in DCCTCCBlock::unpack(unsigned long const**, unsigned int*, short) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libEventFilterEcalRawToDigi.so
#7  0x00007f3b92073eef in DCCEBEventBlock::unpack(unsigned long const*, unsigned long, unsigned int) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libEventFilterEcalRawToDigi.so
#8  0x00007f3b92100fea in EcalRawToDigi::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/pluginEventFilterEcalRawToDigiPlugins.so
#9  0x00007f3bf03426b3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#10 0x00007f3bf030e312 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#11 0x00007f3bf02104a7 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#12 0x00007f3bf0210693 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#13 0x00007f3bf0210a1b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#14 0x00007f3bf0210fc5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#15 0x00007f3beeea0a13 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f3b77fb7e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#16 0x00007f3beee99a62 in tbb::internal::arena::process (this=0x7f3bebabb780, s=...) at ../../src/tbb/arena.cpp:159
#17 0x00007f3beee9854b in tbb::internal::market::process (this=0x7f3bebad7e00, j=...) at ../../src/tbb/market.cpp:697
#18 0x00007f3beee947f6 in tbb::internal::rml::private_worker::run (this=0x7f3bebacf980) at ../../src/tbb/private_server.cpp:271
#19 0x00007f3beee94a29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:224
#20 0x00007f3bedbd2aa1 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f3bed91fbcd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f3b7967e700 (LWP 20990)):
#0  0x00007f3bed8e3c4d in nanosleep () from /lib64/libc.so.6
#1  0x00007f3bed8e3ac0 in sleep () from /lib64/libc.so.6
#2  0x00007f3bdeb21f3e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f3bef951f5a in std::vector<float, std::allocator<float> >::_M_default_append(unsigned long) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/external/slc6_amd64_gcc630/lib/libRIO.so
#5  0x00007f3b9254de1a in MultiHitGeneratorFromChi2::hitSets(TrackingRegion const&, OrderedMultiHits&, edm::EventSetup const&, HitDoublets const&, RecHitsSortedInPhi const**, std::vector<DetLayer const*, std::allocator<DetLayer const*> > const&, int, std::vector<std::unique_ptr<BaseTrackerRecHit, std::default_delete<BaseTrackerRecHit> >, std::allocator<std::unique_ptr<BaseTrackerRecHit, std::default_delete<BaseTrackerRecHit> > > >&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/pluginRecoTrackerTkSeedGeneratorPlugins.so
#6  0x00007f3b92550567 in MultiHitGeneratorFromChi2::hitSets(TrackingRegion const&, OrderedMultiHits&, edm::Event const&, edm::EventSetup const&, HitDoublets const&, std::vector<SeedingLayerSetsHits::SeedingLayer, std::allocator<SeedingLayerSetsHits::SeedingLayer> > const&, LayerHitMapCache&, std::vector<std::unique_ptr<BaseTrackerRecHit, std::default_delete<BaseTrackerRecHit> >, std::allocator<std::unique_ptr<BaseTrackerRecHit, std::default_delete<BaseTrackerRecHit> > > >&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/pluginRecoTrackerTkSeedGeneratorPlugins.so
#7  0x00007f3b92557916 in MultiHitFromChi2EDProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/pluginRecoTrackerTkSeedGeneratorPlugins.so
#8  0x00007f3bf03426b3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#9  0x00007f3bf030e312 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#10 0x00007f3bf02104a7 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#11 0x00007f3bf0210693 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#12 0x00007f3bf0210a1b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#13 0x00007f3bf0210fc5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#14 0x00007f3beeea0a13 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f3ba0be3e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#15 0x00007f3beee99a62 in tbb::internal::arena::process (this=0x7f3bebabb780, s=...) at ../../src/tbb/arena.cpp:159
#16 0x00007f3beee9854b in tbb::internal::market::process (this=0x7f3bebad7e00, j=...) at ../../src/tbb/market.cpp:697
#17 0x00007f3beee947f6 in tbb::internal::rml::private_worker::run (this=0x7f3bebacfa80) at ../../src/tbb/private_server.cpp:271
#18 0x00007f3beee94a29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:224
#19 0x00007f3bedbd2aa1 in start_thread () from /lib64/libpthread.so.0
#20 0x00007f3bed91fbcd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f3b7a07f700 (LWP 20989)):
#0  0x00007f3bed8e3c4d in nanosleep () from /lib64/libc.so.6
#1  0x00007f3bed8e3ac0 in sleep () from /lib64/libc.so.6
#2  0x00007f3bdeb21f3e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f3bcc956573 in SteppingHelixPropagator::getNextState(SteppingHelixStateInfo const&, SteppingHelixStateInfo&, double, CLHEP::Hep3Vector const&, CLHEP::Hep3Vector const&, double, double, ROOT::Math::SMatrix<double, 5u, 5u, ROOT::Math::MatRepStd<double, 5u, 5u> > const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libTrackPropagationSteppingHelixPropagator.so
#5  0x00007f3bcc956b8a in SteppingHelixPropagator::makeAtomStep(SteppingHelixStateInfo&, SteppingHelixStateInfo&, double, PropagationDirection, SteppingHelixPropagator::Fancy) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libTrackPropagationSteppingHelixPropagator.so
#6  0x00007f3bcc959ddd in SteppingHelixPropagator::propagate(SteppingHelixStateInfo (&) [8], int&, SteppingHelixPropagator::DestType, double const*, double) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libTrackPropagationSteppingHelixPropagator.so
#7  0x00007f3bcc95dc1b in SteppingHelixPropagator::propagate(SteppingHelixStateInfo const&, Plane const&, SteppingHelixStateInfo&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libTrackPropagationSteppingHelixPropagator.so
#8  0x00007f3bcc95e3d7 in SteppingHelixPropagator::propagateWithPath(FreeTrajectoryState const&, Plane const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libTrackPropagationSteppingHelixPropagator.so
#9  0x00007f3bcc961dbd in Propagator::propagateWithPath(TrajectoryStateOnSurface const&, Plane const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libTrackPropagationSteppingHelixPropagator.so
#10 0x00007f3bcd49bc1f in ForwardDetLayer::compatible(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libTrackingToolsDetLayers.so
#11 0x00007f3bcc8f7524 in MuRingForwardLayer::compatibleDets(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libRecoMuonDetLayers.so
#12 0x00007f3bcc8fb69a in MuRingForwardDoubleLayer::groupedCompatibleDets(TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libRecoMuonDetLayers.so
#13 0x00007f3bc51c417e in MuonDetLayerMeasurements::groupedMeasurements(DetLayer const*, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&, edm::Event const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libRecoMuonMeasurementDet.so
#14 0x00007f3bc51c50bc in MuonDetLayerMeasurements::groupedMeasurements(DetLayer const*, TrajectoryStateOnSurface const&, Propagator const&, MeasurementEstimator const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libRecoMuonMeasurementDet.so
#15 0x00007f3b92464fd6 in StandAloneMuonFilter::findBestMeasurements(DetLayer const*, TrajectoryStateOnSurface const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libRecoMuonStandAloneTrackFinder.so
#16 0x00007f3b92466c46 in StandAloneMuonFilter::refit(TrajectoryStateOnSurface const&, DetLayer const*, Trajectory&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libRecoMuonStandAloneTrackFinder.so
#17 0x00007f3b9246db82 in StandAloneMuonTrajectoryBuilder::trajectories(TrajectorySeed const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libRecoMuonStandAloneTrackFinder.so
#18 0x00007f3b92471d9c in ExhaustiveMuonTrajectoryBuilder::trajectories(TrajectorySeed const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libRecoMuonStandAloneTrackFinder.so
#19 0x00007f3bcbb649ee in MuonTrackFinder::reconstruct(edm::Handle<edm::View<TrajectorySeed> > const&, edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libRecoMuonTrackingTools.so
#20 0x00007f3b92991d3a in StandAloneMuonProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/pluginRecoMuonStandAloneMuonProducer.so
#21 0x00007f3bf03426b3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#22 0x00007f3bf030e312 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#23 0x00007f3bf02104a7 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#24 0x00007f3bf0210693 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#25 0x00007f3bf0210a1b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#26 0x00007f3bf0210fc5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#27 0x00007f3beeea0a13 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f3b780b7e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#28 0x00007f3beee99a62 in tbb::internal::arena::process (this=0x7f3bebabb780, s=...) at ../../src/tbb/arena.cpp:159
#29 0x00007f3beee9854b in tbb::internal::market::process (this=0x7f3bebad7e00, j=...) at ../../src/tbb/market.cpp:697
#30 0x00007f3beee947f6 in tbb::internal::rml::private_worker::run (this=0x7f3bebacfa00) at ../../src/tbb/private_server.cpp:271
#31 0x00007f3beee94a29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:224
#32 0x00007f3bedbd2aa1 in start_thread () from /lib64/libpthread.so.0
#33 0x00007f3bed91fbcd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f3bcfa3f700 (LWP 20978)):
#0  0x00007f3bedbda37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f3bdeb221c7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007f3bdeb22765 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007f3bee13ec2f in std::execute_native_thread_routine (__p=0x7f3bdf3549b0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007f3bedbd2aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f3bed91fbcd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f3bec016c20 (LWP 20941)):
#0  0x00007f3bed8e3c4d in nanosleep () from /lib64/libc.so.6
#1  0x00007f3bed8e3ac0 in sleep () from /lib64/libc.so.6
#2  0x00007f3bdeb21f3e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f3bf03dc37b in check_match (sym=0x7f3bf04eccd8) at dl-lookup.c:151
#5  0x00007f3bf03dcc82 in do_lookup_x (new_hash=39592074, old_hash=0x7fff6ae8feb8, ref=<optimized out>, result=0x7fff6ae8fea0, scope=<optimized out>, i=5, flags=5, skip=0x0, undef_map=0x7f3b9294de00) at dl-lookup.c:251
#6  0x00007f3bf03dcf0a in _dl_lookup_symbol_x (undef_name=0x7f3b92524aef "_ZN3edm7RefCoreC1ERKS0_", undef_map=0x7f3b9294de00, ref=0x7fff6ae8ff18, symbol_scope=0x7f3b9294e158, version=0x0, type_class=1, flags=5, skip_map=0x0) at dl-lookup.c:747
#7  0x00007f3bf03e10c0 in _dl_fixup (l=<optimized out>, reloc_arg=<optimized out>) at ../elf/dl-runtime.c:118
#8  0x00007f3bf03e7825 in _dl_runtime_resolve () at ../sysdeps/x86_64/dl-trampoline.S:41
#9  0x00007f3b9254a06e in MultiHitGeneratorFromChi2::hitSets(TrackingRegion const&, OrderedMultiHits&, edm::EventSetup const&, HitDoublets const&, RecHitsSortedInPhi const**, std::vector<DetLayer const*, std::allocator<DetLayer const*> > const&, int, std::vector<std::unique_ptr<BaseTrackerRecHit, std::default_delete<BaseTrackerRecHit> >, std::allocator<std::unique_ptr<BaseTrackerRecHit, std::default_delete<BaseTrackerRecHit> > > >&)::{lambda(TrackingRecHit const*, Vector3DBase<float, GlobalTag> const&)#1}::operator()(TrackingRecHit const*, Vector3DBase<float, GlobalTag> const&) const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/pluginRecoTrackerTkSeedGeneratorPlugins.so
#10 0x00007f3b9254d69f in MultiHitGeneratorFromChi2::hitSets(TrackingRegion const&, OrderedMultiHits&, edm::EventSetup const&, HitDoublets const&, RecHitsSortedInPhi const**, std::vector<DetLayer const*, std::allocator<DetLayer const*> > const&, int, std::vector<std::unique_ptr<BaseTrackerRecHit, std::default_delete<BaseTrackerRecHit> >, std::allocator<std::unique_ptr<BaseTrackerRecHit, std::default_delete<BaseTrackerRecHit> > > >&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/pluginRecoTrackerTkSeedGeneratorPlugins.so
#11 0x00007f3b92550567 in MultiHitGeneratorFromChi2::hitSets(TrackingRegion const&, OrderedMultiHits&, edm::Event const&, edm::EventSetup const&, HitDoublets const&, std::vector<SeedingLayerSetsHits::SeedingLayer, std::allocator<SeedingLayerSetsHits::SeedingLayer> > const&, LayerHitMapCache&, std::vector<std::unique_ptr<BaseTrackerRecHit, std::default_delete<BaseTrackerRecHit> >, std::allocator<std::unique_ptr<BaseTrackerRecHit, std::default_delete<BaseTrackerRecHit> > > >&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/pluginRecoTrackerTkSeedGeneratorPlugins.so
#12 0x00007f3b92557916 in MultiHitFromChi2EDProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/pluginRecoTrackerTkSeedGeneratorPlugins.so
#13 0x00007f3bf03426b3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#14 0x00007f3bf030e312 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#15 0x00007f3bf02104a7 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#16 0x00007f3bf0210693 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#17 0x00007f3bf0210a1b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#18 0x00007f3bf0210fc5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#19 0x00007f3beeea0a13 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f3bebad6400, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#20 0x00007f3bf02b793c in edm::EventProcessor::readAndProcessEvents() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#21 0x00007f3bf02c2291 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-09-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#22 0x000000000040e752 in main::{lambda()#1}::operator()() const ()
#23 0x000000000040d115 in main ()

Current Modules:

Module: EcalRawToDigi:ecalDigis (crashed)
Module: StandAloneMuonProducer:standAloneMuons
Module: MultiHitFromChi2EDProducer:pixelLessStepHitTriplets
Module: MultiHitFromChi2EDProducer:pixelLessStepHitTriplets

A fatal system signal has occurred: segmentation violation


%MSG-w MemoryCheck:  FastjetJetProducer:ak4CaloJets 09-Aug-2017 05:18:25 CEST  Run: 1 Event: 5
MemoryCheck: module FastjetJetProducer:ak4CaloJets VSIZE 2212.22 0 RSS 1392.9 0.0078125
%MSG


A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7fa5d9c3f700 (LWP 22794)):
#0  0x00007fa61fb7fefd in nanosleep () from /lib64/libc.so.6
#1  0x00007fa61fb7fd94 in sleep () from /lib64/libc.so.6
#2  0x00007fa61a059c8e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fa60a291a90 in reco::Jet::Jet(ROOT::Math::LorentzVector<ROOT::Math::PxPyPzE4D<double> > const&, ROOT::Math::PositionVector3D<ROOT::Math::Cartesian3D<double>, ROOT::Math::DefaultCoordinateSystemTag> const&, std::vector<edm::Ptr<reco::Candidate>, std::allocator<edm::Ptr<reco::Candidate> > > const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libDataFormatsJetReco.so
#5  0x00007fa60a297229 in reco::BasicJet::BasicJet(ROOT::Math::LorentzVector<ROOT::Math::PxPyPzE4D<double> > const&, ROOT::Math::PositionVector3D<ROOT::Math::Cartesian3D<double>, ROOT::Math::DefaultCoordinateSystemTag> const&, std::vector<edm::Ptr<reco::Candidate>, std::allocator<edm::Ptr<reco::Candidate> > > const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libDataFormatsJetReco.so
#6  0x00007fa5f0132307 in reco::writeSpecific(reco::BasicJet&, ROOT::Math::LorentzVector<ROOT::Math::PxPyPzE4D<double> > const&, ROOT::Math::PositionVector3D<ROOT::Math::Cartesian3D<double>, ROOT::Math::DefaultCoordinateSystemTag> const&, std::vector<edm::Ptr<reco::Candidate>, std::allocator<edm::Ptr<reco::Candidate> > > const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libRecoJetsJetProducers.so
#7  0x00007fa5f0215604 in void VirtualJetProducer::writeJets<reco::BasicJet>(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginRecoJetsJetProducers_plugins.so
#8  0x00007fa5f01fe53d in VirtualJetProducer::output(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginRecoJetsJetProducers_plugins.so
#9  0x00007fa5f01ff7e3 in VirtualJetProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginRecoJetsJetProducers_plugins.so
#10 0x00007fa5f0272a37 in FastjetJetProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginRecoJetsJetProducers_plugins.so
#11 0x00007fa6226786a3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#12 0x00007fa6225a2de2 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#13 0x00007fa622557307 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#14 0x00007fa6225574db in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#15 0x00007fa6225589fb in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#16 0x00007fa622558fa5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#17 0x00007fa6211d7a43 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fa5de407e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#18 0x00007fa6211d0a92 in tbb::internal::arena::process (this=0x7fa61dcfb780, s=...) at ../../src/tbb/arena.cpp:159
#19 0x00007fa6211cf57b in tbb::internal::market::process (this=0x7fa61dd17e00, j=...) at ../../src/tbb/market.cpp:697
#20 0x00007fa6211cb826 in tbb::internal::rml::private_worker::run (this=0x7fa61dd0f980) at ../../src/tbb/private_server.cpp:271
#21 0x00007fa6211cba59 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:224
#22 0x00007fa61fe8bdc5 in start_thread () from /lib64/libpthread.so.0
#23 0x00007fa61fbb8ced in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fa5da6fe700 (LWP 22793)):
#0  0x00007fa61fb7fefd in nanosleep () from /lib64/libc.so.6
#1  0x00007fa61fb7fd94 in sleep () from /lib64/libc.so.6
#2  0x00007fa61a059c8e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  std::max<int> (__b=<synthetic pointer>: <optimized out>, __a=<synthetic pointer>: <optimized out>) at /build/cmsbld/auto-builds/CMSSW_9_0_0_pre6-slc7_amd64_gcc630/build/CMSSW_9_0_0_pre6-build/slc7_amd64_gcc630/external/gcc/6.3.0/include/c++/6.3.0/bits/stl_algobase.h:224
#5  fastjet::ClusterSequence::_do_ij_recombination_step (this=0x7fa5da6f7b60, jet_i=<optimized out>, jet_j=<optimized out>, dij=1.0125935264371677e+197, newjet_k=@0x7fa5da6f75fc: 7559) at ClusterSequence.cc:1702
#6  0x00007fa5f03b1570 in fastjet::ClusterSequence::plugin_record_ij_recombination (newjet_k=@0x7fa5da6f75fc: 7559, dij=<optimized out>, jet_j=<optimized out>, jet_i=<optimized out>, this=<optimized out>) at ./../include/fastjet/ClusterSequence.hh:319
#7  fastjet::LazyTiling9::run (this=this@entry=0x7fa5da6f7780) at LazyTiling9.cc:652
#8  0x00007fa5f0370c5a in fastjet::ClusterSequence::_initialise_and_run_no_decant (this=this@entry=0x7fa5da6f7b60) at ClusterSequence.cc:384
#9  0x00007fa5f0371242 in fastjet::ClusterSequence::_initialise_and_run (this=this@entry=0x7fa5da6f7b60, jet_def_in=..., writeout_combinations=@0x7fa5da6f7aa0: false) at ClusterSequence.cc:228
#10 0x00007fa5f0429140 in fastjet::ClusterSequenceActiveAreaExplicitGhosts::_initialise<fastjet::PseudoJet> (this=0x7fa5da6f7b60, pseudojets=..., jet_def_in=..., ghost_spec=0x7fa5af4c5228, ghosts=0x0, ghost_area=0, writeout_combinations=@0x7fa5da6f7aa0: false) at ./../include/fastjet/ClusterSequenceActiveAreaExplicitGhosts.hh:227
#11 0x00007fa5f039c3c2 in fastjet::ClusterSequenceActiveAreaExplicitGhosts::ClusterSequenceActiveAreaExplicitGhosts<fastjet::PseudoJet> (writeout_combinations=@0x7fa5da6f7aa0: false, ghost_spec=..., jet_def_in=..., pseudojets=..., this=0x7fa5da6f7b60) at ./../include/fastjet/ClusterSequenceActiveAreaExplicitGhosts.hh:69
#12 fastjet::ClusterSequenceActiveArea::_run_AA (this=this@entry=0x7fa5af4f7380, ghost_spec=...) at ClusterSequenceActiveArea.cc:139
#13 0x00007fa5f039c738 in fastjet::ClusterSequenceActiveArea::_initialise_and_run_AA (this=this@entry=0x7fa5af4f7380, jet_def_in=..., ghost_spec=..., writeout_combinations=@0x7fa5da6f7da0: false) at ClusterSequenceActiveArea.cc:67
#14 0x00007fa5f0429ad1 in fastjet::ClusterSequenceActiveArea::ClusterSequenceActiveArea<fastjet::PseudoJet> (writeout_combinations=@0x7fa5da6f7da0: false, ghost_spec=..., jet_def_in=..., pseudojets=..., this=0x7fa5af4f7380) at ./../include/fastjet/ClusterSequenceActiveArea.hh:217
#15 fastjet::ClusterSequenceArea::initialize_and_run_cswa<fastjet::PseudoJet> (this=0x7fa5af4c5100, pseudojets=..., jet_def_in=...) at ./../include/fastjet/ClusterSequenceArea.hh:222
#16 0x00007fa5f02706ee in FastjetJetProducer::runAlgorithm(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginRecoJetsJetProducers_plugins.so
#17 0x00007fa5f01ff761 in VirtualJetProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginRecoJetsJetProducers_plugins.so
#18 0x00007fa5f0272a37 in FastjetJetProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginRecoJetsJetProducers_plugins.so
#19 0x00007fa6226786a3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#20 0x00007fa6225a2de2 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#21 0x00007fa622557307 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#22 0x00007fa6225574db in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#23 0x00007fa6225589fb in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#24 0x00007fa622558fa5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#25 0x00007fa6211d7a43 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fa5de42fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#26 0x00007fa6211d0a92 in tbb::internal::arena::process (this=0x7fa61dcfb780, s=...) at ../../src/tbb/arena.cpp:159
#27 0x00007fa6211cf57b in tbb::internal::market::process (this=0x7fa61dd17e00, j=...) at ../../src/tbb/market.cpp:697
#28 0x00007fa6211cb826 in tbb::internal::rml::private_worker::run (this=0x7fa61dd0fa80) at ../../src/tbb/private_server.cpp:271
#29 0x00007fa6211cba59 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:224
#30 0x00007fa61fe8bdc5 in start_thread () from /lib64/libpthread.so.0
#31 0x00007fa61fbb8ced in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fa5db0ff700 (LWP 22792)):
#0  0x00007fa61fb7fefd in nanosleep () from /lib64/libc.so.6
#1  0x00007fa61fb7fd94 in sleep () from /lib64/libc.so.6
#2  0x00007fa61a059c8e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fa61a08635d in boost::detail::lcast_ret_unsigned<std::char_traits<char>, unsigned long, char>::convert() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#5  0x00007fa61a084613 in edm::service::ProcInfoFetcher::fetch() const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#6  0x00007fa61a08798d in edm::service::SimpleMemoryCheck::fetch() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#7  0x00007fa61a087d46 in edm::service::SimpleMemoryCheck::update() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#8  0x00007fa61a08ab6b in edm::service::SimpleMemoryCheck::updateAndPrint(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#9  0x00007fa61a08b547 in edm::service::SimpleMemoryCheck::postModule(edm::StreamContext const&, edm::ModuleCallingContext const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#10 0x00007fa62267870b in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#11 0x00007fa6225a2de2 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#12 0x00007fa622557307 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#13 0x00007fa6225574db in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#14 0x00007fa6225589fb in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#15 0x00007fa622558fa5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#16 0x00007fa6211d7a43 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fa5de40fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#17 0x00007fa6211d0a92 in tbb::internal::arena::process (this=0x7fa61dcfb780, s=...) at ../../src/tbb/arena.cpp:159
#18 0x00007fa6211cf57b in tbb::internal::market::process (this=0x7fa61dd17e00, j=...) at ../../src/tbb/market.cpp:697
#19 0x00007fa6211cb826 in tbb::internal::rml::private_worker::run (this=0x7fa61dd0fa00) at ../../src/tbb/private_server.cpp:271
#20 0x00007fa6211cba59 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:224
#21 0x00007fa61fe8bdc5 in start_thread () from /lib64/libpthread.so.0
#22 0x00007fa61fbb8ced in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fa60ce3f700 (LWP 22753)):
#0  0x00007fa61fe92ca9 in waitpid () from /lib64/libpthread.so.0
#1  0x00007fa61a059f17 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007fa61a05a4b5 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007fa620474c3f in std::execute_native_thread_routine (__p=0x7fa61d467fc0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007fa61fe8bdc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fa61fbb8ced in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fa61e28dc80 (LWP 22732)):
#0  0x00007fa61fbae69d in poll () from /lib64/libc.so.6
#1  0x00007fa61a05a304 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007fa61a05aada in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007fa61a05bb35 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fa5fe10e9e0 in DCCEBTCCBlock::addTriggerPrimitivesToCollection() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libEventFilterEcalRawToDigi.so
#6  0x00007fa5fe141a7b in DCCTCCBlock::unpack(unsigned long const**, unsigned int*, short) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libEventFilterEcalRawToDigi.so
#7  0x00007fa5fe104eff in DCCEBEventBlock::unpack(unsigned long const*, unsigned long, unsigned int) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libEventFilterEcalRawToDigi.so
#8  0x00007fa5fe19200a in EcalRawToDigi::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/pluginEventFilterEcalRawToDigiPlugins.so
#9  0x00007fa6226786a3 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#10 0x00007fa6225a2de2 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#11 0x00007fa622557307 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#12 0x00007fa6225574db in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#13 0x00007fa6225589fb in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#14 0x00007fa622558fa5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#15 0x00007fa6211d7a43 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fa61dd16400, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:501
#16 0x00007fa6225e81ec in edm::EventProcessor::readAndProcessEvents() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#17 0x00007fa6225f2b41 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc630/cms/cmssw/CMSSW_9_3_DEVEL_X_2017-08-08-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#18 0x000000000040e6f2 in main::{lambda()#1}::operator()() const ()
#19 0x000000000040d0d5 in main ()

Current Modules:

Module: EcalRawToDigi:ecalDigis (crashed)%MSG-w MemoryCheck:  CaloTowersCreator:caloTowerForTrkPreSplitting  09-Aug-2017 05:18:42 CEST Run: 1 Event: 6
MemoryCheck: module CaloTowersCreator:caloTowerForTrkPreSplitting VSIZE 2212.22 0 RSS 1392.93 0.0273438
%MSG

Module: FastjetJetProducer:ak5CastorJets
Module: CaloTowersCreator:caloTowerForTrkPreSplitting
Module: FastjetJetProducer:ak4PFJets

A fatal system signal has occurred: segmentation violation

[cmsbuild]

A new Issue was created by @dan131riley Dan Riley.

@davidlange6, @Dr15Jones, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[Dr15Jones]

assign reconstruction

[cmsbuild]

New categories assigned: reconstruction

@slava77,@perrotta you have been requested to review this Pull request/Issue and eventually sign? Thanks

[davidlange6]

it looks like it is also a likely cause of problems in IB comparison tests
(#19939) - will revert for now

[Dr15Jones]

The crash probably happens inside the loop

cmssw/EventFilter/EcalRawToDigi/src/DCCEBTCCBlock.cc

Line 72 in 09c3fce

pTP_->setSampleValue(ns, (*tccP_));

There are three pointers whare get dereferenced: pTP_, tccP_ (which is a local variable even thought there is an underscore in its name) and tps_.

pTP_ gets its value from mapper which appears to be this code

cmssw/EventFilter/EcalRawToDigi/interface/EcalElectronicsMapper.h

Line 102 in 09c3fce

EcalTriggerPrimitiveDigi * getTPPointer(unsigned int tccId, unsigned int tower){ return ttTPIds_[tccId-1][tower-1];}

If the array arguments are out of bounds, it could lead to a crash.

tccP_ is just data_ recast. So to cause a crash, data_ would have to point to invalid memory. The value of data_ is set by the function call one up on the stack

cmssw/EventFilter/EcalRawToDigi/src/DCCTCCBlock.cc

Line 15 in 09c3fce

data_ = *data;

tps_ is an object returned from the unpacker

cmssw/EventFilter/EcalRawToDigi/src/DCCEBTCCBlock.cc

Line 14 in 09c3fce

tps_ = unpacker_->ecalTpsCollection();

cmssw/EventFilter/EcalRawToDigi/interface/DCCDataUnpacker.h

Line 143 in 09c3fce

{ return ecalTps_; }

[Aside: return a pointer to a std::unique_ptr<> is not a very good idea since it is way to easy to accidently transfer ownership.]

[slava77]

@dan131riley @Dr15Jones
is this still an issue?

[dan131riley]

I don't think it is still an issue. The crashes were resolved by reverting #18322, and the underlying issue of making EcalDigiToRaw thread safe was resolved by Chris in #24105