Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

81X Workaround MT problem in HIPixelMedianVtxProducer #14724

Merged

Conversation

Martin-Grunewald
Copy link
Contributor

81X Workaround MT problem in HIPixelMedianVtxProducer

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 1, 2016

A new Pull Request was created by @Martin-Grunewald (Martin Grunewald) for CMSSW_8_1_X.

It involves the following packages:

RecoHI/HiTracking

@cmsbuild, @cvuosalo, @slava77, @davidlange6 can you please review it and eventually sign? Thanks.
@jazzitup, @dgulhan, @echapon, @yenjie, @kurtejung, @mandrenguyen, @richard-cms, @yetkinyilmaz this is something you requested to watch as well.
@slava77, @Degano, @smuzaffar you are the release manager for this.

cms-bot commands are list here #13028

@Martin-Grunewald
Copy link
Contributor Author

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 1, 2016

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/13304/console

@Martin-Grunewald
Copy link
Contributor Author

@Dr15Jones
Hi Chris, you might have a look, as the fix you provided some time ago to keep this as a stream module did not work in my tests (we moved TSG tests of the HLT menu to multithreading).

A typical traceback is this one, a common feature to the crashes observed seems to be the destructor sequence in the last thread - Thread 1.
Unfortunately the crashes are not always reproducible, but they disappear in case the
module is made a one module...

%MSG-i ThreadSetup:  (NoModuleName) 01-Jun-2016 10:56:29 CEST pre-events
setting # threads 4
%MSG
Warning in <TInterpreter::ReadRootmapFile>: class  edm::Ref<edmNew::DetSetVector<Phase2TrackerCluster1D>,Phase2TrackerCluster1D,edmNew::DetSetVector<Phase2TrackerCluster1D>::FindForDetSetVector> found in libDataFormatsPhase2TrackerCluster.so  is already in libDataFormatsTrackerRecHit2D.so 
Warning in <TInterpreter::ReadRootmapFile>: class  edm::Ref<edmNew::DetSetVector<Phase2TrackerCluster1D>,Phase2TrackerCluster1D> found in libDataFormatsPhase2TrackerCluster.so  is already in libDataFormatsTrackerRecHit2D.so 
Warning in <TInterpreter::ReadRootmapFile>: class  edm::Wrapper<edm::Ref<edmNew::DetSetVector<Phase2TrackerCluster1D>,Phase2TrackerCluster1D,edmNew::DetSetVector<Phase2TrackerCluster1D>::FindForDetSetVector> > found in libDataFormatsPhase2TrackerCluster.so  is already in libDataFormatsTrackerRecHit2D.so 
Warning in <TInterpreter::ReadRootmapFile>: class  edm::Wrapper<edm::Ref<edmNew::DetSetVector<Phase2TrackerCluster1D>,Phase2TrackerCluster1D> > found in libDataFormatsPhase2TrackerCluster.so  is already in libDataFormatsTrackerRecHit2D.so 
DQMStore: MultiThread option is enabled
01-Jun-2016 10:56:38 CEST  Initiating request to open file file:../RelVal_Raw_HIon_MC.root
01-Jun-2016 10:56:39 CEST  Successfully opened file file:../RelVal_Raw_HIon_MC.root
isBH_ 18:13:18:0


A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 5 (Thread 0x7f0d937ff700 (LWP 6235)):
#0  0x000000394980f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f0da3219667 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f0da3219f12 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f0da577ba50 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
#4  0x0000003949807aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000039494e8aad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f0d7593f700 (LWP 6335)):
#0  0x000000394980e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000039498095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#2  0x00000039498094d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f0da5fddf1b in TClass::GetClass(std::type_info const&, bool, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-05-31-2300/external/slc6_amd64_gcc530/lib/libCore.so
#4  0x00007f0da67f9ff8 in edm::TypeWithDict::TypeWithDict(std::type_info const&, long) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#5  0x00007f0da67e2cc0 in edm::ObjectWithDict::dynamicType() const () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreUtilities.so
#6  0x00007f0d90a1e26f in reco::parser::LazyInvoker::invokeLast(edm::ObjectWithDict const&, std::vector<edm::ObjectWithDict, std::allocator<edm::ObjectWithDict> >&) const () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#7  0x00007f0d90a25e2f in reco::parser::ExpressionLazyVar::value(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#8  0x00007f0d90a48689 in reco::parser::BinarySelector::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#9  0x00007f0d90a6e816 in reco::parser::LogicalBinaryOperator<std::logical_and<bool> >::operator()(edm::ObjectWithDict const&) const () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libCommonToolsUtils.so
#10 0x00007f0d7f33c611 in ObjectSelector<SingleElementCollectionSelector<edm::View<reco::Candidate>, StringCutObjectSelector<reco::Candidate, true>, edm::PtrVector<reco::Candidate>, edm::PtrVector<reco::Candidate>, helper::SelectionPtrViewAdder<reco::Candidate> >, edm::PtrVector<reco::Candidate>, NonNullNumberSelector, helper::NullPostProcessor<edm::PtrVector<reco::Candidate>, edm::stream::EDFilter<> >, helper::CollectionStoreManager<edm::PtrVector<reco::Candidate>, helper::IteratorToObjectConverter<edm::PtrVector<reco::Candidate> > >, helper::ObjectSelectorBase<edm::PtrVector<reco::Candidate>, edm::stream::EDFilter<> >, reco::modules::SingleElementCollectionSelectorEventSetupInit<SingleElementCollectionSelector<edm::View<reco::Candidate>, StringCutObjectSelector<reco::Candidate, true>, edm::PtrVector<reco::Candidate>, edm::PtrVector<reco::Candidate>, helper::SelectionPtrViewAdder<reco::Candidate> > > >::filter(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/pluginCommonToolsCandAlgos_plugins.so
#11 0x00007f0da6bc1d55 in edm::stream::EDFilterAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f0da6b22eef in edm::WorkerT<edm::stream::EDFilterAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f0da6ae115c in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f0da6ae1322 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f0da6b156ee in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f0da6b159a2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f0da6b15dc5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f0da6b160bb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f0da6b1632f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f0da6b7faf2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#21 0x00007f0da6b80227 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f0da6b868f4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f0da58a69b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f0da4b07e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#24 0x00007f0da58a01a4 in tbb::internal::arena::process (this=this@entry=0x7f0da4bff780, s=...) at ../../src/tbb/arena.cpp:156
#25 0x00007f0da589f264 in tbb::internal::market::process (this=0x7f0da4c27e00, j=...) at ../../src/tbb/market.cpp:502
#26 0x00007f0da589b736 in tbb::internal::rml::private_worker::run (this=0x7f0da4c1fa00) at ../../src/tbb/private_server.cpp:275
#27 0x00007f0da589b989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#28 0x0000003949807aa1 in start_thread () from /lib64/libpthread.so.0
#29 0x00000039494e8aad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f0d74f3e700 (LWP 6336)):
#0  0x000000394980e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000039498095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#2  0x00000039498094d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f0da6b90d3d in edm::SharedResourcesAcquirer::lock() () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#4  0x00007f0da6af25f3 in edm::DelayedReader::getProduct(edm::BranchKey const&, edm::EDProductGetter const*, edm::signalslot::Signal<void (edm::StreamContext const&, edm::ModuleCallingContext const&)> const*, edm::signalslot::Signal<void (edm::StreamContext const&, edm::ModuleCallingContext const&)> const*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#5  0x00007f0da6bb2554 in edm::EventPrincipal::readFromSource_(edm::ProductResolverBase const&, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#6  0x00007f0da6add110 in edm::InputProductResolver::resolveProduct_(edm::ProductResolverBase::ResolveStatus&, edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#7  0x00007f0da6b88791 in edm::Principal::prefetch(unsigned int, bool, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#8  0x00007f0da6ae1137 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9  0x00007f0da6ae1322 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007f0da6b156ee in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f0da6b159a2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f0da6b15edc in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f0da6b160bb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f0da6b1632f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f0da6b7faf2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f0da6b80227 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f0da6b868f4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f0da58a69b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f0da4b17e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#19 0x00007f0da58a01a4 in tbb::internal::arena::process (this=this@entry=0x7f0da4bff780, s=...) at ../../src/tbb/arena.cpp:156
#20 0x00007f0da589f264 in tbb::internal::market::process (this=0x7f0da4c27e00, j=...) at ../../src/tbb/market.cpp:502
#21 0x00007f0da589b736 in tbb::internal::rml::private_worker::run (this=0x7f0da4c1fa80) at ../../src/tbb/private_server.cpp:275
#22 0x00007f0da589b989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#23 0x0000003949807aa1 in start_thread () from /lib64/libpthread.so.0
#24 0x00000039494e8aad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f0d741bf700 (LWP 6337)):
#0  0x000000394980e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00000039498095f3 in _L_lock_892 () from /lib64/libpthread.so.0
#2  0x00000039498094d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f0da5fe5beb in TClass::GetClass(char const*, bool, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-05-31-2300/external/slc6_amd64_gcc530/lib/libCore.so
#4  0x00007f0da6d8e049 in edm::setRefCoreStreamer(edm::EDProductGetter const*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libDataFormatsCommon.so
#5  0x00007f0d8bea664a in edm::RootDelayedReader::getProduct_(edm::BranchKey const&, edm::EDProductGetter const*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/pluginIOPoolInput.so
#6  0x00007f0da6af2562 in edm::DelayedReader::getProduct(edm::BranchKey const&, edm::EDProductGetter const*, edm::signalslot::Signal<void (edm::StreamContext const&, edm::ModuleCallingContext const&)> const*, edm::signalslot::Signal<void (edm::StreamContext const&, edm::ModuleCallingContext const&)> const*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#7  0x00007f0da6bb2554 in edm::EventPrincipal::readFromSource_(edm::ProductResolverBase const&, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#8  0x00007f0da6add110 in edm::InputProductResolver::resolveProduct_(edm::ProductResolverBase::ResolveStatus&, edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#9  0x00007f0da6b88791 in edm::Principal::prefetch(unsigned int, bool, edm::ModuleCallingContext const*) const () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#10 0x00007f0da6ae1137 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#11 0x00007f0da6ae1322 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#12 0x00007f0da6b156ee in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#13 0x00007f0da6b159a2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#14 0x00007f0da6b15edc in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#15 0x00007f0da6b160bb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#16 0x00007f0da6b1632f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#17 0x00007f0da6b7faf2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#18 0x00007f0da6b80227 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#19 0x00007f0da6b868f4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#20 0x00007f0da58a69b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f0da4b0fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#21 0x00007f0da58a01a4 in tbb::internal::arena::process (this=this@entry=0x7f0da4bff780, s=...) at ../../src/tbb/arena.cpp:156
#22 0x00007f0da589f264 in tbb::internal::market::process (this=0x7f0da4c27e00, j=...) at ../../src/tbb/market.cpp:502
#23 0x00007f0da589b736 in tbb::internal::rml::private_worker::run (this=0x7f0da4c1f980) at ../../src/tbb/private_server.cpp:275
#24 0x00007f0da589b989 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:228
#25 0x0000003949807aa1 in start_thread () from /lib64/libpthread.so.0
#26 0x00000039494e8aad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f0da5417bc0 (LWP 6187)):
#0  0x00000039494df283 in poll () from /lib64/libc.so.6
#1  0x00007f0da3219d74 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#2  0x00007f0da3219fda in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#3  0x00007f0da321a15b in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f0da5f8d7b2 in TList::FindLink(TObject const*, int&) const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-05-31-2300/external/slc6_amd64_gcc530/lib/libCore.so
#6  0x00007f0da5f8d8ec in TList::Remove(TObject*) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-05-31-2300/external/slc6_amd64_gcc530/lib/libCore.so
#7  0x00007f0d8ffe545b in TMinuit::~TMinuit() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-05-31-2300/external/slc6_amd64_gcc530/lib/libMinuit.so
#8  0x00007f0d8ffe55d9 in TMinuit::~TMinuit() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-05-31-2300/external/slc6_amd64_gcc530/lib/libMinuit.so
#9  0x00007f0d8ffea116 in TMinuitMinimizer::~TMinuitMinimizer() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-05-31-2300/external/slc6_amd64_gcc530/lib/libMinuit.so
#10 0x00007f0d8ffea139 in TMinuitMinimizer::~TMinuitMinimizer() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-05-31-2300/external/slc6_amd64_gcc530/lib/libMinuit.so
#11 0x00007f0da64589f2 in ROOT::Fit::FitResult::~FitResult() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-05-31-2300/external/slc6_amd64_gcc530/lib/libMathCore.so
#12 0x00007f0da2fd1c3b in std::_Sp_counted_ptr<TFitResult*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-05-31-2300/external/slc6_amd64_gcc530/lib/libHist.so
#13 0x00007f0da64639fa in ROOT::Fit::Fitter::~Fitter() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-05-31-2300/external/slc6_amd64_gcc530/lib/libMathCore.so
#14 0x00007f0da2fd4492 in std::_Sp_counted_ptr<ROOT::Fit::Fitter*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-05-31-2300/external/slc6_amd64_gcc530/lib/libHist.so
#15 0x00007f0da309e0aa in TBackCompFitter::~TBackCompFitter() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-05-31-2300/external/slc6_amd64_gcc530/lib/libHist.so
#16 0x00007f0da309e0f9 in TBackCompFitter::~TBackCompFitter() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-05-31-2300/external/slc6_amd64_gcc530/lib/libHist.so
#17 0x00007f0da2fda189 in TFitResultPtr HFit::Fit<TH1>(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-05-31-2300/external/slc6_amd64_gcc530/lib/libHist.so
#18 0x00007f0da2fd3ee5 in ROOT::Fit::FitObject(TH1*, TF1*, Foption_t&, ROOT::Math::MinimizerOptions const&, char const*, ROOT::Fit::DataRange&) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-05-31-2300/external/slc6_amd64_gcc530/lib/libHist.so
#19 0x00007f0da305e83b in TH1::Fit(TF1*, char const*, char const*, double, double) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc530/cms/cmssw-patch/CMSSW_8_1_X_2016-05-31-2300/external/slc6_amd64_gcc530/lib/libHist.so
#20 0x00007f0d7f11a6ae in HIPixelMedianVtxProducer::produce(edm::Event&, edm::EventSetup const&) () from /data/CMS1/CMSSW_8_1_X_2016-05-31-2300/lib/slc6_amd64_gcc530/pluginRecoHIHiTrackingPlugins.so
#21 0x00007f0da6bbff52 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetup const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#22 0x00007f0da6b2301f in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#23 0x00007f0da6ae115c in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#24 0x00007f0da6ae1322 in bool edm::Worker::doWork<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#25 0x00007f0da6b156ee in decltype ({parm#1}()) edm::convertException::wrap<void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#26 0x00007f0da6b159a2 in void edm::Path::processOneOccurrence<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#27 0x00007f0da6b15dc5 in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#28 0x00007f0da6b160bb in decltype ({parm#1}()) edm::convertException::wrap<void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}>(void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#29 0x00007f0da6b1632f in void edm::StreamSchedule::processOneEvent<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#30 0x00007f0da6b7faf2 in edm::EventProcessor::processEvent(unsigned int) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#31 0x00007f0da6b80227 in edm::EventProcessor::processEventsForStreamAsync(unsigned int, std::atomic<bool>*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#32 0x00007f0da6b868f4 in edm::StreamProcessingTask::execute() () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#33 0x00007f0da58a69b5 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f0da4c26400, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:463
#34 0x00007f0da6b8007e in edm::EventProcessor::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#35 0x00007f0da6ae8a9a in statemachine::HandleEvent::readAndProcessEvent() () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#36 0x00007f0da6aea028 in statemachine::HandleEvent::HandleEvent(boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#37 0x00007f0da6af1d6e in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct(boost::intrusive_ptr<statemachine::HandleLumis> const&, boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>&) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#38 0x00007f0da6af1f3a in boost::statechart::simple_state<statemachine::FirstLumi, statemachine::HandleLumis, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#39 0x00007f0da6b863a8 in boost::statechart::state_machine<statemachine::Machine, statemachine::Starting, std::allocator<void>, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&) () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#40 0x00007f0da6b797ad in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/2016-23/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_1_X_2016-05-30-2300/lib/slc6_amd64_gcc530/libFWCoreFramework.so
#41 0x000000000040c7f4 in main::{lambda()#1}::operator()() const ()
#42 0x000000000040ac74 in main ()

A fatal system signal has occurred: segmentation violation

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 1, 2016

@Dr15Jones
Copy link
Contributor

We are seeing a similar crash in the IB. We think it is a bug in ROOT. Possibly running valgrind using the helgrind tool might pinpoint the problem. [I'm on vacation for the next week.]

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 1, 2016

@slava77
Copy link
Contributor

slava77 commented Jun 1, 2016

valgrind with HI code may have issues if it links to CGAL. There is a bug in valgrind.
::one seems like an easier solution.
Martin, do you happen to know how much we loose in efficiency in HLT because of that?

@Martin-Grunewald
Copy link
Contributor Author

Martin-Grunewald commented Jun 1, 2016

Sorry, no - but somewhat less timing efficiency is better than crashing!
Of course, even with one for this plugin we still gain due to using MT!!

@cvuosalo
Copy link
Contributor

cvuosalo commented Jun 1, 2016

+1

For #14724 cb5eb7c

Converting HIPixelMedianVtxProducer from edm::stream to edm::one to avoid a very elusive multi-threading bug. There should be no change in monitored quantities.

#14725 is the 80X version of this PR.

The code change is satisfactory, and Jenkins tests against baseline CMSSW_8_1_X_2016-05-31-2300 show no significant differences, as expected.

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 1, 2016

This pull request is fully signed and it will be integrated in one of the next CMSSW_8_1_X IBs (tests are also fine). This pull request requires discussion in the ORP meeting before it's merged. @slava77, @davidlange6, @Degano, @smuzaffar

@davidlange6
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit 6aa19bd into cms-sw:CMSSW_8_1_X Jun 2, 2016
@Martin-Grunewald Martin-Grunewald deleted the 81XFixHIPixelMedianVtxProducer branch June 6, 2016 04:28
@dan131riley
Copy link

The crash is from a data race in the TMinuit constructor, fix is here:

https://sft.its.cern.ch/jira/browse/ROOT-8217

valgrind can be run on HI code by defining CGAL_DISABLE_ROUNDING_MATH_CHECK in RecoHI/HiJetAlgos/plugins/VoronoiAlgorithm.h (valgrind doesn't emulate the different rounding modes). However, helgrind doesn't find the problem, as it is ordered by a mutex (just ordered incorrectly), so it had to be found via more labor-intensive means.

@Martin-Grunewald
Copy link
Contributor Author

@dan131riley
Amazing, thanks for tracking this down.
So once this is CMS's version of ROOT, we can restore it to a stream module!
When will that be the case, for 80 and 81?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants