Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Seg Fault seen in DTTSTheta::setConfig in production #44999

Open
Dr15Jones opened this issue May 17, 2024 · 9 comments
Open

Seg Fault seen in DTTSTheta::setConfig in production #44999

Dr15Jones opened this issue May 17, 2024 · 9 comments
Labels
l1-pending pending-signatures

Comments

@Dr15Jones
Copy link
Contributor

Production jobs running CMSSW_10_6_31 running SMP-RunIISummer20UL18wmLHEGEN for the step reading pileup mixing files is seg faulting in DTTSTheta::setConfig(DTConfigManager const*)
@cmsbuild
Copy link
Contributor

cmsbuild commented May 17, 2024 •
edited
Loading

cms-bot internal usage

@cmsbuild
Copy link
Contributor

A new Issue was created by @Dr15Jones.

@makortel, @smuzaffar, @Dr15Jones, @rappoccio, @antoniovilela, @sextonkennedy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@makortel
Copy link
Contributor

assign L1Trigger/DTTriggerServerTheta

@cmsbuild
Copy link
Contributor

New categories assigned: l1

@epalencia,@aloeliger you have been requested to review this Pull request/Issue and eventually sign? Thanks

@Dr15Jones
Copy link
Contributor Author

The full traceback is

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Mon May 13 21:11:02 UTC 2024
Thread 14 (Thread 0x1456c0800700 (LWP 9105)):
[xrootd]

Thread 13 (Thread 0x1456c0c00700 (LWP 9104)):
[xrootd]

Thread 12 (Thread 0x1456c1000700 (LWP 9103)):
[xrootd]

Thread 11 (Thread 0x1456c1400700 (LWP 9102)):
[xrootd]

Thread 10 (Thread 0x1456c1800700 (LWP 9101)):
[xrootd]

Thread 9 (Thread 0x1456cc600700 (LWP 9100)):
#4  0x00001456f77ed3b6 in ?? () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libz.so.1
#5  0x00001456f77eee37 in inflate () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libz.so.1
#6  0x00001456f7ad73c5 in R__unzip () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libCore.so
#7  0x00001456f86a8336 in TBasket::ReadBasketBuffers(long long, int, TFile*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#8  0x00001456f86adcaa in TBranch::GetBasket(int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#9  0x00001456f86ae480 in TBranch::GetEntry(long long, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#10 0x00001456f86c42e9 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#11 0x00001456f86c41b6 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#12 0x00001456d52b842f in edm::RootTree::getEntry(TBranch*, long long) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginIOPoolInput.so
#13 0x00001456d52c2c29 in edm::RootDelayedReader::getProduct_(edm::BranchID const&, edm::EDProductGetter const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginIOPoolInput.so
#14 0x00001456f895d4c7 in edm::DelayedReader::getProduct(edm::BranchID const&, edm::EDProductGetter const*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#15 0x00001456f8942a46 in edm::InputProductResolver::resolveProduct_(edm::Principal const&, bool, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#16 0x00001456f8a0232a in edm::Principal::findProductByLabel(edm::KindOfType, edm::TypeID const&, edm::InputTag const&, edm::EDConsumerBase const*, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#17 0x00001456f8a027c1 in edm::Principal::getByLabel(edm::KindOfType, edm::TypeID const&, edm::InputTag const&, edm::EDConsumerBase const*, edm::SharedResourcesAcquirer*, edm::ModuleCallingContext const*) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#18 0x00001456c842b62f in PreMixingSiStripWorker::addPileups(PileUpEventPrincipal const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiStripDigitizerPlugins.so
#19 0x00001456c83814e5 in edm::PreMixingModule::pileWorker(edm::EventPrincipal const&, int, int, edm::EventSetup const&, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
#20 0x00001456c838701b in unsigned long edm::VectorInputSource::loopOverEvents<edm::RecordEventID<std::_Bind<bool (edm::PreMixingModule::*(std::reference_wrapper<edm::PreMixingModule>, std::_Placeholder<1>, int, std::_Placeholder<2>, std::reference_wrapper<edm::EventSetup const>, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*))(edm::EventPrincipal const&, int, int, edm::EventSetup const&, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*)> > >(edm::EventPrincipal&, unsigned long&, unsigned long, edm::RecordEventID<std::_Bind<bool (edm::PreMixingModule::*(std::reference_wrapper<edm::PreMixingModule>, std::_Placeholder<1>, int, std::_Placeholder<2>, std::reference_wrapper<edm::EventSetup const>, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*))(edm::EventPrincipal const&, int, int, edm::EventSetup const&, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*)> >, CLHEP::HepRandomEngine*, edm::EventID const*, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
#21 0x00001456c8387175 in void edm::PileUp::readPileUp<std::_Bind<bool (edm::PreMixingModule::*(std::reference_wrapper<edm::PreMixingModule>, std::_Placeholder<1>, int, std::_Placeholder<2>, std::reference_wrapper<edm::EventSetup const>, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*))(edm::EventPrincipal const&, int, int, edm::EventSetup const&, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*)> >(edm::EventID const&, std::vector<edm::SecondaryEventIDAndFileInfo, std::allocator<edm::SecondaryEventIDAndFileInfo> >&, std::_Bind<bool (edm::PreMixingModule::*(std::reference_wrapper<edm::PreMixingModule>, std::_Placeholder<1>, int, std::_Placeholder<2>, std::reference_wrapper<edm::EventSetup const>, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*))(edm::EventPrincipal const&, int, int, edm::EventSetup const&, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*)>, int, edm::StreamID const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
#22 0x00001456c838316a in edm::PreMixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
#23 0x00001456c85dbb5d in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libMixingBase.so
...

Thread 8 (Thread 0x1456cde00700 (LWP 9099)):
#3  0x00001456e8d5c468 in sig_dostack_then_abort () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00001456c140497e in DTTSTheta::setConfig(DTConfigManager const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libL1TriggerDTTriggerServerTheta.so
#6  0x00001456c180cfe1 in DTSCTrigUnit::setConfig(DTConfigManager const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libL1TriggerDTSectorCollector.so
#7  0x00001456c144e338 in DTTrig::updateES(edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginL1TriggerDTTrigger.so
#8  0x00001456c144f7c4 in DTTrig::triggerReco(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginL1TriggerDTTrigger.so
#9  0x00001456c1448f48 in DTTrigProd::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginL1TriggerDTTrigger.so

Thread 7 (Thread 0x1456cee00700 (LWP 9098)):
#3  <signal handler called>
#4  0x00001456c8618380 in HcalSiPMShape::operator()(double) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libSimCalorimetryHcalSimAlgos.so
#5  0x00001456c8619e44 in HcalSiPMHitResponse::makeSiPMSignal(DetId const&, std::vector<unsigned int, std::allocator<unsigned int> > const&, CLHEP::HepRandomEngine*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libSimCalorimetryHcalSimAlgos.so
#6  0x00001456c8619782 in HcalSiPMHitResponse::finalizeHits(CLHEP::HepRandomEngine*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libSimCalorimetryHcalSimAlgos.so
#7  0x00001456c865f9d1 in HcalDigitizer::finalizeEvent(edm::Event&, edm::EventSetup const&, CLHEP::HepRandomEngine*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libSimCalorimetryHcalSimProducers.so
#8  0x00001456c864bea4 in HcalDigiProducer::finalizeEvent(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libSimCalorimetryHcalSimProducers.so
#9  0x00001456c837f33d in edm::PreMixingModule::put(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
...

Thread 6 (Thread 0x1456cfc00700 (LWP 9097)):
#3  <signal handler called>
#4  0x00001456f50f8e29 in syscall () from /lib64/libc.so.6
#5  0x00001456f7336b38 in std::__atomic_futex_unsigned_base::_M_futex_wait_until (this=<optimized out>, __addr=0x1456004e2ef0, __val=2147483648, __has_timeout=<optimized out>, __s=..., __ns=...) at ../../../../../libstdc++-v3/src/c++11/futex.cc:55
#6  0x00001456f2a3f200 in XrdFile::read(void*, unsigned long, long) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libUtilitiesXrdAdaptor.so
#7  0x00001456f1ec657a in StorageAccountProxy::read(void*, unsigned long, long) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libUtilitiesStorageFactory.so
#8  0x00001456f1ecf01a in LocalCacheFile::cache(long, long) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libUtilitiesStorageFactory.so
#9  0x00001456f1ecf5ce in LocalCacheFile::prefetch(IOPosBuffer const*, unsigned long) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libUtilitiesStorageFactory.so
#10 0x00001456f1ec6914 in StorageAccountProxy::prefetch(IOPosBuffer const*, unsigned long) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libUtilitiesStorageFactory.so
#11 0x00001456f53ebd9f in TStorageFactoryFile::ReadBuffers(char*, long long*, int*, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libIOPoolTFileAdaptor.so
#12 0x00001456f7f2d87d in TFileCacheRead::ReadBufferExtNormal(char*, long long, int, int&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libRIO.so
#13 0x00001456f7f2d25a in TFileCacheRead::ReadBuffer(char*, long long, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libRIO.so
#14 0x00001456f87287fa in TTreeCache::ReadBufferNormal(char*, long long, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#15 0x00001456f86a7ceb in TBasket::ReadBasketBuffers(long long, int, TFile*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#16 0x00001456f86adcaa in TBranch::GetBasket(int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#17 0x00001456f86ae480 in TBranch::GetEntry(long long, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#18 0x00001456f86c42e9 in TBranchElement::GetEntry(long long, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_6_17_patch1/external/slc7_amd64_gcc700/lib/libTree.so
#19 0x00001456d52b842f in edm::RootTree::getEntry(TBranch*, long long) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginIOPoolInput.so
#20 0x00001456d52e72cc in edm::RootFileEventFinder::getEventNumberOfEntry(long long) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginIOPoolInput.so
#21 0x00001456f8ce85c4 in edm::IndexIntoFile::fillUnsortedEventNumbers() const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libDataFormatsProvenance.so
#22 0x00001456f8ceb5ba in edm::IndexIntoFile::fillEventNumbersOrEntries(bool, bool) const () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libDataFormatsProvenance.so
#23 0x00001456d52e1f59 in edm::RootFile::validateFile(edm::InputType, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginIOPoolInput.so
#24 0x00001456d52e4244 in edm::RootFile::RootFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::ProcessConfiguration const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<edm::InputFile>, std::shared_ptr<edm::EventSkipperByID>, bool, int, int, unsigned int, unsigned int, int, edm::InputSource::ProcessingMode, edm::RunHelperBase*, bool, edm::ProductSelectorRules const&, edm::InputType, std::shared_ptr<edm::BranchIDListHelper>, std::shared_ptr<edm::ThinnedAssociationsHelper>, std::vector<edm::BranchID, std::allocator<edm::BranchID> > const*, std::shared_ptr<edm::DuplicateChecker>, bool, edm::ProcessHistoryRegistry&, std::vector<std::shared_ptr<edm::IndexIntoFile>, std::allocator<std::shared_ptr<edm::IndexIntoFile> > > const&, unsigned long, std::vector<edm::Hash<2>, std::allocator<edm::Hash<2> > >&, bool, bool, bool, bool, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginIOPoolInput.so
#25 0x00001456d52ee7f7 in edm::RootEmbeddedFileSequence::makeRootFile(std::shared_ptr<edm::InputFile>) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginIOPoolInput.so
#26 0x00001456d52a78c0 in edm::RootInputFileSequence::initTheFile(bool, bool, edm::InputSource*, char const*, edm::InputType) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginIOPoolInput.so
#27 0x00001456d52eddac in edm::RootEmbeddedFileSequence::readOneRandom(edm::EventPrincipal&, unsigned long&, CLHEP::HepRandomEngine*, edm::EventID const*, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginIOPoolInput.so
#28 0x00001456c8386fcf in unsigned long edm::VectorInputSource::loopOverEvents<edm::RecordEventID<std::_Bind<bool (edm::PreMixingModule::*(std::reference_wrapper<edm::PreMixingModule>, std::_Placeholder<1>, int, std::_Placeholder<2>, std::reference_wrapper<edm::EventSetup const>, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*))(edm::EventPrincipal const&, int, int, edm::EventSetup const&, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*)> > >(edm::EventPrincipal&, unsigned long&, unsigned long, edm::RecordEventID<std::_Bind<bool (edm::PreMixingModule::*(std::reference_wrapper<edm::PreMixingModule>, std::_Placeholder<1>, int, std::_Placeholder<2>, std::reference_wrapper<edm::EventSetup const>, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*))(edm::EventPrincipal const&, int, int, edm::EventSetup const&, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*)> >, CLHEP::HepRandomEngine*, edm::EventID const*, bool) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
#29 0x00001456c8387175 in void edm::PileUp::readPileUp<std::_Bind<bool (edm::PreMixingModule::*(std::reference_wrapper<edm::PreMixingModule>, std::_Placeholder<1>, int, std::_Placeholder<2>, std::reference_wrapper<edm::EventSetup const>, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*))(edm::EventPrincipal const&, int, int, edm::EventSetup const&, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*)> >(edm::EventID const&, std::vector<edm::SecondaryEventIDAndFileInfo, std::allocator<edm::SecondaryEventIDAndFileInfo> >&, std::_Bind<bool (edm::PreMixingModule::*(std::reference_wrapper<edm::PreMixingModule>, std::_Placeholder<1>, int, std::_Placeholder<2>, std::reference_wrapper<edm::EventSetup const>, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*))(edm::EventPrincipal const&, int, int, edm::EventSetup const&, edm::ModuleCallingContext const*, edm::PreMixingModule::AdjustPileupDistribution const*)>, int, edm::StreamID const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
#30 0x00001456c838316a in edm::PreMixingModule::doPileUp(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
#31 0x00001456c85dbb5d in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libMixingBase.so
...

Thread 5 (Thread 0x1456d0800700 (LWP 9096)):
#3  <signal handler called>
#4  je_tcache_bin_flush_small (tsd=tsd@entry=0x1456d07f9af0, tcache=tcache@entry=0x1456d07f9cb0, tbin=0x1456d07f9d50, binind=binind@entry=6, rem=100) at src/tcache.c:116
#5  0x00001456f74078a6 in tcache_dalloc_small (slow_path=false, binind=6, ptr=0x145677221d20, tcache=<optimized out>, tsd=<optimized out>) at include/jemalloc/internal/tcache_inlines.h:178
#6  arena_dalloc (slow_path=false, alloc_ctx=<synthetic pointer>, tcache=<optimized out>, ptr=0x145677221d20, tsdn=<optimized out>) at include/jemalloc/internal/arena_inlines_b.h:224
#7  idalloctm (slow_path=false, is_internal=false, alloc_ctx=<synthetic pointer>, tcache=<optimized out>, ptr=0x145677221d20, tsdn=<optimized out>) at include/jemalloc/internal/jemalloc_internal_inlines_c.h:118
#8  ifree (slow_path=false, tcache=<optimized out>, ptr=0x145677221d20, tsd=0x1456d07f9af0) at src/jemalloc.c:2222
#9  free (ptr=0x145677221d20) at src/jemalloc.c:2393
#10 0x00001456c8491871 in std::_Rb_tree<int, std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude>, std::_Select1st<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> >, std::less<int>, std::allocator<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> > >::_M_erase(std::_Rb_tree_node<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> >*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiPixelDigitizerPlugins.so
#11 0x00001456c84917f3 in std::_Rb_tree<int, std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude>, std::_Select1st<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> >, std::less<int>, std::allocator<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> > >::_M_erase(std::_Rb_tree_node<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> >*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiPixelDigitizerPlugins.so
#12 0x00001456c84917f3 in std::_Rb_tree<int, std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude>, std::_Select1st<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> >, std::less<int>, std::allocator<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> > >::_M_erase(std::_Rb_tree_node<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> >*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiPixelDigitizerPlugins.so
#13 0x00001456c84917f3 in std::_Rb_tree<int, std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude>, std::_Select1st<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> >, std::less<int>, std::allocator<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> > >::_M_erase(std::_Rb_tree_node<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> >*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiPixelDigitizerPlugins.so
#14 0x00001456c84a2400 in SiPixelDigitizerAlgorithm::hitSignalReweight(PSimHit const&, std::map<int, float, std::less<int>, std::allocator<std::pair<int const, float> > >&, unsigned long, unsigned int, PixelTopology const*, unsigned int, std::map<int, SiPixelDigitizerAlgorithm::Amplitude, std::less<int>, std::allocator<std::pair<int const, SiPixelDigitizerAlgorithm::Amplitude> > >&, unsigned short) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiPixelDigitizerPlugins.so
#15 0x00001456c84a757e in SiPixelDigitizerAlgorithm::induce_signal(__gnu_cxx::__normal_iterator<PSimHit const*, std::vector<PSimHit, std::allocator<PSimHit> > >, __gnu_cxx::__normal_iterator<PSimHit const*, std::vector<PSimHit, std::allocator<PSimHit> > >, PSimHit const&, unsigned long, unsigned int, PixelGeomDetUnit const*, std::vector<SiPixelDigitizerAlgorithm::SignalPoint, std::allocator<SiPixelDigitizerAlgorithm::SignalPoint> > const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiPixelDigitizerPlugins.so
#16 0x00001456c84aa1a9 in SiPixelDigitizerAlgorithm::accumulateSimHits(__gnu_cxx::__normal_iterator<PSimHit const*, std::vector<PSimHit, std::allocator<PSimHit> > >, __gnu_cxx::__normal_iterator<PSimHit const*, std::vector<PSimHit, std::allocator<PSimHit> > >, unsigned long, unsigned int, PixelGeomDetUnit const*, Vector3DBase<float, GlobalTag> const&, TrackerTopology const*, CLHEP::HepRandomEngine*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiPixelDigitizerPlugins.so
#17 0x00001456c848ae97 in cms::SiPixelDigitizer::accumulatePixelHits(edm::Handle<std::vector<PSimHit, std::allocator<PSimHit> > >, unsigned long, unsigned int, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiPixelDigitizerPlugins.so
#18 0x00001456c848b4e6 in cms::SiPixelDigitizer::accumulate(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiPixelDigitizerPlugins.so
#19 0x00001456c86d4e1b in edm::MixingModule::accumulateEvent(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralMixingModulePlugins.so
#20 0x00001456c86d4e3f in edm::MixingModule::addSignals(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralMixingModulePlugins.so
#21 0x00001456c85dbbae in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libMixingBase.so
...

Thread 4 (Thread 0x1456d1a00700 (LWP 9095)):
#3  <signal handler called>
#4  0x00001456c83ba013 in SiStripFedZeroSuppression::suppress(std::vector<SiStripDigi, std::allocator<SiStripDigi> > const&, std::vector<SiStripDigi, std::allocator<SiStripDigi> >&, unsigned int, edm::ESHandle<SiStripNoises>&, edm::ESHandle<SiStripThreshold>&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libRecoLocalTrackerSiStripZeroSuppression.so
#5  0x00001456c842e70b in PreMixingSiStripWorker::put(edm::Event&, edm::EventSetup const&, std::vector<PileupSummaryInfo, std::allocator<PileupSummaryInfo> > const&, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiStripDigitizerPlugins.so
#6  0x00001456c837f33d in edm::PreMixingModule::put(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
#7  0x00001456f8a82c27 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libFWCoreFramework.so

Thread 3 (Thread 0x1456d2600700 (LWP 9094)):
#3  <signal handler called>
#4  0x00001456c8420abf in SiStripDigitizerAlgorithm::accumulateSimHits(__gnu_cxx::__normal_iterator<PSimHit const*, std::vector<PSimHit, std::allocator<PSimHit> > >, __gnu_cxx::__normal_iterator<PSimHit const*, std::vector<PSimHit, std::allocator<PSimHit> > >, unsigned long, unsigned int, StripGeomDetUnit const*, Vector3DBase<float, GlobalTag> const&, TrackerTopology const*, CLHEP::HepRandomEngine*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiStripDigitizerPlugins.so
#5  0x00001456c843fd0f in SiStripDigitizer::accumulateStripHits(edm::Handle<std::vector<PSimHit, std::allocator<PSimHit> > >, TrackerTopology const*, unsigned long, unsigned int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiStripDigitizerPlugins.so
#6  0x00001456c844029a in SiStripDigitizer::accumulate(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiStripDigitizerPlugins.so
#7  0x00001456c86d4e1b in edm::MixingModule::accumulateEvent(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralMixingModulePlugins.so
#8  0x00001456c86d4e3f in edm::MixingModule::addSignals(edm::Event const&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralMixingModulePlugins.so
#9  0x00001456c85dbbae in edm::BMixingModule::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libMixingBase.so

Thread 2 (Thread 0x1456db600700 (LWP 9088)):
[cmsRun helper thread]

Thread 1 (Thread 0x1456f7735480 (LWP 8970)):
#3  <signal handler called>
#4  0x00001456c842ee24 in PreMixingSiStripWorker::put(edm::Event&, edm::EventSetup const&, std::vector<PileupSummaryInfo, std::allocator<PileupSummaryInfo> > const&, int) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimTrackerSiStripDigitizerPlugins.so
#5  0x00001456c837f33d in edm::PreMixingModule::put(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/pluginSimGeneralPreMixingModulePlugins.so
#6  0x00001456f8a82c27 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libFWCoreFramework.so
...
#13 0x00001456f89efa90 in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#14 0x00001456f89f94e1 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms.cern.ch/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_17/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#15 0x000000000040fe78 in main::{lambda()#1}::operator()() const ()
#16 0x000000000040e142 in main ()

Current Modules:

Module: DTTrigProd:simDtTriggerPrimitiveDigis (crashed)
Module: PreMixingModule:mixData
Module: MixingModule:mix
Module: MixingModule:mix
Module: PreMixingModule:mixData
Module: PreMixingModule:mixData
Module: PreMixingModule:mixData
Module: PreMixingModule:mixData

@Dr15Jones
Copy link
Contributor Author

The full log info can be found https://cernbox.cern.ch/files/link/public/CQvjzkONaprof65?tiles-size=1&items-per-page=100&view-mode=resource-table with file vocms0253.cern.ch-283595-0-log.tar.gz

@Dr15Jones
Copy link
Contributor Author

I see at least one (probably unrelated) bug in the code

cmssw/L1Trigger/DTTrigger/src/DTTrig.cc

Lines 227 to 231 in d376eb3

if (iSetup.get<MuonGeometryRecord>().cacheIdentifier() != _configid) {
if (_debug)
std::cout << "DTTrig::updateES updating muon geometry" << std::endl;
_geomid = iSetup.get<MuonGeometryRecord>().cacheIdentifier();

The use of _configid should be changed to _geomid at line 227.

@Dr15Jones
Copy link
Contributor Author

Dr15Jones commented May 17, 2024 •
edited
Loading

So the code crashes in

cmssw/L1Trigger/DTTriggerServerTheta/src/DTTSTheta.cc

Lines 67 to 70 in d376eb3

void DTTSTheta::setConfig(const DTConfigManager* conf) {
DTChamberId sid = ChamberId();
_config = conf->getDTConfigTSTheta(sid);
}

where the only two possible bad pointers are conf (which was obtained from the EventSetup system in such a way that if it were a nullptr, the framework would have thrown an exception) or this. The this pointer is from an object in a std::map<DTChamberId, DTSCTrigUnit, std::less<DTChamberId> > and the call comes from looping over that container which should me the this is OK.

@Dr15Jones
Copy link
Contributor Author

So I used the configurations and ran them at FNAL on a local machine interactively and the jobs processed just fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
l1-pending pending-signatures
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@makortel @Dr15Jones @cmsbuild