Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SEGV on unknown address 0x000000000000, RunManagerMT #21359

Closed
davidlt opened this issue Nov 17, 2017 · 7 comments
Closed

SEGV on unknown address 0x000000000000, RunManagerMT #21359

davidlt opened this issue Nov 17, 2017 · 7 comments
Labels
pending-signatures simulation-pending

Comments

@davidlt
Copy link
Contributor

davidlt commented Nov 17, 2017

CMSSW: CMSSW_10_0_ASAN_X_2017-11-12-1100
SCRAM_ARCH: slc6_amd64_gcc700
Workflow: 500199.0
Step: step1

Detailed report:

Internal FlatRandomEGun is initialzed
ASAN:DEADLYSIGNAL
=================================================================
==14731==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7fe8d15d85a3 bp 0x7fe8d039beb0 sp 0x7fe8d039bea0 T2)
==14731==The signal is caused by a READ memory access.
==14731==Hint: address points to the zero page.
    #0 0x7fe8d15d85a2 in RunManagerMT::terminateRun() /build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/src/SimG4Core/Application/src/RunManagerMT.cc:254
    #1 0x7fe8d15d5866 in RunManagerMT::~RunManagerMT() /build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/src/SimG4Core/Application/src/RunManagerMT.cc:92
    #2 0x7fe8d164f8d3 in void __gnu_cxx::new_allocator<RunManagerMT>::destroy<RunManagerMT>(RunManagerMT*) /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/ext/new_allocator.h:140
    #3 0x7fe8d164f8a6 in void std::allocator_traits<std::allocator<RunManagerMT> >::destroy<RunManagerMT>(std::allocator<RunManagerMT>&, RunManagerMT*) /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/bits/alloc_traits.h:487
    #4 0x7fe8d164f6a8 in std::_Sp_counted_ptr_inplace<RunManagerMT, std::allocator<RunManagerMT>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/bits/shared_ptr_base.h:535
    #5 0x7fe903d9e2bb in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libDataFormatsCommon.so+0x512bb)
    #6 0x7fe8d164d60d in std::__shared_ptr<RunManagerMT, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/bits/shared_ptr_base.h:1123
    #7 0x7fe8d164def9 in std::__shared_ptr<RunManagerMT, (__gnu_cxx::_Lock_policy)2>::reset() /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/bits/shared_ptr_base.h:1235
    #8 0x7fe8d15dfc78 in operator() /build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/src/SimG4Core/Application/src/OscarMTMasterThread.cc:112
    #9 0x7fe8d15e30a7 in __invoke_impl<void, OscarMTMasterThread::OscarMTMasterThread(const edm::ParameterSet&)::<lambda()> > /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/bits/invoke.h:60
    #10 0x7fe8d15e2c62 in __invoke<OscarMTMasterThread::OscarMTMasterThread(const edm::ParameterSet&)::<lambda()> > /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/bits/invoke.h:95
    #11 0x7fe8d15e3483 in _M_invoke<0> /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/thread:234
    #12 0x7fe8d15e3454 in operator() /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/thread:243
    #13 0x7fe8d15e3433 in _M_run /cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/include/c++/7.2.1/thread:186
    #14 0x7fe901e45c7e in execute_native_thread_routine ../../../../../libstdc++-v3/src/c++11/thread.cc:83
    #15 0x326f807aa0 in start_thread (/lib64/libpthread.so.0+0x326f807aa0)
    #16 0x326f0e8bcc in __clone (/lib64/libc.so.6+0x326f0e8bcc)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/src/SimG4Core/Application/src/RunManagerMT.cc:254 in RunManagerMT::terminateRun()
Thread T2 created by T0 here:
    #0 0x7fe90463d210 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cc:243
    #1 0x7fe901e45f74 in __gthread_create /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/gcc/7.0.0-mmelna/gcc-branches_gcc-7-branch-254409/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:662
    #2 0x7fe901e45f74 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) ../../../../../libstdc++-v3/src/c++11/thread.cc:163
    #3 0x7fe8d15e07c6 in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&) /build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/src/SimG4Core/Application/src/OscarMTMasterThread.cc:38
    #4 0x7fe8d1585e11 in OscarMTProducer::initializeGlobalCache(edm::ParameterSet const&) /build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/src/SimG4Core/Application/plugins/OscarMTProducer.cc:139
    #5 0x7fe8d15b1194 in std::unique_ptr<OscarMTMasterThread, std::default_delete<OscarMTMasterThread> > edm::stream::impl::makeGlobal<OscarMTProducer, OscarMTMasterThread>(edm::ParameterSet const&, OscarMTMasterThread const*) (/build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/biglib/slc6_amd64_gcc700/pluginSimulation.so+0xde4194)
    #6 0x7fe8d15b0b23 in edm::stream::ProducingModuleAdaptor<OscarMTProducer, edm::stream::EDProducerBase, edm::stream::EDProducerAdaptorBase>::ProducingModuleAdaptor(edm::ParameterSet const&) (/build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/biglib/slc6_amd64_gcc700/pluginSimulation.so+0xde3b23)
    #7 0x7fe8d15b03a7 in std::_MakeUniq<edm::stream::ProducingModuleAdaptor<OscarMTProducer, edm::stream::EDProducerBase, edm::stream::EDProducerAdaptorBase> >::__single_object std::make_unique<edm::stream::ProducingModuleAdaptor<OscarMTProducer, edm::stream::EDProducerBase, edm::stream::EDProducerAdaptorBase>, edm::ParameterSet const&>(edm::ParameterSet const&) (/build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/biglib/slc6_amd64_gcc700/pluginSimulation.so+0xde33a7)
    #8 0x7fe8d15afc35 in std::unique_ptr<edm::stream::EDProducerAdaptorBase, std::default_delete<edm::stream::EDProducerAdaptorBase> > edm::MakeModuleHelper<edm::stream::EDProducerAdaptorBase>::makeModule<OscarMTProducer>(edm::ParameterSet const&) (/build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/biglib/slc6_amd64_gcc700/pluginSimulation.so+0xde2c35)
    #9 0x7fe8d15af73e in edm::WorkerMaker<OscarMTProducer>::makeModule(edm::ParameterSet const&) const (/build/davidlt/CMSSW_10_0_ASAN_X_2017-11-12-1100/biglib/slc6_amd64_gcc700/pluginSimulation.so+0xde273e)
    #10 0x7fe904172925 in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x362925)
    #11 0x7fe90405d9da in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x24d9da)
    #12 0x7fe90439ccfb in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::string const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x58ccfb)
    #13 0x7fe903f9448f in edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::string const&) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x18448f)
    #14 0x7fe9041db189 in edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::string const&) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x3cb189)
    #15 0x7fe904123339 in edm::StreamSchedule::fillWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::string const&, bool, std::vector<edm::WorkerInPath, std::allocator<edm::WorkerInPath> >&, std::vector<std::string, std::allocator<std::string> > const&) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x313339)
    #16 0x7fe90412639f in edm::StreamSchedule::fillTrigPath(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, int, std::string const&, std::shared_ptr<edm::HLTGlobalStatus>, std::vector<std::string, std::allocator<std::string> > const&) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x31639f)
    #17 0x7fe904133d55 in edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::BranchIDListHelper&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration>, bool, edm::StreamID, edm::ProcessContext const*) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x323d55)
    #18 0x7fe9044146a2 in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::BranchIDListHelper&, edm::ThinnedAssociationsHelper&, edm::SubProcessParentageHelper const*, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration>, bool, edm::PreallocationConfiguration const&, edm::ProcessContext const*) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x6046a2)
    #19 0x7fe9043be02f in edm::ScheduleItems::initSchedule(edm::ParameterSet&, bool, edm::PreallocationConfiguration const&, edm::ProcessContext const*) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x5ae02f)
    #20 0x7fe9042aca32 in edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x49ca32)
    #21 0x7fe9042b630f in edm::EventProcessor::EventProcessor(std::shared_ptr<edm::ProcessDesc>, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x4a630f)
    #22 0x412c3d in main::{lambda()#1}::operator()() const (/cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/bin/slc6_amd64_gcc700/cmsRun+0x412c3d)
    #23 0x40d322 in main (/cvmfs/cms-ib.cern.ch/nweek-02498/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-11-12-1100/bin/slc6_amd64_gcc700/cmsRun+0x40d322)
    #24 0x326f01ed1c in __libc_start_main (/lib64/libc.so.6+0x326f01ed1c)

==14731==ABORTING

The problem happens here:
https://github.com/cms-sw/cmssw/blob/master/SimG4Core/Application/src/RunManagerMT.cc#L252

m_userRunAction pointer value is 0xbebebebebebebebe. Now ASan has 0xbe as pattern for malloc_fill_byte: Value used to fill the newly allocated memory.
@cmsbuild
Copy link
Contributor

A new Issue was created by @davidlt .

@davidlange6, @Dr15Jones, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@Dr15Jones
Copy link
Contributor

assign simulation

@cmsbuild
Copy link
Contributor

New categories assigned: simulation

@mdhildreth,@civanch you have been requested to review this Pull request/Issue and eventually sign? Thanks

@Dr15Jones
Copy link
Contributor

m_userRunAction is only set here
https://github.com/cms-sw/cmssw/blob/master/SimG4Core/Application/src/RunManagerMT.cc#L236

which is called from
https://github.com/cms-sw/cmssw/blob/master/SimG4Core/Application/src/RunManagerMT.cc#L196

Looking at the logic of the code, it is possible for an exception to happen before reading the initialization of m_userRunAction which means the destructor would be called and would see an uninitialized value for the member variable.

I beleive the fix is just to set m_userRunAction to nullptr in the constructor. Or even better, changed m_userRunAction to be a std::shared_ptr<> with a customized delete method.

@davidlt
Copy link
Contributor Author

davidlt commented Nov 19, 2017

Could you propose a pull request?

@civanch civanch mentioned this issue Nov 19, 2017
Merged
@civanch
Copy link
Contributor

civanch commented Nov 19, 2017

There is one #21363

@davidlt
Copy link
Contributor Author

davidlt commented Nov 20, 2017

I can confirm that PR mentioned above resolves the issue.

@davidlt davidlt closed this as completed Nov 21, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
pending-signatures simulation-pending
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@davidlt @Dr15Jones @cmsbuild @civanch