Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ASAN] heap-buffer-overflow in GsfElectronProducer::produce #35760

Closed
iarspider opened this issue Oct 21, 2021 · 10 comments
Closed

[ASAN] heap-buffer-overflow in GsfElectronProducer::produce #35760

iarspider opened this issue Oct 21, 2021 · 10 comments
Labels
fully-signed reconstruction-approved

Comments

@iarspider
Copy link
Contributor

Many RelVals failed in ASAN IB. For example, this one

More detailed stack trace (with debug information enabled):

==27483==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60201413e61c at pc 0x7fa99442baa2 bp 0x7fa9d0d150b0 sp 0x7fa9d0d150a8
READ of size 4 at 0x60201413e61c thread T4
    #0 0x7fa99442baa1 in setMVAOutputs .../RecoEgamma/EgammaElectronProducers/plugins/GsfElectronProducer.cc:64
    #1 0x7fa99442baa1 in GsfElectronProducer::produce(edm::Event&, edm::EventSetup const&) .../src/RecoEgamma/EgammaElectronProducers/plugins/GsfElectronProducer.cc:727
    #2 0x7faa2737ef5e in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (.../lib/slc7_amd64_gcc10/libFWCoreFramework.so+0x8f9f5e)
    #3 0x7faa272e9eb2 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*) (.../lib/slc7_amd64_gcc10/libFWCoreFramework.so+0x864eb2)
    #4 0x7faa26fd2634 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) (.../lib/slc7_amd64_gcc10/libFWCoreFramework.so+0x54d634)
    #5 0x7faa26fd2e7a in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (.../lib/slc7_amd64_gcc10/libFWCoreFramework.so+0x54de7a)
    #6 0x7faa26fde8d6 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (.../lib/slc7_amd64_gcc10/libFWCoreFramework.so+0x5598d6)
    #7 0x7faa262c72a1 in tbb::detail::d1::function_task<edm::WaitingTaskList::announce()::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (.../lib/slc7_amd64_gcc10/libFWCoreConcurrency.so+0x112a1)
    #8 0x7faa24c7406b in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-544b7b2b990e0660abde669a014ec673/tbb-v2021.4.0/src/tbb/task_dispatcher.h:322
    #9 0x7faa24c7406b in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-544b7b2b990e0660abde669a014ec673/tbb-v2021.4.0/src/tbb/task_dispatcher.h:463
    #10 0x7faa24c7406b in tbb::detail::r1::arena::process(tbb::detail::r1::thread_data&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-544b7b2b990e0660abde669a014ec673/tbb-v2021.4.0/src/tbb/arena.cpp:138
    #11 0x7faa24c805b2 in tbb::detail::r1::market::process(rml::job&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-544b7b2b990e0660abde669a014ec673/tbb-v2021.4.0/src/tbb/market.cpp:597
    #12 0x7faa24c805b2 in tbb::detail::r1::rml::private_worker::run() /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-544b7b2b990e0660abde669a014ec673/tbb-v2021.4.0/src/tbb/private_server.cpp:267
    #13 0x7faa24c805b2 in tbb::detail::r1::rml::private_worker::thread_routine(void*) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc10/external/tbb/v2021.4.0-544b7b2b990e0660abde669a014ec673/tbb-v2021.4.0/src/tbb/private_server.cpp:221
    #14 0x7faa23e7fea4 in start_thread (/lib64/libpthread.so.0+0x7ea4)
    #15 0x7faa23ba89fc in __clone (/lib64/libc.so.6+0xfe9fc)

Possibly caused by #35403
@cmsbuild
Copy link
Contributor

A new Issue was created by @iarspider .

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@iarspider
Copy link
Contributor Author

assign reconstruction

@cmsbuild
Copy link
Contributor

New categories assigned: reconstruction

@slava77,@jpata you have been requested to review this Pull request/Issue and eventually sign? Thanks

@slava77
Copy link
Contributor

slava77 commented Oct 21, 2021

@cms-sw/egamma-pog-l2 @wrtabb @SohamBhattacharya
@valsdav

I guess this is coming with #35403

@jpata
Copy link
Contributor

jpata commented Oct 25, 2021

@valsdav please check, the line picked up by ASAN is:

        mvaOutput.dnn_e_bkgTau = values[3];

https://github.com/cms-sw/cmssw/blob/master/RecoEgamma/EgammaElectronProducers/plugins/GsfElectronProducer.cc#L64

@Dr15Jones
Copy link
Contributor

Additional useful information is where the memory came from

0x6020145c97bc is located 0 bytes to the right of 12-byte region [0x6020145c97b0,0x6020145c97bc)
allocated by thread T4 here:
    #0 0x2b96629fa3d7 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x2b96e6b5dce1 in egammaTools::EgammaDNNHelper::evaluate(std::vector<std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, float, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, float> > >, std::allocator<std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, float, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, float> > > > > const&, std::vector<tensorflow::Session*, std::allocator<tensorflow::Session*> > const&) const (/cvmfs/cms-ib.cern.ch/nweek-02703/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-10-20-2300/lib/slc7_amd64_gcc10/libRecoEgammaEgammaTools.so+0x99ce1)
    #2 0x2b96e8fb187a in ElectronDNNEstimator::evaluate(std::vector<reco::GsfElectron, std::allocator<reco::GsfElectron> > const&, std::vector<tensorflow::Session*, std::allocator<tensorflow::Session*> > const&) const (/cvmfs/cms-ib.cern.ch/nweek-02703/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-10-20-2300/lib/slc7_amd64_gcc10/libRecoEgammaElectronIdentification.so+0x2d87a)
    #3 0x2b96f3d01d76 in GsfElectronProducer::produce(edm::Event&, edm::EventSetup const&) (/cvmfs/cms-ib.cern.ch/nweek-02703/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_1_ASAN_X_2021-10-20-2300/lib/slc7_amd64_gcc10/pluginRecoEgammaEgammaElectronProducersPlugins.so+0x163d76)

@swagata87 swagata87 mentioned this issue Nov 4, 2021
Merged
@swagata87
Copy link
Contributor

hopefully the issue will be fixed by this PR #35984

@jpata
Copy link
Contributor

jpata commented May 16, 2022

+reconstruction

  • I think it was indeed fixed by the PR above, as I can't find this ASAN issue anymore in the logs

@jpata
Copy link
Contributor

jpata commented May 16, 2022

@cmsbuild please close

@cmsbuild
Copy link
Contributor

This issue is fully signed and ready to be closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
fully-signed reconstruction-approved
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@jpata @iarspider @Dr15Jones @cmsbuild @slava77 @swagata87