Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

invalid write of 1 byte, heap-use-after-free in xrootd/libUtilitiesXrdAdaptor.so (?) #21110

Closed
davidlt opened this issue Oct 31, 2017 · 10 comments
Closed

invalid write of 1 byte, heap-use-after-free in xrootd/libUtilitiesXrdAdaptor.so (?) #21110

davidlt opened this issue Oct 31, 2017 · 10 comments
Labels
core-approved fully-signed

Comments

@davidlt
Copy link
Contributor

davidlt commented Oct 31, 2017

This is read312RV addon test: https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc700/CMSSW_10_0_ASAN_X_2017-10-30-2300/addOnTests/logs/cmsDriver-read312RV_cmsRun__cvmfs_cms-ib.cern.ch_week0_slc6_amd64_gcc700_cms_cmssw_CMSSW_10_0_ASAN_X_2017-10-30-2300_src_Utilities_ReleaseScripts_scripts_read312RV_cfg.py.log

Report:

=================================================================
==20942==ERROR: AddressSanitizer: heap-use-after-free on address 0x60e0000ca430 at pc 0x2ae60a01e2e5 bp 0x2ae6175fa450 sp 0x2ae6175fa448
WRITE of size 1 at 0x60e0000ca430 thread T4
    #0 0x2ae60a01e2e4 in std::_Function_handler<void (XrdAdaptor::RequestManager::OpenHandler*), XrdAdaptor::RequestManager::OpenHandler::HandleResponseWithHosts(XrdCl::XRootDStatus*, XrdCl::AnyObject*, std::vector<XrdCl::HostInfo, std::allocator<XrdCl::HostInfo> >*)::{lambda(XrdAdaptor::RequestManager::OpenHandler*)#1}>::_M_invoke(std::_Any_data const&, XrdAdaptor::RequestManager::OpenHandler*&&) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libUtilitiesXrdAdaptor.so+0xa72e4)
    #1 0x2ae60a05ac4b in std::unique_ptr<XrdAdaptor::RequestManager::OpenHandler, std::function<void (XrdAdaptor::RequestManager::OpenHandler*)> >::~unique_ptr() (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libUtilitiesXrdAdaptor.so+0xe3c4b)
    #2 0x2ae60a03017f in XrdAdaptor::RequestManager::OpenHandler::HandleResponseWithHosts(XrdCl::XRootDStatus*, XrdCl::AnyObject*, std::vector<XrdCl::HostInfo, std::allocator<XrdCl::HostInfo> >*) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libUtilitiesXrdAdaptor.so+0xb917f)
    #3 0x2ae609f0d872 in HandleResponseWithHosts /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClFileStateHandler.cc:87
    #4 0x2ae609eef372 in XrdCl::XRootDMsgHandler::HandleResponse() /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClXRootDMsgHandler.cc:1144
    #5 0x2ae609ef3f46 in XrdCl::XRootDMsgHandler::Process(XrdCl::Message*) /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClXRootDMsgHandler.cc:363
    #6 0x2ae609ed3421 in XrdCl::Stream::HandleIncMsgJob::Run(void*) /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/./XrdCl/XrdClStream.hh:289
    #7 0x2ae609f3789a in XrdCl::JobManager::RunJobs() /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClJobManager.cc:148
    #8 0x2ae609f37b18 in RunRunnerThread /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClJobManager.cc:33
    #9 0x2ae604370aa0 in start_thread (/lib64/libpthread.so.0+0x7aa0)
    #10 0x2ae60466ebcc in __clone (/lib64/libc.so.6+0xe8bcc)

0x60e0000ca430 is located 48 bytes inside of 152-byte region [0x60e0000ca400,0x60e0000ca498)
freed by thread T4 here:
    #0 0x2ae60010e568 in operator delete(void*, unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cc:140
    #1 0x42192c in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (/cvmfs/cms-ib.cern.ch/nweek-02496/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/bin/slc6_amd64_gcc700/cmsRun+0x42192c)
    #2 0x2ae60a030ca5 in XrdAdaptor::RequestManager::OpenHandler::HandleResponseWithHosts(XrdCl::XRootDStatus*, XrdCl::AnyObject*, std::vector<XrdCl::HostInfo, std::allocator<XrdCl::HostInfo> >*) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libUtilitiesXrdAdaptor.so+0xb9ca5)
    #3 0x2ae609f0d872 in HandleResponseWithHosts /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClFileStateHandler.cc:87
    #4 0x2ae609eef372 in XrdCl::XRootDMsgHandler::HandleResponse() /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClXRootDMsgHandler.cc:1144
    #5 0x2ae609ef3f46 in XrdCl::XRootDMsgHandler::Process(XrdCl::Message*) /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClXRootDMsgHandler.cc:363
    #6 0x2ae609ed3421 in XrdCl::Stream::HandleIncMsgJob::Run(void*) /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/./XrdCl/XrdClStream.hh:289
    #7 0x2ae609f3789a in XrdCl::JobManager::RunJobs() /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClJobManager.cc:148
    #8 0x2ae609f37b18 in RunRunnerThread /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClJobManager.cc:33
    #9 0x2ae604370aa0 in start_thread (/lib64/libpthread.so.0+0x7aa0)

previously allocated by thread T0 here:
    #0 0x2ae60010d1e0 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cc:80
    #1 0x2ae60a0448c1 in XrdAdaptor::RequestManager::initialize(std::weak_ptr<XrdAdaptor::RequestManager>) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libUtilitiesXrdAdaptor.so+0xcd8c1)
    #2 0x2ae609fa6ce2 in XrdFile::open(char const*, int, int) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libUtilitiesXrdAdaptor.so+0x2fce2)
    #3 0x2ae609fad9b3 in XrdFile::XrdFile(std::string const&, int, int) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libUtilitiesXrdAdaptor.so+0x369b3)
    #4 0x2ae609e6072f in XrdStorageMaker::open(std::string const&, std::string const&, int, StorageMaker::AuxSettings const&) const (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/pluginUtilitiesXrdAdaptorPlugin.so+0xc72f)
    #5 0x2ae60a0bcd9e in StorageFactory::open(std::string const&, int) const (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libUtilitiesStorageFactory.so+0x27d9e)
    #6 0x2ae60accffda in TStorageFactoryFile::Initialize(char const*, char const*) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libIOPoolTFileAdaptor.so+0x1ffda)
    #7 0x2ae60acd18ee in TStorageFactoryFile::TStorageFactoryFile(char const*, char const*, char const*, int, int, bool) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libIOPoolTFileAdaptor.so+0x218ee)
    #8 0x2ae6142ed08e  (<unknown module>)
    #9 0x2ae60b5de77e in TClingCallFunc::exec(void*, void*) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/external/slc6_amd64_gcc700/lib/libCling.so+0x3b177e)
    #10 0x604000117a67  (<unknown module>)

Thread T4 created by T0 here:
    #0 0x2ae600069230 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cc:243
    #1 0x2ae609f3773b in XrdCl::JobManager::Start() /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClJobManager.cc:74
    #2 0x2ae609ecb3d8 in XrdCl::PostMaster::Start() /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClPostMaster.cc:126
    #3 0x2ae609eb5f87 in XrdCl::DefaultEnv::GetPostMaster() /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClDefaultEnv.cc:435
    #4 0x2ae609ef8f14 in XrdCl::MessageUtils::SendMessage(XrdCl::URL const&, XrdCl::Message*, XrdCl::ResponseHandler*, XrdCl::MessageSendParams const&) /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClMessageUtils.cc:49
    #5 0x2ae609ede858 in XrdCl::FileSystem::Send(XrdCl::Message*, XrdCl::ResponseHandler*, XrdCl::MessageSendParams&) /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClFileSystem.cc:1419
    #6 0x2ae609ee6368 in XrdCl::FileSystem::Prepare(std::vector<std::string, std::allocator<std::string> > const&, XrdCl::PrepareFlags::Flags, unsigned char, XrdCl::ResponseHandler*, unsigned short) /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClFileSystem.cc:1326
    #7 0x2ae609e68821 in XrdStorageMaker::stagein(std::string const&, std::string const&, StorageMaker::AuxSettings const&) const (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/pluginUtilitiesXrdAdaptorPlugin.so+0x14821)
    #8 0x2ae60a0bde20 in StorageFactory::stagein(std::string const&) const (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libUtilitiesStorageFactory.so+0x28e20)
    #9 0x2ae61550f09f in edm::RootPrimaryFileSequence::RootPrimaryFileSequence(edm::ParameterSet const&, edm::PoolSource&, edm::InputFileCatalog const&) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/pluginIOPoolInput.so+0x17709f)
    #10 0x2ae6153e73ab in edm::PoolSource::PoolSource(edm::ParameterSet const&, edm::InputSourceDescription const&) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/pluginIOPoolInput.so+0x4f3ab)
    #11 0x2ae6154a668d in edmplugin::PluginFactory<edm::InputSource* (edm::ParameterSet const&, edm::InputSourceDescription const&)>::PMaker<edm::PoolSource>::create(edm::ParameterSet const&, edm::InputSourceDescription const&) const (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/pluginIOPoolInput.so+0x10e68d)
    #12 0x2ae601404617 in edm::InputSourceFactory::makeInputSource(edm::ParameterSet const&, edm::InputSourceDescription const&) const (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x621617)
    #13 0x2ae60124ea1d in edm::makeInput(edm::ParameterSet&, edm::CommonParams const&, std::shared_ptr<edm::ProductRegistry>, std::shared_ptr<edm::BranchIDListHelper>, std::shared_ptr<edm::ThinnedAssociationsHelper>, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x46ba1d)
    #14 0x2ae601258fa0 in edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x475fa0)
    #15 0x2ae601262b0f in edm::EventProcessor::EventProcessor(std::shared_ptr<edm::ProcessDesc>, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x47fb0f)
    #16 0x412c3d in main::{lambda()#1}::operator()() const (/cvmfs/cms-ib.cern.ch/nweek-02496/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/bin/slc6_amd64_gcc700/cmsRun+0x412c3d)
    #17 0x40d322 in main (/cvmfs/cms-ib.cern.ch/nweek-02496/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/bin/slc6_amd64_gcc700/cmsRun+0x40d322)
    #18 0x2ae6045a4d1c in __libc_start_main (/lib64/libc.so.6+0x1ed1c)

SUMMARY: AddressSanitizer: heap-use-after-free (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libUtilitiesXrdAdaptor.so+0xa72e4) in std::_Function_handler<void (XrdAdaptor::RequestManager::OpenHandler*), XrdAdaptor::RequestManager::OpenHandler::HandleResponseWithHosts(XrdCl::XRootDStatus*, XrdCl::AnyObject*, std::vector<XrdCl::HostInfo, std::allocator<XrdCl::HostInfo> >*)::{lambda(XrdAdaptor::RequestManager::OpenHandler*)#1}>::_M_invoke(std::_Any_data const&, XrdAdaptor::RequestManager::OpenHandler*&&)
Shadow bytes around the buggy address:
  0x0c1c80011430: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c80011440: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c80011450: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c80011460: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c80011470: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c1c80011480: fd fd fd fd fd fd[fd]fd fd fd fd fd fd fd fd fd
  0x0c1c80011490: fd fd fd fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c800114a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c800114b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c800114c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c800114d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==20942==ABORTING
@cmsbuild
Copy link
Contributor

A new Issue was created by @davidlt .

@davidlange6, @Dr15Jones, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@davidlt davidlt changed the title invalid write of 1 byte, heap-use-after-free in xrootd (?) invalid write of 1 byte, heap-use-after-free in xrootd/libUtilitiesXrdAdaptor.so (?) Oct 31, 2017
@Dr15Jones
Copy link
Contributor

@bbockelm

@davidlt
Copy link
Contributor Author

davidlt commented Nov 1, 2017

I had to run the whole thing in a loop overnight to hit it again (this time with debug and -O0):

while cmsRun /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/src/Utilities/ReleaseScripts/scripts/read312RV_cfg.py; do :; done

Updated report:

=================================================================
==17687==ERROR: AddressSanitizer: heap-use-after-free on address 0x60e000198fb0 at pc 0x7f0930dd2772 bp 0x7f0924319810 sp 0x7f0924319808
WRITE of size 1 at 0x60e000198fb0 thread T4
    #0 0x7f0930dd2771 in std::__atomic_base<bool>::store(bool, std::memory_order) /cvmfs/cms-ib.cern.ch/nweek-02496/slc6_amd64_gcc700/external/gcc/7.0.0-fmblme3/include/c++/7.2.1/bits/atomic_base.h:374
    #1 0x7f0930dd2771 in std::__atomic_base<bool>::operator=(bool) /cvmfs/cms-ib.cern.ch/nweek-02496/slc6_amd64_gcc700/external/gcc/7.0.0-fmblme3/include/c++/7.2.1/bits/atomic_base.h:267
    #2 0x7f0930dce610 in std::atomic<bool>::operator=(bool) /cvmfs/cms-ib.cern.ch/nweek-02496/slc6_amd64_gcc700/external/gcc/7.0.0-fmblme3/include/c++/7.2.1/atomic:79
    #3 0x7f0930dbd72e in operator() /mnt/build/davidlt/debug/CMSSW_10_0_ASAN_X_2017-10-30-2300/src/Utilities/XrdAdaptor/src/XrdRequestManager.cc:1075
    #4 0x7f0930dc236c in _M_invoke /cvmfs/cms-ib.cern.ch/nweek-02496/slc6_amd64_gcc700/external/gcc/7.0.0-fmblme3/include/c++/7.2.1/bits/std_function.h:316
    #5 0x7f0930de1ba8 in std::function<void (XrdAdaptor::RequestManager::OpenHandler*)>::operator()(XrdAdaptor::RequestManager::OpenHandler*) const /cvmfs/cms-ib.cern.ch/nweek-02496/slc6_amd64_gcc700/external/gcc/7.0.0-fmblme3/include/c++/7.2.1/bits/std_function.h:706
    #6 0x7f0930dd8d6e in std::unique_ptr<XrdAdaptor::RequestManager::OpenHandler, std::function<void (XrdAdaptor::RequestManager::OpenHandler*)> >::~unique_ptr() /cvmfs/cms-ib.cern.ch/nweek-02496/slc6_amd64_gcc700/external/gcc/7.0.0-fmblme3/include/c++/7.2.1/bits/unique_ptr.h:268
    #7 0x7f0930dbe372 in XrdAdaptor::RequestManager::OpenHandler::HandleResponseWithHosts(XrdCl::XRootDStatus*, XrdCl::AnyObject*, std::vector<XrdCl::HostInfo, std::allocator<XrdCl::HostInfo> >*) /mnt/build/davidlt/debug/CMSSW_10_0_ASAN_X_2017-10-30-2300/src/Utilities/XrdAdaptor/src/XrdRequestManager.cc:1077
    #8 0x7f0930edf872 in HandleResponseWithHosts /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClFileStateHandler.cc:87
    #9 0x7f0930ec1372 in XrdCl::XRootDMsgHandler::HandleResponse() /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClXRootDMsgHandler.cc:1144
    #10 0x7f0930ec5f46 in XrdCl::XRootDMsgHandler::Process(XrdCl::Message*) /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClXRootDMsgHandler.cc:363
    #11 0x7f0930ea5421 in XrdCl::Stream::HandleIncMsgJob::Run(void*) /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/./XrdCl/XrdClStream.hh:289
    #12 0x7f0930f0989a in XrdCl::JobManager::RunJobs() /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClJobManager.cc:148
    #13 0x7f0930f09b18 in RunRunnerThread /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClJobManager.cc:33
    #14 0x7f093b9d6aa0 in start_thread (/lib64/libpthread.so.0+0x7aa0)
    #15 0x7f093b72393c in clone (/lib64/libc.so.6+0xe893c)

0x60e000198fb0 is located 48 bytes inside of 152-byte region [0x60e000198f80,0x60e000199018)
freed by thread T4 here:
    #0 0x7f093f038568 in operator delete(void*, unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cc:140
    #1 0x7f0930dbd6e2 in XrdAdaptor::RequestManager::OpenHandler::~OpenHandler() /mnt/build/davidlt/debug/CMSSW_10_0_ASAN_X_2017-10-30-2300/src/Utilities/XrdAdaptor/src/XrdRequestManager.cc:1066
    #2 0x7f0930df3832 in std::_Sp_counted_ptr<XrdAdaptor::RequestManager::OpenHandler*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() /cvmfs/cms-ib.cern.ch/nweek-02496/slc6_amd64_gcc700/external/gcc/7.0.0-fmblme3/include/c++/7.2.1/bits/shared_ptr_base.h:376
    #3 0x7f093e6ff2bb in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libDataFormatsCommon.so+0x512bb)
    #4 0x7f0930dcf455 in std::__shared_ptr<XrdAdaptor::RequestManager::OpenHandler, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /cvmfs/cms-ib.cern.ch/nweek-02496/slc6_amd64_gcc700/external/gcc/7.0.0-fmblme3/include/c++/7.2.1/bits/shared_ptr_base.h:1123
    #5 0x7f0930dcf471 in std::shared_ptr<XrdAdaptor::RequestManager::OpenHandler>::~shared_ptr() /cvmfs/cms-ib.cern.ch/nweek-02496/slc6_amd64_gcc700/external/gcc/7.0.0-fmblme3/include/c++/7.2.1/bits/shared_ptr.h:93
    #6 0x7f0930dbe2e2 in XrdAdaptor::RequestManager::OpenHandler::HandleResponseWithHosts(XrdCl::XRootDStatus*, XrdCl::AnyObject*, std::vector<XrdCl::HostInfo, std::allocator<XrdCl::HostInfo> >*) /mnt/build/davidlt/debug/CMSSW_10_0_ASAN_X_2017-10-30-2300/src/Utilities/Xrd
Adaptor/src/XrdRequestManager.cc:1085
    #7 0x7f0930edf872 in HandleResponseWithHosts /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClFileStateHandler.cc:87
    #8 0x7f0930ec1372 in XrdCl::XRootDMsgHandler::HandleResponse() /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClXRootDMsgHandler.cc:1144
    #9 0x7f0930ec5f46 in XrdCl::XRootDMsgHandler::Process(XrdCl::Message*) /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClXRootDMsgHandler.cc:363
    #10 0x7f0930ea5421 in XrdCl::Stream::HandleIncMsgJob::Run(void*) /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/./XrdCl/XrdClStream.hh:289
    #11 0x7f0930f0989a in XrdCl::JobManager::RunJobs() /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClJobManager.cc:148
    #12 0x7f0930f09b18 in RunRunnerThread /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClJobManager.cc:33
    #13 0x7f093b9d6aa0 in start_thread (/lib64/libpthread.so.0+0x7aa0)

previously allocated by thread T0 here:
    #0 0x7f093f0371e0 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cc:80
    #1 0x7f0930dcf535 in XrdAdaptor::RequestManager::OpenHandler::getInstance(std::weak_ptr<XrdAdaptor::RequestManager>) /mnt/build/davidlt/debug/CMSSW_10_0_ASAN_X_2017-10-30-2300/src/Utilities/XrdAdaptor/src/XrdRequestManager.h:247
    #2 0x7f0930dad5e5 in XrdAdaptor::RequestManager::initialize(std::weak_ptr<XrdAdaptor::RequestManager>) /mnt/build/davidlt/debug/CMSSW_10_0_ASAN_X_2017-10-30-2300/src/Utilities/XrdAdaptor/src/XrdRequestManager.cc:131
    #3 0x7f0930d5dc69 in XrdAdaptor::RequestManager::getInstance(std::string const&, XrdCl::OpenFlags::Flags, XrdCl::Access::Mode) (/mnt/build/davidlt/debug/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libUtilitiesXrdAdaptor.so+0xa1c69)
    #4 0x7f0930d5502c in XrdFile::open(char const*, int, int) /mnt/build/davidlt/debug/CMSSW_10_0_ASAN_X_2017-10-30-2300/src/Utilities/XrdAdaptor/src/XrdFile.cc:149
    #5 0x7f0930d53d4b in XrdFile::XrdFile(std::string const&, int, int) /mnt/build/davidlt/debug/CMSSW_10_0_ASAN_X_2017-10-30-2300/src/Utilities/XrdAdaptor/src/XrdFile.cc:52
    #6 0x7f0930f6a838 in std::_MakeUniq<XrdFile>::__single_object std::make_unique<XrdFile, std::string&, int&>(std::string&, int&) /cvmfs/cms-ib.cern.ch/nweek-02496/slc6_amd64_gcc700/external/gcc/7.0.0-fmblme3/include/c++/7.2.1/bits/unique_ptr.h:825
    #7 0x7f0930f673aa in XrdStorageMaker::open(std::string const&, std::string const&, int, StorageMaker::AuxSettings const&) const /mnt/build/davidlt/debug/CMSSW_10_0_ASAN_X_2017-10-30-2300/src/Utilities/XrdAdaptor/plugins/XrdStorageMaker.cc:75
    #8 0x7f0930c42358 in StorageFactory::open(std::string const&, int) const /mnt/build/davidlt/debug/CMSSW_10_0_ASAN_X_2017-10-30-2300/src/Utilities/StorageFactory/src/StorageFactory.cc:185
    #9 0x7f0930021a8f in TStorageFactoryFile::Initialize(char const*, char const*) /mnt/build/davidlt/debug/CMSSW_10_0_ASAN_X_2017-10-30-2300/src/IOPool/TFileAdaptor/src/TStorageFactoryFile.cc:196
    #10 0x7f0930020ffa in TStorageFactoryFile::TStorageFactoryFile(char const*, char const*, char const*, int, int, bool) /mnt/build/davidlt/debug/CMSSW_10_0_ASAN_X_2017-10-30-2300/src/IOPool/TFileAdaptor/src/TStorageFactoryFile.cc:118
    #11 0x7f0926c2d08e  (<unknown module>)
    #12 0x7f092d91077e in TClingCallFunc::exec(void*, void*) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/external/slc6_amd64_gcc700/lib/libCling.so+0x3b177e)
    #13 0x6040000fabe7  (<unknown module>)

Thread T4 created by T0 here:
    #0 0x7f093ef93230 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cc:243
    #1 0x7f0930f0973b in XrdCl::JobManager::Start() /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClJobManager.cc:74
    #2 0x7f0930e9d3d8 in XrdCl::PostMaster::Start() /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClPostMaster.cc:126
    #3 0x7f0930e87f87 in XrdCl::DefaultEnv::GetPostMaster() /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClDefaultEnv.cc:435
    #4 0x7f0930ecaf14 in XrdCl::MessageUtils::SendMessage(XrdCl::URL const&, XrdCl::Message*, XrdCl::ResponseHandler*, XrdCl::MessageSendParams const&) /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.
1/src/XrdCl/XrdClMessageUtils.cc:49
    #5 0x7f0930eb0858 in XrdCl::FileSystem::Send(XrdCl::Message*, XrdCl::ResponseHandler*, XrdCl::MessageSendParams&) /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClFileSystem.cc:141
9
    #6 0x7f0930eb8368 in XrdCl::FileSystem::Prepare(std::vector<std::string, std::allocator<std::string> > const&, XrdCl::PrepareFlags::Flags, unsigned char, XrdCl::ResponseHandler*, unsigned short) /build/cmsbld/jenkins-workarea/workspace/build-any-ib/w/BUILD/slc6_amd64_
gcc700/external/xrootd/4.7.1-mmelna/xrootd-4.7.1/src/XrdCl/XrdClFileSystem.cc:1326
    #7 0x7f0930f67a24 in XrdStorageMaker::stagein(std::string const&, std::string const&, StorageMaker::AuxSettings const&) const /mnt/build/davidlt/debug/CMSSW_10_0_ASAN_X_2017-10-30-2300/src/Utilities/XrdAdaptor/plugins/XrdStorageMaker.cc:89
    #8 0x7f0930c42cf0 in StorageFactory::stagein(std::string const&) const /mnt/build/davidlt/debug/CMSSW_10_0_ASAN_X_2017-10-30-2300/src/Utilities/StorageFactory/src/StorageFactory.cc:226
    #9 0x7f0925b3b09f in edm::RootPrimaryFileSequence::RootPrimaryFileSequence(edm::ParameterSet const&, edm::PoolSource&, edm::InputFileCatalog const&) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/plugin
IOPoolInput.so+0x17709f)
    #10 0x7f0925a133ab in edm::PoolSource::PoolSource(edm::ParameterSet const&, edm::InputSourceDescription const&) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/pluginIOPoolInput.so+0x4f3ab)
    #11 0x7f0925ad268d in edmplugin::PluginFactory<edm::InputSource* (edm::ParameterSet const&, edm::InputSourceDescription const&)>::PMaker<edm::PoolSource>::create(edm::ParameterSet const&, edm::InputSourceDescription const&) const (/cvmfs/cms-ib.cern.ch/week0/slc6_amd6
4_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/pluginIOPoolInput.so+0x10e68d)
    #12 0x7f093ed92617 in edm::InputSourceFactory::makeInputSource(edm::ParameterSet const&, edm::InputSourceDescription const&) const (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x
621617)
    #13 0x7f093ebdca1d in edm::makeInput(edm::ParameterSet&, edm::CommonParams const&, std::shared_ptr<edm::ProductRegistry>, std::shared_ptr<edm::BranchIDListHelper>, std::shared_ptr<edm::ThinnedAssociationsHelper>, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr
<edm::ProcessConfiguration const>, edm::PreallocationConfiguration const&) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libFWCoreFramework.so+0x46ba1d)
    #14 0x7f093ebe6fa0 in edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700/libFWCor
eFramework.so+0x475fa0)
    #15 0x7f093ebf0b0f in edm::EventProcessor::EventProcessor(std::shared_ptr<edm::ProcessDesc>, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) (/cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/lib/slc6_amd64_gcc700
/libFWCoreFramework.so+0x47fb0f)
    #16 0x412c3d in main::{lambda()#1}::operator()() const (/cvmfs/cms-ib.cern.ch/nweek-02496/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/bin/slc6_amd64_gcc700/cmsRun+0x412c3d)
    #17 0x40d322 in main (/cvmfs/cms-ib.cern.ch/nweek-02496/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_0_ASAN_X_2017-10-30-2300/bin/slc6_amd64_gcc700/cmsRun+0x40d322)
    #18 0x7f093b659d5c in __libc_start_main (/lib64/libc.so.6+0x1ed5c)

SUMMARY: AddressSanitizer: heap-use-after-free /cvmfs/cms-ib.cern.ch/nweek-02496/slc6_amd64_gcc700/external/gcc/7.0.0-fmblme3/include/c++/7.2.1/bits/atomic_base.h:374 in std::__atomic_base<bool>::store(bool, std::memory_order)
Shadow bytes around the buggy address:
  0x0c1c8002b1a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c8002b1b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c8002b1c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c8002b1d0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c1c8002b1e0: fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa
=>0x0c1c8002b1f0: fd fd fd fd fd fd[fd]fd fd fd fd fd fd fd fd fd
  0x0c1c8002b200: fd fd fd fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c8002b210: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c8002b220: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c8002b230: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c1c8002b240: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==17687==ABORTING

@davidlt
Copy link
Contributor Author

davidlt commented Nov 1, 2017

Looking at stack strace, I guess, OpenHandler is allocated in CMSSW and then passed/registered (?) to xrootd. Once it is destroyed on CMSSW side xrootd still could call it.

@smuzaffar
Copy link
Contributor

assign core

@cmsbuild
Copy link
Contributor

cmsbuild commented Nov 2, 2017

New categories assigned: core

@Dr15Jones,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

@davidlt
Copy link
Contributor Author

davidlt commented Nov 22, 2017

ping

This stack trace keeps showing up also in more places looking at ASAN logs. Is anyone looking into this?

@Dr15Jones
Copy link
Contributor

This is fixed by #21634

@Dr15Jones
Copy link
Contributor

+1

@cmsbuild
Copy link
Contributor

cmsbuild commented Dec 2, 2017

This issue is fully signed and ready to be closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
core-approved fully-signed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@davidlt @Dr15Jones @smuzaffar @cmsbuild