Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ASAN] Failure in mkfit #37096

Closed
smuzaffar opened this issue Mar 1, 2022 · 11 comments
Closed

[ASAN] Failure in mkfit #37096

smuzaffar opened this issue Mar 1, 2022 · 11 comments
Labels
fully-signed reconstruction-approved

Comments

@smuzaffar
Copy link
Contributor

Hi,
In ASAN IBs, many workflow are failing. ASAN complains about ( when build with USSR_CXXFLAGS=-g)

==3127==ERROR: AddressSanitizer: invalid alignment requested in aligned_alloc: 64, alignment must be a power of two and the requested size 0x68 must be a multiple of alignment (thread T0)
    #0 0x7fcb2d092faf in __interceptor_aligned_alloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:192
    #1 0x7fcadb735c17 in mkfit::Pool<mkfit::CandCloner>::create() /build/muz/asan/CMSSW_12_3_ASAN_X_2022-02-28-1100/src/RecoTracker/MkFitCore/src/Pool.h:41
    #2 0x7fcadb735c17 in mkfit::Pool<mkfit::CandCloner>::populate(int) /build/muz/asan/CMSSW_12_3_ASAN_X_2022-02-28-1100/src/RecoTracker/MkFitCore/src/Pool.h:27
    #3 0x7fcadb735c17 in mkfit::ExecutionContext::populate(int) /build/muz/asan/CMSSW_12_3_ASAN_X_2022-02-28-1100/src/RecoTracker/MkFitCore/src/MkBuilder.cc:46
    #4 0x7fcadb735c17 in mkfit::MkBuilder::populate() /build/muz/asan/CMSSW_12_3_ASAN_X_2022-02-28-1100/src/RecoTracker/MkFitCore/src/MkBuilder.cc:171
    #5 0x7fcadb9f3920 in MkFitProducer::MkFitProducer(edm::ParameterSet const&) (/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-02-28-1100/lib/slc7_amd64_gcc11/pluginRecoTrackerMkFitPlugins.so+0x15a920)
    #6 0x7fcadba0ecc4 in edm::WorkerMaker<MkFitProducer>::makeModule(edm::ParameterSet const&) const (/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-02-28-1100/lib/slc7_amd64_gcc11/pluginRecoTrackerMkFitPlugins.so+0x175cc4)
    #7 0x7fcb2cda922d in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-02-28-1100/lib/slc7_amd64_gcc11/libFWCoreFramework.so+0x82f22d)
    #8 0x7fcb2ca1a116 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const (/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-02-28-1100/lib/slc7_amd64_gcc11/libFWCoreFramework.so+0x4a0116)
    #9 0x7fcb2ca6de09 in edm::ModuleRegistry::getModule(edm::MakeModuleParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) (/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-02-28-1100/lib/slc7_amd64_gcc11/libFWCoreFramework.so+0x4f3e09)
    #10 0x7fcb2cdb6e31 in edm::WorkerRegistry::getWorker(edm::WorkerParams const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-02-28-1100/lib/slc7_amd64_gcc11/libFWCoreFramework.so+0x83ce31)
    #11 0x7fcb2cdab3f6 in edm::WorkerManager::getWorker(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration const>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-02-28-1100/lib/slc7_amd64_gcc11/libFWCoreFramework.so+0x8313f6)
    #12 0x7fcb2cdafd98 in edm::WorkerManager::addToUnscheduledWorkers(edm::ParameterSet&, edm::ProductRegistry&, edm::PreallocationConfiguration const*, std::shared_ptr<edm::ProcessConfiguration>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-02-28-1100/lib/slc7_amd64_gcc11/libFWCoreFramework.so+0x835d98)
    #13 0x7fcb2cd0d4f1 in edm::StreamSchedule::StreamSchedule(std::shared_ptr<edm::TriggerResultInserter>, std::vector<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::PathStatusInserter> > > >&, std::vector<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> >, std::allocator<edm::propagate_const<std::shared_ptr<edm::EndPathStatusInserter> > > >&, std::shared_ptr<edm::ModuleRegistry>, edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::PreallocationConfiguration const&, edm::ProductRegistry&, edm::BranchIDListHelper&, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration>, edm::StreamID, edm::ProcessContext const*) (/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-02-28-1100/lib/slc7_amd64_gcc11/libFWCoreFramework.so+0x7934f1)
    #14 0x7fcb2cc88eaa in edm::Schedule::Schedule(edm::ParameterSet&, edm::service::TriggerNamesService const&, edm::ProductRegistry&, edm::BranchIDListHelper&, edm::ProcessBlockHelperBase&, edm::ThinnedAssociationsHelper&, edm::SubProcessParentageHelper const*, edm::ExceptionToActionTable const&, std::shared_ptr<edm::ActivityRegistry>, std::shared_ptr<edm::ProcessConfiguration>, bool, edm::PreallocationConfiguration const&, edm::ProcessContext const*) (/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-02-28-1100/lib/slc7_amd64_gcc11/libFWCoreFramework.so+0x70eeaa)
    #15 0x7fcb2ccc492e in edm::ScheduleItems::initSchedule(edm::ParameterSet&, bool, edm::PreallocationConfiguration const&, edm::ProcessContext const*, edm::ProcessBlockHelperBase&) (/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-02-28-1100/lib/slc7_amd64_gcc11/libFWCoreFramework.so+0x74a92e)
    #16 0x7fcb2c8264de in edm::EventProcessor::init(std::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) (/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-02-28-1100/lib/slc7_amd64_gcc11/libFWCoreFramework.so+0x2ac4de)
    #17 0x7fcb2c82f8e0 in edm::EventProcessor::EventProcessor(std::shared_ptr<edm::ProcessDesc>, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) (/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-02-28-1100/lib/slc7_amd64_gcc11/libFWCoreFramework.so+0x2b58e0)
    #18 0x40e36b in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (/cvmfs/cms-ib.cern.ch/nweek-02722/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-02-28-1100/bin/slc7_amd64_gcc11/cmsRun+0x40e36b)
    #19 0x7fcb2a756897 in tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc11/external/tbb/v2021.4.0-3d940fce09ce7cacf58c68ba3ce04951/tbb-v2021.4.0/src/tbb/arena.cpp:698
    #20 0x411e7f in main::{lambda()#1}::operator()() const (/cvmfs/cms-ib.cern.ch/nweek-02722/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-02-28-1100/bin/slc7_amd64_gcc11/cmsRun+0x411e7f)
    #21 0x40c615 in main (/cvmfs/cms-ib.cern.ch/nweek-02722/slc7_amd64_gcc11/cms/cmssw/CMSSW_12_3_ASAN_X_2022-02-28-1100/bin/slc7_amd64_gcc11/cmsRun+0x40c615)
    #22 0x7fcb29570554 in __libc_start_main (/lib64/libc.so.6+0x22554)

==3127==HINT: if you don't care about these errors you may set allocator_may_return_null=1
SUMMARY: AddressSanitizer: invalid-aligned-alloc-alignment ../../../../libsanitizer/asan/asan_malloc_linux.cpp:192 in __interceptor_aligned_alloc
==3127==ABORTING
@smuzaffar
Copy link
Contributor Author

smuzaffar commented Mar 1, 2022 •
edited

assign reconstruction

@cmsbuild
Copy link
Contributor

cmsbuild commented Mar 1, 2022

A new Issue was created by @smuzaffar Malik Shahzad Muzaffar.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@cmsbuild
Copy link
Contributor

cmsbuild commented Mar 1, 2022

New categories assigned: reconstruction

@slava77,@jpata,@clacaputo you have been requested to review this Pull request/Issue and eventually sign? Thanks

@iarspider
Copy link
Contributor

@smuzaffar #37017 (comment)

@smuzaffar
Copy link
Contributor Author

right but ASAN issue is still open , so we need a fix for it.

@slava77
Copy link
Contributor

slava77 commented Mar 1, 2022

right but ASAN issue is still open , so we need a fix for it.

@osschar
please clarify on the status/progress of the fix.
Thank you.

@osschar
Copy link
Contributor

osschar commented Mar 1, 2022

i'll work on it today.

@osschar osschar mentioned this issue Mar 1, 2022
Merged
@perrotta
Copy link
Contributor

perrotta commented Mar 8, 2022

This should have been fixed by #37111, merged in 12_3_X since CMSSW_12_3_ASAN_X_2022-03-04-1100 and CMSSW_12_3_X_2022-03-02-2300

Can this issue be closed, if so?

@clacaputo
Copy link
Contributor

clacaputo commented Mar 8, 2022 •
edited

This should have been fixed by #37111, merged in 12_3_X since CMSSW_12_3_ASAN_X_2022-03-04-1100 and CMSSW_12_3_X_2022-03-02-2300

Can this issue be closed, if so?

Hi @perrotta , you're right

@clacaputo
Copy link
Contributor

+reconstruction

@cmsbuild
Copy link
Contributor

cmsbuild commented Mar 8, 2022

This issue is fully signed and ready to be closed.

@qliphy qliphy closed this as completed Mar 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
fully-signed reconstruction-approved
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@iarspider @osschar @perrotta @smuzaffar @cmsbuild @slava77 @qliphy @clacaputo