Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ASAN problem in L1TMuonEndCapTrackProducer #29332

Closed
Dr15Jones opened this issue Mar 28, 2020 · 27 comments
Closed

ASAN problem in L1TMuonEndCapTrackProducer #29332

Dr15Jones opened this issue Mar 28, 2020 · 27 comments
Labels
alca-pending l1-pending pending-signatures

Comments

@Dr15Jones
Copy link
Contributor

The address sanitizer has found the following problem

==2410==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x606001c10ee8 at pc 0x7fffc73d3622 bp 0x7ffffffece90 sp 0x7ffffffece88
READ of size 8 at 0x606001c10ee8 thread T0
    #0 0x7fffc73d3621 in emtf::Node::filterEventToDaughter(emtf::Event*) /build/chrjones/asan/CMSSW_11_1_ASAN_X_2020-03-27-2300/src/L1Trigger/L1TMuonEndCap/src/bdt/Node.cc:327
    #1 0x7fffc73fa108 in emtf::Tree::filterEventRecursive(emtf::Node*, emtf::Event*) /build/chrjones/asan/CMSSW_11_1_ASAN_X_2020-03-27-2300/src/L1Trigger/L1TMuonEndCap/src/bdt/Tree.cc:291
    #2 0x7fffc73fa130 in emtf::Tree::filterEventRecursive(emtf::Node*, emtf::Event*) /build/chrjones/asan/CMSSW_11_1_ASAN_X_2020-03-27-2300/src/L1Trigger/L1TMuonEndCap/src/bdt/Tree.cc:295
    #3 0x7fffc73fa130 in emtf::Tree::filterEventRecursive(emtf::Node*, emtf::Event*) /build/chrjones/asan/CMSSW_11_1_ASAN_X_2020-03-27-2300/src/L1Trigger/L1TMuonEndCap/src/bdt/Tree.cc:295
    #4 0x7fffc73fa130 in emtf::Tree::filterEventRecursive(emtf::Node*, emtf::Event*) /build/chrjones/asan/CMSSW_11_1_ASAN_X_2020-03-27-2300/src/L1Trigger/L1TMuonEndCap/src/bdt/Tree.cc:295
    #5 0x7fffc73fa130 in emtf::Tree::filterEventRecursive(emtf::Node*, emtf::Event*) /build/chrjones/asan/CMSSW_11_1_ASAN_X_2020-03-27-2300/src/L1Trigger/L1TMuonEndCap/src/bdt/Tree.cc:295
    #6 0x7fffc73fa0d6 in emtf::Tree::filterEvent(emtf::Event*) /build/chrjones/asan/CMSSW_11_1_ASAN_X_2020-03-27-2300/src/L1Trigger/L1TMuonEndCap/src/bdt/Tree.cc:281
    #7 0x7fffc73de7d6 in emtf::Forest::appendCorrection(emtf::Event*, int) /build/chrjones/asan/CMSSW_11_1_ASAN_X_2020-03-27-2300/src/L1Trigger/L1TMuonEndCap/src/bdt/Forest.cc:467
    #8 0x7fffc73de762 in emtf::Forest::predictEvent(emtf::Event*, unsigned int) /build/chrjones/asan/CMSSW_11_1_ASAN_X_2020-03-27-2300/src/L1Trigger/L1TMuonEndCap/src/bdt/Forest.cc:455
    #9 0x7fffc73a064a in PtAssignmentEngine2016::calculate_pt_xml(unsigned long const&) const /build/chrjones/asan/CMSSW_11_1_ASAN_X_2020-03-27-2300/src/L1Trigger/L1TMuonEndCap/src/PtAssignmentEngine2016.cc:668
    #10 0x7fffc731f6bb in PtAssignmentEngine::calculate_pt(unsigned long const&) const /build/chrjones/asan/CMSSW_11_1_ASAN_X_2020-03-27-2300/src/L1Trigger/L1TMuonEndCap/src/PtAssignmentEngine.cc:112
    #11 0x7fffc7283ce7 in PtAssignment::process(std::vector<l1t::EMTFTrack, std::allocator<l1t::EMTFTrack> >&) /build/chrjones/asan/CMSSW_11_1_ASAN_X_2020-03-27-2300/src/L1Trigger/L1TMuonEndCap/src/PtAssignment.cc:74
    #12 0x7fffc728d7b8 in SectorProcessor::process_single_bx(int, std::vector<L1TMuon::TriggerPrimitive, std::allocator<L1TMuon::TriggerPrimitive> > const&, std::vector<l1t::EMTFHit, std::allocator<l1t::EMTFHit> >&, std::vector<l1t::EMTFTrack, std::allocator<l1t::EMTFTrack> >&, std::deque<std::vector<l1t::EMTFHit, std::allocator<l1t::EMTFHit> >, std::allocator<std::vector<l1t::EMTFHit, std::allocator<l1t::EMTFHit> > > >&, std::deque<std::vector<l1t::EMTFTrack, std::allocator<l1t::EMTFTrack> >, std::allocator<std::vector<l1t::EMTFTrack, std::allocator<l1t::EMTFTrack> > > >&, std::map<std::array<int, 3ul>, int, std::less<std::array<int, 3ul> >, std::allocator<std::pair<std::array<int, 3ul> const, int> > >&) const /build/chrjones/asan/CMSSW_11_1_ASAN_X_2020-03-27-2300/src/L1Trigger/L1TMuonEndCap/src/SectorProcessor.cc:608
    #13 0x7fffc728b368 in SectorProcessor::process(unsigned long long, std::vector<L1TMuon::TriggerPrimitive, std::allocator<L1TMuon::TriggerPrimitive> > const&, std::vector<l1t::EMTFHit, std::allocator<l1t::EMTFHit> >&, std::vector<l1t::EMTFTrack, std::allocator<l1t::EMTFTrack> >&) const /build/chrjones/asan/CMSSW_11_1_ASAN_X_2020-03-27-2300/src/L1Trigger/L1TMuonEndCap/src/SectorProcessor.cc:432
    #14 0x7fffc7390887 in TrackFinder::process(edm::Event const&, edm::EventSetup const&, std::vector<l1t::EMTFHit, std::allocator<l1t::EMTFHit> >&, std::vector<l1t::EMTFTrack, std::allocator<l1t::EMTFTrack> >&) /build/chrjones/asan/CMSSW_11_1_ASAN_X_2020-03-27-2300/src/L1Trigger/L1TMuonEndCap/src/TrackFinder.cc:197
    #15 0x7fffbd31fbd5 in L1TMuonEndCapTrackProducer::produce(edm::Event&, edm::EventSetup const&) /build/chrjones/asan/CMSSW_11_1_ASAN_X_2020-03-27-2300/src/L1Trigger/L1TMuonEndCap/plugins/L1TMuonEndCapTrackProducer.cc:25
    #16 0x7ffff6e9f108 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_1_ASAN_X_2020-03-27-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x814108)
    #17 0x7ffff6a0cfbc in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_1_ASAN_X_2020-03-27-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x381fbc)
    #18 0x7ffff6863cb7 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) (/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_1_ASAN_X_2020-03-27-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x1d8cb7)
    #19 0x7ffff68641e0 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_1_ASAN_X_2020-03-27-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x1d91e0)
    #20 0x7ffff686dfcc in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) (/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_1_ASAN_X_2020-03-27-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x1e2fcc)
    #21 0x7ffff6873449 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() (/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_1_ASAN_X_2020-03-27-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x1e8449)
    #22 0x7ffff466d27c in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop(tbb::internal::context_guard_helper<false>&, tbb::task*, long) ../../src/tbb/custom_scheduler.h:469
    #23 0x7ffff466d574 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all(tbb::task&, tbb::task*) ../../src/tbb/custom_scheduler.h:631
    #24 0x7ffff6b94a86 in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) (/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_1_ASAN_X_2020-03-27-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x509a86)
    #25 0x7ffff6bb6d89 in edm::EventProcessor::runToCompletion() (/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_1_ASAN_X_2020-03-27-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so+0x52bd89)
    #26 0x41a2f0 in main::{lambda()#1}::operator()() const (/cvmfs/cms-ib.cern.ch/nweek-02621/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_1_ASAN_X_2020-03-27-2300/bin/slc7_amd64_gcc820/cmsRun+0x41a2f0)
    #27 0x4139c5 in main (/cvmfs/cms-ib.cern.ch/nweek-02621/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_1_ASAN_X_2020-03-27-2300/bin/slc7_amd64_gcc820/cmsRun+0x4139c5)
    #28 0x7ffff3304504 in __libc_start_main (/lib64/libc.so.6+0x22504)
    #29 0x413db8  (/cvmfs/cms-ib.cern.ch/nweek-02621/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_1_ASAN_X_2020-03-27-2300/bin/slc7_amd64_gcc820/cmsRun+0x413db8)

0x606001c10ee8 is located 8 bytes to the right of 64-byte region [0x606001c10ea0,0x606001c10ee0)
allocated by thread T0 here:
    #0 0x7ffff70ffd90 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cc:90
    #1 0x7ffff562a012 in std::vector<double, std::allocator<double> >::operator=(std::vector<double, std::allocator<double> > const&) (/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_1_ASAN_X_2020-03-27-2300/external/slc7_amd64_gcc820/lib/libMathCore.so+0xbd012)

SUMMARY: AddressSanitizer: heap-buffer-overflow /build/chrjones/asan/CMSSW_11_1_ASAN_X_2020-03-27-2300/src/L1Trigger/L1TMuonEndCap/src/bdt/Node.cc:327 in emtf::Node::filterEventToDaughter(emtf::Event*)
Shadow bytes around the buggy address:
  0x0c0c8037a180: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
  0x0c0c8037a190: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0c8037a1a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c8037a1b0: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
  0x0c0c8037a1c0: fd fd fd fa fa fa fa fa 00 00 00 00 00 00 00 00
=>0x0c0c8037a1d0: fa fa fa fa 00 00 00 00 00 00 00 00 fa[fa]fa fa
  0x0c0c8037a1e0: 00 00 00 00 00 00 00 fa fa fa fa fa fd fd fd fd
  0x0c0c8037a1f0: fd fd fd fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0c8037a200: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x0c0c8037a210: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c8037a220: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb

I got the stack trace by rebuilding the code with debug information on and then running workflow 1302.0 with just one thread.
@cmsbuild
Copy link
Contributor

A new Issue was created by @Dr15Jones Chris Jones.

@Dr15Jones, @smuzaffar, @silviodonato, @makortel, @davidlange6, @fabiocos can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@Dr15Jones
Copy link
Contributor Author

Adding an assert to the code just before the problem, I was able to look at the variables in the debugger. The problem was at this line

cmssw/L1Trigger/L1TMuonEndCap/src/bdt/Node.cc

Line 327 in fe5b448

if (e->data[sv] < sp)

where the value of sv comes from the member variable splitVariable and has the value of 9 and the rest of the object

 print *this
$4 = {name = "root left left left left", leftDaughter = 0x60d005371dc0, rightDaughter = 0x60d005371e90, parent = 0x60d005371a80, splitValue = 1, splitVariable = 9, errorReduction = -1, totalError = -1, avgError = -1, 
  fitValue = 0.00040209802682511508, numEvents = -1094795586, events = std::vector of length 0, capacity 0}

while e->data has size 8

(gdb) print *e
$3 = {trueValue = 0, predictedValue = 0.13586474777853488, DTPt = 0, CSCPt = 0, tmvaPt = 0, tmvaPt1 = 0, Mode = 0, Quality = 0, static sortingIndex = 1, id = 0, data = std::vector of length 8, capacity 8 = {1, 1.4750000238418579, -9, 
    -11, 1, -1, 0, 0}}

So there is a problem with the algorithm.

@Dr15Jones
Copy link
Contributor Author

assign l1

@cmsbuild
Copy link
Contributor

New categories assigned: l1

@benkrikler,@rekovic you have been requested to review this Pull request/Issue and eventually sign? Thanks

@Dr15Jones
Copy link
Contributor Author

The size of e->data is hard coded by the calling algorithm to by 8. See

cmssw/L1Trigger/L1TMuonEndCap/src/PtAssignmentEngine2016.cc

Lines 635 to 664 in fe5b448

std::vector<double> tree_data;
tree_data.push_back(1.0);
tree_data.push_back(eta);
for (int i = 0; i < 6; i++) { // loop over 6 variables (or less)
int mv = mode_variables[mode_inv - 3][i];
if (mv != -999) {
int v = variables.at(mv);
if (!(mode_inv == 13 && i == 3)) { // somehow this uses CSCID1
if (not(v != -999)) {
edm::LogError("L1T") << "v = " << v;
return -1;
}
}
tree_data.push_back(v);
} else {
tree_data.push_back(0); // pad with zeroes, somehow BDT tries to access out of bounds
}
}
if (verbose_ > 2) {
std::cout << "mode_inv: " << mode_inv << " variables: ";
for (const auto& v : tree_data)
std::cout << v << " ";
std::cout << std::endl;
}
auto tree_event = std::make_unique<emtf::Event>();
tree_event->predictedValue = 0; // must explicitly initialize
tree_event->data = tree_data;

So this seems to imply that the boosted decision tree being used is incompatible with the actual code calling it.

@Dr15Jones
Copy link
Contributor Author

We have 96 failing workflows in the ASAN IB and based on a random sampling of them it appears they all are from this problem.

@Dr15Jones
Copy link
Contributor Author

I've been unable to find old ASAN reports, so based solely on recent pull requests, the most likely culprints are #29260 or #29080

@Dr15Jones
Copy link
Contributor Author

ping

@Dr15Jones
Copy link
Contributor Author

Is anyone looking at this? I think it is also causing crashes in the ROOT6 IBs.

@jiafulow
Copy link
Contributor

I'm sorry for the delay in getting back. Could you check if the failing workflows are all using Run2_2016 era? If so, I think it's related to #29237 . I believe the main problem is due to loading the wrong global tag, and thus the wrong BDT trees. I hope there is no new issue in Node.cc.

@Dr15Jones
Copy link
Contributor Author

@jiafulow you can actually look yourself by going to
https://cmssdt.cern.ch/SDT/html/cmssdt-ib/#/relVal/CMSSW_11_1/2020-04-24-2300?selectedArchs=slc7_amd64_gcc820&selectedFlavors=ASAN_X&selectedStatus=failed

which was the most recent run of address sanitizer (ASAN). It looks like each of the failures labelled 256 are caused by this problem. It appears there are 93 workflows failing under ASAN because of this problem.

@makortel
Copy link
Contributor

makortel commented May 5, 2020

@jiafulow Any updates? The ASAN is still reporting the issue, e.g. here for workflow 1310.0
https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc820/CMSSW_11_1_ASAN_X_2020-05-01-2300/pyRelValMatrixLogs/run/1310.0_ADDMonoJet_d3MD3_13+ADDMonoJet_d3MD3_13INPUT+DIGIUP15+RECOUP15+HARVESTUP15/step2_ADDMonoJet_d3MD3_13+ADDMonoJet_d3MD3_13INPUT+DIGIUP15+RECOUP15+HARVESTUP15.log#/

@makortel
Copy link
Contributor

makortel commented May 5, 2020

assign alca

@cmsbuild
Copy link
Contributor

cmsbuild commented May 5, 2020

New categories assigned: alca

@christopheralanwest,@tlampen,@pohsun,@tocheng you have been requested to review this Pull request/Issue and eventually sign? Thanks

@makortel
Copy link
Contributor

makortel commented May 5, 2020

@christopheralanwest,@tlampen,@pohsun,@tocheng

I believe the main problem is due to loading the wrong global tag, and thus the wrong BDT trees.

Could you please help to double-check if the conditions payload(s) related to L1TMuonEndCapTrackProducer in e.g. workflow 1310.0 are correct or not? Thanks.

@jiafulow
Copy link
Contributor

jiafulow commented May 5, 2020

@makortel using the provided link (updated to 2020-05-01-2300) , the following workflows that failed with error code 256 (incl 1310.0) are all using '--era Run2_2016':

  • 135.1
  • 1302.0
  • 1310.0
  • 1311.0
  • 1312.0
  • 1314.0

(I didn't check beyond 1314.0. Note that 534.0 and 536.0 failures are not due to EMTF.)

As I wrote above, I believe it's related to the issue #29237 which was due to GlobalTag confusion. Unfortunately I don't really know how to fix it. It would require experts who know how to fix the GTs and how to load the correct GTs for the Run2_2016 era in 'runTheMatrix'.

@makortel
Copy link
Contributor

makortel commented May 5, 2020

Thanks @jiafulow, let's try to push #29237 to be fixed then.

@makortel makortel mentioned this issue May 5, 2020
Closed
@makortel
Copy link
Contributor

makortel commented May 7, 2020

Let me add that I find it disturbing that wrong conditions payload causes the code to misbehave this way. It would be much better that the code either always works according to the payload, or would be able check upfront if the payload is inconsistent with the code.

@jiafulow
Copy link
Contributor

jiafulow commented May 7, 2020

Thanks for the suggestion, this is something we need to do better. In this particular case, the payload is a set of BDT trees. The number of trees and the number of variables, as well as the definitions of the variables, are different for different eras (2016 vs 2017/18). When there is a mismatch between the payload and the C++ codes, it crashes.

@makortel
Copy link
Contributor

makortel commented May 7, 2020

When there is a mismatch between the payload and the C++ codes, it crashes.

Now it appears to not to crash but to produce undefined results.

@smuzaffar
Copy link
Contributor

@jiafulow , We still have this error in ASAN IBs[a], I debugged it a bit an it happens when splitVariable >= 8
( https://github.com/cms-sw/cmssw/blob/master/L1Trigger/L1TMuonEndCap/src/bdt/Node.cc#L327 )
as @Dr15Jones mentioned the event data has fixed size of 8 ( https://github.com/cms-sw/cmssw/blob/master/L1Trigger/L1TMuonEndCap/src/PtAssignmentEngine2016.cc#L632-L647 )

[a]
https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc820/CMSSW_11_2_ASAN_X_2020-05-22-2300/pyRelValMatrixLogs/run/1330.0_ZMM_13+ZMM_13INPUT+DIGIUP15+RECOUP15_L1TMuDQM+HARVESTUP15_L1TMuDQM+NANOUP15/step2_ZMM_13+ZMM_13INPUT+DIGIUP15+RECOUP15_L1TMuDQM+HARVESTUP15_L1TMuDQM+NANOUP15.log#/

@jiafulow
Copy link
Contributor

jiafulow commented May 29, 2020 •
edited

@smuzaffar , I explained the issue in #29237. The quick summary is that back in Feb 2020, someone from L1T has made changes to the global tag used in 10_6_X for era=Run2_2016. This revealed some issue in the software (specifically, the BDT loaded from global tag did not match the software after the change). We fixed that and put the fix in both 10_6_X and 11_1_X. But it turned out that the global tag used in 11_1_X for era=Run2_2016 has not been updated accordingly. This means that, after the fix, the loaded BDT did not match the software again, but now in 11_1_X.

We are still waiting for #29237 to be resolved. In the meanwhile, perhaps I can try some incomplete fix? I'm not sure how to debug ASAN errors. If I modify the problematic lines at Node.cc#L327-L330 to do:

  if (sv < e->data.size() && e->data[sv] < sp)
    nextNode = left;
  if (sv < e->data.size() && e->data[sv] >= sp)
    nextNode = right;

Would it make the ASAN errors go away?

@smuzaffar
Copy link
Contributor

smuzaffar commented May 29, 2020 •
edited

Thanks @jiafulow for the explaination. Sorry I did not read the #29237

You can reproduce the error in ASAN IBs by doing this

scram p CMSSW_11_2_ASAN_X_2020-05-22-2300
cd CMSSW_11_2_ASAN_X_2020-05-22-2300/
cmsenv
runTheMatrix.py -i all --ibeos -t 4 -l 1330.0

Yes your proposed fix should avoid the ASAN error but I would suggest the following change

-  int sv = splitVariable;
+  unsigned int sv = splitVariable;
   double sp = splitValue;
 
   Node* left = leftDaughter;
   Node* right = rightDaughter;
   Node* nextNode = nullptr;
 
-  if (left == nullptr || right == nullptr)
+  if (left == nullptr || right == nullptr || sv >= e->data.size())

@smuzaffar smuzaffar mentioned this issue May 29, 2020
Closed
@jiafulow
Copy link
Contributor

@smuzaffar , thanks for the instructions. I tried your suggestion and it seems to work. Before the fix, I got:

1 0 0 0 0 tests passed, 0 1 0 0 0 failed

After the fix, I got:

1 1 1 1 1 tests passed, 0 0 0 0 0 failed

Would you implement the fix? Or should I submit a pull request?

@smuzaffar
Copy link
Contributor

Go ahead @jiafulow and sumbit the PR. Thanks,

@jiafulow jiafulow mentioned this issue May 29, 2020
Merged
@jiafulow
Copy link
Contributor

Ok, I submitted a PR with your suggestion.

@smuzaffar
Copy link
Contributor

ASAN IBs are good now, we do not see this isssue any more

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
alca-pending l1-pending pending-signatures
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@makortel @Dr15Jones @smuzaffar @cmsbuild @jiafulow