RecoVertex/PrimaryVertexProducer: fix heap buffer overflow found by ASAN. #32028

gartung · 2020-11-04T17:01:47Z

PR description:

Change size of pcut_cache from nt to nv to avoid heap buffer overflow when nt < nv.

PR validation:

Ran runTheMatrix in CMSSW_11_2_ASAN_X_2020-11-02-2300.

if this PR is a backport please specify the original PR and why you need to backport that PR:

Before submitting your pull requests, make sure you followed this checklist:

verify that the PR is really intended for the chosen branch
verify that changes follow CMS Naming, Coding, And Style Rules
verify that the PR passes the basic test procedure suggested in the CMSSW PR instructions

cmsbuild · 2020-11-04T17:02:09Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-11-04T17:07:04Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-32028/19573

This PR adds an extra 20KB to repository

cmsbuild · 2020-11-04T17:07:28Z

A new Pull Request was created by @gartung (Patrick Gartung) for master.

It involves the following packages:

RecoVertex/PrimaryVertexProducer

@perrotta, @jpata, @cmsbuild, @slava77 can you please review it and eventually sign? Thanks.
@makortel, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @ebrondol, @mtosi, @dgulhan this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

gartung · 2020-11-04T17:12:26Z

please test

cmsbuild · 2020-11-04T17:12:51Z

@gartung, CMSSW_11_2_X branch is closed for direct updates. cms-bot is going to move this PR to master branch.
In future, please use cmssw master branch to submit your changes.

cmsbuild · 2020-11-04T17:13:12Z

The tests are being triggered in jenkins.

CMSSW_11_2_X_2020-11-04-1100/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/10510/console Started: 2020/11/04 18:15

gartung · 2020-11-04T17:33:44Z

please test for CMSSW_11_2_ASAN_X

cmsbuild · 2020-11-04T17:34:06Z

The tests are being triggered in jenkins.
Test Parameters:

RELEASE_FORMAT = CMSSW_11_2_ASAN_X
CMSSW_11_2_ASAN_X_2020-11-02-2300/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/10511/console Started: 2020/11/04 18:35

RecoVertex/PrimaryVertexProducer/src/DAClusterizerInZ_vect.cc

cmsbuild · 2020-11-04T22:08:12Z

-1

Tested at: 787bba7

CMSSW: CMSSW_11_2_ASAN_X_2020-11-02-2300
SCRAM_ARCH: slc7_amd64_gcc820

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

HCAL Pulse Filtering for OOT PU Subtraction #31661 @JHiltbrand: HCAL Pulse Filtering for OOT PU Subtraction
Clean BuildFiles in OnlineDB/CSCCondDB #31912 @guitargeek: Clean BuildFiles in OnlineDB/CSCCondDB
MTD geometry: add proton ProdCuts to active MTD scenarios #31993 @fabiocos: MTD geometry: add proton ProdCuts to active MTD scenarios
Switch to Geant4 thermal neutron physics #32009 @civanch: Switch to Geant4 thermal neutron physics
Clean BuildFiles in EventFilter #31898 @guitargeek: Clean BuildFiles in EventFilter
Resolve a few dead assignments in CalibrationHcalCalibAlgos plugins #31948 @perrotta: Resolve a few dead assignments in CalibrationHcalCalibAlgos plugins
[MillePedeAlignmentAlgorithm] Increase default constraints precision #31992 @vbotta: [MillePedeAlignmentAlgorithm] Increase default constraints precision
Run3-gex32x Complete the changes to the ECAL geometry code (clean up) #32006 @bsunanda: Run3-gex32x Complete the changes to the ECAL geometry code (clean up)
Run3-hcx269 Utilize ESGetToken in HCAL Hit validation code #32007 @bsunanda: Run3-hcx269 Utilize ESGetToken in HCAL Hit validation code
Phase2-hgx267 Assign part/orient from flat file in the HGCal geometry definition #31994 @bsunanda: Phase2-hgx267 Assign part/orient from flat file in the HGCal geometry definition

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ecae04/10515/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ecae04/10515/git-merge-result

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ecae04/10515/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests RelVals AddOn

Unit Tests:

I found errors in the following unit tests:

---> test testUCTUnpacker had ERRORS
---> test TestDQMServicesDemo had ERRORS

RelVals:

When I ran the RelVals I found an error in the following workflows:
1001.0 step4

runTheMatrix-results/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVDSIPIXELCALRUN1+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5/step4_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVDSIPIXELCALRUN1+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5.log

23234.0 step3

runTheMatrix-results/23234.0_TTbar_14TeV+2026D49+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal/step3_TTbar_14TeV+2026D49+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal.log

28234.0 step3

runTheMatrix-results/28234.0_TTbar_14TeV+2026D60+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal/step3_TTbar_14TeV+2026D60+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal.log

23434.999 step4

runTheMatrix-results/23434.999_TTbar_14TeV+2026D49PU_PMXS1S2PR+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+PREMIX_PremixHLBeamSpot14PU+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU/step4_TTbar_14TeV+2026D49PU_PMXS1S2PR+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+PREMIX_PremixHLBeamSpot14PU+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU.log

AddOn:

I found errors in the following addon tests:

cmsRun /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-02-2300/src/HLTrigger/Configuration/test/OnLine_HLT_GRun.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Wed Nov 4 23:03:26 2020-date Wed Nov 4 22:15:37 2020 s - exit: 256
cmsRun /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-02-2300/src/HLTrigger/Configuration/test/OnLine_HLT_PRef.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Wed Nov 4 22:52:17 2020-date Wed Nov 4 22:15:43 2020 s - exit: 256
cmsRun /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-02-2300/src/HLTrigger/Configuration/test/OnLine_HLT_Fake.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Wed Nov 4 22:51:05 2020-date Wed Nov 4 22:15:53 2020 s - exit: 256
cmsRun /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-02-2300/src/HLTrigger/Configuration/test/OnLine_HLT_HIon.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Wed Nov 4 22:53:56 2020-date Wed Nov 4 22:15:56 2020 s - exit: 256
cmsRun /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-02-2300/src/HLTrigger/Configuration/test/OnLine_HLT_Fake2.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Wed Nov 4 22:54:16 2020-date Wed Nov 4 22:15:59 2020 s - exit: 256
cmsRun /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-02-2300/src/HLTrigger/Configuration/test/OnLine_HLT_Fake1.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Wed Nov 4 22:53:34 2020-date Wed Nov 4 22:16:04 2020 s - exit: 256
cmsRun /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-02-2300/src/HLTrigger/Configuration/test/OnLine_HLT_PIon.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Wed Nov 4 22:59:45 2020-date Wed Nov 4 22:30:16 2020 s - exit: 256

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

HCAL Pulse Filtering for OOT PU Subtraction #31661 @JHiltbrand: HCAL Pulse Filtering for OOT PU Subtraction
Clean BuildFiles in OnlineDB/CSCCondDB #31912 @guitargeek: Clean BuildFiles in OnlineDB/CSCCondDB
MTD geometry: add proton ProdCuts to active MTD scenarios #31993 @fabiocos: MTD geometry: add proton ProdCuts to active MTD scenarios
Switch to Geant4 thermal neutron physics #32009 @civanch: Switch to Geant4 thermal neutron physics
Clean BuildFiles in EventFilter #31898 @guitargeek: Clean BuildFiles in EventFilter
Resolve a few dead assignments in CalibrationHcalCalibAlgos plugins #31948 @perrotta: Resolve a few dead assignments in CalibrationHcalCalibAlgos plugins
[MillePedeAlignmentAlgorithm] Increase default constraints precision #31992 @vbotta: [MillePedeAlignmentAlgorithm] Increase default constraints precision
Run3-gex32x Complete the changes to the ECAL geometry code (clean up) #32006 @bsunanda: Run3-gex32x Complete the changes to the ECAL geometry code (clean up)
Run3-hcx269 Utilize ESGetToken in HCAL Hit validation code #32007 @bsunanda: Run3-hcx269 Utilize ESGetToken in HCAL Hit validation code
Phase2-hgx267 Assign part/orient from flat file in the HGCal geometry definition #31994 @bsunanda: Phase2-hgx267 Assign part/orient from flat file in the HGCal geometry definition

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ecae04/10515/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ecae04/10515/git-merge-result

cmsbuild · 2020-11-04T22:08:14Z

Comparison job queued.

perrotta · 2020-11-05T10:08:59Z

This is a rather obvious and trivial fix (thank you @gartung ) and the remaining errors while running with ASAN are due to other open issues (one of them should be fixed by #32021, but the tests I've tried to launch there together with PR for CMSSW_11_2_ASAN_X did not start... ). I think this could even be merged as such, perhaps after having tested it in the "normal" IB.

perrotta · 2020-11-05T10:09:06Z

please test

cmsbuild · 2020-11-05T10:09:23Z

The tests are being triggered in jenkins.

CMSSW_11_2_X_2020-11-05-1100/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/10526/console Started: 2020/11/05 17:06

cmsbuild · 2020-11-05T17:24:18Z

+1
Tested at: 787bba7
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ecae04/10526/summary.html
CMSSW: CMSSW_11_2_X_2020-11-05-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-11-05T17:24:20Z

Comparison job queued.

cmsbuild · 2020-11-05T18:44:05Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ecae04/10526/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 35
DQMHistoTests: Total histograms compared: 2544144
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2544115
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 34 files compared)
Checked 149 log files, 22 edm output root files, 35 DQM output files

perrotta · 2020-11-06T06:46:27Z

+1

Trivial and logical fix
Jenkins tests pass with no no difference, as expected because it is unlikely that there are less tracks than vertices in an event

cmsbuild · 2020-11-06T06:46:51Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

silviodonato · 2020-11-06T08:26:00Z

+1

cmsbuild added this to the CMSSW_11_2_X milestone Nov 4, 2020

gartung mentioned this pull request Nov 4, 2020

AddressSanitizer: heap-buffer-overflow DAClusterizerInZ_vect::purge #32027

Closed

cmsbuild added code-checks-pending comparison-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Nov 4, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 4, 2020

gartung changed the base branch from master to CMSSW_11_2_X November 4, 2020 17:11

cmsbuild changed the base branch from CMSSW_11_2_X to master November 4, 2020 17:12

cmsbuild added tests-started and removed tests-pending labels Nov 4, 2020

cmsbuild added tests-pending and removed tests-started labels Nov 4, 2020

cmsbuild added tests-started and removed tests-pending labels Nov 4, 2020

dan131riley reviewed Nov 4, 2020

View reviewed changes

RecoVertex/PrimaryVertexProducer/src/DAClusterizerInZ_vect.cc Outdated Show resolved Hide resolved

gartung force-pushed the gartung-RecoVertex-PrimaryVertexProducer-fix-heap-buffer-overflow branch from 033c003 to 45ec02a Compare November 4, 2020 17:43

cmsbuild removed code-checks-approved tests-started labels Nov 4, 2020

cmsbuild added tests-rejected and removed tests-started labels Nov 4, 2020

perrotta mentioned this pull request Nov 5, 2020

[HGCAL] Corrections in noise vector in CLUE #32021

Merged

cmsbuild added tests-pending tests-started and removed tests-rejected tests-pending labels Nov 5, 2020

cmsbuild added tests-approved and removed tests-started labels Nov 5, 2020

cmsbuild added comparison-available and removed comparison-pending labels Nov 5, 2020

cmsbuild added fully-signed reconstruction-approved and removed pending-signatures reconstruction-pending labels Nov 6, 2020

cmsbuild added orp-approved and removed orp-pending labels Nov 6, 2020

cmsbuild merged commit 4a58219 into cms-sw:master Nov 6, 2020

gartung deleted the gartung-RecoVertex-PrimaryVertexProducer-fix-heap-buffer-overflow branch December 16, 2020 23:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RecoVertex/PrimaryVertexProducer: fix heap buffer overflow found by ASAN. #32028

RecoVertex/PrimaryVertexProducer: fix heap buffer overflow found by ASAN. #32028

gartung commented Nov 4, 2020 •

edited

cmsbuild commented Nov 4, 2020

cmsbuild commented Nov 4, 2020

cmsbuild commented Nov 4, 2020

gartung commented Nov 4, 2020

cmsbuild commented Nov 4, 2020

cmsbuild commented Nov 4, 2020 •

edited

gartung commented Nov 4, 2020

cmsbuild commented Nov 4, 2020 •

edited

cmsbuild commented Nov 4, 2020

cmsbuild commented Nov 4, 2020

perrotta commented Nov 5, 2020

perrotta commented Nov 5, 2020

cmsbuild commented Nov 5, 2020 •

edited

cmsbuild commented Nov 5, 2020

cmsbuild commented Nov 5, 2020

cmsbuild commented Nov 5, 2020

perrotta commented Nov 6, 2020

cmsbuild commented Nov 6, 2020

silviodonato commented Nov 6, 2020

RecoVertex/PrimaryVertexProducer: fix heap buffer overflow found by ASAN. #32028

RecoVertex/PrimaryVertexProducer: fix heap buffer overflow found by ASAN. #32028

Conversation

gartung commented Nov 4, 2020 • edited

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented Nov 4, 2020

cmsbuild commented Nov 4, 2020

cmsbuild commented Nov 4, 2020

gartung commented Nov 4, 2020

cmsbuild commented Nov 4, 2020

cmsbuild commented Nov 4, 2020 • edited

gartung commented Nov 4, 2020

cmsbuild commented Nov 4, 2020 • edited

cmsbuild commented Nov 4, 2020

cmsbuild commented Nov 4, 2020

perrotta commented Nov 5, 2020

perrotta commented Nov 5, 2020

cmsbuild commented Nov 5, 2020 • edited

cmsbuild commented Nov 5, 2020

cmsbuild commented Nov 5, 2020

cmsbuild commented Nov 5, 2020

perrotta commented Nov 6, 2020

cmsbuild commented Nov 6, 2020

silviodonato commented Nov 6, 2020

gartung commented Nov 4, 2020 •

edited

cmsbuild commented Nov 4, 2020 •

edited

cmsbuild commented Nov 4, 2020 •

edited

cmsbuild commented Nov 5, 2020 •

edited