Various updates to MultiTrackValidator #19582

makortel · 2017-07-06T11:53:22Z

This PR provides the following updates to MultiTrackValidator

Add 2D histogram showing the originalAlgos of duplicate tracks
- From two best-matching tracks of a single TrackingParticle
Turn the plots vs. production radius to have logarithmic x axis
- To gain vsial sensitivity also for non-strip-seeded iterations
Make resolution and pull plots (and certain other 2D histograms) optional
- The required 2D histograms take lots of memory, but are usually interesting only for "all tracks" collections (not per iteration)
- By default the resolutions are limited to generalTracks, highPurity tracks, and BTV-selected tracks
Move histograms by originalAlgo to the standard sequence (from standalone/trackingOnly)
- We want to have the monitoring also by original iteration in addition to the "best algo assignment"
Move monitoring of built tracks to the standard sequence (from trackingOnly)
- Mainly to monitor the track selection MVAs, but also the effect of track selection in general
Add efficiency plots per iteration for pT < 0.9 GeV TrackingParticles
- To monitor the performance of iterations targeting low-pT tracks
- For this I had to add ptMax parameter to TrackingParticleSelector
- The "efficiency vs. collection without pT cut" histograms were removed as obsolete now
Add profiles of track selection MVAs vs pT and eta (as in Add track selection MVA plots to tracking DQM #19481 for DQM) separately for true and fake tracks
Some cleanup (e.g. merging MultiTrackValidatorBase to MultiTrackValidator)

The net effect on memory (in 10024.0) is going from ~100 MB to ~80 MB per thread.

Tested in 9_3_0_pre1. Expecting removal of resolution and pull plots except for the collections mentioned above, and addition of new histograms. No change is expected for any existing histogram that is not removed.

@rovere @VinInn @ebrondol @felicepantaleo

There are no other deriving classes, so the base class was just additional complication.

…ected

Mainly to monitor the track selection MVA in RelVals.

Made obsolete by the low-pT efficiencty plots per iteration.

It makes no sense to make algoMask selection for generalTracks+HP, nor originalAlgo for generalTracks+duplicateMerge+their HP.

I'm not really sure at which point these were introduced (even before the previous commit the code should have crashed with "multiple calls to Sumw2" exception.

cmsbuild · 2017-07-06T11:53:42Z

A new Pull Request was created by @makortel (Matti Kortelainen) for master.

It involves the following packages:

CommonTools/RecoAlgos
PhysicsTools/RecoAlgos
SLHCUpgradeSimulations/Geometry
SimGeneral/MixingModule
SimGeneral/TrackingAnalysis
SimTracker/Common
SimTracker/VertexAssociation
Validation/RecoMuon
Validation/RecoTrack

@perrotta, @civanch, @Dr15Jones, @vazzolini, @kmaeshima, @ianna, @kpedro88, @dmitrijus, @cmsbuild, @mdhildreth, @slava77, @vanbesien, @monttj, @davidlange6 can you please review it and eventually sign? Thanks.
@TaiSakuma, @felicepantaleo, @schoef, @rappoccio, @Martin-Grunewald, @threus, @mmusich, @seemasharmafnal, @LBeck, @mmarionncern, @battibass, @ahinzmann, @sdevissc, @jlagram, @jhgoh, @cnuttens, @prolay, @HuguesBrun, @trocino, @rociovilar, @abbiendi, @GiacomoSguazzoni, @rovere, @VinInn, @nhanvtran, @gkasieczka, @wmtford, @ebrondol, @mtosi, @dgulhan, @jdolen, @gbenelli, @calderona, @mariadalfonso this is something you requested to watch as well.
@davidlange6 you are the release manager for this.

cms-bot commands are listed here

makortel · 2017-07-06T11:54:08Z

@cmsbuild, please test

cmsbuild · 2017-07-06T11:54:28Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/21213/console Started: 2017/07/06 15:15

cmsbuild · 2017-07-06T14:59:25Z

+1
Tested at: 785930a
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19582/21213/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
6625d2d
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19582/21213/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19582/21213/git-merge-result

cmsbuild · 2017-07-06T14:59:28Z

Comparison job queued.

cmsbuild · 2017-07-06T17:37:05Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19582/21213/summary.html

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/build/cmsbld/jenkins-workarea/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-19582/136.7611_RunJetHT2016E_reminiaod+RunJetHT2016E_reminiaod+REMINIAOD_data2016+HARVESTDR2_REMINIAOD_data2016

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 23
DQMHistoTests: Total histograms compared: 1756345
DQMHistoTests: Total failures: 34118
DQMHistoTests: Total nulls: 3228
DQMHistoTests: Total successes: 1718833
DQMHistoTests: Total skipped: 166
DQMHistoTests: Total Missing objects: 0
Checked 93 log files, 14 edm output root files, 23 DQM output files

ianna · 2017-07-07T08:10:38Z

+1

civanch · 2017-07-07T08:44:31Z

+1

perrotta · 2017-07-07T14:40:27Z

+1
Modifications to reco related configs are trivial
No changes in jenkins outputs: only Tracking related DQM plots show differences in the way they are displayed

dmitrijus · 2017-07-10T08:14:29Z

+1

davidlange6 · 2017-07-10T09:45:03Z

merge

makortel added 18 commits July 5, 2017 13:34

Add duplicate oriAlgo vs oriAlgo histogram

d1d3301

Add duplica algo vs algo to plotting

c9be1e4

Turn plots vs radius to logarithmic x axis

1936fc7

Merge MultiTrackValidatorBase to MultiTrackValidator

0925444

There are no other deriving classes, so the base class was just additional complication.

Make resolution plots and some other 2D histograms optional

afd46df

Limit resolution plots only to generalTracks, highPurity, and BTV-sel…

83b9659

…ected

Move built tracks from trackingOnly to standard

71dda8e

Mainly to monitor the track selection MVA in RelVals.

Move histograms by original algo from standalone to standard sequence

7182f02

Update makeTrackValidationPlots.py --limit-relval

e00c8cf

Add ptMax parameter to TrackingParticleSelector

1b37b87

Add MTV instace for pT < 0.9 GeV efficiency

8f114a8

Remove spurious appearance of Pt09 summary

0462543

Fix duplicate sections in non-tracking pages

cb8fb14

Remove efficiency vs. collection without pT cut as obsolete

5ffcd65

Made obsolete by the low-pT efficiencty plots per iteration.

Remove spurious input collections

424d99a

It makes no sense to make algoMask selection for generalTracks+HP, nor originalAlgo for generalTracks+duplicateMerge+their HP.

Add profiles of selection MVA vs. pT and eta

76f57ca

Check for duplicate input track collections

53a9991

Remove duplicate input collections

785930a

I'm not really sure at which point these were introduced (even before the previous commit the code should have crashed with "multiple calls to Sumw2" exception.

cmsbuild added this to the CMSSW_9_3_X milestone Jul 6, 2017

cmsbuild added analysis-pending comparison-pending dqm-pending geometry-pending orp-pending pending-signatures reconstruction-pending simulation-pending tests-pending upgrade-pending labels Jul 6, 2017

cmsbuild added tests-started and removed tests-pending labels Jul 6, 2017

cmsbuild added tests-approved and removed tests-started labels Jul 6, 2017

cmsbuild added comparison-available and removed comparison-pending labels Jul 6, 2017

cmsbuild added geometry-approved and removed geometry-pending labels Jul 7, 2017

cmsbuild added simulation-approved and removed simulation-pending labels Jul 7, 2017

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Jul 7, 2017

cmsbuild added dqm-approved and removed dqm-pending labels Jul 10, 2017

cmsbuild merged commit bffc6f3 into cms-sw:master Jul 10, 2017

makortel mentioned this pull request Jul 24, 2017

Fix MultiTrackValidator histograms vs. radius #19890

Merged

makortel mentioned this pull request Aug 7, 2017

Add built tracks back to trackingOnly MultiTrackValidator #20073

Merged

makortel mentioned this pull request Sep 12, 2017

Fix summary harvesting for built tracks MTV #20470

Merged

makortel deleted the mtvLowPt branch February 12, 2018 12:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Various updates to MultiTrackValidator #19582

Various updates to MultiTrackValidator #19582

makortel commented Jul 6, 2017

cmsbuild commented Jul 6, 2017

makortel commented Jul 6, 2017

cmsbuild commented Jul 6, 2017 •

edited

cmsbuild commented Jul 6, 2017

cmsbuild commented Jul 6, 2017

cmsbuild commented Jul 6, 2017

ianna commented Jul 7, 2017

civanch commented Jul 7, 2017

perrotta commented Jul 7, 2017

dmitrijus commented Jul 10, 2017

davidlange6 commented Jul 10, 2017

Various updates to MultiTrackValidator #19582

Various updates to MultiTrackValidator #19582

Conversation

makortel commented Jul 6, 2017

cmsbuild commented Jul 6, 2017

makortel commented Jul 6, 2017

cmsbuild commented Jul 6, 2017 • edited

cmsbuild commented Jul 6, 2017

cmsbuild commented Jul 6, 2017

cmsbuild commented Jul 6, 2017

ianna commented Jul 7, 2017

civanch commented Jul 7, 2017

perrotta commented Jul 7, 2017

dmitrijus commented Jul 10, 2017

davidlange6 commented Jul 10, 2017

cmsbuild commented Jul 6, 2017 •

edited