Monitor track selection MVAs in MultiTrackValidator

[makortel]

This PR extends MultiTrackValidator (and the plotting scripts) to optionally monitor track selection MVA variables. The motivation is to provide a monitoring tool for track selection MVA tuning.

The monitoring is enabled only for the trackValidatorBuildingTrackingOnly module (monitoring the "built tracks"), because the MVA variables are easily available for those, and the track collection of an iteration N is not biased by the existing track selection of itereration N. The plots include

• Distributions of the MVA variable for true and fake tracks
• Efficiency and fake rate vs. cut on the MVA variable
  • for this I had to extend DQMGenericClient to be able to do the accumulation for the "cumulative histograms" also in the other direction
    • not sure if the "ascending"/"descending" nomenclature is the clearest one, better suggestions are welcome
  • from these we can make ROC curves
• Plotting includes also the selection efficiencies of true and fake tracks as a function of the cut to allow direct cross-check with TMVA plots
• Iterations with multiple MVA selections (like initialStep) are treated as follows
  • For the first MVA, histograms are filled for all tracks
  • For the second MVA and onwards, histograms are filled only for the not-yet-selected tracks. There are separate histograms for loose-selected and highPurity-selected tracks.
  • The order of MVA's is the one in ClassifierMerger.inputClassifiers of that iteration.

In addition this PR fixes a minor bug in makeTrackValidationPlots.py (without any options it makes plots for the "pT>0.9 GeV" folders, while that should be active only with --ptcut option).

Tested in CMSSW_8_1_0_pre15, no changes expected in standard workflows. New histograms should appear in "trackingOnly" workflows (e.g. 1325.1, 10024.1), but there should be no changes in existing histograms.

@rovere @VinInn

[cmsbuild]

A new Pull Request was created by @makortel (Matti Kortelainen) for CMSSW_8_1_X.

It involves the following packages:

DQMServices/ClientConfig
Validation/RecoTrack

@cmsbuild, @dmitrijus, @vanbesien, @davidlange6 can you please review it and eventually sign? Thanks.
@barvic, @GiacomoSguazzoni, @rovere, @VinInn, @wmtford, @dgulhan this is something you requested to watch as well.
@slava77, @smuzaffar you are the release manager for this.

cms-bot commands are listed here #13028

[VinInn]

@cmsbuild , please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/16312/console

[cmsbuild]

+1
Tested at: e10a1e6
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-16559/16312/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-16559/16312/summary.html

[makortel]

Generation of alternative-comparisons failed for 25202 with

/cvmfs/cms-ib.cern.ch/2016-46/slc6_amd64_gcc530/lcg/root/6.06.08-giojec2/lib/libGraf.so: error adding symbols: Bad address
collect2: error: ld returned 1 exit status
Error in <ACLiC>: Compilation failed!
input_line_18:2:3: error: use of undeclared identifier 'compareAll'
 (compareAll(((*(class TFile **)0x7fddfa076028)), ((*(class TFile **)0x7fddfa076030)), 0, 0, ""))
  ^

(but likely not important-enough to relaunch the tests since the relmon exists)

[slava77]

strange, considering that all other comparisons passed.
Maybe there was a glitch with cvmfs or smth like that.

@smuzaffar
Could you please grep for the same Compilation failed in alternative comparisons.
Thank you.

[smuzaffar]

sorry @slava77 , you want what?

[slava77]

@smuzaffar
as commented in #16559 (comment)
the workflow 25202 alternative-comparisons failed to compile https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_8_1_X_2016-11-11-1100+16559/16894/alternative-comparisons/runDQMComp-25202.0.log.

I wanted to find out if this is happening frequently.
You probably have interactive access to the output files and could check if there are other compilation failures in runDQMComp-*.log files

[smuzaffar]

@slava77 , only workflow 25202.0 has this error.

$ grep 'Compilation failed' alternative-comparisons/*.log
alternative-comparisons/runDQMComp-25202.0.log:Error in <ACLiC>: Compilation failed!

I will add a check for this include some error/warning the comparison ready message.

[dmitrijus]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_8_1_X IBs (tests are also fine). This pull request requires discussion in the ORP meeting before it's merged. @slava77, @davidlange6, @smuzaffar

[davidlange6]

+1

[davidlange6]

Hi @makortel - could you look at fixing this problem, that seems related to this PR. Thanks !

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc7_amd64_gcc530/CMSSW_8_1_X_2016-11-15-2300/pyRelValMatrixLogs/run/10024.5_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFullINPUT+DigiFull_2017+RecoFull_trackingOnlyPhase1PU70_2017+HARVESTFull_trackingOnly_2017/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFullINPUT+DigiFull_2017+RecoFull_trackingOnlyPhase1PU70_2017+HARVESTFull_trackingOnly_2017.log

[makortel]

Thanks David, very likely from this PR. I'll take a look.

[makortel]

Forgot to mention here that the fixes are in #16660 (90X) and #16661 (81X).