RelMon histogram comparison: adapt ranges in a compatible way with the definition of getNbins

[fabiocos]

PR description:

The RelMon DQM histogram comparison is complaining about differences between histograms when comparing a file with itself, which is clearly incorrect, see the discussion in #26496 . This issue looks linked to the inconsistent use in the utility package https://cmssdt.cern.ch/lxr/source/Utilities/RelMon/python/utils.py of number of bins and ranges. In python the range function has as a second argument not the last value of the list but the next to last. This looks likely the motivation for the function getNbins adding 1 unit to the number of bins as returned by the ROOT function getNbinsX(). There seems to be also some inconsistency between the starting point of the different range loops, in several cases bin 0 is used (underflow), sometime bin 1 (the first histogram bin).

This PR tries to solve the issue, having all the range based loops running from the underflow (bin=0) to the overflow (bin=GetNbinsX()+1), which means setting in the range loops as last item GetNbinsX()+2.

PR validation:

Running the test BinToBin to compare a file with itself does not show any more the problem. Using the DQM output of the generator workflow 503 with a syntax similar to cms-bot:

13:51 cmsdev25 4149> compare_using_files.py -B DQM/TimerService@3 -o out -C -R -p --no_successes -s b2b 503.0_MinBias_13TeV_pythia8+MinBias_13TeV_pythia8+HARVESTGEN/DQM_V0001_R000000001__Global__CMSSW_X_Y_Z__RECO.root 503.0_MinBias_13TeV_pythia8+MinBias_13TeV_pythia8+HARVESTGEN/DQM_V0001_R000000001__Global__CMSSW_X_Y_Z__RECO.root
Analysing Histograms located in directory  at: 
 o 503.0_MinBias_13TeV_pythia8+MinBias_13TeV_pythia8+HARVESTGEN/DQM_V0001_R000000001__Global__CMSSW_X_Y_Z__RECO.root
 o 503.0_MinBias_13TeV_pythia8+MinBias_13TeV_pythia8+HARVESTGEN/DQM_V0001_R000000001__Global__CMSSW_X_Y_Z__RECO.root
We have a Blacklist:
 o Directory TimerService at level 3
++++++++++++++++++++++++++++++
Output Directory will be  out
Studying directory Info, 1/2
Studying directory Generator, 2/2
 ->Appending Generator... Appended.
Finished

 - summary of 840 tests:
 o Failiures: 0.00% (0/840)
 o Nulls: 0.00% (0/840) 
 o Successes: 100.00% (840/840) 
 o Skipped: 0.00% (0/840) 
 o Missing objects: 0
Pickleing the directory as out.pkl in dir /build/fabiocos/106X/relmon/CMSSW_10_6_X_2019-04-29-2300/work/out
Reading directory from out.pkl
Calculating stats for the directory...
Producing html...

[cmsbuild]

The code-checks are being triggered in jenkins.

[fabiocos]

@peruzzim FYI

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26593/9521

• This PR adds an extra 16KB to repository

[cmsbuild]

A new Pull Request was created by @fabiocos (Fabio Cossutti) for master.

It involves the following packages:

Utilities/RelMon

@perrotta, @cmsbuild, @slava77 can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @wddgit this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

[fabiocos]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/34456/console Started: 2019/05/02 14:09

[perrotta]

type bugfix

[cmsbuild]

+1
Tested at: 4c3c0ca
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26593/34456/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26593/34456/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 5 differences found in the comparisons
• DQMHistoTests: Total files compared: 33
• DQMHistoTests: Total histograms compared: 3211964
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3211759
• DQMHistoTests: Total skipped: 204
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 32 files compared)
• Checked 137 log files, 14 edm output root files, 33 DQM output files

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26593/9718

• This PR adds an extra 16KB to repository

[cmsbuild]

Pull request #26593 was updated. @perrotta, @cmsbuild, @slava77 can you please check and sign again.

[fabiocos]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/126/console Started: 2019/05/09 16:33

[cmsbuild]

+1
Tested at: 15e4114
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b0ed73/126/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b0ed73/126/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 33
• DQMHistoTests: Total histograms compared: 3211964
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3211759
• DQMHistoTests: Total skipped: 204
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 32 files compared)
• Checked 137 log files, 14 edm output root files, 33 DQM output files

[perrotta]

+1

• Fixed as planned in the description
• Jenkins tests pass

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

[fabiocos]

+1