[DQM] Fix to compareDQMOutput since python3

[jfernan2]

PR description:

Bin By Bin tool broken since python3 change
As pointed out in #34374 (comment)

[jfernan2]

urgent

[jfernan2]

please test

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34411/23814

• This PR adds an extra 12KB to repository

[cmsbuild]

A new Pull Request was created by @jfernan2 for master.

It involves the following packages:

DQMServices/FileIO (dqm)

@andrius-k, @kmaeshima, @ErnestaP, @ahmad3213, @jfernan2, @rvenditti can you please review it and eventually sign? Thanks.
@barvic this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy, @perrotta you are the release manager for this.

cms-bot commands are listed here

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-152b8c/16623/summary.html
COMMIT: 61d2c68
CMSSW: CMSSW_12_0_X_2021-07-08-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34411/16623/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @CMSTrackerDPG make CPEFast to better reproduce Generic (w/o track angle) #34286
• @cvuosalo [DD4hep] Add support for Assembly tags in XML and for creating DB payload #34344
• @cvuosalo [DD4hep] Add DD4hep Run 3 workflow that reads geometry from DB #34343
• @smuzaffar python2 clean: fixed unit test to use python3 #34398
• @Purva-Chaudhari Use std::bind in place of boost:bind #34328
• @OzAmram Remove 2D pixel template B field warning for 0T #34390
• @smuzaffar Use python3 to correct pick sqlalchemy and ROOT #34396
• @smuzaffar python2 cleanup: Updated unit test to use python3 #34397
• @VinInn enforce memory order to avoid crash on Ampere #34386

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-152b8c/16623/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-152b8c/16623/git-merge-result

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 5091 differences found in the comparisons
• DQMHistoTests: Total files compared: 38
• DQMHistoTests: Total histograms compared: 2786260
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 2786231
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: -0.004 KiB( 37 files compared)
• DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
• Checked 160 log files, 37 edm output root files, 38 DQM output files
• TriggerResults: no differences found

[jfernan2]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy, @perrotta (and backports should be raised in the release meeting by the corresponding L2)

[srimanob]

@qliphy @perrotta @silviodonato @jfernan2
I see that 12_0_0_pre4 is built already.

Do you think that this PR is needed when PdmV try to make a comparison using relmon?
https://github.com/cms-sw/cmssw/tree/master/Utilities/RelMon

[jfernan2]

Hi @srimanob
This PR affects only the DQM Bin By Bin tool, not RelMon

[silviodonato]

+1
Thanks @jfernan2 and @srimanob . Indeed I see that this code was added with #26283 and it seems independent on RelMon

[silviodonato]

@jfernan2 I still see

Exception comparing two root files: a bytes-like object is required, not 'str'

See dqmBinByBinLog.log in https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_0_X_2021-07-08-1100+152b8c/43938/

Do you think we will see this fixed only after the merge of the PR?

[smuzaffar]

@silviodonato , yes it should be fixed once we have a newer IB with the fix. I already have started 8h00 Ib

[jfernan2]

I confirm, thanks @smuzaffar

[smuzaffar]

@jfernan2 , looks like we still need more fixes fir dqm bin by bin test. Looks like it managed to upload the result but failed ot generate the html pages , see https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_0_X_2021-07-09-1100+5c6882/43968/dqmBinByBinLog.log

No differences were observed - output will not be written
Baseline file successfully opened
PR file successfully opened
No differences were observed - output will not be written
No differences were observed - output will not be written

Exception comparing two root files: list index out of range
Exception comparing two root files: list index out of range
Exception comparing two root files: list index out of range

[jfernan2]

I have created #34442 to fix the issue.
Thanks for the warning!