DQMIO-related tools

[schneiml]

PR description:

This PR adds some helper tools useful for DQM development:

• root2sqlite.py is a very generic script that takes a ROOT file containing directories and/or trees and transforms the data in SQLite tables. It does not work on all root files, but, among others, DQMIO and legacy DQM output. Nontrivial queries are IMO much easier on SQLite than ROOT data...

• dqmiodatasetharvest.py demonstrates how to read MEs from DQMIO data available via DAS. It is not supposed to be a general tool, but with minor changes it can be used for operations like historic DQM. It does exploit the fact that DQMIO data is saved per job, and therefore per-lumisection granularity data is available in prompt RECO output. It also contains code to convert the resulting file back into a TTree, but this is not done automatically.

PR validation:

No existing code is affected. The tools have proven useful in my daily work.

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26337/9051

• This PR adds an extra 24KB to repository

[cmsbuild]

A new Pull Request was created by @schneiml (Marcel Schneider) for master.

It involves the following packages:

DQMServices/Components

@andrius-k, @kmaeshima, @schneiml, @cmsbuild, @jfernan2, @fioriNTU can you please review it and eventually sign? Thanks.
@barvic this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

[andrius-k]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/33932/console Started: 2019/04/03 16:09

[cmsbuild]

+1
Tested at: afeec9d
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26337/33932/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26337/33932/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 32
• DQMHistoTests: Total histograms compared: 3139747
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3139549
• DQMHistoTests: Total skipped: 197
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 31 files compared)
• Checked 133 log files, 14 edm output root files, 32 DQM output files

[andrius-k]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

[fabiocos]

@schneiml these helper scripts are in principle ok to me, but I see that the print management is not python3 compatible. Could you please fix it and try to compile with python3? Otherwise this will be marked as an error in the python3 check in the IB

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26337/9278

• This PR adds an extra 24KB to repository

[cmsbuild]

Pull request #26337 was updated. @andrius-k, @kmaeshima, @schneiml, @cmsbuild, @jfernan2, @fioriNTU can you please check and sign again.

[andrius-k]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/34209/console Started: 2019/04/15 18:53

[cmsbuild]

+1
Tested at: b6a0bf4
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26337/34209/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26337/34209/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 2 differences found in the comparisons
• DQMHistoTests: Total files compared: 32
• DQMHistoTests: Total histograms compared: 3142783
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3142585
• DQMHistoTests: Total skipped: 197
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 31 files compared)
• Checked 133 log files, 14 edm output root files, 32 DQM output files

[fabiocos]

@schneiml python3 has been enabled in the DEVEL IB for testing, otherwise for the compilation youy may use the test done on the baseline IB

https://github.com/cms-sw/cms-bot/blob/master/run-ib-python3#L13

[andrius-k]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

[fabiocos]

+1