DQMIO lumisection extraction script

[schneiml]

PR description:

This PR introduces a new tool to extract the histograms from DQMIO data (including per-lumi saved MEs), and repacks them into a format acceptable by DQMGUI.

There are various limitations on DQMGUI side regarding this data, and the navigation between lumisections is achieved by swapping run and ls numbers. Such files should never be uploaded to official DQMGUI instances, and are only to view specific data using a local GUI.

PR validation:

Introduces a new tool, no changes expected.

Instructions to run:

# 0. Connect to lxplus with a tunnel.
firefox http://localhost:8070/dqm/online-dev/ &
ssh -L 8070:localhost:8070 $USER@lxplus7.cern.ch

# 1. Setup a local DQMGUI
/bin/bash
export PATH=$PATH:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin
mkdir -p /tmp/$USER/testGui
cd /tmp/$USER/testGui
git clone git://github.com/dmwm/deployment.git
$PWD/deployment/Deploy -A slc7_amd64_gcc630 -r "comp=comp" \
  -R comp@HG1908e -t MYDEV -s "prep sw post" $PWD dqmgui/bare
source current/apps/dqmgui/128/etc/profile.d/env.sh
$PWD/current/config/dqmgui/manage -f online \
  start "I did read documentation"
exit

# 2. Setup the tool (no longer needed once this PR is merged)
cmsrel CMSSW_11_0_0_pre5
cd CMSSW_11_0_0_pre5/src/
cmsenv
git cms-merge-topic schneiml:dqm-harvestlumi-script
scram b -j8

# 3. Run the tool
voms-proxy-init --rfc --voms cms
dqmiolumiharvest.py -u http://localhost:8070/dqm/online-dev/ \
  --perlumionly -j10 \
  -m "JetMET/MET/*/*/METSig" \
  -r 301393 \
  /JetHT/Run2017C-25Jul2019_UL2017-v2/DQMIO

# 4. Wait for the GUI to index and enjoy your per-lumi plots!
# this can take a while, much longer than collecting the data...

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-27903/11707

• This PR adds an extra 24KB to repository

[cmsbuild]

A new Pull Request was created by @schneiml (Marcel Schneider) for master.

It involves the following packages:

DQMServices/Components

@andrius-k, @kmaeshima, @schneiml, @cmsbuild, @jfernan2, @fioriNTU can you please review it and eventually sign? Thanks.
@barvic this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

[jfernan2]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/2290/console Started: 2019/08/31 18:58

[cmsbuild]

+1
Tested at: fcae2f2
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5eb6ba/2290/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5eb6ba/2290/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 2 differences found in the comparisons
• DQMHistoTests: Total files compared: 34
• DQMHistoTests: Total histograms compared: 2955700
• DQMHistoTests: Total failures: 2
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2955357
• DQMHistoTests: Total skipped: 341
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
• Checked 145 log files, 15 edm output root files, 34 DQM output files

[fabiocos]

@jfernan2 @andrius-k is there any issue with this PR? What is the plan for it?

[jfernan2]

+1
@fabiocos
The script is working with test data, we were waiting for official perLS from 2017UL to make a larger test but I believe it can go in this round so far

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

[fabiocos]

@jfernan2 @schneiml up to you to say, in my understanding it is a tool to be used by those who know what to do with it. But if you prefer to keep it on hold to better test it fine for me, I understood it was already proved to work

[fioriNTU]

@fabiocos your understanding is correct, the tool is tested and it is working as expected. It has recently be tested also in UL2017 ZeroBias successfully. So, please, go ahead with this PR, at the end it doesn't touch any production workflow.

[fabiocos]

+1