give OnlineLuminosityRecord info to HLT's LumiMonitor plugin [12_4_X]

[missirol]

backport of #39859

PR description:

From the description of #39859:

This PR address the 2nd item in the description of #39756, plus a related comment in #39756 (comment).

The LumiMonitor DQM plugin, used inside HLT, is updated to read the values of online instantaneous luminosity and mean PU from the so-called 'online-metadata' digis, when the corresponding info from SCAL is not available (i.e. from Run 3 onwards).

To improve clarity, some of the plot labels, transient variables, and configuration parameters are renamed. I changed "scal lumi/PU" to a more generic "online lumi/PU", but of course feel free to suggest better names.

Minor technical improvements (e.g. const-ness of class data members) were done in the process.

(In case this is backported to 12_4_X, it should not warrant a new ConfDB parsing, since the new configuration parameter of LumiMonitor, i.e. onlineMetaDataDigis, already defaults to the intended collection; at the same time, hltOnlineMetaDataDigis should in principle be included in the DQMHistograms EndPath in the HLT menus.)

cee9009 fixes a couple of pixel-related harvesting outputs which depend on the outputs of the LumiMonitor plugin; please see #39859 (comment) for details.

This is not an exact backport because of 357bb68: the renaming of the parameter LumiMonitor.FolderName (-> .folderName), done in #39859, is not backported to 12_4_X in order to avoid extra complications to HLT online operations (see #39859 (comment)).

PR validation:

None beyond the validation done for #39859.

If this PR is a backport, please specify the original PR and why you need to backport that PR. If this PR will be backported, please specify to which release cycle the backport is meant for:

#39859

Fix for HLT online DQM.

[cmsbuild]

A new Pull Request was created by @missirol (Marino Missiroli) for CMSSW_12_4_X.

It involves the following packages:

• DQM/HLTEvF (dqm, hlt)
• DQM/Integration (dqm)
• DQMOffline/Trigger (dqm)
• HLTrigger/Timer (hlt)

@Martin-Grunewald, @emanueleusai, @ahmad3213, @cmsbuild, @missirol, @jfernan2, @syuvivida, @pmandrik, @micsucmed, @rvenditti can you please review it and eventually sign? Thanks.
@batinkov, @battibass, @silviodonato, @mtosi, @Fedespring, @Martin-Grunewald, @HuguesBrun, @jhgoh, @threus, @fwyzard, @rociovilar, @trocino, @cericeci, @francescobrivio this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

• Backported from give OnlineLuminosityRecord info to HLT's LumiMonitor plugin #39859

[missirol]

please test

[cmsbuild]

Pull request #39897 was updated. @Martin-Grunewald, @emanueleusai, @ahmad3213, @cmsbuild, @missirol, @jfernan2, @syuvivida, @pmandrik, @micsucmed, @rvenditti can you please check and sign again.

[missirol]

please test

This now includes the non-update discussed in #39859 (comment), to avoid the need for changes to the 12_4_X online HLT menus (the description of this PR has been updated accordingly).

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-246579/28612/summary.html
COMMIT: 357bb68
CMSSW: CMSSW_12_4_X_2022-10-28-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39897/28612/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 3682416
• DQMHistoTests: Total failures: 17
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 3682376
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 221.01100000000002 KiB( 49 files compared)
• DQMHistoSizes: changed ( 139.001,... ): 27.582 KiB PixelPhase1/Tracks
• DQMHistoSizes: changed ( 139.001,... ): 0.039 KiB DQM/TimerService
• DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
• Checked 208 log files, 45 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[cmsbuild]

Pull request #39897 was updated. @Martin-Grunewald, @emanueleusai, @ahmad3213, @cmsbuild, @missirol, @jfernan2, @syuvivida, @pmandrik, @micsucmed, @rvenditti can you please check and sign again.

[missirol]

please test

See #39859 (comment).

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-246579/28649/summary.html
COMMIT: cee9009
CMSSW: CMSSW_12_4_X_2022-10-30-0000/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39897/28649/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 3682416
• DQMHistoTests: Total failures: 12
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3682382
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 55.515 KiB( 49 files compared)
• DQMHistoSizes: changed ( 1000.0,... ): 0.039 KiB DQM/TimerService
• DQMHistoSizes: changed ( 138.5,... ): 27.582 KiB PixelPhase1/Tracks
• Checked 208 log files, 45 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[cmsbuild]

Pull request #39897 was updated. @Martin-Grunewald, @emanueleusai, @ahmad3213, @cmsbuild, @missirol, @jfernan2, @syuvivida, @pmandrik, @micsucmed, @rvenditti can you please check and sign again.

[missirol]

please test

@emanueleusai , I rebased this PR on 12_4_10_patch3. Could you please retry to test it?

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-246579/28758/summary.html
COMMIT: 3e39cf0
CMSSW: CMSSW_12_4_X_2022-11-02-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39897/28758/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 3719758
• DQMHistoTests: Total failures: 12
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3719724
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 55.515 KiB( 49 files compared)
• DQMHistoSizes: changed ( 1000.0,... ): 0.039 KiB DQM/TimerService
• DQMHistoSizes: changed ( 138.5,... ): 27.582 KiB PixelPhase1/Tracks
• Checked 208 log files, 45 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[missirol]

+hlt

[syuvivida]

@missirol We saw the same errors if testing the same input file from run 361197 with CMSSW_12_5_1 + PR 39597, 39935. Do you happen to know why? Do we have to build CMSSW_12_5_1 with PR 39844 and as well?

Begin processing the 1st record. Run 361197, Event 5811668, LumiSection 4 on stream 0 at 03-Nov-2022 10:23:43.689 CET
----- Begin Fatal Exception 03-Nov-2022 10:23:44 CET-----------------------
An exception of category 'FileInPathError' occurred while
[0] Processing Event run: 361197 lumi: 4 event: 5811668 stream: 0
[1] Running path 'l1tEmulatorMonitorPath'
[2] Calling method for module L1TMuonEndCapTrackProducer/'valEmtfStage2Digis'
Exception Message:
edm::FileInPath unable to find file L1Trigger/L1TMuon/data/emtf_luts/ph_lut_Run3_2022_data/ph_init_neighbor.txt anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cmsnfsdqmdata/dqmdata/dqm_cmssw/playback_1102_CMSSW_12_5_1_39597_39935/poison:/cmsnfsdqmdata/dqmdata/dqm_cmssw/playback_1102_CMSSW_12_5_1_39597_39935/src:/cmsnfsdqmdataBegin processing the 1st record. Run 361197, Event 5811668, LumiSection 4 on stream 0 at 03-Nov-2022 10:23:43.689 CET
----- Begin Fatal Exception 03-Nov-2022 10:23:44 CET-----------------------
An exception of category 'FileInPathError' occurred while
[0] Processing Event run: 361197 lumi: 4 event: 5811668 stream: 0
[1] Running path 'l1tEmulatorMonitorPath'
[2] Calling method for module L1TMuonEndCapTrackProducer/'valEmtfStage2Digis'
Exception Message:
edm::FileInPath unable to find file L1Trigger/L1TMuon/data/emtf_luts/ph_lut_Run3_2022_data/ph_init_neighbor.txt anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cmsnfsdqmdata/dqmdata/dqm_cmssw/playback_1102_CMSSW_12_5_1_39597_39935/poison:/cmsnfsdqmdata/dqmdata/dqm_cmssw/playback_1102_CMSSW_12_5_1_39597_39935/src:/cmsnfsdqmdata/dqmdata/dqm_cmssw/playback_1102_CMSSW_12_5_1_39597_39935/external/slc7_amd64_gcc10/data:/opt/offline/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_5_1/src:/opt/offline/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_5_1/external/slc7_amd64_gcc10/data
Current directory is: /home/dqmdevlocal/output
----- End Fatal Exception -------------------------------------------------
03-Nov-2022 10:23:44 CET Writing DQM Root file: ./upload/DQM_V0001_L1TEMU_R000512692.root
DQMFileSaver::globalEndRun()
03-Nov-2022 10:23:45 CET Closed file /fff/BU0/ramdisk/run512692/run512692_ls0004_streamDQM_sm-c2a11-43-01.dat/dqmdata/dqm_cmssw/playback_1102_CMSSW_12_5_1_39597_39935/external/slc7_amd64_gcc10/data:/opt/offline/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_5_1/src:/opt/offline/slc7_amd64_gcc10/cms/cmssw/CMSSW_12_5_1/external/slc7_amd64_gcc10/data
Current directory is: /home/dqmdevlocal/output
----- End Fatal Exception -------------------------------------------------
03-Nov-2022 10:23:44 CET Writing DQM Root file: ./upload/DQM_V0001_L1TEMU_R000512692.root
DQMFileSaver::globalEndRun()
03-Nov-2022 10:23:45 CET Closed file /fff/BU0/ramdisk/run512692/run512692_ls0004_streamDQM_sm-c2a11-43-01.dat

[missirol]

I don't know, but I can give you my guess.

The branch of PR-39935 includes PR-39844; if you are merging PR-39935 via merge-topic on top of 12_5_1, you are effectively also merging PR-39844 in your local build. The issue is that PR-39844 also requires a cms-data update which did not go into 12_5_1, so you end up with an inconsistent setup (you could try to include this cms-data update in your local area; in general, you could try using git cms-cherry-pick-pr, or requiring that PRs that need to be 'manually' merged in the online DQM be based on a CMSSW release, and not an IB).

That said, this DQM issue in 12_5_X appears entirely unrelated to this PR..

[syuvivida]

Test at p5 was successful

[syuvivida]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_4_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_6_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

[missirol]

@syuvivida , thanks for doing the test.

Could you please also review the 12_5_X (#39896) and 12_6_X (#39859) versions of this PR?

[perrotta]

+1

• "Almost" verbatim backport of the corresponding PR in the master, see give OnlineLuminosityRecord info to HLT's LumiMonitor plugin #39859 (comment): this was evaluated by HLT the safest way to propagate the changed parameters, and DQM approved