various improvements to HLT-development tools

[missirol]

PR description:

This PR contains various improvements to scripts used for HLT development (used both by TSG itself and HLT developers at large).

The main changes are

1. removal of ConfDB queries from the so-called HLT-Validation tests running in IBs (to address hltIntegrationTests tests failing randomly in IBs #37598 and HLT Validation tests fail in all (12_3/4/5/6) IBs since 2022-09-07-1100 IBs #39345);
2. make more HLT-development scripts capable of using the --dbproxy option, to connect to ConfDB from outside CERN through a tunnel connection.

As part of 1., scripts like hltIntegrationTests and hltListPaths are extended to be able to use as input a local cfg file (as an alternative to the standard HLT-menu input, which is the name of a configuration in the ConfDB database).

Other smaller updates are included in the process. More detailed comments will be added as part of the review.

Summary:

./getHLT.sh --dbproxy # will now work
./getFrozenHLT.sh --dbproxy # will now work
./makeSubtables --dbproxy # will now work
./runAll.csh [..] # does not query ConfDB anymore (content of the tests is unchanged)

hltIntegrationTests hlt.py [..] # will now work
hltListPaths hlt.py [..] # will now work

The aim is to add functionalities, and leave current behaviour unchanged (except for tweaks to the printouts of hltIntegrationTests and get{Frozen,}HLT.sh).

Merely technical. No differences expected in the outputs of PR tests (this PR is basically insensitive to PR tests).

PR validation:

Private tests.

If this PR is a backport, please specify the original PR and why you need to backport that PR. If this PR will be backported, please specify to which release cycle the backport is meant for:

CMSSW_12_4_X
CMSSW_12_5_X

(Backports are not necessary, but they could be useful to keep these tools aligned across release cycles.)

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40004/32942

• This PR adds an extra 36KB to repository

[cmsbuild]

A new Pull Request was created by @missirol (Marino Missiroli) for master.

It involves the following packages:

• HLTrigger/Configuration (hlt)

@cmsbuild, @missirol, @Martin-Grunewald can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @silviodonato this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

[Martin-Grunewald]

Given getHLT.sh is changed, I would expect a change also in getFrozenHLT.sh.

[missirol]

Given getHLT.sh is changed, I would expect a change also in getFrozenHLT.sh.

Indeed, I need to add this. Thanks for the catch!

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40004/32971

• This PR adds an extra 16KB to repository

[cmsbuild]

Pull request #40004 was updated. @cmsbuild, @missirol, @Martin-Grunewald can you please check and sign again.

[missirol]

The latest version includes the update of getFrozenHLT.sh (thanks for spotting this, Martin).

I still have a couple of small improvements in mind, and more testing to do. The plan would be to integrate this PR and its (future) backports next week.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40004/33029

• This PR adds an extra 20KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File HLTrigger/Configuration/scripts/hltGetConfiguration modified in PR(s): HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40034, HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40054
  • File HLTrigger/Configuration/scripts/hltIntegrationTests modified in PR(s): HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40034, HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40054
  • File HLTrigger/Configuration/test/cmsDriver.csh modified in PR(s): HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40034, HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40054
  • File HLTrigger/Configuration/test/getEventContent.py modified in PR(s): HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40034, HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40054
  • File HLTrigger/Configuration/test/getFrozenHLT.sh modified in PR(s): HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40034, HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40054
  • File HLTrigger/Configuration/test/getHLT.sh modified in PR(s): HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40034, HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40054
  • File HLTrigger/Configuration/test/runIntegration.csh modified in PR(s): HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40034, HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40054
  • File HLTrigger/Configuration/test/runOne.csh modified in PR(s): HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40034, HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40054

[cmsbuild]

Pull request #40004 was updated. @cmsbuild, @missirol, @Martin-Grunewald can you please check and sign again.

[missirol]

please test

The latest push updated get{Frozen,}HLT.sh such that they can be executed from any directory, not just test/. I don't have other updates planned for this PR (and more testing didn't reveal issues). I'd backport this down to 12_4_X, mainly because those are the cycles with frequent IBs, so most affected by #37598.

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6d5fdb/29020/summary.html
COMMIT: 72347fe
CMSSW: CMSSW_12_6_X_2022-11-15-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40004/29020/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 3 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3417074
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3417046
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 206 log files, 48 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[missirol]

+hlt

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

[rappoccio]

+1