various improvements to HLT-development tools [12_5_X]

[missirol]

backport of #40004

PR description:

From the description of #40004:

This PR contains various improvements to scripts used for HLT development (used both by TSG itself and HLT developers at large).

The main changes are

1. removal of ConfDB queries from the so-called HLT-Validation tests running in IBs (to address hltIntegrationTests tests failing randomly in IBs #37598 and HLT Validation tests fail in all (12_3/4/5/6) IBs since 2022-09-07-1100 IBs #39345);
2. make more HLT-development scripts capable of using the --dbproxy option, to connect to ConfDB from outside CERN through a tunnel connection.

As part of 1., scripts like hltIntegrationTests and hltListPaths are extended to be able to use as input a local cfg file (as an alternative to the standard HLT-menu input, which is the name of a configuration in the ConfDB database).

Other smaller updates are included in the process. More detailed comments will be added as part of the review.

Summary:

./getHLT.sh --dbproxy # will now work
./getFrozenHLT.sh --dbproxy # will now work
./makeSubtables --dbproxy # will now work
./runAll.csh [..] # does not query ConfDB anymore (content of the tests is unchanged)

hltIntegrationTests hlt.py [..] # will now work
hltListPaths hlt.py [..] # will now work

The aim is to add functionalities, and leave current behaviour unchanged (except for tweaks to the printouts of hltIntegrationTests and get{Frozen,}HLT.sh).

Merely technical. No differences expected in the outputs of PR tests (this PR is basically insensitive to PR tests).

PR validation:

None beyond the tests done for #40004.

If this PR is a backport, please specify the original PR and why you need to backport that PR. If this PR will be backported, please specify to which release cycle the backport is meant for:

#40004

Fix for IB tests + improved tools for HLT development.

[cmsbuild]

A new Pull Request was created by @missirol (Marino Missiroli) for CMSSW_12_5_X.

It involves the following packages:

• HLTrigger/Configuration (hlt)

@cmsbuild, @missirol, @Martin-Grunewald can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @silviodonato this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

• Backported from various improvements to HLT-development tools #40004

[missirol]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f3a18f/29062/summary.html
COMMIT: 46a9f81
CMSSW: CMSSW_12_5_X_2022-11-16-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40094/29062/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 51
• DQMHistoTests: Total histograms compared: 3734871
• DQMHistoTests: Total failures: 8
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3734841
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
• Checked 216 log files, 49 edm output root files, 51 DQM output files
• TriggerResults: no differences found

[missirol]

+hlt

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_5_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_6_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

[rappoccio]

Do we have strong motivation to merge this in 12_5 considering that the datataking is done (and presumably, there will be no more trigger development)? Maybe it's best to just leave it in the master branch alone?

[Martin-Grunewald]

Likely HIon will stay and study with 12_5.

[missirol]

@rappoccio

• the main motivation to have this in 12_5_X too is to fix hltIntegrationTests tests failing randomly in IBs #37598 for all cycles where IBs run regularly (and this includes 12_5_X)
• HLT-menu development will still be in 12_5_X for a short bit (fixes to HIon menu post-2022 run)
• HIon might still use this release in the future for their own studies, as mentioned by Martin
• i think it's generally helpful to have these HLT tools aligned across release cycles, when possible

[rappoccio]

+1

[missirol]

thanks @rappoccio