L1T phase-2: Update to histogram (Phase-1 like) jets

[cecilecaillol]

PR description:

Modifications to jet finder to improve the agreement with the corresponding firmware.

Add option for trimmed jets, which improve the 9x9 jet finding performance for nearby/close to merging jets

Add python configuration for 9x9 and trimmed 9x9 jets, in addition to the default 7x7 (perhaps 9x9 should now be made the default configuration)

Add calculation of MET to the jet producer, and add a new producer that calculates HT and MHT from the jets

PR validation:

The presentation at the Correlator meeting was based on this version of the emulator, albeit using CMSSW_10_X_Y. The features have been ported to CMSSW_11_1_7.

I double checked the emulator in CMSSW_11_1_7 still gives the same level of agreement with the HLS code over 10k ttbar+200PU events

The efficiency turn on curves also look the same.

Rebase of cms-l1t-offline#883

if this PR is a backport please specify the original PR and why you need to backport that PR:

Rebase of cms-l1t-offline#883

Before submitting your pull requests, make sure you followed this checklist:

• verify that the PR is really intended for the chosen branch
• verify that changes follow CMS Naming, Coding, And Style Rules
• verify that the PR passes the basic test procedure suggested in the CMSSW PR instructions

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-33606/22429

• This PR adds an extra 64KB to repository

[cmsbuild]

A new Pull Request was created by @cecilecaillol for master.

It involves the following packages:

DataFormats/L1Trigger
L1Trigger/L1CaloTrigger

@cmsbuild, @rekovic, @srimanob, @cecilecaillol, @kpedro88 can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @dinyar, @rovere, @kreczko, @thomreis this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[cecilecaillol]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-bd2a58/14801/summary.html
COMMIT: 607d546
CMSSW: CMSSW_12_0_X_2021-05-03-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/33606/14801/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 7 differences found in the comparisons
• DQMHistoTests: Total files compared: 37
• DQMHistoTests: Total histograms compared: 2662646
• DQMHistoTests: Total failures: 12
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 2662611
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.004 KiB( 36 files compared)
• DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
• Checked 155 log files, 37 edm output root files, 37 DQM output files
• TriggerResults: no differences found

[cecilecaillol]

@EmyrClement @rekovic Can you please check this PR (rebasing of cms-l1t-offline#883) is ok from your side?

[cecilecaillol]

+l1

[EmyrClement]

@cecilecaillol This looks ok from my side, I get the same output before/after the rebase.

[srimanob]

+Upgrade

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[qliphy]

+1

[mrodozov]

Hello there,

after this was merged yesterday 11 unit tests started failing in our python3 IBs, in DEVEL and PY3 IB:

https://cmssdt.cern.ch/SDT/cgi-bin/showBuildLogs.py/slc7_amd64_gcc900/www/tue/12.0.PY3-tue-23/CMSSW_12_0_PY3_X_2021-05-11-2300?utests

https://cmssdt.cern.ch/SDT/cgi-bin/showBuildLogs.py/slc7_amd64_gcc900/www/tue/12.0.DEVEL-tue-23/CMSSW_12_0_DEVEL_X_2021-05-11-2300?utests

I didn't investigated much just enough to see
https://github.com/cms-sw/cmssw/pull/33606/files#diff-75c61e837f10b2ee77ba43a4d660d39177424ff6a69a5930b52badbe8e3f39bdR4

which is what this builds complain about.
@cecilecaillol one can setup this IBs with
scram p CMSSW_12_0_DEVEL_X_2021-05-11-2300
and then add this packages to try fix the missing module :)

[srimanob]

Thanks, @mrodozov for reporting.

Do yu think that we will see the same issue if we do the PR-test in python3? How can we start to do PR-test with python-3 during the transition period? @smuzaffar @davidlange6

[mrodozov]

so one get the latest PY3 or DEVEL release before the change and tests against it
it's not enabled by default but that's why we have the testing by bot in such cases

[mrodozov]

please test

[mrodozov]

abort
i forgot to add a pkg with unit tests failing

[mrodozov]

test parameters:

• release = CMSSW_12_0_PY3_X_2021-05-10-2300
• addpkg = DQMOffline/Configuration

[mrodozov]

please test

[cmsbuild]

-1

Failed Tests: UnitTests RelVals AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-bd2a58/15067/summary.html
COMMIT: 607d546
CMSSW: CMSSW_12_0_PY3_X_2021-05-10-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/33606/15067/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestDQMOfflineConfiguration0 had ERRORS
---> test TestDQMOfflineConfiguration100 had ERRORS
---> test TestDQMOfflineConfiguration50 had ERRORS
---> test TestDQMOfflineConfiguration200 had ERRORS
and more ...

RelVals

• 5.15.1_TTbar+TTbarFS+HARVESTFS/step1_TTbar+TTbarFS+HARVESTFS.log
• 136.731136.731_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2/step2_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2.log
• 135.4135.4_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS/step1_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS.log

Expand to see more relval errors ...

• 136.793
• 136.874
• 136.7611
• 136.8311
• 136.8523
• 4.22
• 136.88811
• 1325.81
• 158.01
• 140.53
• 140.56
• 1000.0
• 1001.0
• 4.53
• 8.0
• 1306.0
• 312.0
• 7.3
• 10042.0
• 1330.0
• 9.0
• 12434.0
• 11634.0
• 25202.0
• 10824.0
• 25.0
• 10224.0
• 10024.0
• 23234.0
• 34634.0
• 28234.0
• 250202.181
• 34834.999

AddOn Tests

• fastsimcmsDriver.py TTbar_8TeV_TuneCUETP8M1_cfi --conditions auto:run1_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot Realistic8TeVCollision : FAILED - time: date Thu May 13 14:31:18 2021-date Thu May 13 14:31:07 2021 s - exit: 256
• fastsim1cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi --conditions auto:run2_mc_l1stage1 --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot NominalCollision2015 --era Run2_25ns : FAILED - time: date Thu May 13 14:31:18 2021-date Thu May 13 14:31:08 2021 s - exit: 256
• fastsim2cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi --conditions auto:run2_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot NominalCollision2015 --era Run2_2016 : FAILED - time: date Thu May 13 14:31:18 2021-date Thu May 13 14:31:09 2021 s - exit: 256

Expand to see more addon errors ...

• hlt_mc_Fake
• hlt_mc_Fake
• hlt_mc_Fake
• hlt_mc_Fake1
• hlt_mc_Fake1
• hlt_mc_Fake1
• hlt_mc_Fake2
• hlt_mc_Fake2
• hlt_mc_Fake2
• hlt_mc_GRun
• hlt_mc_GRun
• hlt_mc_GRun
• hlt_mc_HIon
• hlt_mc_HIon
• hlt_mc_HIon
• hlt_mc_PIon
• hlt_mc_PIon
• hlt_mc_PIon
• hlt_mc_PRef
• hlt_mc_PRef
• hlt_mc_PRef
• hlt_data_Fake1
• hlt_mc_Fake1
• hlt_data_Fake1
• hlt_data_Fake2
• hlt_mc_Fake2
• hlt_data_Fake2
• hlt_data_GRun
• hlt_mc_GRun
• hlt_data_GRun
• hlt_data_HIon
• hlt_mc_HIon
• hlt_data_HIon
• hlt_data_PIon
• hlt_mc_PIon
• hlt_data_PIon
• hlt_data_PRef
• hlt_mc_PRef
• hlt_data_PRef

[mrodozov]

Thanks, @mrodozov for reporting.

Do yu think that we will see the same issue if we do the PR-test in python3? How can we start to do PR-test with python-3 during the transition period? @smuzaffar @davidlange6

to answer how to do it if you are suspecting there might be a problem you can use the
please test for CMSSW_12_0_PY3_X but in this case there was not guarantee you'll catch anything because the unit tests are within the L1Trigger/L1CaloTrigger domain for this PR
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-bd2a58/14801/unitTests/log.txt
- and the failing one are in other packages.

[mrodozov]

voila, the DQM tests failed :)