Modify tau sequences to not pull payloads from outside GT #44685

mbluj · 2024-04-10T09:14:10Z

PR description:

This PR addresses a longstanding issue with tau modules pulling payloads from outside GT (#43797).
The issue at the AOD and miniAOD steps is solved since CMSSW 12 series, so this PR fixes it at the nanoAOD step and it is believed that it closes the issue with current CMSSW series.
Changes introduced by this PR are purely technical and do not change produced samples.

PR validation:

Validated locally with nano configurations for Run3 and Run2,run2_nanoAOD_106Xv2 eras with and without access to frontier (the latter with GTs dumped to sqlite files), and with (limited set of) matrix tests (runTheMatrix.py -l limited -i all --ibeos).

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

This PR can be easily backported to 14_0 series if needed - please let me know.
Backport to 10_6 will require some effort as it will require also modifications at AOD and miniAOD steps and preparation of a set of new GTs therefore I would like to avoid this.

cmsbuild · 2024-04-10T09:14:42Z

cms-bot internal usage

cmsbuild · 2024-04-10T09:22:10Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44685/39882

This PR adds an extra 44KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File PhysicsTools/NanoAOD/python/boostedTaus_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575
- File PhysicsTools/NanoAOD/python/nano_cff.py modified in PR(s): Add reduced lepton time-life info to NanoAOD #44637, Introduce Unified Particle Transformer AK4 jet tagger #44641
- File PhysicsTools/NanoAOD/python/taus_cff.py modified in PR(s): Explicit lazy parsing in SimpleFlatTableProducer #44575

cmsbuild · 2024-04-10T09:22:35Z

A new Pull Request was created by @mbluj for master.

It involves the following packages:

PhysicsTools/NanoAOD (xpog)
RecoTauTag/Configuration (reconstruction)
RecoTauTag/RecoTau (reconstruction)

@hqucms, @vlimant, @jfernan2, @mandrenguyen, @cmsbuild can you please review it and eventually sign? Thanks.
@azotz, @mbluj, @gpetruc, @missirol, @AnnikaStein this is something you requested to watch as well.
@antoniovilela, @sextonkennedy, @rappoccio you are the release manager for this.

cms-bot commands are listed here

vlimant · 2024-04-10T09:34:01Z

enable nano

vlimant · 2024-04-10T09:36:38Z

please test

vlimant · 2024-04-10T09:36:47Z

type tau

cmsbuild · 2024-04-10T12:34:03Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f1591/38733/summary.html
COMMIT: db9191f
CMSSW: CMSSW_14_1_X_2024-04-09-2300/el8_amd64_gcc12
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44685/38733/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 1 errors in the following unit tests:

---> test runtestPhysicsToolsPatAlgos had ERRORS

Comparison Summary

Summary:

You potentially removed 24 lines from the logs
Reco comparison results: 48 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3316263
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3316237
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: no differences found

NANO Comparison Summary

Summary:

You potentially added 2 lines to the logs
Reco comparison results: 13 differences found in the comparisons
DQMHistoTests: Total files compared: 15
DQMHistoTests: Total histograms compared: 16484
DQMHistoTests: Total failures: 14
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 16470
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 14 files compared)
Checked 55 log files, 32 edm output root files, 15 DQM output files

Nano size comparison Summary:

Sample	kb/ev	ref kb/ev	diff kb/ev	ev/s/thd	ref ev/s/thd	diff rate	mem/thd	ref mem/thd
2500.0	2.574	2.574	0.000 ( +0.0% )	4.58	4.62	-0.7%	2.228	2.172
2500.001	2.721	2.722	-0.000 ( -0.0% )	4.12	4.11	+0.2%	2.637	2.593
2500.002	2.667	2.667	0.000 ( +0.0% )	4.28	4.27	+0.3%	2.643	2.588
2500.01	1.318	1.318	0.000 ( +0.0% )	8.42	8.31	+1.3%	2.354	2.267
2500.011	1.739	1.739	0.000 ( +0.0% )	4.61	4.57	+0.7%	2.538	2.411
2500.012	1.580	1.580	0.000 ( +0.0% )	6.59	6.45	+2.2%	2.450	2.403
2500.1	2.218	2.218	0.000 ( +0.0% )	4.61	4.60	+0.3%	2.064	2.067
2500.2	2.333	2.333	0.000 ( +0.0% )	5.24	5.26	-0.4%	1.981	1.975
2500.21	1.182	1.182	0.000 ( +0.0% )	3.53	3.52	+0.5%	2.273	2.266
2500.211	1.546	1.546	0.000 ( +0.0% )	3.17	3.15	+0.7%	2.357	2.249
2500.3	2.089	2.089	0.000 ( +0.0% )	10.20	10.17	+0.3%	1.969	1.972
2500.301	2.667	2.667	0.000 ( +0.0% )	8.64	8.48	+1.9%	1.947	1.956
2500.31	7.159	7.159	0.000 ( +0.0% )	1.42	1.35	+5.3%	1.704	1.709
2500.311	1.564	1.564	0.000 ( +0.0% )	6.93	6.76	+2.5%	1.055	1.055
2500.312	540.453	540.453	0.000 ( +0.0% )	0.53	0.53	-0.0%	1.595	1.633
2500.313	817.689	817.689	0.000 ( +0.0% )	0.73	0.73	+0.4%	1.586	1.583
2500.32	1.256	1.256	0.000 ( +0.0% )	13.31	12.98	+2.5%	2.364	2.317
2500.321	1.643	1.643	0.000 ( +0.0% )	10.01	9.84	+1.7%	2.408	2.397
2500.322	1.163	1.163	0.000 ( +0.0% )	10.48	10.37	+1.1%	2.139	2.202
2500.323	7.768	7.768	0.000 ( +0.0% )	3.40	3.37	+0.8%	1.838	1.818
2500.324	1.759	1.759	0.000 ( +0.0% )	10.18	10.11	+0.8%	2.179	2.177
2500.325	4.144	4.144	0.000 ( +0.0% )	4.26	4.19	+1.8%	2.216	2.208
2500.326	3.095	3.095	0.000 ( +0.0% )	1.80	1.80	-0.0%	2.049	2.129
2500.327	1.791	1.791	0.000 ( +0.0% )	10.29	10.29	+0.1%	2.323	2.334
2500.4	2.245	2.245	0.000 ( +0.0% )	9.78	9.67	+1.2%	1.846	1.829
2500.401	1.778	1.778	0.000 ( +0.0% )	8.50	8.56	-0.7%	1.812	1.705
2500.402	2.795	2.795	0.000 ( +0.0% )	8.45	8.10	+4.2%	1.907	1.910
2500.403	8.752	8.752	0.000 ( +0.0% )	2.84	2.76	+2.8%	1.939	1.931
2500.404	5.105	5.105	0.000 ( +0.0% )	1.34	1.32	+1.2%	1.786	1.777
2500.405	2.803	2.803	0.000 ( +0.0% )	8.41	8.16	+3.2%	1.910	1.802
2500.5	5.192	5.192	0.000 ( +0.0% )	15.75	15.90	-0.9%	1.506	1.563
2500.51	9.119	9.119	0.000 ( +0.0% )	9.62	9.62	-0.0%	1.505	1.511

hqucms · 2024-04-11T13:46:07Z

+1

Failure in the unittest is also in the plain IB and not related to this PR.

hqucms · 2024-04-11T13:47:23Z

@mbluj Could you please make a backport to 14_0_X? Thanks!

mbluj · 2024-04-11T13:49:29Z

@mbluj Could you please make a backport to 14_0_X? Thanks!

Sure, I'll do it.

mbluj · 2024-04-11T13:53:40Z

@mbluj Could you please make a backport to 14_0_X? Thanks!

Sure, I'll do it.

Just a warning: there are several PRs ongoing which are potentially in conflict with this development because of changes in nano configs. This will require rebasing and conflict solving.

jfernan2 · 2024-04-11T20:08:31Z

+1

cmsbuild · 2024-04-11T20:08:51Z

This pull request is fully signed and it will be integrated in one of the next master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @rappoccio, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

antoniovilela · 2024-04-14T22:35:04Z

ignore tests-rejected with ib-failure

antoniovilela · 2024-04-14T22:36:25Z

+1

mbluj added 3 commits April 8, 2024 13:05

Remove unused cff file to update tauIDs in Nano

b5cb8f6

Test tauID rereco with payloads from GT

6a2ec9d

Define but not regiser payloads

dc3a901

cmsbuild added this to the CMSSW_14_1_X milestone Apr 10, 2024

cmsbuild added reconstruction-pending pending-signatures tests-pending orp-pending code-checks-pending xpog-pending labels Apr 10, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 10, 2024

cmsbuild added tests-started tau and removed tests-pending labels Apr 10, 2024

This was referenced Apr 10, 2024

Add reduced lepton time-life info to NanoAOD #44637

Merged

Introduce Unified Particle Transformer AK4 jet tagger #44641

Merged

cmsbuild added tests-rejected and removed tests-started labels Apr 10, 2024

cmsbuild mentioned this pull request Apr 10, 2024

Add Unified ParT to tau NanoAOD #44691

Merged

Use tauIDs with payloads from GT

d6afdcb

mbluj force-pushed the CMSSW_14_1_X_rmTauPayloadsNoGT branch from db9191f to d6afdcb Compare April 10, 2024 14:38

cmsbuild removed the tests-rejected label Apr 10, 2024

cmsbuild added tests-rejected and removed tests-started labels Apr 10, 2024

cmsbuild added xpog-approved and removed xpog-pending labels Apr 11, 2024

mbluj mentioned this pull request Apr 11, 2024

Modify tau sequences to not pull payloads from outside GT (14_0_X) #44715

Merged

cmsbuild mentioned this pull request Apr 11, 2024

Explicit lazy parsing in SimpleFlatTableProducer #44575

Closed

cmsbuild added reconstruction-approved fully-signed and removed reconstruction-pending pending-signatures labels Apr 11, 2024

cmsbuild mentioned this pull request Apr 12, 2024

Fix ParticleNetLegacy when running Nano on Run2UL samples #44726

Merged

cmsbuild added tests-approved tests-ib-failure and removed tests-rejected labels Apr 14, 2024

cmsbuild added orp-approved and removed orp-pending labels Apr 14, 2024

cmsbuild merged commit 95e9ccc into cms-sw:master Apr 14, 2024
12 of 13 checks passed

This was referenced Apr 15, 2024

Rivet 4.0.0 and YODA 2.0.0 cms-sw/cmsdist#9111

Open

Adaptions for Pythia 8.311 #43786

Merged

[14_1_X] Bumped CepGen version to 1.2.3 cms-sw/cmsdist#9122

Merged

Enable multi-arch build for unit tests cms-sw/cmsdist#9137

Merged

vlimant mentioned this pull request May 13, 2024

Developers overriding production location for retrieving conditions in configurations #27393

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modify tau sequences to not pull payloads from outside GT #44685

Modify tau sequences to not pull payloads from outside GT #44685

mbluj commented Apr 10, 2024

cmsbuild commented Apr 10, 2024 •

edited

cmsbuild commented Apr 10, 2024

cmsbuild commented Apr 10, 2024

vlimant commented Apr 10, 2024

vlimant commented Apr 10, 2024

vlimant commented Apr 10, 2024

cmsbuild commented Apr 10, 2024

hqucms commented Apr 11, 2024

hqucms commented Apr 11, 2024

mbluj commented Apr 11, 2024

mbluj commented Apr 11, 2024

jfernan2 commented Apr 11, 2024

cmsbuild commented Apr 11, 2024

antoniovilela commented Apr 14, 2024 •

edited

antoniovilela commented Apr 14, 2024

Modify tau sequences to not pull payloads from outside GT #44685

Modify tau sequences to not pull payloads from outside GT #44685

Conversation

mbluj commented Apr 10, 2024

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

cmsbuild commented Apr 10, 2024 • edited

cmsbuild commented Apr 10, 2024

cmsbuild commented Apr 10, 2024

vlimant commented Apr 10, 2024

vlimant commented Apr 10, 2024

vlimant commented Apr 10, 2024

cmsbuild commented Apr 10, 2024

Unit Tests

Comparison Summary

NANO Comparison Summary

hqucms commented Apr 11, 2024

hqucms commented Apr 11, 2024

mbluj commented Apr 11, 2024

mbluj commented Apr 11, 2024

jfernan2 commented Apr 11, 2024

cmsbuild commented Apr 11, 2024

antoniovilela commented Apr 14, 2024 • edited

antoniovilela commented Apr 14, 2024

cmsbuild commented Apr 10, 2024 •

edited

antoniovilela commented Apr 14, 2024 •

edited