[106X] Using Puppi v15 in Ntuple generator (re-installation of UHH2 required since newer CMSSW version is needed)

[MatthiesC]

This Pull Request will start automatic compilation, then making + testing of ntuples:

• Don't want to run any of this? (e.g. only XML files) Remove the @: [ci @ skip]

• Only want to compile, but don't make + test ntuples? (e.g. change CommonModules.cxx) Remove the @: [only @ compile]

Please include destination branch in title, e.g. [102X], + short meaningful title

Please include an explanation message if it's something more complex than just XML dataset files:

Updates:

• Using Puppi v15 in Ntuple generator, following instructions from https://twiki.cern.ch/twiki/bin/viewauth/CMS/PUPPI#Recipe_for_106X
  I will attach sanity-check plots in a separate comment to this PR
• This recipe requires CMSSW_10_6_>=17, thus updating the UHH2 install scripts
• This new CMSSW version also requires a further update: ONNXRuntime replaces MXNet (frameworks used for DeepBoosted stuff), see e. g. [10_6_X] ONNXRuntime-based implementation of DeepJet, DeepAK8 and DeepDoubleX cms-sw/cmssw#30123 and [10_6_X] Use ONNXRuntime for ParticleNet and update the training for UL cms-sw/cmssw#31037
  However, this comes with a change of the DeepBoosted outputs, see plot in following comment to this PR (I am not sure why, most likely it is due to a change of the training version)
• Replaced source files for testing of UL17 ntuplewriters (old ones do not exist anymore)

[uhh2-bot]

Starting setup & tests, see https://gitlab.cern.ch/mschrode/UHH2-integration/-/pipelines/2343022

[MatthiesC]

As seen in these plots, the update from v14 to v15 has an effect

[MatthiesC]

As seen in this exemplary plot, the change from MXNet to ONNXRuntime has an impact on the DeepBoosted output shapes, meaning that the change likely also comes with a different training. Does anyone know details?
For this plot here, I explicitly chose AK8 CHS jets in order to reject the hypothesis that the different shapes come from the new Puppi version (indeed, other distributions in CHS jets remain unaffected by the Puppi version change)

[uhh2-bot]

Compilation with PR successful

[uhh2-bot]

Report for PR 1550
Webpages with full plots, timing & size info: https://uhh2-integration-results.web.cern.ch/uhh2-integration-results/UHH2integration/test1550

Test samples defined here: https://gitlab.cern.ch/raggleto/UHH2-integration/blob/test1550/scripts/cmsrun_jobs.py

Ntuple comparison report

name	total # hists	diff mean rms	added collections	added hists	removed collections	removed hists	diff entries	no entries	zero rms	zero value	same
data_UL16postVFP_JetHT	1998	71	0	0	0	0	513	188	336	141	749
data_UL16postVFP_SingleElectron	1998	80	0	0	0	0	512	496	279	119	512
data_UL16postVFP_SingleMu	1998	67	0	0	0	0	512	510	280	117	512
data_UL16preVFP_JetHT	1998	103	0	0	0	0	466	314	325	139	651
data_UL16preVFP_SingleElectron	1998	96	0	0	0	0	466	300	326	141	669
data_UL16preVFP_SingleMu	1998	71	0	0	0	0	511	328	318	131	639
data_UL17_JetHT	1998	100	0	0	0	0	466	76	372	157	827
data_UL17_SingleElectron	1998	226	0	0	0	0	327	510	277	126	532
data_UL17_SingleMu	1998	60	0	0	0	0	513	538	261	118	508
data_UL18_EGamma	1998	195	0	0	0	0	376	440	304	129	554
data_UL18_JetHT	1998	73	0	0	0	0	513	62	370	154	826
data_UL18_SingleMu	1998	64	0	0	0	0	513	482	302	123	514
mc_UL16postVFP_TTbar	1794	93	0	0	0	0	466	87	306	145	697
mc_UL16preVFP_TTbar	1794	60	0	0	0	0	513	87	291	137	706
mc_UL17_TTSemiLeptonic	1794	62	0	0	0	0	513	87	288	138	706
mc_UL18_TTSemiLeptonic	1794	62	0	0	0	0	513	87	289	138	705

• diff entries: Differing number of entries
• diff mean rms: Differing means and/or RMS
• no entries: Both hists has 0 entries
• zero rms: One or both RMSs are 0: stores same value
• zero value: One or both hists have only 0s
• same: Histograms are the same (lowest priority)

Timing report

Sample	Reference event loop real/event [s]	PR event loop real/event [s]	diff
data_UL16postVFP_JetHT	0.702	0.490	-0.212 / -30.17 %
data_UL16postVFP_SingleElectron	0.399	0.317	-0.083 / -20.69 %
data_UL16postVFP_SingleMu	0.409	0.327	-0.082 / -20.02 %
data_UL16preVFP_JetHT	0.515	0.448	-0.066 / -12.92 %
data_UL16preVFP_SingleElectron	0.423	0.355	-0.069 / -16.20 %
data_UL16preVFP_SingleMu	0.362	0.302	-0.060 / -16.62 %
data_UL17_JetHT	0.707	0.508	-0.199 / -28.21 %
data_UL17_SingleElectron	0.442	0.357	-0.084 / -19.13 %
data_UL17_SingleMu	0.445	0.344	-0.101 / -22.60 %
data_UL18_EGamma	0.509	0.398	-0.111 / -21.74 %
data_UL18_JetHT	0.828	0.596	-0.232 / -27.98 %
data_UL18_SingleMu	0.460	0.382	-0.078 / -17.00 %
mc_UL16postVFP_TTbar	0.634	0.487	-0.147 / -23.20 %
mc_UL16preVFP_TTbar	0.626	0.522	-0.104 / -16.61 %
mc_UL17_TTSemiLeptonic	0.699	0.565	-0.134 / -19.19 %
mc_UL18_TTSemiLeptonic	0.694	0.503	-0.191 / -27.55 %

Size report
(kB = kilobytes, period is a decimal point not thousands separator)

Sample	Reference total branch size per event [kB]	PR total branch size per event [kB]	diff
data_UL16postVFP_JetHT	6.880	6.905	0.025 / 0.36 %
data_UL16postVFP_SingleElectron	4.461	4.485	0.024 / 0.53 %
data_UL16postVFP_SingleMu	4.578	4.617	0.039 / 0.85 %
data_UL16preVFP_JetHT	5.225	5.246	0.021 / 0.41 %
data_UL16preVFP_SingleElectron	4.790	4.821	0.032 / 0.66 %
data_UL16preVFP_SingleMu	3.812	3.835	0.023 / 0.60 %
data_UL17_JetHT	6.940	6.968	0.028 / 0.40 %
data_UL17_SingleElectron	4.892	4.907	0.015 / 0.31 %
data_UL17_SingleMu	4.932	4.958	0.027 / 0.55 %
data_UL18_EGamma	6.258	6.290	0.031 / 0.50 %
data_UL18_JetHT	8.834	9.043	0.208 / 2.36 %
data_UL18_SingleMu	5.392	5.430	0.038 / 0.70 %
mc_UL16postVFP_TTbar	7.543	7.580	0.037 / 0.49 %
mc_UL16preVFP_TTbar	7.553	7.600	0.047 / 0.62 %
mc_UL17_TTSemiLeptonic	10.141	10.216	0.074 / 0.73 %
mc_UL18_TTSemiLeptonic	10.078	10.153	0.075 / 0.75 %

[MatthiesC]

https://gitlab.cern.ch/raggleto/UHH2-integration/blob/test1550/scripts/cmsrun_jobs.py

I just noticed that this link is not available anymore (the link is given in the uhh2 bot messages above). Needs a fix

[MatthiesC]

We verbally agreed on not merging this PR.
We want to stick with Puppi v14 for now, reasoning:

• Avoid confusions regarding which ntuples in UHH2 pnfs storage will have v14 or v15, and
• JEC and JER for UL17/18 have been determined with v14 so far.

Test source files for ntuplewriters for UL17 don't exist on pnfs/desy.de/cms/tier2/, but maybe somewhere else, thus do not change them without further checks in that direction.
Last but not least, CMSSW_10_6_17 will not be the final UL branch. Thus, avoid updating it right now

[uhh2-bot]

Starting setup & tests, see https://gitlab.cern.ch/mschrode/UHH2-integration/-/pipelines/2345544

[uhh2-bot]

Compilation with PR successful