run2 re-miniAOD setup based on UL inputs

[slava77]

• run2_miniAOD_UL process modifier is introduced for UL reminiAOD needs.
  • note that using proc modifiers has been preferable over eras since a while; I took the chance to switch to the new way
  • I expect that one modifier should be enough for all years; a year-specific config modifications should be possible with standard modifiers in combination, e.g. as (ctpps_2016 & run2_miniAOD_UL).toModify
    • if this will not be enough e.g. for 2016 we can add run2_miniAOD_UL16extra, but hopefully this can be avoided
• only minimal updates to miniAOD parameters compared to the defaults are made to have the workflow running
  • rerun container discriminators as in run2_miniAOD_94XFall17 @mbluj @swozniewski please check and confirm if this needs a simple change (a more complex one would better be done by you in a follow up PR)
  • drop trying to add pixelClusterTagInfos as in other run2 reminis because this collection is not available in 106X AODs (added in 11_X)
  • PUPPI re-addition will be needed after a fixed version of PUPPI MET in RECO #29254 is available
  • I did not add the photon/electron correctors, assuming that these will be added by EGM @jainshilpi either in [RecoEgamma/EgammaTools] Updates to EnergyScaleCorrection class for scales and smearings in UL #29526
• tentative data and MC workflows are set up for 4 periods of Run2 (2 for 2016, 2017, and 2018)
  • the data workflow numbering takes the existing "legacy" remini as "10" as a reference (even though 0 is implicit due to the numeric nature) and assigns this UL to be "11": e.g. 136.8311 (= 136.83110) -> 136.83111. This minimizes conflicts with some nanoAOD workflows and uses a common pattern
  • the MC workflow numbering starts from 1325.5 as a base (this corresponds to the old 2016 MC remini) and appends "160" (1325.516) and "161" (1325.5161) for the 2016 pre and post-VFP, respectively; similarly "17" and "18" are used for the 2017 and 2018 UL workflows. This was an attempt to establish a somewhat intuitive pattern and avoid conflicts with existing wfs.
• the input datasets are not available at CERN for most of the cases, a request will need to be made (somehow, for data the cmsDriver is not generating a site=T2_CH_CERN requirement and the workflows can run out of the box).
  • for TTbar_13_reminiaod2017UL_INPUT I couldn't find an AODSIM relval with PU and use a RECO input instead
• 136.88811, 2018D UL remini wf is added to the short matrix

A summary of limited validation for 1325.517 (MC 2017) by comparing the original 10_6_X MINIAOD with remini in CMSSW_11_1_X_2020-05-04-1100 using FWLite: the differences are apparently in recognizable places where changes were introduced since 10_6_X

• PUPPI updates are visible, including bugfixes in the MET uncertainties and corrections
• muon beta variables are filled
• tau ID variables are hard to compare with a plain branch diff due to reshuffling of the IDs introduced in 11_1_X

@chayanit @pgunnell @wajidalikhan please let me know if you can make PhedEx request for the datasets proposed here in the tests or if an alternative can be transferred (I can make an update to the PR then)

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-29756/15191

• This PR adds an extra 80KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): Deprecate convertToUnscheduled() customize function and --runUnscheduled argument of cmsDriver #29630

[cmsbuild]

A new Pull Request was created by @slava77 (Slava Krutelyov) for master.

It involves the following packages:

Configuration/ProcessModifiers
Configuration/PyReleaseValidation
PhysicsTools/PatAlgos

@perrotta, @pgunnell, @chayanit, @wajidalikhan, @kpedro88, @cmsbuild, @silviodonato, @franzoni, @slava77, @santocch, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@rappoccio, @gouskos, @hatakeyamak, @emilbols, @Martin-Grunewald, @peruzzim, @slomeo, @mmarionncern, @JyothsnaKomaragiri, @makortel, @smoortga, @fabiocos, @jdolen, @seemasharmafnal, @ferencek, @jdamgov, @nhanvtran, @gkasieczka, @schoef, @mariadalfonso, @clelange, @riga, @ahinzmann, @mverzett, @gpetruc, @andrzejnovak this is something you requested to watch as well.
@silviodonato, @dpiparo you are the release manager for this.

cms-bot commands are listed here

[slava77]

@cmsbuild please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/6146/console Started: 2020/05/07 06:19

[cmsbuild]

+1
Tested at: eac0dc2
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4d00f1/6146/summary.html
CMSSW: CMSSW_11_1_X_2020-05-06-2300
SCRAM_ARCH: slc7_amd64_gcc820

[cmsbuild]

Comparison job queued.

[silviodonato]

Do you have any comments ?
xpog: @fgolf @peruzzim
pdmv: @chayanit @pgunnell @wajidalikhan
analysis: @santocch

[chayanit]

+1

[mbluj]

This is not necessarily wrong (I think Tau POG might like it), but I wonder if it is what was expected. Regardless, I think that there is a way to read also discriminators in old (pre 11_1_X) format.

Thank you for checking.
If Tau POG will like it, I would then accept this as expected. smile
In the context of 10_6_X, this will be a bit of a waste of CPU, but that's a rather small fraction.

For 11_X, rerunning may be more forward looking so that we do not commit ourselves to support reading the old discriminators in the future.

It is just a confirmation from conveners (@roger-wolf, @rmanzoni) that Tau POG likes it.

[silviodonato]

do you have any comments?
analysis: @santocch
xpog: @fgolf @peruzzim

[peruzzim]

+xpog

yes, for NanoAOD there will be a follow up in a separate pull request, that will come once the updates for MiniAOD are integrated

[silviodonato]

+operations

[silviodonato]

merge
@slava77 @smuzaffar @mrodozov have you copied the missing input files at CERN?
#29756 (comment)

[slava77]

* PUPPI re-addition will be needed after a fixed version of #29254 is available

ah. So, the tests were not redone while the PUPPI PR was merged. This broke the tests in the IB

[ahinzmann]

The relevant lines to change for these workflow are likely these:
https://github.com/cms-sw/cmssw/pull/29254/files#diff-76dede28e7751d7bb6c5c9542de4155cR430
https://github.com/cms-sw/cmssw/pull/29254/files#diff-416ad16d53535660648a4a4bb98636d6R33
https://github.com/cms-sw/cmssw/pull/29254/files#diff-76dede28e7751d7bb6c5c9542de4155cR374
@slava77 if you could take care that would be awesome

[santocch]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.

[chayanit]

@slava77 sorry I just saw your comment about PheDEX. I think it should be fine even the input is not at CERN. We can switch these lines to search at other sites.
https://github.com/slava77/cmssw/blob/eac0dc26d24ad80f85d4b4bf638ce1b931feac0f/Configuration/PyReleaseValidation/python/MatrixUtil.py#L180-L185
However, for Data AOD we probably need to ask computing to lock them at the site. What do you think?