switch to tau id MVA2017v2 in AOD and MiniAOD #26541

swozniewski · 2019-04-25T19:48:17Z

PR description:

Switch from MVA2017v1 to MVA2017v2 tau ID in AOD and MiniAOD following the recommendation in https://indico.cern.ch/event/810741/contributions/3384093/attachments/1827349/2991114/TauID_CMSweek_10042019.pdf#page=3

PR validation:

Matrix tests passed.
Output of raw value has been compared to corresponding NanoAOD sample where MVA2017v2 is already available.

According differences in raw and WP values of IsolationMVArun2v1DBoldDMwLT are expected.

cmsbuild · 2019-04-25T19:48:52Z

The code-checks are being triggered in jenkins.

cmsbuild · 2019-04-25T19:55:22Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26541/9428

This PR adds an extra 16KB to repository

cmsbuild · 2019-04-25T19:55:46Z

A new Pull Request was created by @swozniewski for master.

It involves the following packages:

RecoTauTag/Configuration

@perrotta, @cmsbuild, @slava77 can you please review it and eventually sign? Thanks.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

slava77 · 2019-04-25T20:11:20Z

@cmsbuild please test

cmsbuild · 2019-04-25T20:11:51Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/34351/console Started: 2019/04/25 22:13

slava77 · 2019-04-25T20:11:58Z

@swozniewski
please add a link to some slides shown in a TAU POG (sub)group meeting to the PR description to document the changes.

cmsbuild · 2019-04-25T21:53:39Z

+1
Tested at: 88bd96b
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26541/34351/summary.html

cmsbuild · 2019-04-25T21:53:44Z

Comparison job queued.

cmsbuild · 2019-04-26T00:01:09Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26541/34351/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 187 differences found in the comparisons
DQMHistoTests: Total files compared: 33
DQMHistoTests: Total histograms compared: 3211964
DQMHistoTests: Total failures: 1607
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3210153
DQMHistoTests: Total skipped: 204
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 32 files compared)
Checked 137 log files, 14 edm output root files, 33 DQM output files

steggema · 2019-04-26T08:02:29Z

@swozniewski
please add a link to some slides shown in a TAU POG (sub)group meeting to the PR description to document the changes.

@slava77
We have recommended the 2017v2 discriminator for a while now. The most recent set of slides on the recommendations is here: https://indico.cern.ch/event/810741/contributions/3384093/attachments/1827349/2991114/TauID_CMSweek_10042019.pdf#page=3

The main purpose of this PR is to have this discriminator also available by default in all MiniAOD to have the most useful baseline for all ultra-legacy analyses and possible new trainings that will be performed with the ultra-legacy samples. @rmanzoni @roger-wolf

swozniewski · 2019-04-26T08:31:49Z

thank you @steggema ! I've added the link to the PR description.

slava77 · 2019-04-29T20:39:36Z

RecoTauTag/Configuration/python/HPSPFTaus_cff.py

@@ -526,7 +526,7 @@
    PFTauProducer = cms.InputTag('hpsPFTauProducer'),
    Prediscriminants = requireDecayMode.clone(),
    loadMVAfromDB = cms.bool(True),
-    mvaName = cms.string("RecoTauTag_tauIdMVAIsoDBoldDMwLT2017v1"),
+    mvaName = cms.string("RecoTauTag_tauIdMVAIsoDBoldDMwLT2017v2"),


what is the rationale to keep the same name of the producer with "v1"?
Should it be changed as well to v2?

I had the impression that this version number is not really used because the year does not even show up, so v1 would not stand for 2017v1. But the naming history goes beyond my scope. @steggema @rmanzoni @mbluj could you comment?

@mbluj may know better, but we have two things to consider,

a) the list of input variables, which I think can be considered to be represented by the tag "MVArun2v1DBoldDMwLT" and then resolved to "mvaOpt = cms.string("DBoldDMwLTwGJ")"
b) the version of the training, which is 2017v2

So I think it may be fine to keep the version as Sebastian has it. OTOH, maybe a case can be made for the training tag (2017v2) representing everything, but that may need to lead to even larger overall changes...

I confirm what @steggema wrote. Name of module in the python configuration represents version of MVA, i.e. list of inputs, while name of payloads stands for a given training of the MVA.
From practical perspective, it is also useful to keep names of modules and change only names of payloads as it reduces number of modifications (in terms of no. of modified lines and files).
BTW, when set of produces has been switched from original training with 2015 MC (w/o specified version in names of payload) to training with early 2017 MC (2017v1) names of producers were kept unchanged.

Looking back to CMSSW_7_6_2, when this naming pattern was introduced, the previous payloads had the version of the MVA in the name (or at least it seemed like it).
Please share a pointer to a past example of a different case of ID where the payload does not contain the "vX" of the MVA in its name, just to be clear of a somewhat established practice.

In this PR #21022 the 2017v1

yet, it was still a "v1".
I just wanted to understand if this "vX" is becoming more formalized just now.
Perhaps for consistency of future changes you can make a note in some tau POG area to have that as a reference.

I see what you mean I agree that some version bookkeeping will be useful for final users who usually know name of a producer or/and access name within PAT (and a release version), but not a name of a related payload. It is up to conveners to decide if it makes sense to change name of producer or it will be enough to provide a table in Tau POG recommendation TWiki.

OK, I can wait until Thursday to signoff on this PR (considering the holiday tomorrow).
I hope that by then the decision to keep the current naming or to change it can be made.

@steggema and I agreed to keep the current naming.

The reason is not to force analysts to adjust their code now while we expect to move to recommend a newer DNN-based Tau ID sometime soon, which will demand code changes.

With that, we’ll have a chance to start right away with a more coherent version naming.

OK, I will sign off, but some better formalized naming would be quite helpful.
Without being aware of this discussion and digging into the git history, it would appear more and more that the configuration may be buggy.

mbluj · 2019-04-30T10:48:22Z

@swozniewski I has been not able to check the PR earlier, but it looks that DBnewDMwLT and DBdR03oldDMwLT are not migrated to the 2017v2 training (whose are not present for 2017v1). Could you confirm that this migration is considered, please? And sorry for the late hint.

swozniewski · 2019-04-30T11:19:49Z

@mbluj I was not aware that there again is a training in 2017v2 in contrast to 2017v1 for these two IDs. I will add this now and create a PR to cms-tau-pog:CMSSW_10_6_X_tau-pog_MVA2017v2, where you can check it. I assume that the naming scheme is the same.

cmsbuild · 2019-04-30T15:12:16Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/34426/console Started: 2019/04/30 17:12

cmsbuild · 2019-04-30T15:12:18Z

Pull request #26541 was updated. @perrotta, @cmsbuild, @slava77 can you please check and sign again.

swozniewski · 2019-04-30T15:12:55Z

The second commit addresses Michals comment that 2017v2 is also available for DBnewDMwLT, DBdR03oldDMwLT and updates them. The raw output has been compared to the existing values in NanoAOD on some test events.

cmsbuild · 2019-04-30T16:51:27Z

+1
Tested at: b11ce72
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26541/34426/summary.html

cmsbuild · 2019-04-30T16:51:33Z

Comparison job queued.

cmsbuild · 2019-04-30T19:05:00Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26541/34426/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 590 differences found in the comparisons
DQMHistoTests: Total files compared: 33
DQMHistoTests: Total histograms compared: 3211964
DQMHistoTests: Total failures: 3531
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3208229
DQMHistoTests: Total skipped: 204
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 32 files compared)
Checked 137 log files, 14 edm output root files, 33 DQM output files

slava77 · 2019-05-02T17:51:01Z

+1

for #26541 b11ce72

code changes are in line with the PR description and the follow up review
jenkins tests pass and comparisons with the baseline show changes only in tau MVA and ID variables
- changes start from reco workflows in hpsPFTauDiscriminationByIsolationMVArun2v1DBoldDMwLTraw, hpsPFTauDiscriminationByIsolationMVArun2v1DBdR03oldDMwLTraw, and hpsPFTauDiscriminationByIsolationMVArun2v1DBnewDMwLTraw raw discriminant values
- the differences then propagate to the tau IDs, including the miniAOD and nanoAOD outputs in cases where the corresponding tau "reco" MVA tag modules are running in the tested workflow.

The performance is changing somewhat moderately, e.g. in 250202.181 (ttbar with PU 2018 pmx setup)

cmsbuild · 2019-05-02T17:51:35Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

fabiocos · 2019-05-03T08:06:06Z

@peruzzim @fgolf FYI

fabiocos · 2019-05-03T08:45:40Z

+1

switch to MVA2017v2 in HPSPFTaus_cff

88bd96b

cmsbuild added this to the CMSSW_10_6_X milestone Apr 25, 2019

cmsbuild added code-checks-pending comparison-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Apr 25, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 25, 2019

cmsbuild added tests-started and removed tests-pending labels Apr 25, 2019

cmsbuild added tests-approved and removed tests-started labels Apr 25, 2019

cmsbuild added comparison-available and removed comparison-pending labels Apr 26, 2019

slava77 reviewed Apr 29, 2019

View reviewed changes

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 30, 2019

cmsbuild added tests-started and removed tests-pending labels Apr 30, 2019

cmsbuild added tests-approved and removed tests-started labels Apr 30, 2019

cmsbuild added comparison-available and removed comparison-pending labels Apr 30, 2019

cmsbuild added fully-signed and removed pending-signatures reconstruction-pending labels May 2, 2019

cmsbuild added the reconstruction-approved label May 2, 2019

cmsbuild added orp-approved and removed orp-pending labels May 3, 2019

cmsbuild merged commit 2413cca into cms-sw:master May 3, 2019

This was referenced May 17, 2019

Followup of switch to TauId MVA2017v2 #26824

Merged

Followup of switch to TauId MVA2017v2 for UL #26859

Merged

mbluj deleted the CMSSW_10_6_X_tau-pog_MVA2017v2 branch October 10, 2023 10:15

mbluj restored the CMSSW_10_6_X_tau-pog_MVA2017v2 branch October 10, 2023 10:16

mbluj deleted the CMSSW_10_6_X_tau-pog_MVA2017v2 branch October 10, 2023 10:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

switch to tau id MVA2017v2 in AOD and MiniAOD #26541

switch to tau id MVA2017v2 in AOD and MiniAOD #26541

swozniewski commented Apr 25, 2019 •

edited

cmsbuild commented Apr 25, 2019

cmsbuild commented Apr 25, 2019

cmsbuild commented Apr 25, 2019

slava77 commented Apr 25, 2019

cmsbuild commented Apr 25, 2019 •

edited

slava77 commented Apr 25, 2019

cmsbuild commented Apr 25, 2019

cmsbuild commented Apr 25, 2019

cmsbuild commented Apr 26, 2019

steggema commented Apr 26, 2019

swozniewski commented Apr 26, 2019

slava77 Apr 29, 2019

swozniewski Apr 30, 2019

steggema Apr 30, 2019

mbluj Apr 30, 2019 •

edited

slava77 Apr 30, 2019

slava77 Apr 30, 2019

mbluj Apr 30, 2019

slava77 Apr 30, 2019

rmanzoni May 2, 2019

slava77 May 2, 2019

mbluj commented Apr 30, 2019

swozniewski commented Apr 30, 2019

cmsbuild commented Apr 30, 2019 •

edited

cmsbuild commented Apr 30, 2019

swozniewski commented Apr 30, 2019

cmsbuild commented Apr 30, 2019

cmsbuild commented Apr 30, 2019

cmsbuild commented Apr 30, 2019

slava77 commented May 2, 2019

cmsbuild commented May 2, 2019

fabiocos commented May 3, 2019

fabiocos commented May 3, 2019

switch to tau id MVA2017v2 in AOD and MiniAOD #26541

switch to tau id MVA2017v2 in AOD and MiniAOD #26541

Conversation

swozniewski commented Apr 25, 2019 • edited

PR description:

PR validation:

cmsbuild commented Apr 25, 2019

cmsbuild commented Apr 25, 2019

cmsbuild commented Apr 25, 2019

slava77 commented Apr 25, 2019

cmsbuild commented Apr 25, 2019 • edited

slava77 commented Apr 25, 2019

cmsbuild commented Apr 25, 2019

cmsbuild commented Apr 25, 2019

cmsbuild commented Apr 26, 2019

steggema commented Apr 26, 2019

swozniewski commented Apr 26, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbluj Apr 30, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mbluj commented Apr 30, 2019

swozniewski commented Apr 30, 2019

cmsbuild commented Apr 30, 2019 • edited

cmsbuild commented Apr 30, 2019

swozniewski commented Apr 30, 2019

cmsbuild commented Apr 30, 2019

cmsbuild commented Apr 30, 2019

cmsbuild commented Apr 30, 2019

slava77 commented May 2, 2019

cmsbuild commented May 2, 2019

fabiocos commented May 3, 2019

fabiocos commented May 3, 2019

swozniewski commented Apr 25, 2019 •

edited

cmsbuild commented Apr 25, 2019 •

edited

mbluj Apr 30, 2019 •

edited

cmsbuild commented Apr 30, 2019 •

edited