Adaptation to enable the use of a new MVAIsolation training #21022

roger-wolf · 2017-10-25T20:57:52Z

Dear Slava and Andrea,

this is a PR that enables the use of a new training adapted to the increase of the pT cut required for PFPhotons used for input variables for the tauID MVA, isolation sums and tau reconstruction in general, as included in our last PR (#20900).

All changes are only on the python configuration level. The anticipated changes in performance have been studied and documented here (*). The new training is based on the Summer17 MC samples and will be referred to as 2017v1.

We hope for an integration still into 94X to have the new training included in the upcoming new MC production campaign. Standard tests show no differences beyond those that we expect to see. For more details and confirmation I refer to @mbluj, @steggema and @isobelojalvo

Cheers,
Roger

(*)
https://indico.cern.ch/event/675184/contributions/2766837/attachments/1546406/2427371/mbluj_TauId_GammaPtGt1GeV_24Oct2017.pdf
https://indico.cern.ch/event/675184/contributions/2767859/attachments/1547023/2428479/mbluj_TauId_80XLegacyMiniAOD_24Oct2017.pdf

…VLoose WP

Tau MVAIso 2017v1

cmsbuild · 2017-10-25T20:58:22Z

The code-checks are being triggered in jenkins.

cmsbuild · 2017-10-25T21:00:12Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/PR-21022/1649

cmsbuild · 2017-10-25T21:00:42Z

A new Pull Request was created by @roger-wolf (Roger Wolf) for master.

It involves the following packages:

PhysicsTools/PatAlgos
RecoTauTag/Configuration

@perrotta, @cmsbuild, @monttj, @slava77 can you please review it and eventually sign? Thanks.
@TaiSakuma, @gouskos, @imarches, @ahinzmann, @acaudron, @mmarionncern, @rappoccio, @jdamgov, @jdolen, @nhanvtran, @gpetruc, @gkasieczka, @schoef, @ferencek, @mverzett, @mariadalfonso, @pvmulder, @seemasharmafnal, @JyothsnaKomaragiri this is something you requested to watch as well.
@davidlange6, @slava77 you are the release manager for this.

cms-bot commands are listed here

perrotta · 2017-10-26T05:17:51Z

please test

cmsbuild · 2017-10-26T05:18:06Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/23991/console Started: 2017/10/26 07:19

cmsbuild · 2017-10-26T07:15:52Z

-1

Tested at: 00e4b6b

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
00e4b6b
56e6674
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21022/23991/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21022/23991/git-merge-result

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21022/23991/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests AddOn

Unit Tests:

I found errors in the following unit tests:

---> test runtestPhysicsToolsPatAlgos had ERRORS

AddOn:

I found errors in the following addon tests:

cmsRun /build/cmsbld/jenkins-workarea/workspace/ib-any-integration/CMSSW_9_4_X_2017-10-25-2300/src/PhysicsTools/PatAlgos/test/IntegrationTest_cfg.py : FAILED - time: date Thu Oct 26 08:29:57 2017-date Thu Oct 26 08:29:18 2017 s - exit: 17920

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
00e4b6b
56e6674
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21022/23991/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21022/23991/git-merge-result

cmsbuild · 2017-10-26T07:15:54Z

Comparison job queued.

perrotta · 2017-10-26T07:19:59Z

Errors are related: @roger-wolf please fix

mbluj · 2017-10-26T07:27:22Z

The error is related to a work-flow where MiniAOD is produced from existing AOD, i.e. AOD with is not rerun, which does not contain a new VVLoose WP of MVAIso (added on request of SUSY and EXO). It was checked that it works with the full production chain. How it can be fixed to satisfy for unit tests?

slava77 · 2017-10-26T07:27:46Z

We hope for an integration still into 94X to have the new training included in the upcoming new MC production campaign.

94X is now closed for new features.
If this is required in 94X, POG validators can invalidate 940pre3 and then we can have this update included.
I expect that some time today this PR is auto-advanced to 10X release and we will eventually need a backport.

cmsbuild · 2017-10-30T09:19:45Z

Pull request #21022 was updated. @perrotta, @cmsbuild, @monttj, @slava77 can you please check and sign again.

mbluj · 2017-10-30T09:21:02Z

On Sun, Oct 29, 2017 at 9:21 PM, perrotta ***@***.***> wrote: @mbluj <https://github.com/mbluj> @roger-wolf <https://github.com/roger-wolf> : latest updates seem to correspond to what was asked for.

Yes, thanks.

But there are still errors in the unit tests which were supposed to have been cured with them: please check if anything was forgotten.

Yes, indeed the (temporary) customize to a few tests were overlooked. It has just been fixed and hopefully this time all tests will be successful.

perrotta · 2017-10-30T09:21:25Z

please test

cmsbuild · 2017-10-30T09:22:44Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/24042/console Started: 2017/10/30 10:23

cmsbuild · 2017-10-30T11:38:20Z

+1
Tested at: 07ebdd3
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21022/24042/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
07ebdd3
e099e08
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21022/24042/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21022/24042/git-merge-result

cmsbuild · 2017-10-30T11:38:23Z

Comparison job queued.

cmsbuild · 2017-10-30T12:47:41Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21022/24042/summary.html

There are some workflows for which there are errors in the baseline:
10024.0 step 5
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Comparison Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 15 differences found in the comparisons
DQMHistoTests: Total files compared: 25
DQMHistoTests: Total histograms compared: 2621497
DQMHistoTests: Total failures: 1652
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2619681
DQMHistoTests: Total skipped: 164
DQMHistoTests: Total Missing objects: 0
Checked 102 log files, 10 edm output root files, 25 DQM output files

perrotta · 2017-10-30T13:11:07Z

+1

Training update results in modified shapes for the distributions of the tau discriminator by isolation, and this further propagates in differences in DQM distributions for all tau isolation working points
The results had been validated by the tau POG, as from the plots attached in the main description of this PR
A flag has been added to the PATTauProducer which prevents, if enabled, code crashing in case a given tau discriminator is not present in the event. This is not enabled by default (i.e. the steering configuration must correspond identically to what in the event), but it is in the addOn tests which read on not-yet updated AOD files, thus preventing them from crashing.

perrotta · 2017-10-30T13:14:10Z

@mbluj @roger-wolf : this is submitted to the master, which now corresponds to CMSSW_10_0_X.
If you need it backported to 94X (previous agreemet of the release manager, PC, etc. because of the modified reco content) you must provide a backport PR.

davidlange6 · 2017-11-03T15:44:37Z

merge

davidlange6 · 2017-11-03T15:44:51Z

(but we need to finalize 80x and 94x AOD support)

Dr15Jones · 2017-11-04T17:58:16Z

This appears to have broken the TopQuarkAnalysis unit tests. E.g.
https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc630/CMSSW_10_0_X_2017-11-03-2300/unitTestLogs/TopQuarkAnalysis/Examples

perrotta · 2017-11-06T10:26:29Z

@roger-wolf @mbluj

Unit tests are broken in

TopQuarkAnalysis/ Examples/
TopQuarkAnalysis/TopEventProducers/
TopQuarkAnalysis/TopEventSelection/
(see e.g. https://cmssdt.cern.ch/SDT/cgi-bin/showBuildLogs.py/slc7_amd64_gcc630/www/sun/10.0-sun-23/CMSSW_10_0_X_2017-11-05-2300)
Could you please apply the patch also there, and prepare a new PR with it?

slava77 · 2017-11-17T08:18:17Z

PhysicsTools/PatAlgos/plugins/PATTauProducer.cc

@@ -315,6 +316,11 @@ void PATTauProducer::produce(edm::Event & iEvent, const edm::EventSetup & iSetup
 	  edm::Handle<reco::PFTauDiscriminator> pfTauIdDiscr;
 	  iEvent.getByToken(pfTauIDTokens_[i], pfTauIdDiscr);

+	  if(skipMissingTauID_ && !pfTauIdDiscr.isValid()){
+	    edm::LogWarning("DataSource") << "Tau discriminator '" << tauIDSrcs_[i].first


Can this be made a LogInfo?
The warning shows up for every tau and just pollutes the logs.
A more informative option that will still be printed is to accumulate the number of dropped IDs and print only the first time it happens.

mbluj and others added 5 commits October 25, 2017 17:37

Add new payloads for MVAIso 2017v1

75e3b6a

Update MVAIso discriminants (oldDMs, R=0.5) to 2017v1 training; add V…

ac57c2e

…VLoose WP

Add new VVLoose MVAIso WP to PAT/MiniAOD

3d7569f

bugfix

f409fc9

Merge pull request #61 from mbluj/CMSSW_9_4_X_tauMVAIso2017v1

00e4b6b

Tau MVAIso 2017v1

cmsbuild added this to the CMSSW_9_4_X milestone Oct 25, 2017

cmsbuild added analysis-pending code-checks-pending comparison-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Oct 25, 2017

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 25, 2017

cmsbuild added tests-started and removed tests-pending labels Oct 26, 2017

cmsbuild added tests-rejected and removed tests-started labels Oct 26, 2017

smuzaffar removed this from the CMSSW_9_4_X milestone Oct 26, 2017

cmsbuild added tests-started and removed tests-pending labels Oct 30, 2017

cmsbuild added tests-approved and removed tests-started labels Oct 30, 2017

cmsbuild added comparison-available and removed comparison-pending labels Oct 30, 2017

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Oct 30, 2017

cmsbuild added orp-approved and removed orp-pending labels Nov 3, 2017

cmsbuild merged commit 5fbe472 into cms-sw:master Nov 3, 2017

mbluj mentioned this pull request Nov 6, 2017

Customize to the unit tests that fail due to old input samples cms-tau-pog/cmssw#68

Merged

roger-wolf mentioned this pull request Nov 6, 2017

Temporary fix of TQAF unit tests after TauID update #21186

Merged

slava77 reviewed Nov 17, 2017

View reviewed changes

roger-wolf deleted the CMSSW_9_4_X_tau-pog_MVA2017v1 branch November 17, 2017 11:23

slava77 mentioned this pull request Feb 20, 2018

New trainings for mvaTauID + one backport from CMSSW_10_0_X #21977

Merged

mbluj mentioned this pull request Apr 30, 2019

switch to tau id MVA2017v2 in AOD and MiniAOD #26541

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adaptation to enable the use of a new MVAIsolation training #21022

Adaptation to enable the use of a new MVAIsolation training #21022

roger-wolf commented Oct 25, 2017

cmsbuild commented Oct 25, 2017

cmsbuild commented Oct 25, 2017

cmsbuild commented Oct 25, 2017

perrotta commented Oct 26, 2017

cmsbuild commented Oct 26, 2017 •

edited

cmsbuild commented Oct 26, 2017

cmsbuild commented Oct 26, 2017

perrotta commented Oct 26, 2017

mbluj commented Oct 26, 2017 •

edited

slava77 commented Oct 26, 2017

cmsbuild commented Oct 30, 2017

mbluj commented Oct 30, 2017 via email

perrotta commented Oct 30, 2017

cmsbuild commented Oct 30, 2017 •

edited

cmsbuild commented Oct 30, 2017

cmsbuild commented Oct 30, 2017

cmsbuild commented Oct 30, 2017

perrotta commented Oct 30, 2017

perrotta commented Oct 30, 2017

davidlange6 commented Nov 3, 2017

davidlange6 commented Nov 3, 2017

Dr15Jones commented Nov 4, 2017

perrotta commented Nov 6, 2017

slava77 Nov 17, 2017

Adaptation to enable the use of a new MVAIsolation training #21022

Adaptation to enable the use of a new MVAIsolation training #21022

Conversation

roger-wolf commented Oct 25, 2017

cmsbuild commented Oct 25, 2017

cmsbuild commented Oct 25, 2017

cmsbuild commented Oct 25, 2017

perrotta commented Oct 26, 2017

cmsbuild commented Oct 26, 2017 • edited

cmsbuild commented Oct 26, 2017

cmsbuild commented Oct 26, 2017

perrotta commented Oct 26, 2017

mbluj commented Oct 26, 2017 • edited

slava77 commented Oct 26, 2017

cmsbuild commented Oct 30, 2017

mbluj commented Oct 30, 2017 via email

perrotta commented Oct 30, 2017

cmsbuild commented Oct 30, 2017 • edited

cmsbuild commented Oct 30, 2017

cmsbuild commented Oct 30, 2017

cmsbuild commented Oct 30, 2017

perrotta commented Oct 30, 2017

perrotta commented Oct 30, 2017

davidlange6 commented Nov 3, 2017

davidlange6 commented Nov 3, 2017

Dr15Jones commented Nov 4, 2017

perrotta commented Nov 6, 2017

slava77 Nov 17, 2017

Choose a reason for hiding this comment

cmsbuild commented Oct 26, 2017 •

edited

mbluj commented Oct 26, 2017 •

edited

cmsbuild commented Oct 30, 2017 •

edited