Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding new Energy Correlation Functions from fastjet-contrib 1.026 #19601

Merged
merged 15 commits into from Sep 23, 2017

Conversation

rappoccio
Copy link
Contributor

The new energy correlation functions are the next generation of the n-subjettiness tau variables. These have similar performance, but are much better behaved as a function of jet pt and mass. However, we prefer to use the GROOMED observables, so this comes with two necessary code changes:

  1. applySubstructure in MiniAOD: Need to move the AK8 PF jets with puppi and soft drop to the top of the file. We then run ECFs on the AK8 Puppi soft-dropped collection, and store as user floats. Then, we need to merge the soft-dropped version with the ungroomed version for final storage.

  2. JetSubstructurePacker: To accomplish the merging of soft-dropped ECFs with the ungroomed version, we also need to "forward" the userFloats from the pat::Jets that are soft-dropped into the ungroomed collection.

The rest are the updates of the ECFs themselves. Once this is integrated, we will prepare a backport to 9.2.x to take advantage of upcoming MINIAOD reproductions.

@cmsbuild
Copy link
Contributor

cmsbuild commented Jul 6, 2017

A new Pull Request was created by @rappoccio for master.

It involves the following packages:

PhysicsTools/PatAlgos
PhysicsTools/PatUtils
RecoJets/JetProducers

@perrotta, @cmsbuild, @slava77, @monttj, @davidlange6 can you please review it and eventually sign? Thanks.
@TaiSakuma, @gouskos, @imarches, @ahinzmann, @acaudron, @gpetruc, @mmarionncern, @jdolen, @nhanvtran, @JyothsnaKomaragiri, @gkasieczka, @yslai, @schoef, @ferencek, @mverzett, @mariadalfonso, @pvmulder, @seemasharmafnal this is something you requested to watch as well.
@davidlange6 you are the release manager for this.

cms-bot commands are listed here

@slava77
Copy link
Contributor

slava77 commented Jul 6, 2017

@cmsbuild please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Jul 6, 2017

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/21236/console Started: 2017/07/07 03:30

@cmsbuild
Copy link
Contributor

cmsbuild commented Jul 7, 2017

-1

Tested at: 86374a1

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
4761895
f36f752
30de1f6
98f552c
5a77c2d
5cea7fe
2dfc973
a906315
fdaea21
6625d2d
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19601/21236/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19601/21236/git-merge-result

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19601/21236/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests

  • Unit Tests:

I found errors in the following unit tests:

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
4761895
f36f752
30de1f6
98f552c
5a77c2d
5cea7fe
2dfc973
a906315
fdaea21
6625d2d
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19601/21236/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19601/21236/git-merge-result

@cmsbuild
Copy link
Contributor

cmsbuild commented Jul 7, 2017

Comparison job queued.

@perrotta
Copy link
Contributor

perrotta commented Jul 7, 2017

please test
(The failing unit test seems unrelated: let try if now less additional commit are tested together)

@cmsbuild
Copy link
Contributor

cmsbuild commented Jul 7, 2017

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/21251/console Started: 2017/07/07 13:23

@cmsbuild
Copy link
Contributor

cmsbuild commented Jul 7, 2017

@cmsbuild
Copy link
Contributor

cmsbuild commented Jul 7, 2017

Comparison job queued.

@cmsbuild
Copy link
Contributor

cmsbuild commented Jul 7, 2017

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19601/21251/summary.html

Comparison Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 23
  • DQMHistoTests: Total histograms compared: 1790682
  • DQMHistoTests: Total failures: 14569
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 1775947
  • DQMHistoTests: Total skipped: 166
  • DQMHistoTests: Total Missing objects: 0
  • Checked 93 log files, 14 edm output root files, 23 DQM output files

produces<edm::ValueMap<float> >(ecfN_str.str().c_str());
routine_.push_back(std::auto_ptr<fastjet::contrib::EnergyCorrelator> ( new fastjet::contrib::EnergyCorrelator( *n, beta_, fastjet::contrib::EnergyCorrelator::pt_R ) ));
}
if ( iConfig.exists("alpha") ) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"alpha" always exist, since you are now using fillDescription()
If you want to treat separately old configs which did not have alpha explicitely set, you must look for a different logic

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh, I didn't realize that. Thanks. Updated now.

src = cms.InputTag("ak8CHSJets"),
ecftype = cms.string("C"),
Njets = cms.vuint32(1, 2, 3),
beta = cms.double(2.0),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"alpha" is still defined by fillDescriptions(), and it will be set as "1" whatever the value of beta.
If you want a different value you must explicitely define it here

The same in the other configs below

@cmsbuild
Copy link
Contributor

The code-checks are being triggered in jenkins.

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/PR-19601/824

@cmsbuild
Copy link
Contributor

Pull request #19601 was updated. @perrotta, @cmsbuild, @monttj, @slava77 can you please check and sign again.

@rappoccio
Copy link
Contributor Author

Fixed the indents now.

@perrotta
Copy link
Contributor

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Sep 19, 2017

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/23087/console Started: 2017/09/19 20:05

@cmsbuild
Copy link
Contributor

@cmsbuild
Copy link
Contributor

Comparison job queued.

@cmsbuild
Copy link
Contributor

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19601/23087/summary.html

Comparison Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 26
  • DQMHistoTests: Total histograms compared: 2649068
  • DQMHistoTests: Total failures: 382
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 2648497
  • DQMHistoTests: Total skipped: 189
  • DQMHistoTests: Total Missing objects: 0
  • Checked 107 log files, 14 edm output root files, 26 DQM output files

@perrotta
Copy link
Contributor

Trying to summarize what changed with the latest commits, since last review (and reco signature) the following updates were applied:

  • Added a pt cut of 250 geV for the 3-particle correlator, which is supposed to be still good for physics, see Optimize new Energy Correlation Functions from fastjet #20558 (comment)
  • Added protection for cases when the nr of constituents are lower than the correlation function power (potentially ending up with nan's)
  • Implemented the suggestion from David to remove the check for userfloat overlaps (done for every jet), which looks correct as it is quite likely not needed to check so often if duplicated algos were given in the configs

Did I miss anything?

The timing measured with the new modules does not seem significantly improved (but I should probably test it in a machine emptier that now cmsdev02 to write here a more quantitative statement)

@rappoccio
Copy link
Contributor Author

I am seeing a factor of ~2 in workflow 10024. Somehow with workflow 10224 on 10 events I saw an even larger improvement, which didn't necessarily make a lot of sense to me.

@rappoccio
Copy link
Contributor Author

So are we okay with this? There is one more request from SMP to add, and I don't want the features to collide ;)

@perrotta
Copy link
Contributor

After some debug I verified that the cut on the third jet pt can save some cpu but not that much, in accordance with the observations. In fact, the selector cut acts on ak8 jets (which are by the way rather energetic in the 10224 wf output), but not on the subjects, which give the most of the combinations.

In any case the observation is that the cut works, although not being as effective as hoped in reducing the cpu time.

I can therefor confirm my previous signature, being fine with the latest adjustments introduced since then,

@rappoccio
Copy link
Contributor Author

Thanks @perrotta, I will put this into the final request from SMP.

@perrotta
Copy link
Contributor

+1

@davidlange6
Copy link
Contributor

merge

@cmsbuild cmsbuild merged commit 1315aa1 into cms-sw:master Sep 23, 2017
@rappoccio rappoccio deleted the ECFMiniAOD branch February 6, 2018 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants