introduce MINIGEN step and output to replace GenOnly workflows #19853

perrozzi · 2017-07-21T09:36:52Z

created a new eventcontent
--eventcontent MINIAODGEN
and a new step
--step PATGEN
to allow MINIAODSIM from GenOnly datasets.
Useful for GEN validation and creation of alternative samples in central production with small event size.

cmsbuild · 2017-07-21T09:37:11Z

A new Pull Request was created by @perrozzi for master.

It involves the following packages:

Configuration/Applications
Configuration/EventContent
PhysicsTools/PatAlgos

@perrotta, @monttj, @cmsbuild, @franzoni, @slava77, @davidlange6 can you please review it and eventually sign? Thanks.
@ghellwig, @mmarionncern, @gouskos, @rappoccio, @imarches, @ahinzmann, @acaudron, @gpetruc, @TaiSakuma, @Martin-Grunewald, @jdolen, @nhanvtran, @JyothsnaKomaragiri, @gkasieczka, @schoef, @ferencek, @mverzett, @mariadalfonso, @pvmulder, @seemasharmafnal this is something you requested to watch as well.
@davidlange6 you are the release manager for this.

cms-bot commands are listed here

perrozzi · 2017-07-21T09:56:16Z

please test

cmsbuild · 2017-07-21T09:56:40Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/21658/console Started: 2017/07/21 11:57

cmsbuild · 2017-07-21T11:36:19Z

+1
Tested at: 2fc59bc
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19853/21658/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
be34a0d
377be83
d2da024
2717209
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19853/21658/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19853/21658/git-merge-result

cmsbuild · 2017-07-21T11:36:22Z

Comparison job queued.

slava77 · 2017-07-21T12:17:27Z

@perrozzi
please add a test configuration for this new cmsDriver step.
A matrix workflow seems more appropriate, but it can also be added as a unit test in PhysicsTools/PatAlgos/test/runtests.sh

perrozzi · 2017-07-21T12:25:03Z

@slava77 I am not sure where to find a cmsDriver example for master/93X.
I have tested it in 80X using this (then I ported and removed the cfg from the commit)

cmssw/Configuration/StandardSequences/test/test_MINIAODGEN_from_GEN.py

Line 5 in 685d6a6

    
           # with command line options: step1 --fileout file:out.root --filein /store/mc/RunIISummer15GS/DYToMuMu_M_50_TuneAZ_8TeV_pythia8/GEN-SIM/GenOnly_MCRUN2_71_V1-v3/100000/006A257F-D140-E711-93E2-0025901C1A92.root --mc --eventcontent MINIAODGEN --runUnscheduled --datatier MINIAODSIM --conditions 80X_mcRun2_asymptotic_2016_TrancheIV_v6 --step PATGEN --nThreads 1 --era Run2_2016 --python_filename test_MINIAODGEN_from_GEN.py --no_exec -n 5

do you have suggestions?

slava77 · 2017-07-21T12:28:08Z

@fabozzi
please take a look at the new steps in cmsDriver and advise @perrozzi on how to add a matrix workflow for it.
Thank you.

cmsbuild · 2017-07-21T14:14:59Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19853/21658/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 340 differences found in the comparisons
DQMHistoTests: Total files compared: 23
DQMHistoTests: Total histograms compared: 2347494
DQMHistoTests: Total failures: 30599
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2316729
DQMHistoTests: Total skipped: 166
DQMHistoTests: Total Missing objects: 0
Checked 93 log files, 14 edm output root files, 23 DQM output files

davidlange6 · 2017-07-21T14:33:47Z

made its worth a bit of thought to first agree on the concept being proposed? [I say without really looking at this]

…

On Jul 21, 2017, at 2:28 PM, Slava Krutelyov ***@***.***> wrote: @fabozzi please take a look at the new steps in cmsDriver and advise @perrozzi on how to add a matrix workflow for it. Thank you. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

slava77 · 2017-07-21T15:12:13Z

Reco comparison results: 340 differences found in the comparisons

if not semi-blandly to ignore the diffs, need to rerun this after another IB.
The PR was ahead of the IB by a bit at the time of testing

perrozzi · 2017-07-22T07:23:56Z

@davidlange6 sure, I can try to summarize here then we can discuss at the ORP meeting or else.

In GEN we are often using GenOnly workflows (i.e. GEN step without SIM) to provide high statistics alternative samples to compare unfolded data and mc or validate new versions of the MC, for instance.
In most of the cases, we use Rivet for this purpose, which relies on the HepMC product.
However, GEN(-SIM) have very short (i.e. 0) lifetime on disk [*] complicating things especially for quick validation turn around.
Thanks to the developments achieved in the last months, Rivet can now be reliably used starting from MINIAOD [**]

Since we foresee very soon one of these such quick validation turn around to validate the latest and greatest GEN updates before to launch the 2017 v2 MC campaign, to count on central production GENOnly+MINIAODSIM workflows (saving only reduced MINIAODSIM output) will be very beneficial, I think.

The validation can be carried on using the current 2016 MC campaign (71X+80X) so this feature is quite more urgent in 80X than master/93X, somehow.

Let me know if you want to discuss more or would like to have further input.
Thanks
Luca

[*] https://hypernews.cern.ch/HyperNews/CMS/get/comp-ops/3623/1/1/1.html

[**] The HepMC collection can be produced on the fly starting from the GenParticle collection, but in MINIAOD it is splitted and slimmed into PackedGenParticles and PrunedGenParticles to reduce disk space.
A "merger" has been written to combine back the two collections restoring missing links, being able to produce a complete HepMC. The Rivet results obtained with this HepMC collection have been compared to those obtained with the original collection and are successfully validated.

slava77 · 2017-07-24T16:13:08Z

@perrozzi
can all of this be simply added to GEN step?

the output data tier better be named MINIGEN or something like that, because "AOD" in the name seems inappropriate.

perrozzi · 2017-09-03T05:03:58Z

errors unrelated, seem to be due to DQM
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19853/22674/runTheMatrix-results/1330.0_ZMM_13+ZMM_13+DIGIUP15+RECOUP15+HARVESTUP15/step4_ZMM_13+ZMM_13+DIGIUP15+RECOUP15+HARVESTUP15.log

perrotta · 2017-09-04T04:55:03Z

please test

cmsbuild · 2017-09-04T04:55:26Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/22676/console Started: 2017/09/04 06:56

cmsbuild · 2017-09-04T07:08:08Z

+1
Tested at: 3781f11
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19853/22676/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
0770659
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19853/22676/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19853/22676/git-merge-result

cmsbuild · 2017-09-04T07:08:11Z

Comparison job queued.

cmsbuild · 2017-09-04T08:00:18Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19853/22676/summary.html

Comparison Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 26
DQMHistoTests: Total histograms compared: 2656522
DQMHistoTests: Total failures: 207
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2656126
DQMHistoTests: Total skipped: 189
DQMHistoTests: Total Missing objects: 0
Checked 107 log files, 14 edm output root files, 26 DQM output files

perrozzi · 2017-09-04T19:34:13Z

Hello, may I kindly ask to review this on time for tomorrow's ORP? I would really like to close this...

slava77 · 2017-09-05T16:59:17Z

+1

for #19853 3781f11

changes since the last signoff for introduce MINIGEN step and output to replace GenOnly workflows #19853 cc7ca51: a consistent rebase was made
jenkins tests pass and comparisons show no differences in monitored products
- there is an addition of a product to miniAODSIM recoGenMETs_genMetTrue

perrozzi · 2017-09-06T23:23:51Z

@perrotta, @monttj, can you please review and sign this PR, unless there are other suggestions?

perrozzi · 2017-09-11T02:34:30Z

Dear @perrotta, @monttj, I would kindly renew my invitation to please review and sign this PR, unless there are other suggestions.

perrotta · 2017-09-11T05:27:53Z

@perozzi : his PR only mosses the "analysis" signature; I can only sign for reco, and @slava77 already did six days ago
@davidlange6 : this PR is fully signed but it only misses the analysis signature since six days already; since GEN apparerntly urges having it merged in the release, if @monttj cannot sign maybe you can bypass and at least start evaluting the PR

davidlange6 · 2017-09-12T19:31:00Z

merge

cmsbuild added this to the CMSSW_9_3_X milestone Jul 21, 2017

perrozzi changed the title ~~allow running MINIAOSIM from GenOnly from cmsDriver~~ allow running MINIAOSIM from GenOnly using cmsDriver Jul 21, 2017

cmsbuild added analysis-pending comparison-pending operations-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Jul 21, 2017

perrozzi changed the title ~~allow running MINIAOSIM from GenOnly using cmsDriver~~ allow running MINIAODSIM from GenOnly using cmsDriver Jul 21, 2017

perrozzi mentioned this pull request Jul 21, 2017

allow running MINIAODSIM from GenOnly using cmsDriver #19854

Closed

cmsbuild added tests-started and removed tests-pending labels Jul 21, 2017

cmsbuild added tests-approved and removed tests-started labels Jul 21, 2017

cmsbuild added comparison-available and removed comparison-pending labels Jul 21, 2017

cmsbuild added comparison-pending tests-pending and removed comparison-notrun tests-rejected labels Sep 4, 2017

cmsbuild added tests-started and removed tests-pending labels Sep 4, 2017

cmsbuild added tests-approved and removed tests-started labels Sep 4, 2017

cmsbuild added comparison-available and removed comparison-pending labels Sep 4, 2017

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Sep 5, 2017

smuzaffar modified the milestones: CMSSW_9_4_X, CMSSW_9_3_X Sep 6, 2017

perrozzi changed the title ~~allow running MINIAODSIM from GenOnly using cmsDriver~~ introduce MINIGEN step and output to replace GenOnly workflows Sep 6, 2017

cmsbuild merged commit beb511f into cms-sw:master Sep 12, 2017

perrozzi mentioned this pull request Sep 12, 2017

Minigen 93x #20485

Merged

slava77 mentioned this pull request Sep 12, 2017

Saving generator protons from pile-up for standard mixing and premixing #20172

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

introduce MINIGEN step and output to replace GenOnly workflows #19853

introduce MINIGEN step and output to replace GenOnly workflows #19853

perrozzi commented Jul 21, 2017

cmsbuild commented Jul 21, 2017

perrozzi commented Jul 21, 2017

cmsbuild commented Jul 21, 2017 •

edited

cmsbuild commented Jul 21, 2017

cmsbuild commented Jul 21, 2017

slava77 commented Jul 21, 2017

perrozzi commented Jul 21, 2017

slava77 commented Jul 21, 2017

cmsbuild commented Jul 21, 2017

davidlange6 commented Jul 21, 2017 via email

slava77 commented Jul 21, 2017

perrozzi commented Jul 22, 2017

slava77 commented Jul 24, 2017

perrozzi commented Sep 3, 2017

perrotta commented Sep 4, 2017

cmsbuild commented Sep 4, 2017 •

edited

cmsbuild commented Sep 4, 2017

cmsbuild commented Sep 4, 2017

cmsbuild commented Sep 4, 2017

perrozzi commented Sep 4, 2017

slava77 commented Sep 5, 2017

perrozzi commented Sep 6, 2017

perrozzi commented Sep 11, 2017

perrotta commented Sep 11, 2017

davidlange6 commented Sep 12, 2017

introduce MINIGEN step and output to replace GenOnly workflows #19853

introduce MINIGEN step and output to replace GenOnly workflows #19853

Conversation

perrozzi commented Jul 21, 2017

cmsbuild commented Jul 21, 2017

perrozzi commented Jul 21, 2017

cmsbuild commented Jul 21, 2017 • edited

cmsbuild commented Jul 21, 2017

cmsbuild commented Jul 21, 2017

slava77 commented Jul 21, 2017

perrozzi commented Jul 21, 2017

slava77 commented Jul 21, 2017

cmsbuild commented Jul 21, 2017

davidlange6 commented Jul 21, 2017 via email

slava77 commented Jul 21, 2017

perrozzi commented Jul 22, 2017

slava77 commented Jul 24, 2017

perrozzi commented Sep 3, 2017

perrotta commented Sep 4, 2017

cmsbuild commented Sep 4, 2017 • edited

cmsbuild commented Sep 4, 2017

cmsbuild commented Sep 4, 2017

cmsbuild commented Sep 4, 2017

perrozzi commented Sep 4, 2017

slava77 commented Sep 5, 2017

perrozzi commented Sep 6, 2017

perrozzi commented Sep 11, 2017

perrotta commented Sep 11, 2017

davidlange6 commented Sep 12, 2017

cmsbuild commented Jul 21, 2017 •

edited

cmsbuild commented Sep 4, 2017 •

edited