Prevent simultaneous execution of Rivet modules #26597

mseidel42 · 2019-05-02T15:41:24Z

PR description:

Several nanoAod workflows crashed when multi-thread jobs were running both Rivet-based ParticleLevelProducer and HTXSRivetProducer.
This PR prevents Rivet-based modules to be executed at the same time.

PR validation:

Updated modules tested on Higgs and non-Higgs sample. All Rivet modules run together within a test job using 10 threads, no crash after 10k events (the old versions crashed after a few events).

RivetAnalyzer:

Rivet.Analysis.Handler: INFO  Finalising analyses
Rivet.Analysis.Handler: INFO  Processed 10000 events

ParticleLevelProducer: Object output (dressed leptons etc.) is as expected.

HTXSRivetProducer:

Rivet.Analysis.HiggsTemplateCrossSections: INFO   ====================================================== 
Rivet.Analysis.HiggsTemplateCrossSections: INFO        Higgs Template X-Sec Categorization Tool          
Rivet.Analysis.HiggsTemplateCrossSections: INFO                  Status Code Summary                     
Rivet.Analysis.HiggsTemplateCrossSections: INFO   ====================================================== 
Rivet.Analysis.HiggsTemplateCrossSections: INFO       >>>> 10000 events successfully categorized
Rivet.Analysis.HiggsTemplateCrossSections: INFO       >>>> --> the following errors occured:
Rivet.Analysis.HiggsTemplateCrossSections: INFO       >>>> 0 had an undefined Higgs production mode.
Rivet.Analysis.HiggsTemplateCrossSections: INFO       >>>> 0 failed momentum conservation.
Rivet.Analysis.HiggsTemplateCrossSections: INFO       >>>> 0 failed to identify a valid Higgs boson.
Rivet.Analysis.HiggsTemplateCrossSections: INFO       >>>> 0 failed to identify the hard scatter vertex.
Rivet.Analysis.HiggsTemplateCrossSections: INFO       >>>> 0 VH: to identify a valid V-boson.
Rivet.Analysis.HiggsTemplateCrossSections: INFO       >>>> 0 failed to identify valid Ws from top decay.
Rivet.Analysis.HiggsTemplateCrossSections: INFO   ====================================================== 
Rivet.Analysis.HiggsTemplateCrossSections: INFO   ======================================================

cmsbuild · 2019-05-02T15:41:58Z

The code-checks are being triggered in jenkins.

cmsbuild · 2019-05-02T15:47:02Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26597/9532

This PR adds an extra 20KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26597/9532/code-format.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26597/9532/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2019-05-02T15:58:34Z

The code-checks are being triggered in jenkins.

mseidel42 · 2019-05-02T15:59:11Z

Code-format makes the PR hard to read now, please just see the first commit for the relevant changes: 37dd72d

cmsbuild · 2019-05-02T16:03:59Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26597/9536

This PR adds an extra 24KB to repository

cmsbuild · 2019-05-02T16:04:31Z

A new Pull Request was created by @intrepid42 (Markus Seidel) for master.

It involves the following packages:

GeneratorInterface/RivetInterface

@alberto-sanchez, @cmsbuild, @qliphy, @perrozzi, @efeyazgan can you please review it and eventually sign? Thanks.
@alberto-sanchez, @agrohsje, @mkirsano this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

efeyazgan · 2019-05-02T16:43:27Z

please test

cmsbuild · 2019-05-02T16:44:01Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/34462/console Started: 2019/05/02 18:45

cmsbuild · 2019-05-02T18:12:36Z

+1
Tested at: 05acdaa
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26597/34462/summary.html

cmsbuild · 2019-05-02T18:12:42Z

Comparison job queued.

cmsbuild · 2019-05-02T19:40:42Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26597/34462/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 33
DQMHistoTests: Total histograms compared: 3211964
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3211759
DQMHistoTests: Total skipped: 204
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 32 files compared)
Checked 137 log files, 14 edm output root files, 33 DQM output files

cmsbuild · 2019-05-06T07:53:15Z

The code-checks are being triggered in jenkins.

cmsbuild · 2019-05-06T08:00:52Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26597/9596

This PR adds an extra 24KB to repository

cmsbuild · 2019-05-06T08:01:25Z

Pull request #26597 was updated. @alberto-sanchez, @cmsbuild, @qliphy, @perrozzi, @efeyazgan can you please check and sign again.

efeyazgan · 2019-05-06T08:08:36Z

please test

cmsbuild · 2019-05-06T08:09:08Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/34508/console Started: 2019/05/06 10:09

cmsbuild · 2019-05-06T09:36:26Z

+1
Tested at: 4818e56
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26597/34508/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
4818e56
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26597/34508/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26597/34508/git-merge-result

cmsbuild · 2019-05-06T09:36:32Z

Comparison job queued.

peruzzim · 2019-05-06T11:35:52Z

I have tested manually a long nanoAOD workflow, and I confirm that the multi-threading issue seems solved by this fix. It would be great if this makes it to pre5.

cmsbuild · 2019-05-06T12:10:09Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26597/34508/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 33
DQMHistoTests: Total histograms compared: 3211964
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3211759
DQMHistoTests: Total skipped: 204
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 32 files compared)
Checked 137 log files, 14 edm output root files, 33 DQM output files

efeyazgan · 2019-05-06T12:10:43Z

+1

cmsbuild · 2019-05-06T12:11:17Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

peruzzim · 2019-05-06T12:53:47Z

Please backport this fix to 10_2_X. Thanks!

fabiocos · 2019-05-06T20:31:47Z

+1

Prevent simultaneous execution of Rivet modules

37dd72d

cmsbuild added this to the CMSSW_10_6_X milestone May 2, 2019

cmsbuild added code-checks-pending comparison-pending generators-pending orp-pending pending-signatures tests-pending labels May 2, 2019

cmsbuild added code-checks-rejected and removed code-checks-pending labels May 2, 2019

Apply code-format

05acdaa

cmsbuild added code-checks-pending and removed code-checks-rejected labels May 2, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels May 2, 2019

cmsbuild added tests-started and removed tests-pending labels May 2, 2019

cmsbuild added tests-approved and removed tests-started labels May 2, 2019

cmsbuild added the tests-pending label May 6, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels May 6, 2019

cmsbuild added tests-started and removed tests-pending labels May 6, 2019

cmsbuild added tests-approved and removed tests-started labels May 6, 2019

cmsbuild added comparison-available and removed comparison-pending labels May 6, 2019

cmsbuild added fully-signed generators-approved and removed generators-pending pending-signatures labels May 6, 2019

mseidel42 mentioned this pull request May 6, 2019

Prevent simultaneous execution of Rivet modules (10_2_X) #26657

Merged

cmsbuild added orp-approved and removed orp-pending labels May 6, 2019

cmsbuild merged commit d6b1c1e into cms-sw:master May 6, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prevent simultaneous execution of Rivet modules #26597

Prevent simultaneous execution of Rivet modules #26597

mseidel42 commented May 2, 2019 •

edited

cmsbuild commented May 2, 2019

cmsbuild commented May 2, 2019

cmsbuild commented May 2, 2019

mseidel42 commented May 2, 2019

cmsbuild commented May 2, 2019

cmsbuild commented May 2, 2019

efeyazgan commented May 2, 2019

cmsbuild commented May 2, 2019 •

edited

cmsbuild commented May 2, 2019

cmsbuild commented May 2, 2019

cmsbuild commented May 2, 2019

cmsbuild commented May 6, 2019

cmsbuild commented May 6, 2019

cmsbuild commented May 6, 2019

efeyazgan commented May 6, 2019

cmsbuild commented May 6, 2019 •

edited

cmsbuild commented May 6, 2019

cmsbuild commented May 6, 2019

peruzzim commented May 6, 2019

cmsbuild commented May 6, 2019

efeyazgan commented May 6, 2019

cmsbuild commented May 6, 2019

peruzzim commented May 6, 2019

fabiocos commented May 6, 2019

Prevent simultaneous execution of Rivet modules #26597

Prevent simultaneous execution of Rivet modules #26597

Conversation

mseidel42 commented May 2, 2019 • edited

PR description:

PR validation:

cmsbuild commented May 2, 2019

cmsbuild commented May 2, 2019

cmsbuild commented May 2, 2019

mseidel42 commented May 2, 2019

cmsbuild commented May 2, 2019

cmsbuild commented May 2, 2019

efeyazgan commented May 2, 2019

cmsbuild commented May 2, 2019 • edited

cmsbuild commented May 2, 2019

cmsbuild commented May 2, 2019

cmsbuild commented May 2, 2019

cmsbuild commented May 6, 2019

cmsbuild commented May 6, 2019

cmsbuild commented May 6, 2019

efeyazgan commented May 6, 2019

cmsbuild commented May 6, 2019 • edited

cmsbuild commented May 6, 2019

cmsbuild commented May 6, 2019

peruzzim commented May 6, 2019

cmsbuild commented May 6, 2019

efeyazgan commented May 6, 2019

cmsbuild commented May 6, 2019

peruzzim commented May 6, 2019

fabiocos commented May 6, 2019

mseidel42 commented May 2, 2019 •

edited

cmsbuild commented May 2, 2019 •

edited

cmsbuild commented May 6, 2019 •

edited