Add a service to handle MPI initialisation #31155

fwyzard · 2020-08-16T12:29:13Z

PR description:

Add an MPIService to handle

MPI initialisation requiring multithreading support
LogInfo of some MPI environment information
MPI finalisation

Add a python configuration to test the MPIService.

Any other service or module that uses MPI should require this service, to guarantee that MPI is initialised/finalised in the correct order.

PR validation:

$ scram b runtests
...
Creating test log file logs/slc7_amd64_gcc820/testing.log
Package HeterogeneousCore/MPIServices: Running test TestHeterogeneousCoreMPIServices
 
===== Test "TestHeterogeneousCoreMPIServices" ====
%MSG-i MPIService:  (NoModuleName) 16-Aug-2020 14:36:28 CEST pre-events
MPI_COMM_WORLD size: 1
MPI_COMM_WORLD rank: 0
MPI processor name:  fu-c2a02-37-02
The MPI library provides the MPI_THREAD_MULTIPLE multithreading support level
MPI successfully initialised
%MSG

=============================================

MessageLogger Summary

Severity    # Occurrences   Total Occurrences
--------    -------------   -----------------

dropped waiting message count 0

---> test TestHeterogeneousCoreMPIServices succeeded
 
^^^^ End Test TestHeterogeneousCoreMPIServices ^^^^

if this PR is a backport please specify the original PR and why you need to backport that PR:

This PR includes the work from:

Add a simple MPIService to handle MPI initialisation cms-patatrack/cmssw#151 Add a simple MPIService to handle MPI initialisation
Add MPIService test file cms-patatrack/cmssw#535 Update messages and add a test file for the MPIService

cmsbuild · 2020-08-16T12:29:34Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-08-16T12:36:18Z

The code-checks are being triggered in jenkins.

fwyzard · 2020-08-16T12:38:08Z

@cmsbuild, please test

cmsbuild · 2020-08-16T12:40:50Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31155/17782

This PR adds an extra 12KB to repository

fwyzard · 2020-08-16T12:40:54Z

@makortel do you think this should go under HeterogeneousCore/MPICore rather than HeterogeneousCore/MPIServices ?

cmsbuild · 2020-08-16T12:41:10Z

The tests are being triggered in jenkins.

CMSSW_11_2_X_2020-08-16-0000/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/8779/console Started: 2020/08/16 15:29

cmsbuild · 2020-08-16T12:41:11Z

A new Pull Request was created by @fwyzard (Andrea Bocci) for master.

It involves the following packages:

HeterogeneousCore/MPIServices

@makortel, @cmsbuild, @fwyzard can you please review it and eventually sign? Thanks.
@makortel, @rovere this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2020-08-16T14:39:27Z

+1
Tested at: bd3aecb
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fe2255/8779/summary.html
CMSSW: CMSSW_11_2_X_2020-08-16-0000
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-08-16T14:39:29Z

Comparison job queued.

cmsbuild · 2020-08-16T16:08:03Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fe2255/8779/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 35
DQMHistoTests: Total histograms compared: 2608246
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2608191
DQMHistoTests: Total skipped: 48
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 34 files compared)
Checked 149 log files, 22 edm output root files, 35 DQM output files

makortel · 2020-08-17T18:44:46Z

do you think this should go under HeterogeneousCore/MPICore rather than HeterogeneousCore/MPIServices ?

MPIServices sounds more clear to me.

HeterogeneousCore/MPIServices/plugins/MPIService.cc

cmsbuild · 2020-08-18T15:51:11Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-08-18T15:58:55Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31155/17794

This PR adds an extra 12KB to repository

cmsbuild · 2020-08-18T15:59:16Z

The tests are being triggered in jenkins.

CMSSW_11_2_X_2020-08-17-2300/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/8796/console Started: 2020/08/18 18:46

cmsbuild · 2020-08-18T15:59:17Z

Pull request #31155 was updated. @cmsbuild can you please check and sign again.

cmsbuild · 2020-08-18T17:56:32Z

+1
Tested at: 7d6b2a6
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fe2255/8796/summary.html
CMSSW: CMSSW_11_2_X_2020-08-17-2300
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-08-18T17:56:35Z

Comparison job queued.

cmsbuild · 2020-08-18T19:25:27Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fe2255/8796/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 35
DQMHistoTests: Total histograms compared: 2608246
DQMHistoTests: Total failures: 5
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2608192
DQMHistoTests: Total skipped: 48
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 34 files compared)
DQMHistoSizes: changed ( 10224.0 ): 0.004 KiB MessageLogger/Warnings
Checked 149 log files, 22 edm output root files, 35 DQM output files

qliphy · 2020-08-19T01:47:11Z

+1

makortel · 2020-08-19T17:10:41Z

The unit test fails in CMSSW_11_2_X_2020-08-19-1100, but the reason is rather unclear. I opened #31180 to follow up.

cmsbuild added this to the CMSSW_11_2_X milestone Aug 16, 2020

cmsbuild added code-checks-pending comparison-pending heterogeneous-pending orp-pending pending-signatures tests-pending labels Aug 16, 2020

fwyzard force-pushed the add_MPI_service branch from 3fa9db8 to bd3aecb Compare August 16, 2020 12:35

cmsbuild added code-checks-approved and removed code-checks-pending labels Aug 16, 2020

cmsbuild added tests-started and removed tests-pending labels Aug 16, 2020

cmsbuild added tests-approved and removed tests-started labels Aug 16, 2020

cmsbuild added comparison-available and removed comparison-pending labels Aug 16, 2020

makortel reviewed Aug 17, 2020

View reviewed changes

HeterogeneousCore/MPIServices/plugins/MPIService.cc Outdated Show resolved Hide resolved

HeterogeneousCore/MPIServices/plugins/MPIService.cc Outdated Show resolved Hide resolved

fwyzard force-pushed the add_MPI_service branch from bd3aecb to 3af4861 Compare August 17, 2020 22:25

cmsbuild removed the code-checks-approved label Aug 17, 2020

cmsbuild added code-checks-pending comparison-pending fully-signed heterogeneous-approved tests-pending and removed comparison-available heterogeneous-pending pending-signatures tests-approved labels Aug 18, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Aug 18, 2020

cmsbuild added tests-started and removed tests-pending labels Aug 18, 2020

cmsbuild added tests-approved and removed tests-started labels Aug 18, 2020

cmsbuild added comparison-available and removed comparison-pending labels Aug 18, 2020

cmsbuild added orp-approved and removed orp-pending labels Aug 19, 2020

cmsbuild merged commit 2a6c55d into cms-sw:master Aug 19, 2020

fwyzard deleted the add_MPI_service branch August 21, 2020 06:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a service to handle MPI initialisation #31155

Add a service to handle MPI initialisation #31155

fwyzard commented Aug 16, 2020 •

edited

cmsbuild commented Aug 16, 2020

cmsbuild commented Aug 16, 2020

fwyzard commented Aug 16, 2020

cmsbuild commented Aug 16, 2020

fwyzard commented Aug 16, 2020

cmsbuild commented Aug 16, 2020 •

edited

cmsbuild commented Aug 16, 2020

cmsbuild commented Aug 16, 2020

cmsbuild commented Aug 16, 2020

cmsbuild commented Aug 16, 2020

makortel commented Aug 17, 2020

cmsbuild commented Aug 18, 2020

cmsbuild commented Aug 18, 2020

cmsbuild commented Aug 18, 2020 •

edited

cmsbuild commented Aug 18, 2020

cmsbuild commented Aug 18, 2020

cmsbuild commented Aug 18, 2020

cmsbuild commented Aug 18, 2020

qliphy commented Aug 19, 2020

makortel commented Aug 19, 2020

Add a service to handle MPI initialisation #31155

Add a service to handle MPI initialisation #31155

Conversation

fwyzard commented Aug 16, 2020 • edited

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented Aug 16, 2020

cmsbuild commented Aug 16, 2020

fwyzard commented Aug 16, 2020

cmsbuild commented Aug 16, 2020

fwyzard commented Aug 16, 2020

cmsbuild commented Aug 16, 2020 • edited

cmsbuild commented Aug 16, 2020

cmsbuild commented Aug 16, 2020

cmsbuild commented Aug 16, 2020

cmsbuild commented Aug 16, 2020

makortel commented Aug 17, 2020

cmsbuild commented Aug 18, 2020

cmsbuild commented Aug 18, 2020

cmsbuild commented Aug 18, 2020 • edited

cmsbuild commented Aug 18, 2020

cmsbuild commented Aug 18, 2020

cmsbuild commented Aug 18, 2020

cmsbuild commented Aug 18, 2020

qliphy commented Aug 19, 2020

makortel commented Aug 19, 2020

fwyzard commented Aug 16, 2020 •

edited

cmsbuild commented Aug 16, 2020 •

edited

cmsbuild commented Aug 18, 2020 •

edited