MPI and OpenMP setup issues #956

buzhiqiuyuebai · 2023-09-18T09:03:25Z

buzhiqiuyuebai
Sep 18, 2023

The memory leak was fixed in 948, but when testing with this example, I found that with MPI.x OpenMP=1.x 54, the cpu utilisation was high at 5397%. with MPI .x OpenMP=1 .x 27, the cpu utilisation was high at 2685%. But when set to 2 .x 27, the cpu utilisation per process seems to be stuck at 200%. Do you know just what causes this please?

Answered by krenzland

Sep 21, 2023

The first one is broken:
Thu Sep 21 15:56:46, Info: OpenMP worker affinity (this process): "0---------|----------|----------|----------|----------|------" Thu Sep 21 15:56:46, Info: OpenMP worker affinity (this node) : "0-2-------|----------|----------|----------|----------|------" Thu Sep 21 15:56:46, Info: Communication thread affinity : "----4-6-8-|0-2-4-6-8-|0-2-4-6-8-|0-2-4-6-8-|0-2-4-6-8-|0-2-4-"

means that you that all OpenMP threads use the same physical core. Are you sure that the enviromental variable OMP_PLACES is set correctly?

View full answer

davschneller · 2023-09-18T09:22:50Z

davschneller
Sep 18, 2023
Maintainer

Hi, how does your pinning mask / OpenMP worker affinity map look like in each of these cases? (it's printed at the start of the simulation) And how have you set the environment variables there?

0 replies

buzhiqiuyuebai · 2023-09-21T08:01:43Z

buzhiqiuyuebai
Sep 21, 2023
Author

Hi, how does your pinning mask / OpenMP worker affinity map look like in each of these cases? (it's printed at the start of the simulation) And how have you set the environment variables there?

the environment variables:

MP_SINGLE_THREAD=no
unset KMP_AFFINITY
export OMP_NUM_THREADS=27
export OMP_PLACES="cores(27)"

MPI .x OpenMP=2 .x 27

Thu Sep 21 15:56:46, Info:  Welcome to SeisSol 
Thu Sep 21 15:56:46, Info:  Copyright (c) 2012-2021, SeisSol Group 
Thu Sep 21 15:56:46, Info:  Built on: Jun 13 2023 11:39:22 
Thu Sep 21 15:56:46, Info:  Version: 202103_Sumatra-1175-g703d36c 
Thu Sep 21 15:56:46, Info:  Running on: node02 
Thu Sep 21 15:56:46, Info:  Using MPI with #ranks: 2 
Thu Sep 21 15:56:46, Info:  Using OMP with #threads/rank: 27 
Thu Sep 21 15:56:46, Info:  OpenMP worker affinity (this process): "0---------|----------|----------|----------|----------|------" 
Thu Sep 21 15:56:46, Info:  OpenMP worker affinity (this node)   : "0-2-------|----------|----------|----------|----------|------" 
Thu Sep 21 15:56:46, Info:  Communication thread affinity        : "----4-6-8-|0-2-4-6-8-|0-2-4-6-8-|0-2-4-6-8-|0-2-4-6-8-|0-2-4-" 
Thu Sep 21 15:56:46, Info:  The stack size ulimit is  20971520 [kb]. 
Thu Sep 21 15:56:46, Info:  Input Parameters:
 0x1795240 
Rank:        0 | Info    | <--------------------------------------------------------->
Rank:        0 | Info    | <                SeisSol MPI initialization               >
Rank:        0 | Info    | <--------------------------------------------------------->
Rank:        0 | Info    |  Double precision used for real.
Rank:        0 | Info    | <--------------------------------------------------------->
 INFORMATION: The assumed unit number is           6 for stdout and           0 for stderr.
              If no information follows, please change the value.
Rank:        0 | Info    | <--------------------------------------------------------->
Rank:        0 | Info    | <     Start ini_SeisSol ...                               >
Rank:        0 | Info    | <--------------------------------------------------------->
Rank:        0 | Info    | <  Parameters read from file: parameters.par              >
Rank:        0 | Info    | <                                                         >
Rank:        0 | Info    | No plasticity assumed. 
Rank:        0 | Info    | No attenuation assumed. 
Rank:        0 | Info    | No adjoint wavefield generated. 
Rank:        0 | Info    | Isotropic material is assumed. 
Rank:        0 | Info    | Read a PUML mesh file
Rank:        0 | Warning | Ignoring space order from parameter file, using           2
Rank:        0 | Info    | Volume output is in XDMF format (new implementation)
Rank:        0 | Info    | Output data are generated at delta T=   0.50000000000000000     
Rank:        0 | Info    | Use POSIX XdmfWriter backend
Rank:        0 | Info    | Refinement strategy for volume output is Face Extraction :  4 subcells per cell
Thu Sep 21 15:56:46, Info:  Running mini SeisSol to determine node weight 
Thu Sep 21 15:56:47, Info:  Node weights: mean = 2.49271  std = 0.00639087  min = 2.48632  median = 2.49271  max = 2.4991 
Thu Sep 21 15:56:47, Info:  Reading PUML mesh loh.puml.h5

MPI .x OpenMP=1 .x 27

Thu Sep 21 15:55:18, Info:  Welcome to SeisSol 
Thu Sep 21 15:55:18, Info:  Copyright (c) 2012-2021, SeisSol Group 
Thu Sep 21 15:55:18, Info:  Built on: Jun 13 2023 11:39:22 
Thu Sep 21 15:55:18, Info:  Version: 202103_Sumatra-1175-g703d36c 
Thu Sep 21 15:55:18, Info:  Running on: node02 
Thu Sep 21 15:55:18, Info:  Using MPI with #ranks: 1 
Thu Sep 21 15:55:18, Info:  Using OMP with #threads/rank: 27 
Thu Sep 21 15:55:18, Info:  OpenMP worker affinity (this process): "0123456789|0123456789|0123456---|----------|----------|------" 
Thu Sep 21 15:55:18, Info:  OpenMP worker affinity (this node)   : "0123456789|0123456789|0123456---|----------|----------|------" 
Thu Sep 21 15:55:18, Info:  Communication thread affinity        : "----------|----------|-------789|0123456789|0123456789|012345" 
Thu Sep 21 15:55:18, Info:  The stack size ulimit is  20971520 [kb]. 
Thu Sep 21 15:55:18, Info:  Input Parameters:
 0x2919650 
Rank:        0 | Info    | <--------------------------------------------------------->
Rank:        0 | Info    | <                SeisSol MPI initialization               >
Rank:        0 | Info    | <--------------------------------------------------------->
Rank:        0 | Info    |  Double precision used for real.
Rank:        0 | Info    | <--------------------------------------------------------->
 INFORMATION: The assumed unit number is           6 for stdout and           0 for stderr.
              If no information follows, please change the value.
Rank:        0 | Info    | <--------------------------------------------------------->
Rank:        0 | Info    | <     Start ini_SeisSol ...                               >
Rank:        0 | Info    | <--------------------------------------------------------->
Rank:        0 | Info    | <  Parameters read from file: parameters.par              >
Rank:        0 | Info    | <                                                         >
Rank:        0 | Info    | No plasticity assumed. 
Rank:        0 | Info    | No attenuation assumed. 
Rank:        0 | Info    | No adjoint wavefield generated. 
Rank:        0 | Info    | Isotropic material is assumed. 
Rank:        0 | Info    | Read a PUML mesh file
Rank:        0 | Warning | Ignoring space order from parameter file, using           2
Rank:        0 | Info    | Volume output is in XDMF format (new implementation)
Rank:        0 | Info    | Output data are generated at delta T=   0.50000000000000000     
Rank:        0 | Info    | Use POSIX XdmfWriter backend
Rank:        0 | Info    | Refinement strategy for volume output is Face Extraction :  4 subcells per cell
Thu Sep 21 15:55:18, Info:  Reading PUML mesh loh.puml.h5

4 replies

krenzland Sep 21, 2023
Maintainer

The first one is broken:
Thu Sep 21 15:56:46, Info: OpenMP worker affinity (this process): "0---------|----------|----------|----------|----------|------" Thu Sep 21 15:56:46, Info: OpenMP worker affinity (this node) : "0-2-------|----------|----------|----------|----------|------" Thu Sep 21 15:56:46, Info: Communication thread affinity : "----4-6-8-|0-2-4-6-8-|0-2-4-6-8-|0-2-4-6-8-|0-2-4-6-8-|0-2-4-"

means that you that all OpenMP threads use the same physical core. Are you sure that the enviromental variable OMP_PLACES is set correctly?

Answer selected by buzhiqiuyuebai

buzhiqiuyuebai Sep 21, 2023
Author

Is this the right setting? I've both written it to ~/.bashrc and executed it in the terminal, so why is the problem still there?

I try to use the following enviromental variable, but it still doesn't work.

krenzland Sep 21, 2023
Maintainer

I don't know. This is definitely not a SeisSol issue - I'd recommend that you discuss this with the admins of your cluster. The settings that you are using should work.

buzhiqiuyuebai Sep 21, 2023
Author

OK,thank you,I will have a try, because I find the process of reading mesh have the similar problem, it is limited to one core.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MPI and OpenMP setup issues #956

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 4 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

MPI and OpenMP setup issues #956

buzhiqiuyuebai Sep 18, 2023

Replies: 2 comments · 4 replies

davschneller Sep 18, 2023 Maintainer

buzhiqiuyuebai Sep 21, 2023 Author

krenzland Sep 21, 2023 Maintainer

buzhiqiuyuebai Sep 21, 2023 Author

krenzland Sep 21, 2023 Maintainer

buzhiqiuyuebai Sep 21, 2023 Author

buzhiqiuyuebai
Sep 18, 2023

Replies: 2 comments 4 replies

davschneller
Sep 18, 2023
Maintainer

buzhiqiuyuebai
Sep 21, 2023
Author

krenzland Sep 21, 2023
Maintainer

buzhiqiuyuebai Sep 21, 2023
Author

krenzland Sep 21, 2023
Maintainer

buzhiqiuyuebai Sep 21, 2023
Author