examples: Update MPI tutorial notebook and scripts #1923

georgebisbas · 2022-05-20T11:09:55Z

Update the setup script and notebook.
Add deterministic output (--no-stream)

Much needed after 3 years.

review-notebook-app · 2022-05-20T11:09:59Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

codecov · 2022-05-20T11:17:28Z

Codecov Report

Merging #1923 (af10183) into master (45c75a7) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1923   +/-   ##
=======================================
  Coverage   89.60%   89.60%           
=======================================
  Files         211      211           
  Lines       35941    35941           
  Branches     5414     5414           
=======================================
  Hits        32205    32205           
  Misses       3232     3232           
  Partials      504      504

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 45c75a7...af10183. Read the comment docs.

FabioLuporini · 2022-05-24T07:47:05Z

scripts/create_ipyparallel_mpi_profile.sh

@@ -6,11 +6,6 @@ IPYTHONDIR=~/.ipython
 # Create a new profile, called "mpi"
 ipython profile create --parallel --profile=mpi

-# Add the following line as per instructions from


out of curiosity, why are you dropping this? unnecessary?

Yes, the machinery in line 12 has been deprecated.
The option engines=mpi I am using here:

https://github.com/devitocodes/devito/pull/1923/files#diff-4acc409654fcff44dde5c96e156d25a86f29d10a612d110f245af6f76a47a443R25

is taking care of instructing the MPI launchers.

FabioLuporini · 2022-05-24T07:47:39Z

Thank you. So, you've tested it locally, right? and it still does work? nice

georgebisbas

Maybe this can be merged if happy

georgebisbas · 2022-05-24T09:10:53Z

scripts/create_ipyparallel_mpi_profile.sh

@@ -6,11 +6,6 @@ IPYTHONDIR=~/.ipython
 # Create a new profile, called "mpi"
 ipython profile create --parallel --profile=mpi

-# Add the following line as per instructions from


Yes, the machinery in line 12 has been deprecated.
The option engines=mpi I am using here:

https://github.com/devitocodes/devito/pull/1923/files#diff-4acc409654fcff44dde5c96e156d25a86f29d10a612d110f245af6f76a47a443R25

is taking care of instructing the MPI launchers.

georgebisbas · 2022-05-24T09:11:54Z

Thank you. So, you've tested it locally, right? and it still does work? nice

Tested for me and alongside Kene as well

mloubout · 2022-05-24T11:06:01Z

Can it be re-enable in the ci or is new version still notnworking

georgebisbas · 2022-05-24T17:28:23Z

scripts/create_ipyparallel_mpi_profile.sh

 ver=$(mpiexec --version)
 if [[ $ver == *"open-mpi"* ]]; then
  # OpenMPI need to be told that is allowed to oversubscribe cores
+  rm $IPYTHONDIR/profile_mpi/ipcluster_config.py


Delete any previous profiles in host.

FabioLuporini · 2022-05-25T13:14:16Z

.github/workflows/examples-mpi.yml

-        # ipcluster start --profile=mpi -n 4 --daemon
-        # py.test --nbval examples/mpi
-        # ipcluster stop --profile=mpi
+        pip3 install "ipyparallel<8.4"


@georgebisbas out of curiosity, why <8.4 ? and shouldn't this rather be added to the "Install dependencies" section above?

Right, thought of it, maybe added in requirements-optional.txt ? (8.3 is current version)

FabioLuporini · 2022-05-25T13:19:04Z

@georgebisbas we also probably need to add some NBVAL_IGNORE_OUTPUT or NBVAL_SKIP?

…iles

mloubout

Almost there, two small comments

mloubout · 2022-06-06T12:14:21Z

.github/workflows/examples-mpi.yml

@@ -43,14 +43,16 @@ jobs:
    - name: Install dependencies
      run: |
        pip install --upgrade pip
-        pip install -e .
+        pip install -e .[extras]


[extras,mpi]?

mloubout · 2022-06-06T12:14:50Z

requirements-optional.txt

@@ -1,3 +1,3 @@
 matplotlib
-ipyparallel
+ipyparallel<8.4


move to mpi requirements

georgebisbas · 2022-06-06T14:37:29Z

requirements-optional.txt

@@ -1,3 +1,3 @@
 matplotlib
-ipyparallel
+ipyparallel<8.4


georgebisbas · 2022-06-06T14:37:38Z

.github/workflows/examples-mpi.yml

@@ -43,14 +43,16 @@ jobs:
    - name: Install dependencies
      run: |
        pip install --upgrade pip
-        pip install -e .
+        pip install -e .[extras]


georgebisbas · 2022-06-06T14:41:57Z

requirements-mpi.txt

@@ -1 +1,2 @@
-mpi4py
+mpi4py<4.0


NFR: this was added because soon mpi4py will go to version 4 after long time of minor upgrades

Does dependabot actually understand this is a dependency file as well?

FabioLuporini · 2022-06-07T10:03:44Z

Merged, thanks

georgebisbas added the MPI mpi-related label May 20, 2022

georgebisbas requested review from FabioLuporini and rhodrin May 20, 2022 11:09

georgebisbas self-assigned this May 20, 2022

georgebisbas force-pushed the mpi_notebook_refresh branch from 53d0c70 to c4e79e5 Compare May 20, 2022 11:16

georgebisbas force-pushed the mpi_notebook_refresh branch from c4e79e5 to 537ef01 Compare May 20, 2022 11:25

FabioLuporini reviewed May 24, 2022

View reviewed changes

FabioLuporini added examples examples and removed MPI mpi-related labels May 24, 2022

georgebisbas commented May 24, 2022

View reviewed changes

georgebisbas force-pushed the mpi_notebook_refresh branch from 3fe080f to 4e57492 Compare May 24, 2022 17:24

georgebisbas commented May 24, 2022

View reviewed changes

georgebisbas linked an issue May 25, 2022 that may be closed by this pull request

mpi notebook and ipyparallel race conditions #859

Closed

FabioLuporini reviewed May 25, 2022

View reviewed changes

georgebisbas requested a review from FabioLuporini May 26, 2022 08:49

georgebisbas force-pushed the mpi_notebook_refresh branch 4 times, most recently from 23a2885 to 2321176 Compare May 30, 2022 09:18

mpi: Update script and notebooks

0e10d80

georgebisbas force-pushed the mpi_notebook_refresh branch from 2321176 to dcbfebf Compare May 31, 2022 18:00

georgebisbas added this to WIP in Minor ideas Jun 2, 2022

georgebisbas force-pushed the mpi_notebook_refresh branch from db616cf to 71c882e Compare June 3, 2022 11:13

examples: Order rank output with --no-stream and remove past mpi prof…

780b8aa

…iles

examples: Update cell output, add nbval

c00887d

georgebisbas force-pushed the mpi_notebook_refresh branch from 71c882e to c00887d Compare June 3, 2022 12:59

mloubout reviewed Jun 6, 2022

View reviewed changes

install: Update requirements -mpi -optional

af10183

georgebisbas changed the title ~~mpi: Update script and notebooks~~ examples: Update MPI tutorial notebook and scripts Jun 6, 2022

georgebisbas commented Jun 6, 2022

View reviewed changes

FabioLuporini approved these changes Jun 7, 2022

View reviewed changes

FabioLuporini merged commit 5e0f6d0 into master Jun 7, 2022

FabioLuporini deleted the mpi_notebook_refresh branch June 7, 2022 10:03

AtilaSaraiva mentioned this pull request Aug 16, 2022

devito: unstable-2022-04-22 -> 4.7.1 NixOS/nixpkgs#186943

Merged

13 tasks

georgebisbas moved this from WIP to DONE in Minor ideas Aug 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

examples: Update MPI tutorial notebook and scripts #1923

examples: Update MPI tutorial notebook and scripts #1923

georgebisbas commented May 20, 2022 •

edited

review-notebook-app bot commented May 20, 2022

codecov bot commented May 20, 2022 •

edited

FabioLuporini May 24, 2022

georgebisbas May 24, 2022

FabioLuporini commented May 24, 2022

georgebisbas left a comment

georgebisbas May 24, 2022

georgebisbas commented May 24, 2022

mloubout commented May 24, 2022

georgebisbas May 24, 2022

FabioLuporini May 25, 2022

georgebisbas May 25, 2022 •

edited

FabioLuporini commented May 25, 2022

mloubout left a comment

mloubout Jun 6, 2022

georgebisbas Jun 6, 2022

mloubout Jun 6, 2022

georgebisbas Jun 6, 2022

georgebisbas Jun 6, 2022

georgebisbas Jun 6, 2022

georgebisbas Jun 6, 2022

mloubout Jun 6, 2022

FabioLuporini commented Jun 7, 2022

examples: Update MPI tutorial notebook and scripts #1923

examples: Update MPI tutorial notebook and scripts #1923

Conversation

georgebisbas commented May 20, 2022 • edited

review-notebook-app bot commented May 20, 2022

codecov bot commented May 20, 2022 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FabioLuporini commented May 24, 2022

georgebisbas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

georgebisbas commented May 24, 2022

mloubout commented May 24, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

georgebisbas May 25, 2022 • edited

Choose a reason for hiding this comment

FabioLuporini commented May 25, 2022

mloubout left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FabioLuporini commented Jun 7, 2022

georgebisbas commented May 20, 2022 •

edited

codecov bot commented May 20, 2022 •

edited

georgebisbas May 25, 2022 •

edited