MFC testing failing on aarch64 MacOS due to false hang when `-j X` with `X` is the max. number of available threads

[sbryngelson]

[I]shb-m1pro: Downloads/MFC $ ./mfc.sh test -j 8
[mfc.sh]: Entering the Python virtual environment (venv).
      ___            ___          ___
     /__/\          /  /\        /  /\       spencer@shb-m1pro.attlocal.net [Darwin]
    |  |::\        /  /:/_      /  /:/       ---------------------------------------
    |  |:|:\      /  /:/ /\    /  /:/
  __|__|:|\:\    /  /:/ /:/   /  /:/  ___
 /__/::::| \:\  /__/:/ /:/   /__/:/  /  /\   --jobs:    8
 \  \:\~~\__\/  \  \:\/:/    \  \:\ /  /:/   --mode:    release-cpu
  \  \:\         \  \::/      \  \:\  /:/
   \  \:\         \  \:\       \  \:\/:/
    \  \:\         \  \:\       \  \::/
     \__\/          \__\/        \__\/       $ ./mfc.sh [build, run, test, clean] --help

Building pre_process:

  $ cmake --build /Users/spencer/Downloads/MFC/build/pre_process --target pre_process -j 8 --config Release

[0/2] Re-checking globbed directories...
ninja: no work to do.

  $ cmake --install /Users/spencer/Downloads/MFC/build/pre_process

-- Install configuration: "Release"
-- Up-to-date: /Users/spencer/Downloads/MFC/build/install/bin/pre_process

Building simulation:

  $ cmake --build /Users/spencer/Downloads/MFC/build/simulation --target simulation -j 8 --config Release

[0/2] Re-checking globbed directories...
ninja: no work to do.

  $ cmake --install /Users/spencer/Downloads/MFC/build/simulation

-- Install configuration: "Release"
-- Up-to-date: /Users/spencer/Downloads/MFC/build/install/bin/simulation

Test | from D79C3E6F to BDD3411B (142 tests)

   tests/UUID    Summary

    3AE495F4    1D -> bc=-5
    C5B79059    1D -> bc=-9
    70DAE9E8    1D -> bc=-4
    D79C3E6F    1D -> bc=-1
    48CCE072    1D -> bc=-7
    5EC236F2    1D -> bc=-6
    AED93D34    1D -> bc=-8
    8A59E8E6    1D -> bc=-2
    727F72ED    1D -> bc=-10
    A60691E7    1D -> bc=-11
    3FC6FC4A    1D -> bc=-12
    2AB32975    1D -> bc=-3
    B3C85904    1D -> weno_order=3 -> mapped_weno=F -> mp_weno=F
    7077C99F    1D -> weno_order=3 -> mapped_weno=T -> mp_weno=F
    84017671    1D -> weno_order=5 -> mapped_weno=F -> mp_weno=F
    F5890628    1D -> weno_order=5 -> mapped_weno=T -> mp_weno=F
    34580912    1D -> weno_order=5 -> mapped_weno=F -> mp_weno=T
    5527832F    1D -> 1 Fluid(s) -> riemann_solver=1 -> mixture_err
    4AEF478A    1D -> 1 Fluid(s) -> riemann_solver=1 -> avg_state=1
    32D0F235    1D -> 1 Fluid(s) -> riemann_solver=1 -> wave_speeds=2
    18BDCBC8    1D -> 1 Fluid(s) -> riemann_solver=2 -> mixture_err
    F97573DB    1D -> 1 Fluid(s) -> riemann_solver=2 -> avg_state=1
    F4F6AC27    1D -> 1 Fluid(s) -> riemann_solver=2 -> wave_speeds=2
    2F35A1FE    1D -> 1 Fluid(s) -> riemann_solver=2 -> model_eqns=3
    1E738705    1D -> 2 Fluid(s) -> riemann_solver=1 -> mixture_err
    0879E062    1D -> 2 Fluid(s) -> riemann_solver=1 -> avg_state=1
    83EFC30C    1D -> 2 Fluid(s) -> riemann_solver=1 -> wave_speeds=2
    1CCA82F5    1D -> 2 Fluid(s) -> riemann_solver=1 -> mpp_lim
    3A8359F6    1D -> 2 Fluid(s) -> riemann_solver=2 -> mixture_err
    6D24B115    1D -> 2 Fluid(s) -> riemann_solver=2 -> avg_state=1
    461DCB09    1D -> 2 Fluid(s) -> riemann_solver=2 -> wave_speeds=2
    FD891191    1D -> 2 Fluid(s) -> riemann_solver=2 -> model_eqns=3
    9DAC4DDC    1D -> 2 Fluid(s) -> riemann_solver=2 -> alt_soundspeed
    C4907722    1D -> 2 Fluid(s) -> riemann_solver=2 -> mpp_lim
    C79E1D3C    1D -> 2 Fluid(s) -> Viscous
    CD9D3050    1D -> 2 Fluid(s) -> Viscous -> weno_Re_flux
    0FCCE9F1    1D -> 2 MPI Ranks
    EF54219C    1D -> Bubbles -> Monopole -> Polytropic -> bubble_model=3
    7FC6826B    1D -> Bubbles -> Monopole -> Polytropic -> bubble_model=2
    6B22A317    1D -> Bubbles -> Monopole -> bubble_model=2
    59D05DE9    1D -> Bubbles -> Monopole -> nb=1
    9EB947DB    1D -> Hypoelasticity -> 1 Fluid(s)
    AF0BCEE4    1D -> Hypoelasticity -> 2 Fluid(s)
    AF46C382    1D -> Bubbles -> Monopole -> QBMM -> bubble_model=3
    55533234    2D -> bc=-1
    EAA53889    2D -> bc=-2
    46AA7AF8    1D -> Bubbles -> Monopole -> QBMM
    20AE0551    2D -> bc=-4
    A6E65782    2D -> bc=-5
    4129A23A    2D -> bc=-6
    E84967E7    2D -> bc=-7
    5F877BC9    2D -> bc=-8
    16C03D8E    2D -> bc=-9
    B96AC58F    2D -> bc=-10
    8FDEE23A    2D -> bc=-11
    BF46F657    2D -> bc=-12
    D972BA0F    2D -> bc=-3
    E4EFEDB2    2D -> weno_order=3 -> mapped_weno=F -> mp_weno=F
    CD3D9660    2D -> weno_order=3 -> mapped_weno=T -> mp_weno=F
    3974AC7B    2D -> weno_order=5 -> mapped_weno=F -> mp_weno=F
    C04741B4    2D -> weno_order=5 -> mapped_weno=T -> mp_weno=F
    E76D41CE    2D -> weno_order=5 -> mapped_weno=F -> mp_weno=T
    7374E266    2D -> 1 Fluid(s) -> riemann_solver=1 -> mixture_err
    3BFEAC19    2D -> 1 Fluid(s) -> riemann_solver=1 -> avg_state=1
    FBF808BE    2D -> 1 Fluid(s) -> riemann_solver=1 -> wave_speeds=2
  [36m[m: Entering the Python virtual environment (venv).
        ___            ___          ___
       /__/\          /  /\        /  /\       spencer@shb-m1pro.attlocal.net [Darwin]
      |  |::\        /  /:/_      /  /:/       ---------------------------------------
      |  |:|:\      /  /:/ /\    /  /:/
    __|__|:|\:\    /  /:/ /:/   /  /:/  ___
   /__/::::| \:\  /__/:/ /:/   /__/:/  /  /\   --jobs:    1
   \  \:\~~\__\/  \  \:\/:/    \  \:\ /  /:/   --mode:    release-cpu
    \  \:\         \  \::/      \  \:\  /:/    --targets: pre_process and simulation
     \  \:\         \  \:\       \  \:\/:/
      \  \:\         \  \:\       \  \::/
       \__\/          \__\/        \__\/       $ ./mfc.sh  --help

  Run
    Acquiring /Users/spencer/Downloads/MFC/tests/043B535A/case.py...
    Configuration:
      Input               /Users/spencer/Downloads/MFC/tests/043B535A/case.py
      Job Name      (-#)  unnamed
      Engine        (-e)  interactive
      Nodes         (-N)  1
      CPUs (/node)  (-n)  1
      GPUs (/node)  (-g)  0
      MPI Binary    (-b)  mpirun

    Running pre_process:
      Ensuring the Interactive Engine works (10s timeout):

  $ mpirun -np 1 hostname



  Error: The Interactive Engine appears to hang or exit with a non-zero status code. This may indicate that the wrong MPI binary is being used
to
  launch parallel jobs. You can specify the correct one for your system using the <-b,--binary> option. For example:
   - ./mfc.sh run <myfile.py> -b mpirun
   - ./mfc.sh run <myfile.py> -b srun
  Reason: Exit code.

  Terminated: 15
  [36m[m: Exiting the Python virtual environment.
  /opt/homebrew/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/resource_tracker.py:224:
UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
    warnings.warn('resource_tracker: There appear to be %d '

    3B414AF0    2D -> 1 Fluid(s) -> riemann_solver=2 -> mixture_err


Error: Test tests/043B535A: 2D -> 1 Fluid(s) -> riemann_solver=2 -> model_eqns=3: Failed to execute MFC. You can find the run's output in
/Users/spencer/Downloads/MFC/tests/043B535A/out.txt, and the case dictionary in /Users/spencer/Downloads/MFC/tests/043B535A/case.py.

Terminated: 15
[mfc.sh]: Exiting the Python virtual environment.
[I]shb-m1pro: Downloads/MFC $
[I]shb-m1pro: Downloads/MFC $ mpif90
gfortran: fatal error: no input files
compilation terminated.
[I]shb-m1pro: Downloads/MFC $ mpif90 --version
GNU Fortran (Homebrew GCC 12.2.0) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

above fails but below works

[I]shb-m1pro: Downloads/MFC $ ./mfc.sh test -j 4
[mfc.sh]: Entering the Python virtual environment (venv).
      ___            ___          ___
     /__/\          /  /\        /  /\       spencer@shb-m1pro.attlocal.net [Darwin]
    |  |::\        /  /:/_      /  /:/       ---------------------------------------
    |  |:|:\      /  /:/ /\    /  /:/
  __|__|:|\:\    /  /:/ /:/   /  /:/  ___
 /__/::::| \:\  /__/:/ /:/   /__/:/  /  /\   --jobs:    4
 \  \:\~~\__\/  \  \:\/:/    \  \:\ /  /:/   --mode:    release-cpu
  \  \:\         \  \::/      \  \:\  /:/
   \  \:\         \  \:\       \  \:\/:/
    \  \:\         \  \:\       \  \::/
     \__\/          \__\/        \__\/       $ ./mfc.sh [build, run, test, clean] --help

Building pre_process:

  $ cmake --build /Users/spencer/Downloads/MFC/build/pre_process --target pre_process -j 4 --config Release

[0/2] Re-checking globbed directories...
ninja: no work to do.

  $ cmake --install /Users/spencer/Downloads/MFC/build/pre_process

-- Install configuration: "Release"
-- Up-to-date: /Users/spencer/Downloads/MFC/build/install/bin/pre_process

Building simulation:

  $ cmake --build /Users/spencer/Downloads/MFC/build/simulation --target simulation -j 4 --config Release

[0/2] Re-checking globbed directories...
ninja: no work to do.

  $ cmake --install /Users/spencer/Downloads/MFC/build/simulation

-- Install configuration: "Release"
-- Up-to-date: /Users/spencer/Downloads/MFC/build/install/bin/simulation

Test | from D79C3E6F to BDD3411B (142 tests)

   tests/UUID    Summary

    3AE495F4    1D -> bc=-5
    70DAE9E8    1D -> bc=-4
    8A59E8E6    1D -> bc=-2
    D79C3E6F    1D -> bc=-1
    5EC236F2    1D -> bc=-6
    48CCE072    1D -> bc=-7
    AED93D34    1D -> bc=-8
    C5B79059    1D -> bc=-9
    727F72ED    1D -> bc=-10
    A60691E7    1D -> bc=-11
    3FC6FC4A    1D -> bc=-12
    2AB32975    1D -> bc=-3
    B3C85904    1D -> weno_order=3 -> mapped_weno=F -> mp_weno=F
    7077C99F    1D -> weno_order=3 -> mapped_weno=T -> mp_weno=F
    84017671    1D -> weno_order=5 -> mapped_weno=F -> mp_weno=F
    F5890628    1D -> weno_order=5 -> mapped_weno=T -> mp_weno=F
    34580912    1D -> weno_order=5 -> mapped_weno=F -> mp_weno=T
    5527832F    1D -> 1 Fluid(s) -> riemann_solver=1 -> mixture_err
    4AEF478A    1D -> 1 Fluid(s) -> riemann_solver=1 -> avg_state=1
    32D0F235    1D -> 1 Fluid(s) -> riemann_solver=1 -> wave_speeds=2
    18BDCBC8    1D -> 1 Fluid(s) -> riemann_solver=2 -> mixture_err
    F97573DB    1D -> 1 Fluid(s) -> riemann_solver=2 -> avg_state=1
    F4F6AC27    1D -> 1 Fluid(s) -> riemann_solver=2 -> wave_speeds=2
    2F35A1FE    1D -> 1 Fluid(s) -> riemann_solver=2 -> model_eqns=3
    1E738705    1D -> 2 Fluid(s) -> riemann_solver=1 -> mixture_err
    0879E062    1D -> 2 Fluid(s) -> riemann_solver=1 -> avg_state=1
    83EFC30C    1D -> 2 Fluid(s) -> riemann_solver=1 -> wave_speeds=2
    1CCA82F5    1D -> 2 Fluid(s) -> riemann_solver=1 -> mpp_lim
    3A8359F6    1D -> 2 Fluid(s) -> riemann_solver=2 -> mixture_err
    6D24B115    1D -> 2 Fluid(s) -> riemann_solver=2 -> avg_state=1
    461DCB09    1D -> 2 Fluid(s) -> riemann_solver=2 -> wave_speeds=2
    FD891191    1D -> 2 Fluid(s) -> riemann_solver=2 -> model_eqns=3
    9DAC4DDC    1D -> 2 Fluid(s) -> riemann_solver=2 -> alt_soundspeed
    C4907722    1D -> 2 Fluid(s) -> riemann_solver=2 -> mpp_lim
    C79E1D3C    1D -> 2 Fluid(s) -> Viscous
    CD9D3050    1D -> 2 Fluid(s) -> Viscous -> weno_Re_flux
    0FCCE9F1    1D -> 2 MPI Ranks
    EF54219C    1D -> Bubbles -> Monopole -> Polytropic -> bubble_model=3
    7FC6826B    1D -> Bubbles -> Monopole -> Polytropic -> bubble_model=2
    59D05DE9    1D -> Bubbles -> Monopole -> nb=1
    6B22A317    1D -> Bubbles -> Monopole -> bubble_model=2
    AF46C382    1D -> Bubbles -> Monopole -> QBMM -> bubble_model=3
    46AA7AF8    1D -> Bubbles -> Monopole -> QBMM
    9EB947DB    1D -> Hypoelasticity -> 1 Fluid(s)
    AF0BCEE4    1D -> Hypoelasticity -> 2 Fluid(s)
    55533234    2D -> bc=-1
    EAA53889    2D -> bc=-2
    20AE0551    2D -> bc=-4
    A6E65782    2D -> bc=-5
    4129A23A    2D -> bc=-6
    E84967E7    2D -> bc=-7
    5F877BC9    2D -> bc=-8
    16C03D8E    2D -> bc=-9
    B96AC58F    2D -> bc=-10
    8FDEE23A    2D -> bc=-11
    BF46F657    2D -> bc=-12
    D972BA0F    2D -> bc=-3
    E4EFEDB2    2D -> weno_order=3 -> mapped_weno=F -> mp_weno=F
    CD3D9660    2D -> weno_order=3 -> mapped_weno=T -> mp_weno=F
    3974AC7B    2D -> weno_order=5 -> mapped_weno=F -> mp_weno=F
    C04741B4    2D -> weno_order=5 -> mapped_weno=T -> mp_weno=F
    E76D41CE    2D -> weno_order=5 -> mapped_weno=F -> mp_weno=T
    7374E266    2D -> 1 Fluid(s) -> riemann_solver=1 -> mixture_err
    3BFEAC19    2D -> 1 Fluid(s) -> riemann_solver=1 -> avg_state=1
    FBF808BE    2D -> 1 Fluid(s) -> riemann_solver=1 -> wave_speeds=2
    3B414AF0    2D -> 1 Fluid(s) -> riemann_solver=2 -> mixture_err
    3C00B89D    2D -> 1 Fluid(s) -> riemann_solver=2 -> avg_state=1
    345A94C0    2D -> 1 Fluid(s) -> riemann_solver=2 -> wave_speeds=2
    043B535A    2D -> 1 Fluid(s) -> riemann_solver=2 -> model_eqns=3
    16FBF4C8    2D -> 2 Fluid(s) -> riemann_solver=1 -> mixture_err
    DC9CB97E    2D -> 2 Fluid(s) -> riemann_solver=1 -> avg_state=1
    A5C93D62    2D -> 2 Fluid(s) -> riemann_solver=1 -> wave_speeds=2
    A6AC2E06    2D -> 2 Fluid(s) -> riemann_solver=1 -> mpp_lim
    5781A4C2    2D -> 2 Fluid(s) -> riemann_solver=2 -> mixture_err
    645A26E3    2D -> 2 Fluid(s) -> riemann_solver=2 -> avg_state=1
    FC4D07B6    2D -> 2 Fluid(s) -> riemann_solver=2 -> wave_speeds=2
    4F2F4ACE    2D -> 2 Fluid(s) -> riemann_solver=2 -> model_eqns=3
    5DAB50B2    2D -> 2 Fluid(s) -> riemann_solver=2 -> alt_soundspeed
    F0F175B2    2D -> 2 Fluid(s) -> riemann_solver=2 -> mpp_lim
    9CB03CEF    2D -> 2 Fluid(s) -> Viscous
    D6BAC936    2D -> 2 Fluid(s) -> Viscous -> weno_Re_flux
    DB670E50    2D -> Axisymmetric -> model_eqns=2
    B89B8C70    2D -> Axisymmetric -> model_eqns=3
    FB822062    2D -> Axisymmetric -> Viscous
    8C7AA13B    2D -> 2 MPI Ranks
    B3AAC9C8    2D -> Axisymmetric -> Viscous -> weno_Re_flux
    34DBFE14    2D -> Bubbles -> Monopole -> Polytropic -> bubble_model=3
    AE37D842    2D -> Bubbles -> Monopole -> nb=1
    14B6198D    2D -> Bubbles -> Monopole -> Polytropic -> bubble_model=2
    CC4F7C44    2D -> Bubbles -> Monopole -> bubble_model=2
    122713AA    2D -> Hypoelasticity -> 1 Fluid(s)
    5281BD7B    2D -> Hypoelasticity -> 2 Fluid(s)
    66CFF8CC    2D -> Bubbles -> Monopole -> QBMM -> bubble_model=3
    6FC6A809    3D -> bc=-1
    09DAFEBA    3D -> bc=-2
    303B925A    2D -> Bubbles -> Monopole -> QBMM
    F99FBB36    3D -> bc=-4
    E09A12D9    3D -> bc=-5
    5010B814    3D -> bc=-6
    730DFD6D    3D -> bc=-7
    ABAC3AE3    3D -> bc=-8
    C93BE9B5    3D -> bc=-9
    D0045756    3D -> bc=-10
    557FF170    3D -> bc=-11
    61FFF3D3    3D -> bc=-12
    6B4B738B    3D -> bc=-3
    E1352143    3D -> weno_order=3 -> mapped_weno=F -> mp_weno=F
    13DFC31D    3D -> weno_order=3 -> mapped_weno=T -> mp_weno=F
    728A2A5B    3D -> weno_order=5 -> mapped_weno=F -> mp_weno=F
    42B169F5    3D -> weno_order=5 -> mapped_weno=F -> mp_weno=T
    19E33853    3D -> weno_order=5 -> mapped_weno=T -> mp_weno=F
    9ACD5174    3D -> 1 Fluid(s) -> riemann_solver=1 -> mixture_err
    73B0539E    3D -> 1 Fluid(s) -> riemann_solver=1 -> avg_state=1
    2A523AC1    3D -> 1 Fluid(s) -> riemann_solver=1 -> wave_speeds=2
    C06849AD    3D -> 1 Fluid(s) -> riemann_solver=2 -> mixture_err
    AB0BE4E4    3D -> 1 Fluid(s) -> riemann_solver=2 -> avg_state=1
    C36F18FB    3D -> 1 Fluid(s) -> riemann_solver=2 -> wave_speeds=2
    6241177B    3D -> 1 Fluid(s) -> riemann_solver=2 -> model_eqns=3
    C4A2FAA3    3D -> 2 Fluid(s) -> riemann_solver=1 -> mixture_err
    851F7AE2    3D -> 2 Fluid(s) -> riemann_solver=1 -> avg_state=1
    BD8004FF    3D -> 2 Fluid(s) -> riemann_solver=1 -> wave_speeds=2
    758D0268    3D -> 2 Fluid(s) -> riemann_solver=1 -> mpp_lim
    AACF1BC5    3D -> 2 Fluid(s) -> riemann_solver=2 -> mixture_err
    B33E256A    3D -> 2 Fluid(s) -> riemann_solver=2 -> avg_state=1
    B8F5F1C8    3D -> 2 Fluid(s) -> riemann_solver=2 -> wave_speeds=2
    7C8F1BA9    3D -> 2 Fluid(s) -> riemann_solver=2 -> alt_soundspeed
    F0E6771E    3D -> 2 Fluid(s) -> riemann_solver=2 -> model_eqns=3
    A0B82851    3D -> 2 Fluid(s) -> riemann_solver=2 -> mpp_lim
    1C0780C8    3D -> 2 Fluid(s) -> Viscous
    301B9153    3D -> Cylindrical -> model_eqns=2
    2060F55A    3D -> 2 Fluid(s) -> Viscous -> weno_Re_flux
    07C33719    3D -> Cylindrical -> Viscous
    CE232828    3D -> 2 MPI Ranks
    939D6718    3D -> Cylindrical -> Viscous -> weno_Re_flux
    36256906    3D -> Bubbles -> Monopole -> Polytropic -> bubble_model=3
    8A341282    3D -> Bubbles -> Monopole -> nb=1
    AD63A4A5    3D -> Bubbles -> Monopole -> Polytropic -> bubble_model=2
    622DEC78    3D -> Bubbles -> Monopole -> bubble_model=2
    7EFBCDAE    3D -> Hypoelasticity -> 1 Fluid(s)
    BDD3411B    3D -> Hypoelasticity -> 2 Fluid(s)
    63850240    3D -> Bubbles -> Monopole -> QBMM -> bubble_model=3
    AB04C64D    3D -> Bubbles -> Monopole -> QBMM

  Tested ✓
[mfc.sh]: Exiting the Python virtual environment.

[henryleberre]

I got my hands on an M1 MacBook pro for a while but couldn't reproduce this issue. Are you sure you aren't oversubscribing? Does it work if you try running the command (with -j 8 or perhaps -j 7) ? I don't see how we can be using more cores than specified with the -j argument. It might be an edge-case I'm not considering but I've never encountered it.

Also, our OpenMPI/Silo/GNU/AppleClang workaround appears to no longer be necessary. Should we keep build instructions as is or simply them?

[sbryngelson]

I can't replicate this anymore either. Not sure what was going on. Closing. Regarding the Silo workaround, please point me to what instructions you are talking about.

[henryleberre]

If my initial testing is to be believed, these instructions

MFC/doc/landing/building.md

Lines 79 to 117 in 0f38045

	**Note:** macOS remains the most difficult platform to consistently compile MFC on.
	If you run into issues, we suggest you try using Docker (instructions above).
	- **MacOS v10.15 (Catalina) or newer [ZSH]** (Verify with `echo $SHELL`)
	```console
	$ touch ~/.zshrc
	$ open ~/.zshrc
	```
	- **Older than MacOS v10.15 (Catalina) [BASH]** (Verify with `echo $SHELL`)
	```console
	$ touch ~/.bash_profile
	$ open ~/.bash_profile
	```
	An editor should open. Please paste the following lines into it before saving the file. If you wish to use a version of GNU's GCC other than 11, modify the first assignment. These lines ensure that LLVM's Clang, and Apple's modified version of GCC, won't be used to compile MFC. Further reading on `open-mpi` incompatibility with `clang`-based `gcc` on macOS: [here](https://stackoverflow.com/questions/27930481/how-to-build-openmpi-with-homebrew-and-gcc-4-9). We do *not* support `clang` due to conflicts with our Silo dependency.
	```console
	# === MFC MPI Installation ===
	export MFC_GCC_VER=11
	export OMPI_MPICC=gcc-$MFC_GCC_VER
	export OMPI_CXX=g++-$MFC_GCC_VER
	export OMPI_FC=gfortran-$MFC_GCC_VER
	export CC=gcc-$MFC_GCC_VER
	export CXX=g++-$MFC_GCC_VER
	export FC=gfortran-$MFC_GCC_VER
	# === MFC MPI Installation ===
	```
	**Close the open editor and terminal window**. Open a **new terminal** window before executing the commands bellow.
	```console
	$ brew install wget make python make cmake coreutils gcc@$MFC_GCC_VER
	$ HOMEBREW_MAKE_JOBS=$(nproc) brew install --cc=gcc-$MFC_GCC_VER --verbose --build-from-source open-mpi
	```
	They will download the dependencies MFC requires to build itself. `open-mpi` will be compiled from source, using the version of GCC we specified above with the environment variables `HOMEBREW_CC` and `HOMEBREW_CXX`. Building this package might take a while.

could be replaced with:

$ brew install wget make python make cmake coreutils gcc openmpi

This might only work on more recent versions of macOS. It still built fine with AppleClang. I would have to check. The current instructions are known always work so it might not be worth it to replace them, given the uncertainty.

We could consider adding prebuilt binaries to our GitHub releases for common operating systems (Windows/macOS/Linux) and architectures (x86/arm) at least for the CPU versions. In this case, I assume we could statically link to our fftw/hdf5/silo dependencies, but dynamically link to MPI.

[sbryngelson]

I would rather have the more verbose option for now.