Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mpi4py installation #172

Closed
jasmainak opened this issue Sep 12, 2020 · 9 comments · Fixed by #188
Closed

mpi4py installation #172

jasmainak opened this issue Sep 12, 2020 · 9 comments · Fixed by #188
Labels
docs Extra attention is needed parallelization This doesn't seem right

Comments

@jasmainak
Copy link
Collaborator

jasmainak commented Sep 12, 2020 •
edited

I have been struggling with mpi4py installation in #164. I followed all the instructions here and ran the following command:

$ mpiexec -np 2 nrniv -mpi -python -c 'from neuron import h; from mpi4py import MPI; \
                                       print("Hello from proc %d" % MPI.COMM_WORLD.Get_rank()); \
                                           h.quit()'

which runs as expected and gives:

numprocs=2
NEURON -- VERSION 7.8.1-1-gda13bb7c HEAD (da13bb7c) 2020-07-22
Duke, Yale, and the BlueBrain Project -- Copyright 1984-2019
See http://neuron.yale.edu/neuron/credits

Hello from proc 0
Hello from proc 1

However, when I run the test:

$ pytest ./hnn_core/tests/test_compare_hnn.py::test_mpi --pdb

I get:

hnn_core/tests/test_compare_hnn.py::test_mpi
  /Users/mainak/Documents/github_repos/hnn-core/hnn_core/parallel_backends.py:186: UserWarning: mpi4py not installed. will run on single processor
    warn('mpi4py not installed. will run on single processor')

Also I think when mpi4py is not installed, the test should be skipped instead of passing.
@jasmainak jasmainak added parallelization This doesn't seem right docs Extra attention is needed labels Sep 12, 2020
@klankinen
Copy link
Contributor

I am not able to install mpi4py either.

@blakecaldwell
Copy link
Member

blakecaldwell commented Sep 19, 2020 •
edited

@jasmainak mpi4py is different than the MPI installation on your system so there is no connection between the mpiexec test succeeding and mpi4py not being installed. The check in parallel_backends.py just does an import mpi4py. If you debug this as with any python module, can you find what happened to the mpi4py that you installed per the instructions?

@blakecaldwell
Copy link
Member

Also I think when mpi4py is not installed, the test should be skipped instead of passing.

Yes, indeed. Thanks for catching this.

@blakecaldwell
Copy link
Member

I am not able to install mpi4py either.

@jasmainak @klankinen I don't think the conda version of mpi4py is very good. I found some problems with it causing a seg fault on my system with Python 3.7.9

Can you remove the conda version and install it with pip? The PyPi version was having trouble installing on mac several months back. However, now it seems working again.

Yes @jasmainak this is not great stability. I think Python iterations outpace the mpi4py verification.

blakecaldwell pushed a commit to blakecaldwell/hnn-core that referenced this issue Sep 19, 2020
DOC: update mpi4py installation on mac
cd76666 
Fixes jonescompneurolab#172
@jasmainak
Copy link
Collaborator Author

I removed the conda install and tried with pip but I get a scary error:

(testenv) PHS008348:figures mainak$ pip install --upgrade mpi4py
Collecting mpi4py
  Using cached mpi4py-3.0.3.tar.gz (1.4 MB)
Building wheels for collected packages: mpi4py
  Building wheel for mpi4py (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/mainak/anaconda3/envs/testenv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/vy/v2l8b_011rn1nt1nrx8ffb_00000gp/T/pip-install-y7nqbqak/mpi4py/setup.py'"'"'; __file__='"'"'/private/var/folders/vy/v2l8b_011rn1nt1nrx8ffb_00000gp/T/pip-install-y7nqbqak/mpi4py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/vy/v2l8b_011rn1nt1nrx8ffb_00000gp/T/pip-wheel-6y_gnret
       cwd: /private/var/folders/vy/v2l8b_011rn1nt1nrx8ffb_00000gp/T/pip-install-y7nqbqak/mpi4py/
  Complete output (206 lines):
  running bdist_wheel
  running build
  running build_src
  running build_py
  creating build
  creating build/lib.macosx-10.9-x86_64-3.7
  creating build/lib.macosx-10.9-x86_64-3.7/mpi4py
  copying src/mpi4py/run.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py
  copying src/mpi4py/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py
  copying src/mpi4py/bench.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py
  copying src/mpi4py/__main__.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py
  creating build/lib.macosx-10.9-x86_64-3.7/mpi4py/futures
  copying src/mpi4py/futures/_base.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/futures
  copying src/mpi4py/futures/server.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/futures
  copying src/mpi4py/futures/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/futures
  copying src/mpi4py/futures/pool.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/futures
  copying src/mpi4py/futures/aplus.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/futures
  copying src/mpi4py/futures/__main__.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/futures
  copying src/mpi4py/futures/_lib.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/futures
  copying src/mpi4py/__init__.pxd -> build/lib.macosx-10.9-x86_64-3.7/mpi4py
  copying src/mpi4py/libmpi.pxd -> build/lib.macosx-10.9-x86_64-3.7/mpi4py
  copying src/mpi4py/MPI.pxd -> build/lib.macosx-10.9-x86_64-3.7/mpi4py
  creating build/lib.macosx-10.9-x86_64-3.7/mpi4py/include
  creating build/lib.macosx-10.9-x86_64-3.7/mpi4py/include/mpi4py
  copying src/mpi4py/include/mpi4py/mpi4py.MPI.h -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/include/mpi4py
  copying src/mpi4py/include/mpi4py/mpi4py.MPI_api.h -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/include/mpi4py
  copying src/mpi4py/include/mpi4py/mpi4py.h -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/include/mpi4py
  copying src/mpi4py/include/mpi4py/mpi4py.i -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/include/mpi4py
  copying src/mpi4py/include/mpi4py/mpi.pxi -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/include/mpi4py
  running build_clib
  MPI configuration: [mpi] from 'mpi.cfg'
  MPI C compiler:    /Users/mainak/anaconda3/envs/testenv/bin/mpicc
  MPI C++ compiler:  /Users/mainak/anaconda3/envs/testenv/bin/mpicxx
  MPI F compiler:    /Users/mainak/anaconda3/envs/testenv/bin/mpifort
  MPI F90 compiler:  /Users/mainak/anaconda3/envs/testenv/bin/mpif90
  MPI F77 compiler:  /Users/mainak/anaconda3/envs/testenv/bin/mpif77
  checking for library 'lmpe' ...
  /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c _configtest.c -o _configtest.o
  --------------------------------------------------------------------------
  The Open MPI wrapper compiler was unable to find the specified compiler
  x86_64-apple-darwin13.4.0-clang in your PATH.
  
  Note that this compiler was either specified at configure time or in
  one of several possible environment variables.
  --------------------------------------------------------------------------
  failure.
  removing: _configtest.c _configtest.o
  building 'mpe' dylib library
  creating build/temp.macosx-10.9-x86_64-3.7
  creating build/temp.macosx-10.9-x86_64-3.7/src
  creating build/temp.macosx-10.9-x86_64-3.7/src/lib-pmpi
  /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c src/lib-pmpi/mpe.c -o build/temp.macosx-10.9-x86_64-3.7/src/lib-pmpi/mpe.o
  --------------------------------------------------------------------------
  The Open MPI wrapper compiler was unable to find the specified compiler
  x86_64-apple-darwin13.4.0-clang in your PATH.
  
  Note that this compiler was either specified at configure time or in
  one of several possible environment variables.
  --------------------------------------------------------------------------
  warning: build_clib: command '/Users/mainak/anaconda3/envs/testenv/bin/mpicc' failed with exit status 1
  
  warning: build_clib: building optional library "mpe" failed
  
  checking for library 'vt-mpi' ...
  /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c _configtest.c -o _configtest.o
  --------------------------------------------------------------------------
  The Open MPI wrapper compiler was unable to find the specified compiler
  x86_64-apple-darwin13.4.0-clang in your PATH.
  
  Note that this compiler was either specified at configure time or in
  one of several possible environment variables.
  --------------------------------------------------------------------------
  failure.
  removing: _configtest.c _configtest.o
  checking for library 'vt.mpi' ...
  /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c _configtest.c -o _configtest.o
  --------------------------------------------------------------------------
  The Open MPI wrapper compiler was unable to find the specified compiler
  x86_64-apple-darwin13.4.0-clang in your PATH.
  
  Note that this compiler was either specified at configure time or in
  one of several possible environment variables.
  --------------------------------------------------------------------------
  failure.
  removing: _configtest.c _configtest.o
  building 'vt' dylib library
  /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c src/lib-pmpi/vt.c -o build/temp.macosx-10.9-x86_64-3.7/src/lib-pmpi/vt.o
  --------------------------------------------------------------------------
  The Open MPI wrapper compiler was unable to find the specified compiler
  x86_64-apple-darwin13.4.0-clang in your PATH.
  
  Note that this compiler was either specified at configure time or in
  one of several possible environment variables.
  --------------------------------------------------------------------------
  warning: build_clib: command '/Users/mainak/anaconda3/envs/testenv/bin/mpicc' failed with exit status 1
  
  warning: build_clib: building optional library "vt" failed
  
  checking for library 'vt-mpi' ...
  /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c _configtest.c -o _configtest.o
  --------------------------------------------------------------------------
  The Open MPI wrapper compiler was unable to find the specified compiler
  x86_64-apple-darwin13.4.0-clang in your PATH.
  
  Note that this compiler was either specified at configure time or in
  one of several possible environment variables.
  --------------------------------------------------------------------------
  failure.
  removing: _configtest.c _configtest.o
  checking for library 'vt.mpi' ...
  /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c _configtest.c -o _configtest.o
  --------------------------------------------------------------------------
  The Open MPI wrapper compiler was unable to find the specified compiler
  x86_64-apple-darwin13.4.0-clang in your PATH.
  
  Note that this compiler was either specified at configure time or in
  one of several possible environment variables.
  --------------------------------------------------------------------------
  failure.
  removing: _configtest.c _configtest.o
  building 'vt-mpi' dylib library
  /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c src/lib-pmpi/vt-mpi.c -o build/temp.macosx-10.9-x86_64-3.7/src/lib-pmpi/vt-mpi.o
  --------------------------------------------------------------------------
  The Open MPI wrapper compiler was unable to find the specified compiler
  x86_64-apple-darwin13.4.0-clang in your PATH.
  
  Note that this compiler was either specified at configure time or in
  one of several possible environment variables.
  --------------------------------------------------------------------------
  warning: build_clib: command '/Users/mainak/anaconda3/envs/testenv/bin/mpicc' failed with exit status 1
  
  warning: build_clib: building optional library "vt-mpi" failed
  
  checking for library 'vt-hyb' ...
  /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c _configtest.c -o _configtest.o
  --------------------------------------------------------------------------
  The Open MPI wrapper compiler was unable to find the specified compiler
  x86_64-apple-darwin13.4.0-clang in your PATH.
  
  Note that this compiler was either specified at configure time or in
  one of several possible environment variables.
  --------------------------------------------------------------------------
  failure.
  removing: _configtest.c _configtest.o
  checking for library 'vt.ompi' ...
  /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c _configtest.c -o _configtest.o
  --------------------------------------------------------------------------
  The Open MPI wrapper compiler was unable to find the specified compiler
  x86_64-apple-darwin13.4.0-clang in your PATH.
  
  Note that this compiler was either specified at configure time or in
  one of several possible environment variables.
  --------------------------------------------------------------------------
  failure.
  removing: _configtest.c _configtest.o
  building 'vt-hyb' dylib library
  /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c src/lib-pmpi/vt-hyb.c -o build/temp.macosx-10.9-x86_64-3.7/src/lib-pmpi/vt-hyb.o
  --------------------------------------------------------------------------
  The Open MPI wrapper compiler was unable to find the specified compiler
  x86_64-apple-darwin13.4.0-clang in your PATH.
  
  Note that this compiler was either specified at configure time or in
  one of several possible environment variables.
  --------------------------------------------------------------------------
  warning: build_clib: command '/Users/mainak/anaconda3/envs/testenv/bin/mpicc' failed with exit status 1
  
  warning: build_clib: building optional library "vt-hyb" failed
  
  running build_ext
  MPI configuration: [mpi] from 'mpi.cfg'
  MPI C compiler:    /Users/mainak/anaconda3/envs/testenv/bin/mpicc
  MPI C++ compiler:  /Users/mainak/anaconda3/envs/testenv/bin/mpicxx
  MPI F compiler:    /Users/mainak/anaconda3/envs/testenv/bin/mpifort
  MPI F90 compiler:  /Users/mainak/anaconda3/envs/testenv/bin/mpif90
  MPI F77 compiler:  /Users/mainak/anaconda3/envs/testenv/bin/mpif77
  checking for dlopen() availability ...
  checking for header 'dlfcn.h' ...
  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include/python3.7m -c _configtest.c -o _configtest.o
  success!
  removing: _configtest.c _configtest.o
  success!
  checking for library 'dl' ...
  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include/python3.7m -c _configtest.c -o _configtest.o
  gcc -flat_namespace -undefined suppress -arch x86_64 _configtest.o -Lbuild/temp.macosx-10.9-x86_64-3.7 -ldl -o _configtest
  success!
  removing: _configtest.c _configtest.o _configtest
  checking for function 'dlopen' ...
  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include/python3.7m -c _configtest.c -o _configtest.o
  gcc -arch x86_64 _configtest.o -Lbuild/temp.macosx-10.9-x86_64-3.7 -ldl -o _configtest
  success!
  removing: _configtest.c _configtest.o _configtest
  building 'mpi4py.dl' extension
  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -DHAVE_DLFCN_H=1 -DHAVE_DLOPEN=1 -I/Users/mainak/anaconda3/envs/testenv/include/python3.7m -c src/dynload.c -o build/temp.macosx-10.9-x86_64-3.7/src/dynload.o
  gcc -bundle -undefined dynamic_lookup -L/Users/mainak/anaconda3/envs/testenv/lib -arch x86_64 -L/Users/mainak/anaconda3/envs/testenv/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.9-x86_64-3.7/src/dynload.o -Lbuild/temp.macosx-10.9-x86_64-3.7 -ldl -o build/lib.macosx-10.9-x86_64-3.7/mpi4py/dl.cpython-37m-darwin.so
  checking for MPI compile and link ...
  /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include/python3.7m -c _configtest.c -o _configtest.o
  --------------------------------------------------------------------------
  The Open MPI wrapper compiler was unable to find the specified compiler
  x86_64-apple-darwin13.4.0-clang in your PATH.
  
  Note that this compiler was either specified at configure time or in
  one of several possible environment variables.
  --------------------------------------------------------------------------
  failure.
  removing: _configtest.c _configtest.o
  error: Cannot compile MPI programs. Check your configuration!!!
  ----------------------------------------
  ERROR: Failed building wheel for mpi4py
  Running setup.py clean for mpi4py
Failed to build mpi4py
Installing collected packages: mpi4py
    Running setup.py install for mpi4py ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/mainak/anaconda3/envs/testenv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/vy/v2l8b_011rn1nt1nrx8ffb_00000gp/T/pip-install-y7nqbqak/mpi4py/setup.py'"'"'; __file__='"'"'/private/var/folders/vy/v2l8b_011rn1nt1nrx8ffb_00000gp/T/pip-install-y7nqbqak/mpi4py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/vy/v2l8b_011rn1nt1nrx8ffb_00000gp/T/pip-record-gnkszmk8/install-record.txt --single-version-externally-managed --compile --install-headers /Users/mainak/anaconda3/envs/testenv/include/python3.7m/mpi4py
         cwd: /private/var/folders/vy/v2l8b_011rn1nt1nrx8ffb_00000gp/T/pip-install-y7nqbqak/mpi4py/
    Complete output (206 lines):
    running install
    running build
    running build_src
    running build_py
    creating build
    creating build/lib.macosx-10.9-x86_64-3.7
    creating build/lib.macosx-10.9-x86_64-3.7/mpi4py
    copying src/mpi4py/run.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py
    copying src/mpi4py/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py
    copying src/mpi4py/bench.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py
    copying src/mpi4py/__main__.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py
    creating build/lib.macosx-10.9-x86_64-3.7/mpi4py/futures
    copying src/mpi4py/futures/_base.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/futures
    copying src/mpi4py/futures/server.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/futures
    copying src/mpi4py/futures/__init__.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/futures
    copying src/mpi4py/futures/pool.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/futures
    copying src/mpi4py/futures/aplus.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/futures
    copying src/mpi4py/futures/__main__.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/futures
    copying src/mpi4py/futures/_lib.py -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/futures
    copying src/mpi4py/__init__.pxd -> build/lib.macosx-10.9-x86_64-3.7/mpi4py
    copying src/mpi4py/libmpi.pxd -> build/lib.macosx-10.9-x86_64-3.7/mpi4py
    copying src/mpi4py/MPI.pxd -> build/lib.macosx-10.9-x86_64-3.7/mpi4py
    creating build/lib.macosx-10.9-x86_64-3.7/mpi4py/include
    creating build/lib.macosx-10.9-x86_64-3.7/mpi4py/include/mpi4py
    copying src/mpi4py/include/mpi4py/mpi4py.MPI.h -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/include/mpi4py
    copying src/mpi4py/include/mpi4py/mpi4py.MPI_api.h -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/include/mpi4py
    copying src/mpi4py/include/mpi4py/mpi4py.h -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/include/mpi4py
    copying src/mpi4py/include/mpi4py/mpi4py.i -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/include/mpi4py
    copying src/mpi4py/include/mpi4py/mpi.pxi -> build/lib.macosx-10.9-x86_64-3.7/mpi4py/include/mpi4py
    running build_clib
    MPI configuration: [mpi] from 'mpi.cfg'
    MPI C compiler:    /Users/mainak/anaconda3/envs/testenv/bin/mpicc
    MPI C++ compiler:  /Users/mainak/anaconda3/envs/testenv/bin/mpicxx
    MPI F compiler:    /Users/mainak/anaconda3/envs/testenv/bin/mpifort
    MPI F90 compiler:  /Users/mainak/anaconda3/envs/testenv/bin/mpif90
    MPI F77 compiler:  /Users/mainak/anaconda3/envs/testenv/bin/mpif77
    checking for library 'lmpe' ...
    /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c _configtest.c -o _configtest.o
    --------------------------------------------------------------------------
    The Open MPI wrapper compiler was unable to find the specified compiler
    x86_64-apple-darwin13.4.0-clang in your PATH.
    
    Note that this compiler was either specified at configure time or in
    one of several possible environment variables.
    --------------------------------------------------------------------------
    failure.
    removing: _configtest.c _configtest.o
    building 'mpe' dylib library
    creating build/temp.macosx-10.9-x86_64-3.7
    creating build/temp.macosx-10.9-x86_64-3.7/src
    creating build/temp.macosx-10.9-x86_64-3.7/src/lib-pmpi
    /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c src/lib-pmpi/mpe.c -o build/temp.macosx-10.9-x86_64-3.7/src/lib-pmpi/mpe.o
    --------------------------------------------------------------------------
    The Open MPI wrapper compiler was unable to find the specified compiler
    x86_64-apple-darwin13.4.0-clang in your PATH.
    
    Note that this compiler was either specified at configure time or in
    one of several possible environment variables.
    --------------------------------------------------------------------------
    warning: build_clib: command '/Users/mainak/anaconda3/envs/testenv/bin/mpicc' failed with exit status 1
    
    warning: build_clib: building optional library "mpe" failed
    
    checking for library 'vt-mpi' ...
    /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c _configtest.c -o _configtest.o
    --------------------------------------------------------------------------
    The Open MPI wrapper compiler was unable to find the specified compiler
    x86_64-apple-darwin13.4.0-clang in your PATH.
    
    Note that this compiler was either specified at configure time or in
    one of several possible environment variables.
    --------------------------------------------------------------------------
    failure.
    removing: _configtest.c _configtest.o
    checking for library 'vt.mpi' ...
    /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c _configtest.c -o _configtest.o
    --------------------------------------------------------------------------
    The Open MPI wrapper compiler was unable to find the specified compiler
    x86_64-apple-darwin13.4.0-clang in your PATH.
    
    Note that this compiler was either specified at configure time or in
    one of several possible environment variables.
    --------------------------------------------------------------------------
    failure.
    removing: _configtest.c _configtest.o
    building 'vt' dylib library
    /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c src/lib-pmpi/vt.c -o build/temp.macosx-10.9-x86_64-3.7/src/lib-pmpi/vt.o
    --------------------------------------------------------------------------
    The Open MPI wrapper compiler was unable to find the specified compiler
    x86_64-apple-darwin13.4.0-clang in your PATH.
    
    Note that this compiler was either specified at configure time or in
    one of several possible environment variables.
    --------------------------------------------------------------------------
    warning: build_clib: command '/Users/mainak/anaconda3/envs/testenv/bin/mpicc' failed with exit status 1
    
    warning: build_clib: building optional library "vt" failed
    
    checking for library 'vt-mpi' ...
    /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c _configtest.c -o _configtest.o
    --------------------------------------------------------------------------
    The Open MPI wrapper compiler was unable to find the specified compiler
    x86_64-apple-darwin13.4.0-clang in your PATH.
    
    Note that this compiler was either specified at configure time or in
    one of several possible environment variables.
    --------------------------------------------------------------------------
    failure.
    removing: _configtest.c _configtest.o
    checking for library 'vt.mpi' ...
    /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c _configtest.c -o _configtest.o
    --------------------------------------------------------------------------
    The Open MPI wrapper compiler was unable to find the specified compiler
    x86_64-apple-darwin13.4.0-clang in your PATH.
    
    Note that this compiler was either specified at configure time or in
    one of several possible environment variables.
    --------------------------------------------------------------------------
    failure.
    removing: _configtest.c _configtest.o
    building 'vt-mpi' dylib library
    /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c src/lib-pmpi/vt-mpi.c -o build/temp.macosx-10.9-x86_64-3.7/src/lib-pmpi/vt-mpi.o
    --------------------------------------------------------------------------
    The Open MPI wrapper compiler was unable to find the specified compiler
    x86_64-apple-darwin13.4.0-clang in your PATH.
    
    Note that this compiler was either specified at configure time or in
    one of several possible environment variables.
    --------------------------------------------------------------------------
    warning: build_clib: command '/Users/mainak/anaconda3/envs/testenv/bin/mpicc' failed with exit status 1
    
    warning: build_clib: building optional library "vt-mpi" failed
    
    checking for library 'vt-hyb' ...
    /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c _configtest.c -o _configtest.o
    --------------------------------------------------------------------------
    The Open MPI wrapper compiler was unable to find the specified compiler
    x86_64-apple-darwin13.4.0-clang in your PATH.
    
    Note that this compiler was either specified at configure time or in
    one of several possible environment variables.
    --------------------------------------------------------------------------
    failure.
    removing: _configtest.c _configtest.o
    checking for library 'vt.ompi' ...
    /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c _configtest.c -o _configtest.o
    --------------------------------------------------------------------------
    The Open MPI wrapper compiler was unable to find the specified compiler
    x86_64-apple-darwin13.4.0-clang in your PATH.
    
    Note that this compiler was either specified at configure time or in
    one of several possible environment variables.
    --------------------------------------------------------------------------
    failure.
    removing: _configtest.c _configtest.o
    building 'vt-hyb' dylib library
    /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -c src/lib-pmpi/vt-hyb.c -o build/temp.macosx-10.9-x86_64-3.7/src/lib-pmpi/vt-hyb.o
    --------------------------------------------------------------------------
    The Open MPI wrapper compiler was unable to find the specified compiler
    x86_64-apple-darwin13.4.0-clang in your PATH.
    
    Note that this compiler was either specified at configure time or in
    one of several possible environment variables.
    --------------------------------------------------------------------------
    warning: build_clib: command '/Users/mainak/anaconda3/envs/testenv/bin/mpicc' failed with exit status 1
    
    warning: build_clib: building optional library "vt-hyb" failed
    
    running build_ext
    MPI configuration: [mpi] from 'mpi.cfg'
    MPI C compiler:    /Users/mainak/anaconda3/envs/testenv/bin/mpicc
    MPI C++ compiler:  /Users/mainak/anaconda3/envs/testenv/bin/mpicxx
    MPI F compiler:    /Users/mainak/anaconda3/envs/testenv/bin/mpifort
    MPI F90 compiler:  /Users/mainak/anaconda3/envs/testenv/bin/mpif90
    MPI F77 compiler:  /Users/mainak/anaconda3/envs/testenv/bin/mpif77
    checking for dlopen() availability ...
    checking for header 'dlfcn.h' ...
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include/python3.7m -c _configtest.c -o _configtest.o
    success!
    removing: _configtest.c _configtest.o
    success!
    checking for library 'dl' ...
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include/python3.7m -c _configtest.c -o _configtest.o
    gcc -flat_namespace -undefined suppress -arch x86_64 _configtest.o -Lbuild/temp.macosx-10.9-x86_64-3.7 -ldl -o _configtest
    success!
    removing: _configtest.c _configtest.o _configtest
    checking for function 'dlopen' ...
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include/python3.7m -c _configtest.c -o _configtest.o
    gcc -arch x86_64 _configtest.o -Lbuild/temp.macosx-10.9-x86_64-3.7 -ldl -o _configtest
    success!
    removing: _configtest.c _configtest.o _configtest
    building 'mpi4py.dl' extension
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -DHAVE_DLFCN_H=1 -DHAVE_DLOPEN=1 -I/Users/mainak/anaconda3/envs/testenv/include/python3.7m -c src/dynload.c -o build/temp.macosx-10.9-x86_64-3.7/src/dynload.o
    gcc -bundle -undefined dynamic_lookup -L/Users/mainak/anaconda3/envs/testenv/lib -arch x86_64 -L/Users/mainak/anaconda3/envs/testenv/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.9-x86_64-3.7/src/dynload.o -Lbuild/temp.macosx-10.9-x86_64-3.7 -ldl -o build/lib.macosx-10.9-x86_64-3.7/mpi4py/dl.cpython-37m-darwin.so
    checking for MPI compile and link ...
    /Users/mainak/anaconda3/envs/testenv/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include -arch x86_64 -I/Users/mainak/anaconda3/envs/testenv/include/python3.7m -c _configtest.c -o _configtest.o
    --------------------------------------------------------------------------
    The Open MPI wrapper compiler was unable to find the specified compiler
    x86_64-apple-darwin13.4.0-clang in your PATH.
    
    Note that this compiler was either specified at configure time or in
    one of several possible environment variables.
    --------------------------------------------------------------------------
    failure.
    removing: _configtest.c _configtest.o
    error: Cannot compile MPI programs. Check your configuration!!!
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/mainak/anaconda3/envs/testenv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/vy/v2l8b_011rn1nt1nrx8ffb_00000gp/T/pip-install-y7nqbqak/mpi4py/setup.py'"'"'; __file__='"'"'/private/var/folders/vy/v2l8b_011rn1nt1nrx8ffb_00000gp/T/pip-install-y7nqbqak/mpi4py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/vy/v2l8b_011rn1nt1nrx8ffb_00000gp/T/pip-record-gnkszmk8/install-record.txt --single-version-externally-managed --compile --install-headers /Users/mainak/anaconda3/envs/testenv/include/python3.7m/mpi4py Check the logs for full command output.

@jasmainak
Copy link
Collaborator Author

I also tried with the "--upgrade" flag, no luck

@blakecaldwell
Copy link
Member

Uck. Now I remember. That's the reason I went with conda in the first place. It requires the mpicc compiler, which I conveniently happen to have installed on my laptop. Let me package it in a docker container... ;)

Back to conda... Did you figure out why it won't let you import it? Also, perhaps use a fresh conda env for this troubleshooting.

@jasmainak
Copy link
Collaborator Author

jasmainak commented Sep 20, 2020 •
edited

okay I figured out the issue finally ... conda was tripping me, had encountered this the other day while helping @kohl-carmen

If you make a new environment in conda like this:

$ conda create -n test_mpi python=3.8

the pytest from the other environment is still available in the path and it uses the python from the other environment. The fix was to update pytest in the new environment. I'm not sure if this is the best solution though. Might have to look around on stackoverflow. So it's not really related to MPI installation. Happy to close the issue. Thanks for helping out!

ps: there is a minor typo in the install instructions, it should be "--yes" not "yes" in the conda installation command.

@jasmainak
Copy link
Collaborator Author

@klankinen if you still face issues with mpi4py, we can look into it but I am pretty confident it will work

jasmainak pushed a commit that referenced this issue Sep 22, 2020
@jasmainak
DOC: update mpi4py installation on mac
2f542ce 
Fixes #172
@jasmainak jasmainak mentioned this issue Nov 29, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
docs Extra attention is needed parallelization This doesn't seem right
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

postprocessing kohl-carmen/hnn-core
3 participants
@blakecaldwell @jasmainak @klankinen