Simulations aborted on Mac #11

axiezai · 2018-09-20T19:58:45Z

Hello,

I just did a fresh install of HNN on a macOS High Sierra.

When I tried to run the Simulating Rhythmic Proximal Inputs: Alpha Only tutorial, the simulation instantly aborts and the terminal gives off the following message:

Starting simulation. . .

mpiexec noticed that the job aborted, but has no info as to the process
that caused that situation.

WARN: could not read simulation outputs: dict_values(['/Users/ashish/hnn/data/default/dpl.txt', '/Users/ashish/hnn/data/default/rawspec.npz', '/Users/ashish/hnn/data/default/spk.txt', '/Users/ashish/hnn/data/default/param.txt'])

Not really informative as the error message doesn't know what exactly caused the simulation to abort. Please let me know if you need any other info to solve this. Thank you!

samnemo · 2018-09-20T20:18:58Z

Robert - any suggestions?

…

On Thu, Sep 20, 2018 at 3:58 PM Xi He ***@***.***> wrote: Hello, I just did a fresh install of HNN on a macOS High Sierra. When I tried to run the Simulating Rhythmic Proximal Inputs: Alpha Only tutorial, the simulation instantly aborts and the terminal gives off the following message: ` Starting simulation. . . mpiexec noticed that the job aborted, but has no info as to the process that caused that situation. WARN: could not read simulation outputs: dict_values(['/Users/ashish/hnn/data/default/dpl.txt', '/Users/ashish/hnn/data/default/rawspec.npz', '/Users/ashish/hnn/data/default/spk.txt', '/Users/ashish/hnn/data/default/param.txt']) ` Not really informative as the error message doesn't know what exactly caused the simulation to abort. Please let me know if you need any other info to solve this. Thank you! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#11>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKXT6CMeL47EpXH-Z5rlEiQ4XrUkJsmVks5uc_N1gaJpZM4Wy1e8> .

ramcdougal · 2018-09-20T20:30:10Z

Do any of the tutorials work?

If not, does NEURON load successfully?

e.g. if you do

cd ~/hnn
python

and then in python try "import neuron", does it succeed or fail?

If it fails, try:

cd ~/hnn
rm -fr x86_64
nrnivmodl mod/

and then see if the previous test now succeeds or fails.

ramcdougal · 2018-09-20T20:31:04Z

One other thing to check:

Does typing "python" load Python 2 or Python 3? (It should say in the banner.)

axiezai · 2018-09-20T21:04:00Z

After some digging, it seems that the newer versions of MacOS doesn't include openMPI, which is required to run HNN simulations I assume? After installing I am now seeing the following:

Starting simulation. . .
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
[iMac-Pro.radiology.ucsf.edu:44424] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
[iMac-Pro.radiology.ucsf.edu:44424] OPAL ERROR: Not initialized in file pmix2x_client.c at line 109
[iMac-Pro.radiology.ucsf.edu:44425] OPAL ERROR: Not initialized in file pmix2x_client.c at line 109

does this help?

axiezai · 2018-09-20T21:05:23Z

hi @ramcdougal
"import neuron" works

Python version is 3.6.5

ramcdougal · 2018-09-21T15:31:22Z

Just to confirm: Are you using Anaconda Python?

I would have expected doing the conda install of mpi4py to install openmpi, but you can try removing the openmpi you installed and explicitly doing the conda install of openmpi:

sudo conda install -c mpi4py openmpi

(sudo is assuming you've installed for all users; if it's a local install, do not use "sudo").

axiezai · 2018-09-21T19:44:45Z

I did the conda install for mpi4py and openmpi, loaded the tutorial parameters, clicked run simulation and now i'm getting the following:

Starting simulation. . .

[iMac-Pro.radiology.ucsf.edu:46680] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 122
[iMac-Pro.radiology.ucsf.edu:46680] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46683] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 122

[iMac-Pro.radiology.ucsf.edu:46683] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 122
[iMac-Pro.radiology.ucsf.edu:46683] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46684] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 122
[iMac-Pro.radiology.ucsf.edu:46684] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46685] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 122

[iMac-Pro.radiology.ucsf.edu:46685] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 122
[iMac-Pro.radiology.ucsf.edu:46685] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46688] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 122

[iMac-Pro.radiology.ucsf.edu:46688] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 122
[iMac-Pro.radiology.ucsf.edu:46688] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46689] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 122
[iMac-Pro.radiology.ucsf.edu:46689] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46690] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 122

[iMac-Pro.radiology.ucsf.edu:46690] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 122
[iMac-Pro.radiology.ucsf.edu:46690] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46691] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 122

[iMac-Pro.radiology.ucsf.edu:46691] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 122
[iMac-Pro.radiology.ucsf.edu:46691] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46692] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 122
[iMac-Pro.radiology.ucsf.edu:46692] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46681] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46682] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46686] PMIX ERROR: UNPACK-PAST-END in file unpack.c at line 122
[iMac-Pro.radiology.ucsf.edu:46686] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46687] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46693] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46694] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46695] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46684] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46686] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46688] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46687] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
[iMac-Pro.radiology.ucsf.edu:46691] PMIX ERROR: SUCCESS in file event/pmix_event_registration.c at line 91
WARN: could not read simulation outputs: dict_values(['/Users/ashish/hnn/data/OnlyRhythmicProx/dpl.txt', '/Users/ashish/hnn/data/OnlyRhythmicProx/rawspec.npz', '/Users/ashish/hnn/data/OnlyRhythmicProx/spk.txt', '/Users/ashish/hnn/data/OnlyRhythmicProx/param.txt'])

nrnhines · 2018-09-24T18:57:25Z

To help diagnose. results of the following tests may be useful. Run tests in a terminal.

Does mpi work?
<
$ mpiexec -n 2 echo "hello"
hello
hello

does nrniv work without mpi and python
<
nrniv -nobanner -nopython -c '{print "hello" quit()}'
hello

do nrniv and mpi work together but without python
<
$ mpiexec -n 2 nrniv -nobanner -nopython -mpi -c '{print "hello" quit()}'
numprocs=2
hello
hello

can you launch nrniv and use python
<
$ nrniv -nobanner -python -c 'from neuron import h; print("Hello"); h.quit()'
Hello

do all three work when launching nrniv
<
$ mpiexec -n 2 nrniv -nobanner -mpi -python -c 'from neuron import h; pc = h.ParallelContext(); print ("I am %d of %d" % (int(pc.id()), int(pc.nhost()))); h.quit()'
numprocs=2
I am 0 of 2
I am 1 of 2

vogdb · 2018-11-21T16:24:58Z

@axiezai

I've recently installed HNN. I had the same problem as you. Additionally if you look View->View Simulation Log, you might see a more detailed message. In my case it was

Try loading openmpi
load_mpi: dlopen(libmpi.dylib, 10): image not found
Is openmpi installed? If not in default location, need a LD_LIBRARY_PATH.
could not dynamically load libmpi.so or libmpich2.so

To solve it I pointed LD_LIBRARY_PATH to the dir containing libmpi.dylib. I had a separate conda's env hnn for HNN, so I added the below line to my ~/.profile(don't forget to log out) and everything started working.

export LD_LIBRARY_PATH=~/anaconda2/envs/hnn/lib/

You also can use Mac's search utilities to find your libmpi.dylib location.

blakecaldwell · 2019-03-27T16:57:53Z

@vogdb thanks for the instructions above.

This error can come about in at least a couple of ways. Even if LD_LIBRARY_PATH is set to where libmpi.dylib exists, the error may not be resolved if libmpi.10.dylib does not exist. I needed to create a link:

cd $CONDA_PREFIX/lib
ln -s libmpi.1.dylib libmpi.10.dylib

I don't know where the 10 comes from. Perhaps @ramcdougal has some insight from the NEURON macOS installer?

axiezai · 2019-04-13T00:05:41Z

I'm glad @vogdb and @blakecaldwell provided some easy solutions!

To be honest, I don't have access to the Mac machine I did this install on anymore, it wasn't my Mac :(

So I can't provide more info on my end, apologies. If we can confirm thatextra 10 in libmpi.10.dylib isn't an issue, perhaps we can close this?

Thanks again for all the help and debugging! Really grateful :)

samnemo added bug Something isn't working question Further information is requested and removed bug Something isn't working labels Nov 15, 2018

blakecaldwell closed this as completed Aug 16, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Simulations aborted on Mac #11

Simulations aborted on Mac #11

axiezai commented Sep 20, 2018 •

edited

samnemo commented Sep 20, 2018 via email

ramcdougal commented Sep 20, 2018

ramcdougal commented Sep 20, 2018

axiezai commented Sep 20, 2018

axiezai commented Sep 20, 2018

ramcdougal commented Sep 21, 2018

axiezai commented Sep 21, 2018

nrnhines commented Sep 24, 2018 •

edited

vogdb commented Nov 21, 2018

blakecaldwell commented Mar 27, 2019

axiezai commented Apr 13, 2019

Simulations aborted on Mac #11

Simulations aborted on Mac #11

Comments

axiezai commented Sep 20, 2018 • edited

Starting simulation. . .

mpiexec noticed that the job aborted, but has no info as to the process that caused that situation.

samnemo commented Sep 20, 2018 via email

ramcdougal commented Sep 20, 2018

ramcdougal commented Sep 20, 2018

axiezai commented Sep 20, 2018

axiezai commented Sep 20, 2018

ramcdougal commented Sep 21, 2018

axiezai commented Sep 21, 2018

nrnhines commented Sep 24, 2018 • edited

vogdb commented Nov 21, 2018

blakecaldwell commented Mar 27, 2019

axiezai commented Apr 13, 2019

axiezai commented Sep 20, 2018 •

edited

mpiexec noticed that the job aborted, but has no info as to the process
that caused that situation.

nrnhines commented Sep 24, 2018 •

edited