Tool attach failure

[lee218llnl]

When using the latest OpenMPI commit (d782542) and trying to attach to an MPI job via LaunchMON (https://github.com/LLNL/LaunchMON) I get the following error:

--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 60 slots
that were requested by the application:
  /nfs/tmp2/lee218/prefix/stat-travis/bin/STATD

Either request fewer slots for your application, or make more slots available
for use.
--------------------------------------------------------------------------

I am on a system that allocated nodes via SLURM and this is on 2 nodes with 64 tasks. The application was launched with mpirun/orterun with 4 MPI processes, which might explain why it's trying to fill the node with 64-4=60 slots. However, it should only try launching 1 STATD daemon process per node. Let me know if there are more diagnostics that I can gather to help diagnose this. Note I am able to attach TotalView to the a similarly launched MPI job, so LaunchMON is doing something different than TotalView for process acquisition/daemon launch.

Perhaps similarly related, I notice that if I instead launch the MPI application using all 64 tasks that were allocated, my attempt to attach a tool results in:

--------------------------------------------------------------------------
All nodes which are allocated for this job are already filled.
--------------------------------------------------------------------------

Let me know if this should be submitted as a separate issue. I can provide additional diagnostics for this too if need be.

[lee218llnl]

@rhc54 can you help me look at this one or find someone else who can help?

[rhc54]

I can help, but need another day or two to get to it. Is this with OMPI's "master" branch?

[lee218llnl]

Thanks, and yes, this was on OMPI's master branch.

[rhc54]

@ggouaillardet I'm stuck resolving another issue - would you have a chance to look at this and/or perhaps #3247?

[ggouaillardet]

@lee218llnl can you please provide detailled instructions on how to reproduce the issue

• slurm script and sbatch command line
• launchmon command line

[lee218llnl]

The same steps outlined in #3247 should demonstrate this issue, albeit pointing to different versions of OpenMPI. Here's what I did:

git clone https://github.com/llnl/launchmon.git
cd launchmon/
export PATH=/collab/usr/global/tools/openmpi/toss_3_x86_64_ib/openmpi-test/bin:$PATH
./configure --with-test-rm=orte --with-test-rm-launcher=/collab/usr/global/tools/openmpi/toss_3_x86_64_ib/openmpi-test/bin/mpirun --with-test-installed --with-test-ncore-per-CN=2 --with-test-nnodes=1 --prefix=/nfs/tmp2/lee218/prefix/launchmon-openmpi-test && make clean && make -j 8 install && make -j 8 check
cd test/src
./test.launch_1

The error I get is:

[LMON_FE] launching the job/daemons via /collab/usr/global/tools/openmpi/toss_3_x86_64_ib/openmpi-test/bin/orterun

[LMON FE] 6 RM types are supported
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 30 slots
that were requested by the application:
  /g/g0/lee218/src/LaunchMON/test/src/be_kicker


Either request fewer slots for your application, or make more slots available
for use.
--------------------------------------------------------------------------
<Apr 05 08:36:18> <LMON FE API> (INFO): FE-ENGINE connection timed out: 120
[LMON FE] FAILED

Thanks for looking into this.

[lee218llnl]

Also, if you modify the test.launch_1 script to make NUMTASKS fill the allocation, then you will get:

[LMON_FE] launching the job/daemons via /collab/usr/global/tools/openmpi/toss_3_x86_64_ib/openmpi-test/bin/orterun

[LMON FE] 6 RM types are supported
--------------------------------------------------------------------------
All nodes which are allocated for this job are already filled.
--------------------------------------------------------------------------

[rhc54]

Hmmm...just tried this with current head of master repo:

$ ./test.attach_1
This RM is not supported yet
./test.attach_1: line 95: fe_attach_smoketest: command not found
$

Did I miss something, or perhaps something has changed in master that fixed the problem? I configured with:

  $ ./configure --prefix=/home/common/launchmon/build --with-test-rm=orte --with-test-rm-launcher=mpirun --with-test-installed --with-test-nnodes=1

I have mpirun in my PATH.

[lee218llnl]

@rhc54 you may want to try giving configure the full path to mpirun

[rhc54]

Got different output, but it still ran:

$ ./test.attach_1
[rhc001:177054] CHECKING TOPOS
[rhc001:177054] SKIPPING
[rhc001:177054] CHECKING TOPOS
[LMON APP] singal handler installed for signum=10
<Apr 11 10:48:49> APP (INFO): stall for 3 secs
<Apr 11 10:48:52> APP (INFO): stall for 3 secs
<Apr 11 10:48:55> APP (INFO): stall for 3 secs
<Apr 11 10:48:58> APP (INFO): stall for 3 secs
<Apr 11 10:49:01> APP (INFO): stall for 3 secs
<Apr 11 10:49:04> APP (INFO): stall for 3 secs
./test.attach_1: line 95: fe_attach_smoketest: command not found
<Apr 11 10:49:07> APP (INFO): stall for 3 secs
<Apr 11 10:49:10> APP (INFO): stall for 3 secs
<Apr 11 10:49:13> APP (INFO): stall for 3 secs
<Apr 11 10:49:16> APP (INFO): stall for 3 secs
<Apr 11 10:49:19> APP (INFO): stall for 3 secs
<Apr 11 10:49:22> APP (INFO): stall for 3 secs
<Apr 11 10:49:25> APP (INFO): stall for 3 secs
$

Note that I had a bunch of leftover processes "hang_on_SIGUSR1" that I had to manually kill

[rhc54]

aha - I can now replicate. The test/src directory has to also be in the PATH. Will take a look at this.

[rhc54]

Just an FYI: I'm making progress on this, but will probably need another day to complete it

[rhc54]

I'm afraid I am going to need some help here - can you please outline what test.attach_1 is doing? I got past the first error message that you reported, but now I'm getting another one:

--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 25 slots
that were requested by the application:
  /home/common/launchmon/master/test/src/be_kicker

Either request fewer slots for your application, or make more slots available
for use.
--------------------------------------------------------------------------

It would really help if I understood exactly what you are trying to launch (e.g., is be_kicker a debugger daemon or another job? did the first job already terminate?), and how you are launching them (e.g., nprocs and mapping pattern).