Debugger support for launch, detach, then reattach

[lee218llnl]

I have noticed that if I launch an MPI job under a debugger, then orterun does not create the attach fifo. This is OK for the initial debug session, however, if I detach the debugger and then later try to attach to the same MPI job, the attach fifo does not exist and thus my debugger fails. Is it possible for orterun to generate the attach fifo after the initial debugger session detaches?

[rhc54]

Assuming we can detect that the debugger detached, then sure - it would be relatively easy to start the fifo at that time. Alternatively, I see no reason why the fifo can't simply be created at the beginning of time, though I suppose we would have to add some protection to ensure you didn't attach to a job that was already launched under a debugger. At the moment, we have that protection for the attach use-case because the fifo won't respond while you are attached, so you can't attach twice to the same job.

[lee218llnl]

Thanks @rhc54. I trust that you can figure out the best path forward. Let me know if you need me to test anything.

[rhc54]

@lee218llnl Please give this a try: #3222

[lee218llnl]

I'm not sure what else has changed, but I am having a few issues with #3222. When I try to launch my job I get this error:

--------------------------------------------------------------------------
All nodes which are allocated for this job are already filled.
--------------------------------------------------------------------------

And when I try to attach I get this error:

There are not enough slots available in the system to satisfy the 28 slots
that were requested by the application:
  /nfs/tmp2/lee218/prefix/stat-travis/bin/STATD

Either request fewer slots for your application, or make more slots available
for use.

[rhc54]

Odd - can you tell me a bit about the scenario? I assume there is a resource manager involved? What was the allocation? And what was the cmd line to launch the job?

[lee218llnl]

My Launch command is:

/nfs/tmp2/lee218/prefix/stat-travis/bin/stat-cl -t 1 -C /collab/usr/global/tools/openmpi/openmpi-test/bin/mpirun mpi_ringtopo.ompi-test

And for the attach case, I ran:

/collab/usr/global/tools/openmpi/openmpi-test/bin/mpirun -n 4 mpi_ringtopo.ompi-test

And then try to attach to the PID of mpirun with the STAT tool (which uses LaunchMON).

[lee218llnl]

Here's another scenario (launching and specifying the number of MPI tasks):

bash-4.2$ /nfs/tmp2/lee218/prefix/stat-travis/bin/stat-cl -t 1 -C /collab/usr/global/tools/openmpi/openmpi-test/bin/mpirun -n 4 mpi_ringtopo.ompi-test 
STAT started at 2017-03-22-08:51:01
Launching application and tool daemons...
--------------------------------------------------------------------------
There are not enough slots available in the system to satisfy the 28 slots
that were requested by the application:
  /nfs/tmp2/lee218/prefix/stat-travis/bin/STATD

Either request fewer slots for your application, or make more slots available
for use.
--------------------------------------------------------------------------

[rhc54]

Just to be clear: if you attempt to run the app under the debugger on the current HEAD of OMPI master (minus this PR), does it work? I would honestly be surprised - all I changed here was to ensure that the attach fifo is always created. My guess is that the change to the mapping system (done a few weeks ago) is the culprit

[lee218llnl]

I have not tried with the HEAD of OMPI master, but my guess is that it would break in the same way. Perhaps I will just take a patch from your commit and apply it to the last working OpenMPI that I was able to use to see if the FIFO issue is resolved and then submit a separate issue for these attach/launch errors.

[rhc54]

Kewl - that makes sense. Sorry for the problem.

[lee218llnl]

I checked out the v2.0.2 tag and added the code from #3222 to orterun.c and was able to reattach to a job that was launched under the debugger, so this issue appears to be resolved in #3222. I will do some more testing w.r.t. the other errors and submit bug reports as necessary. Thanks!

[lee218llnl]

Note I created #3243 for the attach failure on the master branch.