Ensure that we always send a SIGTERM prior to SIGKILL to give child processes a chance to cleanup

[rhc54]

Thanks to Noel Rycroft for the report

Signed-off-by: Ralph Castain rhc@open-mpi.org

[rhc54]

Replaces #2498

[rhc54]

@noelrycroft Had to replace the earlier PR due to a repo issue.

[noelrycroft]

@rhc54, @jsquyres sent me another tarball.. this branch looks good to me. I can verify that the SIGTERM is being propagated to the MPI tasks as expected. Thanks for fixing this issue so quickly..

./sleeptest.sh
[24018] : Installing signal handler for signal : INT
[24018] : Installing signal handler for signal : TERM
[24018] : Installing signal handler for signal : 0
[24018] : Parallel Sleep Test MPI
[24018] : /users/xxx/data/RUN/377/openmpi/openmpiinstall/gitclone2/bin/mpirun --prefix /users/xxx/data/RUN/377/openmpi/openmpiinstall/gitclone2 -np 2 sleep.sh 30 &
[24024] : Installing signal handler for signal : INT
[24024] : Installing signal handler for signal : TERM
[24024] : Installing signal handler for signal : 0
[24024] : ./sleep.openmpi 30
[24025] : Installing signal handler for signal : INT
[24025] : Installing signal handler for signal : TERM
[24025] : Installing signal handler for signal : 0
[24025] : ./sleep.openmpi 30
Hello world! I'm 0 out of 2 on dougie.lebanon.xxx
My rank = 0 PID = 24026
0 sleeping for 1 seconds
Hello world! I'm 1 out of 2 on dougie.lebanon.xxx
My rank = 1 PID = 24027
1 sleeping for 1 seconds
2 sleeping for 1 seconds
3 sleeping for 1 seconds
4 sleeping for 1 seconds
5 sleeping for 1 seconds
6 sleeping for 1 seconds
7 sleeping for 1 seconds
8 sleeping for 1 seconds
9 sleeping for 1 seconds
10 sleeping for 1 seconds
11 sleeping for 1 seconds
12 sleeping for 1 seconds
# top level sleeptest.sh receives SIGTERM
[24018] : sleeptest.sh Terminated
[24018] : Killing child : 24019
# explicitly propagates SIGTERM to mpirun
[24018] : kill -TERM 24019
13 sleeping for 1 seconds
# 1st MPI task receives SIGTERM and propagates to child
[24024] : sleep.sh Terminated
[24024] : Killing child : 24026
[24024] : kill -TERM 24026
[24024] : Normal Exit : 143
# 2nd MPI task receives SIGTERM and propagates to child
[24025] : sleep.sh Terminated
[24025] : Killing child : 24027
[24025] : kill -TERM 24027
[24025] : Normal Exit : 143
# top level sleeptest.sh gets exit code from mpirun and exits..
[24018] : Normal Exit : 1

[jsquyres]

@noelrycroft Thanks for testing so quickly!

[jsquyres]

Reviewed by testing / functionality.

[jsquyres]

@hppritcha and I discussed this the other day; I'm confident that he's ok with me merging it. I'd like to get this merged so that it can hit MTT.

[jsquyres]

@hppritcha It looks like LANL Jenkins is wedged.

@rhc54 Is there a master version of this commit?

[rhc54]

85a6349

[hppritcha]

bot:lanl:retest

[hppritcha]

@jsquyres is this ready to merge?