Q: Supported commands

[sebschlicht]

Sorry for using the issue section but I could not find a mailing list or similar.

I am using PSSH to execute a script on - for now - a single node

sshKey=paramiko.RSAKey.from_private_key_file('mysshkeypath',password='mypw'))
ParallelSSHClient(hosts=['myipaddress'], user='node', pkey=sshKey)
client.exec_command(`/home/node/circus/start.sh`)

with /home/node/circus/start.sh being

touch /tmp/test1
circusd /home/node/circus/circus.ini &>/tmp/console.log &
touch /tmp/test2

While the touches get executed, the circusd is not. No console.log is created.
When I execute the above call using a single SSH client everything works just fine.
If I remove the & to make it a synchronous call, Circus spawns and PSSH blocks until I kill the process on the remote machine manually - as expected.

Can someone explain me what is the problem here?
Am I using PSSH wrong? This does not seem to be a Circus-issue since I can start the scripts locally and via SSH client without any issue.

[pkittenis]

Can you show an example with the single SSHClient that works as expected? Might be a race condition when checking result via ParallelSSHClient

Can you please also show the version of ParallelSSH you are using.

[sebschlicht]

I am sorry for using this misleading phrase. With "using a single SSH client" I meant using the terminal and executing the above script on the remote machine via ssh.

$ ssh node@myipaddress
node@nodePC $ /home/node/circus/start.sh

With sshpt, for instance, the start script is executed as expected.

[pkittenis]

Not sure I understand what the issue is. Keep in mind the output redirection is done by the shell. If the shell exits then the redirection ends.

Is circusd running but you have no log output or is circusd not running?

I would not be surprised if you do not have anything in the log but circusd is running as the redirection happens outside the daemon (the circusd executable) and ends when the script ends or when the shell that spawned the script exits.

If you login via SSH and run the script you are holding the shell open (your SSH connection) so the redirection will continue to work. If you exit the shell the redirection ends but the daemon keeps going..

[sebschlicht]

The issue does not target the redirection. As you can see I write it to a local file and do not care further.
Actually there will be no redirection.

My only goal is to start the circusd process and keep it alive.
You might be right. Maybe the process is shutdown when the script returns. However, I thought the & would prevent the process from doing so and using sshpt the process indeed keeps alive.

[pkittenis]

Try client.exec_command(nohup /home/node/circus/start.sh) instead, does that fix your issue? & just puts the command in background, doesn't do any forking.

If circusd is not forking and running as daemon by itself it will exit when the shell exits. Given that it hangs when you do not have the & that sounds likely.

[sebschlicht]

Sorry, I edited my answer a couple of times and removed a clear statement on that:

No, cirucsd is not running.

[pkittenis]

Here's an example with a daemon:
script test.sh:

#!/bin/bash

while true; do echo "Running"; sleep 5; done

script starter.sh

#!/bin/bash
touch /tmp/test1
start-stop-daemon --start -b -x /home/XX/test.sh
touch /tmp/test2

Run it:

client = ParallelSSHClient(['localhost'])
cmds = client.exec_command('/home/XX/starter.sh')
client.pool.join()
# ps aux | grep test.sh
XXX    26803  0.0  0.0  12404  1192 ?        S    14:18   0:00 /bin/bash /home/XXX/test.sh
ls -lh /tmp/test*
-rw-rw-r-- 1 XX  XX  0 Jan  5 14:18 test1
-rw-rw-r-- 1 XX XX    0 Jan  5 14:18 test2

If test.sh is changed to:

#!/bin/bash
touch /tmp/test1
start-stop-daemon --start -x /home/XX/test.sh -- &
touch /tmp/test2

Which seems to be what your script is doing, then test.sh does not run as it is backgrounded, not forked, and exits when the shell exits.

A 'proper' daemon with a wait for the child process:
test.sh:

while true; do echo "Running"; sleep 5; done &> /tmp/console.log &

# Capture the child process PID
CHILD="$!"

# Kill the child process when start-stop-daemon sends us a kill signal
trap "kill $CHILD" exit INT TERM

# Wait for child process to exit
wait

Run it:

client = ParallelSSHClient(['localhost'])
cmds = client.exec_command('/home/XX/starter.sh')
client.pool.join()
# ps -ef | grep test.sh
XX    28155     1  0 14:44 ?        00:00:00 /bin/bash /home/XX/test.sh
XX    28157 28155  0 14:44 ?        00:00:00 /bin/bash /home/XX/test.sh

PID 28157 is the backgrounded command within the test.sh process and its parent is PID 28155. PID 28155 is test.sh which was forked by start-stop-daemon making its parent PID 1, the init process.

Redirection needs to happen within the forked process or it disappears when the shell exits.

[pkittenis]

Closing as this is a bash question - not related to ParallelSSH.

[pkittenis]

There's a good explanation of what you are seeing here - http://stackoverflow.com/questions/881388/what-is-the-reason-for-performing-a-double-fork-when-creating-a-daemon

Have a look at the pid and parent pid of the process when it is spawned and the above explanation and it should become clear what is happening on your system.

[sebschlicht]

Since it is working both in ssh and sshpt I thought it was related to parallel SSH. If such workaround is not necessary in other tools there might be an issue as well. If this isn't a bug one can see it as pointer to a possible enhancement.
There is no hint concerning this limitation in the docs and I would consider & to be supported by parallel SSH clients by default.

[pkittenis]

Your issue is with bash, not ParallelSSH. Backgrounded processes, what & does, will exit when the shell exits unless they are correctly daemonized per POSIX standard. This has nothing to do with ParallelSSH.

ParallelSSH knows nothing about & or the shell you are running, it simply executes commands in a new shell process.

If you can provide a test case showing ParallelSSH not working correctly with a daemon please do so and will re-open and investigate. Have posted a test case with a daemon above which works as expected - feel free to test it out on your system.

Again, if your issue is fixed by using nohup your issue is with bash, not ParallelSSH.

Here's some documentation explaining daemons and backgrounded processes - http://www.unix.com/unix-for-advanced-and-expert-users/42558-nohup-background-process.html

Specifically:

When you run a command from an interactive shell you have a "controlling terminal" managed by sshd, telnetd, xterm or whatever.

When that controlling terminal closes it sends the processes under it's umbrella a 'SIGHUP' signal, this will normally kill those processes. 'nohup' traps this signal and may detach from the controlling terminal.

[pkittenis]

Here's some detailed explanation..

Backgrounded process will exit when their parent process exits. When backgrounding a process within a shell, the parent is the shell. The parent needs to explicitly be told to wait for its children if the intention is for a background process to not exit.

Run these commands in a new bash shell

1. Background then exit

$ touch /tmp/test
$ tail -f /tmp/test & > /dev/null
$ exit

Backgrounded process tail exits when its parent, the shell running it, exits. The shell exits immediately after kicking off the backgrounded process, which immediately kills the tail child process.

2. Background and wait

$ touch /tmp/test
$ tail -f /tmp/test & > /dev/null
$ wait
$ exit

Now the parent process is told to wait for its children to finish executing before exiting. The tail process keeps running indefinitely until either it or the parent process are killed.

Scenario (1) is what you are doing. Scenario (2) is an example of how to do a quick and dirty 'daemon' out of a backgrounded process. You still need nohup in both cases if you want the process to remain alive when the user running it logs out. There are tons of similar recipes on the net though you are better off with something like start-stop-daemon.

Neither ParallelSSH nor SSH in general come into play here, this is all shell code. I trust that explains things, if not then I would suggest going through the Advanced Bash Scripting Guide as this is not a forum for bash questions.

[sebschlicht]

Thank you, for your active support! It is not a pssh issue, you explained that very well. Though I first thought it would be, I am now convinced it is not.
What I am just saying now is: It works out-of-the-box with other tools and maybe here is a possible enhancement.

[pkittenis]

Yes, it looks like the 'other tools' are doing the same thing but the process is zombied and kept alive by the SSHd controlled pty rather than being spawned in a new pty which ParallelSSH does by default.

For example, with ssh remote command:

$ ssh localhost "tail -f xx & > /dev/null"

Command blocks and waits for tail -f to finish, although shell is supposed to have exited. Looking at process list, we indeed see tail with a PPID of 1, so the backgrounded process got turned into a zombie when the shell exited and was picked up by init.

xx    19833 15437  0 17:46 pts/0    00:00:00 ssh localhost tail -f xx & > /dev/null
xx    19876     1  0 17:46 ?        00:00:00 tail -f xx

Forcing a new tty with ssh confirms it:

ssh -t localhost "tail -f xx & > /dev/null"
Connection to localhost closed.

Process exits, tail exits, which is correct.

Zombying backgrounded processes is improper behaviour so would be loathe to do that. The intended usage for ParallelSSH is non-interactive, therefor the use of a new pty is enforced.

Saying 'it works' with other tools is not really the case here as the behaviour when 'it works' is improper. The process should not be kept alive, but it is.

The left alive process will eventually be killed by init when it starts checking for zombie processes so your service will just stop running after some amount of time and you won't know why.

If a new pty is not used other issues will crop up depending on user and terminal configuration such as real daemons that are killed when the SSH connection terminates and ParallelSSH blocking without exiting the shell because the shell has children that are still running, exactly like in the first remote ssh command.

Some documentation on this scenario would probably be good tho, thank you for bringing it up.