Run runc container over SSH

[mouri11]

I am working on a cluster of runc containers running in AWS EC2 Ubuntu 18.04 x86_64 instances. I intend to have one master node run the containers in all the other nodes via SSH. I am using the following bash script in the master node:

#!/bin/bash
while read node
do
      ssh user@$node "cd ~/mycont; sudo runc run mycont"
done <active

The "active" file contains list of remote IPs for the EC2 instances. Running this results in an error similar to the one below
level=error msg="open /dev/tty: no such device or address"

Running the runc command by directly accessing the remote terminal via SSH works, but not in this inline format. Also, using ssh -t works as inline in the local terminal, but not in the bash script.

I am not sure what I am missing here. I don't have an in-depth knowledge of how SSH issues the commands: in the remote terminal, or by creating a terminal in the local machine. Also, will restoring the container via inline SSH have any issues? Any help would be appreciated.

Edit: Does it have anything to do with terminal: true in config.json?

[cyphar]

Yeah, runc has kinda special terminal handling. We have some documentation on how it works though I don't think it describes the situation you're in. I'll need to test it out to see what exactly is going on -- I think runc in foreground new-terminal mode expects stdin to be a terminal hence the error.

You can either disable new terminal creation ("terminal": false) -- which might be okay for you unless you need to run programs that expect a proper terminal like sudo -- or you can create a terminal for runc to run in (I think script -q or something like that is the way you used to do this -- but I'd need to double-check).

[kolyshkin]

Yes, the best way is to disable the terminal in config.json.

There are a few ways to create a terminal, the simplest one being ssh -t. As you reported, it fails when run via bash script -- which probably means that bash script process do no not have a terminal itself (for example, when run via cron or other script). Anyway, for such cases where is ssh -tt; as described in ssh(1), "multiple -t options force tty allocation, even if ssh has no local tty".

The alternative is using script(3), something like script -e -c 'runc run mycont' should work. Note that older versions of script utility might be buggy.

[cyphar]

Might be a good idea to document some of this in docs/terminal.md?

[kolyshkin]

which probably means that bash script process do no not have a terminal itself

Nope; in your case it happens because stdin is getting redirected:

#!/bin/bash
while read node
do
      ssh user@$node "cd ~/mycont; sudo runc run mycont"
done <active

so stdin (fd 0) is not a terminal (and I guess that ssh only looks at stdin while searching for a terminal), and thus ssh deduces that there's no terminal to pass through.

Perhaps the easiest solution is to rewrite your script so stdin won't be a redirect.

[kolyshkin]

Might be a good idea to document some of this in docs/terminal.md?

Good idea, @cyphar! PTAL #2985