How to run multiple user docker daemons

Terminology

Term	Definition
Dind	Docker in docker, this is the key to our realization of this feature.
User docker, UD	The user-docker on RancherOS
Other user docker, OUD	The other user-docker daemons we create, these user-docker daemons are Dind mode

Solution Overview

Introduce some CLIs to allow users to create OUDs.

Rancher1.6 server and rancher-agent can work on user-docker and dind container.

Architecture

UD still runs in the console container, OUD runs on system-docker. Each OUD is a container, and has standclone fs and namespace. Users can access each OUD in the console container.

                                    RancherOS
+-------------------------------------------+
|                 docker.sock               |
|  +------UD------+   |   +--------------+  |
|  |  console     ----+---> OUD container|  |
|  |  container   |   |   +--------------+  |
|  +--------------+   |   +--------------+  |
|                     +---> OUD container|  |
|                         +--------------+  |
|  +-------------------------------------+  |
|  |              +--------------+       |  |
|  | ROOTFS       | system-docker|       |  |
|  |              +--------------+       |  | 
|  +-------------------------------------+  |
|                                           |
+-------------------------------------------+

How to run

Download ISO and install

Users can download the ISO here. https://storage.googleapis.com/ros-images/rancheros-mud-3.iso

If users want to use it on AWS, they should boot a VM with RancherOS1.4 and upgrade to a new version:

ros os upgrade -i niusmallnan/os:mud-3

Prepare

Users must switch user-docker to 17.12.1 or earlier version.

ros engine switch docker-17.12.1-ce

Otherwise, it may get these error when creating an user-defined network on system-docker.

Error response from daemon: unable to insert jump to DOCKER-ISOLATION rule in FORWARD chain:  (iptables failed: iptables --wait -I FORWARD -j DOCKER-ISOLATION: iptables v1.6.0: Couldn't find target `DOCKER-ISOLATION'

Try `iptables -h' or 'iptables --help' for more information.
 (exit status 2))

Create an user-define network, need to use this network when creating an OUD:

system-docker network create --subnet=172.20.0.0/16 dind

Create OUD

Just use ros engine create. For the OUD image, currently only support docker 17.12.1 and 18.03.1.

ros engine create dind1 --network=dind --fixed-ip=172.20.0.2

After the OUD service is created, users can query the OUD service as usual.

ros service list
...
...
disabled volume-efs
disabled volume-nfs
enabled  dind1

To make the dind1 service running, can use:

ros service up dind1

After the OUD service is started, users can interact with it as if they were using the docker command.

docker-dind1 ps -a

SSH into OUD container

User can specify an external ssh port with --ssh-port, and ssh keys with --authorized-keys. Both of them are optional.

ros engine create  -h
...
...
OPTIONS:
    --ssh-port value
    --authorized-keys value

For --authorized-keys, user needs to put the key file in one of the following directories:

/var/lib/rancher/
/opt/
/home/

We will generate a random password for each OUD container, which users can see in the container logs. This password is useful when the users do not set the keys.

system-docker logs dind1
======================================
chpasswd: password for 'root' changed
password: xCrw6fEG
======================================

Users can ssh into any OUD container like this:

system-docker ps
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS                             NAMES
2ca07a25799b        rancher/os-dind:17.12.1          "docker-entrypoint..."   5 seconds ago       Up 3 seconds        2375/tcp, 0.0.0.0:34791->22/tcp   dind1

ssh -p 34791 root@<host-external-ip>

ssh root@<OUD-container-ip>

Remove OUD

Just use ros engine rm:

ros engine rm dind1

Rancher1.6 setup

Run rancher/server with UD

docker run -d --restart=unless-stopped -p 8080:8080 rancher/server:stable

Run rancher/agent in OUD container

# exec into OUD container, also ssh
system-docker exec -it dind1 /bin/sh

# Add an agent to this rancher-server
# "-e CATTLE_CHECK_NAMESERVER=false" need to be added
docker run -e CATTLE_AGENT_IP="172.20.0.2" -e CATTLE_CHECK_NAMESERVER=false --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.2.10 http://<RancherOS-IPADDRESS>:8080/v1/scripts/xxxxxx

Limitations

As Rancher uses the overlay network by default, only one RancherOS host can be used in one Env.

Others

Please submit any bugs, issues for this feature to rancher/os, and add this label area/oud would be very friendly.