How to run multiple user docker daemons

Terminology

Term	Definition
Dind	Docker in docker, this is the key to our realization of this feature.
User docker, UD	The user-docker on RancherOS
Other user docker, OUD	The other user-docker daemons we create, these user-docker daemons are Dind mode

Solution Overview

Introduce some CLIs to allow users to create OUDs.

Rancher1.6 server and rancher-agent can work on user-docker and dind container.

Architecture

UD still runs in the console container, OUD runs on system-docker. Each OUD is a container, and has standclone fs and namespace. Users can access each OUD in the console container.

                                    RancherOS
+-------------------------------------------+
|                                           |
|  +------UD------+       +--------------+  |
|  |  console     <---+---- OUD container|  |
|  |  container   |   |   +--------------+  |
|  +--------------+   |   +--------------+  |
|                     +---- OUD container|  |
|                         +--------------+  |
|  +-------------------------------------+  |
|  |              +--------------+       |  |
|  | ROOTFS       | system-docker|       |  |
|  |              +--------------+       |  | 
|  +-------------------------------------+  |
|                                           |
+-------------------------------------------+

How to run

Download ISO and install

You can download the ISO here.

If you want to use it on AWS, you should boot a VM with RancherOS1.4 and upgrade to a new version:

ros os upgrade -i niusmallnan/os:mud-1

Prepare

You must switch user-docker to 17.12.1 or earlier version.

ros engine switch docker-17.12.1-ce

Otherwise, you may get these error when you create an user-defined network on system-docker.

Error response from daemon: unable to insert jump to DOCKER-ISOLATION rule in FORWARD chain:  (iptables failed: iptables --wait -I FORWARD -j DOCKER-ISOLATION: iptables v1.6.0: Couldn't find target `DOCKER-ISOLATION'

Try `iptables -h' or 'iptables --help' for more information.
 (exit status 2))

Create an user-define network, need to use this network when creating an OUD:

system-docker network create --subnet=172.20.0.0/16 dind

Create OUD

Just use ros engine create.

ros engine create dind1 --network=dind --fix-ip=172.20.0.2

After the OUD service is created, users can query the OUD service as usual.

ros service list


ros service up dind1

After the OUD service is started, users can interact with it as if they were using the docker command.

docker-dind1 ps -a

Remove OUD

Just use ros engine rm:

ros engine rm dind1

SSH into OUD container

Rancher1.6 setup

Run rancher/server with UD

docker run -d --restart=unless-stopped -p 8080:8080 rancher/server:stable

Run rancher/agent in OUD container

# -e CATTLE_CHECK_NAMESERVER=false`need to be added

docker run -e CATTLE_AGENT_IP="172.20.0.2"  --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher -e CATTLE_CHECK_NAMESERVER=false rancher/agent:v1.2.10 http://<RancherOS-IPADDRESS>:8080/v1/scripts/xxxxxx

Limitations

Others