Expose agent options in server command

[mortenlj]

Describe the bug
I'm trying to run k3s on a Raspberry Pi 3 running RancherOS. RancherOS uses docker, so I need to pass the --docker option to the agent. When starting the server, it starts an agent, but there is no way to pass the --docker option (at least not that I can see).

To Reproduce
On a machine with just docker available, start the server with agent enabled and see that it errors out.

Expected behavior
All agent options needs to be possible to pass to the server, so that it can start the agent with those options.

[ibuildthecloud]

Thanks, we will ensure that all agent options are available in the server command too. For the short time being you can run the server and agent as separate containers. I'll see if I can cook up a config for RancherOS that does this.

[ibuildthecloud]

This is a work in progress. The seems to initially work but I need to do more testing. We will make sure we document and have a good approach for running k3s on RancherOS, and address the agent flags.

rancher:
  services:
    server:
      image: rancher/k3s:v0.1.0
      command: server --disable-agent
      environment:
      - K3S_CLUSTER_SECRET=somethingrandom
      volumes:
      - /etc/rancher/k3s:/etc/rancher/k3s
      - /var/lib/rancher/k3s:/var/lib/rancher/k3s
      net: host
  
    agent:
      image: rancher/k3s:v0.1.0
      command: agent --docker -s https://localhost:6443
      environment:
      - K3S_CLUSTER_SECRET=somethingrandom
      volumes:
      - /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
      - /var/lib/docker:/var/lib/docker:rshared
      - /var/run:/var/run:rshared
      - /run:/run:rshared
      privileged: true
      net: host
      pid: host
      ipc: host

[mortenlj]

Thanks. I was going in that direction myself. I've tried the config, but it didn't work. With some changes, I have something that looks like a working k3s cluster, but pods go into CrashLoopBackOff with errors related to failure to create sandbox because the ResolvConfPath doesn't exist.

My service definition:

k3s-server:
  image: rancher/k3s:v0.1.0
  command: server --no-deploy=servicelb --disable-agent
  environment:
    - K3S_CLUSTER_SECRET=somethingrandom
    - K3S_KUBECONFIG_OUTPUT=/var/lib/rancher/k3s/kubeconfig.yml
    - K3S_KUBECONFIG_MODE=666
  labels:
    io.rancher.os.scope: "system"
  net: host
  volumes:
    - /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
k3s-agent:
  image: rancher/k3s:v0.1.0
  command: agent --docker -s https://localhost:6443
  environment:
    - K3S_CLUSTER_SECRET=somethingrandom
  labels:
    io.rancher.os.scope: "system"
  volumes:
    - /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
    - /var/lib/docker:/var/lib/docker:rshared
    - /var/run/docker.sock:/var/run/docker.sock
  privileged: true
  net: host
  pid: host
  ipc: host

Most notable changes is that mounting /etc/rancher/k3s in the server didn't work for some reason (I couldn't find any error messages, but it simply wasn't mounted), and mounting /var/run and /run fails because the mount is not a shared mount.

[mortenlj]

The ResolvConfPath problem was caused by me using /mnt/docker for container storage, so unrelated to the compose file. This setup sort of works as a workaround (I have found another problem that might be a separate issue, I will investigate and possibly create a new issue for that if it really is a bug).

[ibuildthecloud]

Since you are running in system-docker /etc/rancher/k3s will not be visible inside the console. When I was testing I was running the services on user docker. I'm putting in a couple fixes to make this easier.

[ibuildthecloud]

@mortenlj All agent arguments are available on the server command line now in v0.2.0-rc3. Could you please test that this works for you. Also I've tested the following two configuration for k3s in RancherOS

rancher:
  services:
    k3s:
      image: rancher/k3s:v0.2.0-rc2
      command: server --docker
      volumes:
        - /etc/rancher/k3s:/etc/rancher/k3s
        - /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
        - /var/lib/docker:/var/lib/docker:rshared
        - /var/run:/var/run
        - /run:/run
      privileged: true
      net: host
      pid: host
      ipc: host

And similar to above but separating agent and server

rancher:
  services:
    k3s-server:
      image: ibuildthecloud/k3s:f90cbed-dirty-amd64
      command: server --disable-agent
      net: host
      volumes:
        - /etc/rancher/k3s:/etc/rancher/k3s
        - /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
    k3s-agent:
      image: ibuildthecloud/k3s:f90cbed-dirty-amd64
      command: agent --docker -s https://localhost:6443 --token-file /var/lib/rancher/k3s/server/node-token
      volumes:
        - /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
        - /var/lib/docker:/var/lib/docker:rshared
        - /var/run:/var/run
        - /run:/run
      privileged: true
      net: host
      pid: host
      ipc: host

[mortenlj]

Thanks! I'll see if I get a chance to try it out tomorrow.

[erikwilson]

Verified with using k3s server --docker and --no-flannel that the arguments are passed to the agent.

[mortenlj]

I got around to testing this now, and it works fine. Thanks!

[zimme]

@mortenlj All agent arguments are available on the server command line now in v0.2.0-rc3. Could you please test that this works for you. Also I've tested the following two configuration for k3s in RancherOS

rancher:
  services:
    k3s:
      image: rancher/k3s:v0.2.0-rc2
      command: server --docker
      volumes:
        - /etc/rancher/k3s:/etc/rancher/k3s
        - /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
        - /var/lib/docker:/var/lib/docker:rshared
        - /var/run:/var/run
        - /run:/run
      privileged: true
      net: host
      pid: host
      ipc: host

And similar to above but separating agent and server

rancher:
  services:
    k3s-server:
      image: ibuildthecloud/k3s:f90cbed-dirty-amd64
      command: server --disable-agent
      net: host
      volumes:
        - /etc/rancher/k3s:/etc/rancher/k3s
        - /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
    k3s-agent:
      image: ibuildthecloud/k3s:f90cbed-dirty-amd64
      command: agent --docker -s https://localhost:6443 --token-file /var/lib/rancher/k3s/server/node-token
      volumes:
        - /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
        - /var/lib/docker:/var/lib/docker:rshared
        - /var/run:/var/run
        - /run:/run
      privileged: true
      net: host
      pid: host
      ipc: host

When trying this and restarting the k3s(-server/-agent) service it seems to close the docker socket. Is this a k3s, crictl or kubelet problem? I've been trying to follow the code to find what would close the docker socket on exit but I can't find it.