Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose agent options in server command #73

Closed
mortenlj opened this Issue Feb 27, 2019 · 9 comments

Comments

4 participants
@mortenlj
Copy link
Contributor

mortenlj commented Feb 27, 2019

Describe the bug
I'm trying to run k3s on a Raspberry Pi 3 running RancherOS. RancherOS uses docker, so I need to pass the --docker option to the agent. When starting the server, it starts an agent, but there is no way to pass the --docker option (at least not that I can see).

To Reproduce
On a machine with just docker available, start the server with agent enabled and see that it errors out.

Expected behavior
All agent options needs to be possible to pass to the server, so that it can start the agent with those options.

@ibuildthecloud

This comment has been minimized.

Copy link
Member

ibuildthecloud commented Feb 27, 2019

Thanks, we will ensure that all agent options are available in the server command too. For the short time being you can run the server and agent as separate containers. I'll see if I can cook up a config for RancherOS that does this.

@ibuildthecloud

This comment has been minimized.

Copy link
Member

ibuildthecloud commented Feb 28, 2019

This is a work in progress. The seems to initially work but I need to do more testing. We will make sure we document and have a good approach for running k3s on RancherOS, and address the agent flags.

rancher:
  services:
    server:
      image: rancher/k3s:v0.1.0
      command: server --disable-agent
      environment:
      - K3S_CLUSTER_SECRET=somethingrandom
      volumes:
      - /etc/rancher/k3s:/etc/rancher/k3s
      - /var/lib/rancher/k3s:/var/lib/rancher/k3s
      net: host
  
    agent:
      image: rancher/k3s:v0.1.0
      command: agent --docker -s https://localhost:6443
      environment:
      - K3S_CLUSTER_SECRET=somethingrandom
      volumes:
      - /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
      - /var/lib/docker:/var/lib/docker:rshared
      - /var/run:/var/run:rshared
      - /run:/run:rshared
      privileged: true
      net: host
      pid: host
      ipc: host
@mortenlj

This comment has been minimized.

Copy link
Contributor Author

mortenlj commented Feb 28, 2019

Thanks. I was going in that direction myself. I've tried the config, but it didn't work. With some changes, I have something that looks like a working k3s cluster, but pods go into CrashLoopBackOff with errors related to failure to create sandbox because the ResolvConfPath doesn't exist.

My service definition:

k3s-server:
  image: rancher/k3s:v0.1.0
  command: server --no-deploy=servicelb --disable-agent
  environment:
    - K3S_CLUSTER_SECRET=somethingrandom
    - K3S_KUBECONFIG_OUTPUT=/var/lib/rancher/k3s/kubeconfig.yml
    - K3S_KUBECONFIG_MODE=666
  labels:
    io.rancher.os.scope: "system"
  net: host
  volumes:
    - /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
k3s-agent:
  image: rancher/k3s:v0.1.0
  command: agent --docker -s https://localhost:6443
  environment:
    - K3S_CLUSTER_SECRET=somethingrandom
  labels:
    io.rancher.os.scope: "system"
  volumes:
    - /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
    - /var/lib/docker:/var/lib/docker:rshared
    - /var/run/docker.sock:/var/run/docker.sock
  privileged: true
  net: host
  pid: host
  ipc: host

Most notable changes is that mounting /etc/rancher/k3s in the server didn't work for some reason (I couldn't find any error messages, but it simply wasn't mounted), and mounting /var/run and /run fails because the mount is not a shared mount.

@ibuildthecloud ibuildthecloud added this to Backlog in K3S Development Feb 28, 2019

@mortenlj

This comment has been minimized.

Copy link
Contributor Author

mortenlj commented Mar 1, 2019

The ResolvConfPath problem was caused by me using /mnt/docker for container storage, so unrelated to the compose file. This setup sort of works as a workaround (I have found another problem that might be a separate issue, I will investigate and possibly create a new issue for that if it really is a bug).

@ibuildthecloud

This comment has been minimized.

Copy link
Member

ibuildthecloud commented Mar 1, 2019

Since you are running in system-docker /etc/rancher/k3s will not be visible inside the console. When I was testing I was running the services on user docker. I'm putting in a couple fixes to make this easier.

@ibuildthecloud

This comment has been minimized.

Copy link
Member

ibuildthecloud commented Mar 4, 2019

@mortenlj All agent arguments are available on the server command line now in v0.2.0-rc3. Could you please test that this works for you. Also I've tested the following two configuration for k3s in RancherOS

rancher:
  services:
    k3s:
      image: rancher/k3s:v0.2.0-rc2
      command: server --docker
      volumes:
        - /etc/rancher/k3s:/etc/rancher/k3s
        - /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
        - /var/lib/docker:/var/lib/docker:rshared
        - /var/run:/var/run
        - /run:/run
      privileged: true
      net: host
      pid: host
      ipc: host

And similar to above but separating agent and server

rancher:
  services:
    k3s-server:
      image: ibuildthecloud/k3s:f90cbed-dirty-amd64
      command: server --disable-agent
      net: host
      volumes:
        - /etc/rancher/k3s:/etc/rancher/k3s
        - /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
    k3s-agent:
      image: ibuildthecloud/k3s:f90cbed-dirty-amd64
      command: agent --docker -s https://localhost:6443 --token-file /var/lib/rancher/k3s/server/node-token
      volumes:
        - /var/lib/rancher/k3s:/var/lib/rancher/k3s:rshared
        - /var/lib/docker:/var/lib/docker:rshared
        - /var/run:/var/run
        - /run:/run
      privileged: true
      net: host
      pid: host
      ipc: host

@ibuildthecloud ibuildthecloud moved this from Needs review to Reviewer approved in K3S Development Mar 4, 2019

@ibuildthecloud ibuildthecloud moved this from Reviewer approved to Testing in K3S Development Mar 4, 2019

@ibuildthecloud ibuildthecloud moved this from Testing to Reviewer approved in K3S Development Mar 4, 2019

@mortenlj

This comment has been minimized.

Copy link
Contributor Author

mortenlj commented Mar 6, 2019

Thanks! I'll see if I get a chance to try it out tomorrow.

@cjellick cjellick added this to the v0.2.0 milestone Mar 8, 2019

@cjellick cjellick moved this from Reviewer approved to Testing in K3S Development Mar 8, 2019

@cjellick cjellick changed the title Server needs to expose agent options Expose agent options in server command Mar 8, 2019

@erikwilson

This comment has been minimized.

Copy link
Member

erikwilson commented Mar 8, 2019

Verified with using k3s server --docker and --no-flannel that the arguments are passed to the agent.

@erikwilson erikwilson closed this Mar 8, 2019

K3S Development automation moved this from Testing to Done Mar 8, 2019

@mortenlj

This comment has been minimized.

Copy link
Contributor Author

mortenlj commented Mar 10, 2019

I got around to testing this now, and it works fine. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.