/bundle-canonical-kubernetes

Kubernetes with OVN #350

Closed
AakashKT opened this Issue Jul 12, 2017 · 9 comments

Comments

Assignees
No one assigned
Labels
kind/feature
Projects
None yet
Milestone
  2017.08.04
3 participants
@AakashKT @tvansteenburgh @marcoceppi
@AakashKT

AakashKT commented Jul 12, 2017 •
Edited 1 time

Hello everyone,
This is a thread to discuss a charm for deploying Kubernetes with OVN which I am working on.
Documentation reference : https://github.com/openvswitch/ovn-kubernetes
Charm : cs:~aakashkt/ovn-2
Charm code : https://github.com/AakashKT/ovn-kubernetes-charm
The bundle file for deploying the whole of kubernetes using OVN is as below:

series: xenial

applications:

  kubernetes-master:
    charm: "cs:~containers/kubernetes-master-35"
    num_units: 1
    to:
      - "0"
    expose: true
    options:
      service-cidr: 192.168.200.0/24
      channel: 1.5/stable

  kubernetes-worker:
    charm: "cs:~containers/kubernetes-worker-40"
    num_units: 1
    to:
      - "1"
    expose: true
    options:
      channel: 1.5/stable

  ovn:
    charm: "cs:~aakashkt/ovn-2"
    options:
      gateway-physical-interface: "ens4"

  etcd:
    charm: "cs:~containers/etcd-40"
    num_units: 1
    to:
      - "0"

  easyrsa:
    charm: "cs:~containers/easyrsa-12"
    num_units: 1
    to:
      - "1"

relations:
  - - "kubernetes-master:kube-api-endpoint"
    - "kubernetes-worker:kube-api-endpoint"
  - - "kubernetes-master:kube-control"
    - "kubernetes-worker:kube-control"
  - - "kubernetes-master:cni"
    - "ovn:cni"
  - - "kubernetes-worker:cni"
    - "ovn:cni"
  - - "etcd:certificates"
    - "easyrsa:client"
  - - "kubernetes-worker:certificates"
    - "easyrsa:client"
  - - "kubernetes-master:etcd"
    - "etcd:db"
  - - "kubernetes-master:certificates"
    - "easyrsa:client"

machines:
  "0":
    series: xenial
  "1":
    series: xenial

Note1 : If you plan to deploy the above, make sure you change the 'gateway-physical-interface' config option in ovn charm to your default interface on the machine. 'ip route | grep default' should get you the name of default interface.
Note2 : Also, you will have to grant access to ports 6641 and 6642 on master and port 8080 both on master and worker, in your cloud providers firewall rules

The above setup gets stuck for master at 'waiting for kube-system pods to start'. On looking further, I found that for some reason, the worker node does not register with master, and I need to pass specific args while starting the services.

@ktsakalozos suggested to restart snap services on master and worker providing custom arguments by editing the /var/snap/service/current/args file, kill services, using /snap/service/3/service --args to restart them
On doing that, the service runs perfectly.
I am providing the exact arguments below, for each service:
Master

kube-apiserver --service-cluster-ip-range=192.168.200.0/24 --address=0.0.0.0 --etcd-servers=<your_etcd_server_ip> --v=9
kube-controller-manager --master=127.0.0.1:8080 --v=9
kube-scheduler --master=127.0.0.1:8080 --v=9

Worker

kubelet --api-servers=http://$MASTER_IP:8080 --v=2 --address=0.0.0.0 --enable-server=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir="/opt/cni/bin/"

This should start kubernetes normally (It may not show up that way in 'juju status')
Now, the main issue here is how do I get these args passed in the main kubernetes-worker and kubernetes-master charm, and if that is done, does Kubernetes still start normally?

I have not put SSL in OVN, but maybe that is required? Also, I am aware that starting k8s by the above options starts it without SSL, but I guess we can figure out how to start k8s with SSL and correspondingly OVN too.
@tvansteenburgh
Member

tvansteenburgh commented Jul 12, 2017

@AakashKT Hi, thanks for taking the time to write this up.

We have a PR open (kubernetes/kubernetes#47912) that will allow you to pass arbitrary args to kube-apiserver via charm config, once it's merged and released.

We would need to add similar support for manipulating the args of the other services. A bug for this was opened recently (#339), but we haven't done any work on that yet.

Seeing that you have a specific use case that requires those features, we'll take a look at in our next development sprint. I can't promise that it will get finished in that sprint, but we'll at least do some planning around it and get started.

@tvansteenburgh tvansteenburgh added the kind/feature label Jul 12, 2017

@tvansteenburgh tvansteenburgh added this to the 2017.08.04 milestone Jul 12, 2017

@marcoceppi
Owner

marcoceppi commented Jul 12, 2017

@tvansteenburgh
Member

tvansteenburgh commented Jul 13, 2017

@marcoceppi We don't have an interface that does that today, and I think that allowing args to be set via config is more generally useful (two other people have already asked for it).
@marcoceppi
Owner

marcoceppi commented Jul 13, 2017

I don't disagree this feature of config set is needed, but surely flannel charm is setting args somehow? The idea that deploying a subordinate /then/ changing configuration is a step backwards in the Juju model
@marcoceppi
Owner

marcoceppi commented Jul 13, 2017

So, there's a kubernetes-cni interface, but it only has a cidr key. It might be worth having that interface expanded if there's anything missing for this, or allowing to pass args through so the setup is pluggable
@AakashKT

AakashKT commented Jul 13, 2017

@tvansteenburgh @marcoceppi Thanks for the update! Good hear that there is a PR already addressing this issue.
I guess till this config options get released, I can have a workaround? I had something like the following in mind:
In the subordinate charm, I can wait for the master and correspondingly worker services to start, then the I can restart the snaps as mentioned earlier. Is there a way for me to know, in the subordinate charm, when that happens in the principal? And do you think this will be a good approach for now? I am trying to implement exactly this and test whether it actually works, but do tell me if there is something else we can do.

@marcoceppi Hi! First off,I would like to say that your talks/tutorials on juju charms helped me a lot to get things started. Thanks a lot for those.
You mentioned the kubernetes-cni interface, I am already using that, though just to determine if the principal is master or worker. And I guess you are right that it should be expanded. Would provide an easy way to pass args then.
@AakashKT

AakashKT commented Jul 25, 2017

An update :
The final bundle is not working, which I have attached below, do deploy and see if you wish

series: xenial

applications:

  kubernetes-master:
    charm: "cs:~containers/kubernetes-master-35"
    num_units: 1
    to:
      - "0"
    expose: true
    options:
      service-cidr: 192.168.200.0/24
      channel: 1.5/stable

  kubernetes-worker:
    charm: "cs:~containers/kubernetes-worker-40"
    num_units: 1
    to:
      - "1"
    expose: true
    options:
      channel: 1.5/stable

  ovn:
    charm: "cs:~aakashkt/ovn-13"
    options:
      gateway-physical-interface: "none"

  etcd:
    charm: "cs:~containers/etcd-40"
    num_units: 1
    to:
      - "0"

  easyrsa:
    charm: "cs:~containers/easyrsa-12"
    num_units: 1
    to:
      - "1"

relations:
  - - "kubernetes-master:kube-api-endpoint"
    - "kubernetes-worker:kube-api-endpoint"
  - - "kubernetes-master:kube-control"
    - "kubernetes-worker:kube-control"
  - - "kubernetes-master:cni"
    - "ovn:cni"
  - - "kubernetes-worker:cni"
    - "ovn:cni"
  - - "etcd:certificates"
    - "easyrsa:client"
  - - "kubernetes-worker:certificates"
    - "easyrsa:client"
  - - "kubernetes-master:etcd"
    - "etcd:db"
  - - "kubernetes-master:certificates"
    - "easyrsa:client"

machines:
  "0":
    series: xenial
  "1":
    series: xenial

@ktsakalozos deployed the bundle, and noticed some of the pods were not starting, and has created an issue here : AakashKT/ovn-kubernetes-charm#3
Neither I nor him are entirely sure why the pods don't start. Posting it here since we can have more help then getting to the bottom of this.
@tvansteenburgh
Member

tvansteenburgh commented Aug 7, 2017

@AakashKT Can we close this and just use AakashKT/ovn-kubernetes-charm#3 ?
@AakashKT

AakashKT commented Aug 12, 2017

@tvansteenburgh Yeah! Sorry, I just saw you reply here.

@AakashKT AakashKT closed this Aug 12, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment