[WIP] Kubernetes integration

[Zogg]

[Work in Progress]
Kubernetes integration into Mantl. This is an integration of kubernetes-ansible.

DNS

We run etcd/skydns on every node. DNS entries are managed by kube2sky. DNS queries are routed through dnsmasq>skydns>etcd on local node to decrease dns network traffic.

Networking

Kubernetes may employ either Flannel or Calico for containter networking.

Calico

• Pitfall: Currently calico cannot assign IP to container/pod which has been launched from manifest file. Only pods launched through kalicoctl/kubeapi gets an IP assigned. Track the bug in kubernetes repo and calico-kubernetes repo.

Flanneld

• Blocker: flanneld cannot probperly start because of race condition between services. Flannel needs to start before Docker, Etcd must be started before Flannel, but currently it is started as Docker container. Hence the lockdown. Track the issue.

[stevendborrelli]

@Zogg what is the impact of the calico issue? How often do we deploy via manifest vs. the API.

[langston-barrett]

Related: #643

[eirslett]

Is it possible to run Kubernetes on Mantl, on top of Mesos instead of using the bundled Kubernetes scheduler? (Using the Kubernetes mesos framework)
In which case, it could probably be run from Marathon instead of as a system process?

[stevendborrelli]

@eirslett The general feeling is that it is less complex to run kubernetes on its own than by using the k8s-mesos framework. Mantl support for k8s experimental at this point, and we are going to keep it separate from Mesos installs until we can integrate it better.

The idea is to migrate K8s to use calico and consul, so that we have common networking and service discovery for different frameworks.

[ryane]

could we use an official package for this or move it to ciscocloud?

[Zogg]

Yes, I'm on to it.

[Zogg]

Moved into asteris account.

[ryane]

I was able to successfully run the kubernetes playbook.

but the changes in the etcd and calico roles make them unusable on a non-kubernetes cluster.

[stevendborrelli]

@Zogg @avnik we need to get the etcd playbook aligned so the same one is used for kubernetes and calico.

[Zogg]

@stevendborrelli The roles are fixed and OK.

[ryane]

etcd playbook does seem to work well in a non-kubernetes mantl cluster. and survives restarts so this fixes #737.

calico is still problematic on a non-kubernetes cluster. no containers can be launched with mesos/marathon when calico is enabled.

[ryane]

we're going to address the calico problems in a separate issue