Make it easy to run Kubernetes on top of the Kubelet (aka self-hosting)

[smarterclayton]

Pulling this out from #167

I'd love to start using the kubelet in a more "static" mode to handle the master containers. That would consist of getting the kubelet running on the master (with out looking an etcd) and then have it run a pod for the master components by reading out of a manifest file on disk.

Something I'd be happy to work on

[jjhuff]

The kubelet already supports reading config from a file, does that work?

[lavalamp]

Yeah, I think the work to do here is:

1. Finish up jbeda's work to dockerize all our components.
2. Adjust our startup scripts to use kubelet to launch apiserver/replication-controller via the config file.

[smarterclayton]

As a further item (possibly as a separate issue), being able to change code on your devenv and either one-line a command (hack/update-local) or have the image/source automatically reloaded would be valuable for reducing code-test loop time. Probably the former though

[bgrant0607]

/cc @proppy, since we were discussing this recently.

The Dockerization issue is #19.

[bgrant0607]

Self-hosting proposal:

Kubernetes components are just applications, too. They have much the same needs for packaging/image-building, configuration, deployment, auth[nz], naming/discovery, process management, and so on. Leveraging the systems/APIs we build for our users avoids needing to build and maintain 2 systems to do the same thing, and helps us to eat our own dogfood.

The recipe for building a bootstrappable system is fairly straightforward:

• Minimize the number of dependencies, particularly those required for steady-state operation,
• Stratify the dependencies that remain via principled layering, and
• Break any circular dependencies by converting hard dependencies to soft dependencies.
  • Also accept that data from other components from another source, such as local files, which can then be manually populated at bootstrap time and then continuously updated once those other components are available.
  • Make all state rediscoverable and/or reconstructable.
  • Make it easy to run temporary, bootstrap instances of all components in order to create the runtime state needed to run the components in the steady state; use a lock (master election for distributed components, file lock for local components like Kubelet) to coordinate handoff. We call this technique "pivoting".
  • Have a solution to restart dead components. For distributed components, replication works well. For local components such as Kubelet, a process manager or even a simple shell loop works.

A more concrete sketch:

Master components:

1. Startup instances of the components all on the same host
2. Create the services and replication controllers needed to run the components "for real" via kubectl
3. Either kill the bootstrap instances or enable them to be adopted by the replication controllers submitted through the API. Replication controller deliberately does not care how the pods it controls were created.

The only tricky part is transferring the etcd state. We don't have great solutions for stateful services yet, in general (#260, #1515), but if just running on the same host or same PD, etcd would just pick it up from the volume. The data could also be replicated to other instances.

A step towards this could be to run the components on Kubelet using a local pod file.

Kubelet:

1. Kubelet should support caching the pods read from the registry in a hostdir volume on local disk (Kubelet checkpointing or something #489). This cache would be updated when updates were received from the apiserver or etcd.
2. Distribute an old, fallback/bootstrap Kubelet to the nodes any way we like: baked into the host/VM image, Salt, Bosh, rsync, whatever.
3. Use the runonce feature of Kubelet to use the first-step bootstrap Kubelet to pull and start a containerized, kube-ified Kubelet. We may want to extend runonce to read from apiserver as well as from local sources.
4. Leverage Docker restarts to restart Kubelet when it dies (Adopt Docker's own container restarts #816).
5. Use the per-node controller (Daemon (was Feature: run-on-every-node scheduling/replication (aka per-node controller or daemon controller)) #1518, WIP - a Per-Node controller #2491) to deploy new Kubelet images and/or configuration. The new Kubelet must be started before the old one terminates.

[proppy]

A step towards this could be to run the components on Kubelet using a local pod file.

There was an example for that for 0.4 here: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/build/run-images/bootstrap/run.sh#L22

Use the runonce feature of Kubelet to use the first-step bootstrap Kubelet to pull and start a containerized, kube-ified Kubelet. We may want to extend runonce to read from apiserver as well as from local sources.

Last time I tried you couldn't run a kubelet in a pod w/ kubelet --runonce, because the child kubelet would kill its parent (because of the lack of namespace, he thought it was a leftover k8s container it didn't know about)