Investigate k8s-in-k8s

[jhunt]

My good friend and Kubernetes expert @dmolik wants to try running the components of k8s on top of Kubernetes. The basic gist is that we move as much out from under the control of monit as possible, leaving behind just the kubelet process and whatever the containerd + runtime chosen needs. The kubelet then leverages the containerd + runtime to run Static Pods for control plane components (API server / Controller Manager / Scheduler).

I have started the k8s-in-k8s branch for this.

Upsides

1. Less stuff under the control of monit
2. More stuff visible from an admin with nothing but kubectl (i.e. no BOSH access necessary to troubleshoot)
3. We're already relying on the kubelet to properly start, monitor, and tear down scheduled Pods, so why not leverage it for other things.
4. It might allow for customization of the runtime-y bits of k8s by letting operators swap out the images that run for control plane components?

Downsides

1. The approach relies HEAVILY on pre-start / post-deploy hooks in BOSH's lifecycle, with minimal usage of the "meat" of a BOSH release.
2. Operators accustomed to troubleshooting BOSH deployments will be out of their element.
3. Requires pulling container images from somewhere, which may prove problematic for air-gapped environments.
4. Unless we can embed / re-inflate these images to mitigate (3) above, we are only shipping half of the software necessary to make the cluster operate, in the release. This is contrary to what BOSH stands for.