Private cluster support

[andrewosh]

As 👍 'd by a number of people in the Gitter (@rahuldave, @mcburton, and @ctb, in yesterday's discussion), there's a variety of use-cases for Binder that can't be supported by the public cluster due to resource limitations, and where private cluster support becomes critical. Some examples:

1. Supporting a large class, where a large number of slots (approaching the number available on the public cluster!) need to be reliably available.
2. Giving substantial resources to external services, such as giving each deployment access to 10 Spark workers (the public cluster only creates a single Spark worker/master, which is only useful for demo purposes)
3. Needing finer control over hardware details (node size, storage...), such as if your binder needs to take full advantage of a multi-core environment (@mrocklin)
4. Testing. It hasn't been much of an issue yet, but right now there's only a single test cluster to run PRs on....'nuff said.

While supporting private clusters on the Google Compute Engine would be very straightforward, as that's what the public cluster is running on, there are a few issues to figure out before this will be possible on a wider variety of cloud providers, or on internal clusters:

1. GCE-specific features - We've tried to minimize the number of GCE-specific features we use, but we are using Google's container registry for image hosting, and GCE's load balancers for supporting our external Kubernetes services (the proxy lookup/registration services). The image registry wouldn't be an issue if Run a Docker registry kubernetes/kubernetes#1319 were completed, but spinning up a private registry isn't much trouble either (http://kubernetes.io/v1.0/docs/user-guide/images.html#using-a-private-registry). The load balancers are currently not being used, but as cluster size increases the configurable-http-proxy might face websocket issues (@rgbkrk), and load balancing between multiple proxies might be necessary.
2. Startup scripts - The Kubernetes launch scripts vary greatly across cloud providers, and Binder might want to add light modifications to these scripts, like we currently do for GCE. This would be very cumbersome to manage as the number of supported cloud providers grows, so one alternative is to pass Kubernetes setup to the user, like:
   1. Download Kubernetes, modify the startup scripts to your liking (setting node size, etc.)
   2. Run the Binder docker container, which on launch will ask for a Kubernetes directory and cloud provider credentials (if necessary)
   3. Call binder cluster start, which implicitly calls kube-up, and we assume everything's been configured properly.
3. Proxying - Right now, all GitHub badges will redirect to app.mybinder.org, which points directly to the proxy lookup service. Other possible way of handling proxying:
4. mybinder.org maintains an index of private clusters, and app.mybinder.org routes requests to the correct proxy lookup service. This is more complicated, but there are definitely benefits to a federated approach.
5. All private clusters are completely independent, and when the API server image is launched, it asks for an IP or a URL for proxy lookup service.

I'm probably (definitely) overlooking a few things, so any feedback/thoughts would be appreciated!

[ctb]

@andrewosh, thanks -- I'm looking for a place to dig my claws into this, and it would be most helpful if there was pretty explicit install instructions for me to play with, so I can get a better sense of how all the puzzles pieces fit together. (I'm happy to go learn GCE is that's where those install instructions work.) Note that I'm completely unfamiliar with kubernetes so that's presumably part of the learning curve. Guidance appreciated!