This project is no longer maintained
We're keeping this repository live as it pointed the way to now more common idea of being able to push to a live environment every single version (commit) of an application. Grand Central is a really powerful way of fast forwarding/rewinding through the development of a project, and see your committed changes exposed live in a running system. GC has been one of the most interesting efforts I've participated in to build better quality software! - Eric Pugh (2/1/19)
Quepid's automated review deployment tool. Gut-check in the cloud
Grand Central is a tool for automated deployment and cleanup of developer review environments / containers. Requests are parsed and routed based on their URL structure. If the target container exists the request is proxied along. If not Grand Central will spin up a container and forward the request once it comes online.
The appropriate container is determined by parsing the first part of the domain name.
*.review.quepid.com in DNS is directed at the Grand Central service. The application parses the domain name to retrieve the appropriate Git version to deploy.
http://db139cf.review.quepid.com/secure would route to a container running version
db139cf if it exists.
When a request is received by the system the following processing takes place.
- Validate the git version supplied in the
Hostheader. Does it match a valid short hash signature?
- Verify if the version is currently running
- If so, proxy the request
- If not, continue
- Verify version exists in Container Registry. *We can't deploy a version which doesn't exist.
- If so, continue
- If not, 404
- Create Pod containing app version, database, and loader
- Create Service for Pod (routes requests internally within the cluster)
- Proxy the original request
Note that all requests will have some metrics stored to determine activity for a give pod. This is useful when reaping old pods.
BUT WAIT! What happens when two requests come in for the same version?
Easy, state is maintained within an Atomically accessed Map. As soon as a version is detected to not be running we instantiate it before releasing the lock. If a version exists, but isn't running we pause the request until the creation process is complete on another thread.
Should a container not be running in the Kubernetes cluster when the request is performed the following process starts.
- Create a Pod with the following components
- Rails Application
- MySQL Instance
- Data dump loader (golden review image stored block store)
- Create a service referencing that pod
Every hour a janitorial task runs which cleans up pods / services that have not received a request in the past x seconds. This keeps cluster resources available. Since pods are trivial to create they can be re-instantiated easily.
There is a hard limit to the number of simultaneous pods running on the cluster. To prevent Denial of Service by our own team the maximum number of review environments is capped at y. Should a new pod be requested when the currently running count is already maxed the pod with the oldest most recent request will be removed.
- Persistent hashes - the ability to mark a version as persistent. This prevents the janitor from reaping the pod.
- Admin interface - allow a RESTful interface to manage pods. List all pods, create a new one, delete an old one out of the janitorial process etc.
Local K8S Route
Adds a route into the OS X routing table to forward all requests to the internal K8S IP space through the K8S virtual machine's IP.
sudo route add -net 10.2.47 172.17.4.99
Local K8S Certificate
The K8S cluster has a self-signed SSL certificate. It must be added to a keystore as a trusted certificate before requests are permitted.
brew install openssl echo -n | /usr/local/Cellar/openssl/1.0.2e/bin/openssl s_client -connect <kubernetes master ip>:443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > config/local.pem keytool -importkeystore -srckeystore $JAVA_HOME/jre/lib/security/cacerts -destkeystore config/grandcentral.jks -srcstorepass changeit -deststorepass changeit ho "yes" | keytool -import -v -trustcacerts -alias local_k8s -file k8s/local.pem -keystore config/grandcentral.jks -keypass changeit -storepass changeit
echo -n | openssl s_client -connect 172.17.4.99:443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > config/local.pem keytool -importkeystore -srckeystore $JAVA_HOME/jre/lib/security/cacerts -destkeystore config/grandcentral.jks -srcstorepass changeit -deststorepass changeit echo "yes" | keytool -import -v -trustcacerts -alias local_k8s -file k8s/local.pem -keystore config/grandcentral.jks -keypass changeit -storepass changeit
Logging in to GCR.io
Setup a GCP service account and download it's JSON key file
docker login -e email@example.com -u _json_key -p "$(cat keyfile.json)" https://gcr.io
Building the Docker Container
mvn clean package docker build -t grand-central .