Colossus — An example microservice architecture for Kubernetes powered by Bazel, Go, Java, Docker, and gRPC
This is an example project that combines several cloud native technologies that I really like and have been meaning to get working in a meaningful way:
- The Bazel build tool
- Kubernetes and Minikube
- Protocol Buffers and gRPC
- Docker
Colossus is basically a microservice architecture consisting of three services:
Service | Where the code lives | Language |
---|---|---|
An HTTP service that takes web requests. This is the entry point into the backend services. | web |
Go |
An authentication/authorization service | auth |
Go |
A "data" service that handles data requests | data |
Java |
A "user info" service that doesn't do anything interesting yet, but it works and it's in C++! | userinfo |
C++ |
What do these services actually do?
- The web service requires a
Password
header and aString
header. ThePassword
header is used for authentication and theString
header is used as a data input. You need to makePOST
requests to the/string
endpoint. - The auth service verifies the password passed to the web service. There is only one password that actually works:
tonydanza
. Use any other password and you'll get a401 Unauthorized
HTTP error. - The data service handles words or strings that you pass to the web service using the
String
header. The data service simply capitalizes whatever you pass via that header and returns it.
Wait a second, these services don't do anything meaningful! Nope, they sure don't. But that's okay because the point of this project is to show you how to get the basic (yet not-at-all-trivial) plumbing to work. Colossus is a boilerplate project that's meant as a springboard to more complex and meaningful projects.
Getting all of these technologies to work together was a real challenge. I had to dig through countless GitHub issues and dozens of example projects to make all these things work together. I'm offering this repo as a starter pack for other people with a Bazel monorepo targeting Kubernetes.
In order to run Colossus locally, you'll need to run a local Docker registry. If your Docker daemon is started up, you can run the local registry like this:
$ docker run -d -p 5000:5000 --restart=always --name registry registry:2
# Alternatively
$ make docker-registry
Once the registry is running, you'll need to start up Minikube in conjunction with an insecure registry (i.e. the Docker registry running locally):
$ minikube start --insecure-registry localhost:5000
# Alternatively
$ make minikube-start
Once Minikube is up and running (use minikube status
to check), you'll need to enable the ingress add-on and set the Docker environment:
$ minikube addons enable ingress
# Alternatively
$ make minikube-setup
# Set up the Docker environment for Minikube
$ eval $(minikube docker-env)
Now Minikube is all set. The one required dependency for Colossus is a Redis cluster. To run a one-node Redis cluster in Kubernetes-on-Minikube (configuration in k8s/redis.yaml
):
$ kubectl apply -f k8s/redis.yaml
# Alternatively
$ make k8s-redis-deploy
Once the Redis pod is up and running (you can check using kubectl get pods -w -l app=redis
), you need to set a password for the authentication service. To set the password as tonydanza
(which the later curl examples assume):
$ REDIS_POD=$(kubectl get pods -l app=redis -o jsonpath='{.items[0].metadata.name}')
$ kubectl exec -it $REDIS_POD -- redis-cli SET password tonydanza
OK
# Alternatively
$ make redis-set-password
You can then verify that the password has been set throughout the cluster by running a GET password
query from a different pod in the cluster:
$ kubectl exec -it $REDIS_POD -- redis-cli GET password
"tonydanza"
# Alternative
$ make redis-get-password
Now that Redis is all set up, you can deploy Colossus using one command:
$ make deploy
Okay, that's suspiciously magical so I'll break it down into pieces. make deploy
will do the following:
- Build Docker images for each service using Bazel
- Upload those images to the local Docker registry (in each case the image will be run using the
--norun
flag, which will upload the images without actually running them) - Apply the Kubernetes configuration in
k8s/colossus.yaml
, which has KubernetesService
andDeployment
configurations for each of the three services (each of which runs on three instances) as well as anIngress
configuration for access to the HTTP service
Run kubectl get pods
and if all of the pods have the status Running
then Colossus is ready to take requests!
In order to access the web service, you'll need to get an IP address for Minikube. I recommend setting it as an environment variable:
$ MINIKUBE_IP=$(minikube ip)
Now let's make a request to our web service:
$ curl -i -XPOST $MINIKUBE_IP/string
HTTP/1.1 401 Unauthorized
Server: nginx/1.13.12
Date: Sun, 27 May 2018 22:30:02 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 32
Connection: keep-alive
X-Content-Type-Options: nosniff
You cannot access this resource
Oops! We need to specify a password using the Password
header. The password that we supply will be sent to the auth service for verification.
$ curl -i -XPOST -H Password:foo $MINIKUBE_IP/string
Oops! Denied again. Remember: the only password that works is tonydanza
. Let's try this again:
$ curl -i -XPOST -H Password:tonydanza $MINIKUBE_IP/string
HTTP/1.1 400 Bad Request
Server: nginx/1.13.12
Date: Sun, 27 May 2018 22:33:31 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 50
Connection: keep-alive
X-Content-Type-Options: nosniff
You must specify a string using the String header
Oops! Forgot to specify a string using the String
header, which means that our data service isn't even being access. Let's supply a string:
$ curl -i -XPOST -H Password:tonydanza -H String:"Hello, world" $MINIKUBE_IP/string
HTTP/1.1 200 OK
Server: nginx/1.13.12
Date: Sun, 27 May 2018 22:50:19 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 12
Connection: keep-alive
HELLO, WORLD%
Success! Our Password
header is authenticating us via the auth service and the data service is handling our data request the way that we would expect. Colossus is a rousing success, folks 👍
Just to show that our auth service is working correctly (no false positives!), let's change the password and make another request to our web service using the old password:
$ kubectl exec -it $REDIS_POD -- redis-cli SET password somethingelse
$ curl -i -XPOST -H Password:tonydanza -H String:"This should fail" $MINIKUBE_IP/string
Unauthorized! Ruh roh. Now let's use the proper password:
$ curl -i -XPOST -H Password:somethingelse -H String:"This should work now" $MINIKUBE_IP/string
Success 😎.
Create a config map for Prometheus using the prometheus.yml
configuration file:
$ kubectl create configmap prometheus-config --from-file=configs/prometheus.yml
That file contains the proper configs to make Prometheus periodically scrape metrics from the web and auth services. Once the config has been uploaded, you can start up both Prometheus and Grafana in the Kubernetes cluster:
$ kubectl apply -f k8s/monitoring.yaml
# Alternative
$ make k8s-monitoring-deploy
This is a humble start but I'd like to expand it a great deal in the future. In particular I'd like to add:
- A service mesh like Conduit or Istio.
- A Helm chart
- Some "real" services that do meaningful things, like interact with databases running on k8s or even a cloud service like Google BigTable.
- Real REST capabilities. Right now our web service doesn't do anything cool. At the very least it should provide some interesting CRUD operations.
- More languages. Right now Go and Java are pretty much the only languages that can be easily incorporated into a gRPC-plus-Bazel setup. I'm sure that support for Python, C++, and others is on the way, and I'll use those capabilities as the opportunity arises.
- Componentize service building. Right now each service is its own self-contained universe. I'd like to create a reusable service abstraction (or re-use abstractions built by others) for creating new services, especially more robust configuration management.
- Integration testing for specific services and the whole thing
The good news is that the hard part---especially getting Bazel to build the right things and Kubernetes to use a local image registry---is already behind me, so adding new services is fairly trivial.