Go GRPC telemetry ingestion service and client. This is a project I decided to work on to get more experience with concepts I have been working on professionally. I wanted to prove to myself that I understood these concepts and could apply them functionally without any support. This is not a real product.
Data is passed from client <-> server via GRPC. Shared proto files are compiled in the deploy script to both server and client.
Docker images are publicly hosted on docker hub.
There are a handful of kubernetes features at play in the server. Minikube is leveraged here because I had no experience with it and thought it would be interesting to use locally (Most of my experience is with Google Kubernetes Engine). A LoadBalancer is used to expose the GRPC server to the world. TLS secrets are mounted via volumes and secrets. An initContainers is used to deploy migrations via a custom github.com/golang-migrate/migrate docker image. A PersistentVolumeClaim and PersistentVolume are used to host the postgres data. The postgres server is running within the cluster for low latency queries.
Normally to store massive amounts of telemetry data I would not have gone with a relational database. I would have chosen something like Google Cloud Datastore, Google BigQuery or Cassandra. However having worked professionally in NoSQL databases the past few years I wanted to get my hands dirty in SQL again.
Its rad.
- Document Code
- Unit Testing
- Integration Testing
- CI/CD
- Prometheus postgres_exporter (shared_preload_libraries) enable on docker
- Go Channels
- Working TLS in Kubernetes
openssl genrsa -out server.key 2048
openssl req -new -x509 -sha256 -key server.key -out server.crt -days 3650
openssl req -new -sha256 -key server.key -out server.csr
openssl x509 -req -sha256 -in server.csr -signkey server.key -out server.crt -days 3650
kubectl create secret generic tls --from-file=server.crt --from-file=server.key=server.key
minikube start
minikube stop
minikube ip
minikube status
minikube service grpc-lb --url
./deploy.sh
brew install golang-migrate
kubectl exec -it <pod> -- sh
psql -h localhost -U test --password -p 5432
create database telemetry;
// Migration create example
migrate create -ext sql -dir services/server/migrations/ -seq create_locations_table
namespace: monitoring
minikube service prometheus-service --url -n monitoring
brew install helm
helm repo add stable https://kubernetes-charts.storage.googleapis.com/
helm repo update
helm install prometheus-postgres-exporter -f helm/prometheus-postgres-exporter.yaml stable/prometheus-postgres-exporter
helm delete prometheus-postgres-exporter