Demo setup for Grafana Loki in microservices mode using MinIO as storage backend.
The following pre-requisites must be installed:
Run ./setup.sh
.
This will create a kind cluster and install all services.
A kind cluster with one master and three worker nodes is created. The cluster uses a local pull-through Docker registry for faster image pulling on the various kind nodes and to avoid rate-limiting issue with Docker hub.
The setup creates an almost production-ready, highly available, distributed Loki setup. However, there are no resources configured for pods in this setup because we have to be able to spin up a lot of pods on a local Docker setup. You may want to tweak your Docker configuration increasing the number of CPUs and the amount of memory it can use. In a production setup resources should definitely be configured.
Port 30123 is mapped to the same port on the master node in order to enable ingress.
- ingress-nginx
- metrics-server
- kube-prometheus-stack
- minio
- loki-distributed
- Heptio Eventrouter
- promtail
- loki-canary
- test-logger
- gateway
- ingester
- distributor
- querier
- query-frontend
- table-manager
- compactor
- ruler
- memcached-chunks
- memcached-frontend
- memcached-index-queries
Single-Store (boltdb-shipper) is used for index storage with MinIO as storage backend.
-
Grafana: http://localhost:30123/grafana
-
Prometheus: http://localhost:30123/prom
-
Alertmanager: http://localhost:30123/alerts
-
MinIO: http://localhost:30123/minio
MinIO access key and secret key for logging into the UI can be retrieved with the following commands.
kubectl get secret -n minio minio-credentials -o jsonpath="{.data.accesskey}" | base64 --decode | xargs echo kubectl get secret -n minio minio-credentials -o jsonpath="{.data.secretkey}" | base64 --decode | xargs echo
Two instances of the test-logger chart are installed which simply log test messages for demo purposes, one in JSON format and one in logfmt format. The Promtail configuration demonstrates how to extract the log level as label.
Besides a number of dashboards that come with the kube-prometheus-stack, the following dashboards are provided for Loki:
- Loki Operational (adapted from Grafana's Tanka setup): http://localhost:30123/grafana/d/loki-operational/
- Loki Canary: http://localhost:30123/grafana/d/loki-canary/
- Kubernetes Events: http://localhost:30123/grafana/d/kubernetes-events/