A Service Mesh in Go
Learning distributed systems through building production-grade infrastructure
Ananse is a service mesh built in Go, named after the Akan folktale spider known for wisdom and cleverness. It provides transparent traffic interception, load balancing, and observability for microservices.
| Mode | Purpose | Use Case |
|---|---|---|
| Sidecar | Transparent proxy using iptables | Injected into pods/containers, intercepts all traffic |
| Gateway | Reverse proxy with routing | Edge proxy, API gateway, explicit proxying |
┌─────────────────────────────────────────────────────────────┐
│ Control Plane │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ gRPC │ │ Webhook │ │ Config Watcher │ │
│ │ Server │ │ (K8s) │ │ (File/K8s) │ │
│ └─────────────┘ └─────────────┘ └─────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
│
│ gRPC Stream
▼
┌─────────────────────────────────────────────────────────────┐
│ Data Plane │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Sidecar Proxy │ │
│ │ ┌───────────┐ ┌───────────┐ │ │
│ │ │ Inbound │ │ Outbound │ │ │
│ │ │ :15006 │ │ :15001 │ │ │
│ │ └───────────┘ └───────────┘ │ │
│ │ ▲ │ │ │
│ │ │ iptables REDIRECT │ SO_ORIGINAL_DST│ │
│ │ │ ▼ │ │
│ │ [External Traffic] [Original Destination] │ │
│ └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
- Transparent Proxying: iptables-based traffic interception (sidecar mode)
- Load Balancing: Round-robin and least-connections algorithms
- Circuit Breaker: Automatic failure detection with Open/HalfOpen/Closed states
- Health Checking: Active and passive health monitoring with exponential backoff
- Automatic Sidecar Injection: MutatingWebhook injects proxy into annotated pods
- Namespace Exclusions: Skips system namespaces to prevent deadlocks
- Security Hardened: Non-root containers, dropped capabilities, read-only filesystem
- Kubernetes: Native service discovery via EndpointSlices
- Consul: Watches Consul catalog for service changes
- File-based: Static configuration from YAML/JSON files
- Prometheus Metrics: Request counts, latencies, circuit breaker states
- Distributed Tracing: OpenTelemetry integration with Tempo/Jaeger
- Structured Logging: JSON logs with trace correlation
Prerequisites: Helm 3.0+, kubectl, Kubernetes 1.19+
# 1. Add the Helm repo
helm repo add ananse https://ananselabs.github.io/ananse
helm repo update
# 2. Generate TLS certs for the mutating webhook
bash <(curl -sL https://raw.githubusercontent.com/ananselabs/ananse/main/scripts/generate-certs.sh)
# 3. Install Ananse
helm install ananse ananse/ananse \
--set-file caBundle=./ca.crt.b64 \
-n ananse-system --create-namespace
# 4. Label your namespace — injection activates on next pod create
kubectl label namespace default ananse.io/inject=enabled
# 5. Restart your workloads to get sidecars
kubectl rollout restart deployment -n defaultVerify injection:
kubectl get pod -n default -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].name}{"\n"}{end}'
# Each pod should show: <app-name> ananse-proxyWith tracing (Tempo/Jaeger):
helm install ananse ananse/ananse \
--set-file caBundle=./ca.crt.b64 \
--set observability.tracing.enabled=true \
--set observability.tracing.endpoint=tempo.monitoring.svc:4317 \
-n ananse-system --create-namespaceWith Prometheus Operator (PodMonitor auto-scraping):
If your cluster has Prometheus Operator, the PodMonitor is deployed automatically when serviceMonitor.enabled: true (default). Prometheus will discover all injected sidecars across all namespaces via the sidecar.ananse.io/status: injected label.
If your Prometheus CR doesn't watch all namespaces, apply RBAC for cross-namespace discovery:
# Give Prometheus SA cluster-scope pod read access
kubectl apply -f https://raw.githubusercontent.com/ananselabs/ananse/main/k8s/prometheus-rbac.yamlSee ananse-chart/README.md for full configuration options.
Prerequisites: kubectl, kind/minikube, Docker
# Build and push images
docker build -f docker/Dockerfile.controlplane -t anthony4m/ananse-controlplane:v1 .
docker build -f docker/Dockerfile.proxy -t anthony4m/ananse-proxy:v1 .
docker build -f docker/Dockerfile.init -t anthony4m/ananse-init:v1 .
docker push anthony4m/ananse-controlplane:v1
docker push anthony4m/ananse-proxy:v1
docker push anthony4m/ananse-init:v1
# Generate TLS certificates
./scripts/generate-certs.sh
# Deploy to cluster
kubectl apply -f deploy/namespace.yaml
kubectl apply -f deploy/injector-config.yaml
kubectl apply -f deploy/rbac.yaml
kubectl apply -f deploy/webhook-deployment.yaml
kubectl apply -f deploy/webhook-service.yaml
kubectl apply -f deploy/webhook-config.yaml
# Test injection - deploy a pod with the annotation
kubectl run test-app --image=nginx \
--annotations="sidecar.ananse.io/inject=true"
# Verify sidecar was injected
kubectl get pod test-app -o jsonpath='{.spec.containers[*].name}'
# Should show: nginx ananse-proxySidecar mode - transparent proxying:
# docker-compose.yml
version: '3.8'
services:
sidecar:
image: anthony4m/ananse-proxy:v1
environment:
- ANANSE_MODE=sidecar
cap_add:
- NET_ADMIN
- NET_RAW
volumes:
- ./scripts/iptables-init.sh:/iptables-init.sh
entrypoint: ["/bin/sh", "-c", "/iptables-init.sh && /ananse-proxy"]
my-app:
image: nginx
network_mode: "service:sidecar"
depends_on:
- sidecarGateway mode - reverse proxy:
version: '3.8'
services:
controlplane:
image: anthony4m/ananse-controlplane:v1
ports:
- "50051:50051"
volumes:
- ./config:/config
proxy:
image: anthony4m/ananse-proxy:v1
environment:
- ANANSE_MODE=gateway
- CONTROL_PLANE_ENDPOINT=controlplane:50051
ports:
- "8080:8080"For non-Kubernetes environments using Consul for service discovery:
# Run control plane with Consul discovery
./controlplane -consul -consul-addr consul.example.com:8500
# With tag filtering (only services tagged "ananse")
./controlplane -consul -consul-addr consul.example.com:8500 -consul-tag ananse
# Run proxy in gateway mode
./proxy# Run control plane with Kubernetes discovery
go run ./controlplane/cmd/ -k8s
# Run control plane with Consul discovery
go run ./controlplane/cmd/ -consul -consul-addr localhost:8500
# Run control plane with file-based config
go run ./controlplane/cmd/ -config-path ./config -config-name services
# Run proxy in gateway mode (default)
go run ./proxy/
# Run proxy in sidecar mode (requires Linux + iptables)
ANANSE_MODE=sidecar go run ./proxy/| Variable | Default | Description |
|---|---|---|
ANANSE_MODE |
gateway |
Operating mode: gateway or sidecar |
SIDECAR_IMAGE |
anthony4m/ananse-proxy:v1 |
Image for injected sidecars |
INIT_IMAGE |
anthony4m/ananse-init:v1 |
Image for init container |
PROXY_PORT |
15001 |
Outbound listener port |
INBOUND_PORT |
15006 |
Inbound listener port |
PROXY_UID |
1337 |
UID for sidecar (iptables bypass) |
ANANSE_TRACING_ENABLED |
"" |
Set to "false" to disable tracing entirely |
OTEL_EXPORTER_OTLP_ENDPOINT |
localhost:4317 |
OTLP gRPC endpoint for traces |
FILTER_HEALTH_CHECKS |
"false" |
Set to "true" to drop successful health probe spans from Tempo |
| Annotation | Values | Description |
|---|---|---|
sidecar.ananse.io/inject |
true/false |
Enable/disable sidecar injection |
sidecar.ananse.io/status |
injected |
Set automatically after injection |
Injection is automatically skipped for:
kube-systemkube-publiccert-managerananse-system
ananse/
├── controlplane/
│ ├── cmd/
│ │ ├── main.go # Control plane entry point
│ │ └── server.go # gRPC server
│ ├── injector/
│ │ ├── injector.go # Sidecar injection logic
│ │ └── webhook.go # Webhook server
│ ├── consul-client.go # Consul service discovery
│ ├── file-client.go # File-based config watcher
│ └── k8s-client.go # K8s service discovery
│
├── pkg/proxy/
│ ├── listener.go # Inbound/outbound listeners
│ ├── originaldst.go # SO_ORIGINAL_DST syscall
│ ├── handler.go # Request handling
│ ├── backend.go # Backend pool management
│ ├── health.go # Health checking
│ └── circuit.go # Circuit breaker
│
├── proxy/
│ └── main.go # Proxy entry point
│
├── ananse-chart/ # Helm chart
│ ├── Chart.yaml # Dependencies (observability stack)
│ ├── values.yaml # Configuration
│ └── templates/ # K8s manifests
│
├── scripts/
│ ├── iptables-init.sh # Traffic interception rules
│ └── generate-certs.sh # TLS certificate generation
│
├── deploy/ # Raw K8s manifests (use Helm instead)
│ ├── namespace.yaml
│ ├── rbac.yaml
│ ├── injector-config.yaml
│ ├── webhook-deployment.yaml
│ ├── webhook-service.yaml
│ └── webhook-config.yaml
│
└── docker/
├── Dockerfile.controlplane
├── Dockerfile.proxy
└── Dockerfile.init
| Mode | Requirements |
|---|---|
| Sidecar | Linux, iptables, NET_ADMIN capability |
| Gateway | Any OS (Linux, macOS, Windows) |
| Control Plane | Any OS |
The sidecar mode uses SO_ORIGINAL_DST to recover original destinations after iptables REDIRECT. This is a Linux-only syscall.
Each sidecar proxy exposes Prometheus metrics on port 15021 at /metrics. The controlplane does not expose metrics.
# Scrape a sidecar directly
curl http://<pod-ip>:15021/metricsKey metrics:
ananse_sidecar_connections_active- Active connections (gauge, by direction)ananse_sidecar_connections_total- Total connections since start (counter)ananse_sidecar_request_duration_seconds- Latency histogramananse_http_requests_in_flight- Requests currently being processedananse_circuit_breaker_failures_total- Circuit breaker trip countananse_sidecar_connections_by_tls_total- Connections by TLS mode
Prometheus scraping:
| Setup | How |
|---|---|
| Prometheus Operator installed | PodMonitor auto-deployed by Helm chart (serviceMonitor.enabled: true) |
| Standalone | kubectl apply -f k8s/ (includes Prometheus with kubernetes_sd scraping port 15021) |
The PodMonitor uses namespaceSelector: any: true to discover injected pods across all namespaces. It matches pods by label sidecar.ananse.io/status: injected (set automatically on injection) on the named port ananse-admin.
Promtail ships pod logs to Loki with namespace, pod, and container labels.
# Deploy standalone observability stack
kubectl create ns monitoring
kubectl apply -f k8s/The sidecar sends traces via OpenTelemetry OTLP gRPC (port 4317) to any compatible backend (Tempo, Jaeger).
Configure via Helm values:
observability:
tracing:
enabled: "true"
endpoint: "tempo.monitoring.svc.cluster.local:4317"
# Optional: drop successful health check spans from Tempo.
# Only errored health probes (5xx / transport failure) are exported.
# Reduces Tempo noise from kubelet liveness/readiness probes.
filterHealthChecks: falseOr set directly on the injector ConfigMap:
kubectl patch configmap ananse-injector-config -n ananse-system \
--type merge -p '{"data":{"TRACING_ENABLED":"true","OTEL_ENDPOINT":"tempo.monitoring.svc:4317"}}'Health check trace filtering (filterHealthChecks: true): kubelet probes (/management/health, /healthz, /ready) fire every few seconds per pod. By default these generate a trace each. Enabling this filter drops successful health probe spans at the sidecar before they reach Tempo — only spans where the probe returned 5xx are exported, which is exactly when you need the trace for debugging.
The Ananse dashboard is auto-provisioned on startup — no manual import needed. Datasources provisioned automatically: Prometheus, Loki (with TraceID links to Tempo), Tempo (with log links to Loki).
With ingress (recommended for persistent access — add a host entry to your ingress pointing to jhipster-grafana:3000):
https://grafana.your-domain.io
With port-forward (standalone / no ingress):
kubectl port-forward svc/grafana 3000:3000 -n monitoring
# Open http://localhost:3000Login: admin / jhipster (from jhipster-grafana-credentials secret)
Before go-live, stress the mesh with the bundled k6 test suite (kubernetes/load-test/ in your deployment repo). Tests run as a Kubernetes Job inside the cluster — traffic goes through the real pod network → iptables → sidecar path.
Three scenarios:
| Scenario | Duration | VUs | Purpose |
|---|---|---|---|
smoke |
3 min | 5 | Sanity check — confirm all endpoints reachable |
ramp |
19 min | 0→200 | Find capacity ceiling |
soak |
7h | 50 | Overnight — goroutine/memory leak detection |
Setup (one time):
# Store credentials
kubectl create secret generic k6-credentials \
--from-literal=username=YOUR_USER \
--from-literal=password=YOUR_PASS \
-n default
# Load the test script
kubectl create configmap k6-test-script \
--from-file=test.js=kubernetes/load-test/k6-test.js \
-n default --dry-run=client -o yaml | kubectl apply -f -Run:
# Smoke (edit K6_SCENARIO=smoke in k6-job.yaml first)
kubectl apply -f kubernetes/load-test/k6-job.yaml
kubectl logs -f job/k6-load-test -n default
kubectl delete job k6-load-test -n default
# Ramp (K6_SCENARIO=ramp)
# ... repeat
# Soak overnight (K6_SCENARIO=soak is the default)
kubectl apply -f kubernetes/load-test/k6-job.yaml
# Morning check
kubectl logs job/k6-load-test -n default | tail -40What to watch in Grafana during soak:
ananse_sidecar_connections_active— should plateau, not drift upward (goroutine leak)kubectl top pods— sidecar memory should stay flat- p99 latency — stable means the cluster is healthy; climbing means something is accumulating
The k6 pod runs outside the mesh (sidecar.ananse.io/inject: "false") but every service it hits IS in the mesh — so you're testing the real inbound proxy path on every request. Auth goes through Keycloak the same way the frontend does.
Ananse v0.3.3 was validated under sustained load in a real production Kubernetes cluster (DigitalOcean, 4 Spring Boot microservices, Prometheus Operator):
| Metric | Result |
|---|---|
| Total requests processed | 2,634,971 |
| Sustained throughput | 87 req/s for 8+ hours |
| Concurrent users | 50 |
| Sidecar crashes | 0 |
| Pod restarts | 0 |
| Memory growth over 8h | flat (no goroutine leak) |
ananse_sidecar_connections_active |
Stable plateau throughout |
The mesh was transparent — errors observed during testing originated from application-layer configuration (Spring Cloud Gateway connection pool), not the proxy. Confirmed via per-pod Prometheus metrics during the live test.
MIT License