Kubernetes Metrics Agent

A lightweight agent for collecting essential Kubernetes metrics, similar to kubectl top but with export capabilities and scheduling.

Features

• Minimalist: Only collects essential metrics (CPU, memory, basic cluster information)
• Secure: Minimal RBAC permissions, runs as non-root user
• Flexible: Output in JSON, HTTP POST or Prometheus format
• Lightweight: Optimized Docker image < 20MB
• Configurable: Can run as deployment, cronjob or single command
• Prometheus Integration: Native support using prometheus/client_golang
• HTTP Server: Endpoints /health, /api/metrics and /metrics (Prometheus)
• Async Execution: Metrics collection in background while serving HTTP

Collected Metrics

Cluster Information

• API Server URL
• Kubernetes version
• Number of nodes

Node Metrics

• CPU usage (cores)
• Memory usage (bytes/GB)
• Node name

Pod Metrics

• CPU usage per pod
• Memory usage per pod
• Namespace and pod name

Aggregated Summary

• Total nodes and pods
• Total CPU and memory usage of the cluster

Compilation

# Compile locally
go build -o jetmesh-agent

# Build Docker image
docker build -t jetmesh-agent:latest .

# Build for multiple architectures
docker buildx build --platform linux/amd64,linux/arm64 -t jetmesh-agent:latest .

Usage

Local Execution

# Start HTTP server (port 8080 by default)
./jetmesh-agent

# With custom port
SERVER_PORT=9090 ./jetmesh-agent

# JSON format (for debugging)
OUTPUT_FORMAT=json ./jetmesh-agent

# HTTP POST to backend
OUTPUT_FORMAT=http METRICS_POST_URL=https://my-backend/metrics ./jetmesh-agent

# With custom kubeconfig
KUBECONFIG=/path/to/kubeconfig ./jetmesh-agent

HTTP Endpoints

The agent exposes the following endpoints:

/health

Health check endpoint:

{
  "status": "healthy",
  "timestamp": "2024-01-20T10:30:00Z",
  "service": "jetmesh-agent"
}

/api/metrics

Endpoint that returns metrics in JSON format:

{
  "timestamp": "2024-01-20T10:30:00Z",
  "cluster_info": {...},
  "nodes": [...],
  "pods": [...],
  "summary": {...}
}

/metrics

Endpoint that returns metrics in Prometheus format using the official library:

# HELP k8s_cluster_nodes_total Total number of nodes
# TYPE k8s_cluster_nodes_total gauge
k8s_cluster_nodes_total 3
...

Note: The /metrics endpoint uses the official Prometheus library (prometheus/client_golang) and is served on port 9090 by default.

Kubernetes Deployment

# Apply manifests
kubectl apply -f kubernetes-deployment.yml

# View logs
kubectl logs -f deployment/jetmesh-agent

# Run as single job
kubectl create job --from=cronjob/jetmesh-agent-cron metrics-job-$(date +%s)

Configuration

Environment Variables

Variable	Description	Values	Default
OUTPUT_FORMAT	Output format	json, http, post	http
KUBECONFIG	Path to kubeconfig	file path	~/.kube/config
METRICS_POST_URL	Backend URL for POST	URL	(empty)
METRICS_API_TOKEN	Authentication token for POST	string	(empty)
METRICS_CLIENT_ID	Client ID for POST	string	(empty)
SERVER_PORT	HTTP server port	string	8080
LOG_LEVEL	Logging level	debug, info, warn, error, fatal, panic	info

Logging System

The agent uses Logrus for log handling with the following features:

• JSON Format: Structured logs in JSON format for easy parsing
• Configurable levels: debug, info, warn, error, fatal, panic
• Structured fields: Includes timestamp, level, message and additional fields
• Environment variable configuration: LOG_LEVEL

Log examples:

{
  "level": "info",
  "msg": "Starting JetMesh Agent",
  "time": "2024-01-20T10:30:00Z"
}
{
  "level": "info",
  "msg": "Starting HTTP server on port 8080",
  "time": "2024-01-20T10:30:00Z"
}
{
  "level": "info",
  "msg": "Starting Prometheus metrics server on port 9090",
  "time": "2024-01-20T10:30:00Z"
}

Logging configuration:

# Detailed logs for debugging
LOG_LEVEL=debug ./jetmesh-agent

# Only errors and warnings
LOG_LEVEL=warn ./jetmesh-agent

# In Kubernetes
env:
  - name: LOG_LEVEL
    value: "info"

Output Modes

1. JSON (OUTPUT_FORMAT=json)

Prints metrics in JSON format to console. Useful for debugging.

OUTPUT_FORMAT=json ./jetmesh-agent

2. HTTP POST (OUTPUT_FORMAT=http or post)

Sends metrics in JSON format via HTTP POST to the configured backend.

env:
  - name: OUTPUT_FORMAT
    value: "http"
  - name: METRICS_POST_URL
    value: "https://my-backend/metrics"
  - name: METRICS_API_TOKEN
    valueFrom:
      secretKeyRef:
        name: metrics-agent-secrets
        key: api-token
  - name: METRICS_CLIENT_ID
    value: "my-cluster-123"

Included headers:

• Authorization: Bearer <API_TOKEN>
• X-Client-ID: <CLIENT_ID>
• Content-Type: application/json

Prometheus Integration

The agent includes native Prometheus integration using the official prometheus/client_golang library:

Available Metrics

Cluster Metrics:

• k8s_cluster_nodes_total - Total nodes
• k8s_cluster_pods_total - Total pods
• k8s_cluster_cpu_usage_cores - Total cluster CPU
• k8s_cluster_memory_usage_gb - Total cluster memory

Node Metrics:

• k8s_node_cpu_usage_cores{node="..."} - CPU per node
• k8s_node_memory_usage_gb{node="..."} - Memory per node

Pod Metrics (top 10):

• k8s_pod_cpu_usage_cores{namespace="...",pod="..."} - CPU per pod
• k8s_pod_memory_usage_gb{namespace="...",pod="..."} - Memory per pod

Prometheus Configuration

To scrape metrics from the agent, configure Prometheus:

scrape_configs:
  - job_name: 'jetmesh-agent'
    static_configs:
      - targets: ['jetmesh-agent:9090']
    metrics_path: /metrics
    scrape_interval: 30s

Accessing Metrics

# Metrics in Prometheus format
curl http://localhost:9090/metrics

# Metrics in JSON format
curl http://localhost:8080/api/metrics

Kubernetes Authentication

JetMesh Agent automatically detects whether it is running inside a Kubernetes cluster or in local mode:

• In-cluster: Uses the ServiceAccount and RBAC permissions assigned to the pod.
• Local: Uses your user's ~/.kube/config file.

Never mount an admin kubeconfig in production! Always use ServiceAccounts and minimal permissions.