Skip to content

Monitoring the CloudZero Agent

Jump to bottom
evan-cz edited this page Oct 17, 2025 · 5 revisions

Note: This document is under construction. We are actively working on ways to improve observability within the CloudZero Agent. This document currently focuses on monitoring the webhook server using Kubernetes API server metrics; we will be expanding this document in the future with additional methods for monitoring other components of the agent.

Monitoring Webhook Performance in CloudZero Agent

This document explains how to monitor the CloudZero Agent webhook's performance impact on your Kubernetes cluster using Kubernetes API server metrics.

Why Monitor Webhook Performance?

The CloudZero Agent webhook operates as a Kubernetes validating admission controller that intercepts resource operations (CREATE, UPDATE, DELETE) before they're persisted to etcd. While the webhook is designed to be fast and non-blocking, it's important to monitor its performance impact to ensure it doesn't introduce significant latency into your cluster operations.

Key concerns include:

  • API request latency: How much time does the webhook add to resource operations?
  • Network overhead: Time spent on TLS handshakes, network transit, and proxy components
  • Service mesh impact: Additional latency introduced by service meshes (e.g., Istio, Linkerd)
  • Operational visibility: Understanding the webhook's behavior during high-load scenarios

Why API Server Metrics?

Monitoring webhook performance from the webhook server itself only captures part of the picture. The webhook server can measure how long it takes to process a request once received, but it cannot measure:

  • TLS handshake time
  • Network transit time (both directions)
  • Service mesh overhead (Istio, Linkerd, etc.)
  • API server queuing time
  • Connection setup overhead

The API server metrics provide the complete end-to-end latency from when the API server initiates the webhook call to when it receives the response. This is the metric that matters for understanding the webhook's impact on cluster operations.

The Right Metric: apiserver_admission_webhook_admission_duration_seconds

Kubernetes exposes a STABLE metric specifically for tracking admission webhook latency (see the Kubernetes Metrics Reference):

apiserver_admission_webhook_admission_duration_seconds

Metric Details:

  • Type: Histogram
  • Stability: STABLE
  • Labels:
    • name: Webhook name (e.g., "cz-agent-cloudzero-agent-webhook-server-webhook.cloudzero-agent.svc")
    • operation: API operation (CREATE, UPDATE, DELETE, CONNECT)
    • rejected: Whether the request was rejected ("true" or "false")
    • type: Webhook type ("validating" or "admit")
  • Buckets: [0.005, 0.025, 0.1, 0.5, 1, 2.5, 10, 25] seconds

This metric captures the complete round-trip time including:

  • TLS handshake and connection setup
  • Network transit (to webhook server and back)
  • Service mesh proxy processing (if applicable)
  • Webhook server processing time
  • Any queuing or retry logic

Accessing the Metrics

The metric is exposed by the Kubernetes API server on the /metrics endpoint. If you have Prometheus configured to scrape the API server, this metric will be available automatically. You can query it directly using:

kubectl get --raw /metrics | grep apiserver_admission_webhook_admission_duration_seconds

To see metrics specific to the CloudZero webhook, filter by the webhook name in your Prometheus queries.

Webhook Server Metrics (Supplementary)

While API server metrics provide the complete picture, the webhook server itself also exposes metrics that can help diagnose issues:

Available at: http://<webhook-pod-ip>:9090/metrics

Key metrics:

  • http_request_duration_seconds: Server-side processing time (excludes network/TLS)
  • http_requests_total: Request count by status code
  • czo_webhook_types_total: Webhook events by resource type and operation

Note: These metrics only show webhook server processing time and don't include network latency or TLS overhead. They're useful for isolating whether high latency is due to webhook processing or network/infrastructure issues.

Comparison: API Server vs. Webhook Server Metrics

Aspect API Server Metrics Webhook Server Metrics
Scope Complete end-to-end latency Server processing only
Includes TLS ✅ Yes ❌ No
Includes Network ✅ Yes ❌ No
Includes Sidecars ✅ Yes ❌ No
Granularity Per webhook name All requests
Recommended for Performance monitoring Debugging webhook logic

Recommendation: Use API server metrics for understanding the webhook's impact on cluster operations. Use webhook server metrics only for debugging specific webhook processing issues.

Conclusion

Monitoring webhook performance is essential for maintaining cluster health and understanding the CloudZero Agent's operational impact. The apiserver_admission_webhook_admission_duration_seconds metric provides complete visibility into webhook latency, including all network and infrastructure overhead that affects actual cluster operations.

By monitoring this metric and setting appropriate alerts, you can ensure the CloudZero Agent webhook maintains minimal impact on your cluster's performance while providing valuable cost attribution data.

Clone this wiki locally