CloudZero Agent Replicas and Resources

Why 3 Replicas?

The CloudZero Agent defaults to 3 replicas for both the aggregator and webhook server. This isn't arbitrary - it provides important operational benefits:

Zero-downtime node migration: When Kubernetes needs to shut down or replace a node, it can gradually migrate pods to other nodes without interruption. With only one replica, this creates a dilemma: either accept downtime during node maintenance, or configure a Pod Disruption Budget that prevents Kubernetes from draining the node at all. See Pod Disruption Budgets for details on this tradeoff.

Data collection reliability: If a pod crashes or restarts, that replica stops collecting data until it recovers. Multiple replicas ensure continuous data collection during routine pod lifecycle events (upgrades, restarts, node failures).

Webhook server capacity: The webhook server receives notifications for every resource create/update/delete in your cluster. Multiple replicas distribute this load and prevent requests from queuing during pod restarts. Running a single webhook replica can introduce latency into cluster operations. See Monitoring the CloudZero Agent for details.

Why These Resource Settings?

The default resource requests and limits account for how the agent components actually behave:

Aggregator collector container:

• Memory grows during metric collection, then drops sharply after flushing to disk
• Limits must accommodate these periodic spikes, not just average usage
• Default: 512Mi request, 768Mi limit

Aggregator shipper container:

• Memory stays low except during brief spikes when loading metrics for transmission
• Default: 256Mi request, 512Mi limit

Webhook server:

• Load varies with cluster activity (pod churn rate)
• High pod churn (frequent creates/deletes) requires more resources
• Default: 256Mi request, 512Mi limit

Why sufficient resources matter: Insufficient memory limits cause OOM kills, which means lost data. When a pod is destroyed, any cached data that hasn't been shipped yet is lost - while graceful shutdown attempts to send remaining data, network issues or unexpected failures mean that data cannot be recovered once the pod is gone. The defaults include enough padding to handle normal variability and avoid unnecessary pod churn. Having sufficient replicas to handle load spikes prevents pods from being killed and reduces redundancy loss in the system.

Tuning for Your Environment

These defaults work for typical clusters, but you may want to adjust them based on your specific environment.

Measuring Actual Usage

Check what your pods are actually consuming:

kubectl top pod -n cloudzero -l app.kubernetes.io/component=aggregator
kubectl top pod -n cloudzero -l app.kubernetes.io/component=webhook-server

Monitor over time to capture the full pattern, including memory spikes during flush/ship operations.

Adjusting Resources

Set requests and limits with 20-50% padding above observed peaks:

components:
  aggregator:
    collector:
      resources:
        requests:
          memory: "512Mi"
          cpu: "200m"
        limits:
          memory: "768Mi"
          cpu: "500m"
    shipper:
      resources:
        requests:
          memory: "256Mi"
          cpu: "100m"
        limits:
          memory: "512Mi"
          cpu: "250m"

  webhookServer:
    resources:
      requests:
        memory: "256Mi"
        cpu: "100m"
      limits:
        memory: "512Mi"
        cpu: "250m"

Adjusting Replica Counts

components:
  aggregator:
    replicas: 3

  webhookServer:
    replicas: 3  # Minimum of 2 strongly recommended

When to increase: Large clusters (100+ nodes) or very high pod churn may need more replicas for capacity.

When to decrease: Small test/dev clusters where data gaps and downtime during maintenance are acceptable may use fewer replicas. Note that running a single webhook replica can introduce latency in cluster operations, and single-replica components require Pod Disruption Budgets to prevent downtime - which then blocks node draining during maintenance.