CP-37207: Per-Check Type Configuration for Validator Diagnostics

[evan-cz]

The Problem

The CloudZero Agent validator runs diagnostic checks during pod startup to verify the deployment is configured correctly. Until now, these checks existed but offered no user-facing configuration. All checks ran, but there was no supported way to:

• Control whether a failing check blocked pod startup
• Disable checks that don't apply to a particular environment, or are broken
• Distinguish between critical misconfigurations and minor issues

This made it difficult to be aggressive about validation. We couldn't enforce checks that detect serious problems (like Istio cross-cluster misconfigurations) without also risking blocked deployments for less critical issues.

The Solution

This PR introduces the first user-facing API for controlling validator diagnostic behavior. Each check can now be configured independently with one of four types:

Type	On Failure	Use Case
required	Blocks pod startup (non-zero exit)	Critical misconfigurations that will cause problems
optional	Warning logged, pod continues	Important validation but may have transient failures
informative	Always passes	Pure telemetry gathering
disabled	Check not run	Doesn't apply to this environment

Key behavior: All checks run before any exit decision—we collect all diagnostics first, then determine the exit code based on whether any required checks failed.

The API

The definitive reference is helm/values.yaml. Here's the current configuration:

components:
  validator:
    checks:
      pre-start:
        api_key_valid: optional
        istio_xcluster_lb: required
      post-start:
        k8s_version: informative
        k8s_namespace: informative
        k8s_provider: informative
        kube_state_metrics_reachable: optional
        prometheus_version: informative
        scrape_cfg: optional
        webhook_server_reachable: optional
      pre-stop: {}
      config-load:
        api_key_valid: optional
        k8s_version: informative
        k8s_namespace: informative
        k8s_provider: informative
        kube_state_metrics_reachable: optional

This interface was chosen because it provides a Helm-friendly way to override the settings; trying to modify a list in Helm is difficult, you generally have to replace the whole list. With this design, you can disable any check you want (e.g., components.validator.checks.pre-start.istio_xcluster_lb=disabled), add it to another location, etc. The JSON Schema prevents specification of invalid validator checks.

Design Decisions

istio_xcluster_lb as required

This is the exemplar of what required is for. The Istio cross-cluster load balancing check:

• Passes silently for non-Istio clusters (no false positives)
• Only fails when it detects a genuine Istio misconfiguration
• Prevents serious problems — without this check, cross-cluster metrics get misattributed

This is exactly the kind of aggressive validation we want. If this check fails, there's a real configuration problem that will cause incorrect cost allocation.

api_key_valid as optional

We want to start collecting data as soon as possible. If the API key is invalid:

• The customer can fix it without restarting the collector
• Data collected during that window may be recoverable
• Blocking startup only delays problem discovery

Informative Checks

Pure telemetry: k8s_version, k8s_namespace, k8s_provider, prometheus_version

These gather environment information sent to CloudZero for diagnostics. They never block startup and always report passing—their job is information gathering, not validation.

Optional Checks

Important but not critical: kube_state_metrics_reachable, scrape_cfg, webhook_server_reachable

These validate connectivity and configuration but may have transient failures during cluster startup. We log warnings without blocking.

What This Enables

1. More aggressive validation — We can add required checks for serious misconfigurations without fear of blocking all deployments

2. Customer-specific configuration — Users can disable checks that don't apply (webhook check when webhooks are disabled, Istio check when not using Istio)

3. Graceful degradation — Optional checks warn about issues without preventing data collection

Discussion

The check type assignments represent our best judgment, but we're open to feedback:

• Should any checks move between categories?
• Are there checks that should be required by default but aren't?
• Are there checks where optional is too aggressive?

The API structure itself (components.validator.checks.<stage>.<check>: <type>) is intended to be stable across releases.

Validation

• Go unit tests: All existing tests updated and passing, plus new tests for CheckType validation, CheckConfig validation, and requiredFailures tracking in the runner
• Helm schema validation: All schema tests passing with new CheckConfig type definitions
• Helm unit tests: all tests passing, including new tests for per-check configuration (helm/tests/validator_checks_test.yaml)
• Manual deployment: Deployed to GKE cluster in many different configurations to verify that all
  options were being respected and handled correctly.

Backwards Compatibility

No concerns — the previous enforce flag was never exposed in values.yaml. This is a new user-facing API, not a migration from an existing one.

Technical Details

For reviewers interested in implementation:

• Go config: app/config/validator/diagnostics.go — CheckType enum, CheckConfig struct
• Runner logic: app/domain/diagnostic/runner/runner.go — Type-aware exit code determination
• Helm template: helm/templates/_helpers.tpl — cloudzero-agent.validator.stageCheck helper
• Schema: helm/values.schema.yaml — Full documentation for each check

Check types are tracked internally in the runner for exit code determination. They're not stored in the status protobuf or exposed in the API.

[josephbarnett]

What about a default - for example if type: is not defined on the yaml?

[josephbarnett]

I might recommend adding support for when type is not defined in a yaml file at all - to optional .... but otherwise looks good.

[evan-cz]

I might recommend adding support for when type is not defined in a yaml file at all - to optional .... but otherwise looks good.

had the default set to "required"; I changed it to "optional"

But the thing is that due to how the code is structured it can't really come up... because the user doesn't have direct access (unless they are engaging in shenanigans) to the data generated in the validator CM; they specify stuff in their overrides, then the template helpers generate the actual name/type properties.

If they do something like api_key_valid: (aka api_key_valid: null) it just won't get merged in by Helm and the default will be used.