validation_gateway_solo_io_valid_config metric not working

[asayah]

validation_gateway_solo_io_valid_config metric doesn't report an issue even if the control plane configuration is invalid

[chrisgaun]

@asayah can you add steps to replicate it

[jameshbarton]

Steps to reproduce:

1. Set up port-forward for glooe prom server:

kubectl port-forward -n gloo-system deployment/glooe-prometheus-server 9090

2. Add a VS that points to a non-existent Upstream:

kubectl apply -f https://raw.githubusercontent.com/solo-io/solo-blog/main/prometheus-auth-intg/httpbin-vs-original.yaml

3. Confirm that control plane is invalid with glooctl check:

Error: 3 errors occurred:
	* Found virtual service with warnings: gloo-system httpbin-vs (Reason: warning:
  Route Warning: InvalidDestinationWarning. Reason: *v1.Upstream { gloo-system.default-httpbin-8000 } not found)
	* Virtual service references unknown upstream: (Virtual service: gloo-system httpbin-vs | Upstream: gloo-system default-httpbin-8000)
	* Found proxy with warnings: gloo-system gateway-proxy
Reason: warning:
  Route Warning: InvalidDestinationWarning. Reason: *v1.Upstream { gloo-system.default-httpbin-8000 } not found

4. Confirm that the metric's value is set to 1 (expected value with invalid control plane is 0):

% curl "http://localhost:9090/federate?match[]=\{gloo!=''\}" | grep validation_gateway_solo_io_valid_config
<snip>
# TYPE validation_gateway_solo_io_valid_config untyped
validation_gateway_solo_io_valid_config{gloo="gateway",instance="10.84.1.5:9091",job="kubernetes-pods",kubernetes_namespace="gloo-system",kubernetes_pod_name="gateway-7956b65865-rzctb",pod_template_hash="7956b65865"} 1 1629754615305

Context:

% glooctl version
Client: {"version":"1.8.6"}
Server: {"type":"Gateway","enterprise":true,"kubernetes":{"containers":[{"Tag":"1.8.6","Name":"discovery","Registry":"quay.io/solo-io"},{"Tag":"1.8.5","Name":"extauth-ee","Registry":"quay.io/solo-io"},{"Tag":"1.8.6","Name":"gateway","Registry":"quay.io/solo-io"},{"Tag":"1.8.5","Name":"gloo-ee-envoy-wrapper","Registry":"quay.io/solo-io"},{"Tag":"1.8.5","Name":"gloo-ee","Registry":"quay.io/solo-io"},{"Tag":"1.8.5","Name":"observability-ee","Registry":"quay.io/solo-io"},{"Tag":"1.8.5","Name":"rate-limit-ee","Registry":"quay.io/solo-io"},{"Tag":"5","Name":"redis","Registry":"docker.io"}],"namespace":"gloo-system"}}

[chrisgaun]

For making all glooctl check into metrics...

Metrics will not give same granularity.

Can give boolean on "something is wrong"

[sam-heilbron]

Where we define the gauge: https://github.com/solo-io/gloo/blob/master/projects/gateway/pkg/validation/validator.go#L51

[jenshu]

Verified the fix is working in gloo-ee v1.9.0-beta14. Note, following the above steps produces only warnings, so the valid config metric will still show as 1 unless you install gloo with allowWarnings=false (which will treat the warnings as errors), e.g.:

helm install -n gloo-system gloo-ee gloo-ee/gloo-ee --create-namespace --set-string license_key=$GLOO_LICENSE_KEY --set gloo-fed.enabled=false --set gloo.gateway.validation.allowWarnings=false  --version 1.9.0-beta14

After setting allowWarnings=false and applying the above config, I can confirm that the validation_gateway_solo_io_valid_config shows 0.

[jenshu]

this has been backported to 1.8 as well