Ensure metrics are deleted for resources in a deleted namespace #124
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This MR fixes bug DVO-31: Deleted Namespace and Kind still show in DVO grafana / prometheus.
Reconcile watches objects and triggers validations.RunValidations(), where DVO output is shipped to DVO's /metrics endpoint. Delete metric labels operations are also handled in RunValidations().
On deletion, gvk.Kind is an empty string for all resource types besides "Pod". This string is passed into RunValidations and is used to construct the prometheus labels for the associated metric.
When a deletion event is passed to RunValidations for non-pod resources (Deployment, ReplicaSet), promLabels is missing the kind field, and as such can't associate the provided deletion event with the existing metrics. For an example nginx deployment, the prom labels do not exist / will return multiple labels.
Passing
gr.reconciledKindinto RunValidations instead ofgvk.Kindfixes this problem. Deployment and ReplicaSet kinds are passed to the validation method, and the promLabels are found and deleted.