provide configuration to include custom node and pod labels on metrics

[a7i]

/cc @jklaw90 @damemi

In order to improve our observability/monitoring, we would like to include certain pod labels in our metrics.

use-cases:

1. We have compliance tags to identify cost-center/teams/projects (e.g. team, project) and we need these associated with eviction metrics
2. We would like to include labels such as toplogy.kubernetes.io/zone as it relates to our reliability/resilience efforts

I realize that in the new descheduler framework efforts, v1.Node is no longer being passed in EvictPod. If this is a deal breaker, we can just continue with podLabels but flatten out the metricsConfig structure.

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[k8s-ci-robot]

@a7i: GitHub didn't allow me to request PR reviews from the following users: jklaw90.

Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @jklaw90 @damemi

In order to improve our observability/monitoring, we would like to include certain pod labels in our metrics.

use-cases:

1. We have compliance tags to identify cost centers (e.g. team, project) and we need these associated with eviction metrics
2. We would like to include labels such as toplogy.kubernetes.io/zone as it relates to our reliability/resilience efforts

I realize that in the new descheduler framework efforts, v1.Node is no longer being passed in EvictPod. If this is a deal breaker, we can just continue with podLabels but flatten out the metricsConfig structure.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign seanmalloy after the PR has been reviewed.
You can assign the PR to them by writing /assign @seanmalloy in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[a7i]

/cc @knelasevero @JaneLiuL

[k8s-ci-robot]

@a7i: GitHub didn't allow me to request PR reviews from the following users: knelasevero.

Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @knelasevero @JaneLiuL

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[damemi]

Thanks for the work @a7i. This is an interesting idea. I do have some concerns, though.

As I understand it, prometheus metrics are strict in their definitions such that 2 metrics with different label sets are incompatible. I think this means that, with something like this, we could technically never mark these metrics as stable.

I think if there is a case for adding some known k8s pod/node labels to our metrics, we should just add them, or introduce them as an entirely new metric. If there is another example or precedent for dynamic re-labeling, that would reassure me. But this approach is pretty new and unique to me so I am just a little unsure about it.

For custom arbitrary pod labels, I think there is probably a better way to extract those into your metrics server without piping them through the descheduler. It might be something you're better off instrumenting in the applications themselves. Then, with the info already exported from descheduler's metrics, you can get enough to correlate the data. Have you tried something like that?

[ingvagabund]

Hey @a7i. Thank You for providing detailed description and an example for better understanding.

As I understand it, prometheus metrics are strict in their definitions such that 2 metrics with different label sets are incompatible. I think this means that, with something like this, we could technically never mark these metrics as stable.

From https://kubernetes.io/blog/2021/04/23/kubernetes-release-1.21-metrics-stability-ga/#what-are-stable-metrics-and-why-do-we-need-them:

A stable metric is one which, from a consumption point of view, can be reliably consumed across a number of Kubernetes versions without risk of ingestion failure.

I don't think there's a way to say "this set of labels is guaranteed, the rest is not". So rather than extending the current one I agree with @damemi on creating a new set of metrics instead. Which we might indefinitely keep in the alpha level. Something we might call a customizable metric for custom consumption. Though I'd like to have a conformation from the metrics community that this approach is considered acceptable so we do not end up maintaining something that's not recommended by the community.

I realize that in the new descheduler framework efforts, v1.Node is no longer being passed in EvictPod.

Given the v1.Node object would be only required by the customizable metrics, we might just move the metrics out of the EvictPod. I.e. have the EvictPod return a Status struct with an eviction result (e.g. "maximum number of pods per node reached") and an eviction state (evicted, eviction rejected). Then have higher logic read the Status and record the metrics.

Once all the strategies are migrated, we can move the metrics out. The framework will create a wrapper around the EvictPod method. We can then implement any "special" handling of the eviction status transparently to all plugins. E.g. extend them as needed after we have convincing discussion with the monitoring community. @a7i what's your though on this approach?

[a7i]

@ingvagabund and @damemi thanks for taking a look and the feedback, it makes sense to me.

Per your suggestion, I will go ahead and move this to draft and rebase/adjust as necessary once all strategies have migrated.

[k8s-ci-robot]

@a7i: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@a7i: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-descheduler-test-e2e-k8s-master-1-25	24f1486	link	true	/test pull-descheduler-test-e2e-k8s-master-1-25

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.