-
Notifications
You must be signed in to change notification settings - Fork 662
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
provide configuration to include custom node and pod labels on metrics #859
Conversation
Skipping CI for Draft Pull Request. |
@a7i: GitHub didn't allow me to request PR reviews from the following users: jklaw90. Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
f552b91
to
e6523ba
Compare
e6523ba
to
24f1486
Compare
@a7i: GitHub didn't allow me to request PR reviews from the following users: knelasevero. Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs. In response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the work @a7i. This is an interesting idea. I do have some concerns, though.
As I understand it, prometheus metrics are strict in their definitions such that 2 metrics with different label sets are incompatible. I think this means that, with something like this, we could technically never mark these metrics as stable.
I think if there is a case for adding some known k8s pod/node labels to our metrics, we should just add them, or introduce them as an entirely new metric. If there is another example or precedent for dynamic re-labeling, that would reassure me. But this approach is pretty new and unique to me so I am just a little unsure about it.
For custom arbitrary pod labels, I think there is probably a better way to extract those into your metrics server without piping them through the descheduler. It might be something you're better off instrumenting in the applications themselves. Then, with the info already exported from descheduler's metrics, you can get enough to correlate the data. Have you tried something like that?
Hey @a7i. Thank You for providing detailed description and an example for better understanding.
I don't think there's a way to say "this set of labels is guaranteed, the rest is not". So rather than extending the current one I agree with @damemi on creating a new set of metrics instead. Which we might indefinitely keep in the alpha level. Something we might call a customizable metric for custom consumption. Though I'd like to have a conformation from the metrics community that this approach is considered acceptable so we do not end up maintaining something that's not recommended by the community.
Given the v1.Node object would be only required by the customizable metrics, we might just move the metrics out of the Once all the strategies are migrated, we can move the metrics out. The framework will create a wrapper around the |
@ingvagabund and @damemi thanks for taking a look and the feedback, it makes sense to me. Per your suggestion, I will go ahead and move this to draft and rebase/adjust as necessary once all strategies have migrated. |
@a7i: PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
@a7i: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
/cc @jklaw90 @damemi
In order to improve our observability/monitoring, we would like to include certain pod labels in our metrics.
use-cases:
team
,project
) and we need these associated with eviction metricstoplogy.kubernetes.io/zone
as it relates to our reliability/resilience effortsI realize that in the new descheduler framework efforts,
v1.Node
is no longer being passed inEvictPod
. If this is a deal breaker, we can just continue withpodLabels
but flatten out themetricsConfig
structure.