add metrics for provisioner usage #1872
Conversation
This adds three metrics for provisioners - Limits - Usage - Usage Percentage (0-100)
✅ Deploy Preview for karpenter-docs-prod ready!
To edit notification comments on pull requests, go to your Netlify site settings. |
|
The usage_pct metric is easy to wire up into Grafana for a gauge based vec that shows you if you are close to exceeding any limit on any provisioner. The other two metrics are just the limits & status, but reported to prometheus.
New metrics docs can be seen at https://deploy-preview-1872--karpenter-docs-prod.netlify.app/preview/tasks/metrics/ |
tzneal
force-pushed
the
add-provisioner-metrics
branch
2 times, most recently
from
988fa28
to
7e08df4
Compare
tzneal
force-pushed
the
add-provisioner-metrics
branch
from
21f97ad
to
93b9db1
Compare
|
|
||
| // metrics are all package global variables | ||
| var allMetrics []metricInfo | ||
| for _, pkg := range packages { |
There was a problem hiding this comment.
It would be nice to have a Karpenter dry-run mode where we could just parse the metrics endpoint rather than making all these assumptions about the pkg structure.
There was a problem hiding this comment.
Ah, yeah, that could work too. I could go either way, parsing the source isn't too bad and we walk through all of the controller packages.
There was a problem hiding this comment.
I'm fine with this approach for now. This is missing cloudprovider metrics: https://github.com/aws/karpenter/blob/main/pkg/cloudprovider/metrics/cloudprovider.go
| subSystemSortOrder["pods"] = 3 | ||
| subSystemSortOrder["cloudprovider"] = 4 | ||
| subSystemSortOrder["allocation_controller"] = 5 | ||
| return func(i, j int) bool { |
There was a problem hiding this comment.
If the subsystem isn't one of the enum keys above, it'll show up as the first sorted, right? (Value of 0), can we make it the other way around?
| } | ||
| } | ||
|
|
||
| func handleVariableDeclaration(v *ast.GenDecl) []metricInfo { |
There was a problem hiding this comment.
Can you comment this function so it's more readable to newcomers?
1. Issue, if available:
Fixes #840
2. Description of changes:
This adds three metrics for provisioners
3. How was this change tested?
Deploying to EKS
4. Does this change impact docs?
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.