Add Allocation Service Metrics #414
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…undernetes into allocation_metrics
|
|
dgkanatsios
reviewed
Sep 24, 2022
|
Great work! Can you add a small description on the PR? Thanks! |
|
Hi @dgkanatsios, I've added a description of the PR. Also a heads up, we are currently still adding some code changes so currently the PR is in draft status. |
|
hey @abbasahmed, let me know how I can help to land this PR! Appreciate all the hard work, thanks! |
|
@abbasahmed @ghov kind ping, we want to release 0.6 next week and it would be great to include your changes! |
|
@dgkanatsios will resolve the conflicts and publish the PR today. Sorry for the hold up on this PR! |
|
thanks @abbasahmed, let me know if you need any help! |
dgkanatsios
approved these changes
Oct 12, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem: We currently do not have a lot of metrics around Allocations API service in Thundernetes. Reference: Issue #384
Solution:
This PR adds in 6 metrics:
The time taken for a successful allocation to complete.
The number of retries taken for an allocation to complete.
The number of 429 (too many requests) errors during allocation
The number of 404 (not found) errors during allocation
The number of 500 (internal) errors during allocation
The number of 409 (request conflict) errors during allocation
These new metrics can help us monitor the allocation service in ways such as monitoring the performance of allocation service in terms of speed and reliability. The metrics allow us to quickly monitor the errors of allocation service helping us to make quicker inferences/decisions.
Along with the metrics, we have also added a couple of panels in the Grafana dashboard to visualize these events.