Add a gRPC interceptor for function metrics #5006
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment was marked as outdated.
This comment was marked as outdated.
negz
commented
Nov 16, 2023
sttts
reviewed
Nov 17, 2023
| if err != nil { | ||
| return errors.Wrap(err, "cannot load client TLS certificates") | ||
| } | ||
|
|
||
| m := xfn.NewMetrics() | ||
| metrics.Registry.MustRegister(m) |
There was a problem hiding this comment.
metrics registration should go to an init func. Otherwise, you will never be able to create multiple instances of this (e.g. for testing) without a panic.
There was a problem hiding this comment.
I don't think it's that cut-and-dry, and would generally prefer to setup things like metrics, loggers etc here in "main" 1 and plumb them down rather than relying on init functions hidden away in individual packages.
Footnotes
-
This isn't literally
func mainbut it's close enough in practice. I can't see us unit testing this plumbing. ↩
sttts
reviewed
Nov 17, 2023
This adds some basic metrics for functions: * RPCs started (count) * RPCs handled (count) * RPC compleition time (histogram) While we probably want metrics like this, I don't think this premade interceptor will work for us. It's close, but we probably want function name/target as a label and this doesn't seem to give us that. Signed-off-by: Nic Cope <nicc@rk0n.org>
This allows us to add useful, function-specific metric labels including what function is being called and the severity of the result. Signed-off-by: Nic Cope <nicc@rk0n.org>
This is useful to be able to look at metrics after an E2E run. Signed-off-by: Nic Cope <nicc@rk0n.org>
Signed-off-by: Nic Cope <nicc@rk0n.org>
Signed-off-by: Nic Cope <nicc@rk0n.org>
negz
force-pushed
the
the-metric-system
branch
from
62c7f5e
to
4c9ebcc
Compare
bobh66
approved these changes
Nov 20, 2023
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of your changes
Fixes #5001
This adds some basic RED metrics for functions:
All metrics are labelled by function name, package, and gRPC target. Responses are also labelled with gRPC status code and "severity". The gRPC status code represents whether the function successfully returned a response. The severity indicates whether the response was successful. We consider the severity of the response to be the severity of the most severe result in the response. So a response that returns a fatal result has a severity of fatal. This will allow folks to monitor how often a function is successfully returning a response, but a fatal response.
Example metrics from an E2E test run:
I have:
make reviewableto ensure this PR is ready for review.Added or updated unit tests.Added or updated e2e tests.Linked a PR or a docs tracking issue to document this change.Addedbackport release-x.ylabels to auto-backport this PR.Need help with this checklist? See the cheat sheet.