[secretfilter] Add metrics to track the redaction of secrets#2985
[secretfilter] Add metrics to track the redaction of secrets#2985
Conversation
Necessary as otherwise updating the labels being tracked would lead to prometheus creating multiple metrics with different labels to track.
|
💻 Deploy preview deleted. |
Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>
|
Docs look OK for now. I'll wait on final approve for docs until after code review. |
wildum
left a comment
There was a problem hiding this comment.
Thanks for adding this and for all the tests, this is great!
Only two small remarks:
- the changelog entry is missing
- it would be nice to have a little bit more test coverage for the update behavior in the tests: ensuring that existing metrics are not reset between two updates + what happens if I update with a different origin label and then update back to the previous origin label? Will I start from the previous value or from 0?
|
|
|
@wildum hope the new test matches your expectations - helpfully it also revealed a bug with updating the allowlists (it didn't clear the old allowlist) 👍 The failing tests seem completely unrelated to my changes post-merging in main, and are possibly just flakey tests. |
|
Re-running the test to confirm flakiness, and we'll need an updated changelog as this just missed the RC for 1.8. |
Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>
* Initial implementation of secretfilters metrics * Add summary metric for performance monitoring * Update label metrics to use user-specified list * Update Loki label metric to a single origin label Necessary as otherwise updating the labels being tracked would lead to prometheus creating multiple metrics with different labels to track. * Update documentation to reflect new metrics * Apply docs suggestions from code review Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Revert change to bullet points style * Validate behavior from repeated changes via Update * Add changelog entry for secretfilter metrics * Move changelog entry to correct heading * Apply suggestions from code review on docs Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com> * Revert unnecessary formating changes --------- Co-authored-by: Clayton Cornell <131809008+clayton-cornell@users.noreply.github.com>
PR Description
When the secretfilter component reacts secrets from log data, it is currently challenging to track how many secrets have been redacted, where they were redacted from, and how the component performs without expensive queries over Loki. This PR introduces multiple metrics to enable these to be tracked.
Which issue(s) this PR fixes
Notes to the Reviewer
I'd be very interested in your thoughts on the value of each of these metrics and my design choices around them. I would consider the
loki_secretfilter_secrets_redacted_by_rule_totalandloki_secretfilter_secrets_redacted_by_originto be the key metrics - but I think each of the others would also have some value. Arguablyloki_secretfilter_secrets_redacted_totalcould be determined directly fromloki_secretfilter_secrets_redacted_by_rule_total- would it be better to just have the latter?PR Checklist