Alertmanager Integration #129
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces a Loki tester charm for the purpose of integration testing the Loki charm. The loki tester charm forwards its own Python debug logs to Loki, when a Juju relation is created between it and the Loki charm. Normally the tester charm only forwards Python debug logs to Loki. The tester charm also provides and Juju action "log-error" which logs and error message to loki. The tester charm contains an alert rule which is also forwarded to Loki. This alert rule triggers an alert any time an error log is received by Loki.
This commit adds an integration test to check that LokiPushApiConsumer can forward logs and alert rules to LokiPushApiProvider. The test further checks conditions satisfying the alert rules do indeed raise alerts.
rbarry82
requested review from
Abuelodelanada,
simskij,
dstathis,
sed-i and
a team
|
Libraries are not up to date with their remote counterparts. If this was stdoutstderr |
sed-i
previously approved these changes
Apr 1, 2022
mmanciop
reviewed
Apr 1, 2022
balbirthomas
previously approved these changes
Apr 1, 2022
|
Libraries are not up to date with their remote counterparts. If this was stdoutstderr |
Add a helper method to also check the Prometheus-compatible alerts endpoint from Loki itself, so we can check there as a bisection point between Alertmanager. Add a test which exercises this endpoint. Typos and cleanup. Add a (very temporary) FIXME so the "other" test which uses Alertmananager can be moved to the bundle when this merges
|
Libraries are not up to date with their remote counterparts. If this was stdoutstderr |
balbirthomas
previously approved these changes
Apr 1, 2022
|
Libraries are not up to date with their remote counterparts. If this was stdoutstderr |
balbirthomas
approved these changes
Apr 1, 2022
sed-i
approved these changes
Apr 5, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue
Implement integration between Alertmanager and Loki, so any alerting rules actually fire alerts.
Solution
Primarily this was simply additions to
metadata.yamland small changes to the charm. The bulk of it was the creation of a workload-less tester charm which can generate messages in an arbitrary format through Juju actions.Testing Instructions
Deploy Loki and Alertmanager, as well as some charm which includes alerting rules (or
cos-configwith rules which match). See if alerts fire.Release Notes
Integrate Loki and Alertmanager, so that alerts fired by Loki based on LogQL alert rules are sent to Alertmanager for routing.