-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use managed Prometheus with in-cluster Alertmanager #6
Comments
Hey @razvan-moj, Thank you for posting this issue! I would love to better understand this use case, and had a few follow up questions for you.
Thank you for submitting this feature proposal! Excited to learn a bit more about the use case from you. |
Hey @ampabhi-aws !
We have many (4000 odd as listed above) rules and alerts defined by users in their namespaces; those rules are picked up by Alertmanager once the .yaml is applied. Users do not have access to the AWS API. To change their setup to AMP it seems we need to need to change their workflow from
I don't think this is a problem for us, if we decide to use the AWS API to interact with Prometheus, we can also allow SNS and our users are familiar with how it works. |
Thanks for the clarification @razvan-moj! If we provided you a way to use CRDs to configure the rules directly via the Kubernetes APIs, and similarly configure your AMP alert manager via those CRDs, would that alleviate the need for an in-cluster Alertmanager? Trying to get a better sense of if the problem is (1) the AMP Alert manager is inconvenient to use, or (2) Doesn't fit the use case entirely. |
A definite +1 for this! |
@ampabhi-aws what are the next steps here? |
Prometheus (as deployed by the commonly used operator chart) is difficult to maintain and a resource hog; because of that AMP is very attractive. Alertmanager though works fine for our purposes, and we have
alerts defined, by teams which can each access only one namespace (https://github.com/ministryofjustice/cloud-platform-environments/search?q=prometheusrule), so no shared visibility. Alerts go directly to eg Slack, each team having control of its channel and hooks.
It would be ideal for us to use the managed Prometheus but keep alert definitions and Alertmanager as they are right now (presumably, AMP would need a configuration option to reach the cluster's AM).
The text was updated successfully, but these errors were encountered: