Add proposal for Prometheus Alerting in Observatorium #453
Conversation
Signed-off-by: Prem Saraswat <prmsrswt@gmail.com>
|
|
||
|  | ||
|
|
||
| 1. Alerts will be evaluated based on periodic requests to any replica of HTTP Query API of Query-frontend (Question: Or querier?). |
There was a problem hiding this comment.
I think this could be simplified and simply say against Query API, regardless of which component.
There was a problem hiding this comment.
Nice start @onprem - the bulk of the thinking is there and seems sensible 👍
The biggest bits that are missing for me at the moment are about how we construct the routing trees that alert manager consumes. Users will write routing table configuration to us, and we will store that somehow, but how do we create the final blob of YAML that we sync into AM?
Also - I would be careful about drawing the line between the Rules API proposal and this Alerting proposal, which lets us keep this proposal much more focused. For example - when the Rules API is implemented Ruler will naturally start evaluating alerts, all we need to do is configure Ruler to point at an Alertmanager URL.
| * Scaling Rulers | ||
| * Scaling Alertmanager beyond 3 replicas. |
There was a problem hiding this comment.
I don't understand why these are relevant / non-goals?
There was a problem hiding this comment.
Because it might be important for usability of whole platform (:
|
Just chatting with @onprem - there will likely be some lag in responding to all the comments in here as we're currently focused on shipping the Rules API. That is a hard dependency of this proposal & unblocks other workstreams. Watch this space :) |
This is indeed possible but I think this might involve more work. Going through Obs. API means we have take care of authenticating all requests, which might involve running token-refresher as sidecar for every Ruler, configuring the tenant secrets, etc. |
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
| * Scaling Rulers | ||
| * Scaling Alertmanager beyond 3 replicas. |
There was a problem hiding this comment.
Because it might be important for usability of whole platform (:
|
|
||
|  | ||
|
|
||
| 1. Alerts will be evaluated based on periodic requests to any replica of HTTP Query API of Query-frontend (Question: Or querier?). |
There was a problem hiding this comment.
All addressed, except this:
My understanding is that when the Rules API is active, alerts will already be evaluated in Thanos Ruler? The outcome is that Ruler will create new rules with name = ALERT?
@ianbillett could you elaborate on what do you mean? Outcome is that Recording Rule is evaluated and it's output is pushed to receiver. Similarly, Alert is evaluated and if triggered it is sending __name__ = ALERT? and pushing Alert to AM.
|
|
||
| ### Why thanos-ruler-syncer is not currently sufficient for our use-case? | ||
|
|
||
| Thanos rule syncer only support syncing Rules. Does not support AM Routing configuration. It also requires separate service thanos-objstore to be deployed. This unnecessary microservice architecture will become a problem and a fuss to operate and maintain. We don't need to scale those components separatedly, so I propose to keep it in a single binary for now and expected future. |
|
|
||
| 1. Alerts will be evaluated based on periodic requests to any replica of HTTP Query API, potentially through Query-frontend. | ||
| 2. Ruler evaluates Alerts that can be triggered. Triggered alerts will be pushed to all replicas of the Alertmanager (discussed later) through AM HTTP API. | ||
| 3. It exposes gRPC Rules API which is consumed by Queriers. Querier than can show this API via it's HTTP Rule API. |
There was a problem hiding this comment.
What is meant here by show? Maybe specifying that it's consumed by queriers is enough?
| 3. It exposes gRPC Rules API which is consumed by Queriers. Querier than can show this API via it's HTTP Rule API. | ||
| 4. It saves samples to any replica of Receiver Router (or RouterIngestor). | ||
|
|
||
| ### Rules API (non row one) |
There was a problem hiding this comment.
I'm finding this non-raw vs. raw distinction a bit confusing, maybe we could call this one just gRPC Rules API?
There was a problem hiding this comment.
Hmm both are HTTP in most cases.
One is raw configuration you can GET, PUT and DELETE, second is currently LOADED rules and alerts (+ instances of active alerts) you can GET
Co-authored-by: Jéssica Lins <jlins@redhat.com> Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
| Goals: | ||
|
|
||
| * Tenants should be able to read the status of active alerts | ||
| * via ([Prometheus Alert API GET](https://prometheus.io/docs/prometheus/latest/querying/api/#alerts)) |
There was a problem hiding this comment.
TODO: Remove, since Rules contains already same info!.
|
|
||
| We propose defining an additional HTTP API for routing on `/api/metrics/v1/{tenant}/api/v1/routing/raw` path in Observatorium API. Path that supports both GET, DELETE and PUT. | ||
|
|
||
| We propose to base the type on [Prometheus Operator AM Config](https://github.com/prometheus-operator/prometheus-operator/blob/9c0db5656f04e005de6a0413fd8eb8f11ec99757/pkg/apis/monitoring/v1alpha1/alertmanager_config_types.go#L69). Proposed adding this API to upstream Alertmanager too (TODO). |
There was a problem hiding this comment.
TODO: I will start a discussion on the Alertmanager repo.
TODO: Clarify what API should look like exactly, before merging
There was a problem hiding this comment.
I think we should clarify what exactly do we want here from the upstream Alertmanager. The fact is that AM already has an API which would enable us to get currently loaded configuration, see the /status endpoint in the spec.
| * **Other docs:* | ||
| * None | ||
|
|
||
| > TL;DR: For monitoring use cases, especially tenants of the metric signal of Observatorium, users want to be able to configure Prometheus-based alerts that will be evaluated for a given interval. If triggered they should notify the desired target (receiver) (e.g PagerDuty using Alertmanager Configuration). Users should also be able to see the status of all alerts through Prometheus-like alerts HTTP API. This proposal specifies how such a system can be deployed within Observatorium. |
There was a problem hiding this comment.
TODO:
- How often do we reload AM conf
- How do we trigger reload?
- Does triggering reload affects AM
- Exact config file? (type)
- How tenancy is enabled?
There was a problem hiding this comment.
I believe the biggest question mark we have is still around multi-tenancy and Alertmanager configuration. I'll try to summarize the discussions so far.
Main concern:
- We do not have (yet) a good idea on how multi-tenancy with configuration should be implemented. The fact is that Alertmanager does not recognize tenancy, meaning the configuration, from global perspective, represents a configuration of a single user. To make tenancy possible, we need to design a solution which would makes it possible to isolate configurations of multiple users (tenants) while still being represented in the global configuration (on top of this, there are related questions such as how to enable such isolation when it comes to UI and other APIs not included in this proposal - i.e. silences, inhibitions - the design choices here might have impact on these as well).
Ideas:
- Have separate Alertmanager instance for each tenant (hard tenancy)
- With this, we would completely side-step the issue of configuration, since the global config = user config. Downsides? Extra overhead of running Alertmanager instance for each tenant?
- Combine configurations provided by users into a single global configuration (soft tenancy)
- This aligns with how we deal with e.g. rules (see proposal). This would mean creating one global file with each tenant being represented in the global configuration. The main issue with this solution is how to combine tenants configuration:
- Alertmanager has a few global options that are applied if the child routes have no configuration set. If tenants are to provide us with their global parameters, it is not clear how it would be possible to combine global parameters from multiple tenants into one config file.
- An additional suggestion how to solve this would be to accept only non-global parameters, i.e.
routesandreceiversand combine these into a global config file. The downside is obviously that users cannot leverage global options and would need to specify configuration for all routes explicitly.
In this context it is also worth looking at how Cortex works, in which Alertmanager is running as a component within Cortex. This makes it possible to run a 'multitenant Alertmanager' as a single component, which encompasses multiple Alertmanagers, each dedicated to a single tenant (meaning each tenant can provide their isolated, global configuration)
|
The image is weirdly rendered on the website
|
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
- Add more alternatives from disucssion - Clarify text in some places - Disable invalid links for now to make CI pass Signed-off-by: Matej Gera <matejgera@gmail.com>
This proposal talks about adding Prometheus alerting capabilities to Observatorium.
Signed-off-by: Prem Saraswat prmsrswt@gmail.com