Feature (What you would like to be added):
We are exposing metrics for compaction job in ETCD druid by #569 . Now we need to whitelist the metrics in g/g and raise correct alerts based on the metrics. We have decided to raise alerts when job_duration_seconds and jobs_total with failed labels cross certain thresholds.
Motivation (Why is this needed?):

Approach/Hint to the implement solution (optional):
Threshold for jobs_total with failed label in a seed : 10% of aggregated jobs_total (This alert would be raised per seed)
Threshold for jobs_total with failed label in a shoot: 10% of jobs_total(This alert would be raised per shoot)

To raise the alert at seed level, Aggregate prometheus can be used. The metrics jobs_total with failed label is scraped by cache prometheus from etcd-druid. Aggregate prometheus can aggregate metrics jobs_total from cache prometheus. Seed level alerts can be raised on this aggregated metrics from aggregated prometheus.

To raise the alert at shoot level, alerts can be raised in control plane prometheus. Control plane prometheus already federate shoot specific jobs_total from cache prometheus. So, to raise the alert for jobs_total at shoot level, we need to add an alert for jobs_total in here

Another idea is that we aggregate the alert data that is already raised on shoot control plane prometheus. Alerts from the shoot control plane prometheus are passed to aggregate prometheus in garden namespace. These are just alert data. We can aggregate these alert data streaming from multiple shoots to aggregate prometheus and send alert for jobs_total at seed level.