Team based monitoring alert notification emails

• The alerts are configured in a way to ensure teams get alert notifications for services they own
• This is achieved using route feature provided by alert manager

This is achieved by attaching service_name label to the metrics by configuring jobs in prometheus.yml. Example

# prometheus.yml
- job_name: 'availability_static_site'
  ...    
  labels:
    "service_name": "sunbird_static_site"

- job_name: 'availability_cassandra'
  ...    
  labels:
    "service_name": "cassandra"

Routes are configured as

# alertmanagerconfig.yml
route:
  receiver: 'devops_team' # default receiver when route defined below is NOT matched
  routes:
  - receiver: slack # always send alerts to slack channel
    continue: true
  - match_re:
      service_name: "actor-service|learner-service|cassandra"
    receiver: app_team
  - match_re:
      service_name: "sunbird_static_site"
    receiver: site_team
...

Alert receivers are defined as

# alertmanagerconfig.yml
receivers:
  - name: 'devops_team'
    email_configs:
        to: '{{ devops_alerts_mailing_list }}'

  - name: 'app_team'
    email_configs:
        to: '{{ app_alerts_mailing_list }}'

  - name: 'site_team'
    email_configs:
        to: '{{ site_alerts_mailing_list }}'

The email ids receiving alerts are defines by ansible variables

devops_alerts_mailing_list: 'foo@bar.com, qux@baz.com'
app_alerts_mailing_list: 
site_alerts_mailing_list: