Skip to content

AlertmanagerConfig conversion fails if matcher has not receiver defined #2185

@vdavidoff

Description

@vdavidoff

We have a Prometheus AlertmanagerConfiguration CRD that in part configures a route that has no receiver, with nested routes. e.g.

    route:
      groupBy:
      - alertname
      - cluster
      - env
      - job
      - region
      groupInterval: 10m
      groupWait: 30s
      receiver: blackhole
      repeatInterval: 12h
      routes:
      - matchers:
        - matchType: =~
          name: namespace
          value: namespace1|namespace2|namespace3|namespace4
        routes:
        - matchers:
          - matchType: '!~'
            name: team
            value: teamname1|teamname2
          receiver: blackhole

The Prometheus operator parses this fine and generates config (as viewed in the Alertmanager UI) as such:

route:
  receiver: blackhole
  group_by:
  - alertname
  - cluster
  - env
  - job
  - region
  continue: false
  routes:
  - receiver: monitoring/app-team/blackhole
    group_by:
    - alertname
    - cluster
    - env
    - job
    - region
    continue: true
    routes:
    - matchers:
      - namespace=~"namespace1|namespace2|namespace3|namespace4"
      continue: false
      routes:
      - receiver: monitoring/app-team/blackhole
        matchers:
        - team!~"teamname1|teamname2"
        continue: false

The VM operator fails to generate a valid configuration, however. It looks to me like the missing receiver results in that route being assigned a receiver that is the empty string, which then breaks configuration generation/validation since the built receiver name (including CRD namespace and name) doesn't exist when referenced.

Relevant section of VMAlertmanagerConfig (converted by the operator):

  route:                                                                                                                          
    group_by:                                                                                                                     
    - alertname                                                                                                                   
    - cluster                                                                                                                     
    - env                                                                                                                         
    - job                                                                                                                         
    - region                                                                                                                      
    group_interval: 10m                                                                                                           
    group_wait: 30s                                                                                                               
    receiver: blackhole                                                                                                           
    repeat_interval: 12h                                                                                                          
    routes:                                                                                                                                                                                                                
    - matchers:                                                                                                                   
      - namespace=~"namespace1|namespace2|namespace3|namespace4"
      receiver: ""                                                                                                                
      routes:                                                                                                                     
      - matchers:                                                                                                                 
        - team!~"teamname1|teamname2"                                                                                                         
        receiver: blackhole

Conversion error logged by the operator:

{"level":"error","ts":"2026-05-18T21:33:10Z","logger":"manager","msg":"Reconciler error","controller":"vmalertmanagerconfig","controllerGroup":"operator.victoriametrics.com","controllerKind":"VMAlertmanagerConfig","VMAlertmanagerConfig":{"name":"monitoring","namespace":"app-team"},"namespace":"monitoring","name":"app-team","reconcileID":"6dd23fc4-ede4-40a9-8589-0565a2be0c90","error":"incorrect result configuration, config source=inline configuration at configRawYaml: undefined receiver \"monitoring-app-team-\" used in route"}

I imagine this comment in a different bug report is the same issue.

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions