AlertmanagerConfig Pattern for PagerDuty Service/Global Orchestration

[argvk]

Hi all,

A common pattern with PagerDuty is leveraging its service/global orchestration features. In setups where AlertmanagerConfig is used per namespace (e.g., one service per namespace), the PagerDuty configuration typically includes a label for routing on PagerDuty's side, with all services sending events to the same endpoint.

However, this approach requires setting the routing_key or grouping_key (along with the secret) individually for each namespace.

Does anyone have suggestions for achieving this without duplicating the secret configuration across all namespaces?

A related discussion on global receivers: GitHub Issue #4606.

[mg1986jp]

If all your namespaces need to send to the same PagerDuty orchestration endpoint with the same routing key, you could use an AlertmanagerConfig at the global/cluster level instead of per-namespace. The Alertmanager CR supports alertmanagerConfiguration which lets you point to a single AlertmanagerConfig that acts as the global config:

apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
spec:
  alertmanagerConfiguration:
    name: global-alertmanager-config

For the routing key secret duplication issue, one approach is to create the secret in every namespace using something like Reflector or sealed-secrets with a template. Or if you're using PagerDuty Event Orchestration, you can use a single integration key and do all the routing on PagerDuty's side — in that case every namespace can share the same key and you only need one secret replicated around.

Another option is to skip AlertmanagerConfig entirely and use alertmanagerConfigSecret on the Alertmanager CR to mount a traditional alertmanager.yaml as a Secret. That gives you full control over the routing tree without per-namespace duplication.