Issue with PD integration (or how to debug)

[ut0mt8]

What did you do?

I try to send alert from prometheus to PD. Some of them are failing and don't are ingested by PD.
Summary : there are surely a bad json generated but how to debug it ?

What did you expect to see?

An alert in PD.

What did you see instead? Under which circumstances?

so the alerts is not sent and here the log file :

level=error ts=2019-10-24T13:29:13.862Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="cancelling notify retry for \"pagerduty\" due to unrecoverable error: unexpected status code 400: Event object format is unrecognized: JSON parse error" context_err=null
level=error ts=2019-10-24T13:29:13.862Z caller=dispatch.go:266 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="cancelling notify retry for \"pagerduty\" due to unrecoverable error: unexpected status code 400: Event object format is unrecognized: JSON parse error"

Environment

docker / linux basics

• Alertmanager version:

0.19

• Prometheus version:

2.9.2

• Alertmanager configuration file:

global:
    resolve_timeout: 5m
inhibit_rules:
-   equal:
    - cloud
    - region
    - env
    - role
    - alertgroups
    source_match:
        severity: critical
    target_match:
        severity: warning
receivers:
-   name: default
    slack_configs:
    -   api_url: https://hooks.slack.com/services/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
        channel: '#alerts'
        send_resolved: true
-   name: slack
    slack_configs:
    -   actions:
        -   style: danger
            text: '{{ template "csq.slack.silence.text" . }}'
            type: button
            url: '{{ template "csq.slack.silence.link" . }}'
        -   style: primary
            text: '{{ template "csq.slack.dashboard.text" . }}'
            type: button
            url: '{{ template "csq.slack.dashboard.link" . }}'
        -   style: primary
            text: '{{ template "csq.slack.confluence.text" . }}'
            type: button
            url: '{{ template "csq.slack.confluence.link" . }}'
        -   style: primary
            text: '{{ template "csq.slack.grafana.text" . }}'
            type: button
            url: '{{ template "csq.slack.grafana.link" . }}'
        api_url: https://hooks.slack.com/services/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
        channel: '{{ template "csq.slack.channel" . }}'
        color: '{{ template "csq.slack.color" . }}'
        fields:
        -   short: true
            title: cloud
            value: '{{ .CommonLabels.cloud }}'
        -   short: true
            title: region
            value: '{{ .CommonLabels.region }}'
        -   short: true
            title: env
            value: '{{ .CommonLabels.env }}'
        -   short: true
            title: role
            value: '{{ .CommonLabels.role }}'
        icon_emoji: '{{ template "csq.slack.emoji" . }}'
        image_url: '{{ template "csq.slack.image" . }}'
        send_resolved: true
        short_fields: true
        text: '{{ template "csq.slack.text" . }}'
        title: '{{ template "csq.slack.title" . }}'
-   name: pagerduty
    pagerduty_configs:
    -   description: '{{ template "csq.pagerduty.description" . }}'
        details:
            alertname: '{{ .CommonLabels.alertname }}'
            cloud: '{{ .CommonLabels.cloud }}'
            env: '{{ .CommonLabels.env }}'
            region: '{{ .CommonLabels.region }}'
            role: '{{ .CommonLabels.role }}'
        send_resolved: true
        service_key: '{{ template "csq.pagerduty.service_key" . }}'
        severity: '{{ .CommonLabels.severity }}'
route:
    group_by:
    - cloud
    - region
    - env
    - role
    - alertname
    group_interval: 5m
    group_wait: 1m
    receiver: default
    repeat_interval: 1h
    routes:
    -   continue: false
        match:
            test: 'yes'
        receiver: slack
    -   continue: false
        group_by:
        - '...'
        match:
            env: production
        receiver: pagerduty
    -   continue: false
        receiver: slack
templates:
- /etc/alertmanager/templates.tmpl

• Prometheus configuration file:

NA

• Logs:

level=error ts=2019-10-24T13:29:13.862Z caller=notify.go:372 component=dispatcher msg="Error on notify" err="cancelling notify retry for \"pagerduty\" due to unrecoverable error: unexpected status code 400: Event object format is unrecognized: JSON parse error" context_err=null
level=error ts=2019-10-24T13:29:13.862Z caller=dispatch.go:266 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="cancelling notify retry for \"pagerduty\" due to unrecoverable error: unexpected status code 400: Event object format is unrecognized: JSON parse error"

[simonpasquier]

Hmm, it could be that there's network hiccup resulting in the posted payload being truncated? AFAICT the notify/pagerduty code itself can't generate invalid JSON.

[ut0mt8]

No network issue, this is reproductible with the alert. My question is still the same : how to debug ?
I think I will finish by hack the code by myself to add some debug log.

Another interesting one :

level=error ts=2019-10-24T14:21:08.917Z caller=dispatch.go:266 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="cancelling notify retry for \"pagerduty\" due to unrecoverable error: unexpected status code 400: Event object is invalid: 'routing_key' must be provided in the body, or provided in the headers using 'x-routing-key'"

with this configuration

-   name: pagerduty
    pagerduty_configs:
    -   description: '{{ template "csq.pagerduty.description" . }}'
        send_resolved: true
        severity: '{{ .CommonLabels.severity }}'
        details:
            alertname: '{{ .CommonLabels.alertname }}'
            cloud: '{{ .CommonLabels.cloud }}'
            env: '{{ .CommonLabels.env }}'
            region: '{{ .CommonLabels.region }}'
            role: '{{ .CommonLabels.role }}'
    routing_key: '{{ template "csq.pagerduty.service_key" . }}'

and this work with this modified conf :

-   name: pagerduty
    pagerduty_configs:
    -   description: '{{ template "csq.pagerduty.description" . }}'
        routing_key: '{{ template "csq.pagerduty.service_key" . }}'
        send_resolved: true
        severity: '{{ .CommonLabels.severity }}'
        details:
            alertname: '{{ .CommonLabels.alertname }}'
            cloud: '{{ .CommonLabels.cloud }}'
            env: '{{ .CommonLabels.env }}'
            region: '{{ .CommonLabels.region }}'
            role: '{{ .CommonLabels.role }}'

just change the order of the routing key.

I know yaml and templating are ugly but still.
I really think we miss some error checking in the integration code.

[simonpasquier]

Re your last comment, I suspect that {{ template "csq.pagerduty.service_key" . }} expands to the empty string. There needs to be a check in the notifier about this.

As for debugging, yes you'll have to add some logs yourself and rebuild Alertmanager.

[ut0mt8]

two comment : so yes it expands to nothing but due to previous error in the templating.
So maybe some code refacto is needed to the templating in multiple pass ?!
also why don't add some debug info ? and more generaly an option to debug templating isssue ?

[simonpasquier]

Sorry but I'm not following, you didn't mention any templating errors before. Can you provide more details?

[ut0mt8]

Ok I will to be more clear.

So we state that the order of the field in the yaml change something.
For example if we had a problem in the templating certainly in details (for example we miss some fields) and the routing key is after in the conf file the message with be not send cause the routing key field is empty.

This is why I suggest to do a two pass: First with mandatory field, and second with optionnal; cause we are never sure that are fields will be filled.

Anyway : I persist that having an option to see the result of the templating in a debug file would be really helpful.

[simonpasquier]

So we state that the order of the field in the yaml change something.

It shouldn't.

For example if we had a problem in the templating certainly in details (for example we miss some fields) and the routing key is after in the conf file the message with be not send cause the routing key field is empty.

In case of templating errors, Alertmanager should return immediately and not send anything to PagerDuty.

[simonpasquier]

Closing it. With #2085, Alertmanager should now detect when routing/service keys are empty after templating and it should report the error in the logs.