Opsgenie Alerts cannot attach image url with external S3 configured

[oneoneonepig]

What happened:

Cannot create Opsgenie Alerts with S3 image url attached

What you expected to happen:

Opsgenie Alerts are created and image URL are attached in details section

How to reproduce it (as minimally and precisely as possible):

1. Create a Grafana container
2. Create a Grafana Image Renderer container
3. Configure Grafana to use the Grafana Image Renderer by following the tutorial
4. Configure panels, alert rules, contact points, notification policies, etc.
5. Trigger an Grafana Alert
6. An image is created in the S3 bucket
7. Grafana log shows 422 Unprocessable Entity
8. No Opsgenie Alert is created

Anything else we need to know?:

The following section is the error log found in Grafana

logger=alertmanager org=1 t=2023-04-25T03:52:08.926659281Z level=error component=alertmanager orgID=1 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="Opsgenie/opsgenie[0]: notify retry canceled due to unrecoverable error after 1 attempts: send notification to Opsgenie: webhook response status 422 Unprocessable Entity"

I've tried sending the alerts to MockAPI, then send the same payload to Opsgenie by directly invoking the Opsgenie Alerts API. It turns out that the detail section seems to only accept K/V with string value, does not accept array in the value.

Environment:

• Grafana version: Grafana v9.4.3
• Data source type & version: Prometheus 2.42.0
• OS Grafana is installed on: Ubuntu 22.04.1 LTS Docker CE 23.0.1
• User OS & Browser: Chrome Version 112.0.5615.138
• Grafana plugins: External Grafana Image Renderer v3.7.0

[oneoneonepig]

This seems like a beginner-friendly issue and I'm gonna start trying to fix it.

[vCra]

Have started seeing this issue with Google Cloud storage, however I never got images from OG before. Having alerts not being able to go to OG is not good, so I think removing image_urls from the request should allow us to start receiving alerts

[vCra]

This can probably be fixed by reverting #49379

[grobinson-grafana]

@oneoneonepig thanks for working on the fix! @vCra if needed you can disable images in notifications until this is fixed (https://grafana.com/docs/grafana/latest/alerting/manage-notifications/images-in-notifications/#configuration). Looking at the code, the image_urls field will be omitted if there are no images.

[oneoneonepig]

Indeed, it works as expected until I've configured image renderer

[grobinson-grafana]

@oneoneonepig have you decided how to fix it? Is there anything I can do to help?

[oneoneonepig]

If multiple images were attached, since the rule mentioned in the doc says that all images should be provided:

When multiple alerts are sent in a single notification a screenshot might be included for each alert. The order the images are shown is random.

I'll try to change from sending this:

details: {
    "image_urls":[
        "https://bucket_name.s3.region.amazonaws.com/image1.png",
        "https://bucket_name.s3.region.amazonaws.com/image2.png",
        "https://bucket_name.s3.region.amazonaws.com/image3.png"
    ]
}

to a comma separated value such as:

details: {
    "image_urls": "https://bucket_name.s3.region.amazonaws.com/image1.png, https://bucket_name.s3.region.amazonaws.com/image2.png, https://bucket_name.s3.region.amazonaws.com/image3.png"
}

The final result should show multiple clickable URLs. By directly invoking the Opsgenie Create Alert API, the results will be like this:

@grobinson-grafana Thanks for asking! This will be my first attempt to build Grafana, I'll dig through the docs and let you know if I need any help or any doc improvements that I would like to add 🙂

[oneoneonepig]

Hi @grobinson-grafana, seems like the corresponding code were refactored to grafana/alerting

In my forked repo, I have successfully patched the alerting library by joining the URLs into a single string, and it works like how I described above. Single image attached and two images attached scenarios are both tested, both were able to send Opsgenie Alerts successfully. The following screenshot is an example of two images attached alert:

The fix in my local dev is tested by directly modifying code under ~/go/pkg/mod/github.com/grafana/alerting@XXX/receivers/opsgenie/opsgenie.go, which apparently is not the correct way to fix the library.

Mind if I ask what is the correct way of sending the PR, should I send the patch to grafana/alerting? And what is the correct way to develop fixes if the code is in external libraries (to grafana/grafana, grafana/alerting seems to be external)?

Thanks for your help!

[grobinson-grafana]

Hi @oneoneonepig! This looks great, well done! 🎉 Please send the fix to grafana/alerting. I will take care of updating grafana/grafana from there!

[oneoneonepig]

Instead of directly changing the code under ~/go/pkg/mod/github.com/grafana/alerting, we've found a better (or the correct) development workflow when fixing libraries:

1. Clone grafana/grafana
2. Fork and clone grafana/alerting
3. Use the replace directive. In grafana/go.mod file, add line replace github.com/grafana/alerting => ../alerting to reference local code
4. Under grafana directory, run go mod tidy
5. Fix the code in alerting directory and build Grafana by running make run under directory grafana
6. Confirm the fix, commit to fork and create PR

[grobinson-grafana]

That is 100% correct. I don't think we've documented this so I will make sure that's in CONTRIBUTING.md.