How to increase the timeout for Outgoing Webhooks?

[augustovan]

Hello,

We are experiencing an issue with Outgoing Webhooks because our ServiceNow API is taking more than 7 seconds to respond. Since the current timeout seems to be shorter than that, we are noticing an unexpected behavior: four identical requests being sent to the API, likely due to automatic retry attempts.

We would like to know:
1. What is the default timeout for Outgoing Webhooks? ( 4 Seconds?)
2. Is there a way to increase it to prevent duplicate calls?
3. If increasing the timeout is not possible, what would be the best approach to handle this scenario?

We appreciate any help you can provide!

Expected behavior:
We would like the webhook to wait longer before considering the request as failed and retrying it.

Current behavior:
The webhook seems to hit the timeout quickly and generates multiple duplicate requests.

Additional information:
• Logs

HTTPSConnectionPool(host='contoso.service-now.com', port=443): Read timed out. (read timeout=4)

OnCall Version == 1.15.2
Grafana Version == v11.5.2 (598e0338d5)

Thanks in advance for any guidance! 😊

Product Area

API

Anything else to add?

No response

[github-actions]

The current version of Grafana OnCall, at the time this issue was opened, is v1.15.2. If your issue pertains to an older version of Grafana OnCall, please be sure to list it in the PR description. Thank you 😄!

[padupe]

I'm having the same problem.
Has anyone managed to solve it?

[brewle]

A similar problem exists in version 1.15.6

The issue occurs with real alerts from updown.io, which trigger duplicate notifications due to webhook timeout and retry. When a webhook POST request to our server times out (after 4 seconds), Grafana OnCall automatically retries the webhook, resulting in duplicate notifications.

Moreover, if I send a test event from the integration interface, then there is no such problem - only one notification is sent.