webhooks do not work #22782

zs-dima · 2020-12-17T19:47:45Z

Reproduced with Saas (sentry.io)

In case of several projects: .Net, Flutter, etc and setup webhooks for several projects at once - Flutter project webhooks do not work.
However 'Test' works well for all project + Alerts are the same.
And same webhooks works for other projects.

ceorourke · 2021-01-06T01:28:45Z

@zs-dima can you send a ticket to support@sentry.io referencing this so we can take a look at your specifics? I tested this out and can't reproduce the issue, but being able to look at your setup for the Flutter project would help.

McSneaky · 2021-01-15T21:52:54Z

My case is resolved. Didn't notice there is all dropdown nead when. Can ignore my comment

Alright, similar problem on self-hosted. But non of the alerts work when there are multiple organizations

How I got to that state:

Started fresh instance of Sentry
Noticed that I can't create organizations. Stopped the Sentry, sent that multi_organization env variable to true, started Sentry again
Created new organization
Made new project in that organization
Set up alerting like this
Send error from NodeJS SDK to Sentry. Error appears nicely in dashboard, but nothing is triggered. Tried emails and legacy integrations. Nothing

Went back to initial organization. Set up exactly the same alerting in new project in there and it works perfectly

Sentry version: Sentry 21.1.0.dev 06a4de6d

Edit: Aha. maybe it's more of an issue to getsentry/onpremise ?
Edit2: Just noticed that I had all under when, but issue can't be new and changed from resolved to unresolved at same time..

ceorourke · 2021-01-15T22:42:41Z

@McSneaky great - can this be closed then? Are you also zs-dima?

zs-dima · 2021-01-15T22:45:14Z

Still have this issue
Support looking into

zs-dima · 2021-02-22T18:12:48Z

Still have this issue
Support still looking into...

chadwhitacre · 2021-02-22T19:46:10Z

Thanks @zs-dima, sorry for the bot noise.

liupcoder · 2021-02-24T08:47:23Z

I have the same problem. Sentry 20.12.1bade073

zs-dima · 2021-02-24T08:52:28Z

Strange that issue affects Flutter projects, but .Net / .Net Core projects works well.

BYK · 2021-03-04T11:12:50Z

Ping @getsentry/visibility

BYK · 2021-03-04T16:48:56Z

Correction, @getsentry/workflow is the one responsible for alerts, notifications etc. :D

wedamija · 2021-03-04T22:13:34Z

Correction, @getsentry/workflow is the one responsible for alerts, notifications etc. :D

Is the issue here a specific integration isn't working, or that alerts aren't working in general? Wasn't clear after reading through this. If it's a specific integration, then @getsentry/ecosystem will be able to help.

zs-dima · 2021-03-05T05:16:46Z

@wedamija alerts (WebHooks) do not work for the Flutter projects

manuzope · 2021-03-08T20:47:56Z

Hi @zs-dima, just tested this out with a Flutter project and the webhooks worked fine for me. I have a few ways we can move forward:

Instead of creating a webhook, try creating an internal integration (Developer Settings > Internal integration). And then configure your alert rules to send to the internal integration. Internal integrations have a dashboard that you can see which events are sent and the response code we saw. Might help us narrow down what's happening.
If the above doesn't work, the best way for us to replicate this problem is if you could provide some code samples (ideally a repository with a button we can click to trigger an event) with which we can trigger events and see why these webhooks are not sending.

zs-dima · 2021-03-09T13:30:01Z

@manuzope thanks a lot for looking into.
I sent Sentry support samples in previous year already - but looks support had no time to look into yet.
Here I created new flutter project and set it up according your documentation:
stest.zip

Webhook still does not work.

I could share my dns url and etc in email or etc, just let me know please.

& 2. Webhook still does not work and I want to make then work therefore.

paalandreassen · 2021-03-10T09:25:28Z

Can confirm issues with webhooks from sentry.io.

I've tested the URL via curl and it works as expected.

When trying via the integration UI on sentry.io I get Test Results "No errors returned". But no message appears in slack which is my integration.

manuzope · 2021-03-10T20:02:55Z

Hi @zs-dima, mind putting that on GitHub? I can't securely open zip files.

@paalandreassen: just want to confirm - after setting up the webhook, did you also configure an alert rule to send to that webhook?

zs-dima · 2021-03-19T09:29:27Z

@BYK Could I kindly ask you to provide any working solution for Flutter project then?
I do not know anyone who could confirm that webhooks works for Flutter projects - I tried many.
And I could confirm that webhooks works for .Net Core projects well on other hand.

BYK · 2021-03-19T10:42:04Z

@zs-dima as we've mentioned earlier, webhooks should work for all projects. There is not project-type specific code around alerting or webhooks so let's shift our focus from "webhooks do not work with Flutter" to something else.

We need a lot more information to be able to assist you:

Are you getting the events/new issues from your Flutter project into your Sentry project?
How are your alert rules configured?

zs-dima · 2021-03-19T16:38:53Z

@BYK

webhooks should work for all projects

Totally agree

Are you getting the events/new issues from your Flutter project into your Sentry project?

Yes, as I told already

How are your alert rules configured?

Just 1 rule: Send a notification (for all legacy integrations)
I created several .Net Core projects and several Flutter projects with 100% same alert rules.
All .Net Core projects webhooks works well.
All Flutter projects webhooks does NOT work. However issues are created in Sentry.
I provided sentry.io login details to Sentry support few months before already and projects samples, hope they will find time to look into.

zs-dima · 2021-03-19T16:57:10Z

For example here 'bb76c1eca4b14124ae0b8b2280163dd7' printed out and I could see Sentry issue with the same event value:

    await sentry.SentryFlutter.init(
      (options) {
        options.dsn = 'https://***@***.ingest.sentry.io/***';
      },
    );

    try {
      throw Exception('Test exception');
    } catch (exception, stackTrace) {
      final ll = await Sentry.captureException(exception, stackTrace: stackTrace);
      print(ll);
    }

BYK · 2021-03-19T20:31:08Z

@zs-dima do you allow us to add a custom webhook that we control to your project alert to do further testing and analysis? The webhook will be on a third-party service and the contents may be readable to that service but not to the public.

zs-dima · 2021-03-19T20:37:09Z

@BYK Sure I will happy to do it to fix this strange issue finally.
Thanks a lot for care

chadwhitacre · 2021-03-19T20:51:54Z

I still haven't seen a clear bug report here, with well-defined steps to reproduce, expected result, and actual result.

Here is what I think your steps to reproduce are, @zs-dima:

Steps to Reproduce

Build a flutter app with Sentry installed. stest.zip above contains this (yes?).
Configure a webhook ... here's how I'm doing this:
1. Run httpdump, expose via ngrok.
2. Sentry.io > Settings > Search > WebHooks > Project Configurations > add ngrok URL
3. (Also make sure the plugin configuration is enabled I guess?)
4. "Test Plugin", should see a request-000 show up on local filesystem that looks like this.
Configure an alert notification ...
1. Alerts > Create Alert Rule
2. Alert name: Whatever
3. Alert Conditions: Then Perform These Actions > Send a notification via an integration > ... via WebHooks
Trigger an event from the Flutter app.

If this is not what you are doing then please advise.

All .Net Core projects webhooks works well.
All Flutter projects webhooks does NOT work.

Here is what I think your definitions of "works" (i.e., expected result) and "does not work" (i.e., actual result) are:

Expected Result

Local request-NNN file captured looks like this.

Actual Result

No idea ... what is the actual result you are seeing?

BYK · 2021-03-19T20:56:28Z

@zs-dima thanks for the consent. We have added our webhook to both of your projects so once you start triggering them, we should get some useful data for debugging.

Also if you can try to explain the issue in more detail as @chadwhitacre suggested, it would still help as the webhook debugging may not turn out to be productive.

chadwhitacre · 2021-03-19T20:59:26Z

@zs-dima Are you running afoul of action interval?

chadwhitacre · 2021-03-19T21:01:29Z

@zs-dima I'm confused by your thumbs up on my comment above where I asked for clarification on your bug report. Kindly please answer these questions:

Are the steps to reproduce the problem accurate?
What is the actual result you are seeing, in detail?

zs-dima · 2021-03-19T21:03:28Z

@BYK I just triggered webhook as before, Event Id is '41ee8cc575d440b6bee7939d283411da'
@chadwhitacre I will prepare more details for you shortly, as I have to re-setup environment back.

zs-dima · 2021-03-19T21:04:32Z

@chadwhitacre it is not action interval case as for the Net Core project I have the same settings and I tried to change Flutter projects action interval as well

zs-dima · 2021-03-19T21:11:32Z

and by the way - 'Test Plugin' button works well for all projects, I just retested it.

chadwhitacre · 2021-03-19T21:14:16Z

I will prepare more details for you shortly, as I have to re-setup environment back.

Thank you. Please provide as much information as you can to help us debug.

My best hypothesis at this point is that you have a misconfiguration. I find it highly unlikely that the source of the issue/event is related to the failure of the webhook, but I've been surprised before. :o)

it is not action interval case as for the Net Core project I have the same settings and I tried to change Flutter projects action interval as well

Say more about this ... is the alert shared between the Net Core and Flutter projects? Are you triggering the alert via Net Core, and then, within the alert interval, triggering it again from Flutter? If so is the alert interval suppressing the second webhook? In your steps to reproduce do you create the alert, then hit it immediately from the Flutter app?

chadwhitacre · 2021-03-19T21:14:21Z

Zooming out @manuzope @BYK ... do we have an audit of integration payloads? I'm thinking of something like GitHub does with their webhooks:

Is there a possible product improvement around that to make this kind of thing easier to (self-service) debug in the future?

Another minor wart: I am seeing unrendered markdown in the WebHooks plugin description:

zs-dima · 2021-03-19T21:18:02Z

@chadwhitacre
I triggered webhook few times but I could see nothing in the ngrok:

Version                       2.3.35                                                                                    Region                        United States (us)                                                                        
Web Interface                 http://127.0.0.1:4040                                                                     Forwarding                    
http://2f4c9de31a6b.ngrok.io -> http://localhost:7071                                     Forwarding                    
https://2f4c9de31a6b.ngrok.io -> http://localhost:7071                                                                                                                                                            
Connections                   ttl     opn     rt1     rt5     p50     p90                                                                             0       0       0.00    0.00    0.00    0.00

However 'Test Plugin' button triggering well.

manuzope · 2021-03-19T21:19:02Z

@chadwhitacre we have something like that for internal integrations (Developer Settings -> Dashboard) but not for webhooks. Internal integrations can also be used to send webhooks so we can get an audit log going if we move over to using those.

It will tell us response codes but not request/response bodies.

chadwhitacre · 2021-03-19T21:23:14Z

International integration doesn't work as well.

@zs-dima Why?

BYK · 2021-03-19T21:27:44Z

I triggered webhook few times but I could see nothing in the ngrok:

I can confirm that there is 0 activity on the webhook that we installed for our testing.

zs-dima · 2021-03-19T21:33:57Z

@chadwhitacre I checked all steps you provided above and everything setup as you described
ngrok empty result I provided above
'International integration' - behave in the same way - nothing in the ngrok log

zs-dima · 2021-03-19T21:34:56Z

@BYK

I can confirm that there is 0 activity on the webhook that we installed for our testing.

And you could see that projects had been setup correctly and 'Test Plugin' button triggering well
And Sentry issues are created

chadwhitacre · 2021-03-19T21:36:19Z

'Test Plugin' button triggering well

To confirm: this means that you do get something in the ngrok log from "Test Plugin"?

Do you get something in the ngrok log when triggering the alert from Net Core?

zs-dima · 2021-03-19T21:39:03Z

@chadwhitacre
Sure I receive correct webhook with all details in the ngrok and my service from "Test Plugin"
And I receive correct webhook with all details for all my Net Core Sentry projects as well
It is quite strange

chadwhitacre · 2021-03-19T21:40:28Z

I just sent an event, name 'heck' is not defined, did it arrive in your ngrok log?

zs-dima · 2021-03-19T21:45:52Z

@chadwhitacre I just received event but looks it is related to 'Test exception 8' I fired to test 'International integration'
however json format for this event looks a bit different

zs-dima · 2021-03-19T21:56:28Z

@chadwhitacre @BYK @manuzope looks 'International integration' works but it have different json format so I misses messages.
Lets I will test it in details but it could be really solution that will help me.
However could be nice to warn users that default setup does not work.

chadwhitacre · 2021-03-19T22:01:59Z

Sounds like good news. The internal integration is the preferred and default setup, the old webhooks plugin is legacy at this point. Good luck!

zs-dima · 2021-03-19T23:55:59Z

@chadwhitacre @BYK @manuzope Thanks a lot for care and help, all projects work well now :)

BYK added the Status: On Hold label Jan 19, 2021

github-actions bot added the Status: Stale label Feb 22, 2021

getsentry deleted a comment from github-actions bot Feb 22, 2021

chadwhitacre removed the Status: Stale label Feb 22, 2021

BYK added Component: Notifications Hosting: Self-hosted and removed Status: On Hold labels Mar 4, 2021

BYK added Component: Workflow Team: Workflow and removed Team: Visibility labels Mar 4, 2021

manuzope added Status: Needs More Information and removed Status: Untriaged labels Mar 8, 2021

zs-dima closed this as completed Mar 19, 2021

github-actions bot locked and limited conversation to collaborators Apr 4, 2021

webhooks do not work #22782

webhooks do not work #22782

Comments

zs-dima commented Dec 17, 2020 • edited

ceorourke commented Jan 6, 2021 • edited

McSneaky commented Jan 15, 2021 • edited

ceorourke commented Jan 15, 2021

zs-dima commented Jan 15, 2021

zs-dima commented Feb 22, 2021

chadwhitacre commented Feb 22, 2021

liupcoder commented Feb 24, 2021

zs-dima commented Feb 24, 2021 • edited

BYK commented Mar 4, 2021

BYK commented Mar 4, 2021

wedamija commented Mar 4, 2021

zs-dima commented Mar 5, 2021 • edited

manuzope commented Mar 8, 2021

zs-dima commented Mar 9, 2021

paalandreassen commented Mar 10, 2021 • edited

manuzope commented Mar 10, 2021

zs-dima commented Mar 19, 2021 • edited

BYK commented Mar 19, 2021

zs-dima commented Mar 19, 2021 • edited

zs-dima commented Mar 19, 2021

BYK commented Mar 19, 2021

zs-dima commented Mar 19, 2021

chadwhitacre commented Mar 19, 2021

Steps to Reproduce

Expected Result

Actual Result

BYK commented Mar 19, 2021

chadwhitacre commented Mar 19, 2021

chadwhitacre commented Mar 19, 2021

zs-dima commented Mar 19, 2021 • edited

zs-dima commented Mar 19, 2021 • edited

zs-dima commented Mar 19, 2021

chadwhitacre commented Mar 19, 2021

chadwhitacre commented Mar 19, 2021 • edited

zs-dima commented Mar 19, 2021 • edited

manuzope commented Mar 19, 2021

chadwhitacre commented Mar 19, 2021

BYK commented Mar 19, 2021

zs-dima commented Mar 19, 2021 • edited

zs-dima commented Mar 19, 2021 • edited

chadwhitacre commented Mar 19, 2021

zs-dima commented Mar 19, 2021

chadwhitacre commented Mar 19, 2021

zs-dima commented Mar 19, 2021

zs-dima commented Mar 19, 2021

chadwhitacre commented Mar 19, 2021

zs-dima commented Mar 19, 2021

zs-dima commented Dec 17, 2020 •

edited

ceorourke commented Jan 6, 2021 •

edited

McSneaky commented Jan 15, 2021 •

edited

zs-dima commented Feb 24, 2021 •

edited

zs-dima commented Mar 5, 2021 •

edited

paalandreassen commented Mar 10, 2021 •

edited

zs-dima commented Mar 19, 2021 •

edited

zs-dima commented Mar 19, 2021 •

edited

zs-dima commented Mar 19, 2021 •

edited

zs-dima commented Mar 19, 2021 •

edited

chadwhitacre commented Mar 19, 2021 •

edited

zs-dima commented Mar 19, 2021 •

edited

zs-dima commented Mar 19, 2021 •

edited

zs-dima commented Mar 19, 2021 •

edited