-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Event trigger webhooks are not runned concurrently #5189
Comments
We're running into this issue as well on the latest builds (including v1.3.0-beta.4). It's crushing performance and it's a blocking issue for us, as our application counts on events to be processed in a timely manner. We weren't running into it before, so it seems like a regression. Hasura is running in a docker container, so it would be surprising for this to be a platform issue, but let me know if you need more specifics about our environment. |
@MichalKalita @mousetraps Thank you so much for reporting this regression. |
@codingkarthik I've seen the PR (#5352) where you solve this regression. We're currently experiencing a bunch of webhook executions on production (on version We're seeing a lot errors like this one and I'm wondering what could be causing it:
Note: Our Hasura setup is running multiple instances at the same time to provide HA. |
@cusspvz The issue you're facing is a different issue, which happens when you're running event triggers on multiple instances of hasura. We have fixed this in v1.3.3 which has been released yesterday. Do let us know if you continue facing the same issue with the newer version also. |
@codingkarthik I will update it and let you know. Thanks for that quick answer @codingkarthik ! |
…1237) This essentially restores the original code from c425b55 (#4013). Prior to this commit we would slurp messages as fast as possible from the database (one thing c425b55 fixed). Another thing broken as a consequence of the same logic was the removeEventFromLockedEvents logic which unlocks in-flight events (breaking at-least-once delivery) Some archeology, post-c425b55: - cc8e2cc erroneously attempted to refactor using `bracket`, resulting in the same slurp-all-events behavior (since we don't ever wait for processEvent to complete) - at some point event processing within a batch is made serial, this reported as a bug. See: #5189 - in 0ef5229 (which I approved...) an `async` is added, again causing the same issue... GitOrigin-RevId: d8cbaab
I have insert event in table. It call webhook and response is in 10s. Actually webhooks are called sequential. But I want to call it concurrently. How to do it?
I am not sure if this behavior is bug in Hasura/heroku or some Hasura feature and how to change it.
Steps to reproduce:
Actual behavior:
After 10s 1 pending event is done. It is not runningconcurrently.
Expected behavior:
Run all webhooks in same time. All should be done in 10 seconds
Can be reproduced on
v1.3.0-beta.1
,v1.3.0-beta.2
It cannot be reproduced on
v1.2.2
The text was updated successfully, but these errors were encountered: