Add Async Webhooks (sent in background)

[wcurtis]

Without async webhooks, this module is pretty much a no-go for production shops - the dependency on third party systems is just too risky to do synchronously. The best practice for performing tasks asynchronously is to queue it up on a memory store (redis, memcache, etc) and have a background worker pick up the job and perform it, meanwhile the synchronous request returns immediately. Since there's no native queueing for Magento 2, our best bet might be to use the database as a 'queue' ("Blasphemy!" you say. Well, I agree. However, magento does something similar in the newsletter module) then use the cron to pick up the jobs every minute.

Ideas and feedback welcome - let's kick this conversation off before getting to building.

[Stumblor]

Bit of a PHP noob, but what about using a 3rd party non-blocking library, such as:
https://github.com/stil/curl-easy
?

[milsanore]

Firstly, thanks for building this library, I desperately need product created/updated/deleted webhooks in Magento.

1. THREADS
   Could you trigger a job on a separate php thread (https://secure.php.net/manual/en/class.thread.php)? Since Magento is not expecting a response from webhooks, the thread does not need to 'join', so you could just kick off a thread and walk away. In my head this saves you having to deal with the extra infrastructure of a message-queue, etc.

Regarding reliability, I noticed with Shopify that if a webhook does not respond with a 200 code then it is treated as an error and retried several times. If the webhook still fails (or times out) after several attemps, the webhook is considered broken. I don't think a message queue is necessary for this logic, in theory all of it could be done on a thread.

2. QUEUES
   The queue would solve other problems, however, like running out of memory if too many webhook events are triggered simultaneously. Can you package up your code in a Laravel project? Laravel has off-the-shelf support for a DB queue (including a migration to create the table, see https://laravel.com/docs/5.4/queues#driver-prerequisites).
   The only problem is that Laravel needs to run a queue-listener process (basically a CLI PHP script). Could your plugin explose a GUI for starting/stopping the queue-listener process?

Thanks,

[Baachi]

Hey guys,

I might need this functionality for an upcoming project, so i do some research regarding this issue.
Using pthreads is the easiest way as it required not really a code change, but there is one big disadvantage. There is no way to reschedule a failed webhook and i think this should be a required feature, too.
Because it can occur some network trouble, timeout issues or just high load.

Regarding to queues, Magento2 EE supports ships already with a queue, so this would be one solution. If we want to support the CE Version too, i found this module
which enqueues jobs in the database.

If i need this functionality for my project, i might have time to work on it.