Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Create an event queue for important, must run events #16971
An events queue would allow WC to run several different requests in sequence without encountering race conditions.
The queue would likely take the form of a custom table:
Example use case; processing orders.
This use case would prevent issues with concurrent checkout. The big challenge is ensuring the queue is processed correctly and without delays.
When we implement this queue we should ensure it covers the following:
This was referenced
Sep 28, 2017
In general, I'm definitely in favor of queuing such things as you have examples for. For things like email, this would work just fine even on a shutdown hook that could be processed after the response is sent (with the hotfix in wc-api-dev for this making that situation better).
However, for the scenario of avoiding concurrent inventory adjustment, this becomes a bit more complex. If we "lock" somehow to prevent multiple processes from adjusting the inventory at the same time, this does create a performance bottleneck, so that's something to keep in mind. (For most smaller stores, this shouldn't be much of a problem, actually.) This would manifest in slow HTTP responses to the customer upon checkout.
I'd definitely recommend keeping any locking as closely tied to the data as possible. I'm not sure how possible this is though, with the API and with products being in the post and post_meta tables.
Locking will crash things, it will create scenarios of backed up or failed requests because of it. Queues at least will queue up the changes to a product or order which for most things will be perfectly fine (like inventory, 2+3-6+1 = 2-6+1+3
Also it could have a priority option for basic queue handling and making sure high priority stuff happens first (emails being lower priority vs say processing an order).
Having things abtracted so while the default of queues is table driven but for large stores could utilize beanstalkd or amazon sqs or other options would be critical IMO as well.
The locking should only be needed for processes that should not run more than once, ie upgrade/install routines.
I like the idea of queuing as that solves the stock issue, but we will also need something that we can fire async inside a queue, and that async job should be able to fire more queues of its own.
Excuse the low level diagram above, but hopefully, that gives an idea around this. Customers will place orders, this will then create a queue item for each order, this top-level queue item could take care of the stock checking and initial save to DB. From there the queue item can spawn an async job, at that point it can move onto the next queue item ie order #2. While the queue is processing the async jobs runs in the background independent of the queues, which can be used for sending emails, it is important to note that an async job should be able to also spawn another queue item that should be blocked within the job's context. Think processing download permissions or generating products keys or generating vanity order numbers before sending the order complete email.
Hopefully, that makes sense, the idea is that we can offload a lot of the heavy processing for orders to async jobs that run in the background.
Think of the flow of larger stores you have purchased from lately. It is typical that you would add to cart, enter payment details and then click the place order button which then simply tells you that your order was placed. Then minutes, or seconds depending on the store, later they send you your order completion email and download links if you have purchased digital products.
Technically we only need the blocking functionality to check for stock, so the idea Mike has about storing all the of the order in json format seems good, we can even go as far as to instead of queue every order simply check existing queue items via the json and see if a matching product is in there, and only them block the next order, otherwise process the order via an async job.
I think this should work well. We may want some sort of checking in case e.g. the same order gets enqueued twice we only want to process one and discard the second one. That can probably be done with a timestamp/user_id/guid column or something like that.
That may not even end up being a problem, but if concurrent users cause #14541 I think it may be reasonable to expect concurrent users can cause the same event to get enqueued twice.
FWIW, Action Schedule is very similar to what is being proposed here.
Specifically, it has:
Action Scheduler also has additional features this event queue would benefit from, and most likely have added in future if a new system was designed. Most notably:
There are some potential issues and/or mismatches though, including:
So it's not a perfect match, but IMO it's a well tested system that might at very least be a good jumping off point for this (it's already reliably processing hundreds of thousands of renewal payments on sites/servers outside of our control every month).
Also a redux of #3467. Which is the reason Action Scheduler's APIs are already prefixed with
A continued caution that you will need atomic operations and probably locks to implement a reliable queue infrastructure.
Warning! The default MyISAM database engine in MySQL does not support transactions. Without transaction support, I doubt you can create atomic handlers for queue work items. This leads to a change in server requirements for Woocommerce. https://www.sitepoint.com/mysql-transactions-php-emulation/ and https://dev.mysql.com/doc/refman/5.6/en/myisam-storage-engine.html
When using a SQL engine like InnoDB, SQL stored procedures are a reliable way to implement these handlers rather than piecing it together with PHP script. I acknowledge that a potential problem is that in some Wp/Woo installations, the database may not support/authorize stored procedures. Woo would need market research to understand their customers regarding this.
@kloon it's been very reliable. It's been responsible for all subscription events since Subscriptions v1.5 release a few years ago. The most notable of the events is handlers are renewal payments, which it reliably processes hundreds of thousands, if not millions, every month across all the sites using Subscriptions.
The main cases where it's had issues with reliability are:
@kloon yes but this is sort of "by design". WP-Cron is considered the best choice for ways to initiate the queue processing, because that's the most widely used and tested background processing system across the diversity of hosts and configurations available in WordPress.
It's not 100% reliably, but I doubt anything is and I doubt anything else would do as well as WP-Cron. Especially for the cases where it's not working, as there are abundant existing resources to help diagnose and fix issues with it, from the WP-Crontrol plugin to the numerous docs/guides for working with it. That's why we stick with it.
Action Scheduler has no hardcoded dependency on WP-Cron though. Or any other way to initiate the queue. The way jobs are run can be augmented or replaced without modifying any code in Action Scheduler.
It's also possible to have multiple queue runners. For example, we're introducing a WP CLI command to run actions in v1.6. WooCommerce core could have multiple systems for initiating queues. If it finds a certain server/install doesn't support required features for one, it could fallback to another.
@thenbrent thanks for all the info, one last question. Do you perhaps know how many other WooCommerce extensions make use of the action scheduler? Obviously it would make sense to include a library that is widely used by the community rather then one that is not so wanting to get some details around usage as well.
With regards to the custom post type usage, this is definitely something that will need to be revised should we decide to include the library into core, so glad to hear you are moving to a custom table design.
@pmgarman that library is already in core
Just to note that using that is also on the cards since it is already in core.
Chiming in here to report that we started using the wp-background-processing library in a rather demanding use case recently and I'm quite happy with its simplicity and how easy it is to extend (already used it for DB updates before).
Our use case: Maintaining a many-to-many database relationship between the stock (status) of any product and the stock status of all bundles that contain it. It's demanding in the sense that it needs to run trouble-free at a very high frequency, especially on high-traffic sites where product stock can change frequently (many orders/minute during peaks).
It's still simple in the sense that nobody is actually waiting for a result/response to be returned -- the core problem it solves is the need to update the
However, after playing with it I think it could be utilized to queue "order placement tasks" -- and this queue utilized to
Note that I've never had to build an actual request/response queue, where the outcome of each queued task alters the content of subsequent responses. It seems to me that this is what we're dealing with here (?).
This could be solved perhaps by hashing tasks, similar to what wp-background-processing is already doing?
Also note that wp-background-processing "task processing" locks are implemented using transients. These locks work reliably when a fast external object cache is present, but the approach might be prone to concurrency issues (same task being worked on multiple times) otherwise?
@kloon I know for sure that Follow-ups and Memberships also use it. I know the devs behind Zapier and at least two other extensions have also looked into using it as they've asked me about it over the years.
Based on the recent date of this gist it looks like Zapier still uses WordPress cron though. It might be worth asking @om4james why he chose not to move to AS, as it may give an idea of why it shouldn't be used in WC (or at least, what needs to be changed if it is used).
@pmgarman what specifically would you like abstracted that isn't already?
@pmgarman in general terms, that's all possible already.
The queue runners are already abstracted so much that you could run queues both stored in other systems, or even on other systems via API calls. Have a read over the README, especially this FAQ and this one or take a look at this PR to see an example of a custom queue runner/worker.
The data stores are also abstracted, so you can store the events anywhere you want, not just locally in MySQL, but also in other servers or services. Take a look at the custom tables plugin to get an idea of using custom stores.
We didn't end up using Action Scheduler in the WC Zapier extension because the extension only needs to do one-off async requests, and not scheduled recurring events.
For now we use wp_schedule_single_event() to asynchronously send data to Zapier after orders are paid for or order statuses change.
We do find that there a small proportion of sites that have issues with these events running reliably though. In some cases, WP-Cron never runs at all, and in other cases events get executed multiple times (usually because of an object caching issue).
Ideally I'd love to have an official WooCommerce-supported way to reliably schedule important events to run once and only once.
@om4james Have a look at the WP_Async_Request class in WC Core, that might be what you are after here. There is also WP_Background_Process which is an extension of WP_Async_Request which handles queues in a background job.
I just completed #17694 using WP_Background_Process and happy with the result, it seems reliable in all my tests I did.