New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create an event queue for important, must run events #16971

Closed
mikejolley opened this Issue Sep 28, 2017 · 23 comments

Comments

Projects
None yet
@mikejolley
Member

mikejolley commented Sep 28, 2017

An events queue would allow WC to run several different requests in sequence without encountering race conditions.

This is an alternative solution to the locks idea: #15780 / #15967

The queue would likely take the form of a custom table:

Column name Type Description Example
event_id int Auto increment ID 1
event_name varchar Name of the event to process reduce_stock
event_type varchar An event type so a certain event type can be processed if needed order_updates
event_data longtext Data passed to the event. { "order_id" : 1 }
  • Event handlers would be coded to handle success and failure of each event.
  • Completed events are removed from the queue.
  • Queues run in sequence; first in, first out.
  • Some kind of lock will be needed to ensure only one process runs through the queue.

Potential uses:

  • Queueing emails to send out in sequence.
  • Queueing order processing so stock levels reduce once only.

Example use case; processing orders.

  1. Customer goes to checkout.
  2. Customer enters details and places order.
  3. Order is created and a processing event is queued. All checkout data is stored alongside the queued item in JSON format.
  4. Checkout polls for updates or can show a thanks page which updates later.
  5. Queue process works through each item in the processing queue, processing each order in sequence.
  6. Processing of each order consists of:
    1. Check item stock
    2. Processing payment
    3. Reduce stock on success and update order
    4. Add error if failed
  7. Poll sees update and shows status on the checkout.

This use case would prevent issues with concurrent checkout. The big challenge is ensuring the queue is processed correctly and without delays.


When we implement this queue we should ensure it covers the following:

  • Concurrent checkouts can cause negative inventory for items with backorder disabled (#12467) - To fix this we need an order processing/stock reduction queue.
  • IPN/PDT could come in at the same time causing double order update #15773

cc @kloon

@coderkevin

This comment has been minimized.

Show comment
Hide comment
@coderkevin

coderkevin Sep 28, 2017

Member

In general, I'm definitely in favor of queuing such things as you have examples for. For things like email, this would work just fine even on a shutdown hook that could be processed after the response is sent (with the hotfix in wc-api-dev for this making that situation better).

However, for the scenario of avoiding concurrent inventory adjustment, this becomes a bit more complex. If we "lock" somehow to prevent multiple processes from adjusting the inventory at the same time, this does create a performance bottleneck, so that's something to keep in mind. (For most smaller stores, this shouldn't be much of a problem, actually.) This would manifest in slow HTTP responses to the customer upon checkout.

I'd definitely recommend keeping any locking as closely tied to the data as possible. I'm not sure how possible this is though, with the API and with products being in the post and post_meta tables.

Member

coderkevin commented Sep 28, 2017

In general, I'm definitely in favor of queuing such things as you have examples for. For things like email, this would work just fine even on a shutdown hook that could be processed after the response is sent (with the hotfix in wc-api-dev for this making that situation better).

However, for the scenario of avoiding concurrent inventory adjustment, this becomes a bit more complex. If we "lock" somehow to prevent multiple processes from adjusting the inventory at the same time, this does create a performance bottleneck, so that's something to keep in mind. (For most smaller stores, this shouldn't be much of a problem, actually.) This would manifest in slow HTTP responses to the customer upon checkout.

I'd definitely recommend keeping any locking as closely tied to the data as possible. I'm not sure how possible this is though, with the API and with products being in the post and post_meta tables.

@pmgarman

This comment has been minimized.

Show comment
Hide comment
@pmgarman

pmgarman Sep 28, 2017

Contributor

Locking will crash things, it will create scenarios of backed up or failed requests because of it. Queues at least will queue up the changes to a product or order which for most things will be perfectly fine (like inventory, 2+3-6+1 = 2-6+1+3

Also it could have a priority option for basic queue handling and making sure high priority stuff happens first (emails being lower priority vs say processing an order).

Having things abtracted so while the default of queues is table driven but for large stores could utilize beanstalkd or amazon sqs or other options would be critical IMO as well.

Contributor

pmgarman commented Sep 28, 2017

Locking will crash things, it will create scenarios of backed up or failed requests because of it. Queues at least will queue up the changes to a product or order which for most things will be perfectly fine (like inventory, 2+3-6+1 = 2-6+1+3

Also it could have a priority option for basic queue handling and making sure high priority stuff happens first (emails being lower priority vs say processing an order).

Having things abtracted so while the default of queues is table driven but for large stores could utilize beanstalkd or amazon sqs or other options would be critical IMO as well.

@kloon

This comment has been minimized.

Show comment
Hide comment
@kloon

kloon Sep 29, 2017

Member

The locking should only be needed for processes that should not run more than once, ie upgrade/install routines.

I like the idea of queuing as that solves the stock issue, but we will also need something that we can fire async inside a queue, and that async job should be able to fire more queues of its own.

Low level flow

Excuse the low level diagram above, but hopefully, that gives an idea around this. Customers will place orders, this will then create a queue item for each order, this top-level queue item could take care of the stock checking and initial save to DB. From there the queue item can spawn an async job, at that point it can move onto the next queue item ie order #2. While the queue is processing the async jobs runs in the background independent of the queues, which can be used for sending emails, it is important to note that an async job should be able to also spawn another queue item that should be blocked within the job's context. Think processing download permissions or generating products keys or generating vanity order numbers before sending the order complete email.

Hopefully, that makes sense, the idea is that we can offload a lot of the heavy processing for orders to async jobs that run in the background.

Think of the flow of larger stores you have purchased from lately. It is typical that you would add to cart, enter payment details and then click the place order button which then simply tells you that your order was placed. Then minutes, or seconds depending on the store, later they send you your order completion email and download links if you have purchased digital products.

Technically we only need the blocking functionality to check for stock, so the idea Mike has about storing all the of the order in json format seems good, we can even go as far as to instead of queue every order simply check existing queue items via the json and see if a matching product is in there, and only them block the next order, otherwise process the order via an async job.

Member

kloon commented Sep 29, 2017

The locking should only be needed for processes that should not run more than once, ie upgrade/install routines.

I like the idea of queuing as that solves the stock issue, but we will also need something that we can fire async inside a queue, and that async job should be able to fire more queues of its own.

Low level flow

Excuse the low level diagram above, but hopefully, that gives an idea around this. Customers will place orders, this will then create a queue item for each order, this top-level queue item could take care of the stock checking and initial save to DB. From there the queue item can spawn an async job, at that point it can move onto the next queue item ie order #2. While the queue is processing the async jobs runs in the background independent of the queues, which can be used for sending emails, it is important to note that an async job should be able to also spawn another queue item that should be blocked within the job's context. Think processing download permissions or generating products keys or generating vanity order numbers before sending the order complete email.

Hopefully, that makes sense, the idea is that we can offload a lot of the heavy processing for orders to async jobs that run in the background.

Think of the flow of larger stores you have purchased from lately. It is typical that you would add to cart, enter payment details and then click the place order button which then simply tells you that your order was placed. Then minutes, or seconds depending on the store, later they send you your order completion email and download links if you have purchased digital products.

Technically we only need the blocking functionality to check for stock, so the idea Mike has about storing all the of the order in json format seems good, we can even go as far as to instead of queue every order simply check existing queue items via the json and see if a matching product is in there, and only them block the next order, otherwise process the order via an async job.

@claudiulodro

This comment has been minimized.

Show comment
Hide comment
@claudiulodro

claudiulodro Sep 29, 2017

Contributor

I think this should work well. We may want some sort of checking in case e.g. the same order gets enqueued twice we only want to process one and discard the second one. That can probably be done with a timestamp/user_id/guid column or something like that.

That may not even end up being a problem, but if concurrent users cause #14541 I think it may be reasonable to expect concurrent users can cause the same event to get enqueued twice.

Contributor

claudiulodro commented Sep 29, 2017

I think this should work well. We may want some sort of checking in case e.g. the same order gets enqueued twice we only want to process one and discard the second one. That can probably be done with a timestamp/user_id/guid column or something like that.

That may not even end up being a problem, but if concurrent users cause #14541 I think it may be reasonable to expect concurrent users can cause the same event to get enqueued twice.

@thenbrent

This comment has been minimized.

Show comment
Hide comment
@thenbrent

thenbrent Oct 2, 2017

Member

FWIW, Action Schedule is very similar to what is being proposed here.

Specifically, it has:

Event handlers to handle success and failure of each event.
Completed events are removed from the queue.
Queues run in sequence; first in, first out. - Although it also allows a scheduled date to be set to delay the job/event's execution time.
Some kind of lock will be needed to ensure only one process runs through the queue. - Except Action Scheduler's locking allows for multiple queues to run through jobs concurrently, which is important to speed up execution time.
All checkout data is stored alongside the queued item in JSON format. - _The $args param for scheduling functions already stores a JSON encoded string of that data alongside the action.

Action Scheduler also has additional features this event queue would benefit from, and most likely have added in future if a new system was designed. Most notably:

  1. Admin UI to keep track of queues and scheduled events/actions (this admin UI is even being moved to the WooCommerce > Status admin section with Action Scheduler v1.6).
  2. configurable batch size and queue concurrency to increase speed at which large queues can be processed on high volume sites.
  3. logging of fatal errors during event execution to avoid having to blindly debug issues on a site with no PHP error logs (as events happen in the background, it's very difficult to trace errors without error logs because they might not be reproducible easily).
  4. logging event creation, start and completion times to make it possible to track the length of time it takes to process an event and trace the lifecycle of it, which is especially handy if an even is duplicated because of a tricky bug, like mishandling of PHP timezone changes (which FWIW, doesn't happen with Action Scheduler, but could be easy to accidentally introduced with a new system designed from scratch).

Issues

There are some potential issues and/or mismatches though, including:

  1. By default, Action Scheduler only initiates a new queue at most once per minute. That wouldn't be sufficient frequency for processing orders after checkout. But it's easy to change that with code to run queues via other methods.
  2. By default, Action Scheduler relies on WP-Cron to initiate queues, because that's the most battle tested background processing system in WordPress. As seen in the early 3.0 release period, even solid alternatives have issues when pushed to production across different hosting environments. But WP-Cron won't be a suitable system to run queues more frequently. As above, the way jobs are run has no dependency on WP-Cron though and is easily augmented or replaced. We're also introducing a WP CLI command to run actions in v1.6.
  3. It uses a custom post type not custom tables, so it doesn't scale. That's also possible to change as it's had abstract data stores since v1.0. We also have a prototype of custom tables in alpha status already. That is in active development and slated to be included in Action Scheduler core in future.
  4. It doesn't prevent scheduling identical events by default, which would be helpful to fix #15773 (although calling code can easily check if there is a scheduled action already, and callback code can/should check if the job it needs to be done has already been completed too). That's relatively straightforward to implement. Especially if a new method like wc_schedule_immediate_action() was introduced to be used specifically for scheduling actions to be processed ASAP (ahead even of the scheduled actions, perhaps also in the new runner that operates outside of WP-Cron).

So it's not a perfect match, but IMO it's a well tested system that might at very least be a good jumping off point for this (it's already reliably processing hundreds of thousands of renewal payments on sites/servers outside of our control every month).

This is an alternative solution to the locks idea: #15780 / #15967

Also a redux of #3467. Which is the reason Action Scheduler's APIs are already prefixed with wc_, so you have API function names like wc_schedule_single_action(), which are perfectly suited for inclusion in WC and already in use by at least 3 popular WooCommerce plugins. 🙂

Member

thenbrent commented Oct 2, 2017

FWIW, Action Schedule is very similar to what is being proposed here.

Specifically, it has:

Event handlers to handle success and failure of each event.
Completed events are removed from the queue.
Queues run in sequence; first in, first out. - Although it also allows a scheduled date to be set to delay the job/event's execution time.
Some kind of lock will be needed to ensure only one process runs through the queue. - Except Action Scheduler's locking allows for multiple queues to run through jobs concurrently, which is important to speed up execution time.
All checkout data is stored alongside the queued item in JSON format. - _The $args param for scheduling functions already stores a JSON encoded string of that data alongside the action.

Action Scheduler also has additional features this event queue would benefit from, and most likely have added in future if a new system was designed. Most notably:

  1. Admin UI to keep track of queues and scheduled events/actions (this admin UI is even being moved to the WooCommerce > Status admin section with Action Scheduler v1.6).
  2. configurable batch size and queue concurrency to increase speed at which large queues can be processed on high volume sites.
  3. logging of fatal errors during event execution to avoid having to blindly debug issues on a site with no PHP error logs (as events happen in the background, it's very difficult to trace errors without error logs because they might not be reproducible easily).
  4. logging event creation, start and completion times to make it possible to track the length of time it takes to process an event and trace the lifecycle of it, which is especially handy if an even is duplicated because of a tricky bug, like mishandling of PHP timezone changes (which FWIW, doesn't happen with Action Scheduler, but could be easy to accidentally introduced with a new system designed from scratch).

Issues

There are some potential issues and/or mismatches though, including:

  1. By default, Action Scheduler only initiates a new queue at most once per minute. That wouldn't be sufficient frequency for processing orders after checkout. But it's easy to change that with code to run queues via other methods.
  2. By default, Action Scheduler relies on WP-Cron to initiate queues, because that's the most battle tested background processing system in WordPress. As seen in the early 3.0 release period, even solid alternatives have issues when pushed to production across different hosting environments. But WP-Cron won't be a suitable system to run queues more frequently. As above, the way jobs are run has no dependency on WP-Cron though and is easily augmented or replaced. We're also introducing a WP CLI command to run actions in v1.6.
  3. It uses a custom post type not custom tables, so it doesn't scale. That's also possible to change as it's had abstract data stores since v1.0. We also have a prototype of custom tables in alpha status already. That is in active development and slated to be included in Action Scheduler core in future.
  4. It doesn't prevent scheduling identical events by default, which would be helpful to fix #15773 (although calling code can easily check if there is a scheduled action already, and callback code can/should check if the job it needs to be done has already been completed too). That's relatively straightforward to implement. Especially if a new method like wc_schedule_immediate_action() was introduced to be used specifically for scheduling actions to be processed ASAP (ahead even of the scheduled actions, perhaps also in the new runner that operates outside of WP-Cron).

So it's not a perfect match, but IMO it's a well tested system that might at very least be a good jumping off point for this (it's already reliably processing hundreds of thousands of renewal payments on sites/servers outside of our control every month).

This is an alternative solution to the locks idea: #15780 / #15967

Also a redux of #3467. Which is the reason Action Scheduler's APIs are already prefixed with wc_, so you have API function names like wc_schedule_single_action(), which are perfectly suited for inclusion in WC and already in use by at least 3 popular WooCommerce plugins. 🙂

@diablodale

This comment has been minimized.

Show comment
Hide comment
@diablodale

diablodale Oct 15, 2017

Contributor

A continued caution that you will need atomic operations and probably locks to implement a reliable queue infrastructure.

  • adding an item to a queue requires the queue to manipulate internal structures in a thread-safe way. Multiple threads can simultaneously want to addToQueue() at the exact same microsecond due to multi-core CPU and from multiple webservers operating with a single Wordpress/Woocommerce database. Therefore, the queue itself has to be constructed as thread-safe to operate in today's multhreaded environments. Not doing so results in unpredictable and errant queue behavior including, but not limited to, loss and corruption of queue data. See every doc on "how to implement a thread-safe queue". Our friendly SQL databases are designed to be thread-safe and have locks. If one implements a queue using the SQL database's capabilities, you can likely have a reliable queue infrastructure which can be exposed to PHP and therefore to WC.
  • when a work item in the queue is being processed, e.g. @mikejolley example above "Processing of each order", that work item needs to be processed as a single atomic transaction. The work item in the queue is either completely processed or completely un-processed. Nothing between. Therefore, one needs atomic or transactional operations. Our friendly SQL databases like MySQL have these features with start transaction, commit, rollback, etc. This is a good thing. Using these SQL atomic transaction features, one can implement the work item handler by...
    a) defining the work item "Processing of each order" as the..
    b) group of four actions Mike listed and...
    c) implementing all four within a single atomic SQL transaction http://php.net/manual/en/pdo.begintransaction.php and surround it all with a try/catch

Warning! The default MyISAM database engine in MySQL does not support transactions. Without transaction support, I doubt you can create atomic handlers for queue work items. This leads to a change in server requirements for Woocommerce. https://www.sitepoint.com/mysql-transactions-php-emulation/ and https://dev.mysql.com/doc/refman/5.6/en/myisam-storage-engine.html

When using a SQL engine like InnoDB, SQL stored procedures are a reliable way to implement these handlers rather than piecing it together with PHP script. I acknowledge that a potential problem is that in some Wp/Woo installations, the database may not support/authorize stored procedures. Woo would need market research to understand their customers regarding this.

Contributor

diablodale commented Oct 15, 2017

A continued caution that you will need atomic operations and probably locks to implement a reliable queue infrastructure.

  • adding an item to a queue requires the queue to manipulate internal structures in a thread-safe way. Multiple threads can simultaneously want to addToQueue() at the exact same microsecond due to multi-core CPU and from multiple webservers operating with a single Wordpress/Woocommerce database. Therefore, the queue itself has to be constructed as thread-safe to operate in today's multhreaded environments. Not doing so results in unpredictable and errant queue behavior including, but not limited to, loss and corruption of queue data. See every doc on "how to implement a thread-safe queue". Our friendly SQL databases are designed to be thread-safe and have locks. If one implements a queue using the SQL database's capabilities, you can likely have a reliable queue infrastructure which can be exposed to PHP and therefore to WC.
  • when a work item in the queue is being processed, e.g. @mikejolley example above "Processing of each order", that work item needs to be processed as a single atomic transaction. The work item in the queue is either completely processed or completely un-processed. Nothing between. Therefore, one needs atomic or transactional operations. Our friendly SQL databases like MySQL have these features with start transaction, commit, rollback, etc. This is a good thing. Using these SQL atomic transaction features, one can implement the work item handler by...
    a) defining the work item "Processing of each order" as the..
    b) group of four actions Mike listed and...
    c) implementing all four within a single atomic SQL transaction http://php.net/manual/en/pdo.begintransaction.php and surround it all with a try/catch

Warning! The default MyISAM database engine in MySQL does not support transactions. Without transaction support, I doubt you can create atomic handlers for queue work items. This leads to a change in server requirements for Woocommerce. https://www.sitepoint.com/mysql-transactions-php-emulation/ and https://dev.mysql.com/doc/refman/5.6/en/myisam-storage-engine.html

When using a SQL engine like InnoDB, SQL stored procedures are a reliable way to implement these handlers rather than piecing it together with PHP script. I acknowledge that a potential problem is that in some Wp/Woo installations, the database may not support/authorize stored procedures. Woo would need market research to understand their customers regarding this.

@mikejolley mikejolley referenced this issue Oct 30, 2017

Closed

Stock Reduction Events #13359

3 of 3 tasks complete
@kloon

This comment has been minimized.

Show comment
Hide comment
@kloon

kloon Oct 31, 2017

Member

@thenbrent Curious to know how the action scheduler's reliability has been, you are using that in Subscriptions if I recall? Do you still experience issues like in the past with the cron where events would just not fire?

Member

kloon commented Oct 31, 2017

@thenbrent Curious to know how the action scheduler's reliability has been, you are using that in Subscriptions if I recall? Do you still experience issues like in the past with the cron where events would just not fire?

@thenbrent

This comment has been minimized.

Show comment
Hide comment
@thenbrent

thenbrent Oct 31, 2017

Member

Curious to know how the action scheduler's reliability has been, you are using that in Subscriptions if I recall?

@kloon it's been very reliable. It's been responsible for all subscription events since Subscriptions v1.5 release a few years ago. The most notable of the events is handlers are renewal payments, which it reliably processes hundreds of thousands, if not millions, every month across all the sites using Subscriptions.

The main cases where it's had issues with reliability are:

  1. WP-Cron isn't running (more on this soon to answer your 2nd question)
  2. large queues (it works, but is slower than it should be due to the use of custom post types, which is why we're moving it to use custom tables)
  3. 3rd party plugin conflicts causing fatal errors when attempting to process an action. This general issue is really outside the scope of the event queue system and unavoidable on a WordPress/WooCommerce site, but Action Scheduler does include built-in logging to make it easier to diagnose the cause of this issue when it happens during a scheduled action.

Do you still experience issues like in the past with the cron where events would just not fire?

@kloon yes but this is sort of "by design". WP-Cron is considered the best choice for ways to initiate the queue processing, because that's the most widely used and tested background processing system across the diversity of hosts and configurations available in WordPress.

It's not 100% reliably, but I doubt anything is and I doubt anything else would do as well as WP-Cron. Especially for the cases where it's not working, as there are abundant existing resources to help diagnose and fix issues with it, from the WP-Crontrol plugin to the numerous docs/guides for working with it. That's why we stick with it.

Action Scheduler has no hardcoded dependency on WP-Cron though. Or any other way to initiate the queue. The way jobs are run can be augmented or replaced without modifying any code in Action Scheduler.

It's also possible to have multiple queue runners. For example, we're introducing a WP CLI command to run actions in v1.6. WooCommerce core could have multiple systems for initiating queues. If it finds a certain server/install doesn't support required features for one, it could fallback to another.

Member

thenbrent commented Oct 31, 2017

Curious to know how the action scheduler's reliability has been, you are using that in Subscriptions if I recall?

@kloon it's been very reliable. It's been responsible for all subscription events since Subscriptions v1.5 release a few years ago. The most notable of the events is handlers are renewal payments, which it reliably processes hundreds of thousands, if not millions, every month across all the sites using Subscriptions.

The main cases where it's had issues with reliability are:

  1. WP-Cron isn't running (more on this soon to answer your 2nd question)
  2. large queues (it works, but is slower than it should be due to the use of custom post types, which is why we're moving it to use custom tables)
  3. 3rd party plugin conflicts causing fatal errors when attempting to process an action. This general issue is really outside the scope of the event queue system and unavoidable on a WordPress/WooCommerce site, but Action Scheduler does include built-in logging to make it easier to diagnose the cause of this issue when it happens during a scheduled action.

Do you still experience issues like in the past with the cron where events would just not fire?

@kloon yes but this is sort of "by design". WP-Cron is considered the best choice for ways to initiate the queue processing, because that's the most widely used and tested background processing system across the diversity of hosts and configurations available in WordPress.

It's not 100% reliably, but I doubt anything is and I doubt anything else would do as well as WP-Cron. Especially for the cases where it's not working, as there are abundant existing resources to help diagnose and fix issues with it, from the WP-Crontrol plugin to the numerous docs/guides for working with it. That's why we stick with it.

Action Scheduler has no hardcoded dependency on WP-Cron though. Or any other way to initiate the queue. The way jobs are run can be augmented or replaced without modifying any code in Action Scheduler.

It's also possible to have multiple queue runners. For example, we're introducing a WP CLI command to run actions in v1.6. WooCommerce core could have multiple systems for initiating queues. If it finds a certain server/install doesn't support required features for one, it could fallback to another.

@kloon

This comment has been minimized.

Show comment
Hide comment
@kloon

kloon Nov 1, 2017

Member

@thenbrent thanks for all the info, one last question. Do you perhaps know how many other WooCommerce extensions make use of the action scheduler? Obviously it would make sense to include a library that is widely used by the community rather then one that is not so wanting to get some details around usage as well.

With regards to the custom post type usage, this is definitely something that will need to be revised should we decide to include the library into core, so glad to hear you are moving to a custom table design.

Member

kloon commented Nov 1, 2017

@thenbrent thanks for all the info, one last question. Do you perhaps know how many other WooCommerce extensions make use of the action scheduler? Obviously it would make sense to include a library that is widely used by the community rather then one that is not so wanting to get some details around usage as well.

With regards to the custom post type usage, this is definitely something that will need to be revised should we decide to include the library into core, so glad to hear you are moving to a custom table design.

@pmgarman

This comment has been minimized.

Show comment
Hide comment
@pmgarman

pmgarman Nov 1, 2017

Contributor

With some additional abstraction I'm definitely interested in seeing AS make it into core as well. It's the most battle tested solution we have in the WC space.

May also be worth looking at wp-background-processing as an example as well.

Contributor

pmgarman commented Nov 1, 2017

With some additional abstraction I'm definitely interested in seeing AS make it into core as well. It's the most battle tested solution we have in the WC space.

May also be worth looking at wp-background-processing as an example as well.

@kloon

This comment has been minimized.

Show comment
Hide comment
@kloon

kloon Nov 1, 2017

Member

@pmgarman that library is already in core 😄 https://github.com/woocommerce/woocommerce/blob/master/includes/libraries/wp-background-process.php It is used for deferred email sending which we had enabled by default at some point but had to set it to disabled by default due to some issues customers were experiencing.

Just to note that using that is also on the cards since it is already in core.

Member

kloon commented Nov 1, 2017

@pmgarman that library is already in core 😄 https://github.com/woocommerce/woocommerce/blob/master/includes/libraries/wp-background-process.php It is used for deferred email sending which we had enabled by default at some point but had to set it to disabled by default due to some issues customers were experiencing.

Just to note that using that is also on the cards since it is already in core.

@franticpsyx

This comment has been minimized.

Show comment
Hide comment
@franticpsyx

franticpsyx Nov 5, 2017

Member

Just to note that using that is also on the cards since it is already in core.

Chiming in here to report that we started using the wp-background-processing library in a rather demanding use case recently and I'm quite happy with its simplicity and how easy it is to extend (already used it for DB updates before).

Our use case: Maintaining a many-to-many database relationship between the stock (status) of any product and the stock status of all bundles that contain it. It's demanding in the sense that it needs to run trouble-free at a very high frequency, especially on high-traffic sites where product stock can change frequently (many orders/minute during peaks).

It's still simple in the sense that nobody is actually waiting for a result/response to be returned -- the core problem it solves is the need to update the outofstock visibility taxonomy of bundle-type products depending on what's happening to the stock quantity/status of their children.

However, after playing with it I think it could be utilized to queue "order placement tasks" -- and this queue utilized to

  • validate stock,
  • create pending orders and
  • fire off responses to customers waiting "in line".

Note that I've never had to build an actual request/response queue, where the outcome of each queued task alters the content of subsequent responses. It seems to me that this is what we're dealing with here (?).

That may not even end up being a problem, but if concurrent users cause #14541 I think it may be reasonable to expect concurrent users can cause the same event to get enqueued twice.

This could be solved perhaps by hashing tasks, similar to what wp-background-processing is already doing?

Also note that wp-background-processing "task processing" locks are implemented using transients. These locks work reliably when a fast external object cache is present, but the approach might be prone to concurrency issues (same task being worked on multiple times) otherwise?

Member

franticpsyx commented Nov 5, 2017

Just to note that using that is also on the cards since it is already in core.

Chiming in here to report that we started using the wp-background-processing library in a rather demanding use case recently and I'm quite happy with its simplicity and how easy it is to extend (already used it for DB updates before).

Our use case: Maintaining a many-to-many database relationship between the stock (status) of any product and the stock status of all bundles that contain it. It's demanding in the sense that it needs to run trouble-free at a very high frequency, especially on high-traffic sites where product stock can change frequently (many orders/minute during peaks).

It's still simple in the sense that nobody is actually waiting for a result/response to be returned -- the core problem it solves is the need to update the outofstock visibility taxonomy of bundle-type products depending on what's happening to the stock quantity/status of their children.

However, after playing with it I think it could be utilized to queue "order placement tasks" -- and this queue utilized to

  • validate stock,
  • create pending orders and
  • fire off responses to customers waiting "in line".

Note that I've never had to build an actual request/response queue, where the outcome of each queued task alters the content of subsequent responses. It seems to me that this is what we're dealing with here (?).

That may not even end up being a problem, but if concurrent users cause #14541 I think it may be reasonable to expect concurrent users can cause the same event to get enqueued twice.

This could be solved perhaps by hashing tasks, similar to what wp-background-processing is already doing?

Also note that wp-background-processing "task processing" locks are implemented using transients. These locks work reliably when a fast external object cache is present, but the approach might be prone to concurrency issues (same task being worked on multiple times) otherwise?

@thenbrent

This comment has been minimized.

Show comment
Hide comment
@thenbrent

thenbrent Nov 6, 2017

Member

Do you perhaps know how many other WooCommerce extensions make use of the action scheduler?

@kloon I know for sure that Follow-ups and Memberships also use it. I know the devs behind Zapier and at least two other extensions have also looked into using it as they've asked me about it over the years.

Based on the recent date of this gist it looks like Zapier still uses WordPress cron though. It might be worth asking @om4james why he chose not to move to AS, as it may give an idea of why it shouldn't be used in WC (or at least, what needs to be changed if it is used).

With some additional abstraction

@pmgarman what specifically would you like abstracted that isn't already?

Member

thenbrent commented Nov 6, 2017

Do you perhaps know how many other WooCommerce extensions make use of the action scheduler?

@kloon I know for sure that Follow-ups and Memberships also use it. I know the devs behind Zapier and at least two other extensions have also looked into using it as they've asked me about it over the years.

Based on the recent date of this gist it looks like Zapier still uses WordPress cron though. It might be worth asking @om4james why he chose not to move to AS, as it may give an idea of why it shouldn't be used in WC (or at least, what needs to be changed if it is used).

With some additional abstraction

@pmgarman what specifically would you like abstracted that isn't already?

@pmgarman

This comment has been minimized.

Show comment
Hide comment
@pmgarman

pmgarman Nov 6, 2017

Contributor

I want to be able to push all my background processing into months WP controlled systems. Beanstalk, Amazon, etc. I then want to hook up a queue worker to those systems to run those jobs

Contributor

pmgarman commented Nov 6, 2017

I want to be able to push all my background processing into months WP controlled systems. Beanstalk, Amazon, etc. I then want to hook up a queue worker to those systems to run those jobs

@pmgarman

This comment has been minimized.

Show comment
Hide comment
@pmgarman

pmgarman Nov 6, 2017

Contributor

Not months, “non WP controlled”

Contributor

pmgarman commented Nov 6, 2017

Not months, “non WP controlled”

@thenbrent

This comment has been minimized.

Show comment
Hide comment
@thenbrent

thenbrent Nov 7, 2017

Member

@pmgarman in general terms, that's all possible already.

The queue runners are already abstracted so much that you could run queues both stored in other systems, or even on other systems via API calls. Have a read over the README, especially this FAQ and this one or take a look at this PR to see an example of a custom queue runner/worker.

The data stores are also abstracted, so you can store the events anywhere you want, not just locally in MySQL, but also in other servers or services. Take a look at the custom tables plugin to get an idea of using custom stores.

Member

thenbrent commented Nov 7, 2017

@pmgarman in general terms, that's all possible already.

The queue runners are already abstracted so much that you could run queues both stored in other systems, or even on other systems via API calls. Have a read over the README, especially this FAQ and this one or take a look at this PR to see an example of a custom queue runner/worker.

The data stores are also abstracted, so you can store the events anywhere you want, not just locally in MySQL, but also in other servers or services. Take a look at the custom tables plugin to get an idea of using custom stores.

@om4james

This comment has been minimized.

Show comment
Hide comment
@om4james

om4james Nov 7, 2017

Contributor

Hi all,

We didn't end up using Action Scheduler in the WC Zapier extension because the extension only needs to do one-off async requests, and not scheduled recurring events.

For now we use wp_schedule_single_event() to asynchronously send data to Zapier after orders are paid for or order statuses change.

We do find that there a small proportion of sites that have issues with these events running reliably though. In some cases, WP-Cron never runs at all, and in other cases events get executed multiple times (usually because of an object caching issue).

Ideally I'd love to have an official WooCommerce-supported way to reliably schedule important events to run once and only once.

Contributor

om4james commented Nov 7, 2017

Hi all,

We didn't end up using Action Scheduler in the WC Zapier extension because the extension only needs to do one-off async requests, and not scheduled recurring events.

For now we use wp_schedule_single_event() to asynchronously send data to Zapier after orders are paid for or order statuses change.

We do find that there a small proportion of sites that have issues with these events running reliably though. In some cases, WP-Cron never runs at all, and in other cases events get executed multiple times (usually because of an object caching issue).

Ideally I'd love to have an official WooCommerce-supported way to reliably schedule important events to run once and only once.

@kloon

This comment has been minimized.

Show comment
Hide comment
@kloon

kloon Nov 15, 2017

Member

@om4james Have a look at the WP_Async_Request class in WC Core, that might be what you are after here. There is also WP_Background_Process which is an extension of WP_Async_Request which handles queues in a background job.

I just completed #17694 using WP_Background_Process and happy with the result, it seems reliable in all my tests I did.

Member

kloon commented Nov 15, 2017

@om4james Have a look at the WP_Async_Request class in WC Core, that might be what you are after here. There is also WP_Background_Process which is an extension of WP_Async_Request which handles queues in a background job.

I just completed #17694 using WP_Background_Process and happy with the result, it seems reliable in all my tests I did.

@dynamicJake

This comment has been minimized.

Show comment
Hide comment
@dynamicJake

dynamicJake Jan 8, 2018

Hello, I found this thread initially from duplicate orders happening, I was just making sure this thread is still related and in development?

Thank you

dynamicJake commented Jan 8, 2018

Hello, I found this thread initially from duplicate orders happening, I was just making sure this thread is still related and in development?

Thank you

@mikejolley mikejolley added this to the 3.5.0 milestone May 9, 2018

@mikejolley mikejolley changed the title from Create an event queue for important, must run events to Event queue: Create an event queue for important, must run events May 9, 2018

@mikejolley mikejolley changed the title from Event queue: Create an event queue for important, must run events to Create an event queue for important, must run events Jun 5, 2018

@sandrodz

This comment has been minimized.

Show comment
Hide comment
@sandrodz

sandrodz Jun 14, 2018

@mikejolley Hi, is 3.5.0 a real target? I saw this issue and other suggestions go to bin multiple times. Would love to hear we are taking this issue head on. I've encountered this problem in multiple projects.

sandrodz commented Jun 14, 2018

@mikejolley Hi, is 3.5.0 a real target? I saw this issue and other suggestions go to bin multiple times. Would love to hear we are taking this issue head on. I've encountered this problem in multiple projects.

@mikejolley

This comment has been minimized.

Show comment
Hide comment
@mikejolley

mikejolley Jun 14, 2018

Member

@sandrodz No but a queue PR exists to implement actions scheduler. I'll remove the milestone.

Member

mikejolley commented Jun 14, 2018

@sandrodz No but a queue PR exists to implement actions scheduler. I'll remove the milestone.

@mikejolley mikejolley removed this from the 3.5.0 milestone Jun 14, 2018

@claudiulodro claudiulodro added this to the API v3 ⚡Sprint 1 milestone Jul 13, 2018

@kloon

This comment has been minimized.

Show comment
Hide comment
@kloon

kloon Jul 20, 2018

Member

#20030 as a solution to this.

Member

kloon commented Jul 20, 2018

#20030 as a solution to this.

@claudiulodro

This comment has been minimized.

Show comment
Hide comment
@claudiulodro
Contributor

claudiulodro commented Aug 1, 2018

#20030

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment