Custom order tables + new orders UI

[mikejolley]

These are the steps I think we need to cover (in sequence) in order to migrate orders to a custom DB table (#9735).

Phase 1: Data Abstraction - 2.x.x

Phase 1 will be to remove any reliance for core and 3rd party plugins on WP functions, namely post and post meta functions. Also (@claudiosmweb) we really need to start consolidating code used in the API/CLI and WP Backend to update orders. Our current codebase is not DRY at all.

• Introduce wc_get_orders() function to replace get_posts().
• Introduce meta setter and getter functions to avoid any kind of get_post_meta usage, where needed.
• Expand CRUD operations for order classes.
• Replace core usage of get_posts, get_post_meta and direct SQL queries where possible (reports cannot be changed however due to the nature of the complex queries currently required).
• API CRUD usage for order endpoints.
• CLI CRUD usage for order endpoints.

Phase 2: UI Refresh - 3.x.x

Next we need to redo the UI to again remove the reliance on WP core. Since we'll have a decent CRUD by this point, we can make use of React/Backbone to make something that performs well and gives a better experience for the store owner.

This will have the side affect of breaking any plugin which adds custom meta boxes, so new hooks and ways of adding custom content will be needed to ensure things are still extensible.

@jameskoster will design new screens to show order data. Without the constraints of WP admin + meta boxes we can do pretty much anything we like here.

• Replace posts list with custom list table.
• Replace view/edit order screen with custom UI. This will include new panels for downloadable products to get around the performance issues originality reported in Download Permissions Performance #8589

Phase 3: Data Migration - 3.x.x

• Create new tables; woocommerce_orders and woocommerce_ordermeta
• Separate payment status from shipped status. Just throwing this on the list but needs more consideration ( @maxrice @justinstern ) to make statuses more flexible,
• Create migration script (dedicated page with progress bar, or background operation) to migrate posts to orders.
• Switch CRUD operations to new tables.
• Rewrite reports to use new tables. Also a good opportunity to do a refresh of the entire reports section once data is easier to access.

I will create issues for the above points in the relevant milestones, however, I wanted to get feedback prior and ensure I have not missed anything. @pmgarman @thenbrent @maxrice @claudiosmweb @mattyza @allendav @justinshreve @jameskoster .

[rahul286]

Wow! :-) 👍

[claudiosanches]

😃

[lukecav]

😃

[mikejolley]

Non-smiley feedback welcome :D

[pmgarman]

What kind of parameters will wc_get_orders take in an arguments array. My thinking is that what would be taken in while using get_posts would be pretty different from what is taken in on custom tables. Unless the initial version of it while using get_posts was pretty limited and the more advanced parameteters would be added later.

For example: status. If @ 3.0 there becomes a payment and a shipping status, will the wc_get_orders arguments taken a status or go straight to payment_status? Or will status just get deprecated pretty quick in order to move it to payment_status? Obviously it could just stay status but isn't not as obvious when looking at the array of parameters that it is the payment or shipping status.

[mikejolley]

We'd need some of the get_posts args, limit, per page etc + status (we can handle bw compat if we change this in the future), customer_id, etc API has 'status' https://woothemes.github.io/woocommerce-rest-api-docs/#view-list-of-orders

[mikejolley]

@pmgarman did you look at these functions at all - you said you were keen the other day?

[pmgarman]

I've not yet, I'm working through some unrelated reporting work today, tomorrow/Monday I'm planning to get into some abstraction. Was out of office Tuesday/Wednesday.

[mikejolley]

👍 I'll make a branch once I've taken care of 2.5 release

[jkudish]

API CRUD usage for order endpoints.
CLI CRUD usage for order endpoints.

The CLI and API Cruds can probably be one and the same, with the CLI just being a wrapper on top of the API.

I'm keen to help out with API and UI/React parts of this.

[mikejolley]

@jkudish +1

[thenbrent]

Scary (but exciting) stuff.

The 3 phases sounds like a sane way to tackle it.

What kind of parameters will wc_get_orders take in an arguments array.

For reference, Subscriptions has a wcs_get_subscriptions() method which accepts the following args:

 * 'customer_id' The user ID of a customer on the site.
 * 'product_id' The post ID of a WC_Product object
 * 'subscription_status' Any valid subscription status. Can be 'any', 'active', 'cancelled', 'suspended', 'expired', 'pending' or 'trash'. Defaults to 'any'.
 * 'order_id' The post ID of a shop_order post/WC_Order object which was used to create the subscription
 * 'orderby' The field which the subscriptions should be ordered by. Can be 'start_date', 'trial_end_date', 'end_date', 'status' or 'order_id'. Defaults to 'start_date'.
 * 'order' The order of the values returned. Can be 'ASC' or 'DESC'. Defaults to 'DESC'
 * 'subscriptions_per_page' The number of subscriptions to return. Set to -1 for unlimited. Default 10.
 * 'offset' An optional number of subscription to displace or pass over. Default 0.

Those have been sufficient for our needs, which are obviously more limited than the use cases for wc_get_orders().

You'll also notice we opted for explicit subscription param keys for any subscription data instead of accepting the same param keys get_posts() uses (e.g. subscription_status instead of post_status). But we support the same param keys for query related args, like order and offset. We also extend some get_posts() params, like orderby to accept additional values that relate to a subscription (like ordering by trial_end_date).

I can't say whether this is the best approach for wc_get_orders(), though it works for us pretty well.

This will have the side affect of breaking any plugin which adds custom meta boxes

That doesn't necessarily have to be the case. It will be possible (albeit pretty hacky) to maintain backward compatibility even if using React components for displaying new meta boxes. Something we can look at in more detail when the time comes.

[webdados]

Is this only for orders for now, or are you planning on doing the same for products down the road?

I've got a HUGE website where the products (and the WPML translations of each product) are loaded/updated via a excel file with 40+ fields per product and I use wp_update_post and update_post_meta A LOT!

[claudiosanches]

@webdados only orders for now.

[webdados]

@claudiosmweb "for now" :-/

[BuggeringHell]

This is great for stores with large order volumes, it'd be nice to eventually have the same treatment for products as stores with a large, regularly updated catalog (e.g. CSV imports) would really benefit from it.

[shivapoudel]

👍

[webdados]

@BuggeringHell yes, I see it coming along the way. Huge amounts of products is what makes WC websites very slow. Moving them to separate tables would speed things up. I just hope import/export features/API are created. (I know there are tools like WP All Import, but they don't play well with WC+WPML, for example)

[mikejolley]

Products suit posts tables. We also don't need to run complex queries across products like we do orders. So my vote is to leave products where they are.

[BuggeringHell]

Possibly not the product/post rows itself, but the transients/cache? MySQL gets swamped with queries called from delete_version_transients() located in woocommerce/includes/class-wc-cache-helper.php. Unless I am mistaken, it is called every time a product is updated? With very large product catalogues, imports become impossibly slow.

[mikejolley]

@BuggeringHell read the release post for 2.5. Transients have been optimised.

[BuggeringHell]

Oh! I recently updated and tried an import to the same result, maybe I should clear all WC transients? My wp_options table hasn't budged much from ~1,200,000 rows.

[mikejolley]

Yeah, flush them all (WP normally does after updates). Sessions are also no longer stored in options table.

[webdados]

@mikejolley 👍

[stevegrunwell]

@mikejolley I'm happy to volunteer from the Liquid Web side, since I've been driving development for the last month or so. I can also pull in others from this side as necessary (right now they're busy giving ❤️ s to my earlier comment).

[webdados]

@mikejolley Nice!

[marcusquinn]

I think some people here and #11467 #11913 #12065 #17241 might find this site and related patch plugins interesting:

800,000 products in Woocommerce searched & filtered in ~0.5s on a $40 Digital Ocean droplet...

• https://foundthru.com/

Should alleviate some misconceptions on the EAV data-model being an issue for performance too ;)

The only thing we want separate order tables for is dev-ops to make the dev > staging > live process easier.

Performance with correct indexes and queries is only as limited as the hardware with this tuned setup.

[pabl0rg]

@surferking i've purchased your scalability pro plugin and it causes problems with wp_sessions

Having proper tables for products and orders would be better than hacking around the EAV data-model.

[kevin25]

Anh ETA for this, @mikejolley ?

[WPprodigy]

See #10071 (comment) :)

[jonasskafte]

Thanks @WPprodigy – but @kevin25, did you mean an ETA for the implementation of a data migration solution, which @stevegrunwell was kindly offering to help with here and which @mikejolley is suggesting to move forward with here? -or am I missing something? Anyway, if so, then I'm also super interested! 😃

[kevin25]

@jonasskafte If you don't know right now their plugin doesn't work. I can't even migrate any orders. I need it as the core so it can work stablely.

[NathanBarnard]

Looking forward to this!

[milosh416]

We are going to start year 3 of the wait for the fix that stops the ballooning of order tables. How this massive problem that prevents scalability has not made it a higher on the priority list of the core developers is very questionable.

[alexrowesoap]

I've given up on waiting for this after #19937; switched our recommended ecom platform to Sylius for large merchants.

[milosh416]

Who is served by ignoring the issue for 3 years ?

If @mikejolley is running the project, what guides his decision making process on what developers work on ?

In open source transparency goes a long way and this is starting to look like a big corporate game of coming soon in 2016, 2017, 2018, 2019. Give a job to a manager that understands the needs of users and is willing to see the solution completed.

[thisisbbc]

To this day importing and exporting orders still is a nightmare. Wish this was a higher priority.

[pmgarman]

The work done to support this by adding data stores enables developers to put their data wherever they want.

We are now running stores with millions of orders in their databases without the issues we previously experienced previously.

If you are having issues scaling your store I suggest you get data about what is slowing your store down, chances are it is not WC core 9/10 times.

This thread should probably get locked though at this point.

[milosh416]

You have stores running with millions of orders in wp_postmeta ?

[pmgarman]

Hundreds of millions of postmetas in databases. 50-100+ gb in size. Running on our own flavor of auto scaling infrastructure at <$5k/mo driving millions of dollars worth of monthly sales.

I don’t accept that WC can’t scale or drive enterprise e-commerce or high volume e-commerce because we do it regularly.

[crstauf]

The work done to support this by adding data stores enables developers to put their data wherever they want. ...

a.k.a. hire someone to setup custom data stores and tables for you. 👎

@pmgarman why not see the work that's already been done to it's logical conclusion then in core? if all the work for it to happen has been done (which #19937 might suggest it's not been), why stop short?

does someone need to be paid to put this into core? how much? I know it's been done in real-world applications... why hasn't anyone kindly provided their work to be merged into core? why limit it to the big budget projects, or teams with database structure experts? open source seems to lend itself to quality product for the masses, not just those (elites and experts) that know enough and can afford to do things themselves.

I know enough about WooCommerce to customize it, but implementing custom data stores for clients would likely take me 100s of hours, which most clients do not have the budget for.

I get it, the groundwork is there, and the amount of time to implement the custom tables is exhausting; who needs to be paid, and how much, to get this into core for the rest of us devs without expertise, and to the rest of the world?

[pmgarman]

Go look at the LW custom tables plugin, a feature plugin already exists.

Get involved in the existing projects and help continue to test and improve them so someday they may be ready enough for core.

Also yes, I do think that if you are running a high volume or large store then you should have a person or team who can support it properly. A race car can’t succeed without a pit crew to support it.

[milosh416]

Yes we experienced the same, you need $5K per month in CPU power to query a wp_postmeta table with 160 million records > this is the definition of "does not scale"

[crstauf]

@pmgarman If I recall, the LW plugin is no longer supported or working properly. Am I incorrect? Just checked on the LW plugin, and it does appear to be active: https://github.com/liquidweb/woocommerce-custom-orders-table

I certainly (unfortunately) do not have the expertise to structure a database schema. I'm very willing to test out how I can., but as far as I can tell, there are no active projects on this (as indicated by this issue itself).

I think maintenance and creation are two very different things: I certainly can't build a car, but I can care for its basic needs. That's what most of devs do: we (gratefully) receive the code and product that's been built, and care for its installation on our client's sites. If something minor goes wrong (like shipping methods won't save), we can handle that, but we don't have to worry about WooCommerce processing and storing orders; that functionality is robust and stable. That's all we're asking for: custom database tables implemented to the point where we can care for them.

[crstauf]

@milosh416 I think you may have misread @pmgarman comment: he indicated their site costs less than $5k/month.

[milosh416]

@crstauf They are paid by Automattic, regularly. Such a sad state of affairs.

[crstauf]

@milosh416 My question is rather, who can I (or the community) pay to work on custom order tables for WooCommerce core, and how much would it cost (not by whom are the core lead devs paid).

[milosh416]

It's a lost cause after 3 years ...

[crstauf]

@milosh416 I'm not so sure... this is one of those projects that does take time to implement... the entire first phase has been completed, and the second phase (milestone 3.x.x) seems that it might be underway (?). It is certainly disheartening that we may never see this in WooCommerce core, and especially confusing to see products getting custom database tables ahead of orders, but I think it should be acknowledged that changes of this scale do take time.

And let's not forget: WooCommerce is open source... core devs are not the only ones that move the project forward, and certainly cannot take on every new idea or project. Like I said, I am disheartened by the situation, but hopeful that the LiquidWeb project may be the needed jumpstart into core.

[stevegrunwell]

A custom orders table within WooCommerce is far from a lost cause. Around a year ago, @liquidweb and @Mindsize began a collaboration on the WooCommerce Custom Orders Table plugin, which is currently in a Release Candidate state.

This plugin is being built as a drop-in for WooCommerce, making it easy for stores to store common order data in a custom table, taking advantage of the WooCommerce CRUD APIs. We also want to make sure it's easy to migrate existing orders, too, so it ships with a WP-CLI command for automatically migrating orders (or migrating back, if you decide the plugin's not right for you).

As @mikejolley mentioned further up in this thread, the plan is to treat WooCommerce Custom Orders Table as a feature plugin, learn from it, and eventually move towards integrating it into WooCommerce core. In order to do that, we need to get an official 1.0.0 release out, which we're working on (while juggling other projects). If you'd like to help, we'd love to get more people testing the release candidate — moving order data around isn't something we take lightly, so we want to make sure that this is 100% production ready before tagging an official 1.0.0 release.

[crstauf]

Thanks for the update, @stevegrunwell! I for one had thought the LW plugin's development had stalled, so I was surprised (and very glad) to learn that it's moving and in RC! Will definitely set aside some of my personal dev time to testing.

[JJJ]

I invented a relatively sophisticated API for all of the custom database handling we tend to do. This technology was designed to be forked and project agnostic. It will part of Easy Digital Downloads 3.0, Sugar Calendar 2.0, Restrict Content Pro 2.0, and others.

You can find the base of that code here:

https://github.com/easydigitaldownloads/easy-digital-downloads/tree/release/3.0/includes/database

It handles all of the unfun and repetitive database management and registration tasks that come with owning your own storage, including meta-data pairings and caching.

After 13 years of WordPress development, I continue to recommend everyone moves towards this for any application that goes beyond being a blog.

Much like Theme Compatibility in bbPress (around 2010) this set of database APIs was architected as a bit of a hidden gift to other WordPress developers trying to solve the same problems. I hope y'all maybe find it useful in this situation also. 💜

(Ninja Edit: it works really well. There are features I'd like to add, things I'd like to clean up, but time hasn't permitted it on my end.)

[claudiulodro]

This whole thing is not going to be a fast process. Doing it fast would break many stores and would cause much lost revenue for store owners and many problems for plugin developers. The process will span multiple releases and include a lengthy feature-plugin stage to minimize the disruption to users and plugins.

We needed to give the development ecosystem time to stabilize after the initial groundwork laid down for custom tables in the WC 3.0 release, and introducing another giant breaking change shortly after would be counterproductive. Order tables are still on our radar, and improving performance is one of our main priorities for 2019.

Lots of good suggestions for things you can do in the meantime have already been stated here: your own custom data layer, this order tables plugin, this custom indexes plugin should be ready soon-ish, etc.

We'll look into your recommendation @JJJ; thanks for the heads-up. :)

I'm closing comments on this, as it seems we're treading the same conversations over and over again.