Improve Customer Export Performance

[pippinsplugins]

When there are a lot of customers, the Customer > CSV export tends to time out. We need to look at making this able to handle 5k+ users.

[pippinsplugins]

It can now successfully export at least 1500 customers, whereas 1500 would timeout.

For some reason though, the purchase count and spent totals aren't working at all on WP Engine.

Anyone able to test this on a live or staging server?

[pippinsplugins]

It's still not good enough and does actually still (on some servers) have a limit of about 1500. I'd like to get it up to 20-30k for the limit.

[chriscct7]

WPEngine issue fixed.

[chriscct7]

With joining of multiple tables, a 20k to 30k record amount simply will not happen. You'll hit perfomance limits of memory limits or timeout limits or most likely both.
There's only so much you can optomize.

There is one way, however, it would require us to make our own table.
Basically, we'd need a purchase table, with all the data we'd normally get from joining.

If you're willing to do it, I'll make it work.

[pippinsplugins]

No custom tables, for several reasons:

1. I really don't want to
2. That would require we migrate all of the data. If it was one or the other, I'd go with not being able to export all at once than migrating the data.

There is definitely a way to do this, we'll just have to get more creative.

Right now we're trying to do it in a single call, which simply won't work. We need to implement a "stepping" system where it does the export in batches of 1000 users at a time. With each batch, a new PHP process is started, thus getting rid of the time out problem.

[pippinsplugins]

I think it will need to all happen via ajax.

[chriscct7]

That still causes problems. Lets say you get around the execution time limit, then you'll hit CPU limitations, or your host will axe the process. Guaranteed.

If you move to custom table, I'll be happy to write a import/export for purchases to handle numbers way larger than your goal.

[pippinsplugins]

With batches like I"m envisioning, you'd have to be exporting probably 100k plus customers before it'd run into any limits, if even then.

[pippinsplugins]

Let's add a radio button to set whether purchase counts and amount spent should be included. Without those two data points, we should be able to export 10k+ pretty easily as it's just a single SQL DISTINCT call.

[pippinsplugins]

Maybe even an option to just export email addresses.

[pippinsplugins]

Providing an option to do just the emails allows a 1500+ customer site to generate all emails in about 2 seconds, as opposed to timing out.

[pippinsplugins]

Names and emails is about 5 seconds.

[pippinsplugins]

With the emails-only option now, I suspect 10-20k users isn't out of the question.