Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Campaign summary statistics (performance boost and date range specificity) #6651
TL;DR Make big campaigns practical.
If you have a campaign with 20 events, and 1700 leads a day, that's over a million new rows, per month and if the campaign is still active the queries on the campaign view page will take several minutes on a good day.
Even with various recent improvements, the problem is still growing exponentially as you scale up. This PR aims to fix the problem permanently by summarizing historical campaign data by hours (as services like Google Analytics do). The view will behave exactly the same (same charts, same tabular data, no delays), but will have drastically fewer rows to crunch through to generate the data (no joins, just one flat table with some sums).
A command line function
I've merged in and refactored work from #6021 (Optionally display campaign data by date range) for a number of reasons. This allows you to enable "Use date range" so that all data on the Campaign view respects the date range provided. This makes the view faster and more useful for long-running campaigns. This also adds support for query caching (for whoever may use that feature by modifying their config).
We've been running a variation of this PR (as a patch) in production via mautic-eb since Sept 2018. It's pretty critical for us, as we have campaigns with ~28 million leads in them. Viewing such a campaign without this is literally impossible.
Steps to test this PR:
Before this the count query was still being ran, which in many cases is a huge overhead when there are millions of leads.
We already have the lead count, and that extra lead count costs us many seconds to query. In addition supporting a dynamic orderBy allows us to avoid a tmp table file sort on millions of rows, but we’ll get effectively the same intended result, being the most recent leads added to the campaign in 3ms vs 89s.
After talking with John, I realized that if there is a SQL error at high load, this would be one way to remedy the issue by cron.
@imihandstand if you are using a recent version like 2.14.2+ you may be able to apply this as a git patch. Go into your mautic folder (not in production) and run
That will overlay these code changes with your mautic install.
If you want to be on the safe side, wait till version 2.16.0 which will likely have this in it.
I've resolved conflicts, but note that pending counts are recalculated as of 2.15.1 beta (which is good). Giving pending figures is impractical when filtering by date, so I've disabled that for now when summary/dates are enabled. May revisit this in the future, but if not, just keep in mind that isn't supported with this optional feature.