-
-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Custom date range reports are slow: how to make them archive faster? #4768
Comments
Example today's email to feedback@piwik.org: on the All Websites dashboard when default URL is Update dec 2014: was fixed in #6672 |
…nge: do not archive sub-tables (only the parent table). The sub-tables will be archived only when idSubtable is found, or flat=1, or expanded=1
@tsteur this request for example is slow: http://demo.piwik.org/index.php?module=CoreHome&action=index&date=2013-03-05,2015-03-11&period=range&idSite=1#/module=Actions&action=menuGetPageUrls&date=2013-03-05,2015-03-11&period=range&idSite=1 - it took about 40 seconds to archive. Maybe we could make this kind of large date range much faster? |
This request takes 2.3 seconds when I request it... and it should be even faster once #7409 is merged |
One thing that I noticed and took me a while to figure out was that, if someone actually uses range dates, one should disable browser archiving. Otherwise it will always re-archive the last day, week, month or year depending on the range. We might have to do this automatically (disable browser archiving for some subperiods if range is used and an archive already exists) Note: Once we do pre-archive range dates this can become a problem as it would always pre-archive the last year / month / week / day as it will be always authorized to archive |
A lot of improvements were made here. We have to decide next week how we want to continue with this problem. It might make sense to make further improvements when working on #7470 (refactoring the Archiver). One idea was for example to build the range only the requested record. This is not easy to add to the current implementation of the archiver but would bring quite a bit of improvement. Another idea could be to sometimes substract range dates. Eg if today is Easiest to implement and probably the fastest solution as well would be to only fetch the requested recordName and only the requested 1st level table or only the requested subtable. I tested it and it is very fast and easy to implement. Problem is it does not work with subtableIds. We'd have to use labels as subtables as we can generate the subtableIds only if we build the expanded table. Building the expanded table is expensive (in terms of needed time) again. |
Created follow up issues: (in order of importance)
|
The goal of this ticket is to discuss how we could improve the speed and efficiency of the custom date range report aggregation. Currently, archiving custom date ranges is slow.
The text was updated successfully, but these errors were encountered: