Make "All Websites Dashboard" usable with 20,000+ websites

[diosmosis]

Currently the All Websites dashboard will work well when there are hundreds or even thousands of websites, but after a point it becomes unusable. It should, however, be able to scale to any number of websites.

There are currently many barriers to scaling this area, including:

• Lack of server-side pagination means all site data must be queried at once.
• The design of the Array Archive types will result in unnecessary object creation (in this case several objects per website).
• There's no way to archive metrics for all websites, so summary & evolution data has to be calculated when queried.

These issues will be addressed in this ticket.

[mattab]

What is the status of this ticket, and how much work do you estimate is left to finish it?

[diosmosis]

Still quite a bit of work left. I have to finish the refactoring of the Archive types, then commit the changes, piece by piece. One of the other two changes may not be as hard to implement. I'll send you a patch of what I've done so far so you can take a look (w/o having to review it).

[mattab]

Please contact us if you are interested in this fix!

[pebosi]

When using 100-200 sites and changing period to month it's also unusable slow, even with sparklines off. Are there any options to optimize on server-side or in piwik it self?

[diosmosis]

Replying to pebosi:

When using 100-200 sites and changing period to month it's also unusable slow, even with sparklines off. Are there any options to optimize on server-side or in piwik it self?

Hi @pebosi, the dashboard should be usable for ~1000 websites on any period except range, but ONLY if browser-initiated archiving is disabled (controlled by the first setting in the General Settings admin page). Is this disabled for you?

[diosmosis]

Fixed by Archive.php refactor. When archiving is disabled and there are 20,000 sites each w/ visits, MultiSites.getAll takes 12-14s to return the result on my laptop. Memory usage is at 118 MB.

[anonymous-matomo-user]

In 3a3517e: Refs #3124, add benchmark for MultiSites.getAll and benchmark fixture that adds 20,000 sites to the DB then tracks one pageview per site.

[mattab]

Awesome Benaka, truly beautiful!