Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Make "All Websites Dashboard" usable with 20,000+ websites #3124

Closed
diosmosis opened this Issue · 8 comments

4 participants

@diosmosis
Collaborator

Currently the All Websites dashboard will work well when there are hundreds or even thousands of websites, but after a point it becomes unusable. It should, however, be able to scale to any number of websites.

There are currently many barriers to scaling this area, including:

  • Lack of server-side pagination means all site data must be queried at once.
  • The design of the Array Archive types will result in unnecessary object creation (in this case several objects per website).
  • There's no way to archive metrics for all websites, so summary & evolution data has to be calculated when queried.

These issues will be addressed in this ticket.

@mattab
Owner

What is the status of this ticket, and how much work do you estimate is left to finish it?

@diosmosis
Collaborator

Still quite a bit of work left. I have to finish the refactoring of the Archive types, then commit the changes, piece by piece. One of the other two changes may not be as hard to implement. I'll send you a patch of what I've done so far so you can take a look (w/o having to review it).

@mattab
Owner

Please contact us if you are interested in this fix!

@pebosi

When using 100-200 sites and changing period to month it's also unusable slow, even with sparklines off. Are there any options to optimize on server-side or in piwik it self?

@diosmosis
Collaborator

Replying to pebosi:

When using 100-200 sites and changing period to month it's also unusable slow, even with sparklines off. Are there any options to optimize on server-side or in piwik it self?

Hi @pebosi, the dashboard should be usable for ~1000 websites on any period except range, but ONLY if browser-initiated archiving is disabled (controlled by the first setting in the General Settings admin page). Is this disabled for you?

@diosmosis
Collaborator

Fixed by Archive.php refactor. When archiving is disabled and there are 20,000 sites each w/ visits, MultiSites.getAll takes 12-14s to return the result on my laptop. Memory usage is at 118 MB.

@anonymous-piwik-user

In 3a3517e: Refs #3124, add benchmark for MultiSites.getAll and benchmark fixture that adds 20,000 sites to the DB then tracks one pageview per site.

@mattab
Owner

Awesome Benaka, truly beautiful!

@diosmosis diosmosis self-assigned this
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@diosmosis diosmosis Refs #3124, add benchmark for MultiSites.getAll and benchmark fixture…
… that adds 20,000 sites to the DB then tracks one pageview per site.
3a3517e
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.