when archive.php is run multiple times, each concurrent run should archive different websites #3405

Closed
mattab opened this Issue Sep 23, 2012 · 10 comments

2 participants

@mattab
Piwik Open Source Analytics member

Reported in forums

The goal is to make archiving faster for high traffic piwik server, especially those with thousands of websites. They can exceute php /path/to/cron/archive.php multiple times in parallel, and each instance will archive different websites. Overall it should be much faster than archiving one after the other.

NOte: it used to work as I tested this case when building the feature, but was broken at some point.

@mattab
Piwik Open Source Analytics member

(In [7042]) Fixes #3405 Skipping websites which have been processed by another concurrent archive.php

@anonymous-piwik-user

Will it work if archive.php is launched on different servers but for the "same" database ?

@mattab
Piwik Open Source Analytics member

It should work yes!

@anonymous-piwik-user

This is a great thing :)

@mattab
Piwik Open Source Analytics member

It's not working in 2.0

@mattab
Piwik Open Source Analytics member

In 0ea3b0e: Refs #3405 Refactoring archive.php so I can understand it well

@mattab
Piwik Open Source Analytics member

In 5db45de: Fixes #3405 Clearning up the code and fixing the logic so archive.php will process websites properly when executed concurrently!

@mattab
Piwik Open Source Analytics member

In ea612f9: Refs #3405 Fixing build

@mattab
Piwik Open Source Analytics member

In ac3b5bf: Fixes #4309 adding --force-idsites=1,2,n parameter
Refs #3405 fixing regression in concurrent runs

@mattab
Piwik Open Source Analytics member

See follow up #4903 Add possibility to run multiple archiver in parallel

@mattab mattab added this to the 2.0 - Piwik 2.0 milestone Jul 8, 2014
@mattab mattab self-assigned this Jul 8, 2014
@sabl0r sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
@mattab mattab Refs #3405 Refactoring archive.php so I can understand it well 0ea3b0e
@sabl0r sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
@mattab mattab Fixes #3405 Clearning up the code and fixing the logic so archive.php…
… will process websites properly when executed concurrently!
5db45de
@sabl0r sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
@mattab mattab Refs #3405 Fixing build ea612f9
@sabl0r sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
@mattab mattab Fixes #4309 adding --force-idsites=1,2,n parameter
Refs #3405 fixing regression in concurrent runs
ac3b5bf
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment