Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

when archive.php is run multiple times, each concurrent run should archive different websites #3405

Closed
mattab opened this Issue · 10 comments

2 participants

@mattab
Owner

Reported in forums

The goal is to make archiving faster for high traffic piwik server, especially those with thousands of websites. They can exceute php /path/to/cron/archive.php multiple times in parallel, and each instance will archive different websites. Overall it should be much faster than archiving one after the other.

NOte: it used to work as I tested this case when building the feature, but was broken at some point.

@mattab
Owner

(In [7042]) Fixes #3405 Skipping websites which have been processed by another concurrent archive.php

@anonymous-piwik-user

Will it work if archive.php is launched on different servers but for the "same" database ?

@mattab
Owner

It should work yes!

@anonymous-piwik-user

This is a great thing :)

@mattab
Owner

It's not working in 2.0

@mattab
Owner

In 0ea3b0e: Refs #3405 Refactoring archive.php so I can understand it well

@mattab
Owner

In 5db45de: Fixes #3405 Clearning up the code and fixing the logic so archive.php will process websites properly when executed concurrently!

@mattab
Owner

In ea612f9: Refs #3405 Fixing build

@mattab
Owner

In ac3b5bf: Fixes #4309 adding --force-idsites=1,2,n parameter
Refs #3405 fixing regression in concurrent runs

@mattab
Owner

See follow up #4903 Add possibility to run multiple archiver in parallel

@mattab mattab added this to the 2.0 - Piwik 2.0 milestone
@mattab mattab self-assigned this
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@mattab mattab Fixes #3405 Clearning up the code and fixing the logic so archive.php…
… will process websites properly when executed concurrently!
5db45de
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@mattab mattab Refs #3405 Fixing build ea612f9
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@mattab mattab Fixes #4309 adding --force-idsites=1,2,n parameter
Refs #3405 fixing regression in concurrent runs
ac3b5bf
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.