Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

when archive.php is run multiple times, each concurrent run should archive different websites #3405

Closed
mattab opened this Issue · 10 comments

2 participants

Matthieu Aubry Anonymous Piwik user
Matthieu Aubry
Owner

Reported in forums

The goal is to make archiving faster for high traffic piwik server, especially those with thousands of websites. They can exceute php /path/to/cron/archive.php multiple times in parallel, and each instance will archive different websites. Overall it should be much faster than archiving one after the other.

NOte: it used to work as I tested this case when building the feature, but was broken at some point.

Matthieu Aubry
Owner

(In [7042]) Fixes #3405 Skipping websites which have been processed by another concurrent archive.php

Anonymous Piwik user

Will it work if archive.php is launched on different servers but for the "same" database ?

Matthieu Aubry
Owner

It should work yes!

Anonymous Piwik user

This is a great thing :)

Matthieu Aubry
Owner

It's not working in 2.0

Matthieu Aubry
Owner

In 0ea3b0e: Refs #3405 Refactoring archive.php so I can understand it well

Matthieu Aubry
Owner

In 5db45de: Fixes #3405 Clearning up the code and fixing the logic so archive.php will process websites properly when executed concurrently!

Matthieu Aubry
Owner

In ea612f9: Refs #3405 Fixing build

Matthieu Aubry
Owner

In ac3b5bf: Fixes #4309 adding --force-idsites=1,2,n parameter
Refs #3405 fixing regression in concurrent runs

Matthieu Aubry
Owner

See follow up #4903 Add possibility to run multiple archiver in parallel

Matthieu Aubry mattab added this to the 2.0 - Piwik 2.0 milestone
Matthieu Aubry mattab self-assigned this
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Matthieu Aubry mattab Fixes #3405 Clearning up the code and fixing the logic so archive.php…
… will process websites properly when executed concurrently!
5db45de
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Matthieu Aubry mattab Refs #3405 Fixing build ea612f9
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Matthieu Aubry mattab Fixes #4309 adding --force-idsites=1,2,n parameter
Refs #3405 fixing regression in concurrent runs
ac3b5bf
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.