Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

archive.sh script should loop over all websites and trigger archiving in a separate request for each #1227

Closed
mattab opened this Issue · 6 comments

2 participants

@mattab
Owner

Currently, archive.sh simply loops over the periods and triggers archiving for all websites: https://github.com/piwik/piwik/blob/master/misc/cron/archive.sh

because there are memory leaks issues during archiving, see #766, when a Piwik has hundreds or thousands of websites in the system, the memory usage peaks and can lead to ridiculous values.

Until #766 is addressed and as a useful safe guard, we should looop over each website and trigger archiving separately for each. This is a quick fix that will help many users get past the memory usage issues.

@anonymous-piwik-user

Subscribing to this ticket as I would really appreciate that method. Let me know if I can / should test something :)

@mattab
Owner

(In [2025]) Fixes #1227
the archive script now loops over all websites and triggers a different request for each archive, helping out with the archive memory exhausted issue (refs #766)

@anonymous-piwik-user

Thanks for your work on this. Overall I would say its better. Its still aching slow running over the sites and my most active site (15k visits / 20k actions per day, out of around 150k per day) is taking 2 GB to process the yearly stats.

@anonymous-piwik-user

I tried the new archive.sh script, but it didn't work for me, because the list of site id's seems to be utf encoded it doesn't pass the isnumeric test. The convertToUnicode=0 param to the api call doesn't seem to do anything here.

@mattab
Owner

alx, did you apply the full patch? only applying a subset will obviously fail. does it work for you?

@anonymous-piwik-user

Oops, my bad. I didn't apply the full patch, sorry 'bout that.

@mattab mattab added this to the Piwik 0.6 milestone
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.