Add possibility to run multiple archiver in parallel

[tsteur]

Add possibility to run multiple archiver via cli in parallel to make archiving of >10k websites faster.

If a server has enough power / resources a user will be able to start for instance 5 archiver in parallel and they will all archive different site ids. Currently, all started archiver would archive the same site ids.

[tsteur]

In 5fcbc09: refs #4903 started to work on the possibility to run multiple archivers in parallel for faster archiving. There were multiple issues, for instance there were arrays of siteIds read and written in Options but options do cache all values in a class property so an update of an option does not get updated on another running archiver. Also all sites were reprocessed because of the time_before_today_archive_considered_outdated setting if the last archiving by another archivier was 10 seconds or longer ago. To prevent this only maintaining a list of to be processed siteids in db / filesystem helps so far

[tsteur]

In 3827424: refs #4903 added some comments and better naming

[hpvd]

Great idea!!
What Do you think about a setting where the advanced User can put in the max number of the cores/threads which are available in the server/should be used max by Piwik?

• this may allow max performance for piwik
• but do not slow down other tasks of the server/ the complete server...

Maybe this is not only interesting for archiving, but also for other processes in Piwik...
=> should be open a other ticket for the general task and collection of ideas for the global topic "parallelization of processes for best performance"?

[hpvd]

have opened a new global ticket for the collection of ideas:
#4905

[tsteur]

In 08a0e53: refs #4903 added tests and fixed some issues

[tsteur]

In 70cd91b: refs #4903 added cli command to trigger archiving

[tsteur]

In 77d9bd0: refs #4903 sleep 25 milliseconds not microseconds

[tsteur]

To run for instance 10 archiver in parallel one can execute the following command:

for ((n=0;n<10;n++)); do (/path/to/php /path/to/piwik/console core:archive --url=http://example.org/piwik &); done

FYI: It is not a good idea to do this in case your archiving tables are using the Myisam engine. More on this will follow

[tsteur]

In c2b6c25: refs #4903 forward archive.php to new console command

[tsteur]

In 42ff0ac: refs #4903 those tests are actually database tests

[tsteur]

In 25c3a80: refs #4903 this might fix the tests

[tsteur]

In 213b5c7: refs #4903 make archiving via web possible again, will provide a better solution later

[tsteur]

In 7a2bf71: refs #4903 better solution for not working web archiving

[tsteur]

In 4bbdd90: refs #4903 removed no longer needed methods, make sure logger uses correct log level in case logger instance was already created, if the test proxy is used do not go two directories upwards

[tsteur]

In d40e217: refs #4903 log if we resume an already existing queue