Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Make Archive.php use PHP CLI instead of Web, more reliable / performant #4610

Closed
mattab opened this Issue · 28 comments

2 participants

Matthieu Aubry Thomas Steur
Matthieu Aubry
Owner

The implementation of our Cron Archive script should use Php CLI instead of doing Http requests to the webserver. This would remove one system in the middle and should make archive script slightly more reliable.

Because the piwik.php webcron also uses the CronArchive logic then either we change this logic, or keep the CURL wrapper in the CronArchive (when Php CLI not available or exec() disabled).

Thomas Steur
Owner

In 238e71a: refs #4610 trigger archiving using cli instead of http

Thomas Steur
Owner

In 19385b0: refs #4610 worked on executing cli commands in parallel but not tested yet

Thomas Steur
Owner

In 65d4083: refs #4610 simplified the fallback if async is not supported, tests and fixes for output class

Thomas Steur
Owner

In 095af95: refs #4610 fix headers

Thomas Steur
Owner

In 9cba66f: refs #4610 use existing methods, lock does not work on Windows (yet), save locks in a separate folder

Thomas Steur
Owner

In 279a4d6: refs #4610 I moved the Lock class to CliMulti as it does not 100% work on windows and is only used in CliMulti so far. As the Lock class can be quite useful and maybe needed in other use cases it would be nice to have it somewhere under core or so as it is not directly related to CLI. We might have to move it later again. Need to find a better solution for windows

Thomas Steur
Owner

In 113e403: refs #4610 tests for cliMulti and loads of bugfixes

Thomas Steur
Owner

In f06a7a1: refs #4610 Pid => Process

Thomas Steur
Owner

In a2a586a: refs #4610 make sure the fallback mode works in case async is not supported

Thomas Steur
Owner

In dffb297: refs #4610 use a command to execute a request, this way we do not have to bootstrap everything again and we already make sure it is only executed from the command line

Thomas Steur
Owner

In fdc866a: refs #4610 make sure the command awk exist

Thomas Steur
Owner

In 6247a10: refs #4610 store everything in climulti folder for easier cleanup later, some other optimizations

Thomas Steur
Owner

In cb994c9: refs #4610 prevent some notices like "failed to read from stream" which happens if a subprocess tries to write to a file and the archiver just tries to file_get_content at the same time. So file_exists returned true just a few ms before but while trying to read it fails

Thomas Steur
Owner

In 7947a56: refs #4610 if a process does not start after 8 seconds, finish it as the process probably will not start anyway

Thomas Steur
Owner

In f86af8b: refs #4610 do not start the actual process in case it was marked as finished meanwhile

Thomas Steur
Owner

In 3c5a91d: refs #4610 simplified some logic, fallback to http on windows, redirect the output of php to get output even if piwik exits

Thomas Steur
Owner

In 329115b: refs #4610 simplified, delete old files that were not cleaned up

Thomas Steur
Owner

In fc57db8: refs #4610 I do not like to have the climulti in http class as one could assume it works on all systems etc. Only use it for archiving for now

Thomas Steur
Owner

In 531b357: refs #4610 some code tweaks and optimizations

Thomas Steur
Owner

In 626dcee: refs #4610 another way to get the php binary

Thomas Steur
Owner

In 1fba8ff: refs #4610 added missing groups

Thomas Steur
Owner

In a2d44f5: refs #4610 might fix integration tests

Thomas Steur
Owner

In f39d3cc: refs #4610 fix tests

Thomas Steur
Owner

In 3510045: refs #4610 make sure to rewrite tmp path if needed

Thomas Steur
Owner

In d09986a: refs #4610 create climulti tmp dir before running test

Matthieu Aubry
Owner

In 89b9cfb: Refs #4610 do not require this directory writable, in the UI, only in the archive.php run

Matthieu Aubry
Owner

Great work Thomas, we believe this ticket is now implemented. Please report if you have any feedback or issue!

Thomas Steur
Owner

In d78a9c5: refs #4610 this should fix the tests

Matthieu Aubry mattab added this to the 2.1 - Piwik 2.1 milestone
Thomas Steur tsteur was assigned by mattab
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 trigger archiving using cli instead of http 238e71a
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 simplified the fallback if async is not supported, tests a…
…nd fixes for output class
65d4083
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 fix headers 095af95
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 use existing methods, lock does not work on Windows (yet),…
… save locks in a separate folder
9cba66f
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 I moved the Lock class to CliMulti as it does not 100% wor…
…k on windows and is only used in CliMulti so far. As the Lock class can be quite useful and maybe needed in other use cases it would be nice to have it somewhere under core or so as it is not directly related to CLI. We might have to move it later again. Need to find a better solution for windows
279a4d6
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 tests for cliMulti and loads of bugfixes 113e403
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 Pid => Process f06a7a1
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 use a command to execute a request, this way we do not hav…
…e to bootstrap everything again and we already make sure it is only executed from the command line
dffb297
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 make sure the command awk exist fdc866a
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 store everything in climulti folder for easier cleanup lat…
…er, some other optimizations
6247a10
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 prevent some notices like "failed to read from stream" whi…
…ch happens if a subprocess tries to write to a file and the archiver just tries to file_get_content at the same time. So file_exists returned true just a few ms before but while trying to read it fails
cb994c9
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 if a process does not start after 8 seconds, finish it as …
…the process probably will not start anyway
7947a56
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 do not start the actual process in case it was marked as f…
…inished meanwhile
f86af8b
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 simplified some logic, fallback to http on windows, redire…
…ct the output of php to get output even if piwik exits
3c5a91d
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 I do not like to have the climulti in http class as one co…
…uld assume it works on all systems etc. Only use it for archiving for now
fc57db8
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 some code tweaks and optimizations 531b357
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 another way to get the php binary 626dcee
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 added missing groups 1fba8ff
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 might fix integration tests a2d44f5
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 fix tests f39d3cc
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 make sure to rewrite tmp path if needed 3510045
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 create climulti tmp dir before running test d09986a
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Matthieu Aubry mattab Refs #4610 do not require this directory writable, in the UI, only in…
… the archive.php run
89b9cfb
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Thomas Steur tsteur refs #4610 this should fix the tests d78a9c5
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.