Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Make Archive.php use PHP CLI instead of Web, more reliable / performant #4610

Closed
mattab opened this Issue · 28 comments

2 participants

@mattab
Owner

The implementation of our Cron Archive script should use Php CLI instead of doing Http requests to the webserver. This would remove one system in the middle and should make archive script slightly more reliable.

Because the piwik.php webcron also uses the CronArchive logic then either we change this logic, or keep the CURL wrapper in the CronArchive (when Php CLI not available or exec() disabled).

@tsteur
Owner

In 238e71a: refs #4610 trigger archiving using cli instead of http

@tsteur
Owner

In 19385b0: refs #4610 worked on executing cli commands in parallel but not tested yet

@tsteur
Owner

In 65d4083: refs #4610 simplified the fallback if async is not supported, tests and fixes for output class

@tsteur
Owner

In 095af95: refs #4610 fix headers

@tsteur
Owner

In 9cba66f: refs #4610 use existing methods, lock does not work on Windows (yet), save locks in a separate folder

@tsteur
Owner

In 279a4d6: refs #4610 I moved the Lock class to CliMulti as it does not 100% work on windows and is only used in CliMulti so far. As the Lock class can be quite useful and maybe needed in other use cases it would be nice to have it somewhere under core or so as it is not directly related to CLI. We might have to move it later again. Need to find a better solution for windows

@tsteur
Owner

In 113e403: refs #4610 tests for cliMulti and loads of bugfixes

@tsteur
Owner

In f06a7a1: refs #4610 Pid => Process

@tsteur
Owner

In a2a586a: refs #4610 make sure the fallback mode works in case async is not supported

@tsteur
Owner

In dffb297: refs #4610 use a command to execute a request, this way we do not have to bootstrap everything again and we already make sure it is only executed from the command line

@tsteur
Owner

In fdc866a: refs #4610 make sure the command awk exist

@tsteur
Owner

In 6247a10: refs #4610 store everything in climulti folder for easier cleanup later, some other optimizations

@tsteur
Owner

In cb994c9: refs #4610 prevent some notices like "failed to read from stream" which happens if a subprocess tries to write to a file and the archiver just tries to file_get_content at the same time. So file_exists returned true just a few ms before but while trying to read it fails

@tsteur
Owner

In 7947a56: refs #4610 if a process does not start after 8 seconds, finish it as the process probably will not start anyway

@tsteur
Owner

In f86af8b: refs #4610 do not start the actual process in case it was marked as finished meanwhile

@tsteur
Owner

In 3c5a91d: refs #4610 simplified some logic, fallback to http on windows, redirect the output of php to get output even if piwik exits

@tsteur
Owner

In 329115b: refs #4610 simplified, delete old files that were not cleaned up

@tsteur
Owner

In fc57db8: refs #4610 I do not like to have the climulti in http class as one could assume it works on all systems etc. Only use it for archiving for now

@tsteur
Owner

In 531b357: refs #4610 some code tweaks and optimizations

@tsteur
Owner

In 626dcee: refs #4610 another way to get the php binary

@tsteur
Owner

In 1fba8ff: refs #4610 added missing groups

@tsteur
Owner

In a2d44f5: refs #4610 might fix integration tests

@tsteur
Owner

In f39d3cc: refs #4610 fix tests

@tsteur
Owner

In 3510045: refs #4610 make sure to rewrite tmp path if needed

@tsteur
Owner

In d09986a: refs #4610 create climulti tmp dir before running test

@mattab
Owner

In 89b9cfb: Refs #4610 do not require this directory writable, in the UI, only in the archive.php run

@mattab
Owner

Great work Thomas, we believe this ticket is now implemented. Please report if you have any feedback or issue!

@tsteur
Owner

In d78a9c5: refs #4610 this should fix the tests

@mattab mattab added this to the 2.1 - Piwik 2.1 milestone
@tsteur tsteur was assigned by mattab
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 trigger archiving using cli instead of http 238e71a
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 simplified the fallback if async is not supported, tests a…
…nd fixes for output class
65d4083
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 fix headers 095af95
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 use existing methods, lock does not work on Windows (yet),…
… save locks in a separate folder
9cba66f
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 I moved the Lock class to CliMulti as it does not 100% wor…
…k on windows and is only used in CliMulti so far. As the Lock class can be quite useful and maybe needed in other use cases it would be nice to have it somewhere under core or so as it is not directly related to CLI. We might have to move it later again. Need to find a better solution for windows
279a4d6
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 tests for cliMulti and loads of bugfixes 113e403
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 Pid => Process f06a7a1
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 use a command to execute a request, this way we do not hav…
…e to bootstrap everything again and we already make sure it is only executed from the command line
dffb297
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 make sure the command awk exist fdc866a
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 store everything in climulti folder for easier cleanup lat…
…er, some other optimizations
6247a10
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 prevent some notices like "failed to read from stream" whi…
…ch happens if a subprocess tries to write to a file and the archiver just tries to file_get_content at the same time. So file_exists returned true just a few ms before but while trying to read it fails
cb994c9
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 if a process does not start after 8 seconds, finish it as …
…the process probably will not start anyway
7947a56
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 do not start the actual process in case it was marked as f…
…inished meanwhile
f86af8b
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 simplified some logic, fallback to http on windows, redire…
…ct the output of php to get output even if piwik exits
3c5a91d
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 I do not like to have the climulti in http class as one co…
…uld assume it works on all systems etc. Only use it for archiving for now
fc57db8
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 some code tweaks and optimizations 531b357
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 another way to get the php binary 626dcee
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 added missing groups 1fba8ff
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 might fix integration tests a2d44f5
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 fix tests f39d3cc
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 make sure to rewrite tmp path if needed 3510045
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 create climulti tmp dir before running test d09986a
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@mattab mattab Refs #4610 do not require this directory writable, in the UI, only in…
… the archive.php run
89b9cfb
@sabl0r sabl0r referenced this issue from a commit in sabl0r/piwik
@tsteur tsteur refs #4610 this should fix the tests d78a9c5
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.