Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Archive.php use PHP CLI instead of Web, more reliable / performant #4610

Closed
mattab opened this issue Feb 2, 2014 · 28 comments
Closed

Make Archive.php use PHP CLI instead of Web, more reliable / performant #4610

mattab opened this issue Feb 2, 2014 · 28 comments
Assignees
Labels
Milestone

Comments

@mattab
Copy link
Member

@mattab mattab commented Feb 2, 2014

The implementation of our Cron Archive script should use Php CLI instead of doing Http requests to the webserver. This would remove one system in the middle and should make archive script slightly more reliable.

Because the piwik.php webcron also uses the CronArchive logic then either we change this logic, or keep the CURL wrapper in the CronArchive (when Php CLI not available or exec() disabled).

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 5, 2014

In 238e71a: refs #4610 trigger archiving using cli instead of http

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 10, 2014

In 19385b0: refs #4610 worked on executing cli commands in parallel but not tested yet

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 10, 2014

In 65d4083: refs #4610 simplified the fallback if async is not supported, tests and fixes for output class

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 10, 2014

In 095af95: refs #4610 fix headers

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 10, 2014

In 9cba66f: refs #4610 use existing methods, lock does not work on Windows (yet), save locks in a separate folder

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 10, 2014

In 279a4d6: refs #4610 I moved the Lock class to CliMulti as it does not 100% work on windows and is only used in CliMulti so far. As the Lock class can be quite useful and maybe needed in other use cases it would be nice to have it somewhere under core or so as it is not directly related to CLI. We might have to move it later again. Need to find a better solution for windows

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 11, 2014

In 113e403: refs #4610 tests for cliMulti and loads of bugfixes

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 11, 2014

In f06a7a1: refs #4610 Pid => Process

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 11, 2014

In a2a586a: refs #4610 make sure the fallback mode works in case async is not supported

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 11, 2014

In dffb297: refs #4610 use a command to execute a request, this way we do not have to bootstrap everything again and we already make sure it is only executed from the command line

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 11, 2014

In fdc866a: refs #4610 make sure the command awk exist

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 11, 2014

In 6247a10: refs #4610 store everything in climulti folder for easier cleanup later, some other optimizations

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 11, 2014

In cb994c9: refs #4610 prevent some notices like "failed to read from stream" which happens if a subprocess tries to write to a file and the archiver just tries to file_get_content at the same time. So file_exists returned true just a few ms before but while trying to read it fails

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 11, 2014

In 7947a56: refs #4610 if a process does not start after 8 seconds, finish it as the process probably will not start anyway

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 11, 2014

In f86af8b: refs #4610 do not start the actual process in case it was marked as finished meanwhile

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 11, 2014

In 3c5a91d: refs #4610 simplified some logic, fallback to http on windows, redirect the output of php to get output even if piwik exits

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 11, 2014

In 329115b: refs #4610 simplified, delete old files that were not cleaned up

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 11, 2014

In fc57db8: refs #4610 I do not like to have the climulti in http class as one could assume it works on all systems etc. Only use it for archiving for now

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 11, 2014

In 531b357: refs #4610 some code tweaks and optimizations

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 12, 2014

In 626dcee: refs #4610 another way to get the php binary

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 12, 2014

In 1fba8ff: refs #4610 added missing groups

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 12, 2014

In a2d44f5: refs #4610 might fix integration tests

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 12, 2014

In f39d3cc: refs #4610 fix tests

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 12, 2014

In 3510045: refs #4610 make sure to rewrite tmp path if needed

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 12, 2014

In d09986a: refs #4610 create climulti tmp dir before running test

@mattab
Copy link
Member Author

@mattab mattab commented Feb 12, 2014

In 89b9cfb: Refs #4610 do not require this directory writable, in the UI, only in the archive.php run

@mattab
Copy link
Member Author

@mattab mattab commented Feb 16, 2014

Great work Thomas, we believe this ticket is now implemented. Please report if you have any feedback or issue!

@tsteur
Copy link
Member

@tsteur tsteur commented Feb 17, 2014

In d78a9c5: refs #4610 this should fix the tests

@mattab mattab added this to the 2.1 - Piwik 2.1 milestone Jul 8, 2014
@mattab mattab added the T: Task label Jul 8, 2014
@tsteur tsteur modified the milestones: 2.1 - Piwik 2.1, 2.4.0 - Piwik 2.4.0 Jul 11, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…ows (yet), save locks in a separate folder
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…t 100% work on windows and is only used in CliMulti so far. As the Lock class can be quite useful and maybe needed in other use cases it would be nice to have it somewhere under core or so as it is not directly related to CLI. We might have to move it later again. Need to find a better solution for windows
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…do not have to bootstrap everything again and we already make sure it is only executed from the command line
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…tream" which happens if a subprocess tries to write to a file and the archiver just tries to file_get_content at the same time. So file_exists returned true just a few ms before but while trying to read it fails
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…ish it as the process probably will not start anyway
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…ws, redirect the output of php to get output even if piwik exits
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
… as one could assume it works on all systems etc. Only use it for archiving for now
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants