High priority/ Super useful projects
Run code coverage during the tests and produce reports:
The tests are already sometimes failing without running the integration tests.
Enable SchemaSpy http://qa.piwik.org:8080/schema/
Created as a followup / summary of the 2 year old ticket #1470
More information about piwik QA: http://piwik.org/qa/
Other feature ideas
Note: I read on PHP sonar "PHP at least 5.2.6 is required for this procedure and the plugins to work smoothly". It should probably only run in one build anyway (more recent php version).
I would say that following features are important to have to make Hudson very useful:
Hudson doesn't archive artifacts for failed builds. This doesn't appear to be configurable.
I have scripted a partial workaround, copying the webtest results to http://qa.piwik.org/webtest/ and the Piwik documentation to http://qa.piwik.org/phpdocs/ (if generated).
Build results are now emailed to the piwik-builds mailing list.
To test release update:
Set up build server to:
Add web test:
I created a master job called "CI" which does a single "svn up". This then triggers two jobs, "Docs" (phpdoc generation) and "Piwik" (unit tests, web tests, and schemaspy analysis), which can run in parallel.
The SchemaSpy analysis can now be found at http://qa.piwik.org/schema/
Note: since the integration unit tests are now run 3 times (non-webtest pdo_mysql, non-webtest mysqli, and webtest pdo_mysql), the "processed folder" is renamed between runs. If you browse to http://qa.piwik.org:8080/hudson/job/Piwik/ws/build/tests/integration/ you'll see the folder contains:
Ok, default error_reporting is now E_ALL | E_STRICT. all_tests.php is now failing with/without the webtest wrapper on php 5.2.14.
Anthon, unit tests are failing now, it seems the tracking is not working anymore.
Relevant part of http://qa.piwik.org/webtest/002_UnitTests/001_response_invoke.xml is:
<span class="fail">Fail</span>: ../tests/integration/Main.test.php -> Test_Piwik_Integration_Main -> test_noVisit -> at [/home/www/data/root/hudson.private/jobs/Piwik/workspace/build/tests/integration/Integration.php line 159]<br />
Expected GIF beacon, got: <br/>
is there a way to access error logs of the build? would be great to have them viewable online (or inside the zip artifacts).
I expect that something is wrong there. AT the same time, it is on display_errors On so we should see the error in the response, but it seems tracker is just returning blank response? any idea or anything in error logs?
Unit tests are finally green. (Setting allow_url_fopen=On solves the PiwikTracker problem.)
php-cgi errors are now logged to /var/log/jetty/error.log
Great job Anthon! it's all green now :) http://qa.piwik.org:8080/hudson/
Updated webtest to R_1807 and hudson to 1.369.
Rolled comment:2 and comment:6 ideas into main description. (LDAP login for Hudson isn't doable at this time per matt's email.)
Updated server to hudson 1.388, and switched to a single executor. Two executors allowed doc generation and unit testing in parallel. But having 2 also sometimes resulting in double builds, and false build failures.
(In ) refs #1470 - add nightly build script; tweak the webtest build script
(In ) refs #1470 - build environment changes
A good article pointing out Code coverage report in Hudson (via phpunit again, should we migrate...) and : http://techportal.ibuildings.com/2010/09/20/building-a-continuous-integration-server-for-php-with-hudson/
Also interesting: PHP Mess detector and PHP Copy Paste.
We have to install the Chuck Norris plugin
+1 for PHP Unit, simple test really bugs me sometimes.
For the locale, we should check a locale other than german/english. Maybe some chinese/japanese/cyrillic.
Matt: We're now running Jenkins. For the Hudson Trac plugin, please update the URL and re-enable it. Thanks.
Looking at jenkins-php.org I have the feeling we will have to migrate to PHPUnit to benefit all the cool tools around PHP QA in Jenkins. Not sure how easy it will be to change the testing library, hopefuly not too difficult.
(In ) refs #1470 - add WebTests for piwik.js (tracking-related tests are disabled because HtmlUnit behaves differently from native browser)
(In ) refs #1470 - guard against a false positive, e.g., 0 tests of 0 passed, 0 failed.
(In ) refs #1470
(In ) refs #1470
Tracking requests are disabled in WebTest because the HtmlUnit wrapper loads the href on simulated clicks into the current response instead of the designated iframe (WT-428); setting evt.returnValue to false or calling evt.preventDefault() doesn't work as intended, so can't be used as a workaround. Also, GET requests (fetching the web bug/beacon/pixel) are silently ignored (by design).
It would be nice that the build runs faster, for example by only running unit tests first, then another build would run the webtests (which seem to take few minutes).
It is sometimes useful to have the unit tests feedback after a few seconds than the current 7minutes build time, especially because running the full tests on my box takes 3min and I often just run one or two and rely on Jenkins for the full run.
Good point. I'll break this up into separate jobs, and only trigger the webtests if the unit tests pass.
(In ) refs #1470 - fail build early if unit tests fail
Reminder to self of recent changes on dev6 that I have to update in the slack files:
jsphpdoc - does some text transformations and feeds the output to phpDocumentor; since it doesn't parse the JS, it doesn't recognize some of the syntactic sugar to define methods. The result is missing the public methods, but includes many private methods.
jsdoc-toolkit - uses rhino/java; despite failing miserably (initial output has only three functions: cvt_hex, piwik_log, and piwik_track), it's probably the more promising of the two given its feature set (and usage by other projects). It appears to we have to add more tags as hints...
(In ) refs #1470 - add jsdoc-toolkit
ok, Piwik Mobile docs are online at http://qa.piwik.org:8080/jsdocs/
I've updated the QA page, too.
I allready migrated most core and plugin simpletests to phpunit on my local copy. When I got some time I'll have a close look at the integration tests. don't know wether they are easy to migrate or not.
Great to hear about your tests with Phpunit!!! exciting.
Migration tests should be the same to migrate as other tests, they don't use any special logic.
There is a small problem with the integration tests and some other tests. As simpletests are called in a browser there is a local URL to send api-requests to. phpunit is executed on command line and so there is no url. guess we need to put that to a config file in order to have the tests run correctly with phpunit.
Right. Jenkins sets the following environment vars:
Upgraded qa.piwik.org to WebTest R_1812 last nite. No regressions. http://qa.piwik.org:8080/jenkins/job/Piwik/2372/
Work for Simpletest to PHPUnit migration is done in: #3227
Will be a great improvement to have code coverage metrics, can't wait to see how we are doing...
EDIT: see also this presentation about Jenkins and PHP http://www.slideshare.net/sebastian_bergmann/integrate-your-php-project-with-jenkins-oscon-2012 by PHPUnit dev
Many times our sql update code wasn't failure-friendly and would fail when re-executed as it left the DB in a half-updated state.
Having the following test would ensure that all future update scripts written are working in a failure scenario.
A new idea for a webtest:
Dropping Sonar from the todo list.
See ticket for Webtests request: #2935
This ticket is mostly implemented after 2 years! See #3606 for the follow up ticket listing all ideas and improvements to our continuous integration server and processes. Good times!