Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Continuous Integration server improvements #1470

Closed
mattab opened this Issue · 44 comments

5 participants

@mattab
Owner

High priority/ Super useful projects

Improvement ideas

  • Run code coverage during the tests and produce reports:
    The tests are already sometimes failing without running the integration tests.

  • Enable SchemaSpy http://qa.piwik.org:8080/schema/

  • Enable Nightly Build at: http://qa.piwik.org:8080/nightly/
  • Currently we must manually run the /tests/javascript/. We could use Selenium (and Sauce Labs) to run ./tests/javascript/. Selenium has the advantage of native browser testing.
  • Nightly builds for Piwik Mobile (Android), see test install script
  • automated API doc generation for piwik.js tracker
    • jsphpdoc - does some text transformations and feeds the output to phpDocumentor; since it doesn't parse the JS, it doesn't recognize some of the syntactic sugar to define methods. The result is missing the public methods, but includes many private methods.
    • jsdoc-toolkit - uses rhino/java; despite failing miserably (initial output has only three functions: cvt_hex, piwik_log, and piwik_track), it's probably the more promising of the two given its feature set (and usage by other projects). It appears to we have to add more tags as hints...

Created as a followup / summary of the 2 year old ticket #1470

More information about piwik QA: http://piwik.org/qa/

@mattab
Owner

Other feature ideas

Note: I read on PHP sonar "PHP at least 5.2.6 is required for this procedure and the plugins to work smoothly". It should probably only run in one build anyway (more recent php version).

I would say that following features are important to have to make Hudson very useful:

  • the console output doesn't include the webtest output
    • also need to provide links to artifacts
  • hudson can send out email notifications re: build results
    • shall I create a mailing list similar to piwik-svn?
    • Hudson could send all build results to the committer email (do you have access to this info?), as well as to the mailing list.
@robocoder

Hudson doesn't archive artifacts for failed builds. This doesn't appear to be configurable.

I have scripted a partial workaround, copying the webtest results to http://qa.piwik.org/webtest/ and the Piwik documentation to http://qa.piwik.org/phpdocs/ (if generated).

@robocoder

Build results are now emailed to the piwik-builds mailing list.

@robocoder

To test release update:

Set up build server to:

  • provide service which mimics api.piwik.org/1.0/getLatestVersion
  • serve up latest.zip (svn export trunk and bundle it up in a zip file)
  • restrict to local access (127.0.0.1) to prevent abuse

Add web test:

  • if core/Version.php differs from (real) api.piwik.org/1.0/getLatestVersion
    • svn export using tag
    • install
    • generate fake visits
    • override piwik_url and piwik_host in config.ini.php
    • delete last update check timestamp (in options table)
    • visit dashboard and start the auto-update
    • run through the updater
    • check for errors
@robocoder

I created a master job called "CI" which does a single "svn up". This then triggers two jobs, "Docs" (phpdoc generation) and "Piwik" (unit tests, web tests, and schemaspy analysis), which can run in parallel.

The SchemaSpy analysis can now be found at http://qa.piwik.org/schema/

Note: since the integration unit tests are now run 3 times (non-webtest pdo_mysql, non-webtest mysqli, and webtest pdo_mysql), the "processed folder" is renamed between runs. If you browse to http://qa.piwik.org:8080/hudson/job/Piwik/ws/build/tests/integration/ you'll see the folder contains:

  • expected
  • processed
  • processed-mysqli
  • processed-pdo-mysql
  • Integration.php
  • Main.test.php
  • proxy-piwik.php
@robocoder

Ok, default error_reporting is now E_ALL | E_STRICT. all_tests.php is now failing with/without the webtest wrapper on php 5.2.14.

@mattab
Owner

Anthon, unit tests are failing now, it seems the tracking is not working anymore.

Relevant part of http://qa.piwik.org/webtest/002_UnitTests/001_response_invoke.xml is:

<span class="fail">Fail</span>: ../tests/integration/Main.test.php -&gt; Test_Piwik_Integration_Main -&gt; test_noVisit -&gt;  at [/home/www/data/root/hudson.private/jobs/Piwik/workspace/build/tests/integration/Integration.php line 159]<br />

Expected GIF beacon, got: <br/>

is there a way to access error logs of the build? would be great to have them viewable online (or inside the zip artifacts).

I expect that something is wrong there. AT the same time, it is on display_errors On so we should see the error in the response, but it seems tracker is just returning blank response? any idea or anything in error logs?

@robocoder

Unit tests are finally green. (Setting allow_url_fopen=On solves the PiwikTracker problem.)

php-cgi errors are now logged to /var/log/jetty/error.log

@mattab
Owner

Great job Anthon! it's all green now :) http://qa.piwik.org:8080/hudson/

@robocoder

Updated webtest to R_1807 and hudson to 1.369.

@robocoder

Rolled comment:2 and comment:6 ideas into main description. (LDAP login for Hudson isn't doable at this time per matt's email.)

@robocoder

Updated server to hudson 1.388, and switched to a single executor. Two executors allowed doc generation and unit testing in parallel. But having 2 also sometimes resulting in double builds, and false build failures.

@robocoder

(In [3433]) refs #1470 - add nightly build script; tweak the webtest build script

@robocoder

(In [3434]) refs #1470 - build environment changes

@mattab
Owner

A good article pointing out Code coverage report in Hudson (via phpunit again, should we migrate...) and : http://techportal.ibuildings.com/2010/09/20/building-a-continuous-integration-server-for-php-with-hudson/

Also interesting: PHP Mess detector and PHP Copy Paste.

@mattab
Owner

We have to install the Chuck Norris plugin

@halfdan
Collaborator

+1 for PHP Unit, simple test really bugs me sometimes.

For the locale, we should check a locale other than german/english. Maybe some chinese/japanese/cyrillic.

@robocoder

Matt: We're now running Jenkins. For the Hudson Trac plugin, please update the URL and re-enable it. Thanks.

@mattab
Owner

done.

Looking at jenkins-php.org I have the feeling we will have to migrate to PHPUnit to benefit all the cool tools around PHP QA in Jenkins. Not sure how easy it will be to change the testing library, hopefuly not too difficult.

@robocoder

(In [3889]) refs #1470 - add WebTests for piwik.js (tracking-related tests are disabled because HtmlUnit behaves differently from native browser)

@robocoder

(In [3891]) refs #1470 - guard against a false positive, e.g., 0 tests of 0 passed, 0 failed.

@robocoder

(In [4109]) refs #1470 - test piwik.js against third-party javascript frameworks (dojo, extjs, jquery, prototype, mootools)

@robocoder

(In [4110]) refs #1470

@robocoder

(In [4111]) refs #1470

@robocoder

Tracking requests are disabled in WebTest because the HtmlUnit wrapper loads the href on simulated clicks into the current response instead of the designated iframe (WT-428); setting evt.returnValue to false or calling evt.preventDefault() doesn't work as intended, so can't be used as a workaround. Also, GET requests (fetching the web bug/beacon/pixel) are silently ignored (by design).

@mattab
Owner

It would be nice that the build runs faster, for example by only running unit tests first, then another build would run the webtests (which seem to take few minutes).

It is sometimes useful to have the unit tests feedback after a few seconds than the current 7minutes build time, especially because running the full tests on my box takes 3min and I often just run one or two and rely on Jenkins for the full run.

@robocoder

Good point. I'll break this up into separate jobs, and only trigger the webtests if the unit tests pass.

@robocoder

(In [4286]) refs #1470 - fail build early if unit tests fail

@robocoder

Reminder to self of recent changes on dev6 that I have to update in the slack files:

  • php-cli: set memory_limit to -1
  • piwik_qa db user: GRANT FILE
@robocoder

jsphpdoc - does some text transformations and feeds the output to phpDocumentor; since it doesn't parse the JS, it doesn't recognize some of the syntactic sugar to define methods. The result is missing the public methods, but includes many private methods.

jsdoc-toolkit - uses rhino/java; despite failing miserably (initial output has only three functions: cvt_hex, piwik_log, and piwik_track), it's probably the more promising of the two given its feature set (and usage by other projects). It appears to we have to add more tags as hints...

@robocoder

(In [4380]) refs #1470 - add jsdoc-toolkit

@tsteur
Owner

Attachment:
install.sh

@robocoder

ok, Piwik Mobile docs are online at http://qa.piwik.org:8080/jsdocs/

I've updated the QA page, too.

@robocoder

PHP 5.4 dropped the sqlite extension -- required by tests/javascript/.

@mattab
Owner
@sgiehl
Collaborator

I allready migrated most core and plugin simpletests to phpunit on my local copy. When I got some time I'll have a close look at the integration tests. don't know wether they are easy to migrate or not.

@mattab
Owner

Great to hear about your tests with Phpunit!!! exciting.

Migration tests should be the same to migrate as other tests, they don't use any special logic.

@sgiehl
Collaborator

There is a small problem with the integration tests and some other tests. As simpletests are called in a browser there is a local URL to send api-requests to. phpunit is executed on command line and so there is no url. guess we need to put that to a config file in order to have the tests run correctly with phpunit.

@robocoder

Right. Jenkins sets the following environment vars:

export DISPLAY=:0.0
export REMOTE_ADDR=127.0.0.1
export HTTP_HOST=localhost:8080
export REQUEST_URI=/path/to/all_tests.php
@robocoder

Upgraded qa.piwik.org to WebTest R_1812 last nite. No regressions. http://qa.piwik.org:8080/jenkins/job/Piwik/2372/

@mattab
Owner

Work for Simpletest to PHPUnit migration is done in: #3227
Will be a great improvement to have code coverage metrics, can't wait to see how we are doing...

EDIT: see also this presentation about Jenkins and PHP http://www.slideshare.net/sebastian_bergmann/integrate-your-php-project-with-jenkins-oscon-2012 by PHPUnit dev

@mattab
Owner

Many times our sql update code wasn't failure-friendly and would fail when re-executed as it left the DB in a half-updated state.

Having the following test would ensure that all future update scripts written are working in a failure scenario.

A new idea for a webtest:

  • Download 1.8.3 release
  • Download latest
  • Run Auto update - but make sure the auto update code fails at the last step
  • Re-run auto update: the code should still run and the SQL not fail. For this, some queries will throw exceptions which must be specifically ignored, eg. "1060 Duplicate column name 'xx'" can be ignored for an ALTER that adds a column.
  • The update re-running after initial failure should succeed.
@robocoder

Dropping Sonar from the todo list.

@mattab
Owner

See ticket for Webtests request: #2935

This ticket is mostly implemented after 2 years! See #3606 for the follow up ticket listing all ideas and improvements to our continuous integration server and processes. Good times!

@mattab mattab added this to the 1.10 - Piwik 1.10 milestone
@robocoder robocoder was assigned by mattab
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.