Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Setup continuous integration server for Piwik builds #818

Closed
mattab opened this Issue · 52 comments

3 participants

@mattab
Owner

We use bamboo for continuous integration: https://bamboo.openx.org/

currently it runs the webtest that tests the installation process + reset password feature.
it should also run the unit tests and fail when at least one test doesn't pass. there is some magic involved to rewrite simpletests results in XML to be parsed by the tool.

The tool has been failing lately and we would like to setup our own Build integration software on 2 servers we have available for the task.

@robocoder
Collaborator

bamboo.openx.org server appears to be no longer available; should we close this or look into setting up our own (eg using xinc)?

@mattab
Owner

vipsoft, we should consider running our own bamboo, or maybe Hudson as it seems better. We could have 1 or 2 boxes by our sponsor Free to support it - if you want to have a try I can give you the server details

@mattab
Owner

I looked at xing, interesting to build this in php! the project hasn't seen an update for a while though.. I used Hudson in the past, and people were very happy with it; !http://wiki.hudson-ci.org/display/HUDSON/Meet+Hudson

@robocoder
Collaborator

I thought we could use xinc to avoid using a java server, but since webtest is java-based, we might as well use bamboo or hudson.

@mattab
Owner

Note: the current Bamboo is at http://bamboo.openx.org:8085/browse/PIWIK-TRUNK but seems to fail due to some errors. We would like to setup our own integration build server at eg. tests.piwik.org

@mattab
Owner

I asked atlassian and they offered a free license of Bamboo. However, it might be better for us to pay for hosted continuous integration server (bamboo or hudson). If anyone knows such a hosted service, let us know!

@robocoder
Collaborator

System requirements are:

  • multiple PHP CGI versions (e.g., 5.1.3, 5.2.0, 5.2.6, 5.2-SVN, 5.3.1, 5.3-SVN)
  • Tomcat (or other Java server) to support bamboo or hudson; or Apache if we go with xinc
  • Java runtime, ant, JDK for WebTest
  • MySQL (and in the future, other databases)

Is Free still offering a box (or two)? Maybe we can set this up ourselves.

@mattab
Owner

Anthon,
Yes the servers are still available. I can give you the details by email if you'd like to give it a try.

@robocoder
Collaborator

Should test Piwik with:

  • safe_mode disabled and no open_basedir restriction
  • safe_mode enabled
  • open_basedir restriction
  • safe_mode enabled and open_basedir restriction

To test the one click update process, a script could download and install the latest release (latest.zip). Then instead of api.piwik.org/1.0/getLatestVersion, it could check api.piwik.org/1.0/getLatestSvnRevision, and update to the latest svn snapshot, e.g., !http://dev.piwik.org/trac/changeset/{latest revision}/trunk?old_path=%2F&format=zip. (Unlike the build script, Trac packages it in the "trunk" folder instead of "piwik".) It gets a little trickier because the piwik_option table stores the version number in UpdateCheck_LatestVersion, not the SVN version.

@robocoder
Collaborator

It would be nice if there was some support for tinderbox-like scripts to collect build results from remote build servers because it would be desirable to have builds continuously tested on Windows IIS and XAMPP.

@robocoder
Collaborator

Should also include testing with E_STRICT because some environments don't permit the error_reporting level to be overridden by .htaccess or scripts.

@robocoder
Collaborator

(In [2269]) refs #818 - unit tests with both pdo_mysql and mysqli adapters

@robocoder
Collaborator

(In [2270]) refs #818 - fix typo in target

@robocoder
Collaborator

(In [2271]) refs #818 - bamboo.openx.org doesn't have mysqli extension ... comment out for now

@robocoder
Collaborator

Replying to vipsoft:

It would be nice if there was some support for tinderbox-like scripts to collect build results from remote build servers because it would be desirable to have builds continuously tested on Windows IIS and XAMPP.

Cool. Hudson and Bamboo support remote agents, or on the cloud (Amazon EC2) -- what Atlassian calls "elastic agents". Neither provide Amazon Machine Images for Windows, nor support for Microsoft Azure.

@robocoder
Collaborator

Minor issue with Hudson on Jetty: http://issues.hudson-ci.org/browse/HUDSON-6760

@robocoder
Collaborator

(In [2318]) refs #818 - build directory missing a folder

@robocoder
Collaborator

Note to self: It appears there's a bug in HtmlUnit 2.7 that causes WebTest R_1804 to fail the web test after sign in. Sizzle.filter (in jquery.js 1.4.2) is throwing an exception (Syntax error, unrecognized expression: ul li) for this selector in superfish-modified.js, "li:has(ul li)".

@mattab
Owner

Anthon, I have a few questions

  • Will Hudson run on Mysql, mysqli all tests?
  • Is there a way to trigger the JS tests for piwik.js, as well as unit tests and webtests?
  • Would it be better if we migrate to PhpUnit if we want to have the unit tests ran easily (right now Bamboo doesn't parse Simpletest output)

I'm working on acceptance tests that will test Tracker with known data sets, and then call all API methods and Widgets, and check the XML/ HTML output. I hope to finish this in the next few days, which would give great coverage to Piwik testing. Do you know when Hudson service could be available, any chance before end of Month/1.0 release?

Thanks

@robocoder
Collaborator

Replying to matt:

Anthon, I have a few questions

  • Will Hudson run on Mysql, mysqli all tests?

Eventually. I commented out the mysqli unit test because bamboo.openx.org (or rather its elastic agent) doesn't have the mysqli extension installed.

  • Is there a way to trigger the JS tests for piwik.js, as well as unit tests and webtests?

We could use webtest (via HtmlUnit browser simulation). Need to investigate further.

  • Would it be better if we migrate to PhpUnit if we want to have the unit tests ran easily (right now Bamboo doesn't parse Simpletest output)

I might be able to wrap it in a webtest testcase.

I'm working on acceptance tests that will test Tracker with known data sets, and then call all API methods and Widgets, and check the XML/ HTML output. I hope to finish this in the next few days, which would give great coverage to Piwik testing. Do you know when Hudson service could be available, any chance before end of Month/1.0 release?

As of last nite, I have it running locally but the webtest is failing on the second-to-last step. I tried using Webtest 3.0 and the same version of webtest as on bamboo.openx.org (r 1783) but both choked, so I'm now running the latest snapshot and trying to locate the source of the failure in HtmlUnit. Right now, it's not looking hopeful for end-of-the-month.

@robocoder
Collaborator

Re: piwik.js. It looks like we can use Selenium for native browser testing. (There's a Hudson plugin for this.)

@robocoder
Collaborator

(In [2414]) refs #818 - phpdoc 1.4.3 expects this to be an absolute path

@robocoder
Collaborator

Looks like #660 works around the problem in HtmlUnit.

I'm going to start moving my build setup over to dev6.

@mattab
Owner

Awesome.

The continuous build server running Unit tests as well as webtests will be a life saver!! (not to note the possibility of running mysqli and later postgresql...)

@robocoder
Collaborator

Another option for native browser testing of piwik.js is to run our own Testswarm instance. http://wiki.github.com/jeresig/testswarm/

@robocoder
Collaborator

It looks like bamboo.openx.org is down, so there's greater urgency to get this setup on dev6.

In r2429, Matt indicates that Config.test.php fails when run standalone. We should definitely consider iterating through the unit tests, in addition to all_tests.php, as this can catch some side-effects (due to globals).

@mattab
Owner

Fixed the config.test.php in r2433

@robocoder
Collaborator

(In [2445]) refs #818 - add webtest wrapper around unit tests

@robocoder
Collaborator

Hudson is now running at http://qa.piwik.org:8080/ with some the unit tests are failing. (See other tickets.)

I'll open a new ticket for improvements.

@halfdan
Collaborator

Nice work! The feeds are broken though:

http://qa.piwik.org:8080/hudson/rssFailed

The items links to iproxy.ca:8080.

@robocoder
Collaborator

Thanks. Hopefully that's the last config migration change.

@robocoder
Collaborator

(In [2471]) refs #818 - set-up database_tests section for unit testing via webtest wrapper

@mattab
Owner

(In [2472]) Refs #1446 Refs #818

  • Fixing unit tests on local box by ignoring ExampleAPI.getPiwikVersion which would fail at every new version

Note: on my box, when I run all_tests.php, I don't see the green bar at the bottom of the page anymore. It simply displays the Time and Memory delta. I didn't see any commit that coudl have broken this so I'm a bit confused as to why the green/red bar doesn't display anymore?

@mattab
Owner

Reopened as it seems unit tests are not being executed?
See for example output in: http://qa.piwik.org:8080/hudson/job/Piwik/93/console

Also few other questions

  • can we access webtests artifacts? this is important to see what webtests failed, and see what the output was like.
  • can we access integration tests artifacts (eg. in tests/integration/processed/ ?). This is important in case an integration test fails on Hudson but not in local box.
  • are you planning to use Piwik_QA twitter to post build results?

thanks!

@mattab
Owner

(In [2474]) Refs #1125

  • fixing integration tests
  • fixing bug when no visit (called function on non object) Refs #818: all_tests.php is now green on my box
@mattab
Owner

(In [2478]) Refs #818
When tests fail with a fatal error, it will now display as expected in the test page. This was the cause for the green/red bar not displaying (a fatal error was thrown in the UserSettings API called from the Integration tests, and was hidden).

@robocoder
Collaborator

(In [2485]) Refs #818 - upcase words in installation steps; use "JavaScript" consistently; fixes WebTest (installation)

Artifacts can be access through the "Workspace" link, once you're logged in, e.g.,

(There's a configuration option to archive artifacts somewhere...)

Yes, the plan is to use Piwik_QA to tweet build results.

@halfdan
Collaborator

Is it possible to make Artifacts public? I can't login and would very much like to see the current build status and possible errors.

http://trac-hacks.org/wiki/HudsonTracPlugin would be nice for trac to see the build history from hudson in the trac timeline.

@robocoder
Collaborator

re: public artifacts are possible, but we'll have to review the artifacts to ensure no passwords and internal paths are disclosed.

re: hudson trac plugin. Good idea. We don't currently use the Bamboo Trac plugin that's currently installed.

@robocoder
Collaborator

I've rebuilt php with --enable-mbstring, so the mb_convert_encoding calls work (requires for .csv and .tsv output).

The webtest wrapper on the unit tests is currently failing because the output isn't proper html, eg text before the <html> and after the </html>.

http://qa.piwik.org:8080/hudson/job/Piwik/ws/default/tests/results/002_UnitTests/001_response_invoke.xml/*view*/

@robocoder
Collaborator

(In [2488]) refs #818 - clean up output

@robocoder
Collaborator

(In [2489]) refs #818

@robocoder
Collaborator

Current status of our Hudson server:

  • php 5.2.13 and pdo_mysql
  • unit tests via php-cli (excludes tests/integration) passes
  • webtest install passes
  • webtest unit tests (includes tests/integration) passes
  • phpdocs generated
@mattab
Owner

Are the phpdocs artifacts available for download? It would be cool to link the http://piwik.org/docs/tracking-api/ doc page to the PiwikTracker API page (to the doc from latest successful build maybe?).

@robocoder
Collaborator

I made some configuration changes to the job, but they're not having any effect. It appears that webtest results are treated as build artifacts, and artifacts are not published if the build fails.

I'll try scripting a workaround.

@robocoder
Collaborator
@robocoder
Collaborator

In [2502], exclude integration tests if REQUEST_UR or HTTP_HOST not defined; this means we can now fully test with php-cli if these CGI environment variables are defined.

@robocoder
Collaborator

(In [2503]) refs #818 - PATH_INFO and SCRIPT_NAME are also accepted

@mattab
Owner

Great for artifacts. Should the URL be prefixed by the job ID? (or do we only keep artifacts for last build?)

@mattab
Owner

When you say, "if generated", are the latest successful build doc pages always going to be there (ie. failing builds don't delete old docs). Can I safely link to http://qa.piwik.org/phpdocs/Piwik/PiwikTracker.html ?

@robocoder
Collaborator

Webtest results are only for the latest build.

Yes, should be safe to link to the docs. It's only overwritten if the build succeeds.

@mattab
Owner

I linked the page from the bottom of: http://piwik.org/docs/tracking-api/

@mattab mattab added this to the Piwik 0.6.4 milestone
@robocoder robocoder was assigned by mattab
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.