Allow plugins to bundle unit tests files

[robocoder]

Currently it is not possible for plugins to have tests. plugin developers have to put the tests in the piwik/tests/PHPUnit/* folders. This is not practical because the tests cannot be bundled with the plugin.

Goal: give Piwik developers the ability to bundle tests in their plugins. This will provide community with ability to build plugins with high quality and QA standards.

Tasks:

• Allow plugins to bundle tests in their tests/* directories. Especially unit tests, and integration tests
• Developers can execute their plugin tests
• Add basic doc to plugins developer documentation #testing
• Add console command to create test file and to execute tests for a specific plugin

Out of scope:

• moving tests from PHPUnit/Plugins/* to the respective plugins/*/tests/ directory is not desired and out of scope.
• supporting Benchmark, UI/screenshot tests is out of scope
• Run these tests in Travis CI covered in Let plugins developers run their plugins tests continuously in Travis CI #4280

[mattab]

I think it would be ideal to implement this using an Ant task running on our Hudson server.

[robocoder]

re: comment:1 That's possible. I'll need to secure the CI environment a bit more than currently (e.g., suhosin, open basedir restriction, etc), and limit integration testing to some pre-approved set of third-party plugins.

The build job would iterate through each plugin in the repository:

• install Piwik trunk
• download and install a third-party plugin from the repository
• check for errors
• activate plugin
• check for errors
• run plugin-specific unit tests

[mattab]

From #2017: WP has some interesting plugins for checking themes and plugins that we can learn from.

• Theme Authenticity Checker
• Exploit Scanner
• Theme Check

Lots of possibilities: we could run a test before activating a plugin, test plugins submitted to the repository, svn commit hook, or via jenkins.

[tsteur]

In 53fc1b9: refs #1673 allow plugins to bundle tests

[tsteur]

In bce736f: refs #1673 move it to correct testsuite

[tsteur]

In 793c3f8: refs #1673 added a simple test to example plugin, added test generator

[tsteur]

Test generator: done. ./console generate:test

Executing a test using ./console tests:run ExamplePlugin

[tsteur]

TBD: do we need different options to generate StandardTest (Unit Test), DB Test, Integration Test?

[tsteur]

In 9d48059: refs #1673 always add plugins group

[tsteur]

Link to documentation:

http://developer.piwik.org/guides/plugins#writing-tests-for-your-plugin
https://github.com/piwik/developer-documentation/blob/master/docs/plugins.md#writing-tests-for-your-plugin

@diosmosis Feel free to improve the doc.

[tsteur]

In a1c6c2e: refs #1673 automatically attach "Test" to testname if needed, added optional option to define type of test

[mattab]

In 6156407: Refs #1673 Also allow plugin tests in the /Test folder