Make UI tests work on all servers

[tsteur]

Currently, some screenshot tests containing the PIWIK_INCLUDE_PATH which makes the tests fail on some servers if for instance not the same linux user is used as on travis. Solution: See #6429 as suggested by @diosmosis we could grep the html output for PiwikIncludePath and replace it with '/' for instance.

Currently failing tests on AWS

Installation_system_check
UIIntegrationTest_admin_manage_tracking_code
UIIntegrationTest_admin_plugins Expected    
UIIntegrationTest_admin_security_info   
UIIntegrationTest_admin_visitor_generator
UIIntegrationTest_db_connect_error
Updater_main

System check and security infos both fail as well. Not sure how to make them work on all servers. Probably we'll have to "inject/mock" the security result in PHP layer to see if it is rendered well. For this we could use DI. Mocking it is fine as we are only interested in testing the Controller / template / CSS / JS layer anyway. The actual logic of the security check should be tested in Unit/Integration tests.

[diosmosis]

If SystemCheck & SecurityInfo are migrated to use angularjs, we could use $httpBackend. Which would be closer to what I wanted for UI tests (ie, don't test PHP API too, just JS + UI related PHP code).

[mattab]

@diosmosis you won't easily convince me to skip testing PHP API in UI tests. IMO it's a major benefit of our screenshot tests :)

[diosmosis]

Technically API should be tested by system + Integration tests (and maybe sometimes unit). Ideally, UI shouldn't. And since it currently loads data from a SQL dump that has been pre-archived, it doesn't really.

[tsteur]

@diosmosis 👍 100% true. Such API's have to be covered by unit and integration tests and not by UI tests. UI tests should be only there to "test" Controller, Templates, CSS and partially JS . To be clear: If not, we waste a lot of time (and money) and end up having more bugs etc