Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

UI unit tests #4189

Closed
diosmosis opened this Issue · 17 comments

2 participants

Benaka Matthieu Aubry
Benaka
Collaborator

We should increase our JavaScript code coverage by adding unit tests for individual UI controls. To do this, we need to:

  • Allow JavaScript to be injected before taking screenshots.
  • Create unit tests of all individual UI widgets (eg, site selector, datatable) that compare screenshots after triggering certain events.
Matthieu Aubry
Owner

In 5e644be: Refs #4087 #4189 We would like ui tests build to run every commit even when the Integration tests fail. This will help know exactly which commit introduced a regression.

Benaka
Collaborator

In a68ed87: Refs #4189, use new Omni test fixture for UI tests that combines all other fixtures.

Matthieu Aubry
Owner

@capedfuzz the omni fixture broke the UI tests: https://travis-ci.org/piwik/piwik-ui-tests

Benaka
Collaborator

In be9b410: Refs #4189, added UI unit test base class and tests for ViewDataTable. Refactored capture.js to make new type of testing possible.

Benaka
Collaborator

In 84a22be: Refs #4189, add segment selector/editor unit tests & tweak CSS so tests look right.

Benaka
Collaborator

In f4e39c9: Refs #4739, #4189, remove persist_fixture_data config option code, placed OmniFixture into a namespace, removed PHP screenshot testcase classes and added fixture for use w/ JavaScript screenshot tests.

Benaka
Collaborator

In 7c7a877: Refs #4189, add Annotations.deleteAll API method, work around popover issue in phantomjs (modifying window.location.href directly erases JavaScript state for page even if the URL is the same) and tweak user creation in test fixture setup.

Benaka
Collaborator

In 13a08d5: Refs #4189, added expected screenshot for DBStats, added actual & mock data access class to DBStats, fix bug that occurs when deleting merged assets that do not exist and fix diffviewer generation regression (expected URL was incorrect).

Benaka
Collaborator

In 7161069: Refs #4189, add action to render site selector, add more sites for UI tests for site selector and make site selector has a margin top in UI tests.

Benaka
Collaborator

In e1d338c: Refs #4189, add controller method to CoreHome to render period selector, add periodSelector CSS class to period selector.

Benaka
Collaborator

In 67d34eb: Refs #4189, allow widget factory to be included in getDashboardSettingsControl() action output for tests, make sure last dashboard always overwritten by UI test start.

Benaka
Collaborator

In 57bf5d0: Refs #4189, bug fixes for dashboard & changes to make dashboard UI tests possible. Includes:

  • fix for bug where top controls not displayed correctly when dashboard is displayed in a widget iframe.
  • do not display dashboard manager after dashboard action selected.
  • truncate tables when running UI tests w/ --persist-fixture-data so existing database data won't be used if the fixture setup didn't complete before.
  • do not create super user more than once in Fixture.
  • don't include Test_Piwik_Fixture_CustomAlerts in omni fixture.
  • make sure UITestFixture executes the correct code every time regardless of whether --persist-fixture-data isn't used.
  • add mouseup/mousedown events to screenshot testing page rendere.
  • add ability to call controller methods in screenshot tests and fix API calling mechanism.
Benaka
Collaborator

In 8b48ff2: Refs #4189, allow screenshot tests to change viewport size.

Benaka
Collaborator

In c8d8818: Refs #4189, allow default config file paths to be overridden for tests, rewrite proxy/index.php to use hooks for config instead of disabling dispatch.php and remove no longer used config override code.

Benaka
Collaborator

In 8a525ad: Refs #4189, more work getting UI tests to pass and adding admin screenshots, including:

  • make updatetoken.php use TestingEnvironment in testmode,
  • allow persisted fixture to be dropped using new --drop command line option (for ui test runner),
  • fix debugging mistake in realtime map, fix merge error in empty.twig,
  • make sure Fixture prints to screen only for UI test setup, re-enable CustomAlerts fixture in OmniFixture,
  • make sure forced now value for realtime map is constant,
  • use *.getDatabaseConfig events to override database name in TestingEnvironment
  • use TestingEnvironment in proxy/piwik.php
  • add wait() method to screenshot testing framework's PageRenderer
  • hide current UTC time in manage websites admin screenshot
Benaka
Collaborator

In 5c66663: Refs #4189, adding Overlay screenshot tests:

  • Make 'idsite' parameter in certain Overlay URLs 'idSite'.
  • Make sure Overlay sidebar has width so screenshot test framework can find position.
  • Add overlay test site so Overlay can be loaded in phantomjs.
  • Remove createSuperUser call from TrackerTest (done in Fixture setUp).
  • Make sure piwik.js link is created for screenshot tests & make sure the test environment JSON file is loaded after the database is setup.
  • Modify logic that waits for images to load in screenshot testing framework's page renderer to wait for CSS image URLs.
Matthieu Aubry
Owner

UI Unit Tests Project Summary:

  • Testing framework for UI testing using mocha & chai, both cutting edge libraries.
    • Framework increases the speed of running tests. Even with more than double the amount of tests (from 82 -> 181), the new tests run in less time (5 min less time).
    • Ability to perform benchmarking of UI code, both page load, AJAX requests and individual JavaScript actions.
    • Framework can be used to write other UI tests that run in the browser (such as tests for the AngularJS code).
    • Plugins can now have their own UI tests.
    • Writing UI tests is easier & w/ included README creating tests will be easier for other team members to start writing them.
  • Refactoring of Fixture/IntegrationTest setup code.
    • Opens up the possibility more easily re-introducing benchmarks.
    • Consistent use of TestingEnvironment so testing code is in one place.
  • Significantly increased code coverage, including tests to following logic:
    • Overlay
    • Row Evolution + Annotations (not just load, but switching series, create/update annotations, etc.)
    • Dashboard + Dashboard Manager
    • Site Selector
    • Pie Graph / Bar Graph / Evolution Graph / Actions Data Table / Goals Table
    • Login + Reset Password
    • + Others including DBStats + No Config message + other normal & admin pages, etc.

Intended Outcomes

  • Reduced time to release.
  • Reduced regressions.
  • Reduced time to finished feature.
  • Reduced time to refactor.
  • Increased developer involvement in QA.

Next Steps:

  • We should try and measure the effectiveness of UI unit tests by:
    • Counting all regressions & bugs that appear in 2.1.0 release & recording the amount of days between 2.1.0's intended release day and actual release day. This is the before stats.
    • Counting all regressions & bugs that appear in 2.2.0 release & recording how long it takes to go from 'done' to 'release'.
  • There is currently a large list of UI tests that must be manually done before a list. This entire list could be translated into screenshot tests.

Room for improvement:

  • waiting for one build to finish before starting the UI tests build is a problem
  • debugging travis is still a problem
  • could add benchmarking stats for AJAX (overall & per specific API/Controller method)
  • TestingEnvironment integration can be improved (ie, move Tracker::setTestEnvironment to file)
  • the screenshot testing lib could use an async action JS lib since calling multiple API methods in before()/after() is used
  • when new fixtures are added, the new data is used in the screenshot tests, changing the screenshot. not an issue for us, but potentially one for plugin developers.
  • allowing plugins to add their own PHP code to the UITestFixture isn't possible. can be done w/ an event.
  • to make it exactly like manual UI testing, we can use real data for the tests (ie, from demo.piwik.org). the data could be in an SQL dump we'd modify when making new test cases. this way integration tests can test tracker/aggregation/API, and UI tests can test just the UI.
Benaka diosmosis added this to the 2.2.1 - Piwik 2.2.1 milestone
Benaka diosmosis self-assigned this
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Matthieu Aubry mattab Refs #4087 #4189 We would like ui tests build to run every commit eve…
…n when the Integration tests fail. This will help know exactly which commit introduced a regression.
5e644be
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Benaka diosmosis Refs #4189, use new Omni test fixture for UI tests that combines all …
…other fixtures.
a68ed87
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Benaka diosmosis Refs #4189, added UI unit test base class and tests for ViewDataTable…
…. Refactored capture.js to make new type of testing possible.
be9b410
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Benaka diosmosis Refs #4189, add segment selector/editor unit tests & tweak CSS so tes…
…ts look right.
84a22be
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Benaka diosmosis Refs #4739, #4189, remove persist_fixture_data config option code, pl…
…aced OmniFixture into a namespace, removed PHP screenshot testcase classes and added fixture for use w/ JavaScript screenshot tests.
f4e39c9
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Benaka diosmosis Refs #4189, add Annotations.deleteAll API method, work around popover…
… issue in phantomjs (modifying window.location.href directly erases JavaScript state for page even if the URL is the same) and tweak user creation in test fixture setup.
7c7a877
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Benaka diosmosis Refs #4189, added expected screenshot for DBStats, added actual & moc…
…k data access class to DBStats, fix bug that occurs when deleting merged assets that do not exist and fix diffviewer generation regression (expected URL was incorrect).
13a08d5
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Benaka diosmosis Refs #4189, add action to render site selector, add more sites for UI…
… tests for site selector and make site selector has a margin top in UI tests.
7161069
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Benaka diosmosis Refs #4189, add controller method to CoreHome to render period select…
…or, add periodSelector CSS class to period selector.
e1d338c
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Benaka diosmosis Refs #4189, allow widget factory to be included in getDashboardSettin…
…gsControl() action output for tests, make sure last dashboard always overwritten by UI test start.
67d34eb
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Benaka diosmosis Refs #4189, bug fixes for dashboard & changes to make dashboard UI te…
…sts possible. Includes:

- fix for bug where top controls not displayed correctly when dashboard is displayed in a widget iframe.
- do not display dashboard manager after dashboard action selected.
- truncate tables when running UI tests w/ --persist-fixture-data so existing database data won't be used if the fixture setup didn't complete before.
- do not create super user more than once in Fixture.
- don't include Test_Piwik_Fixture_CustomAlerts in omni fixture.
- make sure UITestFixture executes the correct code every time regardless of whether --persist-fixture-data isn't used.
- add mouseup/mousedown events to screenshot testing page rendere.
- add ability to call controller methods in screenshot tests and fix API calling mechanism.
57bf5d0
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Benaka diosmosis Refs #4189, allow default config file paths to be overridden for test…
…s, rewrite proxy/index.php to use hooks for config instead of disabling dispatch.php and remove no longer used config override code.
c8d8818
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Benaka diosmosis Refs #4189, more work getting UI tests to pass and adding admin scree…
…nshots, including:

- make updatetoken.php use TestingEnvironment in testmode,
- allow persisted fixture to be dropped using new --drop command line option (for ui test runner),
- fix debugging mistake in realtime map, fix merge error in empty.twig,
- make sure Fixture prints to screen only for UI test setup, re-enable CustomAlerts fixture in OmniFixture,
- make sure forced now value for realtime map is constant,
- use *.getDatabaseConfig events to override database name in TestingEnvironment
- use TestingEnvironment in proxy/piwik.php
- add wait() method to screenshot testing framework's PageRenderer
- hide current UTC time in manage websites admin screenshot
8a525ad
Philip Taffner sabl0r referenced this issue from a commit in sabl0r/piwik
Benaka diosmosis Refs #4189, adding Overlay screenshot tests:
- Make 'idsite' parameter in certain Overlay URLs 'idSite'.
- Make sure Overlay sidebar has width so screenshot test framework can find position.
- Add overlay test site so Overlay can be loaded in phantomjs.
- Remove createSuperUser call from TrackerTest (done in Fixture setUp).
- Make sure piwik.js link is created for screenshot tests & make sure the test environment JSON file is loaded after the database is setup.
- Modify logic that waits for images to load in screenshot testing framework's page renderer to wait for CSS image URLs.
5c66663
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.