Skip to content

Latest commit

 

History

History
185 lines (136 loc) · 8.84 KB

tests.md

File metadata and controls

185 lines (136 loc) · 8.84 KB
Raw

Tests

About tests

Tests use the mocha test framework. Start by installing mocha globally:

npm install --global mocha

Most tests use webdriver.io framework (via Spectron) to bring up the browser and run through the tests. Please check out the API documentation for a comprehensive list of commands we can use in our tests.

Tests are located in the top level test directory and their filenames have a suffix of Test.js.

Watching for changes for tests

To stay close to production, tests do not use the webpack dev server. To keep the webpack bundle up to date as you make changes you should run:

npm run watch-test

Running all tests

You can run ALL of the tests using:

npm run test

If you'd prefer to only run the unit tests, you can do that using:

npm run unittest

Running a subset of tests

You can run a subset of tests which match a description or it with:

npm run test -- --grep="expression"

Where expression could be for example ^tabs to match all tests which start with the word tabs. This works for all testing modes (test, unittest).

Things you should know

Background image for new tab page is disabled by default

To speed-up tests, background image for new tab page is disabled by default. If your new webdriver test needs the background to be visible you'll need to enable this setting again, for example:

it('shows new tab page background', function * () {
  yield this.app.client
    // enable setting again:
    .changeSetting('tabs.show-dashboard-images', true)
    // keep testing...
})

Best practices for writing tests

  • If you do anything that opens a new tab, you have to validate that the tab has opened before trying to switch to another tab/window. This is the most common cause of race conditions that I find in tests. webdriver automatically switches contexts to any new "window" that is created. chromedriver can’t differentiate between tabs and windows in brave which is why we have special helpers. If there is more than one tab with the same url, you can’t use waitForUrl: it will return on the tab/window that already exists and not wait for the new one. getTabCount is a good alternative because it doesn’t require a particular window context to run

  • Don’t try to interact with any dom element (click, moveTo, etc…) until you have verified that it exists. Verifying the page or some parent element is not sufficient

  • Avoid using external sites unless it’s absolutely necessary. For example where we need to do this: the httpse and other ssl related tests make external calls because we don’t current have ssl capabilities on the test server.

    Adding an external dependency causes these issues:

    • tests will run slower since external requests need to be made
    • tests are not runnable if network is down
    • if the site's content changes, the test will break

  • Never assume your tests will run in a particular order. Before each test is run, you should do the setup or cleanup required in order for the test to run properly.

  • Where it makes sense, prefer to use a before() instead of beforeEach(). Because the webdriver initialization takes a while, having it set up once and being able to run multiple tests is going to help speed things up. Please note that this isn't always an option though, because the test might alter the environment in a way that you need to redo the setup (in which case a beforeEach() OR moving that test into it's own describe()/before() is a better choice).

Useful helper methods

These are methods we own which can be used along side the existing webdriver methods. Please note that it's fairly easy to add new helper methods. For more info, please check out the addCommands() method in at brave.js.

IPC

  • ipcSend: send an IPC message, To see more message types, check out componentDidMount in main.js
  • ipcSendRenderer

Tab management

  • tabHandles
  • tabByIndex
  • getTabCount
  • tabByUrl
  • waitForTabCount
  • pinTabByIndex

Window management

  • waitForBrowserWindow
  • setContextMenuDetail: hides any context menus which are actively showing
  • showFindbar
  • getDefaultWindowHeight
  • getDefaultWindowWidth
  • resizeWindow
  • windowParentByUrl
  • windowByUrl

State management

  • getAppState
  • getWindowState

Site management

  • addSite: add a bookmark/folder or history entry
  • addSiteList
  • removeSite

Preferences

  • changeSetting
  • changeSiteSetting

and many more! If you're not sure how to use the methods, search our existing tests to find an example.

Please note that many of these methods do NOT wait until the action is complete. Until the helper methods are updated, you'll have to ensure of this in the test. Click here to see a good example which presented itself with intermittent failures (because sometimes action would be complete, sometimes it wouldn't). In the example linked to above, the condition is tested by waiting until the tab count increases (which means the IPC message was received and processed).

Debugging tests

Enabling verbose mode

You can get more verbose output from some of our commands by setting the BRAVE_TEST_COMMAND_LOGS=1 env variable. If BRAVE_TEST_COMMAND_LOGS=1 is set, tests will output extra information which can be very valuable when trying to understand what is going wrong with a failing test.

As an example, here's the output you get when running the blocks custom adblock resources in private tab test in test/components/braveryPanelTest.js with verbose logging enabled:

  Bravery Panel
    Stats
waitForUrl("chrome-extension://mnojpmjdmbbfmejpflffifhffcmidifd/about-newtab.html")
tabByUrl("chrome-extension://mnojpmjdmbbfmejpflffifhffcmidifd/about-newtab.html") => {"sessionId":"4a74bef383e5d2abba1c0640aaca81e8","status":0,"value":null}
waitForBrowserWindow()
waitForBrowserWindow() => {"sessionId":"4a74bef383e5d2abba1c0640aaca81e8","status":0,"value":null}
waitForDataFile("adblock")
waitForDataFile("adblock") => undefined
waitForDataFile("adblock") => undefined
waitForDataFile("adblock") => {"etag":"\"215b010f2f5ff8b102896957564862d7\"","lastCheckDate":1477083465282,"lastCheckVersion":"2"}
tabByIndex(0)
tabHandles() => handles.length = 1; handles[0] = "CDwindow-e532c598-ab85-4114-9bef-8cfbfc14035e";
loadUrl("chrome-extension://mnojpmjdmbbfmejpflffifhffcmidifd/about-adblock.html")
loadUrl.url() => {"sessionId":"4a74bef383e5d2abba1c0640aaca81e8","status":0,"value":null}
waitForUrl("chrome-extension://mnojpmjdmbbfmejpflffifhffcmidifd/about-adblock.html")
tabByUrl("chrome-extension://mnojpmjdmbbfmejpflffifhffcmidifd/about-adblock.html") => {"sessionId":"4a74bef383e5d2abba1c0640aaca81e8","status":0,"value":null}
waitForTabCount(2)
getTabCount() => 1
getTabCount() => 2
waitForUrl("http://localhost:23188/adblock.html")
tabByUrl("http://localhost:23188/adblock.html") => {"sessionId":"4a74bef383e5d2abba1c0640aaca81e8","status":0,"value":null}
openBraveMenu()
      ✓ blocks custom adblock resources in private tab (4569ms)

Using the debug() webdriver command

You can debug tests by using the command yield this.app.client.debug() in a test. Or most of the time just append .debug() to a series of commands.

This will pause the browser from running tests, and you can open up browser dev tools or content dev tools to inspect logs, console, and other things. You should act fast or else adjust the timeout or the test will fail though.

To get browser process logs, run tests with the BRAVE_TEST_BROWSER_LOGS=1 envrionment variable. To get renderer process logs, run tests with the BRAVE_TEST_RENDERER_LOGS=1 envrionment variable.

Avoiding intermittent failures in automated UI tests

UI tests are very easy to get wrong and introduce intermittent failures. For that reason, and because UI tests are much slower, unit tests are always preferred.

Here are some strategies to avoid intermittent failures:

  • Always be explicit.
  • Never assume that state will be reflected right away, for example if loading a URL, don't assume an entry is already added to the app's sites state. Instead use waitForSiteEntry.
  • Enable verbose mode as described above to get better information.
  • Prefer adding Webdriver IO commands when possible, especially for waiting operations, and add logging to them.
  • When possible, make the Webdriver commands you add wait for a result. Example: Changing a seting also waits for the changed value to be reflected in state.
  • Look out for elements that can be disabled temporarily, in this case, a click could for example happen while the element is still disabled.