Testing
In order to support continuous development/integration, taking advantage of automated testing is a must. While the goal of automated testing is to provide full coverage, it also needs to be seamlessly integrated into the developer environment. This document describes the testing setup for EncoreUI.
More details on these files are found in the files themselves
karma.conf.js - Used by component unit tests
protractor.conf.js - Used by our midway/E2E tests
protractor.visual.regression.js - Used by our visual regression tests (Travis CI only)
Goal: Tests smallest piece of functionality or method
- Code level tests
- Uses Karma + Mocha + Chai + Sinon
- Stored in same location as code (separate file with *.spec.js)
- Best for testing services, classes and objects
- Does contain "functional" tests (can test browser interactions like 'click')
- Sandboxed & Isolated testing
- Mocking & Stubbing required
- Fast
Unit tests automatically execute when running grunt server. It's best to run your tests in this fashion, as they automatically re-run on file changes.
If you wish to run them separately, use grunt test. To aid in debugging unit tests, you can use grunt test:debug. Details on this are explained below.
When developing a specific component, you likely don't want to run the entire test suite on every change. In order to test a single set of functionality, use the 'only' function when describing your test. For example:
describe.only('Login', function () {
// ... tests go here ...
});Be sure to remove the only once you're done.
Code coverage stats for our component tests are generated every time the test suite is executed. To view the stats, simply open the index.html file in any of the browser directories in the 'coverage' directory. Note: the coverage directory only exists when the server is running.
When writing or debugging unit tests and code, it can be helpful to enter the Chrome debugger to step through lines of code and inspect variables. The command grunt test:debug configures your environment to do this. To use it:
- Run
grunt test:debug. This will open a new Chrome window and runs all the tests - Open the Inspector in Chrome (
Command-Option-ion a Mac), and reload the Chrome page (Command-R/Ctrl-R/ etc.). This will re-run all the tests - Insert a
debuggerstatement somewhere in your code or your tests. The tests will automatically re-run when you save your file. Now when the tests re-run, they will pause in the Inspector at thedebuggerstatement (automatically switching to the "Sources" tab), allowing you to step through and inspect!
Especially when debugging tests, it can be useful to use the individual component testing technique above, so you don't have to wait for all 600+ tests to run every time you save a file or reload the Chrome page.
Goal: Validate our component in a full browser
- Uses an example instance of the component
- Uses Selenium, Protractor, Astrolabe
- Somewhat slow
- Story Based
The test code for this project isn't just to validate the correctness of the encore-ui component library. Like the encore-ui project itself, it is designed to be shared between various projects.
Because of this, the test code (called "rx-page-objects") is organized to be exported as a stand alone node package. The directory where this project is hosted can be found in ./utils/rx-page-objects. This directory contains its own package.json file, .gitignore file, and other project-specific files.
As a contributor, when you run npm install in the root of this project, it will automatically install the rx-page-object dependencies for you too. If you find that you're having trouble running end to end tests, try deleteing the utils/rx-page-objects/node_modules directory and reinstalling dependencies there.
In order to run the midway test suite, you will need a Selenium server running. To install and run Selenium, execute the webdriver-manager. To do this, open a new terminal window and run the following from the encore-ui directory:
cd utils/rx-page-objects/
./node_modules/.bin/webdriver-manager update # First time only
./node_modules/.bin/webdriver-manager start
This will launch the selenium server. It can sometimes take 25-30 seconds to fully launch, and you'll know it's ready when you see this at the bottom of the terminal:
INFO - Started SocketListener on 0.0.0.0:4444
INFO - Started org.openqa.jetty.jetty.Server@401363ff
Make sure you keep this terminal window running!
To run all tests, enter ./node_modules/.bin/protractor from the encore-ui/utils/rx-page-objects directory, in a different terminal. You need to ensure that you already have a development server running. If you haven't already, run grunt server in a separate terminal. You will need to keep this running in the background through the entirety of the midway tests.
In other words, to run the tests, you need a total of three terminal windows.
- One terminal running the Selenium server
- One terminal running
grunt server - One terminal running
./utils/rx-page-objects/node_modules/.bin/protractor
When developing a specific component, it's much quicker to run tests only for that component (rather than run through the entire suite every time). To do this, pass in path to the file as a 'specs' option in your protractor command. For example:
cd utils/rx-page-objects/
./node_modules/.bin/protractor --specs test/rxComponent.midway.js
In order to help developers more easily test their own apps/projects, each EncoreUI component should provide a page object file for itself. This file will provide convenience methods for the tester to use when writing midway tests for their app.
The filename for these page objects follows the componentName.page.js convention, and will automatically be generated if you create a new EncoreUI component with the Component Scaffolding
On a build of EncoreUI, all page object files are concatenated and tarballed into the dist directory. They are then published either manually or via Travis. To use these page objects, developers should include the following dependency in their app's package.json file:
"rx-page-objects": "^2.0.0"
Alternatively, they can install the file using this command:
npm install --save-dev rx-page-objects@2
Once installed, the page objects can be pulled in to any midway test via:
var someEncoreUIComponent = require('rx-page-objects').someEncoreUIComponent;
// ...
expect(someEncoreUIComponent.initialize($('.my-component')).isDisplayed()).to.eventually.be.true;Alternatively, you could place this helper library in the global scope of all tests. This is the recommended way. In your project's protractor config file, add this to your onPrepare section.
onPrepare: function () {
encore = require('rx-page-objects');
}Keep in mind you'll need to register a global variable declaration in your linting file (.jshintrc, etc.).
Convenience page objects exist to abstract away many of the tedious commands needed to direct a component during tests that prove high-level business functionality in your tests. However, many tests should still be run to ensure that your app's encore-ui components were "wired up" correctly. They should respond to activities that many users will exercise during normal operation. Instead of duplicating these tests in every instance of a component in your app, you can include that page object's exercise using this pattern:
// assuming you've included `encore = require('rx-page-objects')` in your protractor conf's onPrepare step
describe('Home Page', function () {
before(function () {
loginPage.login();
homePage.search('My Term');
});
it('should have a pagination component present', function () {
expect(homePage.resultsTable.pagination.rootElement.isPresent()).to.eventually.be.true;
});
// you can skip many tests by simply doing this instead
describe('search result pagination', encore.exercise.rxPaginate({
instance: homePage.resultsTable.pagination
}));
});The above snippet would run a couple dozen basic tests, assuming there are no edge cases in your app's implementation of the pagination component. Examples of such edge cases might be having less than three pages of total pagination space, or defaulting to a very large or small number of results by default.
If you do have edge cases, many of the exercises support passing in an options argument, which should be documented in the docs page for rxPaginate's exercise. This allows you to specify aspects of a particular component's implementation to the exercise function, allowing it to skip certain tests that are not valid.
Warning: you can easily over-write any pre-existing npm credentials stored in ~/.npmrc.
Make a backup of it first before continuing!
First, you'll need to get the npm user encore-ui's auth token from passwordstore. Once you have that, go ahead and make an .npmrc file in the root of the project so you're not authenticating as this user for all of your publishing use cases!
cd ~/code/js/encore-ui # or wherever you put the project
touch .npmrc
npm login
Username: encore-ui
Password: # paste the password you got from password safe here
Email: (this IS public) encore-ui@lists.rackspace.com
You'll want to visually confirm that the .npmrc file in the root of the project was the one that actually go updated. You'll know because it'll look something like this:
//registry.npmjs.org/:_authToken=12345678-1234-1234-1234-1234567890ab
If it's not there, check ~/.npmrc, and if needed, copy it over into the root of the project.
From there, head over to utils/rx-page-objects, and run the following commands:
npm version from-git # assuming you've released encore-ui already!
npm publish
Goal: Prevent visual regressions by making it easier to compare changes
encore-ui-screenshots is used in EncoreUI for UI regression testing. Currently, it will compare the visual elements of the master branch with your branch during Travis CI pull request runs.
Simply put, you shouldn't run these tests! They run invisibly on the Travis CI server only. This keeps contributors from checking out megabytes of image comparison files that aren't necessary for development.
But if you do want to run the tests, feel free to run them by themselves to populate a fresh copy of screenshots locally:
cd utils/rx-page-objects/
npm install snappit-mocha-protractor
./node_modules/.bin/protractor protractor.visual.regression.conf.js
This is useful only for experimenting with what your screenshots look like on your current operating system, at your current resolution, and your specific version of your browser! This is why it is recommended that you refrain from running these tests locally. They are designed to run on one machine and one machine only, to ensure that any visual regressions are triggered by visual changes, and not differences in hardware or software.
Note, that there are detailed install instructions for snappit-mocha-protractor ocated on its wiki page
In order to get a visual regression to run against your PR, you must push your pull request directly to the remote repository located at rackerlabs/encore-ui, and not a fork. codeship.io will not run its tests against pull requests originating from forks.
Visual regressions include posting an reference link from a pull request in the encore-ui-screenshots repository to the current pull request. It looks like this:
Click the reference link to the encore-ui-screenshots pull request, and check out the "Files" tab to see any visual changes introduced in the pull request.
Once a pull request has been visually signed off on, it should become the new baseline which all new pull requests are compared against. Do this by visiting the link to the encore-ui-screenshots pull request and merging it. From then on, Travis CI pull request builds will pull those screenshots and compare against them from now on.
The visual regression tests are located in the utils/visual-regression directory. Inside that, there is a directory for layouts (typically reserved for pages that demonstrate how a typical EncoreUI layout should look), and a directory for components.
When adding a new visual regression test, remember to keep the names of the tests short. The screenshot comparison library creates the image names based on the test names, so there's no need to be very specific.
Each it() block describes an individual visual regression test. screenshot.snap() is used to grab an area of the screen, and compare it to the current visual baseline, checking for changes. Any detected change will cause a visual diff to be created, with a link to it automatically added to the PR.
Here's an example of a basic visual regression test for a single component.
// utils/visual-regression/components/rxComponent.midway.js
var myPage = {
get myComponent() {
return encore.rxComponent.initialize($('.my-component'));
}
};
describe('demo component', function () {
before(function () {
demoPage.go('#/components/rxComponent');
});
it('default', function () {
// default screenshot of just the component -- no interaction yet
screenshot.snap(this, $('.tab-content rx-component'));
});
it('interactive', function () {
myPage.myComponent.interactWith(); // trigger some click that expands something
// this will capture the component in its current "interacted" state
screenshot.snap(this, myPage.myComponent.rootElement);
});
});