Automated end-to-end tests for WordPress.com
JavaScript Shell
Permalink
Failed to load latest commit information.
config Update Jetpack site and remove jetpackid (#371) Jan 20, 2017
image-uploads Added sample mp3 file for uploading Jan 11, 2017
lib Update Jetpack site and remove jetpackid (#371) Jan 20, 2017
specs-i18n Screenshot-generating test for get.blog i18n Nov 30, 2016
specs-ios Cleaned up reset app comment, removed page spec (to be rebuilt later)… Jan 5, 2017
specs-jetpack-calypso Update Jetpack site and remove jetpackid (#371) Jan 20, 2017
specs-jetpack Jetpack: Adds a check that related posts are shown in the customizer (#… Dec 21, 2016
specs-visdiff Added error handling and debug support for Applitools / Sauce Labs (#293 Oct 13, 2016
specs Move existing jetpack calypso spec to own directory (#370) Jan 20, 2017
test Initial commit to master May 4, 2016
.babelrc Initial commit to master May 4, 2016
.editorconfig Initial commit to master May 4, 2016
.eslintrc Initial commit to master May 4, 2016
.gitignore Added .envrc and cleaned up xunit.xml in .gitignore [skip ci] (#104) Jun 24, 2016
.jsfmtrc Initial commit to master May 4, 2016
.nvmrc Update node to 6.9.1 to match Calypso (#314) Oct 27, 2016
Gruntfile-ios.js Created parallel run Gruntfile for iOS, but need to work through Sauc… Sep 1, 2016
Gruntfile.js Registered missing grunt task for notChrome (#271) Sep 29, 2016
LICENSE.md Initial commit to master May 4, 2016
README.md Update Jetpack site and remove jetpackid (#371) Jan 20, 2017
circle.yml Added special handling for Jetpack branches Jan 20, 2017
fix-saucelabs-etc-hosts.bat Override Sauce Labs /etc/hosts file to prevent loading external images ( Sep 30, 2016
fix-saucelabs-etc-hosts.sh Override Sauce Labs /etc/hosts file to prevent loading external images ( Sep 30, 2016
package.json With WebDriverAgent w/a the .com login is working Dec 21, 2016
randomize.sh randomize.sh spits out the new file order [skip ci] (#105) Jun 24, 2016
report.sh Updated dependencies on report.sh [skip ci] (#219) Aug 29, 2016
run-mobile.sh Misc changes for TestObject Nov 28, 2016
run-wrapper.sh Added special handling for Jetpack branches Jan 20, 2017
run.sh Added special handling for Jetpack branches Jan 20, 2017
trigger-circle-ci-build.sh Initial commit to master May 4, 2016
trigger-circle-ci-live-branch.sh Try higher timeouts for live branches (#100) Jun 24, 2016
trigger-circle-ci-specific.sh Initial commit to master May 4, 2016
trigger-circle-ci-visdiff.sh Separate cross-browser visdiffs (#256) Sep 27, 2016

README.md

WordPress.com End to End Tests

Automated end-to-end acceptance tests for the wp-calypso client and WordPress.com in general.

Circle CI

Pre-requisites

Install NodeJS and ChromeDriver

brew install node chromedriver

Install dependencies

npm install
npm pack lib/reporter
npm install ./spec-xunit-slack-reporter-0.0.1.tgz

Note - One of the dependencies for mobile app testing is Appium, which in turn requires Java. If you don't plan on running any mobile tests and don't have Java installed you can safely ignore the warnings given by npm install

To run the specs (in default browser size)

./run.sh -g

To run an individual spec

./node_modules/mocha/bin/mocha specs/wp-log-in-out-spec.js lib/after.js

Note: you can also change the spec temporarily the use the .only syntax so it is the only spec that runs (making sure this isn't committed)

eg.

test.describe.only( 'Logging In and Out:', function() {

To run a specific suite of specs

The run.sh script takes the following parameters, which can be combined to execute a variety of suites

-R                    - Use custom Slack/Spec/XUnit reporter, otherwise just use Spec reporter
-p [jobs]         - Execute [num] jobs in parallel (experimental)
-s                    - Screensizes in a comma-separated list (defaults to mobile,desktop,tablet)
-g                    - Execute general tests in the specs/ directory
-i                    - Execute i18n tests in the specs-i18n/ directory
-v [all/critical] - Execute the visdiff tests in specs-visdiff[/critical].  Must specify either 'all' or 'critical'.
-h                    - This help listing

To run

All tests should be written to work in three modes: desktop (1440 wide), tablet (1024 wide) and mobile (375 wide).

You can run tests in different modes by setting an environment variable BROWSERSIZE to either desktop, tablet or mobile.

Eg:

env BROWSERSIZE=tablet ./node_modules/mocha/bin/mocha --compilers js:babel-register specs lib/after.js

Or you can use the -s option on the run.sh script:

./run.sh -g -s mobile ./run.sh -g -s desktop,tablet

Config / Environment Variables

The tests use the node config library to specify config values for the tests.

Under the config directory, there are files for each environment: default.json is the base for all environments, then development.json for local, and test.json for CI.

You can also use local config files that are not committed.

The config files should be added under the config/ tree and should follow the naming scheme: local-<env>.json

The properties in the local configuration override the default.json properties. This is useful for local testing of different configurations on local, e.g. testing on local Calypso instance, instead of production, by setting the calypsoBaseURL property to http://calypso.localhost:3000.

If the configuration doesn't exist, the code falls back to using the environmental variables.

Note: NODE_ENV is still required, as it is used to determine what <env> config to load.

For example: export NODE_ENV='personal'

An example configuration file is provided.

The local configurations are excluded from the repository, in order to prevent accidental commit of sensitive data.

Please don't commit usernames and passwords in these files!

Note: As the NODE_CONFIG variable is growing unwieldy for use in CircleCI, some environment variables are also being split out into separate entries. These are listed below the Config Values section.

Config Values

A full list of config values are:

Name Description Example Required Store in file?
browser The browser to use: either firefox or chrome chrome Yes Yes
proxy The type of proxy to use: either system to use whatever your system is configured to use, or direct to use no proxy. direct Yes Yes
saveAllScreenshots Whether screenshots should be saved for all steps, including those that pass false Yes Yes
neverSaveScreenshots Overrides the screenshot function so nothing is captured. This is intended for use with the Applitools visual diff specs, since the screenshots are handled via their utilities instead. false Yes Yes
checkForConsoleErrors Automatically report on console errors in the browser true Yes Yes
reportWarningsToSlack Specifies whether warnings should be reported to Slack - should be used for CI builds false Yes Yes
authURL The page where you authenticate to use calypso - this should not include the redirect https://wordpress.com/wp-login.php Yes Yes
calypsoBaseURL The home page for calypso https://wpcalypso.wordpress.com Yes Yes
explicitWaitMS The explicit wait time in milliseconds to wait for an element to appear - for example a widget loading data via an API 10000 Yes Yes
mochaTimeoutMS This is the maximum total time in milliseconds a single mocha end to end test can take to finish - otherwise it will time out. 120000 Yes Yes
startBrowserTimeoutMS This is the maximum total time in milliseconds that the browser can take to start - this is different from test time as we want it to fail fast 10000 Yes Yes
afterHookTimeoutMS This is the maximum total time in milliseconds that an after test hook can take including capturing the screenshots 20000 Yes Yes
knownABTestKeys An array of expected, known AB testing keys used in localstorage. If a key is found on any page that isn't in here, the test will fail [ "freeTrials_20160112", "plansPageBusinessAATest_20160108" ] Yes Yes
testUserName This is an existing test WordPress.com account for testing purposes - this account should have a single site testuser123 Yes NO
testPassword This is the password for the test WordPress.com account testpassword$$$%### Yes NO
testUserNameMultiSite This is an existing test WordPress.com account for testing purposes that has multiple sites testuser123 Yes NO
testPasswordMultiSite This is the password for the test WordPress.com account that has multiple sites testpassword$$$%### Yes NO
testUserNameJetpack This is an existing test WordPress.com account for testing purposes that has a single jetpack site testuser123 Yes NO
testPasswordJetpack This is the password for the test WordPress.com account that has a single jetpack site testpassword$$$%### Yes NO
testSiteForInvites This is wordpress.com site that is used for testing invitations e2eflowtesting.wordpress.com Yes NO
privateSiteForInvites This is a wordpress.com private site that is used for testing invitations e2eflowtestingprivate.wordpress.com Yes NO
mailosaurAPIKey This is the API key from mailosaur used for testing email hsjdhsjdh7878sd Yes NO
inviteInboxId This is an inbox id from mailosaur used for invite email testing sad34id44ss Yes NO
signupInboxId This is an inbox id from mailosaur used for signup email testing sad34id44ss Yes NO
domainsInboxId This is an inbox id from mailosaur used for domains email testing sad34id44ss Yes NO
publicizeTwitterAccount This is the name of the test twitter account connected to your test username @endtoendtesting Yes NO
passwordForNewTestSignUps This is the password that will be set for new sign ups created for testing purposes alongcomplexpassword%### Yes NO
storeSandboxCookieValue This is a secret cookie value used for testing payments No NO
slackHook This is a Slack incoming webhook where notifications are sent for new accounts that are created (https://my.slack.com/services/new/incoming-webhook -- requires Slack login) https://hooks.slack.com/services/XXXXXXXX/XXXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXX No NO
emailPrefix A string to stick on the beginning of the e-mail addresses used for invites and signups username No NO
testAccounts A JSON object with username/password pairs assigned to keynames for easy retrieval. The necessary keys are shown in the example to the right. {"defaultUser": ["username1","password1"], "multiSiteUser": ["username2","password2"], "jetpackUser": ["username3","password3"], "visualUser": ["username4", "password4"]}, "privateSiteUser": ["username5", "password5"] No NO
highlightElements Boolean to indicate whether to visually highlight elements being interacted with on the page true No Yes

Standalone Environment Variables

Name Description Example Required Store in file?
slackToken This is a token for uploading files/screenshots to Slack (https://api.slack.com/tokens -- requires Slack login) xxxx-##########-##########-###########-XXXXXXXXXX No NO
EYESDEBUG If this is set, no connection is opened to Applitools, only local screenshots are taken 1 No NO
SAUCEDEBUG If this is set, on test failure a breakpoint will be set in SauceLabs, enabling you to continue interacting with the browser for troubleshooting 1 No NO

CircleCI Environment Variables

These environment variables are intended for use inside CircleCI, to control which tests are being run

Name Description Default Required
DISABLE_EMAIL Setting this to true will cause the Invite and Signup tests to be skipped false No
RUN_VISDIFF Setting this to false will cause the visual diff tests to be skipped true No
SKIP_TEST_REGEX The value of this variable will be used in the -i -g ***** parameter, to skip any tests that match the given RegEx. List multiple keywords separated by a | (i.e. Invite|Domain|Theme) Empty String No