Automated Testing (Ghost Inspector)

The Tool

Before any deploy, we want to make sure that all the code is working as expected, especially for experiences core to the Do Something experience (creating an account, signing up for a campaign, reporting back, etc.). To do this, we use Ghost 👻 Inspector, a tool that runs customizable, automated tests and test suites on our site, mimicking the actions of a user and checking for expected behavior.

You can create an account at https://ghostinspector.com/ and ask an existing administrator to invite you to the DoSomething.org organization within the tool.

The Old Days

Before using this tool, this set of tests was performed manually by a member of the product team, using this guide (current as of January 2016). If you take a peek 👀, you can see the kinds of things we would look for while testing -- everything from functionality to aesthetic to transactional emails.

The Thor Suite

This suite tests all the core experiences on the Do Something site:

• Static Content (11 Facts page, Our Team page)
• Logging in and going to your profile as an existing user
• Homepage (Campaign Finder, Tiles)
• Signing up for a closed campaign
• Signing up for a campaign via an alpha permalink
• Signing up for an SMS campaign
• Signing up for a "regular" campaign
• Kudos Functionality ❤️
• Header Navigation
• Forgot Password Flow
• Internal Search
• Action Page Social Shares
• Account Creation
• Submitting a Help Ticket
• Reporting Back

The suite runs every time Thor is deployed as part of the Jenkins job, and results are reported in both in the 👻 Inspector UI and into the #ghost-inspector channel in Slack. In addition to the test suite passing, 4 transactional emails must be sent to the Ghost Inspector channel, a test ZenDesk ticket has to appear in the channel, and a text must be received. The full checklist can be viewed here.

As new features get added to the site, new tests should be written to test that functionality.

The Production Suite

There are a set of tests that test behavior on production, found here. There is a test for the donate flow which runs daily, and hourly tests run against the homepage loading and internal search working.

Quirks + Not Quirky Facts

• Because we wanted to be able to see the result of the test suite in the #deploys Slack channel (vs. the results of each individual test), we set up a meta-suite that would both trigger the entire suite of tests and then report into the #deploys channel that the suite had kicked off, and if the tests have passed or failed. The method for doing setting that up (and fixing it) is detailed in two email exchanges now pinned to the #ghost-inspector channel in Slack. Even if one test in the suite fails, the notification that reports into Slack will appear as having failed.

• To make the transactional emails report into the #ghost-inspector channel in Slack, we set up the Email Integration for Slack. Emails sent to t0l5c3c2e0a0x9e9@dosomething.slack.com will appear in the #ghost-inspector channel.

• In order to run certain tests, you need to create a new user each time. To do so, a dummy email address was created testrecipient@dosomething.org. To generate a new user each time, we added a timestamp token to the email. The email that is input into the test is: testrecipient+{{TIMESTAMP}}@dosomething.org.

• To test the account of an existing user, we use testrecipient@dosomething.org. The password for this email can be found in step 10.

• When Thor gets refreshed with production content, several tests on Thor might fail, and transactional emails might stop sending on Thor. Ask the messaging team to update the Message Broker settings on Thor so they continue sending. You may also need to turn off Optimizely on Thor after a content refresh, because any experiments running will not be expected behavior for the automated tests. Any changes made in the Drupal configuration on production will also appear after this content refresh. Lastly, the campaign that the SMS campaign sign up test runs on will eventually close, and will need to be re-set up on a different SMS campaign.

• 👻 Inspector has the ability to compare how the screenshot of the test compares to how the screen looks when the test is being run now. Most tests in the Thor suite do not compare screenshots, simply because they increase the likelihood that something will error without adding value.

• ZenDesk (our member support software), is set up to tag and route tickets to campaign leads based on the campaign the ticket came in through. Because we test the submission of ZenDesk tickets through the site and to ZenDesk, we didn't want to inundate one campaign lead with ZenDesk tickets, or have it appear as though one campaign is getting an inordinate number of questions. Therefore, we created a tag titled test that gets added to all ZenDesk tickets that come in with the phrase causeyhippo.