Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Spike] Automated visual regression testing #458

Closed
wants to merge 10 commits into from
Closed

Conversation

@fofr
Copy link
Contributor

@fofr fofr commented Aug 18, 2017

https://trello.com/c/TYuS8AGP/82-spike-investigate-a-suitable-automated-visual-regression-testing-tool-2-days

Spike into automated visual regression testing

Aims

  • Run automatic visual regression tests between deployed review app and production
  • Run regression tests against components and fixtures as well as full pages
  • Start with wraith as we have existing config
  • Investigate other options

Wraith findings

It's possible to deploy to a Heroku review app and to use a postdeploy script to run a rake task that runs Wraith. By using the HEROKU_APP_NAME environment variable we can point wraith at the current review app and compare with production. Wraith runs successfully (once the PhantomJS buildpack has been included) and generates the expected gallery, however that gallery cannot be viewed.

I tried putting the generated gallery into the /public directory so the Rails app would be able to render it and so we'd have a consistent URL to check the output of. However – by running the script in the postdeploy context, as soon as the script finishes all temporary files are cleaned up – so the thumbnails and gallery get deleted, so we can't see them. The same happens if you SSH into the box and run the rake task, on exit the files are removed. This is probably because of Heroku’s ephemeral filesystem.

Running wraith in a different context, ie on the core dyno, isn't possible. Using something like the rake task buildpack or by running a task on Rails initialise, these tasks always run before the app is available at the specified Heroku URL, or before phantomJS has been installed by Heroku.

This will be an issue for whatever visual regression tool we use if we run it on Heroku – either as part of the deployed review app or a separate app running elsewhere on Heroku that somehow gets notified of the review app’s deployment (this is probably possible using Github deployment webhooks)

screen shot 2017-08-22 at 10 34 42

Possible solution

  • Send generated files from Heroku to S3
  • Report the gallery URL
  • Do this in a separate app or in a way that doesn't add weight to any individual app that needs regression testing.

Other tools

Both are drop in replacements for Wraith but built for the Javascript/npm environment. They don't solve the issue of running on Heroku and/or pushing screenshots to a place they can be viewed.

Spectre with client is a tool for managing and viewing screenshots. You can set up runs and post screenshots to each run. It's supposed to also run on Heroku but there are reports of images being deleted when doing so. I also can't see how its secure – ie anyone can post any screenshot, from what I can tell. There are no tokens.

Percy

percy.io was a suggested alternative. Their pricing will be an issue. For each component we will be generating 3 images per fixture, per environment, per PR/PR sync. That's ~200 screenshots per run. Running the tests against a set of live content pages will be a further ~200 images per run. We will quickly get through a quota of snapshots.

Another approach

Something like a wraith-server. An app that runs in a dedicated place that can accept commands through an API. Using a webhook the server could be pinged with URLs and/or config for whatever app needs regression tests. The app would run the tests, created the images, manage the gallery and report back a URL for the hosted gallery.

fofr added 5 commits Aug 18, 2017
This config was written for migration and is no longer used
Change local name to one based on the HEROKU_APP_NAME environment
variable. This lets us compare the review app with live.

Move the shots directory into `/public` so they can be viewed from the
app.
@boffbowsh boffbowsh temporarily deployed to government-frontend-pr-458 Aug 18, 2017 Inactive
@fofr fofr force-pushed the wraith-review-app branch from dd15239 to b4f79f5 Aug 18, 2017
@fofr fofr had a problem deploying to government-frontend-pr-458 Aug 18, 2017 Failure
@fofr fofr force-pushed the wraith-review-app branch from b4f79f5 to 93bf6e3 Aug 18, 2017
@fofr fofr had a problem deploying to government-frontend-pr-458 Aug 18, 2017 Failure
@boffbowsh boffbowsh temporarily deployed to government-frontend-pr-458 Aug 18, 2017 Inactive
The post deploy script can only run once, when the app is created, not
on subsequent deploys (eg updated commits, force pushes, etc.)

Use the ruby rake deploy tasks build pack to run the task set in the
`DEPLOY_TASKS` environment variable.
@fofr fofr force-pushed the wraith-review-app branch from 93bf6e3 to 942a2ec Aug 18, 2017
@fofr fofr temporarily deployed to government-frontend-pr-458 Aug 18, 2017 Inactive
@fofr fofr temporarily deployed to government-frontend-pr-458 Aug 18, 2017 Inactive
@boffbowsh boffbowsh temporarily deployed to government-frontend-pr-458 Aug 18, 2017 Inactive
@fofr fofr force-pushed the wraith-review-app branch from 1b5bcba to 659ad0e Aug 18, 2017
@fofr fofr temporarily deployed to government-frontend-pr-458 Aug 18, 2017 Inactive
@fofr fofr force-pushed the wraith-review-app branch from 659ad0e to 79a9b52 Aug 18, 2017
@boffbowsh boffbowsh temporarily deployed to government-frontend-pr-458 Aug 18, 2017 Inactive
@fofr fofr force-pushed the wraith-review-app branch from 79a9b52 to b7d6188 Aug 18, 2017
@boffbowsh boffbowsh temporarily deployed to government-frontend-pr-458 Aug 18, 2017 Inactive
@fofr fofr force-pushed the wraith-review-app branch from b7d6188 to b146161 Aug 18, 2017
@boffbowsh boffbowsh temporarily deployed to government-frontend-pr-458 Aug 18, 2017 Inactive
@boffbowsh boffbowsh temporarily deployed to government-frontend-pr-458 Aug 21, 2017 Inactive
@boffbowsh boffbowsh temporarily deployed to government-frontend-pr-458 Aug 21, 2017 Inactive
@boffbowsh boffbowsh temporarily deployed to government-frontend-pr-458 Aug 21, 2017 Inactive
@fofr fofr force-pushed the wraith-review-app branch from 1ec7fd1 to 30603bd Aug 21, 2017
@fofr fofr force-pushed the wraith-review-app branch from 30603bd to 94828e6 Aug 21, 2017
@boffbowsh boffbowsh temporarily deployed to government-frontend-pr-458 Aug 21, 2017 Inactive
@fofr fofr force-pushed the wraith-review-app branch from 94828e6 to 3309fe3 Aug 21, 2017
@boffbowsh boffbowsh temporarily deployed to government-frontend-pr-458 Aug 21, 2017 Inactive
@fofr fofr force-pushed the wraith-review-app branch from 3309fe3 to 4e9ef38 Aug 21, 2017
@fofr fofr temporarily deployed to government-frontend-pr-458 Aug 21, 2017 Inactive
@boffbowsh boffbowsh temporarily deployed to government-frontend-pr-458 Aug 21, 2017 Inactive
@fofr fofr force-pushed the wraith-review-app branch from 4e9ef38 to 1fe80f3 Aug 21, 2017
@fofr fofr temporarily deployed to government-frontend-pr-458 Aug 21, 2017 Inactive
@boffbowsh boffbowsh temporarily deployed to government-frontend-pr-458 Aug 21, 2017 Inactive
@fofr fofr force-pushed the wraith-review-app branch from 1fe80f3 to 3a4eda4 Aug 21, 2017
@fofr fofr temporarily deployed to government-frontend-pr-458 Aug 21, 2017 Inactive
@boffbowsh boffbowsh temporarily deployed to government-frontend-pr-458 Aug 21, 2017 Inactive
@fofr fofr force-pushed the wraith-review-app branch from 3a4eda4 to 1a69c4e Aug 21, 2017
@fofr fofr temporarily deployed to government-frontend-pr-458 Aug 21, 2017 Inactive
@boffbowsh boffbowsh temporarily deployed to government-frontend-pr-458 Aug 21, 2017 Inactive
@fofr fofr had a problem deploying to government-frontend-pr-458 Aug 21, 2017 Failure
@fofr fofr force-pushed the wraith-review-app branch from 9ea31f4 to 0893b56 Aug 21, 2017
@fofr fofr had a problem deploying to government-frontend-pr-458 Aug 21, 2017 Failure
@fofr fofr force-pushed the wraith-review-app branch from 0893b56 to 1a69c4e Aug 21, 2017
@boffbowsh boffbowsh temporarily deployed to government-frontend-pr-458 Aug 21, 2017 Inactive
@fofr fofr changed the title [WIP] Run visual regression tests on Heroku review app [Spike] Automated visual regression testing Aug 22, 2017
@fofr fofr changed the title [Spike] Automated visual regression testing [DO NOT MERGE] [Spike] Automated visual regression testing Aug 22, 2017
@fofr fofr closed this Aug 22, 2017
@fofr fofr changed the title [DO NOT MERGE] [Spike] Automated visual regression testing [Spike] Automated visual regression testing Aug 22, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.