New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UI Test Status Page #9118

Merged
merged 16 commits into from Jun 22, 2016

Conversation

Projects
None yet
5 participants
@islemaster
Member

islemaster commented Jun 22, 2016

Introduce a --with-status-page option to runner.rb which generates a live-updating summary page for the test run for monitoring test progress.

screenshot from 2016-06-22 13-01-45

This page starts with a summary showing the branch name, commit under test and test run timestamp, and an overall test progress bar.

The rest of the page is a listing of all the tests by browser, displaying current status. Each browser gets its own progress bar, too.

screenshot from 2016-06-22 13-02-02

The whole thing is backed by the test logs that we upload to S3, so this only works for test runs where we use the --html flag. When a test completes (success or failure now) we attach some metadata and push it to S3, where it will replace any previous test log for that branch-browser-feature combination (though the bucket is versioned, so the old logs are still available).

The test status page gets generated as a static file by runner.rb before any tests are run. It is generated with test run metadata, including the branch name, commit, and the list of tests that are part of this test run. Once opened, the page makes async requests to a new API {origin}/test_status/{branch}/since/{timestamp} to find out whether any tests have new output, and if so it hits {origin}/test_status/{branch}/{test_name} to retrieve results for a particular test.

I've hooked this up to CI so that on a DTT both the UI tests and the Eyes tests should put a link in Slack to their respective test status pages.

islemaster added some commits Jun 10, 2016

Upload success logs to S3 too
We want to do this upload for every log, not just the failures, to allow us to use them for status later.
Add metadata to uploaded logs
Include:
* commit hash
* duration in seconds
* attempt number (0 is initial attempt, 1+ are reruns)
* success boolean
More fixups
Remove unused test status route

Put log link in separate column

Report rerun # on summary page
@islemaster

This comment has been minimized.

Show comment
Hide comment
@islemaster

islemaster Jun 22, 2016

Member

PS: I know some of this is pretty hacky. I just wanted to see if we could get something up and running quick, since it's an internal tool.

Member

islemaster commented Jun 22, 2016

PS: I know some of this is pretty hacky. I just wanted to see if we could get something up and running quick, since it's an internal tool.

@@ -146,6 +148,9 @@ def upload_log_and_get_public_link(filename)
f = `egrep -r "Given I am on .*#{scriptname.delete(' ').downcase}" . | cut -f1 -d ':' | sort | uniq | tr '\n' ,`
$options.feature = f.split ','
end
opts.on('--with-status-page', 'Generate a test status summary page for this test run') do

This comment has been minimized.

@mehalshah

mehalshah Jun 22, 2016

Contributor

If this needs to be run with --html then you should force it here

@mehalshah

mehalshah Jun 22, 2016

Contributor

If this needs to be run with --html then you should force it here

@breville

This comment has been minimized.

Show comment
Hide comment
@breville

breville Jun 22, 2016

Member

wow, this looks great.

Member

breville commented Jun 22, 2016

wow, this looks great.

@bcjordan

This comment has been minimized.

Show comment
Hide comment
@bcjordan

bcjordan Jun 22, 2016

Collaborator

LGTM. Excited to see this in action on next test run!

Collaborator

bcjordan commented Jun 22, 2016

LGTM. Excited to see this in action on next test run!

@islemaster islemaster merged commit f343f5c into staging Jun 22, 2016

3 checks passed

ci/circleci Your tests passed on CircleCI!
Details
coverage/coveralls First build on all-ui-test-logs at 88.059%
Details
hound No violations found. Woof!

@islemaster islemaster deleted the all-ui-test-logs branch Jun 22, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment