retry eyes test failures only when non-eyes errors occur #19659
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
👁️🗨️ Reading now.... |
islemaster
approved these changes
Dec 14, 2017
| # Note that applitools strips the trailing dot on this url for us, so we | ||
| # don't need to strip it here. | ||
| def linkify(str) | ||
| str.to_s.gsub(/(https?:\/\/[^ ]+)/, '<a href="\1">\1</a>') |
There was a problem hiding this comment.
I'd replace [^ ] with \S so any whitespace ends the link. Alternatively, there are a number of gems for this sort of thing.
| begin | ||
| @eyes.close(fail_on_mismatch) | ||
| rescue Applitools::TestFailedError => e | ||
| puts "<span style=\"color: red;\">#{EYES_ERROR_PREFIX} #{linkify(e)}</span>" |
There was a problem hiding this comment.
Which log does this puts go to?
There was a problem hiding this comment.
This html ends up in the cucumber log output, e.g. including:
- the html output (e.g. ChromeLatestWin7_angleHelper_eyes_output.html), and
- the stdout stream (which is consumed by runner.rb, and may also be what you'd end up with in
success.logif you didn't specifyrunner.rb --html).
| @@ -8,6 +8,10 @@ | |||
| # See http://support.applitools.com/customer/en/portal/articles/2099488-match-timeout | |||
| MATCH_TIMEOUT = 5 | |||
|
|
|||
| # Prefix to print before any eyes errors. We depend on this in runner.rb to detect | |||
| # whether there were any eyes errors. | |||
| EYES_ERROR_PREFIX = '[Applitools Eyes error]' | |||
There was a problem hiding this comment.
Could we load this constant from a shared location in both eyes_steps.rb and runner.rb?
There was a problem hiding this comment.
but then I'd have to pick a name for the file this gets extracted into :-) PTAL
islemaster
approved these changes
Dec 14, 2017
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background
the following call tree illustrates roughly how test flakiness is computed:
"magic retry" uses test flakiness data to decide how many times to rerun each UI test feature. the goal is to enable this for eyes tests during DTTs (but not on Circle), only counting selenium errors (not eyes diffs) as flaky failures.
Description
(a) make eyes mismatches no longer cause the cucumber run to fail, causing these runs to be reported as successful in terms of test flakiness (described briefly above).
(b) print eyes mismatch errors to the cucumber log
(a) parse the cucumber output to detect eyes mismatches
(b) fail the test run if eyes mismatch is detected
Screenshots
The following looks different when the only eyes failures occur:
in slack (note "1 eyes mismatches")
on the command line (note "1 eyes mismatches"):
in the cucumber logs (note red warning text and linkified eyes link):
Testing
Manually verified locally and on the test machine: