Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarify warning when making changes to UI tests #38645

Merged
merged 4 commits into from Jan 21, 2021

Conversation

bencodeorg
Copy link
Contributor

@bencodeorg bencodeorg commented Jan 20, 2021

Any change to a UI test file currently gives this warning:

image

This is a useful warning for folks well versed in our UI testing to prevent blocking a test build with changes that could have been tested in Drone, but not as great for people with less experience who are making small changes and might view this warning as a requirement for merging a PR with changes to a UI test file.

New warning (updated to 2021 from 2020 since screenshot was taken 馃う ) -- message is kind of verbose, if others have ideas on how to make it more concise/clear.:

image

Currently, Drone will not pass with [test all browsers] specified -- there's more discussion here about differences between our test environment and Drone that might cause this problem, but looking at a couple recent failures using the [test all browsers] tag (here's one), I wonder if things might just time out on Drone when testing all browsers. Here's some of the log:

19 Jan 23:37:51 - Cleaning up the tunnel connection to maki1162.miso.saucelabs.com
--
11781 | 19 Jan 23:37:52 - Cleaning up.
11782 | 19 Jan 23:37:53 - Removing tunnel 31a3f1e9f99c4decba6f4a48d7c674bf.
11783 | 19 Jan 23:37:54 - Waiting for any active jobs using this tunnel to finish.
11784 | 19 Jan 23:37:56 - Press CTRL-C again to shut down immediately.
11785 | 19 Jan 23:37:58 - Note: if you do this, tests that are still running will fail.
11786 | 19 Jan 23:38:08 - Error when removing tunnel: couldn't connect to https://saucelabs.com/rest/v1/******/tunnels/31a3f1e9f99c4decba6f4a48d7c674bf?wait_for_jobs=1: Delete https://saucelabs.com/rest/v1/******/tunnels/31a3f1e9f99c4decba6f4a48d7c674bf?wait_for_jobs=1: dial tcp: lookup saucelabs.com on 127.0.0.11:53: read udp 127.0.0.1:32814->127.0.0.11:53: i/o timeout. Treating error as fatal, client will now exit.
11787 | 19 Jan 23:38:08 - Waiting for the connection to terminate...
11788 | 19 Jan 23:38:28 - Connection closed (0).
11789 | 19 Jan 23:38:28 - Goodbye.
11790 | [test] iPhone_hour_of_code_hoc_reset first selenium error: Failed to open TCP connection to localhost:4445 (Cannot assign requested address - connect(2) for "localhost" port 4445) (Errno::EADDRNOTAVAIL)
11791 | [test] [hour_of_code/hoc_reset] Failing Scenarios:
11792 | [hour_of_code/hoc_reset] cucumber features/hour_of_code/hoc_reset.feature:3 # Scenario: hoc/reset resets videos, callouts, level progress
11793 | [hour_of_code/hoc_reset]
11794 | [hour_of_code/hoc_reset] 1 scenario (1 failed)
11795 | [hour_of_code/hoc_reset] 13 steps (13 skipped)
11796 | [hour_of_code/hoc_reset] 2m11.912s

@bencodeorg
Copy link
Contributor Author

@sureshc FYI -- Winter had taking the lead on understanding/resolving differences between our test environment and Drone, in case any of the information I've included in the PR description is relevant (eg, timeouts).

Copy link
Contributor

@Erin007 Erin007 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for causing the initial confusion, @bencodeorg and @hannahbergam! Thanks for the follow through to make it clearer to folks in the future.

@bencodeorg bencodeorg merged commit 567dadb into staging Jan 21, 2021
@bencodeorg bencodeorg deleted the ben/test-all-browsers-warning branch January 21, 2021 20:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants