Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix acceptance tests false positives #778

Merged
merged 3 commits into from
May 12, 2021

Conversation

oshi97
Copy link
Contributor

@oshi97 oshi97 commented May 12, 2021

Previously, the acceptance test github check would always pass, even
if tests failed. There currently is some issue with how we load mapbox-gl
and firefox:headless, which would cause a "Failed to initialize WebGL."
error, which only occurs for firefox:headless, and not chrome:headless
or regular firefox.

As a short term fix we now run firefox in browserstack.

J=SLAP-1319
TEST=auto

see that the process will exit with 1 (by checking $? in terminal) if
I throw an error inside of acceptance/index.js's try block, and also
that the github acceptance test check will report a failure

see that the acceptance test check will report a failure if any tests fail, and
exit with status code 1

On github, I clicked into the headless acceptance test
status check on a different PR, just for curiosity's sake.
I noticed that all the tests were failing even though
the github status check was passing (green check mark).

J=SLAP-1313
TEST=auto
@oshi97 oshi97 force-pushed the dev/acceptance-tests-error-code branch from d93163e to 10b8125 Compare May 12, 2021 17:41
@oshi97 oshi97 requested review from tmeyer2115 and cea2aj May 12, 2021 17:55
.src('tests/acceptance/suites/*.js')
.browsers(browsers)
.startApp(`npx serve -p ${PORT} test-site/public`, 4000)
.run({ quarantineMode: true });
if (result > 0) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Is result the exit code of the runner? If so, I think we could exit with that code so that we don't lose that info. If not, this works!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

result is the number of failed tests here!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah I should rename this

@oshi97 oshi97 force-pushed the dev/acceptance-tests-error-code branch from 2379bbf to a4abb5e Compare May 12, 2021 18:03
@oshi97 oshi97 merged commit 1e7549c into feature/test-infra-fixes May 12, 2021
@oshi97 oshi97 deleted the dev/acceptance-tests-error-code branch May 12, 2021 18:07
@coveralls
Copy link

coveralls commented May 12, 2021

Coverage Status

Coverage remained the same at 4.724% when pulling a4abb5e on dev/acceptance-tests-error-code into 2eda3c4 on feature/test-infra-fixes.

@oshi97 oshi97 mentioned this pull request May 13, 2021
oshi97 added a commit that referenced this pull request May 13, 2021
use express instead of serve-handler for snapshots (#766)
remove + signs from percy snapshot query params (#765)
fix test-site's mock date (#764)
fix acceptance tests false positives (#778)
update acceptance test workflow name (#779)

J=SLAP-1313
@cea2aj cea2aj mentioned this pull request Jun 23, 2021
cea2aj added a commit that referenced this pull request Jun 23, 2021
## Version 1.22.0
### Changes
- Code coverage reports are now generated automatically for the Theme. (#759)
- A number of dependencies were updated in accordance with Dependabot's suggestions. (#780)
- The build time in `Development` mode was reduced significantly. (#782, #829) Build times in 
`Production` mode were also decreased. (#783, #825)
- Two new cards supporting videos were added: `product-prominentvideo` and `multilang-product-prominentvideo`. These cards
work with either YouTube or Vimeo content. (#794)
- The process for formatting hours was made more efficient, resulting in faster renders for cards with hours. (#784)
- Any SDK branch can now be specified in the `global_config.sdkVersion` attribute. (#791)
- Unused CSS was removed from the Theme. (#793, #798)
- To reduce flashing related to the SDK Component lifecycle, placeholder skeletons were added for the `SearchBar` and
`Navigation` components. (#797)
- Additional acceptance tests were added for the `VerticalFullPageMap` and Percy snapshots are now generated for all
cards. (#830, #820)
- Multi-language tests were added for the Theme. (#826)
- The `documentsearch-standard` card now supports rich text featured snippets. (#838)
- A new concept, `RuntimeConfig` was added. This allows configuration, computed at run-time, to be injected into the
Theme and SDK. This new concept enables JWT integrations as well. (#843)
### Bug Fixes
- A number of changes were made to make the Theme's visual regression tests more reliable. (#764, #765, #766, #778)
- Corrected how the CTA URLs for the `menuitem-standard` and `multilang-menuitem-standard` cards are computed. (#787)
- The Google Streetview person icon is now visible when using the `VerticalFullPageMap` layout. (#801)
- The size of the cluster and result pins are now configurable. (#807)
- A double encoding error for `verticalUrl` was fixed. (#836)
- The `Opens At` open status message is now properly translated in all languages. (#834)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants