-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix acceptance tests false positives #778
fix acceptance tests false positives #778
Conversation
a729e57
to
2eda3c4
Compare
On github, I clicked into the headless acceptance test status check on a different PR, just for curiosity's sake. I noticed that all the tests were failing even though the github status check was passing (green check mark). J=SLAP-1313 TEST=auto
d93163e
to
10b8125
Compare
tests/acceptance/index.js
Outdated
.src('tests/acceptance/suites/*.js') | ||
.browsers(browsers) | ||
.startApp(`npx serve -p ${PORT} test-site/public`, 4000) | ||
.run({ quarantineMode: true }); | ||
if (result > 0) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Is result the exit code of the runner? If so, I think we could exit with that code so that we don't lose that info. If not, this works!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
result is the number of failed tests here!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ah I should rename this
2379bbf
to
a4abb5e
Compare
## Version 1.22.0 ### Changes - Code coverage reports are now generated automatically for the Theme. (#759) - A number of dependencies were updated in accordance with Dependabot's suggestions. (#780) - The build time in `Development` mode was reduced significantly. (#782, #829) Build times in `Production` mode were also decreased. (#783, #825) - Two new cards supporting videos were added: `product-prominentvideo` and `multilang-product-prominentvideo`. These cards work with either YouTube or Vimeo content. (#794) - The process for formatting hours was made more efficient, resulting in faster renders for cards with hours. (#784) - Any SDK branch can now be specified in the `global_config.sdkVersion` attribute. (#791) - Unused CSS was removed from the Theme. (#793, #798) - To reduce flashing related to the SDK Component lifecycle, placeholder skeletons were added for the `SearchBar` and `Navigation` components. (#797) - Additional acceptance tests were added for the `VerticalFullPageMap` and Percy snapshots are now generated for all cards. (#830, #820) - Multi-language tests were added for the Theme. (#826) - The `documentsearch-standard` card now supports rich text featured snippets. (#838) - A new concept, `RuntimeConfig` was added. This allows configuration, computed at run-time, to be injected into the Theme and SDK. This new concept enables JWT integrations as well. (#843) ### Bug Fixes - A number of changes were made to make the Theme's visual regression tests more reliable. (#764, #765, #766, #778) - Corrected how the CTA URLs for the `menuitem-standard` and `multilang-menuitem-standard` cards are computed. (#787) - The Google Streetview person icon is now visible when using the `VerticalFullPageMap` layout. (#801) - The size of the cluster and result pins are now configurable. (#807) - A double encoding error for `verticalUrl` was fixed. (#836) - The `Opens At` open status message is now properly translated in all languages. (#834)
Previously, the acceptance test github check would always pass, even
if tests failed. There currently is some issue with how we load mapbox-gl
and firefox:headless, which would cause a "Failed to initialize WebGL."
error, which only occurs for firefox:headless, and not chrome:headless
or regular firefox.
As a short term fix we now run firefox in browserstack.
J=SLAP-1319
TEST=auto
see that the process will exit with 1 (by checking
$?
in terminal) ifI throw an error inside of acceptance/index.js's
try
block, and alsothat the github acceptance test check will report a failure
see that the acceptance test check will report a failure if any tests fail, and
exit with status code 1