Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip flaky story analytics integration tests #24638

Merged
merged 4 commits into from Sep 19, 2019

Conversation

Enriqe
Copy link
Contributor

@Enriqe Enriqe commented Sep 19, 2019

They are flaking on master

https://travis-ci.org/ampproject/amphtml/jobs/587116429

      IT => should send analytics event when navigating
        ✗ AssertionError: expected 'page-1' to equal 'page-2'
            at _callee2$ (home/travis/build/ampproject/amphtml/test/integration/test-amp-story-analytics.js:143:35)
            at tryCatch (home/travis/build/ampproject/amphtml/node_modules/regenerator-runtime/runtime.js:65:40)
            at Generator.invoke [as _invoke] (home/travis/build/ampproject/amphtml/node_modules/regenerator-runtime/runtime.js:303:22)
            at Generator.prototype.<computed> [as next] (home/travis/build/ampproject/amphtml/node_modules/regenerator-runtime/runtime.js:117:21)
            at asyncGeneratorStep (/tmp/637d6ae55d488af8c723bc9ffe1cea60.browserify.js:99961:103)
            at _next (/tmp/637d6ae55d488af8c723bc9ffe1cea60.browserify.js:99963:194)

https://travis-ci.org/ampproject/amphtml/jobs/587121381

DESCRIBE => amp-story analytics
  DESCRIBE => amp-story analytics
    DESCRIBE =>  
      IT => should send analytics event when entering bookend
        ✗ AssertionError: expected 'false' to equal 'true'
            at _callee3$ (home/travis/build/ampproject/amphtml/test/integration/test-amp-story-analytics.js:154:36)
            at tryCatch (home/travis/build/ampproject/amphtml/node_modules/regenerator-runtime/runtime.js:65:40)
            at Generator.invoke [as _invoke] (home/travis/build/ampproject/amphtml/node_modules/regenerator-runtime/runtime.js:303:22)
            at Generator.prototype.<computed> [as next] (home/travis/build/ampproject/amphtml/node_modules/regenerator-runtime/runtime.js:117:21)
            at asyncGeneratorStep (/tmp/d871b543b0ca58abfa25877e17f239cc.browserify.js:99961:103)
            at _next (/tmp/d871b543b0ca58abfa25877e17f239cc.browserify.js:99963:194)

/cc @choumx

@dreamofabear
Copy link

Looks like flakiness is just on Chrome 77.0.3865 (Windows 10.0.0) FYI.

@dreamofabear
Copy link

Do we have a "skip windows" test option?

@Enriqe
Copy link
Contributor Author

Enriqe commented Sep 19, 2019

That's true, unfortunately I don't think there is a "skip windows" test option

@dreamofabear
Copy link

We can add one. ;)

@estherkim
Copy link
Collaborator

Wait no, I removed skipWindows() as a test config because we run almost all Sauce Labs tests on Windows. Skipping should be browser specific at most.

Also I was looking at these tests just now, I'm not sure the OS is the reason for the flakiness.

@@ -148,6 +149,10 @@ class TestConfig {
return this.skip(this.runOnIos);
}

skipWindows() {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:(

@Enriqe
Copy link
Contributor Author

Enriqe commented Sep 19, 2019

Alright I reverted the last commit which introduces the skip for windows. Let's just skip them
altogether to unblock master and I'll take a closer look for the root cause.

@dreamofabear
Copy link

Good point, what we want is to skip Chrome on Windows specifically. Either way.

@Enriqe Enriqe merged commit 2d5e175 into ampproject:master Sep 19, 2019
@Enriqe Enriqe deleted the skip-integration branch September 19, 2019 20:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants