Stabilize dashboard selenium tests #11004

stacey-gammon · 2017-04-03T14:25:14Z

Attempt to fix #10980. Suspect a toast disappearing around the time
this gets clicked may be messing it up.

stacey-gammon · 2017-04-03T15:20:43Z

Passed once.

Jenkins, test this.

stacey-gammon · 2017-04-03T16:09:17Z

Passed twice

Jenkins, test this.

stacey-gammon · 2017-04-03T20:00:34Z

Third times not a charm. I'm wondering if this time another click failed because of the disappearing toast message. Will try more try wraps.

16:51:26.283: clickDashboardByLinkText: Dashboard Test Time
16:51:26.358: in findTestSubject: [data-test-subj~="globalLoadingIndicator"]
16:51:26.562: in findTestSubject: [data-test-subj~="globalTimepickerRange"]
>> PASS: chrome on any platform - kibana - dashboard app - dashboard time - dashboard with stored timed - sets quick time on open (4881ms)
16:51:26.627: Clicking edit
16:51:26.627: in findTestSubject: [data-test-subj~="dashboardEditMode"]
16:51:26.699: Taking screenshot "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-selenium/test/screenshots/failure/failure_1491238286699_dashboard with stored timed_is saved with absolute time.png"
>> FAIL: chrome on any platform - kibana - dashboard app - dashboard time - dashboard with stored timed - is saved with absolute time (190ms)
UnknownError: [POST http://localhost:4444/wd/hub/session/66e4710624e95522f956d241d7eff0b9/element/0.03164142997168784-39/click] unknown error: Element is not clickable at point (1025, 96). Other element would receive the click: <input parse-query="" input-focus="" kbn-typeahead-input="" ng-model="model.query" placeholder="Filter..." aria-label="Filter input" data-test-subj="dashboardQuery" type="text" class="kuiLocalSearchInput ng-pristine ng-untouched ng-valid ng-isolate-scope" ng-class="{'kuiLocalSearchInput-isInvalid': queryInput.$invalid}" autocomplete="off">
  (Session info: chrome=56.0.2924.87)
  (Driver info: chromedriver=2.24.417424 (c5c5ea873213ee72e3d0929b47482681555340c3),platform=Linux 4.4.0-45-generic x86_64)
  at runRequest  <node_modules/leadfoot/Session.js:88:40>
  at <node_modules/leadfoot/Session.js:109:39>
  at new Promise  <node_modules/dojo/_debug/Promise.ts:411:4>
  at ProxiedSession._post  <node_modules/leadfoot/Session.js:63:10>
  at Element._post  <node_modules/leadfoot/Element.js:23:31>
  at Element.click  <node_modules/leadfoot/Element.js:163:15>
  at Command.<anonymous>  <node_modules/leadfoot/Command.js:680:19>
  at <node_modules/dojo/_debug/Promise.ts:393:16>

stacey-gammon · 2017-04-03T20:58:26Z

Iteration 2: passed once.

Jenkins, test this

stacey-gammon · 2017-04-04T01:29:12Z

Iteration 2: Passed twice

Jenkins, test this

stacey-gammon · 2017-04-04T11:23:15Z

Iteration 2: Third build was aborted.

Jenkins, test this.

stacey-gammon · 2017-04-04T12:34:32Z

Iteration 2: Passed three times.

Jenkins, test this.

* Convert default to named exports - errors * Make all error classes use class syntax and extends add tests * Extending Error apparently doesn't work * Merge PR #11004 to try to fix test failures

stacey-gammon · 2017-04-04T14:04:22Z

Iteration 2: Passed four times.
One final run.
Jenkins, test this.

stacey-gammon · 2017-04-04T14:52:40Z

Well, that's too bad. Still failing:

14:49:26.481: Setting absolute range to 2015-09-19 06:31:44.000 to 2015-09-23 18:31:44.000
14:49:28.516: time picker open: false
14:49:28.516: --Opening time picker
14:49:28.516: in findTestSubject: [data-test-subj~="globalTimepickerButton"]
14:49:28.647: --Clicking Absolute button
14:49:28.735: --Setting From Time : 2015-09-19 06:31:44.000
14:49:38.788: Taking screenshot "/var/lib/jenkins/workspace/elastic+kibana+pull-request+multijob-selenium/test/screenshots/failure/failure_1491317378787_dashboard without stored timed_Does not set the time picker on open.png"
>> FAIL: chrome on any platform - kibana - dashboard app - dashboard time - dashboard without stored timed - Does not set the time picker on open (12431ms)
NoSuchElement: [POST http://localhost:4444/wd/hub/session/531d6a26b123d07c89a7037e571ba092/element / {"using":"css selector","value":"input[ng-model='absolute.from']"}] no such element: Unable to locate element: {"method":"css selector","selector":"input[ng-model='absolute.from']"}
  (Session info: chrome=56.0.2924.87)
  (Driver info: chromedriver=2.24.417424 (c5c5ea873213ee72e3d0929b47482681555340c3),platform=Linux 3.13.0-74-generic x86_64)
  at runRequest  <node_modules/leadfoot/Session.js:88:40>
  at <node_modules/leadfoot/Session.js:109:39>
  at new Promise  <node_modules/dojo/_debug/Promise.ts:411:4>
  at ProxiedSession._post  <node_modules/leadfoot/Session.js:63:10>
  at ProxiedSession.find  <node_modules/leadfoot/Session.js:1227:15>
  at Command.<anonymous>  <node_modules/leadfoot/Command.js:42:36>
  at <node_modules/dojo/_debug/Promise.ts:393:16>
  at run  <node_modules/dojo/_debug/Promise.ts:237:8>
  at <node_modules/dojo/_debug/nextTick.ts:44:4>

Screenshot:

That message has been up for ~20 seconds so I think it might be unrelated to the errors. Looks like clicking the absolute link failed, but the clickAbsoluteButton function (which is wrapped in a try) unfortunately didn't report an error.

stacey-gammon · 2017-04-04T16:58:50Z

Iteration 3: Passed Once
Jenkins, test this.

stacey-gammon · 2017-04-04T17:43:07Z

Iteration 3: Passed Twice

Jenkins, test this.

* Convert default to named exports - errors * Make all error classes use class syntax and extends add tests * Extending Error apparently doesn't work * Merge PR elastic#11004 to try to fix test failures

stacey-gammon · 2017-04-04T18:31:16Z

Iteration 3: Passed Thrice.

Jenkins, test this.

* Convert default to named exports - errors * Make all error classes use class syntax and extends add tests * Extending Error apparently doesn't work * Merge PR #11004 to try to fix test failures

stacey-gammon · 2017-04-04T19:25:00Z

Iteration 3: Fourth time aborted.

Jenkins, test this.

stacey-gammon · 2017-04-04T20:14:46Z

Iteration 3: Fourth time passing! I think this is ready to go.

Rebasing to kick off another test.

stacey-gammon · 2017-04-05T00:24:43Z

build aborted.

jenkins, test this

* Convert default to named exports - errors * Make all error classes use class syntax and extends add tests * Extending Error apparently doesn't work * Merge PR elastic#11004 to try to fix test failures

rhoboat · 2017-04-05T19:23:48Z

@stacey-gammon Looks promising! Please don't merge this up yet. I think #10910 may supersede some of the these fixes. I'd also like to pull your fixes that my PR doesn't have. If this branch went into master, however, merging with mine will be nontrivial, so I'd like to avoid that if possible.

rhoboat · 2017-04-05T19:28:28Z

@stacey-gammon Which flaky dashboard test(s) does this fix? I can keep an eye out for that as I run tests with these changes.

stacey-gammon · 2017-04-05T19:30:59Z

np @archanid, sorry for the mix up, didn't realize you were actively working on this.

One small additional change was fixing text add new visualization link -- At some point it because a test that was never run because of mixed up parens. lmk if you want to put that in your PR, otherwise I can create a separate PR.

Other than that it was mostly the toast notifications that were causing issues, so if your PR doesn't address that specifically, may want to close them purposefully.

One of the most common failing tests was:

FAIL: chrome on any platform - kibana - dashboard app - dashboard time - dashboard without stored timed - Does not set the time picker on open (51837ms)

Thanks!

rhoboat · 2017-04-05T20:51:50Z

test/functional/apps/dashboard/_dashboard_save.js

+  // To avoid load errors in subsequent tests that refresh the index.
+  bdd.it('Finish each test on the landing page', async () => {
+    await PageObjects.dashboard.gotoDashboardLandingPage();
+  });


Hmm.. Why make this an it? There is no expect here. Shouldn't this just be an after hook? Sounds like you want this to run at the end, so it would be an after all hook.

Yup, that would make more sense. 👍 and be much more stable.

rhoboat · 2017-04-05T22:13:55Z

@stacey-gammon I've incorporated all of your changes into my PR. I haven't run it yet, will update when I do.

spalger · 2017-04-07T01:43:59Z

Tests passing 🎉 🕺 🥇

stacey-gammon · 2017-04-12T11:07:47Z

Closing in favor of #10910

stacey-gammon mentioned this pull request Apr 3, 2017

Dashboard fails 'Could not locate that dashboard (id: ...)' #10980

Closed

stacey-gammon added the :Sharing label Apr 3, 2017

stacey-gammon added Feature:Dashboard Dashboard related features test labels Apr 3, 2017

stacey-gammon added a commit to stacey-gammon/kibana that referenced this pull request Apr 3, 2017

Merge PR elastic#11004 to try to fix test failures

5d9d1ef

stacey-gammon mentioned this pull request Apr 3, 2017

Convert default to named exports - errors #10986

Merged

stacey-gammon mentioned this pull request Apr 4, 2017

[5.x] Convert default to named exports - errors (#10986) #11024

Merged

stacey-gammon changed the title ~~Put click dashboard by link text in a try.~~ Stabilize dashboard selenium tests Apr 4, 2017

stacey-gammon added the review label Apr 4, 2017

stacey-gammon requested review from LeeDr and bhavyarm April 4, 2017 20:14

stacey-gammon force-pushed the fix-flaky-dashboard-test branch from c11bf6c to fe4b4a4 Compare April 4, 2017 20:15

stacey-gammon force-pushed the fix-flaky-dashboard-test branch from fe4b4a4 to be2f1b1 Compare April 5, 2017 11:10

stacey-gammon added 2 commits April 5, 2017 10:27

Add more trys, especially around clicks

3481be7

Add more trys and remove toast messages immediately

e9cd1bd

stacey-gammon force-pushed the fix-flaky-dashboard-test branch from be2f1b1 to e9cd1bd Compare April 5, 2017 14:32

stacey-gammon requested a review from rhoboat April 5, 2017 18:56

rhoboat reviewed Apr 5, 2017

View reviewed changes

stacey-gammon closed this Apr 12, 2017

stacey-gammon deleted the fix-flaky-dashboard-test branch April 12, 2017 11:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stabilize dashboard selenium tests #11004

Stabilize dashboard selenium tests #11004

stacey-gammon commented Apr 3, 2017

stacey-gammon commented Apr 3, 2017

stacey-gammon commented Apr 3, 2017

stacey-gammon commented Apr 3, 2017 •

edited

stacey-gammon commented Apr 3, 2017 •

edited

stacey-gammon commented Apr 4, 2017

stacey-gammon commented Apr 4, 2017

stacey-gammon commented Apr 4, 2017

stacey-gammon commented Apr 4, 2017

stacey-gammon commented Apr 4, 2017 •

edited

stacey-gammon commented Apr 4, 2017

stacey-gammon commented Apr 4, 2017

stacey-gammon commented Apr 4, 2017

stacey-gammon commented Apr 4, 2017

stacey-gammon commented Apr 4, 2017 •

edited

stacey-gammon commented Apr 5, 2017

rhoboat commented Apr 5, 2017

rhoboat commented Apr 5, 2017

stacey-gammon commented Apr 5, 2017

rhoboat Apr 5, 2017

stacey-gammon Apr 5, 2017 •

edited

rhoboat commented Apr 5, 2017

spalger commented Apr 7, 2017

stacey-gammon commented Apr 12, 2017

Stabilize dashboard selenium tests #11004

Stabilize dashboard selenium tests #11004

Conversation

stacey-gammon commented Apr 3, 2017

stacey-gammon commented Apr 3, 2017

stacey-gammon commented Apr 3, 2017

stacey-gammon commented Apr 3, 2017 • edited

stacey-gammon commented Apr 3, 2017 • edited

stacey-gammon commented Apr 4, 2017

stacey-gammon commented Apr 4, 2017

stacey-gammon commented Apr 4, 2017

stacey-gammon commented Apr 4, 2017

stacey-gammon commented Apr 4, 2017 • edited

stacey-gammon commented Apr 4, 2017

stacey-gammon commented Apr 4, 2017

stacey-gammon commented Apr 4, 2017

stacey-gammon commented Apr 4, 2017

stacey-gammon commented Apr 4, 2017 • edited

stacey-gammon commented Apr 5, 2017

rhoboat commented Apr 5, 2017

rhoboat commented Apr 5, 2017

stacey-gammon commented Apr 5, 2017

rhoboat Apr 5, 2017

Choose a reason for hiding this comment

stacey-gammon Apr 5, 2017 • edited

Choose a reason for hiding this comment

rhoboat commented Apr 5, 2017

spalger commented Apr 7, 2017

stacey-gammon commented Apr 12, 2017

stacey-gammon commented Apr 3, 2017 •

edited

stacey-gammon commented Apr 3, 2017 •

edited

stacey-gammon commented Apr 4, 2017 •

edited

stacey-gammon commented Apr 4, 2017 •

edited

stacey-gammon Apr 5, 2017 •

edited