Failing test: Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/discover/async_scripted_fields·js - discover async search with scripted fields query return results with valid scripted field #78553

kibanamachine · 2020-09-26T10:23:31Z

A test failed on a tracked branch

Error: retry.tryForTime timeout: Error: expected '14,005' to equal '13,301'
    at Assertion.assert (/dev/shm/workspace/parallel/2/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.be.Assertion.equal (/dev/shm/workspace/parallel/2/kibana/packages/kbn-expect/expect.js:227:8)
    at Assertion.be (/dev/shm/workspace/parallel/2/kibana/packages/kbn-expect/expect.js:69:22)
    at /dev/shm/workspace/parallel/2/kibana/x-pack/test/functional/apps/discover/async_scripted_fields.js:122:61
    at process._tickCallback (internal/process/next_tick.js:68:7)
    at onFailure (/dev/shm/workspace/parallel/2/kibana/test/common/services/retry/retry_for_success.ts:28:9)
    at retryForSuccess (/dev/shm/workspace/parallel/2/kibana/test/common/services/retry/retry_for_success.ts:68:13)

First failure: Jenkins Build

The text was updated successfully, but these errors were encountered:

elasticmachine · 2020-09-26T10:23:32Z

Pinging @elastic/kibana-test-triage (failed-test)

elasticmachine · 2020-09-28T07:04:29Z

Pinging @elastic/kibana-app (Team:KibanaApp)

wylieconlon · 2020-09-28T16:54:59Z

This fails every day or so, and I think it could be related to the timeout-related changes from async search. Running a flaky test suite to see if it's easy to reproduce.

wylieconlon · 2020-09-28T19:36:08Z

Ran 42 runs on master and they all succeeded. I will test 7.x separately, and see if I can reproduce in cloud at all.

kertal · 2020-09-30T10:19:52Z

FYI @LeeDr there seems to be a kind a flakiness, from time to time showing a false hit count

kibanamachine · 2020-10-01T15:24:08Z

New failure: Jenkins Build

LeeDr · 2020-10-01T16:10:15Z

I added this test on June 29th 2020. And Rashmi revised to setRoles to the minimal roles required on July 16th.
The first failure was on August 4th. I'm finding 14 failures since then with 2 of those on 7.x builds and the other 12 on PRs. So that's out of about 100,000 builds.
Here's the most recent (October 1st 2020, 10:24:35.506) failure console output (slightly edited to remove spurious info). In this case, it never loaded any results and shows an error dialog;

[00:08:28]             └-> query return results with valid scripted field
[00:08:28]               └-> "before each" hook: global before each
[00:08:28]               │ debg TestSubjects.click(indexPattern-switch-link)
[00:08:28]               │ debg Find.clickByCssSelector('[data-test-subj="indexPattern-switch-link"]') with timeout=10000
[00:08:28]               │ debg Find.findByCssSelector('[data-test-subj="indexPattern-switch-link"]') with timeout=10000
[00:08:28]               │ debg Find.setValue('[data-test-subj="indexPattern-switcher"] input', 'logstash-*')
[00:08:28]               │ debg Find.findByCssSelector('[data-test-subj="indexPattern-switcher"] input') with timeout=10000
[00:08:28]               │ debg Find.clickByCssSelector('[data-test-subj="indexPattern-switcher"] [title="logstash-*"]') with timeout=10000
[00:08:28]               │ debg Find.findByCssSelector('[data-test-subj="indexPattern-switcher"] [title="logstash-*"]') with timeout=10000
[00:08:28]               │ debg isGlobalLoadingIndicatorVisible
[00:08:28]               │ debg TestSubjects.exists(globalLoadingIndicator)
[00:08:28]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:08:28]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:08:28]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:08:31]               │ debg QueryBar.setQuery(php* OR *jpg OR *css*)
[00:08:31]               │ debg TestSubjects.click(queryInput)
[00:08:31]               │ debg Find.clickByCssSelector('[data-test-subj="queryInput"]') with timeout=10000
[00:08:31]               │ debg Find.findByCssSelector('[data-test-subj="queryInput"]') with timeout=10000
[00:08:32]               │ debg TestSubjects.getAttribute(queryInput, value)
[00:08:32]               │ debg TestSubjects.find(queryInput)
[00:08:32]               │ debg Find.findByCssSelector('[data-test-subj="queryInput"]') with timeout=10000
[00:08:32]               │ debg TestSubjects.click(querySubmitButton)
[00:08:32]               │ debg Find.clickByCssSelector('[data-test-subj="querySubmitButton"]') with timeout=10000
[00:08:32]               │ debg Find.findByCssSelector('[data-test-subj="querySubmitButton"]') with timeout=10000
[00:08:32]               │ debg isGlobalLoadingIndicatorVisible
[00:08:32]               │ debg TestSubjects.exists(globalLoadingIndicator)
[00:08:32]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:08:34]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:08:34]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:08:35]               │ info [o.e.x.c.a.AsyncResultsService] [kibana-ci-immutable-centos-tests-xxl-1601560039535949685] failed to update expiration time for async-search [FlpWWlc2THlNUUtHYk1WelB1UzdnOVEcODF0SkRwQjdUMzZ6Q0x2YjBwREdJUToxMDg1Mg==]
[00:08:35]               │      org.elasticsearch.transport.RemoteTransportException: [kibana-ci-immutable-centos-tests-xxl-1601560039535949685][127.0.0.1:6133][indices:data/write/update[s]]
[00:08:35]               │      Caused by: org.elasticsearch.index.engine.VersionConflictEngineException: [ZVZW6LyMQKGbMVzPuS7g9Q]: version conflict, required seqNo [15], primary term [1]. current document has seqNo [16] and primary term [1]
[00:08:35]               │ proc [kibana]   log   [14:27:38.084] [error][data][elasticsearch] [resource_not_found_exception]: FlpWWlc2THlNUUtHYk1WelB1UzdnOVEcODF0SkRwQjdUMzZ6Q0x2YjBwREdJUToxMDg1Mg==
[00:08:35]               │ERROR browser[SEVERE] http://localhost:6131/internal/search/ese/FlpWWlc2THlNUUtHYk1WelB1UzdnOVEcODF0SkRwQjdUMzZ6Q0x2YjBwREdJUToxMDg1Mg== - Failed to load resource: the server responded with a status of 404 (Not Found)
[00:08:35]               │ debg TestSubjects.getVisibleText(discoverQueryHits)
[00:08:35]               │ debg TestSubjects.find(discoverQueryHits)
[00:08:35]               │ debg Find.findByCssSelector('[data-test-subj="discoverQueryHits"]') with timeout=10000
[00:08:45]               │ debg --- retry.tryForTime error: Waiting for element to be located By(css selector, [data-test-subj="discoverQueryHits"])
[00:08:45]               │      Wait timed out after 10021ms
[00:08:45]               │ debg isGlobalLoadingIndicatorVisible
[00:08:45]               │ debg TestSubjects.exists(globalLoadingIndicator)
[00:08:45]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:08:47]               │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:08:47]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:08:47]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:08:47]               │ debg TestSubjects.getVisibleText(discoverQueryHits)
[00:08:47]               │ debg TestSubjects.find(discoverQueryHits)
[00:08:47]               │ debg Find.findByCssSelector('[data-test-subj="discoverQueryHits"]') with timeout=10000

[00:08:58]               │ debg --- retry.tryForTime error: Waiting for element to be located By(css selector, [data-test-subj="discoverQueryHits"])
[00:08:58]               │      Wait timed out after 10037ms
[00:08:58]               │ debg isGlobalLoadingIndicatorVisible
[00:08:58]               │ debg TestSubjects.exists(globalLoadingIndicator)
[00:08:58]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:09:00]               │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:09:00]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:09:00]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:09:00]               │ debg TestSubjects.getVisibleText(discoverQueryHits)
[00:09:00]               │ debg TestSubjects.find(discoverQueryHits)
[00:09:00]               │ debg Find.findByCssSelector('[data-test-subj="discoverQueryHits"]') with timeout=10000
[00:09:10]               │ debg --- retry.tryForTime error: Waiting for element to be located By(css selector, [data-test-subj="discoverQueryHits"])
[00:09:10]               │      Wait timed out after 10023ms
[00:09:11]               │ info Taking screenshot "/dev/shm/workspace/parallel/3/kibana/x-pack/test/functional/screenshots/failure/discover async search with scripted fields query return results with valid scripted field.png"
[00:09:11]               │ info Current URL is: http://localhost:6131/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:%27Sep%2018,%202015%20@%2019:37:13.000%27,to:%27Sep%2023,%202015%20@%2002:30:09.000%27))&_a=(columns:!(_source),filters:!(),index:eddf62c0-85bb-11ea-a555-0b6139dddc19,interval:auto,query:(language:kuery,query:%27php*%20OR%20*jpg%20OR%20*css*%27),sort:!())
[00:09:11]               │ info Saving page source to: /dev/shm/workspace/parallel/3/kibana/x-pack/test/functional/failure_debug/html/discover async search with scripted fields query return results with valid scripted field.html
[00:09:11]               └- ✖ fail: discover async search with scripted fields query return results with valid scripted field
[00:09:11]               │      Error: retry.tryForTime timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="discoverQueryHits"])
[00:09:11]               │ Wait timed out after 10023ms
[00:09:11]               │     at /dev/shm/workspace/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
[00:09:11]               │     at process._tickCallback (internal/process/next_tick.js:68:7)
[00:09:11]               │       at onFailure (/dev/shm/workspace/parallel/3/kibana/test/common/services/retry/retry_for_success.ts:28:9)
[00:09:11]               │       at retryForSuccess (/dev/shm/workspace/parallel/3/kibana/test/common/services/retry/retry_for_success.ts:68:13)
[00:09:11]               │ 
[00:09:11]               │

And the screenshot;

LeeDr · 2020-10-01T16:25:39Z

Failure on September 30th 2020, 09:49:14.546 looked the same.
. same on September 29th 2020, 11:20:22.365

On September 28th 2020, 15:13:11.238 the failure screenshot looked different. It looks like it loaded the page with 14,005 hits (which is the expected pre-query hit count), and then did the query, hit an error, and didn't clear the previous results.

I checked 3 more before this and they had the same resource_not_found error dialog.

I think I need to make the test click on the See the full error button to capture what's there.

LeeDr · 2020-10-01T16:50:04Z

Found this in the Jenkins failed test log;

[00:08:47]               │ info [o.e.x.c.a.AsyncResultsService] [kibana-ci-immutable-debian-tests-xxl-1601110029429513606] failed to update expiration time for async-search [FmNISDNMTlRMVFd5RUptMF9fVjRBZnccc1FKMHRac0VSTC1USFlQSFVrZ0lqZzoxMDg0Ng==]
[00:08:47]               │      org.elasticsearch.transport.RemoteTransportException: [kibana-ci-immutable-debian-tests-xxl-1601110029429513606][127.0.0.1:6123][indices:data/write/update[s]]
[00:08:47]               │      Caused by: org.elasticsearch.index.engine.VersionConflictEngineException: [cHH3LNTLTWyEJm0__V4Afw]: version conflict, required seqNo [21], primary term [1]. current document has seqNo [22] and primary term [1]
[00:08:47]               │ proc [kibana]   log   [09:26:37.381] [error][data][elasticsearch] [resource_not_found_exception]: FmNISDNMTlRMVFd5RUptMF9fVjRBZnccc1FKMHRac0VSTC1USFlQSFVrZ0lqZzoxMDg0Ng==
[00:08:47]               │ERROR browser[SEVERE] http://localhost:6121/internal/search/ese/FmNISDNMTlRMVFd5RUptMF9fVjRBZnccc1FKMHRac0VSTC1USFlQSFVrZ0lqZzoxMDg0Ng== - Failed to load resource: the server responded with a status of 404 (Not Found)
[00:08:47]               │ debg TestSubjects.getVisibleText(discoverQueryHits)
[00:08:47]               │ debg TestSubjects.find(discoverQueryHits)
[00:08:47]               │ debg Find.findByCssSelector('[data-test-subj="discoverQueryHits"]') with timeout=10000
[00:08:47]               │ debg --- retry.tryForTime error: expected '14,005' to equal '13,301'

LeeDr · 2020-10-01T16:53:15Z

@lizozom with this additional info from the failed test, is this something you could look into? It's not failing all that often so it may be pretty hard to reproduce.

It initially started failing around Aug 4th with the resource_no_found_exception dialog and log messages failed to update expiration time for async-search, version conflict.

But seems to have changed around September 29th to just show Not Found, with the same underlying log messages in the test debug log.

elasticmachine · 2020-10-01T17:05:56Z

Pinging @elastic/kibana-app-arch (Team:AppArch)

tylersmalley · 2020-12-15T04:52:06Z

Skipped, blocking master/7.x

master: 003396e
7.x: 1a8a6db

kibanamachine · 2020-12-15T04:58:53Z

New failure: Jenkins Build

kibanamachine · 2020-12-15T04:58:54Z

New failure: Jenkins Build

* master: (66 commits) [Alerting] fixes broken Alerting Example plugin (elastic#85774) [APM] Service overview instances table (elastic#85770) [Security Solution] Unskip timeline creation Cypress test (elastic#85871) properly recognize enterprise licenses (elastic#85849) [SecuritySolution][Detections] Adds SavedObject persistence to Signals Migrations (elastic#85690) [TSVB] Fix functional tests flakiness and unskip them (elastic#85388) [Fleet] Change permissions for Fleet enroll role (elastic#85802) Gauge visualization can no longer be clicked to filter on values since Kibana 7.10.0 (elastic#84768) [Security Solution][Detections] Add alert source to detection rule action context (elastic#85488) [Discover] Don't display hide/show button for histogram when there's no time filter (elastic#85424) skip flaky suite (elastic#78553) License checks for alerts plugin (elastic#85649) skip flaky suite (elastic#84992) skip 'query return results valid for scripted field' elastic#78553 Allow action types to perform their own mustache variable escaping in parameter templates (elastic#83919) [ML] More machine learning links in doc_links_service.ts (elastic#85365) Removed Alerting & Event Log deprecated fields that should not be using (elastic#85652) Closes elastic#79995 by adding new tab in transaction details to show related trace logs. (elastic#85859) Fix outdated jest snapshot [Maps] Surface on prem EMS (elastic#85729) ...

lukasolson · 2021-05-04T20:55:38Z

Resolved by #94653.

kibanamachine · 2022-07-18T09:08:14Z

New failure: CI Build - main

tsullivan · 2022-08-12T16:21:20Z

Not sure if this test is still truly flaky.

The cause of the last failure is Error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="indexPattern-switcher"] [title="logstash-*"])

kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Sep 26, 2020

azasypkin added Team:Visualizations Visualization editors, elastic-charts and infrastructure triage_needed labels Sep 28, 2020

kertal added Feature:Discover Discover Application :KibanaApp/fix-it-week labels Sep 28, 2020

wylieconlon self-assigned this Sep 28, 2020

wylieconlon mentioned this issue Sep 29, 2020

Failing test: Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/discover/async_scripted_fields·js - discover async search with scripted fields query should show failed shards pop up #78743

Closed

wylieconlon removed their assignment Oct 1, 2020

LeeDr self-assigned this Oct 1, 2020

LeeDr assigned lizozom and unassigned LeeDr Oct 1, 2020

wylieconlon added Project:AsyncSearch Background search, partial results, async search services. Team:AppArch labels Oct 1, 2020

wylieconlon removed Feature:Discover Discover Application Team:Visualizations Visualization editors, elastic-charts and infrastructure triage_needed labels Oct 1, 2020

wylieconlon added the bug Fixes for quality problems that affect the customer experience label Oct 1, 2020

lukasolson self-assigned this Oct 1, 2020

tylersmalley added a commit that referenced this issue Dec 15, 2020

skip flaky suite (#78553)

1a8a6db

tylersmalley added blocker skipped-test v7.11.0 v8.0.0 labels Dec 15, 2020

Dosant added SharedUX/fix-it-week Bugs that have been groomed and queued up for the team's next fix it week and removed bug Fixes for quality problems that affect the customer experience labels Feb 24, 2021

lukasolson mentioned this issue Mar 15, 2021

Re-enable skipped test (discover with async scripted fields) #94653

Merged

9 tasks

lukasolson closed this as completed May 4, 2021

jloleysens mentioned this issue Jul 9, 2021

Failing ES Promotion: discover async search with scripted fields query return results with valid scripted field #104362

Closed

kibanamachine reopened this Jul 18, 2022

exalate-issue-sync bot added the impact:critical This issue should be addressed immediately due to a critical level of impact on the product. label Jul 21, 2022

lukasolson removed their assignment Sep 26, 2022

exalate-issue-sync bot added the loe:small Small Level of Effort label Oct 4, 2022

jloleysens mentioned this issue Oct 17, 2022

Fix flaky test 78553 #143346

Merged

jloleysens closed this as completed in #143346 Oct 19, 2022

kibanamachine removed v8.0.0 blocker v7.11.0 skipped-test labels Oct 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failing test: Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/discover/async_scripted_fields·js - discover async search with scripted fields query return results with valid scripted field #78553

Failing test: Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/discover/async_scripted_fields·js - discover async search with scripted fields query return results with valid scripted field #78553

kibanamachine commented Sep 26, 2020 •

edited

Loading

elasticmachine commented Sep 26, 2020

elasticmachine commented Sep 28, 2020

wylieconlon commented Sep 28, 2020

wylieconlon commented Sep 28, 2020

kertal commented Sep 30, 2020

kibanamachine commented Oct 1, 2020

LeeDr commented Oct 1, 2020 •

edited

Loading

LeeDr commented Oct 1, 2020

LeeDr commented Oct 1, 2020

LeeDr commented Oct 1, 2020 •

edited

Loading

elasticmachine commented Oct 1, 2020

tylersmalley commented Dec 15, 2020 •

edited

Loading

kibanamachine commented Dec 15, 2020

kibanamachine commented Dec 15, 2020

lukasolson commented May 4, 2021

kibanamachine commented Jul 18, 2022

tsullivan commented Aug 12, 2022

Failing test: Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/discover/async_scripted_fields·js - discover async search with scripted fields query return results with valid scripted field #78553

Failing test: Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/discover/async_scripted_fields·js - discover async search with scripted fields query return results with valid scripted field #78553

Comments

kibanamachine commented Sep 26, 2020 • edited Loading

elasticmachine commented Sep 26, 2020

elasticmachine commented Sep 28, 2020

wylieconlon commented Sep 28, 2020

wylieconlon commented Sep 28, 2020

kertal commented Sep 30, 2020

kibanamachine commented Oct 1, 2020

LeeDr commented Oct 1, 2020 • edited Loading

LeeDr commented Oct 1, 2020

LeeDr commented Oct 1, 2020

LeeDr commented Oct 1, 2020 • edited Loading

elasticmachine commented Oct 1, 2020

tylersmalley commented Dec 15, 2020 • edited Loading

kibanamachine commented Dec 15, 2020

kibanamachine commented Dec 15, 2020

lukasolson commented May 4, 2021

kibanamachine commented Jul 18, 2022

tsullivan commented Aug 12, 2022

kibanamachine commented Sep 26, 2020 •

edited

Loading

LeeDr commented Oct 1, 2020 •

edited

Loading

LeeDr commented Oct 1, 2020 •

edited

Loading

tylersmalley commented Dec 15, 2020 •

edited

Loading