Failing test: Firefox UI Functional Tests.test/functional/apps/dashboard/dashboard_filtering·js - dashboard app using current data dashboard filtering disabling a filter unfilters the data on saved searches

[kibanamachine]

A test failed on a tracked branch

        Error: retry.try timeout: Error: expected 0 to equal 1
    at Assertion.assert (packages/kbn-expect/expect.js:100:11)
    at Assertion.be.Assertion.equal (packages/kbn-expect/expect.js:221:8)
    at Assertion.(anonymous function) [as be] (packages/kbn-expect/expect.js:69:22)
    at be (test/functional/services/dashboard/expectations.js:217:43)
    at process._tickCallback (internal/process/next_tick.js:68:7)
    at lastError (test/common/services/retry/retry_for_success.ts:28:9)
    at onFailure (test/common/services/retry/retry_for_success.ts:68:13)
      

First failure: Jenkins Build

[elasticmachine]

Pinging @elastic/kibana-test-triage

[kibanamachine]

New failure: Jenkins Build

[kibanamachine]

New failure: Jenkins Build

[elasticmachine]

Pinging @elastic/kibana-app

[spalger]

skipped

master: f0e7b9e
7.x/7.4: 3b59476
7.3: 3c0426d

[kibanamachine]

New failure: Jenkins Build

[stacey-gammon]

- ✓ pass  (74ms) "dashboard app using current data dashboard filtering disabling a filter unfilters the data on tsvb markdown"
[00:09:42]                 └-> saved searches
[00:09:42]                   └-> "before each" hook: global before each
[00:09:42]                   │ debg DashboardExpect.savedSearchRowCount(1)
[00:09:42]                   │ debg TestSubjects.findAll(docTableExpandToggleColumn)
[00:09:42]                   │ debg Find.allByCssSelector('[data-test-subj~="docTableExpandToggleColumn"]') with timeout=2500
[00:09:45]                   │ debg --- retry.try error: expected 0 to equal 1
[00:09:45]                   │ debg TestSubjects.findAll(docTableExpandToggleColumn)
[00:09:45]                   │ debg Find.allByCssSelector('[data-test-subj~="docTableExpandToggleColumn"]') with timeout=2500
[00:09:48]                   │ debg --- retry.try failed again with the same message...
[00:09:48]                   │ debg TestSubjects.findAll(docTableExpandToggleColumn)
[00:09:48]                   │ debg Find.allByCssSelector('[data-test-subj~="docTableExpandToggleColumn"]') with timeout=2500
[00:09:51]                   │ debg --- retry.try failed again with the same message...

screenshot won't help bc the saved search is below the page.

Maybe there is a timing issue here that is just very difficult to repro (it passed like 80x in a row before I turned it back on).

[kibanamachine]

New failure: Jenkins Build

[dmlemeshko]

Exact the same behaviour:

[00:08:52]                 └-> saved searches
[00:08:52]                   └-> "before each" hook: global before each
[00:08:52]                   │ debg DashboardExpect.savedSearchRowCount(1)
[00:08:52]                   │ debg TestSubjects.findAll(docTableExpandToggleColumn)
[00:08:52]                   │ debg Find.allByCssSelector('[data-test-subj~="docTableExpandToggleColumn"]') with timeout=2500
[00:08:55]                   │ debg --- retry.try error: expected 0 to equal 1
[00:08:55]                   │ debg TestSubjects.findAll(docTableExpandToggleColumn)
[00:08:55]                   │ debg Find.allByCssSelector('[data-test-subj~="docTableExpandToggleColumn"]') with timeout=2500
[00:08:58]                   │ debg --- retry.try failed again with the same message...
...
[00:10:50]                   │ debg --- retry.try failed again with the same message...
[00:10:50]                   │ debg TestSubjects.findAll(docTableExpandToggleColumn)
[00:10:50]                   │ debg Find.allByCssSelector('[data-test-subj~="docTableExpandToggleColumn"]') with timeout=2500
[00:10:53]                   │ debg --- retry.try failed again with the same message...
[00:10:53]                   │ info Taking screenshot "/var/lib/jenkins/workspace/elastic+kibana+master/JOB/kibana-firefoxSmoke/node/linux-immutable/kibana/test/functional/screenshots/failure/dashboard app using current data dashboard filtering disabling a filter unfilters the data on saved searches.png"

@stacey-gammon I don't think it is timing issue, because we wait for rendering completed in before:

      before(async () => {
        await filterBar.toggleFilterEnabled('bytes');
        await PageObjects.header.waitUntilLoadingHasFinished();
        await PageObjects.dashboard.waitForRenderComplete();
      });

And it is not the first test in a suite, so it has 7 additional seconds to wait.

I think it is important to notice it fails on Firefox, not Chrome. I recently submitted FF filtering bug, not sure it can be related: #41567

[stacey-gammon]

@dmlemeshko - I meant more a timing issues in the dashboard app itself, not with the tests. Like a legit bug. Updating the filters happens async, and updating the dashboard container happens async too. I'm wondering if it's possible something like this is going on:

• filters load with value x
• dashboard container initializes with filters x
• filters async update to y
• dashboard container update finishes, dashboard app compares with container filters and sets itself back to x.

This is what I wanted to try to figure out when debugging this in the first place but I couldn't repro. :(

I think it is important to notice it fails on Firefox, not Chrome.

are you sure that is the case? Can you explain to me how the tests run on both chrome and firefox? Looking at the tests from the PR that turned these back on, and I only see firefox running:

[stacey-gammon]

I was about to say that that timing situation wouldn't account for why it always fails on saved searches and not the other visualizations, but I just remembered that visualizations bypass the embeddables and get their filters direct from the angular service so that would actually explain it!

I will try to see if I can force this situation to occur by adding in some sleeps/timeouts.

[dmlemeshko]

are you sure that is the case? Can you explain to me how the tests run on both chrome and firefox? Looking at the tests from the PR that turned these back on, and I only see firefox running:

I'm pretty sure since failures have title Firefox UI Functional Tests, it comes from firefox config.

All the functional tests are run on Chrome by default, particular this one belongs to ciGroup3: index.js

To run specific tests in Firefox (in addition to Chrome) we use smoke tag, it is applied in dashboard_filtering.js and we use custom config.firefox.js to run it on dedicated worker firefoxSmoke (the same setup is for xpack tests)

So when we say it runs on both Chrome and Firefox, it means we run the same tests in different workers.

To run any test on Firefox locally you need to pass config: --config test/functional/config.firefox.js

I know there is a lack of details around Firefox run, I'm finishing PR with updated docs around functional test runner to fix it.

[kibanamachine]

New failure: Jenkins Build

[kibanamachine]

New failure: Jenkins Build

[dmlemeshko]

@spalger @stacey-gammon should we disable (skip) it for Firefox?

[stacey-gammon]

We are able to skip it only for firefox but it will still run on chrome? That wouldn't be too terrible, most bugs should still be caught.