Investigate flaky tests in CI #3538

web-padawan · 2022-03-10T13:24:50Z

This ticket is a list of tests that fail in GitHub Actions occasionally. We need to investigate if they can be fixed, or skipped.

Unit tests

Chrome

packages/date-picker/test/keyboard-navigation.test.js:

 ❌ keyboard navigation > overlay > should focus the closest allowed date with pagedown when selected date is disabled
      TypeError: Cannot read properties of null (reading 'date')
        at qy.<anonymous> (packages/date-picker/test/keyboard-navigation.test.js:383:19)

Firefox

packages/date-picker/test/keyboard-navigation.test.js:

 ❌ keyboard navigation > overlay > should focus min date with shift and pageup
      TypeError: cell is null
        at packages/date-picker/test/keyboard-navigation.test.js:363:7

packages/date-picker/test/keyboard-navigation.test.js:

 ❌ keyboard navigation > overlay > should focus the closest allowed date with shift pageup when selected date is disabled
      TypeError: cell is null
        at packages/date-picker/test/keyboard-navigation.test.js:397:7

packages/date-picker/test/keyboard-input.test.js:

 ❌ keyboard > change event > should fire change on selecting date with Enter
      AssertionError: expected false to be true
      + expected - actual
      
      -false
      +true
      
      at r (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9574:12)
      at n.exports<[3]</t.exports/i.prototype.assert (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:250:12)
      at n.exports<[5]</t.exports/< (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:1037:9)
      at e (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:7959:28)
      at e (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9372:21)
      at packages/date-picker/test/keyboard-input.test.js:524:13

packages/form-layout/test/form-layout.test.js:

 ❌ form layout > container overflow > no spacing on edges (rtl) > should not have spacing on the right edge
      AssertionError: expected 288 to be close to 300 +/- 0.5
        at r (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9574:12)
        at n.exports<[3]</t.exports/i.prototype.assert (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:250:12)
        at S (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:[33](https://github.com/vaadin/web-components/runs/5445517447?check_suite_focus=true#step:6:33)50:9)
        at c (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:7884:24)
        at packages/form-layout/test/form-layout.test.js:278:61
        at packages/form-layout/test/form-layout.test.js:277:59

packages/menu-bar/test/menu-bar.test.js:

 ❌ item components > should teleport the same component to overflow sub-menu and back
      AssertionError: expected undefined to equal <vaadin-context-menu-item theme="menu-bar-item" aria-selected="false" role="menuitem" class="" aria-haspopup="false" tabindex="0"><div style="width: 100px;">Item 3</div></vaadin-context-menu-item>
        at r (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9574:12)
        at n.exports<[3]</t.exports/i.prototype.assert (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:250:12)
        at p (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:1409:11)
        at c (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:7884:24)
        at handleDom (node_modules/@open-wc/semantic-dom-diff/chai-dom-diff.js:102:16)
        at f (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9164:32)
        at packages/menu-bar/test/menu-bar.test.js:577:33

WebKit

packages/date-picker/test/keyboard-input.test.js:

 ❌ keyboard > focused date > should display focused date while overlay focused
      Error: Timeout of 10000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.
        at createTimeoutError (node_modules/@web/test-runner-mocha/dist/autorun.js:1:243204)
        at node_modules/@web/test-runner-mocha/dist/autorun.js:1:257230

packages/upload/test/upload.test.js:

 ❌ upload > Abort Files > should fire `file-remove` and remove from files
      AssertionError: expected false to be true
      + expected - actual
      
      -false
      +true
      
      at r (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9576:21)
      at node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:2[51](https://github.com/vaadin/web-components/runs/5446269040?check_suite_focus=true#step:6:51):8
      at node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:1037:9
      at e (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:7959:33)
      at packages/upload/test/upload.test.js:[52](https://github.com/vaadin/web-components/runs/5446269040?check_suite_focus=true#step:6:52)0:35

packages/date-picker/test/keyboard-navigation.test.js:

 ❌ keyboard navigation > overlay > should focus the closest allowed date with shift pageup when selected date is disabled
      AssertionError: expected Thu, 10 Jun 1999 00:00:00 GMT to deeply equal Sat, 25 Dec 1999 00:00:00 GMT
      + expected - actual
      
      -[Date: 1999-06-10T00:00:00.000Z]
      +[Date: 1999-12-25T00:00:00.000Z]
      
      at qy.<anonymous> (packages/date-picker/test/keyboard-navigation.test.js:397:28)

packages/grid/test/scrolling-mode.test.js:

 ❌ scrolling mode > overflow attribute > update on resize
      Error: Timeout of 10000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.
        at createTimeoutError (node_modules/@web/test-runner-mocha/dist/autorun.js:1:243204)
        at node_modules/@web/test-runner-mocha/dist/autorun.js:1:257230

packages/upload/test/upload.test.js:

 🚧 Browser logs:
      Error: Cannot clear timer: timer created with requestAnimationFrame() but cleared with clearTimeout()
        at clearTimer (node_modules/sinon/pkg/sinon-esm.js:4809:32)
        at packages/upload/src/vaadin-upload.js:606:19
        at progress (packages/upload/test/common.js:89:30)

 ❌ upload > Abort Files > should fire `file-remove` and remove from files
      Error: Error: Cannot clear timer: timer created with requestAnimationFrame() but cleared with clearTimeout() (http://localhost:8000/node_modules/sinon/pkg/sinon-esm.js:4815)
        at node_modules/@web/test-runner-mocha/dist/autorun.js:1:326426

Visual tests

Menu Bar

packages/menu-bar/test/visual/material/menu-bar.test.js:

 ❌ menu-bar > ltr > theme > contained
      Error: Visual diff failed. New screenshot is 0.003600230414746544% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/menu-bar/test/visual/material/screenshots/menu-bar/failed/ltr-contained-diff.png
        at async o.<anonymous> (packages/menu-bar/test/visual/material/menu-bar.test.js:[94](https://github.com/vaadin/web-components/runs/5495396052?check_suite_focus=true#step:5:94):11)

 ❌ menu-bar > rtl > theme > contained
      Error: Visual diff failed. New screenshot is 0.001350086405529[95](https://github.com/vaadin/web-components/runs/5495396052?check_suite_focus=true#step:5:95)4% different.
      See diff for details: /home/runner/work/web-components/web-components/packages/menu-bar/test/visual/material/screenshots/menu-bar/failed/rtl-contained-diff.png
        at async o.<anonymous> (packages/menu-bar/test/visual/material/menu-bar.test.js:94:11)

The text was updated successfully, but these errors were encountered:

vursen · 2022-04-26T10:49:02Z

Here is an updated list of still flaky tests after several improvements have been made (see the referenced PRs).

The tests are not stable in WebKit and Firefox.

menu-bar

packages/menu-bar/test/menu-bar.test.js:

 ❌ item components > should teleport the same component to overflow sub-menu and back
      AssertionError: expected undefined to equal <vaadin-context-menu-item theme="menu-bar-item" aria-selected="false" role="menuitem" class="" aria-haspopup="false" tabindex="0"><div style="width: 100px;">Item 3</div></vaadin-context-menu-item>
        at r (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9574:12)
        at n.exports<[3]</t.exports/i.prototype.assert (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:250:12)
        at p (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:1409:11)
        at c (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:7884:24)
        at handleDom (node_modules/@open-wc/semantic-dom-diff/chai-dom-diff.js:102:16)
        at f (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9164:32)
        at packages/menu-bar/test/menu-bar.test.js:577:33

form-layout

packages/form-layout/test/form-layout.test.js:

 ❌ form layout > container overflow > no spacing on edges (rtl) > should not have spacing on the right edge
      AssertionError: expected 288 to be close to 300 +/- 0.5
        at r (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9574:12)
        at n.exports<[3]</t.exports/i.prototype.assert (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:250:12)
        at S (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:[33](https://github.com/vaadin/web-components/runs/5445517447?check_suite_focus=true#step:6:33)50:9)
        at c (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:7884:24)
        at packages/form-layout/test/form-layout.test.js:278:61
        at packages/form-layout/test/form-layout.test.js:277:59

grid

packages/grid/test/scrolling-mode.test.js:

 ❌ scrolling mode > overflow attribute > update on resize
      Error: Timeout of 10000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.
        at createTimeoutError (node_modules/@web/test-runner-mocha/dist/autorun.js:1:243204)
        at node_modules/@web/test-runner-mocha/dist/autorun.js:1:257230

web-padawan · 2022-08-02T08:28:06Z

Here are some more flaky tests from the recent coverage builds (failed in Chrome):

date-picker

packages/date-picker/test/scroller.test.js:

 ❌ vaadin-infinite-scroller > should reflect currently visible item index as position scrolling up
      Error: Uncaught AssertionError: expected 1.4000000000000057 to be below 1 (http://localhost:8000/node_modules/@esm-bundle/chai/esm/chai.js:[98](https://github.com/vaadin/web-components/runs/7611221226?check_suite_focus=true#step:5:99))
        at t.onerror (node_modules/@web/test-runner-mocha/dist/autorun.js:1:326417)

packages/date-picker/test/keyboard-navigation.test.js:

 ❌ keyboard navigation > overlay > should focus previous year with shift and pageup
      AssertionError: expected Sat, 01 Jan 2000 00:00:00 GMT to deeply equal Fri, 01 Jan 1999 00:00:00 GMT
      + expected - actual
      
      -[Date: 2000-01-01T00:00:00.000Z]
      +[Date: 1999-01-01T00:00:00.000Z]
      
      at qy.<anonymous> (packages/date-picker/test/keyboard-navigation.test.js:274:38)

field-highlighter

packages/field-highlighter/test/user-tags.test.js:

 ❌ user-tags > default > should create user tags for each added user
      AssertionError: expected { entries: [Function: entries],
        keys: [Function: keys],
        values: [Function: values],
        forEach: [Function: forEach],
        length: 0,
        item: [Function: item] } to have a length of 2 but got 0
      + expected - actual
      
      -0
      +2
      
      at o.<anonymous> (packages/field-highlighter/test/user-tags.test.js:55:28)

UPD (27.10.2022): added some more flaky tests

Firefox:

packages/combo-box/test/lazy-loading.test.js:

 ❌ lazy loading > combo-box-light > dataProvider > async > should keep the focused item focused when loading more items
      AssertionError: expected -1 to equal 0
      + expected - actual
      
      --1
      +0

Chrome:

packages/grid/test/scrolling-mode.test.js:

 ❌ scrolling mode > overflow attribute > update on resize
      Error: Timeout of 10000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.
        at createTimeoutError (node_modules/@web/test-runner-mocha/dist/autorun.js:1:243195)
        at Gg._timeoutError (node_modules/@web/test-runner-mocha/dist/autorun.js:1:259145)
        at node_modules/@web/test-runner-mocha/dist/autorun.js:1:257217

WebKit:

packages/component-base/test/virtualizer-item-height.test.js:
 ❌ virtualizer - item height > should adjust the placeholder height
      AssertionError: expected 200 to be below 40
      + expected - actual
      
      -200
      +40
      
      at r (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:9575:6)
      at node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:251:8
      at y (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:1781:10)
      at c (node_modules/@esm-bundle/chai/node_modules/.pnpm/chai@4.3.4/node_modules/chai/chai.js:7884:30)

sissbruecker · 2023-06-23T07:20:06Z

Update from 23.06.2023

Went through the workflow runs of the last few weeks and collected failed workflows. Pretty much none of the tests mentioned above caused a workflow to fail in the last four weeks. However I'm not sure if Github keeps track of workflows that have been manually re-run, the number of failed workflows was suspiciously low compared to what you would expect from having to re-run actions regularly.

Anyway, here are some tests that came up regularly in failed workflows, which have not been mentioned so far:

date-picker

Chrome, Firefox, Webkit

packages/date-picker/test/events.test.js:
	❌ events > has-input-value-changed event > without value > with user input > should be fired when selecting a date with Space
	AssertionError: expected false to be true
	+ expected - actual
	-false
	+true
	at n.<anonymous> (packages/date-picker/test/events.test.js:154:51)

integration

Chrome

integration/tests/grid-tooltip.test.js:
	❌ tooltip > scrolling > should hide the tooltip when starting scrolling
	AssertionError: expected false to be true
	+ expected - actual
	-false
	+true
	at n.<anonymous> (integration/tests/grid-tooltip.test.js:356:35)

Chrome

integration/tests/grid-tooltip.test.js:

 ❌ tooltip > mouse > should set tooltip target on cell mouseenter
      AssertionError: expected <vaadin-grid aria-describedby="vaadin-tooltip-4" style="touch-action: none;">
              <vaadin-grid-column path="firstName"></vaadin-grid-column>
              <vaadin-grid-column path="lastName"></vaadin-grid-column>
              <vaadin-tooltip slot="tooltip"></vaadin-tooltip>
            <vaadin-grid-cell-content slot="vaadin-grid-cell-content-0">First name</vaadin-grid-cell-content><vaadin-grid-cell-content slot="vaadin-grid-cell-content-1">Last name</vaadin-grid-cell-content><vaadin-grid-cell-content slot="vaadin-grid-cell-content-2"></vaadin-grid-cell-content><vaadin-grid-cell-content slot="vaadin-grid-cell-content-3"></vaadin-grid-cell-content><vaadin-grid-cell-content slot="vaadin-grid-cell-content-4"></vaadin-grid-cell-content><vaadin-grid-cell-content slot="vaadin-grid-cell-content-5"></vaadin-grid-cell-content><vaadin-grid-cell-content slot="vaadin-grid-cell-content-6">John</vaadin-grid-cell-content><vaadin-grid-cell-content slot="vaadin-grid-cell-content-7">Doe</vaadin-grid-cell-content><vaadin-grid-cell-content slot="vaadin-grid-cell-content-8">Jane</vaadin-grid-cell-content><vaadin-grid-cell-content slot="vaadin-grid-cell-content-9">Doe</vaadin-grid-cell-content></vaadin-grid> to equal <td id="vaadin-grid-cell-6" role="gridcell" tabindex="0" part="cell body-cell first-column-cell first-row-cell even-row-cell drag-disabled-row-cell drop-disabled-row-cell" first-column="" aria-selected="false" style="width: 100px; flex-grow: 1; order: 10000000;"><slot name="vaadin-grid-cell-content-6"></slot></td>
        at n.<anonymous> (integration/tests/grid-tooltip.test.js:105:[36](https://github.com/vaadin/web-components/actions/runs/5314201424/jobs/9621134295#step:5:37))

web-padawan added the internal label Mar 10, 2022

This was referenced Apr 6, 2022

test: skip flaky grid scrolling mode test for WebKit #3637

Merged

test: make keyboard tests more reliable #3659

Merged

This was referenced Apr 18, 2022

test: disable buttons animation in visual tests #3696

Merged

chore: bump sinon to 13.0.2, fix sinon fake timers warnings #3699

Merged

vursen mentioned this issue May 6, 2022

test: make keyboard tests more reliable #3793

Merged

7 tasks

web-padawan self-assigned this Aug 2, 2022

web-padawan added the flaky-test Test randomly failing in CI label Oct 27, 2022

sissbruecker mentioned this issue Jun 23, 2023

Potential flaky integration test (should set tooltip target on cell mouseenter) #5996

Closed

web-padawan removed their assignment Nov 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Investigate flaky tests in CI #3538

Investigate flaky tests in CI #3538

web-padawan commented Mar 10, 2022 •

edited

vursen commented Apr 26, 2022

web-padawan commented Aug 2, 2022 •

edited

sissbruecker commented Jun 23, 2023 •

edited

Investigate flaky tests in CI #3538

Investigate flaky tests in CI #3538

Comments

web-padawan commented Mar 10, 2022 • edited

Unit tests

Chrome

Firefox

WebKit

Visual tests

Menu Bar

vursen commented Apr 26, 2022

menu-bar

form-layout

grid

web-padawan commented Aug 2, 2022 • edited

date-picker

field-highlighter

sissbruecker commented Jun 23, 2023 • edited

date-picker

integration

web-padawan commented Mar 10, 2022 •

edited

web-padawan commented Aug 2, 2022 •

edited

sissbruecker commented Jun 23, 2023 •

edited