rustdoc-gui tests sporadically failing #93784

ehuss · 2022-02-08T17:42:12Z

The src/test/rustdoc-gui tests have been sporadically failing on CI:

These may have started with #93597
cc @GuillaumeGomez

There are various different errors:

/checkout/src/test/rustdoc-gui/search-tab-selection-if-current-is-empty.goml search-tab-selection-if-current-is-empty... FAILED
[ERROR] (line 6) TimeoutError: waiting for selector "#titles" failed: timeout 30000ms exceeded: for command `wait-for: "#titles"`

/checkout/src/test/rustdoc-gui/search-result-display.goml search-result-display... FAILED
[ERROR] (line 6) TimeoutError: waiting for selector "#titles" failed: timeout 30000ms exceeded: for command `wait-for: "#titles"`

/checkout/src/test/rustdoc-gui/src-font-size.goml An exception occured: read ECONNRESET
== STACKTRACE ==
Error
    at innerRunTestCode (/node-v14.4.0-linux-x64/lib/node_modules/browser-ui-test/src/index.js:468:16)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async main (/checkout/src/tools/rustdoc-gui/tester.js:226:13)

/checkout/src/test/rustdoc-gui/mobile.goml An exception occured: Failed to launch the browser process!
Inconsistency detected by ld.so: dl-tls.c: 493: _dl_allocate_tls_init: Assertion `listp->slotinfo[cnt].gen <= GL(dl_tls_generation)' failed!


TROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/master/docs/troubleshooting.md
== STACKTRACE ==
Error
Error
    at innerRunTestCode (/node-v14.4.0-linux-x64/lib/node_modules/browser-ui-test/src/index.js:468:16)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)

The text was updated successfully, but these errors were encountered:

GuillaumeGomez · 2022-02-08T18:21:08Z

That is concerning... If it happens too many times, let's revert it.

…st, r=notriddle Update browser UI test version Might help a bit with rust-lang#93784 (still need to figure out what's going on with the driver). r? `@notriddle`

…st, r=notriddle Update browser UI test version Might help a bit with rust-lang#93784 (still need to figure out what's going on with the driver). r? ``@notriddle``

…st, r=notriddle Update browser UI test version Might help a bit with rust-lang#93784 (still need to figure out what's going on with the driver). r? ```@notriddle```

GuillaumeGomez · 2022-03-07T15:40:42Z

There is also #94417.

nikic · 2022-03-09T19:19:46Z

It looks like rustdoc-gui tests are now consistently failing with:

2022-03-09T18:18:21.7017663Z /checkout/src/test/rustdoc-gui/mobile.goml mobile... FAILED
2022-03-09T18:18:21.7018773Z [ERROR] (line 27) Error: Evaluation failed: assert didn't fail: for command compare-elements-position-near-false: ("#preferred-light-theme .setting-name", "#preferred-light-theme .choice", {"y": 16})

GuillaumeGomez · 2022-03-09T19:43:34Z

I'm checking locally.

…st, r=notriddle Update browser-ui-test version It should help with rust-lang#93784 (the important PR helping in this regard is GuillaumeGomez/browser-UI-test#273). r? `@notriddle`

GuillaumeGomez · 2022-03-28T12:53:34Z

@ehuss Is it still the case?

ehuss · 2022-03-29T02:38:22Z

It seems to still be happening regularly:

2021-11-23T17:42:34 https://github.com/rust-lang-ci/rust/actions/runs/1495820343: [ERROR] (line 4) TimeoutError: waiting for selector
2021-12-14T00:47:00 https://github.com/rust-lang-ci/rust/actions/runs/1575507908: [ERROR] (line 17) TimeoutError: waiting for selector
2022-01-22T19:25:41 https://github.com/rust-lang-ci/rust/actions/runs/1733757860: /checkout/src/test/rustdoc-gui/search-result-description.goml An exception occured: Failed to launch the browser process!
2022-01-26T19:44:04 https://github.com/rust-lang-ci/rust/actions/runs/1752548375: /checkout/src/test/rustdoc-gui/impl-default-expansion.goml An exception occured: Failed to launch the browser process!
2022-01-31T08:11:10 https://github.com/rust-lang-ci/rust/actions/runs/1771336059: [ERROR] (line 6) TimeoutError: waiting for selector
2022-02-04T05:46:21 https://github.com/rust-lang-ci/rust/actions/runs/1793315545: /checkout/src/test/rustdoc-gui/trait-sidebar-item-order.goml An exception occured: Failed to launch the browser process!
2022-02-05T22:03:59 https://github.com/rust-lang-ci/rust/actions/runs/1800584018: /checkout/src/test/rustdoc-gui/src-font-size.goml An exception occured: read ECONNRESET
2022-02-07T19:15:06 https://github.com/rust-lang-ci/rust/actions/runs/1808171078: [ERROR] (line 6) TimeoutError: waiting for selector
2022-02-08T21:52:54 https://github.com/rust-lang-ci/rust/actions/runs/1814714051: [ERROR] (line 6) TimeoutError: waiting for selector
2022-02-08T22:32:08 https://github.com/rust-lang-ci/rust/actions/runs/1814879899: /checkout/src/test/rustdoc-gui/implementors.goml An exception occured: Failed to launch the browser process!
2022-02-14T11:19:42 https://github.com/rust-lang-ci/rust/actions/runs/1840526472: /checkout/src/test/rustdoc-gui/search-filter.goml An exception occured: Failed to launch the browser process!
2022-02-21T07:21:35 https://github.com/rust-lang-ci/rust/actions/runs/1874627943: [ERROR] (line 6) TimeoutError: waiting for selector
2022-02-22T02:47:16 https://github.com/rust-lang-ci/rust/actions/runs/1879224701: /checkout/src/test/rustdoc-gui/search-result-description.goml An exception occured: Failed to launch the browser process!
2022-03-05T14:31:20 https://github.com/rust-lang-ci/rust/actions/runs/1938283620: /checkout/src/test/rustdoc-gui/toggled-open-implementations.goml An exception occured: Failed to launch the browser process!
2022-03-07T21:18:49 https://github.com/rust-lang-ci/rust/actions/runs/1947870857: [ERROR] (line 6) TimeoutError: waiting for selector
2022-03-09T06:18:46 https://github.com/rust-lang-ci/rust/actions/runs/1955562234: [ERROR] (line 6) TimeoutError: waiting for selector
2022-03-09T18:18:21 https://github.com/rust-lang-ci/rust/actions/runs/1958791332: /checkout/src/test/rustdoc-gui/toggle-docs-mobile.goml An exception occured: Failed to launch the browser process!
2022-03-09T18:49:35 https://github.com/rust-lang-ci/rust/actions/runs/1958984691: /checkout/src/test/rustdoc-gui/jump-to-def-background.goml An exception occured: Failed to launch the browser process!
2022-03-10T05:52:57 https://github.com/rust-lang-ci/rust/actions/runs/1961326088: /checkout/src/test/rustdoc-gui/toggle-docs.goml An exception occured: Failed to launch the browser process!
2022-03-20T08:34:38 https://github.com/rust-lang-ci/rust/actions/runs/2011183164: [ERROR] (line 6) TimeoutError: waiting for selector
2022-03-21T18:32:43 https://github.com/rust-lang-ci/rust/actions/runs/2017851280: [ERROR] (line 6) TimeoutError: waiting for selector
2022-03-24T09:11:02 https://github.com/rust-lang-ci/rust/actions/runs/2033118524: [ERROR] (line 6) TimeoutError: waiting for selector
2022-03-27T12:24:59 https://github.com/rust-lang-ci/rust/actions/runs/2047654016: [ERROR] (line 6) TimeoutError: waiting for selector
2022-03-27T15:02:11 https://github.com/rust-lang-ci/rust/actions/runs/2048045097: [ERROR] (line 6) TimeoutError: waiting for selector
2022-03-28T05:47:21 https://github.com/rust-lang-ci/rust/actions/runs/2050300242: [ERROR] (line 6) TimeoutError: waiting for selector

GuillaumeGomez · 2022-03-29T09:18:23Z

The situation seems to have been much better since #95013: there is just one error to fix now. \o/

…t, r=notriddle Fix last rustdoc-gui spurious test This should the last spurious failing GUI test from rust-lang#93784. r? `@notriddle`

…t, r=notriddle Fix last rustdoc-gui spurious test This should the last spurious failing GUI test from rust-lang#93784. r? ``@notriddle``

ehuss · 2022-04-03T20:30:15Z

@GuillaumeGomez Just FYI, there's still timeouts happening after #95470:

/checkout/src/test/rustdoc-gui/search-filter.goml search-filter... FAILED
Error:  (line 6) TimeoutError: waiting for selector "#titles" failed: timeout 30000ms exceeded: for command `wait-for: "#titles"`

https://github.com/rust-lang-ci/rust/actions/runs/2086146075

GuillaumeGomez · 2022-04-03T20:48:03Z

Thanks! Then I'll need for another way to wait for the search results.

GuillaumeGomez · 2022-05-23T14:20:38Z

Nevermind, seems like new bugs appeared recently. I'll take another look.

GuillaumeGomez · 2022-10-06T20:08:32Z

A new spurious failure appeared in #102744.

Attempt rust-lang#93784

…est, r=notriddle Update browser-ui-test version to fix some flaky tests Part of rust-lang#93784. It should fix the new spurious failure found in rust-lang#102744. r? `@notriddle`

…est, r=notriddle Update browser-ui-test version to fix some flaky tests Part of rust-lang#93784. It should fix the new spurious failure found in rust-lang#102744. r? ``@notriddle``

notriddle · 2022-11-21T16:02:58Z

Another flaky GUI test #104647 (comment)

RalfJung · 2022-11-21T19:38:12Z

That was

/checkout/src/test/rustdoc-gui/code-sidebar-toggle.goml code-sidebar-toggle... FAILED

notriddle · 2022-12-01T14:53:43Z

#105125 (comment)

GuillaumeGomez · 2022-12-17T08:22:58Z

It seems to be mostly:

/checkout/src/test/rustdoc-gui/basic-code.goml basic-code... FAILED
Error:  (line 3) Error: Execution context was destroyed, most likely because of a navigation.: for command `assert-count: (".src-line-numbers", 1)`

GuillaumeGomez · 2022-12-17T09:46:15Z

So about "execution context was destroyed": it seems that when puppeteer gets a element from the DOM, it can get destroyed in the meantime (for whatever reason), resulting in the current issue. A solution would be to instead move everything directly into the browser context. I'll check that in the next weeks to see if it improves the situation.

…est, r=notriddle Update browser-ui-test version to reduce GUI tests flakyness Part of rust-lang#93784. r? `@notriddle`

GuillaumeGomez · 2023-01-02T09:57:41Z

Same spurious failure in #106315. So my fix didn't solve this issue unfortunately...

…hanism, r=Mark-Simulacrum Add retry mechanism for rustdoc GUI tests to reduce flakyness Part of rust-lang#93784. I added 3 retries for failing GUI tests. An important note: if more than half of total tests fail, I don't retry because it's very likely not flakyness anymore at this point but a missing update after changes.

…tion-bug, r=notriddle Fix GUI test navigation bug Following GuillaumeGomez/browser-UI-test#438, we can now remove this `wait-for`. Should help with rust-lang#93784. r? `@notriddle`

…tion-bug, r=notriddle Fix GUI test navigation bug Following GuillaumeGomez/browser-UI-test#438, we can now remove this `wait-for`. Should help with rust-lang#93784. r? ``@notriddle``

ehuss added T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) labels Feb 8, 2022

ehuss mentioned this issue Feb 8, 2022

Rollup of 7 pull requests #93778

Merged

lqd mentioned this issue Feb 8, 2022

Fix regression from lazy opaque types #93783

Merged

GuillaumeGomez mentioned this issue Feb 15, 2022

Update browser UI test version #94027

Merged

wesleywiser mentioned this issue Mar 4, 2022

Stabilize const_fn_fn_ptr_basics, const_fn_trait_bound, and const_impl_trait #93827

Merged

GuillaumeGomez mentioned this issue Mar 9, 2022

Rollup of 7 pull requests #94774

Closed

nikic mentioned this issue Mar 10, 2022

Update LLVM submodule #94764

Merged

GuillaumeGomez mentioned this issue Mar 10, 2022

Update browser-ui-test version #94811

Merged

GuillaumeGomez mentioned this issue Mar 30, 2022

Fix last rustdoc-gui spurious test #95470

Merged

tmiasko mentioned this issue Apr 3, 2022

Don't emit non-asm contents error for naked function composed of errors #95553

Merged

GuillaumeGomez mentioned this issue Apr 5, 2022

rustdoc: Fix events not being registered when navigating browser history #95674

Closed

ehuss mentioned this issue Apr 16, 2022

Update mdbook #96088

Merged

nikic mentioned this issue Apr 18, 2022

Respect ranlib specified for target during LLVM build #96080

Merged

tmiasko mentioned this issue May 3, 2022

Use reverse postorder in non_ssa_locals #96601

Merged

GuillaumeGomez reopened this May 23, 2022

GuillaumeGomez mentioned this issue Oct 6, 2022

rustdoc: remove unused CSS .content .item-list #102744

Merged

notriddle mentioned this issue Oct 9, 2022

Rollup of 8 pull requests #102822

Merged

notriddle added a commit to notriddle/rust that referenced this issue Oct 11, 2022

rustdoc: wait for line numbers to show up before counting them

5a0ae6b

Attempt rust-lang#93784

GuillaumeGomez mentioned this issue Oct 14, 2022

Update browser-ui-test version to fix some flaky tests #103047

Merged

notriddle mentioned this issue Oct 14, 2022

Clean up rust-logo rustdoc GUI test #103054

Merged

notriddle mentioned this issue Nov 21, 2022

enable fuzzy_provenance_casts lint in liballoc and libstd #104647

Merged

This was referenced Dec 17, 2022

Rollup of 7 pull requests #104702

Merged

Check AArch64 branch-protection earlier in the pipeline. #105421

Merged

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Dec 18, 2022

Rollup merge of rust-lang#105863 - GuillaumeGomez:update-browser-ui-t…

be48d21

…est, r=notriddle Update browser-ui-test version to reduce GUI tests flakyness Part of rust-lang#93784. r? `@notriddle`

nikic mentioned this issue Jan 2, 2023

Add codegen test for issue 103840 #106272

Merged

GuillaumeGomez mentioned this issue Jan 4, 2023

Add retry mechanism for rustdoc GUI tests to reduce flakyness #106449

Merged

ehuss mentioned this issue Jan 6, 2023

Add vendor to Fuchsia's target triple #106429

Merged

GuillaumeGomez mentioned this issue Feb 24, 2023

Fix GUI test navigation bug #108412

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustdoc-gui tests sporadically failing #93784

rustdoc-gui tests sporadically failing #93784

ehuss commented Feb 8, 2022 •

edited

Loading

GuillaumeGomez commented Feb 8, 2022

GuillaumeGomez commented Mar 7, 2022

nikic commented Mar 9, 2022

GuillaumeGomez commented Mar 9, 2022

GuillaumeGomez commented Mar 28, 2022

ehuss commented Mar 29, 2022

GuillaumeGomez commented Mar 29, 2022

ehuss commented Apr 3, 2022

GuillaumeGomez commented Apr 3, 2022

GuillaumeGomez commented May 23, 2022

GuillaumeGomez commented Oct 6, 2022

notriddle commented Nov 21, 2022

RalfJung commented Nov 21, 2022

notriddle commented Dec 1, 2022

GuillaumeGomez commented Dec 17, 2022

GuillaumeGomez commented Dec 17, 2022

GuillaumeGomez commented Jan 2, 2023

rustdoc-gui tests sporadically failing #93784

rustdoc-gui tests sporadically failing #93784

Comments

ehuss commented Feb 8, 2022 • edited Loading

GuillaumeGomez commented Feb 8, 2022

GuillaumeGomez commented Mar 7, 2022

nikic commented Mar 9, 2022

GuillaumeGomez commented Mar 9, 2022

GuillaumeGomez commented Mar 28, 2022

ehuss commented Mar 29, 2022

GuillaumeGomez commented Mar 29, 2022

ehuss commented Apr 3, 2022

GuillaumeGomez commented Apr 3, 2022

GuillaumeGomez commented May 23, 2022

GuillaumeGomez commented Oct 6, 2022

notriddle commented Nov 21, 2022

RalfJung commented Nov 21, 2022

notriddle commented Dec 1, 2022

GuillaumeGomez commented Dec 17, 2022

GuillaumeGomez commented Dec 17, 2022

GuillaumeGomez commented Jan 2, 2023

ehuss commented Feb 8, 2022 •

edited

Loading