QuickAccessDialogTest.testPreviousChoicesAvailableForExtension fails

[trancexpress]

See:

• https://download.eclipse.org/eclipse/downloads/drops4/I20260514-1800/testresults/html/org.eclipse.ui.tests_ep440I-unit-macosx-x86_64-java21_macosx.cocoa.x86_64_21.html
• https://download.eclipse.org/eclipse/downloads/drops4/I20260515-1811/testresults/html/org.eclipse.ui.tests_ep440I-unit-macosx-x86_64-java21_macosx.cocoa.x86_64_21.html
• https://download.eclipse.org/eclipse/downloads/drops4/I20260516-1800/testresults/html/org.eclipse.ui.tests_ep440I-unit-macosx-x86_64-java21_macosx.cocoa.x86_64_21.html

All are the same fail:

expected: <true> but was: <false>

org.opentest4j.AssertionFailedError: expected: <true> but was: <false>
at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:31)
at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:183)
at org.eclipse.ui.tests.quickaccess.QuickAccessDialogTest.testPreviousChoicesAvailableForExtension(QuickAccessDialogTest.java:302)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) 

In all cases the test failed after ~8 seconds, i.e. the timeout occurred here:

		assertTrue(DisplayHelper.waitForCondition(text.getDisplay(), TIMEOUT,
				() -> dialogContains(dialog, TestQuickAccessComputer.TEST_QUICK_ACCESS_PROPOSAL_LABEL)));

The build before the last 2 fails has no fail:

https://download.eclipse.org/eclipse/downloads/drops4/I20260513-1800/reports.html#tests

[trancexpress]

I'm not seeing recent changes in platform, platform UI or SWT that would explain a stable fail in the last 2 builds. Maybe its just not stable and we got "lucky" with 2 consecutive fails.

[trancexpress]

@HeikoKlare we seem to have another stable MacOS fail during I-builds. Do we have platform UI MacOS committers who can look into such issues? It doesn't seem feasible to keep disabling tests for MacOS.

[trancexpress]

Looks like the dialog is empty when the test fails:

Unexpected dialog contents: [] ==> expected: <true> but was: <false>

org.opentest4j.AssertionFailedError: Unexpected dialog contents: [] ==> expected: <true> but was: <false>
at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:214)
at org.eclipse.ui.tests.quickaccess.QuickAccessDialogTest.testPreviousChoicesAvailableForExtension(QuickAccessDialogTest.java:302)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

At least when running locally, there are 2 entries:

Test Quick Access Computer | Test Quick Access Proposal |
Help | Search 'test quick access proposal' in Help |

[HeikoKlare]

Note that the I-Build on May 12 (I20260512-0925) also had that test failure. The build on the day after didn't show the failure. So it seems to be sporadic (but very often).

There is a correlation between the failure showing up and the restart/reconfiguration of the according Jenkins agent: https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/7455
Difficult to say if there is any causality but we do not seem to see any other reasonable correlations.

Do we have platform UI MacOS committers who can look into such issues? It doesn't seem feasible to keep disabling tests for MacOS.

I agree that we should not just silence one test after another. I don't know about anyone specific who might have a look at such macOS-related issues. @BeckerWdf would anyone from your side be able to do that? But actually I rather see the difficulty in analyzing/reproducing the failure than the need to have macOS-specific knowledge.

[HeikoKlare]

It's not an issue with the macOS tests. The test also sporadically fails on Windows:
https://github.com/eclipse-platform/eclipse.platform.ui/pull/4002/checks?check_run_id=76558469453

With that, it looks more like a regression in production to me now.

Last change to the dialog was done recently via:

• QuickAccessDialog: render filter as native search field #3916

@vogella could it be that your change produced this regression?

[vogella]

I would be surprised but feel free to revert the change and see

[HeikoKlare]

I agree that there is nothing obviously suspicious in that change.

I just noticed that you also touched the test itself (migrated to JUnit 5):

• Migrate QuickAccessDialogTest to JUnit 5 #3911

That's more likely the modification when it started to become (more) flaky.

I executed the test as repeated test locally for hundreds of times without any failure. But it starts to fail with the failure as reported when I click somewhere else once the quick access dialog pops up (such that it loses focus and closes). So maybe it's about the dialog losing focus and thus closing itself during test execution.

[trancexpress]

So maybe it's about the dialog losing focus and thus closing itself during test execution.

We can likely add some SWT listeners to the dialogs shell and record focus in/out and other such events?

[vogella]

We have seen this flakyness in the QuickAccess tests also in the past see #3042 and #80