org.eclipse.ui.tests do not work in parallel to other tests

[laeubi]

As discovered here #917 when running org.eclipse.ui.tests in parallel with other tests.

[iloveeclipse]

@laeubi : what is your expectation from UI tests sharing same host, how they should test in parallel?

With permanent focus stealing etc the only way to run them in parallel is to open a dedicated vnc session and run every UI test inside such vnc session.

If maven/tycho doesn't offer such functionality, one don't need to create multiple bugs here, it will never work reliably.

[laeubi]

what is your expectation from UI tests sharing same host, how they should test in parallel?

The same I can use different Eclipse instances in parallel even with other applications on the same host maschine.

With permanent focus stealing etc the only way to run them in parallel is to open a dedicated vnc session and run every UI test inside such vnc session.

If the test is not about focus itself one usually don't need to rely on focus, e.g. SWTBot offers methods that take a control to interact with the UI in a focus independent way, also one might use some kind of mutext-lock if exclusive screen access is required.

If you look here:
https://ci.eclipse.org/platform/job/eclipse.platform.ui/job/PR-917/1/

only a few tests actually fail so there are others that can work perfectly fine so it is not completely hopeless.

[iloveeclipse]

only a few tests actually fail

... in this concrete execution, since it is running in parallel, it will be permanent "russian roulette" which one will fail.

[HeikoKlare]

What about disabling parallel execution only for "affected" bundles, i.e., those may having race conditions when executed in parallel, for now? I see that these bundles require most of the overall execution time of a build, but starting with whatever can be parallelized now makes the CI even a bit faster.

Finding all the test cases with potential race conditions during parallel execution may be some effort, but maybe it's worth taking a look at the tests / perform some full parallel builds and have a tracking issue for all test cases that probably need to be adapted for parallel execution. Then it's easier to decide whether it's feasible and worth the effort.

In addition, there are still several tests that randomly fail already during sequential execution and I would expect that there are further tests that are faulty but still succeed because the fault does not lead to a failure in the current execution environment (which was the case for many platform tests before moving to GitHub Actions). So there will be a mixture of tests that fail because they cannot be run in parallel and tests that fail because the parallel execution reveals that they are faulty and could have also failed during sequential execution. The latter can/will even be the case when running non-UI tests in parallel.

Just for documentation: This build seems to be one that was also executed in parallel mode, in which some other test cases than posted before failed.