chore: improvements on flakiness of UI tests INTELLIJ-107

[himanshusinghs]

Description

Two things that I identified inducing flakiness in our UI tests were:

1. ComboBox getting updated while tests were trying to use them, made the combobox collapse
2. Tests were not able to recover from unexpected state and eventually failed

To address the first concern, we are now updating the combobox only when there is an actual change, otherwise not and that too wrapped in an invokeLater so that existing interactions are finished before performing the update.

Second concern is addressed in two steps:

1. We do not want to interact with the comboboxes, right after they show up on screen because they subscribe to the StateFlow and eventually a state update follows which leads to the disruption in UI interaction. So we have put a short timeout of 1 second to allow for initial state subscription.
2. The state transitions that depend on related are grouped together in retry-able block so that when the tests end up in an expected state they can retry from the start to achieve the expected state. One such example is openRunQueryPopup.

Update

There were two further problems identified for subsequent failures:

1. Gradle project would never load for the very first start of the test run
2. Tests tearDown will fail complaining about writes happening in write safe context

To address the first problem, we have worked up a series of UI steps to always perform for every test that will ensure that gradle project is in sync.

And for the second problem, we realised that the write unsafe error is more of a warning which can safely be ignored. But to be on the correct step here, we have wrapped all the actions now in an invokeLater with explicit modality state.

Note: This PR does not entirely remove flakes from our CI. There is still one source of problem (happening from time to time) which I haven't been able to understand completely. When we open a file, at times it will happen that the toolbar is not visible. There is no gradle sync or indexing happening in the background but still its almost like the IDE is in dumb mode. Not yet entirely sure why but I am happy to look into that as a follow up.

Checklist

• New tests and/or benchmarks are included.
• Documentation is changed or added.
• Changelog is updated accordingly.
• I have signed the MongoDB Contributor License Agreement (https://www.mongodb.com/legal/contributor-agreement).

Open Questions

[github-actions]

Coverage Report

Overall Project	79.03% -0.13%	✅
Files changed	85.47%	✅

File	Coverage
DatabaseComboBox.kt	58.9% -3.27%	✅
DataSourceComboBox.kt	52.57% -3.08%	✅

[kmruiz]

Suggested change

	eventually(1.minutes.toJavaDuration()) {
	step("Selecting DataSource $title in toolbar") {
	step("Selecting DataSource $title in toolbar") {
	eventually(1.minutes.toJavaDuration()) {

Because eventually will retry every a few milliseconds, until (in this case) 1 minute passes, so we will see the step in the logs for every retry.

[kmruiz]

If we are going to use this pattern of eventually/step a lot, maybe we want to have an eventually that receives the step name and does both things? So we encapsulate both calls and it's easier to use.

[himanshusinghs]

Actually in practice I found that step log for every retries were making it more clear about what is happening. I am thinking of step as very detailed and verbose explanation of what is happening in the UI. Agree that we probably should have eventually with a step block. I will work towards that if my earlier reasoning sounds good?

[kmruiz]

It sounds great, if it's possible can we add then the retry number? Like "Selecting DataSource $title: retry N" so it's clear in the logs that we are retrying?

[himanshusinghs]

Added this here

[himanshusinghs]

Added this to address the suggestion about combining eventually and step descriptions.