tech-debt(schema-engine-tests): Reduce vitess flakeyness

[miguelff]

Vitess DDL is eventually consistent. When DDL is applied, there's a delay until that's reflected on vtgate. That delay is equal to the TABLET_REFRESH_INTERVAL setting in vttestserver + the time it takes for the refresh operation to complete.

What I did to fix this was:

• Add an optional field to the DBUnderTestabstraction, called max_ddl_refresh_delay which is None for all databaes where DDL is applied atomically, and has a value for vitess of 1000 ms.
• I provided access to that setting in SchemaPush, and in case it had a value, make the schema push operation sleep after being applied.

Effects:

• Experiments yield that 1000 ms seems big enough to remove the flakeyness. I tried with lower wait times, such as 100, 200 and 500ms but some times one of the tests failed.
• The wait time increased substantially for this check, from 2m to 8m. Still this is less than the test feedback from other checks, so it's acceptable.
• In the future if the number of schema tests in vitess grew substantially, the runtime will grow by 1s times the number of schema loads. This might be unnacceptable.
• A longer term solution would be to:
  1. Do not apply the sleep strategy on all tests, and instead run tests normally.
  2. In case the test fails retry with some sleep time, up to x (eg. 3) times.
• I don't have the appetite to implement the long-term solution now, the reason is that connector_test macros used in schema tests require a big overhaul, which takes more time than what's now considered a priority for solving the flakeyness, compared to other team priorities.

[codspeed-hq]

CodSpeed Performance Report

Merging #4492 will not alter performance

_{Comparing fix-vitess-flakes (9a19e30) with main (e08be2e)}

Summary

✅ 11 untouched benchmarks

[miguelff]

Unrelated, this test depended on source code positioning of the schema push file, as it is displayed in a backtrace. I think that's wrong and changed it.

[janpio]

✅ on the description of what you implemented

[Weakky]

The button Miguel, click the button 👇🔥