[build] Split integration tests into 18 buckets

[FelixGV]

The buckets are controlled by a map defined in:

internal/venice-test-common/build.gradle

The build can now also dynamically generate the GitHub CI workflow file according to the content of the build, by running:

./gradlew generateGHCI

This reduces the wall-clock time of the build to roughly half an hour.

How was this PR tested?

Example run: https://github.com/FelixGV/venice/actions/runs/3877390120/usage

Does this PR introduce any user-facing changes?

• No. You can skip the rest of this section.
• Yes. Make sure to explain your proposed changes and call out the behavior change.

[adamxchen]

Thank you, Felix! Left one comment. most of integ tests are finished within 10 mins!

[sixpluszero]

I have a general question about this effort. I have a feeling that it seems like the whole build is less performant compared to our original Venice CI, is that true? I do see my newly introduced heavy integration test could not reach ideal state in time in this env (before your diff it seems to fail very often, with your diff I see latest run in suite Z it failed once then complete), but it was constantly passing in Jenkins CI. Is my understanding correct?

[FelixGV]

I have a general question about this effort. I have a feeling that it seems like the whole build is less performant compared to our original Venice CI, is that true? I do see my newly introduced heavy integration test could not reach ideal state in time in this env (before your diff it seems to fail very often, with your diff I see latest run in suite Z it failed once then complete), but it was constantly passing in Jenkins CI. Is my understanding correct?

Yeah so... there's no doubt that the CI we use internally is way more powerful than the cloud hosts we get for GH Actions. That is why the build on GH takes >4h while the internal one takes ~1h.

Regarding flakiness, it is possible that the two environments behave differently. I haven't looked into that yet, so IDK.

[adamxchen]

I have a general question about this effort. I have a feeling that it seems like the whole build is less performant compared to our original Venice CI, is that true? I do see my newly introduced heavy integration test could not reach ideal state in time in this env (before your diff it seems to fail very often, with your diff I see latest run in suite Z it failed once then complete), but it was constantly passing in Jenkins CI. Is my understanding correct?

In case you are wondering, there's the spec on Github runner hosts. Definitely not as powerful as internal CI hosts.

[adamxchen]

Thank you!!! 🚀

[FelixGV]

Thanks for shipping it @adamxchen, but I found an issue... will fix it and update the PR. Stay tuned!

[FelixGV]

Ok, the broken thing is fixed, and I did some more tweaking of the balance between buckets... I think this is good to go. Thanks!

[adamxchen]

🚀