Migrate the robustness tests to prow

[serathius]

What would you like to be added?

After the last robustness team meeting it was clear how superior Prow + TestGrid is over GitHub actions.

https://testgrid.k8s.io/sig-etcd-robustness#Summary vs https://github.com/etcd-io/etcd/actions/workflows/robustness-nightly.yaml

Advantages:

• More stable, 7% vs 56% failure rate running the same code
• Testgrid being much better UI to monitor failures plus addition of more advanced features like creating bugs alerting.
• Prow being much better tool to view logs from tests, it parses logs, groups them by test, doesn't crash browser, doesn't fail on downloading compressed logs, doesn't created archives.

TODO:

• tests/robustness: update documentation to reflect Prow migration #18167
• Add etcd robustness presubmit job kubernetes/test-infra#32729
• Remove non-arm robustness tests from github actions.
• Mark robustness presubmits as blocking, when we have confidence in their stability.
• Add arm workers to run robustness periodic

cc @jmhbnz @ivanvc

Why is this needed?

Migration to Prow opens a new chapter for stability and debuggability of robustness test with the goal of making the process more approachable for new contributors.

[henrybear327]

@ArkaSaha30

[ivanvc]

Do we have access to arm nodes in the Prow infra? The last I remember is that we were waiting for them. I don't see any updates regarding this on kubernetes/k8s.io#6102. So, it may be a blocker for the second point.

[serathius]

Not great, but I will not block the migration regardless. Robustness tests only bring value if there is someone willing to review them. With Prow being much better, no-one will be willing to review arm robustness failures.

[ivanvc]

I can see two options: pause running robustness for the ARM architecture (not ideal) or keep ARM tests running on GitHub actions.

I don't see much activity in kubernetes/k8s.io#6102. Who or where would be a good place to ask for a status update/ETA for ARM nodegroups?

[jmhbnz]

Hi @upodroid - We spoke at KubeCon EU Paris about a dedicated arm64 cluster for prow. Can you please provide an update on the timeline for it being available?

[serathius]

I can see two options: pause running robustness for the ARM architecture (not ideal) or keep ARM tests running on GitHub actions.

I was thinking about the second option, however due to sub-par user experience I expect it would be equal the first one.

[ivanvc]

Discussed on Slack with Arka, we'll be working on the following at the moment:

• Migrate robustness documentation to Prow https://github.com/etcd-io/etcd/tree/main/tests/robustness
• Add robustness tests presubmits

/assign @ArkaSaha30 @ivanvc

[k8s-ci-robot]

@ivanvc: GitHub didn't allow me to assign the following users: ArkaSaha30.

Note that only etcd-io members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

Discussed on Slack with Arka, we'll be working on the following at the moment:

• Migrate robustness documentation to Prow https://github.com/etcd-io/etcd/tree/main/tests/robustness
• Add robustness tests presubmits

/assign @ArkaSaha30 @ivanvc

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ArkaSaha30]

/assign

[ArkaSaha30]

Currently, the robustness tests on Github Actions run only on main or PRs to main. Do we need to run it on release-3.5 and release-3.4?
The existing robustness periodic and presubmit can be configured to handle all the 3 branches.

[serathius]

There are no robustness test on other branches beside main. We develop and run robustness test from main branch and validate binaries build from older branches.

[ivanvc]

We have finished the first and the third tasks. When would you think is a good time to remove the GitHub action @serathius?

We can't move forward with the second, as we don't have a timeline on when ARM runners are going to be available.

[serathius]

We have finished the first and the third tasks. When would you think is a good time to remove the GitHub action @serathius?

We can keep arm64 on Github actions to not block on it.

[ivanvc]

@ArkaSaha30, can you help with

Remove non-arm robustness tests from github actions.?

Thanks.