feat(ci): Add retest workflow. Fixes #12864

[miltalex]

Fixes #12864

Motivation

Due to flaky tests, it would be nice to allow Members to re-run failing workflows.

Modifications

Extend github workflows with retest.yaml. It adds a new workflow run which is triggered on issue_comment:[created,edited]. It requires the comment to contain exactly the /retest. It will execute a sequence of API calls to fetch the latest SHA, the latest run ID and trigger the rerun of the failed jobs.

Note: If the workflow doesn't contain any failed jobs it will fail.

Verification

Used a forked repository https://github.com/miltalex/argo-workflows . Merged the new workflow on main branch and opened PRs (miltalex#1 , miltalex#2 , miltalex#3) in order to:

• Test new comment /retest → Trigger failed jobs
• Test edit comment to retest → Retest workflow is skipped
• Test edit comment to /retest → Trigger failed jobs
• Test new comment without /retest → Retest workflow is skipped
• Test new comment with /retest and no failing jobs → Retest workflow failed due to no failing jobs.

[agilgur5]

• Tested in a forked repository

Would be good to detail where and what you tested.

I know you were testing in miltalex#2 and miltalex#1 as I had been taking a look. It would be good to also add which scenarios you tested in a list. Can see my "Verification" section in #12006 as an example

[agilgur5]

Thanks for all the effort in testing this out and getting it working!

I left some comments below to optimize this a bit. Some token security settings also need adjustments

[agilgur5]

A few small nits, otherwise LGTM

[agilgur5]

Great work, this should be really helpful whenever we have test flakes!

[agilgur5]

Hmm this seemed to have failed on my first try of it in #12732 (comment): https://github.com/argoproj/argo-workflows/actions/runs/8948441124/job/24581552898:

/home/runner/work/_temp/f4de4a64-b8ac-400d-a1ff-042b59deb995.sh: line 2: unexpected EOF while looking for matching `''

[agilgur5]

^That run might actually be too old and may have been GC'd on GH's side. I didn't have a button to rerun the failed jobs myself.

We may want to add some error handling for a more clear error message in some scenarios

[agilgur5]

Hmm failed on a more recent run in #12732 (comment) as well: https://github.com/argoproj/argo-workflows/actions/runs/8949025175/job/24582819712. Same error as above

[miltalex]

I will have a look to amend it