feat: delete pods in parallel to speed up retryworkflow

[shuangkun]

I have a big worklfow. When it failed and retry, it will be slowly. Latency increases with the number of pods. Always letting my automated system（interact with argo server use https) time out.

time="2023-12-26T13:02:01.560Z" level=info duration=1m18.730806898s method=PUT path=/api/v1/workflows/default/catch-workflow-8c48s/retry size=194 status=409

I found that the main time spent was deleting pods.

time="2023-12-26T13:02:00.126Z" level=info msg="Deleting pod" podDeleted=catch-workflow-8c48s-read-ligands-from-vswdb-op-bmvou-9dzxr-2538960512
time="2023-12-26T13:02:00.328Z" level=info msg="Deleting pod" podDeleted=catch-workflow-8c48s-read-ligands-from-vswdb-op-bmvou-9dzxr-2824246174
time="2023-12-26T13:02:00.527Z" level=info msg="Deleting pod" podDeleted=catch-workflow-8c48s-read-ligands-from-vswdb-op-bmvou-9dzxr-3817503941
time="2023-12-26T13:02:00.731Z" level=info msg="Deleting pod" podDeleted=catch-workflow-8c48s-read-ligands-from-vswdb-op-bmvou-9dzxr-1699455227
time="2023-12-26T13:02:00.925Z" level=info msg="Deleting pod" podDeleted=catch-workflow-8c48s-read-ligands-from-vswdb-op-bmvou-9dzxr-4168132452
time="2023-12-26T13:02:01.133Z" level=info msg="Deleting pod" podDeleted=catch-workflow-8c48s-read-ligands-from-vswdb-op-bmvou-9dzxr-1982889595

So I want make deleting pods parallel.

Motivation

Modifications

Verification

[juliev0]

I had no idea the Server deleted pods..that's surprising...anyway, LGTM!

[shuangkun]

I had no idea the Server deleted pods..that's surprising...anyway, LGTM!

Thank you so much!

[agilgur5]

I had no idea the Server deleted pods..that's surprising...

@juliev0 that makes at least 3 of us as neither Terry nor I expected that either 😅 😕

I mentioned this in #12105 (comment), I don't think this logic should belong in the Server as it breaks the separation of duties between Controller and Server right now; this makes the Server necessary for certain operations (retry in this case). Outside of this functionality, the Server and Controller are independent, with the Server mainly signaling to the Controller (via labels) and not strictly required for any operations.

Bala agreed with me there as well and multiple people have been surprised by the current behavior, so I do think we have consensus that this should be refactored

[juliev0]

I had no idea the Server deleted pods..that's surprising...

@juliev0 that makes at least 3 of us as neither Terry nor I expected that either 😅 😕

I mentioned this in #12105 (comment), I don't think this logic should belong in the Server as it breaks the separation of duties between Controller and Server right now; this makes the Server necessary for certain operations (retry in this case). Outside of this functionality, the Server and Controller are independent, with the Server mainly signaling to the Controller (via labels) and not strictly required for any operations.

Bala agreed with me there as well and multiple people have been surprised by the current behavior, so I do think we have consensus that this should be refactored

I completely agree and wanted to make the same judgment actually.

[agilgur5]

I'll file a tracking issue for it for now as it does require some careful refactoring. EDIT: Created #12538

[shuangkun]

Yes, I agree. I also had some doubts at the time, but at that time I was just focused on solving customer problems.
If there's anything I can do, I'm willing to do it.

[agilgur5]

Feel free to take on the issue if you'd like, otherwise I might get to it eventually (it's not the highest priority)