Retrying failed workflow using WorkflowTemplate get stuck in Running phase of the workflow and Pending phase of the node if POD_NAMES=v2

[mikutas]

Summary

What happened/what you expected to happen?

When retrying failed workflow, the old pod is not deleted and the new pod is not created if POD_NAMES=v2.
The UI tells the workflow's phases is Running and the node's phase is Pending. And nothing will change therefrom.

What version of Argo Workflows are you running?

master (d4aa9d1) (includes #7301)

Diagnostics

examples for repro

# templates.yaml
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: workflow-template-whalesay-template
spec:
  entrypoint: whalesay-template
  templates:
  - name: whalesay-template
    inputs:
      parameters:
      - name: message
    container:
      image: docker/whalesay
      command: [cowsay]
      args: ["{{inputs.parameters.message}}"]
---
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: workflow-template-fail-template
spec:
  templates:
  - name: fail-template
    container:
      image: python:alpine3.6
      command: [python, -c]
      # fail with a 100% probability
      args: ["import sys; sys.exit(1)"]

# failure-in-step2.yaml
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: workflow-template-retry-with-steps-
spec:
  entrypoint: retry-with-steps
  templates:
  - name: retry-with-steps
    steps:
    - - name: hello1
        templateRef:
          name: workflow-template-whalesay-template
          template: whalesay-template
        arguments:
          parameters:
          - name: message
            value: "I am hello1"
    - - name: hello2
        templateRef:
          name: workflow-template-fail-template
          template: fail-template

# fish
env POD_NAMES=v2 make start API=true UI=true

kubectl apply -f templates.yaml -n argo
argo submit failure-in-step2.yaml -n argo

What Kubernetes provider are you using?

k3d

What executor are you running?

Emissary? (just ran make argoexec-image)

# Logs from the server:
argo-server | time="2021-12-01T18:44:54.052Z" level=info msg="Deleting pod: workflow-template-retry-with-steps-qzzt8--2273160064"

Actual pod name is workflow-template-retry-with-steps-qzzt8-fail-template-2273160064

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.