continue on fail doesn't work with DAG

[vmanucharyan]

Checklist:

• I've included the version.
• I've included reproduction steps.
• I've included the workflow YAML.
• I've included the logs.

What happened:

The following workflow finishes after task B fails:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: continue-on-failed-dag
  labels:
    argo-e2e: true
spec:
  entrypoint: workflow-ignore
  parallelism: 2
  templates:
    - name: workflow-ignore
      dag:
        failFast: false
        tasks:
          - name: A
            template: whalesay
          - name: B
            template: boom
            continueOn:
              failed: true
            dependencies:
              - A
          - name: C
            template: whalesay
            dependencies:
              - A
          - name: D
            template: whalesay
            dependencies:
              - B
              - C

    - name: boom
      dag:
        tasks:
          - name: B-1
            template: whalesplosion

    - name: whalesay
      container:
        imagePullPolicy: IfNotPresent
        image: docker/whalesay:latest

    - name: whalesplosion
      container:
        imagePullPolicy: IfNotPresent
        image: docker/whalesay:latest
        command: ["sh", "-c", "sleep 10; exit 1"]

What you expected to happen:

Workflow continues and runs task D.

How to reproduce it (as minimally and precisely as possible):

Run the above workflow

Anything else we need to know?:

Environment:

• Argo version:

$ argo version
argo: v2.7.0
  BuildDate: 2020-03-31T23:35:43Z
  GitCommit: 4d1175eb68f6578ed5d599f877be9b4855d33ce9
  GitTreeState: clean
  GitTag: v2.7.0
  GoVersion: go1.13.4
  Compiler: gc
  Platform: darwin/amd64

• Kubernetes version :

$ kubectl version -o yaml
clientVersion:
  buildDate: "2019-08-19T12:36:28Z"
  compiler: gc
  gitCommit: 2d3c76f9091b6bec110a5e63777c332469e0cba2
  gitTreeState: clean
  gitVersion: v1.15.3
  goVersion: go1.12.9
  major: "1"
  minor: "15"
  platform: darwin/amd64
serverVersion:
  buildDate: "2019-10-15T19:07:57Z"
  compiler: gc
  gitCommit: 20c265fef0741dd71a66480e35bd69f18351daea
  gitTreeState: clean
  gitVersion: v1.15.5
  goVersion: go1.12.10
  major: "1"
  minor: "15"
  platform: linux/amd64

Other debugging information (if applicable):

• workflow result:

argo get <workflowname>

• executor logs:

kubectl logs <failedpodname> -c init
kubectl logs <failedpodname> -c wait

• workflow-controller logs:

kubectl logs -n argo $(kubectl get pods -l app=workflow-controller -n argo -o name)

Logs

argo get <workflowname>
kubectl logs <failedpodname> -c init
kubectl logs <failedpodname> -c wait
kubectl logs -n argo $(kubectl get pods -l app=workflow-controller -n argo -o name)

---

Message from the maintainers:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.