Taking time to mark task as succeeded

[cscetbon]

What happened:
Workflow duration is about 10 minutes when retryStrategy is used with backoff. When backoff is not used it takes a few seconds (see debug information)

What you expected to happen:
Workflow duration should be of a few seconds

How to reproduce it (as minimally and precisely as possible):
Use

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: cyril-tests-
spec:
  entrypoint: cyril-template
  templates:
  - name: cyril-template
    steps:
    - - name: my-template
        template: my-template

  - name: my-template
    # retryStrategy:
    #   limit: 5
    #   backoff:
    #     duration: 5m
    #     factor: 2
    #     maxDuration: 30m
    container:
      image: alpine
      command: [sleep, 1]

Submit it with and without the ReplyStrategy and Backoff options.

Anything else we need to know?:

Environment:

• Argo version:

argo: v2.6.1
  BuildDate: 2020-03-04T21:25:55Z
  GitCommit: 842739d7831cc5b417c4f524ed85288408a32bbf
  GitTreeState: clean
  GitTag: v2.6.1
  GoVersion: go1.13.4
  Compiler: gc
  Platform: darwin/amd64

• Kubernetes version :

clientVersion:
  buildDate: "2020-02-13T18:08:14Z"
  compiler: gc
  gitCommit: 06ad960bfd03b39c8310aaf92d1e7c12ce618213
  gitTreeState: clean
  gitVersion: v1.17.3
  goVersion: go1.13.8
  major: "1"
  minor: "17"
  platform: darwin/amd64
serverVersion:
  buildDate: "2019-11-13T11:11:50Z"
  compiler: gc
  gitCommit: 7015f71e75f670eb9e7ebd4b5749639d42e20079
  gitTreeState: clean
  gitVersion: v1.15.6
  goVersion: go1.12.12
  major: "1"
  minor: "15"
  platform: linux/amd64```

**Other debugging information (if applicable)**:
- workflow result:

With RetryStrategy

$ argo get cyril-tests-fv59x
Name: cyril-tests-fv59x
Namespace: reporting
ServiceAccount: default
Status: Succeeded
Created: Mon Mar 09 20:54:28 -0400 (12 minutes ago)
Started: Mon Mar 09 20:54:28 -0400 (12 minutes ago)
Finished: Mon Mar 09 21:04:31 -0400 (2 minutes ago)
Duration: 10 minutes 3 seconds

STEP PODNAME DURATION MESSAGE
✔ cyril-tests-fv59x (cyril-template)
└---✔ my-template(0) (my-template) cyril-tests-fv59x-1552010973 3s

Without RetryStrategy

$ argo get cyril-tests-rpx4s
Name: cyril-tests-rpx4s
Namespace: reporting
ServiceAccount: default
Status: Succeeded
Created: Mon Mar 09 20:57:51 -0400 (11 minutes ago)
Started: Mon Mar 09 20:57:51 -0400 (11 minutes ago)
Finished: Mon Mar 09 20:57:55 -0400 (11 minutes ago)
Duration: 4 seconds

STEP PODNAME DURATION MESSAGE
✔ cyril-tests-rpx4s (cyril-template)
└---✔ my-template (my-template) cyril-tests-rpx4s-2367970429 3s

---
If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[simster7]

I'll check this out