Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Workflow - Could not get container status #5251

Closed
caueasantos opened this issue Mar 2, 2021 · 43 comments · Fixed by #5296
Closed

Workflow - Could not get container status #5251

caueasantos opened this issue Mar 2, 2021 · 43 comments · Fixed by #5296
Assignees
@alexec
Labels
type/bug type/regression Regression from previous behavior (a specific type of bug)

Comments

@caueasantos
Copy link
Contributor

Summary

What happened/what you expected to happen?

The workflow should successfully terminate all of your triggered tasks. It wasn't easy to reproduce the anomalous behaviour outside of the cluster and also not happen every single time, so please find additional configurations that might help to reproduce it

argo-server: 3 replicas
executor: pns
parallelism: 1000

Diagnostics

What Kubernetes provider are you using?
GKE or Kind

What version of Argo Workflows are you running?

❯ argo version
argo: v3.0.0-rc3
  BuildDate: 2021-02-23T21:06:58Z
  GitCommit: c0c364c229e3b72306bd0b73161df090d24e0c31
  GitTreeState: clean
  GitTag: v3.0.0-rc3
  GoVersion: go1.13
  Compiler: gc
  Platform: darwin/amd64

Workflow template

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  creationTimestamp: "2021-03-02T00:36:25Z"
  generateName: ci-
  generation: 10
  labels:
    submit-from-ui: "true"
    workflows.argoproj.io/completed: "true"
    workflows.argoproj.io/creator: system-serviceaccount-argo-argo-server
    workflows.argoproj.io/phase: Error
    workflows.argoproj.io/workflow-template: ci
  managedFields:
  - apiVersion: argoproj.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:generateName: {}
        f:labels:
          .: {}
          f:submit-from-ui: {}
          f:workflows.argoproj.io/creator: {}
          f:workflows.argoproj.io/workflow-template: {}
      f:spec:
        .: {}
        f:arguments: {}
        f:entrypoint: {}
        f:workflowTemplateRef: {}
      f:status:
        .: {}
        f:storedTemplates: {}
    manager: argo
    operation: Update
    time: "2021-03-02T00:36:25Z"
  - apiVersion: argoproj.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          f:workflows.argoproj.io/completed: {}
          f:workflows.argoproj.io/phase: {}
      f:status:
        f:artifactRepositoryRef: {}
        f:conditions: {}
        f:finishedAt: {}
        f:nodes: {}
        f:phase: {}
        f:progress: {}
        f:resourcesDuration: {}
        f:startedAt: {}
        f:storedWorkflowTemplateSpec: {}
    manager: workflow-controller
    operation: Update
    time: "2021-03-02T00:37:47Z"
  name: ci-t8vq2
  namespace: argo
  resourceVersion: "119304"
  uid: fb296b31-b810-42c0-90bc-c4e8bb879b22
spec:
  arguments: {}
  entrypoint: main
  workflowTemplateRef:
    name: ci
status:
  artifactRepositoryRef:
    default: true
  conditions:
  - status: "False"
    type: PodRunning
  - status: "True"
    type: Completed
  finishedAt: "2021-03-02T00:37:47Z"
  nodes:
    ci-t8vq2:
      children:
      - ci-t8vq2-28546012
      - ci-t8vq2-212400199
      displayName: ci-t8vq2
      finishedAt: "2021-03-02T00:37:47Z"
      id: ci-t8vq2
      name: ci-t8vq2
      outboundNodes:
      - ci-t8vq2-2699866519
      - ci-t8vq2-2921795641
      - ci-t8vq2-935633491
      - ci-t8vq2-2460421745
      - ci-t8vq2-1497633967
      - ci-t8vq2-2529876561
      - ci-t8vq2-2517646915
      - ci-t8vq2-2594947737
      - ci-t8vq2-2608532967
      - ci-t8vq2-3834947628
      - ci-t8vq2-1156673745
      - ci-t8vq2-2620463353
      - ci-t8vq2-3303001669
      - ci-t8vq2-139394661
      - ci-t8vq2-1087124617
      - ci-t8vq2-1466005665
      - ci-t8vq2-3097822101
      - ci-t8vq2-3815346741
      - ci-t8vq2-3045333153
      - ci-t8vq2-2135555749
      - ci-t8vq2-1671854890
      - ci-t8vq2-2189885778
      - ci-t8vq2-2454096578
      - ci-t8vq2-3752440098
      - ci-t8vq2-1958506170
      - ci-t8vq2-569544386
      - ci-t8vq2-743674290
      - ci-t8vq2-553546802
      - ci-t8vq2-3806784823
      - ci-t8vq2-13895053
      - ci-t8vq2-2653728782
      - ci-t8vq2-3144997424
      - ci-t8vq2-2221870794
      - ci-t8vq2-2998710952
      - ci-t8vq2-924300910
      - ci-t8vq2-2687447728
      - ci-t8vq2-441930226
      - ci-t8vq2-2027865656
      - ci-t8vq2-857446365
      - ci-t8vq2-1089779475
      - ci-t8vq2-1533362641
      - ci-t8vq2-2193859387
      - ci-t8vq2-2924122029
      - ci-t8vq2-245955437
      - ci-t8vq2-3119937059
      - ci-t8vq2-3119595669
      - ci-t8vq2-418065567
      - ci-t8vq2-2120215045
      - ci-t8vq2-4250566955
      - ci-t8vq2-145289723
      - ci-t8vq2-1642984369
      - ci-t8vq2-1325696727
      - ci-t8vq2-3992325445
      - ci-t8vq2-3744868327
      - ci-t8vq2-1719834657
      - ci-t8vq2-178844961
      phase: Error
      progress: 57/57
      resourcesDuration:
        cpu: 889
        memory: 889
      startedAt: "2021-03-02T00:36:26Z"
      templateName: main
      templateScope: local/
      type: DAG
    ci-t8vq2-13895053:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(29:context:L)
      finishedAt: "2021-03-02T00:36:54Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-13895053
      inputs:
        parameters:
        - name: message
          value: L
      name: ci-t8vq2.echo-multiple-times(29:context:L)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 17
        memory: 17
      startedAt: "2021-03-02T00:36:27Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-28546012:
      boundaryID: ci-t8vq2
      children:
      - ci-t8vq2-2699866519
      - ci-t8vq2-2921795641
      - ci-t8vq2-935633491
      - ci-t8vq2-2460421745
      - ci-t8vq2-1497633967
      - ci-t8vq2-2529876561
      - ci-t8vq2-2517646915
      - ci-t8vq2-2594947737
      - ci-t8vq2-2608532967
      - ci-t8vq2-3834947628
      - ci-t8vq2-1156673745
      - ci-t8vq2-2620463353
      - ci-t8vq2-3303001669
      - ci-t8vq2-139394661
      - ci-t8vq2-1087124617
      - ci-t8vq2-1466005665
      - ci-t8vq2-3097822101
      - ci-t8vq2-3815346741
      - ci-t8vq2-3045333153
      - ci-t8vq2-2135555749
      - ci-t8vq2-1671854890
      - ci-t8vq2-2189885778
      - ci-t8vq2-2454096578
      - ci-t8vq2-3752440098
      - ci-t8vq2-1958506170
      - ci-t8vq2-569544386
      - ci-t8vq2-743674290
      - ci-t8vq2-553546802
      - ci-t8vq2-3806784823
      - ci-t8vq2-13895053
      - ci-t8vq2-2653728782
      - ci-t8vq2-3144997424
      - ci-t8vq2-2221870794
      - ci-t8vq2-2998710952
      - ci-t8vq2-924300910
      - ci-t8vq2-2687447728
      - ci-t8vq2-441930226
      - ci-t8vq2-2027865656
      displayName: echo-multiple-times
      finishedAt: "2021-03-02T00:37:26Z"
      id: ci-t8vq2-28546012
      name: ci-t8vq2.echo-multiple-times
      phase: Succeeded
      progress: 38/38
      resourcesDuration:
        cpu: 705
        memory: 705
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: TaskGroup
    ci-t8vq2-128512104:
      boundaryID: ci-t8vq2
      children:
      - ci-t8vq2-3119937059
      - ci-t8vq2-3119595669
      - ci-t8vq2-418065567
      - ci-t8vq2-2120215045
      - ci-t8vq2-4250566955
      displayName: task4
      finishedAt: "2021-03-02T00:37:47Z"
      id: ci-t8vq2-128512104
      name: ci-t8vq2.task4
      phase: Succeeded
      progress: 5/5
      resourcesDuration:
        cpu: 42
        memory: 42
      startedAt: "2021-03-02T00:37:16Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: TaskGroup
    ci-t8vq2-139394661:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(13:context:O)
      finishedAt: "2021-03-02T00:36:46Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-139394661
      inputs:
        parameters:
        - name: message
          value: O
      name: ci-t8vq2.echo-multiple-times(13:context:O)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 13
        memory: 13
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-145289723:
      boundaryID: ci-t8vq2
      displayName: task5
      finishedAt: "2021-03-02T00:37:33Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-145289723
      inputs:
        parameters:
        - name: message
          value: task5
      name: ci-t8vq2.task5
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 10
        memory: 10
      startedAt: "2021-03-02T00:37:17Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-162067342:
      boundaryID: ci-t8vq2
      children:
      - ci-t8vq2-1642984369
      - ci-t8vq2-1325696727
      - ci-t8vq2-3992325445
      - ci-t8vq2-3744868327
      - ci-t8vq2-1719834657
      displayName: task6
      finishedAt: "2021-03-02T00:37:47Z"
      id: ci-t8vq2-162067342
      name: ci-t8vq2.task6
      phase: Succeeded
      progress: 5/5
      resourcesDuration:
        cpu: 46
        memory: 46
      startedAt: "2021-03-02T00:37:17Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: TaskGroup
    ci-t8vq2-178844961:
      boundaryID: ci-t8vq2
      displayName: task7
      finishedAt: "2021-03-02T00:37:35Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-178844961
      inputs:
        parameters:
        - name: message
          value: task7
      name: ci-t8vq2.task7
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 11
        memory: 11
      startedAt: "2021-03-02T00:37:17Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-212400199:
      boundaryID: ci-t8vq2
      children:
      - ci-t8vq2-229177818
      - ci-t8vq2-245955437
      - ci-t8vq2-128512104
      - ci-t8vq2-145289723
      - ci-t8vq2-162067342
      - ci-t8vq2-178844961
      displayName: task1
      finishedAt: "2021-03-02T00:37:08Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-212400199
      inputs:
        parameters:
        - name: message
          value: task1
      name: ci-t8vq2.task1
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 26
        memory: 26
      startedAt: "2021-03-02T00:36:27Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-229177818:
      boundaryID: ci-t8vq2
      children:
      - ci-t8vq2-857446365
      - ci-t8vq2-1089779475
      - ci-t8vq2-1533362641
      - ci-t8vq2-2193859387
      - ci-t8vq2-2924122029
      displayName: task2
      finishedAt: "2021-03-02T00:37:47Z"
      id: ci-t8vq2-229177818
      name: ci-t8vq2.task2
      phase: Error
      progress: 5/5
      resourcesDuration:
        cpu: 42
        memory: 42
      startedAt: "2021-03-02T00:37:16Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: TaskGroup
    ci-t8vq2-245955437:
      boundaryID: ci-t8vq2
      displayName: task3
      finishedAt: "2021-03-02T00:37:29Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-245955437
      inputs:
        parameters:
        - name: message
          value: task3
      name: ci-t8vq2.task3
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 7
        memory: 7
      startedAt: "2021-03-02T00:37:16Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-418065567:
      boundaryID: ci-t8vq2
      displayName: task4(2:context:C)
      finishedAt: "2021-03-02T00:37:26Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-418065567
      inputs:
        parameters:
        - name: message
          value: task4
      name: ci-t8vq2.task4(2:context:C)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 5
        memory: 5
      startedAt: "2021-03-02T00:37:17Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-441930226:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(36:context:S)
      finishedAt: "2021-03-02T00:37:08Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-441930226
      inputs:
        parameters:
        - name: message
          value: S
      name: ci-t8vq2.echo-multiple-times(36:context:S)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 28
        memory: 28
      startedAt: "2021-03-02T00:36:27Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-553546802:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(27:context:I)
      finishedAt: "2021-03-02T00:37:02Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-553546802
      inputs:
        parameters:
        - name: message
          value: I
      name: ci-t8vq2.echo-multiple-times(27:context:I)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 23
        memory: 23
      startedAt: "2021-03-02T00:36:27Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-569544386:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(25:context:G)
      finishedAt: "2021-03-02T00:37:01Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-569544386
      inputs:
        parameters:
        - name: message
          value: G
      name: ci-t8vq2.echo-multiple-times(25:context:G)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 25
        memory: 25
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-743674290:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(26:context:H)
      finishedAt: "2021-03-02T00:36:56Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-743674290
      inputs:
        parameters:
        - name: message
          value: H
      name: ci-t8vq2.echo-multiple-times(26:context:H)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 16
        memory: 16
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-857446365:
      boundaryID: ci-t8vq2
      displayName: task2(0:context:A)
      finishedAt: "2021-03-02T00:37:29Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-857446365
      inputs:
        parameters:
        - name: message
          value: task2
      name: ci-t8vq2.task2(0:context:A)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 9
        memory: 9
      startedAt: "2021-03-02T00:37:16Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-924300910:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(34:context:Q)
      finishedAt: "2021-03-02T00:37:03Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-924300910
      inputs:
        parameters:
        - name: message
          value: Q
      name: ci-t8vq2.echo-multiple-times(34:context:Q)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 24
        memory: 24
      startedAt: "2021-03-02T00:36:27Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-935633491:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(2:context:C)
      finishedAt: "2021-03-02T00:36:45Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-935633491
      inputs:
        parameters:
        - name: message
          value: C
      name: ci-t8vq2.echo-multiple-times(2:context:C)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 18
        memory: 18
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-1087124617:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(14:context:P)
      finishedAt: "2021-03-02T00:36:49Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-1087124617
      inputs:
        parameters:
        - name: message
          value: P
      name: ci-t8vq2.echo-multiple-times(14:context:P)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 14
        memory: 14
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-1089779475:
      boundaryID: ci-t8vq2
      displayName: task2(1:context:B)
      finishedAt: "2021-03-02T00:37:31Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-1089779475
      inputs:
        parameters:
        - name: message
          value: task2
      name: ci-t8vq2.task2(1:context:B)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 12
        memory: 12
      startedAt: "2021-03-02T00:37:16Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-1156673745:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(10:context:L)
      finishedAt: "2021-03-02T00:36:53Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-1156673745
      inputs:
        parameters:
        - name: message
          value: L
      name: ci-t8vq2.echo-multiple-times(10:context:L)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 20
        memory: 20
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-1325696727:
      boundaryID: ci-t8vq2
      displayName: task6(1:context:B)
      finishedAt: "2021-03-02T00:37:33Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-1325696727
      inputs:
        parameters:
        - name: message
          value: task6
      name: ci-t8vq2.task6(1:context:B)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 11
        memory: 11
      startedAt: "2021-03-02T00:37:17Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-1466005665:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(15:context:Q)
      finishedAt: "2021-03-02T00:36:46Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-1466005665
      inputs:
        parameters:
        - name: message
          value: Q
      name: ci-t8vq2.echo-multiple-times(15:context:Q)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 12
        memory: 12
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-1497633967:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(4:context:E)
      finishedAt: "2021-03-02T00:36:52Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-1497633967
      inputs:
        parameters:
        - name: message
          value: E
      name: ci-t8vq2.echo-multiple-times(4:context:E)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 20
        memory: 20
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-1533362641:
      boundaryID: ci-t8vq2
      displayName: task2(2:context:C)
      finishedAt: "2021-03-02T00:37:24Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-1533362641
      inputs:
        parameters:
        - name: message
          value: task2
      message: 'Error (exit code 1): Could not get container status'
      name: ci-t8vq2.task2(2:context:C)
      phase: Error
      progress: 1/1
      resourcesDuration:
        cpu: 4
        memory: 4
      startedAt: "2021-03-02T00:37:16Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-1642984369:
      boundaryID: ci-t8vq2
      displayName: task6(0:context:A)
      finishedAt: "2021-03-02T00:37:27Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-1642984369
      inputs:
        parameters:
        - name: message
          value: task6
      name: ci-t8vq2.task6(0:context:A)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 5
        memory: 5
      startedAt: "2021-03-02T00:37:17Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-1671854890:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(20:context:B)
      finishedAt: "2021-03-02T00:36:48Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-1671854890
      inputs:
        parameters:
        - name: message
          value: B
      name: ci-t8vq2.echo-multiple-times(20:context:B)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 13
        memory: 13
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-1719834657:
      boundaryID: ci-t8vq2
      displayName: task6(4:context:E)
      finishedAt: "2021-03-02T00:37:32Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-1719834657
      inputs:
        parameters:
        - name: message
          value: task6
      name: ci-t8vq2.task6(4:context:E)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 11
        memory: 11
      startedAt: "2021-03-02T00:37:17Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-1958506170:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(24:context:F)
      finishedAt: "2021-03-02T00:36:50Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-1958506170
      inputs:
        parameters:
        - name: message
          value: F
      name: ci-t8vq2.echo-multiple-times(24:context:F)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 14
        memory: 14
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-2027865656:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(37:context:T)
      finishedAt: "2021-03-02T00:37:02Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-2027865656
      inputs:
        parameters:
        - name: message
          value: T
      name: ci-t8vq2.echo-multiple-times(37:context:T)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 24
        memory: 24
      startedAt: "2021-03-02T00:36:27Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-2120215045:
      boundaryID: ci-t8vq2
      displayName: task4(3:context:D)
      finishedAt: "2021-03-02T00:37:28Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-2120215045
      inputs:
        parameters:
        - name: message
          value: task4
      name: ci-t8vq2.task4(3:context:D)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 8
        memory: 8
      startedAt: "2021-03-02T00:37:17Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-2135555749:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(19:context:A)
      finishedAt: "2021-03-02T00:36:47Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-2135555749
      inputs:
        parameters:
        - name: message
          value: A
      name: ci-t8vq2.echo-multiple-times(19:context:A)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 13
        memory: 13
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-2189885778:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(21:context:C)
      finishedAt: "2021-03-02T00:36:55Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-2189885778
      inputs:
        parameters:
        - name: message
          value: C
      name: ci-t8vq2.echo-multiple-times(21:context:C)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 19
        memory: 19
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-2193859387:
      boundaryID: ci-t8vq2
      displayName: task2(3:context:D)
      finishedAt: "2021-03-02T00:37:30Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-2193859387
      inputs:
        parameters:
        - name: message
          value: task2
      name: ci-t8vq2.task2(3:context:D)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 9
        memory: 9
      startedAt: "2021-03-02T00:37:16Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-2221870794:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(32:context:O)
      finishedAt: "2021-03-02T00:36:53Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-2221870794
      inputs:
        parameters:
        - name: message
          value: O
      name: ci-t8vq2.echo-multiple-times(32:context:O)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 17
        memory: 17
      startedAt: "2021-03-02T00:36:27Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-2454096578:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(22:context:D)
      finishedAt: "2021-03-02T00:36:59Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-2454096578
      inputs:
        parameters:
        - name: message
          value: D
      name: ci-t8vq2.echo-multiple-times(22:context:D)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 22
        memory: 22
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-2460421745:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(3:context:D)
      finishedAt: "2021-03-02T00:36:49Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-2460421745
      inputs:
        parameters:
        - name: message
          value: D
      name: ci-t8vq2.echo-multiple-times(3:context:D)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 18
        memory: 18
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-2517646915:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(6:context:G)
      finishedAt: "2021-03-02T00:36:46Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-2517646915
      inputs:
        parameters:
        - name: message
          value: G
      name: ci-t8vq2.echo-multiple-times(6:context:G)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 13
        memory: 13
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-2529876561:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(5:context:F)
      finishedAt: "2021-03-02T00:36:48Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-2529876561
      inputs:
        parameters:
        - name: message
          value: F
      name: ci-t8vq2.echo-multiple-times(5:context:F)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 18
        memory: 18
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-2594947737:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(7:context:H)
      finishedAt: "2021-03-02T00:36:48Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-2594947737
      inputs:
        parameters:
        - name: message
          value: H
      name: ci-t8vq2.echo-multiple-times(7:context:H)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 18
        memory: 18
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-2608532967:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(8:context:I)
      finishedAt: "2021-03-02T00:36:52Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-2608532967
      inputs:
        parameters:
        - name: message
          value: I
      name: ci-t8vq2.echo-multiple-times(8:context:I)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 18
        memory: 18
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-2620463353:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(11:context:M)
      finishedAt: "2021-03-02T00:36:50Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-2620463353
      inputs:
        parameters:
        - name: message
          value: M
      name: ci-t8vq2.echo-multiple-times(11:context:M)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 14
        memory: 14
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-2653728782:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(30:context:M)
      finishedAt: "2021-03-02T00:37:05Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-2653728782
      inputs:
        parameters:
        - name: message
          value: M
      name: ci-t8vq2.echo-multiple-times(30:context:M)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 25
        memory: 25
      startedAt: "2021-03-02T00:36:27Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-2687447728:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(35:context:R)
      finishedAt: "2021-03-02T00:37:02Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-2687447728
      inputs:
        parameters:
        - name: message
          value: R
      name: ci-t8vq2.echo-multiple-times(35:context:R)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 26
        memory: 26
      startedAt: "2021-03-02T00:36:27Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-2699866519:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(0:context:A)
      finishedAt: "2021-03-02T00:36:54Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-2699866519
      inputs:
        parameters:
        - name: message
          value: A
      name: ci-t8vq2.echo-multiple-times(0:context:A)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 26
        memory: 26
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-2921795641:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(1:context:B)
      finishedAt: "2021-03-02T00:36:45Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-2921795641
      inputs:
        parameters:
        - name: message
          value: B
      name: ci-t8vq2.echo-multiple-times(1:context:B)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 18
        memory: 18
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-2924122029:
      boundaryID: ci-t8vq2
      displayName: task2(4:context:E)
      finishedAt: "2021-03-02T00:37:28Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-2924122029
      inputs:
        parameters:
        - name: message
          value: task2
      name: ci-t8vq2.task2(4:context:E)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 8
        memory: 8
      startedAt: "2021-03-02T00:37:16Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-2998710952:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(33:context:P)
      finishedAt: "2021-03-02T00:37:05Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-2998710952
      inputs:
        parameters:
        - name: message
          value: P
      name: ci-t8vq2.echo-multiple-times(33:context:P)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 29
        memory: 29
      startedAt: "2021-03-02T00:36:27Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-3045333153:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(18:context:T)
      finishedAt: "2021-03-02T00:36:51Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-3045333153
      inputs:
        parameters:
        - name: message
          value: T
      name: ci-t8vq2.echo-multiple-times(18:context:T)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 17
        memory: 17
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-3097822101:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(16:context:R)
      finishedAt: "2021-03-02T00:36:45Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-3097822101
      inputs:
        parameters:
        - name: message
          value: R
      name: ci-t8vq2.echo-multiple-times(16:context:R)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 12
        memory: 12
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-3119595669:
      boundaryID: ci-t8vq2
      displayName: task4(1:context:B)
      finishedAt: "2021-03-02T00:37:31Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-3119595669
      inputs:
        parameters:
        - name: message
          value: task4
      name: ci-t8vq2.task4(1:context:B)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 8
        memory: 8
      startedAt: "2021-03-02T00:37:17Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-3119937059:
      boundaryID: ci-t8vq2
      displayName: task4(0:context:A)
      finishedAt: "2021-03-02T00:37:34Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-3119937059
      inputs:
        parameters:
        - name: message
          value: task4
      name: ci-t8vq2.task4(0:context:A)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 12
        memory: 12
      startedAt: "2021-03-02T00:37:16Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-3144997424:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(31:context:N)
      finishedAt: "2021-03-02T00:37:09Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-3144997424
      inputs:
        parameters:
        - name: message
          value: "N"
      name: ci-t8vq2.echo-multiple-times(31:context:N)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 29
        memory: 29
      startedAt: "2021-03-02T00:36:27Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-3303001669:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(12:context:N)
      finishedAt: "2021-03-02T00:36:52Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-3303001669
      inputs:
        parameters:
        - name: message
          value: "N"
      name: ci-t8vq2.echo-multiple-times(12:context:N)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 20
        memory: 20
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-3744868327:
      boundaryID: ci-t8vq2
      displayName: task6(3:context:D)
      finishedAt: "2021-03-02T00:37:34Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-3744868327
      inputs:
        parameters:
        - name: message
          value: task6
      name: ci-t8vq2.task6(3:context:D)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 11
        memory: 11
      startedAt: "2021-03-02T00:37:17Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-3752440098:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(23:context:E)
      finishedAt: "2021-03-02T00:36:45Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-3752440098
      inputs:
        parameters:
        - name: message
          value: E
      name: ci-t8vq2.echo-multiple-times(23:context:E)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 9
        memory: 9
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-3806784823:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(28:context:K)
      finishedAt: "2021-03-02T00:36:50Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-3806784823
      inputs:
        parameters:
        - name: message
          value: K
      name: ci-t8vq2.echo-multiple-times(28:context:K)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 15
        memory: 15
      startedAt: "2021-03-02T00:36:27Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-3815346741:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(17:context:S)
      finishedAt: "2021-03-02T00:36:51Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-3815346741
      inputs:
        parameters:
        - name: message
          value: S
      name: ci-t8vq2.echo-multiple-times(17:context:S)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 16
        memory: 16
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-3834947628:
      boundaryID: ci-t8vq2
      displayName: echo-multiple-times(9:context:K)
      finishedAt: "2021-03-02T00:36:47Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-3834947628
      inputs:
        parameters:
        - name: message
          value: K
      name: ci-t8vq2.echo-multiple-times(9:context:K)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 12
        memory: 12
      startedAt: "2021-03-02T00:36:26Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-3992325445:
      boundaryID: ci-t8vq2
      displayName: task6(2:context:C)
      finishedAt: "2021-03-02T00:37:27Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-3992325445
      inputs:
        parameters:
        - name: message
          value: task6
      name: ci-t8vq2.task6(2:context:C)
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 8
        memory: 8
      startedAt: "2021-03-02T00:37:17Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
    ci-t8vq2-4250566955:
      boundaryID: ci-t8vq2
      displayName: task4(4:context:E)
      finishedAt: "2021-03-02T00:37:30Z"
      hostNodeName: kind-control-plane
      id: ci-t8vq2-4250566955
      inputs:
        parameters:
        - name: message
          value: task4
      name: ci-t8vq2.task4(4:context:E)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 9
        memory: 9
      startedAt: "2021-03-02T00:37:17Z"
      templateRef:
        name: base
        template: main
      templateScope: local/
      type: Pod
  phase: Error
  progress: 57/57
  resourcesDuration:
    cpu: 889
    memory: 889
  startedAt: "2021-03-02T00:36:25Z"
  storedTemplates:
    namespaced/base/main:
      container:
        args:
        - echo {{inputs.parameters.message}}
        command:
        - sh
        - -c
        image: curlimages/curl:7.75.0
        name: ""
        resources: {}
      inputs:
        parameters:
        - name: message
      metadata: {}
      name: main
      outputs: {}
    namespaced/ci/main:
      dag:
        tasks:
        - arguments:
            parameters:
            - name: message
              value: task1
          name: task1
          templateRef:
            name: base
            template: main
        - arguments:
            parameters:
            - name: message
              value: '{{item.context}}'
          name: echo-multiple-times
          templateRef:
            name: base
            template: main
          withItems:
          - context: A
          - context: B
          - context: C
          - context: D
          - context: E
          - context: F
          - context: G
          - context: H
          - context: I
          - context: K
          - context: L
          - context: M
          - context: "N"
          - context: O
          - context: P
          - context: Q
          - context: R
          - context: S
          - context: T
          - context: A
          - context: B
          - context: C
          - context: D
          - context: E
          - context: F
          - context: G
          - context: H
          - context: I
          - context: K
          - context: L
          - context: M
          - context: "N"
          - context: O
          - context: P
          - context: Q
          - context: R
          - context: S
          - context: T
        - arguments:
            parameters:
            - name: message
              value: task2
          dependencies:
          - task1
          name: task2
          templateRef:
            name: base
            template: main
          withItems:
          - context: A
          - context: B
          - context: C
          - context: D
          - context: E
        - arguments:
            parameters:
            - name: message
              value: task3
          dependencies:
          - task1
          name: task3
          templateRef:
            name: base
            template: main
        - arguments:
            parameters:
            - name: message
              value: task4
          dependencies:
          - task1
          name: task4
          templateRef:
            name: base
            template: main
          withItems:
          - context: A
          - context: B
          - context: C
          - context: D
          - context: E
        - arguments:
            parameters:
            - name: message
              value: task5
          dependencies:
          - task1
          name: task5
          templateRef:
            name: base
            template: main
        - arguments:
            parameters:
            - name: message
              value: task6
          dependencies:
          - task1
          name: task6
          templateRef:
            name: base
            template: main
          withItems:
          - context: A
          - context: B
          - context: C
          - context: D
          - context: E
        - arguments:
            parameters:
            - name: message
              value: task7
          dependencies:
          - task1
          name: task7
          templateRef:
            name: base
            template: main
      inputs: {}
      metadata: {}
      name: main
      outputs: {}
  storedWorkflowTemplateSpec:
    arguments: {}
    entrypoint: main
    parallelism: 1000
    serviceAccountName: argo
    templates:
    - dag:
        tasks:
        - arguments:
            parameters:
            - name: message
              value: task1
          name: task1
          templateRef:
            name: base
            template: main
        - arguments:
            parameters:
            - name: message
              value: '{{item.context}}'
          name: echo-multiple-times
          templateRef:
            name: base
            template: main
          withItems:
          - context: A
          - context: B
          - context: C
          - context: D
          - context: E
          - context: F
          - context: G
          - context: H
          - context: I
          - context: K
          - context: L
          - context: M
          - context: "N"
          - context: O
          - context: P
          - context: Q
          - context: R
          - context: S
          - context: T
          - context: A
          - context: B
          - context: C
          - context: D
          - context: E
          - context: F
          - context: G
          - context: H
          - context: I
          - context: K
          - context: L
          - context: M
          - context: "N"
          - context: O
          - context: P
          - context: Q
          - context: R
          - context: S
          - context: T
        - arguments:
            parameters:
            - name: message
              value: task2
          dependencies:
          - task1
          name: task2
          templateRef:
            name: base
            template: main
          withItems:
          - context: A
          - context: B
          - context: C
          - context: D
          - context: E
        - arguments:
            parameters:
            - name: message
              value: task3
          dependencies:
          - task1
          name: task3
          templateRef:
            name: base
            template: main
        - arguments:
            parameters:
            - name: message
              value: task4
          dependencies:
          - task1
          name: task4
          templateRef:
            name: base
            template: main
          withItems:
          - context: A
          - context: B
          - context: C
          - context: D
          - context: E
        - arguments:
            parameters:
            - name: message
              value: task5
          dependencies:
          - task1
          name: task5
          templateRef:
            name: base
            template: main
        - arguments:
            parameters:
            - name: message
              value: task6
          dependencies:
          - task1
          name: task6
          templateRef:
            name: base
            template: main
          withItems:
          - context: A
          - context: B
          - context: C
          - context: D
          - context: E
        - arguments:
            parameters:
            - name: message
              value: task7
          dependencies:
          - task1
          name: task7
          templateRef:
            name: base
            template: main
      inputs: {}
      metadata: {}
      name: main
      outputs: {}
    workflowTemplateRef:
      name: ci

Controller logs

time="2021-03-02T00:36:25.857Z" level=info msg="Processing workflow" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:25.865Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.010Z" level=info msg="DAG node ci-t8vq2 initialized Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.013Z" level=info msg="TaskGroup node ci-t8vq2-28546012 initialized Running (message: )" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.013Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(0:context:A) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.014Z" level=info msg="Pod node ci-t8vq2-2699866519 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.032Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(0:context:A) (ci-t8vq2-2699866519)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.032Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(1:context:B) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.035Z" level=info msg="Pod node ci-t8vq2-2921795641 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.061Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(1:context:B) (ci-t8vq2-2921795641)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.062Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(2:context:C) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.063Z" level=info msg="Pod node ci-t8vq2-935633491 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.082Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(2:context:C) (ci-t8vq2-935633491)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.082Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(3:context:D) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.087Z" level=info msg="Pod node ci-t8vq2-2460421745 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.097Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(3:context:D) (ci-t8vq2-2460421745)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.097Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(4:context:E) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.105Z" level=info msg="Pod node ci-t8vq2-1497633967 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.150Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(4:context:E) (ci-t8vq2-1497633967)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.151Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(5:context:F) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.157Z" level=info msg="Pod node ci-t8vq2-2529876561 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.184Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(5:context:F) (ci-t8vq2-2529876561)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.184Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(6:context:G) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.185Z" level=info msg="Pod node ci-t8vq2-2517646915 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.199Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(6:context:G) (ci-t8vq2-2517646915)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.199Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(7:context:H) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.199Z" level=info msg="Pod node ci-t8vq2-2594947737 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.222Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(7:context:H) (ci-t8vq2-2594947737)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.222Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(8:context:I) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.222Z" level=info msg="Pod node ci-t8vq2-2608532967 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.243Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(8:context:I) (ci-t8vq2-2608532967)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.244Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(9:context:K) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.264Z" level=info msg="Pod node ci-t8vq2-3834947628 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.274Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(9:context:K) (ci-t8vq2-3834947628)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.274Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(10:context:L) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.275Z" level=info msg="Pod node ci-t8vq2-1156673745 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.294Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(10:context:L) (ci-t8vq2-1156673745)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.294Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(11:context:M) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.294Z" level=info msg="Pod node ci-t8vq2-2620463353 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.315Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(11:context:M) (ci-t8vq2-2620463353)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.315Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(12:context:N) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.315Z" level=info msg="Pod node ci-t8vq2-3303001669 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.347Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(12:context:N) (ci-t8vq2-3303001669)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.347Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(13:context:O) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.347Z" level=info msg="Pod node ci-t8vq2-139394661 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.378Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(13:context:O) (ci-t8vq2-139394661)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.378Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(14:context:P) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.378Z" level=info msg="Pod node ci-t8vq2-1087124617 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.426Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(14:context:P) (ci-t8vq2-1087124617)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.426Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(15:context:Q) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.427Z" level=info msg="Pod node ci-t8vq2-1466005665 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.466Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(15:context:Q) (ci-t8vq2-1466005665)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.466Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(16:context:R) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.479Z" level=info msg="Pod node ci-t8vq2-3097822101 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.506Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(16:context:R) (ci-t8vq2-3097822101)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.506Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(17:context:S) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.506Z" level=info msg="Pod node ci-t8vq2-3815346741 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.554Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(17:context:S) (ci-t8vq2-3815346741)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.556Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(18:context:T) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.569Z" level=info msg="Pod node ci-t8vq2-3045333153 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.623Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(18:context:T) (ci-t8vq2-3045333153)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.623Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(19:context:A) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.624Z" level=info msg="Pod node ci-t8vq2-2135555749 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.657Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(19:context:A) (ci-t8vq2-2135555749)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.657Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(20:context:B) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.664Z" level=info msg="Pod node ci-t8vq2-1671854890 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.696Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(20:context:B) (ci-t8vq2-1671854890)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.696Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(21:context:C) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.697Z" level=info msg="Pod node ci-t8vq2-2189885778 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.711Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(21:context:C) (ci-t8vq2-2189885778)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.712Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(22:context:D) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.712Z" level=info msg="Pod node ci-t8vq2-2454096578 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.732Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(22:context:D) (ci-t8vq2-2454096578)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.732Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(23:context:E) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.732Z" level=info msg="Pod node ci-t8vq2-3752440098 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.764Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(23:context:E) (ci-t8vq2-3752440098)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.764Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(24:context:F) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.766Z" level=info msg="Pod node ci-t8vq2-1958506170 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.874Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(24:context:F) (ci-t8vq2-1958506170)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.874Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(25:context:G) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.875Z" level=info msg="Pod node ci-t8vq2-569544386 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.920Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(25:context:G) (ci-t8vq2-569544386)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.923Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(26:context:H) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:26.924Z" level=info msg="Pod node ci-t8vq2-743674290 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.040Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(26:context:H) (ci-t8vq2-743674290)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.040Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(27:context:I) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.042Z" level=info msg="Pod node ci-t8vq2-553546802 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.100Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(27:context:I) (ci-t8vq2-553546802)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.100Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(28:context:K) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.100Z" level=info msg="Pod node ci-t8vq2-3806784823 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.134Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(28:context:K) (ci-t8vq2-3806784823)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.134Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(29:context:L) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.138Z" level=info msg="Pod node ci-t8vq2-13895053 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.178Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(29:context:L) (ci-t8vq2-13895053)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.178Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(30:context:M) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.178Z" level=info msg="Pod node ci-t8vq2-2653728782 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.198Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(30:context:M) (ci-t8vq2-2653728782)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.207Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(31:context:N) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.208Z" level=info msg="Pod node ci-t8vq2-3144997424 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.230Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(31:context:N) (ci-t8vq2-3144997424)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.230Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(32:context:O) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.231Z" level=info msg="Pod node ci-t8vq2-2221870794 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.268Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(32:context:O) (ci-t8vq2-2221870794)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.268Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(33:context:P) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.268Z" level=info msg="Pod node ci-t8vq2-2998710952 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.298Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(33:context:P) (ci-t8vq2-2998710952)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.298Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(34:context:Q) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.298Z" level=info msg="Pod node ci-t8vq2-924300910 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.340Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(34:context:Q) (ci-t8vq2-924300910)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.340Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(35:context:R) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.340Z" level=info msg="Pod node ci-t8vq2-2687447728 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.362Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(35:context:R) (ci-t8vq2-2687447728)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.362Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(36:context:S) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.366Z" level=info msg="Pod node ci-t8vq2-441930226 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.383Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(36:context:S) (ci-t8vq2-441930226)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.383Z" level=info msg="All of node ci-t8vq2.echo-multiple-times(37:context:T) dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.397Z" level=info msg="Pod node ci-t8vq2-2027865656 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.476Z" level=info msg="Created pod: ci-t8vq2.echo-multiple-times(37:context:T) (ci-t8vq2-2027865656)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.476Z" level=info msg="All of node ci-t8vq2.task1 dependencies [] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.476Z" level=info msg="Pod node ci-t8vq2-212400199 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.494Z" level=info msg="Created pod: ci-t8vq2.task1 (ci-t8vq2-212400199)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:27.620Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=118446 workflow=ci-t8vq2
time="2021-03-02T00:36:36.042Z" level=info msg="Processing workflow" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:36.044Z" level=info msg="Updating node ci-t8vq2-1958506170 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:36.044Z" level=info msg="Updating node ci-t8vq2-2460421745 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:36.044Z" level=info msg="Updating node ci-t8vq2-2189885778 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:36.045Z" level=info msg="Updating node ci-t8vq2-2620463353 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:36.045Z" level=info msg="Updating node ci-t8vq2-3144997424 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:36.045Z" level=info msg="Updating node ci-t8vq2-2594947737 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:36.045Z" level=info msg="Updating node ci-t8vq2-3815346741 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:36.045Z" level=info msg="Updating node ci-t8vq2-2608532967 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:36.045Z" level=info msg="Updating node ci-t8vq2-3806784823 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:36.045Z" level=info msg="Updating node ci-t8vq2-1497633967 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:36.045Z" level=info msg="Updating node ci-t8vq2-2529876561 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:36.045Z" level=info msg="Updating node ci-t8vq2-3303001669 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:36.081Z" level=info msg="Updating node ci-t8vq2-2454096578 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:36.081Z" level=info msg="Updating node ci-t8vq2-743674290 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:36.081Z" level=info msg="Updating node ci-t8vq2-2221870794 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:36.087Z" level=info msg="Updating node ci-t8vq2-1087124617 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:36.087Z" level=info msg="Updating node ci-t8vq2-3045333153 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:36.087Z" level=info msg="Updating node ci-t8vq2-1156673745 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:36.825Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=118536 workflow=ci-t8vq2
time="2021-03-02T00:36:46.220Z" level=info msg="Processing workflow" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.221Z" level=info msg="Updating node ci-t8vq2-212400199 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.221Z" level=info msg="Updating node ci-t8vq2-1466005665 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.221Z" level=info msg="Updating node ci-t8vq2-1671854890 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.222Z" level=info msg="Updating node ci-t8vq2-2687447728 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.222Z" level=info msg="Updating node ci-t8vq2-924300910 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.222Z" level=info msg="Updating node ci-t8vq2-441930226 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.222Z" level=info msg="Updating node ci-t8vq2-2921795641 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.222Z" level=info msg="Updating node ci-t8vq2-3097822101 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.222Z" level=info msg="Updating node ci-t8vq2-2027865656 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.222Z" level=info msg="Updating node ci-t8vq2-2998710952 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.222Z" level=info msg="Updating node ci-t8vq2-2135555749 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.223Z" level=info msg="Updating node ci-t8vq2-3834947628 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.223Z" level=info msg="Updating node ci-t8vq2-3752440098 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.223Z" level=info msg="Updating node ci-t8vq2-569544386 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.223Z" level=info msg="Updating node ci-t8vq2-2517646915 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.224Z" level=info msg="Updating node ci-t8vq2-553546802 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.224Z" level=info msg="Updating node ci-t8vq2-2699866519 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.224Z" level=info msg="Setting node ci-t8vq2-935633491 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.224Z" level=info msg="Updating node ci-t8vq2-935633491 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.224Z" level=info msg="Updating node ci-t8vq2-2653728782 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.224Z" level=info msg="Updating node ci-t8vq2-13895053 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.224Z" level=info msg="Updating node ci-t8vq2-139394661 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:46.418Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=118629 workflow=ci-t8vq2
time="2021-03-02T00:36:56.412Z" level=info msg="Processing workflow" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.413Z" level=info msg="Updating node ci-t8vq2-1958506170 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.413Z" level=info msg="Updating node ci-t8vq2-13895053 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.413Z" level=info msg="Updating node ci-t8vq2-2620463353 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.413Z" level=info msg="Updating node ci-t8vq2-139394661 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.413Z" level=info msg="Updating node ci-t8vq2-1671854890 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.413Z" level=info msg="Setting node ci-t8vq2-2529876561 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.413Z" level=info msg="Updating node ci-t8vq2-2529876561 status Pending -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.413Z" level=info msg="Setting node ci-t8vq2-2189885778 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.414Z" level=info msg="Updating node ci-t8vq2-2189885778 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.414Z" level=info msg="Setting node ci-t8vq2-2699866519 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.414Z" level=info msg="Updating node ci-t8vq2-2699866519 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.414Z" level=info msg="Setting node ci-t8vq2-3303001669 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.414Z" level=info msg="Updating node ci-t8vq2-3303001669 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.416Z" level=info msg="Setting node ci-t8vq2-2594947737 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.416Z" level=info msg="Updating node ci-t8vq2-2594947737 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.416Z" level=info msg="Updating node ci-t8vq2-3815346741 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.416Z" level=info msg="Updating node ci-t8vq2-2608532967 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.416Z" level=info msg="Updating node ci-t8vq2-3834947628 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.416Z" level=info msg="Updating node ci-t8vq2-1466005665 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.417Z" level=info msg="Setting node ci-t8vq2-2460421745 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.417Z" level=info msg="Updating node ci-t8vq2-2460421745 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.417Z" level=info msg="Updating node ci-t8vq2-3045333153 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.418Z" level=info msg="Updating node ci-t8vq2-1497633967 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.418Z" level=info msg="Updating node ci-t8vq2-3806784823 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.418Z" level=info msg="Setting node ci-t8vq2-743674290 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.418Z" level=info msg="Updating node ci-t8vq2-743674290 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.418Z" level=info msg="Updating node ci-t8vq2-1156673745 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.418Z" level=info msg="Updating node ci-t8vq2-2135555749 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.418Z" level=info msg="Updating node ci-t8vq2-1087124617 status Pending -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.419Z" level=info msg="Updating node ci-t8vq2-2221870794 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:36:56.564Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=118736 workflow=ci-t8vq2
time="2021-03-02T00:36:56.620Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-1087124617/labelPodCompleted
time="2021-03-02T00:36:56.620Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-2529876561/labelPodCompleted
time="2021-03-02T00:37:06.639Z" level=info msg="Processing workflow" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.639Z" level=info msg="Setting node ci-t8vq2-2687447728 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.639Z" level=info msg="Updating node ci-t8vq2-2687447728 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.639Z" level=info msg="Updating node ci-t8vq2-2620463353 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.642Z" level=info msg="Updating node ci-t8vq2-2921795641 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.643Z" level=info msg="Updating node ci-t8vq2-935633491 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.644Z" level=info msg="Updating node ci-t8vq2-139394661 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.644Z" level=info msg="Updating node ci-t8vq2-1466005665 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.645Z" level=info msg="Setting node ci-t8vq2-924300910 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.645Z" level=info msg="Updating node ci-t8vq2-924300910 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.645Z" level=info msg="Updating node ci-t8vq2-3806784823 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.647Z" level=info msg="Setting node ci-t8vq2-2653728782 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.647Z" level=info msg="Updating node ci-t8vq2-2653728782 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.648Z" level=info msg="Setting node ci-t8vq2-2998710952 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.648Z" level=info msg="Updating node ci-t8vq2-2998710952 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.649Z" level=info msg="Setting node ci-t8vq2-2454096578 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.650Z" level=info msg="Updating node ci-t8vq2-2454096578 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.651Z" level=info msg="Setting node ci-t8vq2-2027865656 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.651Z" level=info msg="Updating node ci-t8vq2-2027865656 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.653Z" level=info msg="Setting node ci-t8vq2-553546802 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.653Z" level=info msg="Updating node ci-t8vq2-553546802 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.655Z" level=info msg="Updating node ci-t8vq2-1958506170 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.655Z" level=info msg="Updating node ci-t8vq2-2460421745 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.655Z" level=info msg="Setting node ci-t8vq2-569544386 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.656Z" level=info msg="Updating node ci-t8vq2-569544386 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.656Z" level=info msg="Updating node ci-t8vq2-3097822101 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.656Z" level=info msg="Updating node ci-t8vq2-3752440098 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:06.756Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=118850 workflow=ci-t8vq2
time="2021-03-02T00:37:06.778Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-2620463353/labelPodCompleted
time="2021-03-02T00:37:06.778Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-139394661/labelPodCompleted
time="2021-03-02T00:37:06.779Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-1466005665/labelPodCompleted
time="2021-03-02T00:37:06.780Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-2460421745/labelPodCompleted
time="2021-03-02T00:37:06.860Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-3752440098/labelPodCompleted
time="2021-03-02T00:37:06.862Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-2921795641/labelPodCompleted
time="2021-03-02T00:37:06.878Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-935633491/labelPodCompleted
time="2021-03-02T00:37:06.881Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-3806784823/labelPodCompleted
time="2021-03-02T00:37:06.909Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-1958506170/labelPodCompleted
time="2021-03-02T00:37:06.910Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-3097822101/labelPodCompleted
time="2021-03-02T00:37:16.763Z" level=info msg="Processing workflow" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.764Z" level=info msg="Updating node ci-t8vq2-1671854890 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.765Z" level=info msg="Updating node ci-t8vq2-2189885778 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.765Z" level=info msg="Updating node ci-t8vq2-3303001669 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.765Z" level=info msg="Setting node ci-t8vq2-441930226 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.766Z" level=info msg="Updating node ci-t8vq2-441930226 status Pending -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.766Z" level=info msg="Updating node ci-t8vq2-924300910 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.766Z" level=info msg="Updating node ci-t8vq2-2608532967 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.766Z" level=info msg="Updating node ci-t8vq2-2454096578 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.767Z" level=info msg="Updating node ci-t8vq2-2517646915 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.767Z" level=info msg="Updating node ci-t8vq2-2687447728 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.767Z" level=info msg="Updating node ci-t8vq2-2221870794 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.768Z" level=info msg="Setting node ci-t8vq2-3144997424 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.768Z" level=info msg="Updating node ci-t8vq2-3144997424 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.768Z" level=info msg="Setting node ci-t8vq2-212400199 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.768Z" level=info msg="Updating node ci-t8vq2-212400199 status Pending -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.768Z" level=info msg="Updating node ci-t8vq2-553546802 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.769Z" level=info msg="Updating node ci-t8vq2-1156673745 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.769Z" level=info msg="Updating node ci-t8vq2-2594947737 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.770Z" level=info msg="Updating node ci-t8vq2-569544386 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.770Z" level=info msg="Updating node ci-t8vq2-2135555749 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.771Z" level=info msg="Updating node ci-t8vq2-3834947628 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.771Z" level=info msg="Updating node ci-t8vq2-2998710952 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.772Z" level=info msg="Updating node ci-t8vq2-13895053 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.772Z" level=info msg="Updating node ci-t8vq2-2653728782 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.772Z" level=info msg="Updating node ci-t8vq2-2699866519 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.772Z" level=info msg="Updating node ci-t8vq2-2027865656 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.798Z" level=info msg="TaskGroup node ci-t8vq2-229177818 initialized Running (message: )" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.798Z" level=info msg="All of node ci-t8vq2.task2(0:context:A) dependencies [task1] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.800Z" level=info msg="Pod node ci-t8vq2-857446365 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.819Z" level=info msg="Created pod: ci-t8vq2.task2(0:context:A) (ci-t8vq2-857446365)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.820Z" level=info msg="All of node ci-t8vq2.task2(1:context:B) dependencies [task1] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.821Z" level=info msg="Pod node ci-t8vq2-1089779475 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.845Z" level=info msg="Created pod: ci-t8vq2.task2(1:context:B) (ci-t8vq2-1089779475)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.845Z" level=info msg="All of node ci-t8vq2.task2(2:context:C) dependencies [task1] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.849Z" level=info msg="Pod node ci-t8vq2-1533362641 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.889Z" level=info msg="Created pod: ci-t8vq2.task2(2:context:C) (ci-t8vq2-1533362641)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.890Z" level=info msg="All of node ci-t8vq2.task2(3:context:D) dependencies [task1] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.894Z" level=info msg="Pod node ci-t8vq2-2193859387 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.910Z" level=info msg="Created pod: ci-t8vq2.task2(3:context:D) (ci-t8vq2-2193859387)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.911Z" level=info msg="All of node ci-t8vq2.task2(4:context:E) dependencies [task1] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.914Z" level=info msg="Pod node ci-t8vq2-2924122029 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.936Z" level=info msg="Created pod: ci-t8vq2.task2(4:context:E) (ci-t8vq2-2924122029)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.936Z" level=info msg="All of node ci-t8vq2.task3 dependencies [task1] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.937Z" level=info msg="Pod node ci-t8vq2-245955437 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.970Z" level=info msg="Created pod: ci-t8vq2.task3 (ci-t8vq2-245955437)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.971Z" level=info msg="TaskGroup node ci-t8vq2-128512104 initialized Running (message: )" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.971Z" level=info msg="All of node ci-t8vq2.task4(0:context:A) dependencies [task1] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:16.972Z" level=info msg="Pod node ci-t8vq2-3119937059 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.015Z" level=info msg="Created pod: ci-t8vq2.task4(0:context:A) (ci-t8vq2-3119937059)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.015Z" level=info msg="All of node ci-t8vq2.task4(1:context:B) dependencies [task1] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.016Z" level=info msg="Pod node ci-t8vq2-3119595669 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.039Z" level=info msg="Created pod: ci-t8vq2.task4(1:context:B) (ci-t8vq2-3119595669)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.039Z" level=info msg="All of node ci-t8vq2.task4(2:context:C) dependencies [task1] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.040Z" level=info msg="Pod node ci-t8vq2-418065567 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.069Z" level=info msg="Created pod: ci-t8vq2.task4(2:context:C) (ci-t8vq2-418065567)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.070Z" level=info msg="All of node ci-t8vq2.task4(3:context:D) dependencies [task1] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.072Z" level=info msg="Pod node ci-t8vq2-2120215045 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.103Z" level=info msg="Created pod: ci-t8vq2.task4(3:context:D) (ci-t8vq2-2120215045)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.104Z" level=info msg="All of node ci-t8vq2.task4(4:context:E) dependencies [task1] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.104Z" level=info msg="Pod node ci-t8vq2-4250566955 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.124Z" level=info msg="Created pod: ci-t8vq2.task4(4:context:E) (ci-t8vq2-4250566955)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.125Z" level=info msg="All of node ci-t8vq2.task5 dependencies [task1] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.127Z" level=info msg="Pod node ci-t8vq2-145289723 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.171Z" level=info msg="Created pod: ci-t8vq2.task5 (ci-t8vq2-145289723)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.175Z" level=info msg="TaskGroup node ci-t8vq2-162067342 initialized Running (message: )" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.175Z" level=info msg="All of node ci-t8vq2.task6(0:context:A) dependencies [task1] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.176Z" level=info msg="Pod node ci-t8vq2-1642984369 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.262Z" level=info msg="Created pod: ci-t8vq2.task6(0:context:A) (ci-t8vq2-1642984369)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.262Z" level=info msg="All of node ci-t8vq2.task6(1:context:B) dependencies [task1] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.274Z" level=info msg="Pod node ci-t8vq2-1325696727 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.337Z" level=info msg="Created pod: ci-t8vq2.task6(1:context:B) (ci-t8vq2-1325696727)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.337Z" level=info msg="All of node ci-t8vq2.task6(2:context:C) dependencies [task1] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.340Z" level=info msg="Pod node ci-t8vq2-3992325445 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.423Z" level=info msg="Created pod: ci-t8vq2.task6(2:context:C) (ci-t8vq2-3992325445)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.423Z" level=info msg="All of node ci-t8vq2.task6(3:context:D) dependencies [task1] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.437Z" level=info msg="Pod node ci-t8vq2-3744868327 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.497Z" level=info msg="Created pod: ci-t8vq2.task6(3:context:D) (ci-t8vq2-3744868327)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.497Z" level=info msg="All of node ci-t8vq2.task6(4:context:E) dependencies [task1] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.518Z" level=info msg="Pod node ci-t8vq2-1719834657 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.587Z" level=info msg="Created pod: ci-t8vq2.task6(4:context:E) (ci-t8vq2-1719834657)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.587Z" level=info msg="All of node ci-t8vq2.task7 dependencies [task1] completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.587Z" level=info msg="Pod node ci-t8vq2-178844961 initialized Pending" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.607Z" level=info msg="Created pod: ci-t8vq2.task7 (ci-t8vq2-178844961)" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:17.718Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=119020 workflow=ci-t8vq2
time="2021-03-02T00:37:17.779Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-2454096578/labelPodCompleted
time="2021-03-02T00:37:17.785Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-2687447728/labelPodCompleted
time="2021-03-02T00:37:17.789Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-2221870794/labelPodCompleted
time="2021-03-02T00:37:17.795Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-924300910/labelPodCompleted
time="2021-03-02T00:37:17.831Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-2608532967/labelPodCompleted
time="2021-03-02T00:37:17.848Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-553546802/labelPodCompleted
time="2021-03-02T00:37:17.868Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-2135555749/labelPodCompleted
time="2021-03-02T00:37:17.868Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-3834947628/labelPodCompleted
time="2021-03-02T00:37:17.910Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-13895053/labelPodCompleted
time="2021-03-02T00:37:17.911Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-2699866519/labelPodCompleted
time="2021-03-02T00:37:17.976Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-1671854890/labelPodCompleted
time="2021-03-02T00:37:17.977Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-2189885778/labelPodCompleted
time="2021-03-02T00:37:17.986Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-3303001669/labelPodCompleted
time="2021-03-02T00:37:18.022Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-441930226/labelPodCompleted
time="2021-03-02T00:37:18.056Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-2594947737/labelPodCompleted
time="2021-03-02T00:37:18.073Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-2517646915/labelPodCompleted
time="2021-03-02T00:37:18.103Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-1156673745/labelPodCompleted
time="2021-03-02T00:37:18.074Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-212400199/labelPodCompleted
time="2021-03-02T00:37:18.105Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-569544386/labelPodCompleted
time="2021-03-02T00:37:18.120Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-2998710952/labelPodCompleted
time="2021-03-02T00:37:18.182Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-2653728782/labelPodCompleted
time="2021-03-02T00:37:18.204Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-2027865656/labelPodCompleted
time="2021-03-02T00:37:26.829Z" level=info msg="Processing workflow" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.830Z" level=info msg="Updating node ci-t8vq2-743674290 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.830Z" level=info msg="Updating node ci-t8vq2-857446365 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.830Z" level=info msg="Updating node ci-t8vq2-1089779475 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.830Z" level=info msg="Updating node ci-t8vq2-1533362641 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.830Z" level=info msg="Updating node ci-t8vq2-1497633967 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.830Z" level=info msg="Updating node ci-t8vq2-1642984369 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.830Z" level=info msg="Updating node ci-t8vq2-1719834657 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.832Z" level=info msg="Updating node ci-t8vq2-2193859387 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.832Z" level=info msg="Updating node ci-t8vq2-2924122029 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.832Z" level=info msg="Updating node ci-t8vq2-3992325445 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.832Z" level=info msg="Updating node ci-t8vq2-3144997424 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.832Z" level=info msg="Updating node ci-t8vq2-1325696727 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.833Z" level=info msg="Updating node ci-t8vq2-3815346741 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.833Z" level=info msg="Updating node ci-t8vq2-4250566955 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.833Z" level=info msg="Updating node ci-t8vq2-3744868327 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.833Z" level=info msg="Updating node ci-t8vq2-145289723 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.833Z" level=info msg="Updating node ci-t8vq2-3119595669 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.833Z" level=info msg="Updating node ci-t8vq2-245955437 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.833Z" level=info msg="Updating node ci-t8vq2-2120215045 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.833Z" level=info msg="Updating node ci-t8vq2-3045333153 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.833Z" level=info msg="Updating node ci-t8vq2-418065567 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.834Z" level=info msg="Updating node ci-t8vq2-3119937059 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.834Z" level=info msg="Updating node ci-t8vq2-178844961 message: ContainerCreating" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.904Z" level=info msg="node ci-t8vq2-28546012 phase Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:26.915Z" level=info msg="node ci-t8vq2-28546012 finished: 2021-03-02 00:37:26.9150578 +0000 UTC" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:27.012Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=119139 workflow=ci-t8vq2
time="2021-03-02T00:37:27.050Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-743674290/labelPodCompleted
time="2021-03-02T00:37:27.060Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-1497633967/labelPodCompleted
time="2021-03-02T00:37:27.061Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-3144997424/labelPodCompleted
time="2021-03-02T00:37:27.063Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-3815346741/labelPodCompleted
time="2021-03-02T00:37:27.115Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-3045333153/labelPodCompleted
time="2021-03-02T00:37:37.073Z" level=info msg="Processing workflow" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.073Z" level=info msg="Setting node ci-t8vq2-3744868327 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.073Z" level=info msg="Updating node ci-t8vq2-3744868327 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.073Z" level=info msg="Setting node ci-t8vq2-1325696727 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.073Z" level=info msg="Updating node ci-t8vq2-1325696727 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.074Z" level=info msg="Updating node ci-t8vq2-1642984369 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.074Z" level=info msg="Setting node ci-t8vq2-245955437 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.074Z" level=info msg="Updating node ci-t8vq2-245955437 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.074Z" level=info msg="Setting node ci-t8vq2-1089779475 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.074Z" level=info msg="Updating node ci-t8vq2-1089779475 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.074Z" level=info msg="Setting node ci-t8vq2-3119937059 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.074Z" level=info msg="Updating node ci-t8vq2-3119937059 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.074Z" level=info msg="Updating node ci-t8vq2-418065567 status Pending -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.074Z" level=info msg="Setting node ci-t8vq2-1719834657 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.074Z" level=info msg="Updating node ci-t8vq2-1719834657 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.075Z" level=info msg="Setting node ci-t8vq2-178844961 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.075Z" level=info msg="Updating node ci-t8vq2-178844961 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.075Z" level=info msg="Pod failed: Error (exit code 1): Could not get container status" displayName="task2(2:context:C)" namespace=argo pod=ci-t8vq2-1533362641 templateName= workflow=ci-t8vq2
time="2021-03-02T00:37:37.075Z" level=info msg="Updating node ci-t8vq2-1533362641 status Running -> Error" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.075Z" level=info msg="Updating node ci-t8vq2-1533362641 message: Error (exit code 1): Could not get container status" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.075Z" level=info msg="Updating node ci-t8vq2-2924122029 status Pending -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.075Z" level=info msg="Setting node ci-t8vq2-2193859387 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.075Z" level=info msg="Updating node ci-t8vq2-2193859387 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.075Z" level=info msg="Setting node ci-t8vq2-145289723 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.076Z" level=info msg="Updating node ci-t8vq2-145289723 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.076Z" level=info msg="Setting node ci-t8vq2-4250566955 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.076Z" level=info msg="Updating node ci-t8vq2-4250566955 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.076Z" level=info msg="Setting node ci-t8vq2-3119595669 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.076Z" level=info msg="Updating node ci-t8vq2-3119595669 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.076Z" level=info msg="Setting node ci-t8vq2-857446365 outputs" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.076Z" level=info msg="Updating node ci-t8vq2-857446365 status Pending -> Running" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.076Z" level=info msg="Updating node ci-t8vq2-3992325445 status Pending -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.076Z" level=info msg="Updating node ci-t8vq2-2120215045 status Pending -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:37.141Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=119254 workflow=ci-t8vq2
time="2021-03-02T00:37:37.161Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-1533362641/labelPodCompleted
time="2021-03-02T00:37:37.161Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-2924122029/labelPodCompleted
time="2021-03-02T00:37:37.162Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-3992325445/labelPodCompleted
time="2021-03-02T00:37:37.163Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-2120215045/labelPodCompleted
time="2021-03-02T00:37:37.198Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-418065567/labelPodCompleted
time="2021-03-02T00:37:47.113Z" level=info msg="Processing workflow" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.115Z" level=info msg="Updating node ci-t8vq2-3119595669 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.115Z" level=info msg="Updating node ci-t8vq2-3744868327 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.116Z" level=info msg="Updating node ci-t8vq2-1642984369 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.117Z" level=info msg="Updating node ci-t8vq2-2193859387 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.118Z" level=info msg="Updating node ci-t8vq2-4250566955 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.118Z" level=info msg="Updating node ci-t8vq2-1325696727 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.119Z" level=info msg="Updating node ci-t8vq2-857446365 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.119Z" level=info msg="Updating node ci-t8vq2-245955437 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.121Z" level=info msg="Updating node ci-t8vq2-1089779475 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.123Z" level=info msg="Updating node ci-t8vq2-178844961 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.123Z" level=info msg="Updating node ci-t8vq2-145289723 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.124Z" level=info msg="Updating node ci-t8vq2-3119937059 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.124Z" level=info msg="Updating node ci-t8vq2-1719834657 status Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.136Z" level=info msg="node ci-t8vq2-229177818 phase Running -> Error" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.136Z" level=info msg="node ci-t8vq2-229177818 finished: 2021-03-02 00:37:47.1368811 +0000 UTC" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.138Z" level=info msg="node ci-t8vq2-128512104 phase Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.138Z" level=info msg="node ci-t8vq2-128512104 finished: 2021-03-02 00:37:47.1385009 +0000 UTC" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.140Z" level=info msg="node ci-t8vq2-162067342 phase Running -> Succeeded" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.140Z" level=info msg="node ci-t8vq2-162067342 finished: 2021-03-02 00:37:47.1409097 +0000 UTC" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.141Z" level=info msg="Outbound nodes of ci-t8vq2 set to [ci-t8vq2-2699866519 ci-t8vq2-2921795641 ci-t8vq2-935633491 ci-t8vq2-2460421745 ci-t8vq2-1497633967 ci-t8vq2-2529876561 ci-t8vq2-2517646915 ci-t8vq2-2594947737 ci-t8vq2-2608532967 ci-t8vq2-3834947628 ci-t8vq2-1156673745 ci-t8vq2-2620463353 ci-t8vq2-3303001669 ci-t8vq2-139394661 ci-t8vq2-1087124617 ci-t8vq2-1466005665 ci-t8vq2-3097822101 ci-t8vq2-3815346741 ci-t8vq2-3045333153 ci-t8vq2-2135555749 ci-t8vq2-1671854890 ci-t8vq2-2189885778 ci-t8vq2-2454096578 ci-t8vq2-3752440098 ci-t8vq2-1958506170 ci-t8vq2-569544386 ci-t8vq2-743674290 ci-t8vq2-553546802 ci-t8vq2-3806784823 ci-t8vq2-13895053 ci-t8vq2-2653728782 ci-t8vq2-3144997424 ci-t8vq2-2221870794 ci-t8vq2-2998710952 ci-t8vq2-924300910 ci-t8vq2-2687447728 ci-t8vq2-441930226 ci-t8vq2-2027865656 ci-t8vq2-857446365 ci-t8vq2-1089779475 ci-t8vq2-1533362641 ci-t8vq2-2193859387 ci-t8vq2-2924122029 ci-t8vq2-245955437 ci-t8vq2-3119937059 ci-t8vq2-3119595669 ci-t8vq2-418065567 ci-t8vq2-2120215045 ci-t8vq2-4250566955 ci-t8vq2-145289723 ci-t8vq2-1642984369 ci-t8vq2-1325696727 ci-t8vq2-3992325445 ci-t8vq2-3744868327 ci-t8vq2-1719834657 ci-t8vq2-178844961]" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.141Z" level=info msg="node ci-t8vq2 phase Running -> Error" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.141Z" level=info msg="node ci-t8vq2 finished: 2021-03-02 00:37:47.1418331 +0000 UTC" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.142Z" level=info msg="Checking daemoned children of ci-t8vq2" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.142Z" level=info msg="Updated phase Running -> Error" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.142Z" level=info msg="Marking workflow completed" namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.142Z" level=info msg="Checking daemoned children of " namespace=argo workflow=ci-t8vq2
time="2021-03-02T00:37:47.182Z" level=info msg="Workflow update successful" namespace=argo phase=Error resourceVersion=119304 workflow=ci-t8vq2
time="2021-03-02T00:37:47.206Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-1642984369/labelPodCompleted
time="2021-03-02T00:37:47.207Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-1325696727/labelPodCompleted
time="2021-03-02T00:37:47.206Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-2193859387/labelPodCompleted
time="2021-03-02T00:37:47.206Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-4250566955/labelPodCompleted
time="2021-03-02T00:37:47.241Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-857446365/labelPodCompleted
time="2021-03-02T00:37:47.264Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-245955437/labelPodCompleted
time="2021-03-02T00:37:47.274Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-3119595669/labelPodCompleted
time="2021-03-02T00:37:47.305Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-3744868327/labelPodCompleted
time="2021-03-02T00:37:47.313Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-145289723/labelPodCompleted
time="2021-03-02T00:37:47.320Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-3119937059/labelPodCompleted
time="2021-03-02T00:37:47.335Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-1719834657/labelPodCompleted
time="2021-03-02T00:37:47.349Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-1089779475/labelPodCompleted
time="2021-03-02T00:37:47.349Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-t8vq2-178844961/labelPodCompleted

kubectl logs ci-t8vq2-1533362641 -n argo -c wait

time="2021-03-02T00:37:22.325Z" level=info msg="secured root for pid 24 root: runc:[2:INIT]"
time="2021-03-02T00:37:22.327Z" level=info msg="mapped pid 24 to container ID \"42af41ccd93a5ced82d44e0230c07f3cd0592fc368e3694ee8ff400ce257a0b4\""
time="2021-03-02T00:37:22.329Z" level=info msg="mapped container name \"main\" to container ID \"42af41ccd93a5ced82d44e0230c07f3cd0592fc368e3694ee8ff400ce257a0b4\" and pid 24"
time="2021-03-02T00:37:23.568Z" level=info msg="Waiting for \"main\" pid 24 to complete"
time="2021-03-02T00:37:23.568Z" level=info msg="\"main\" pid 24 completed"
time="2021-03-02T00:37:23.568Z" level=info msg="Main container completed"
time="2021-03-02T00:37:23.568Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2021-03-02T00:37:23.569Z" level=info msg="Capturing script exit code"
time="2021-03-02T00:37:23.569Z" level=info msg="Getting exit code of main"
time="2021-03-02T00:37:23.612Z" level=info msg="Get pods 200"
time="2021-03-02T00:37:23.799Z" level=error msg="executor error: Could not get container status\ngithub.com/argoproj/argo-workflows/v3/errors.Wrap\n\t/go/src/github.com/argoproj/argo-workflows/errors/errors.go:88\ngithub.com/argoproj/argo-workflows/v3/errors.InternalWrapError\n\t/go/src/github.com/argoproj/argo-workflows/errors/errors.go:73\ngithub.com/argoproj/argo-workflows/v3/workflow/executor/k8sapi.(*K8sAPIExecutor).GetExitCode\n\t/go/src/github.com/argoproj/argo-workflows/workflow/executor/k8sapi/k8sapi.go:50\ngithub.com/argoproj/argo-workflows/v3/workflow/executor.(*WorkflowExecutor).CaptureScriptExitCode\n\t/go/src/github.com/argoproj/argo-workflows/workflow/executor/executor.go:714\ngithub.com/argoproj/argo-workflows/v3/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo-workflows/cmd/argoexec/commands/wait.go:55\ngithub.com/argoproj/argo-workflows/v3/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo-workflows/cmd/argoexec/commands/wait.go:18\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:846\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\t/go/src/github.com/argoproj/argo-workflows/cmd/argoexec/main.go:14\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1374"
time="2021-03-02T00:37:23.799Z" level=info msg="Alloc=6303 TotalAlloc=11141 Sys=73041 NumGC=3 Goroutines=10"
time="2021-03-02T00:37:23.812Z" level=fatal msg="Could not get container status\ngithub.com/argoproj/argo-workflows/v3/errors.Wrap\n\t/go/src/github.com/argoproj/argo-workflows/errors/errors.go:88\ngithub.com/argoproj/argo-workflows/v3/errors.InternalWrapError\n\t/go/src/github.com/argoproj/argo-workflows/errors/errors.go:73\ngithub.com/argoproj/argo-workflows/v3/workflow/executor/k8sapi.(*K8sAPIExecutor).GetExitCode\n\t/go/src/github.com/argoproj/argo-workflows/workflow/executor/k8sapi/k8sapi.go:50\ngithub.com/argoproj/argo-workflows/v3/workflow/executor.(*WorkflowExecutor).CaptureScriptExitCode\n\t/go/src/github.com/argoproj/argo-workflows/workflow/executor/executor.go:714\ngithub.com/argoproj/argo-workflows/v3/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo-workflows/cmd/argoexec/commands/wait.go:55\ngithub.com/argoproj/argo-workflows/v3/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo-workflows/cmd/argoexec/commands/wait.go:18\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:846\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\t/go/src/github.com/argoproj/argo-workflows/cmd/argoexec/main.go:14\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1374"

image

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.
@alexec
Copy link
Contributor

alexec commented Mar 2, 2021

How odd. You can clearly see Get pods 200 which meant the request was OK.

@alexec alexec added type/regression Regression from previous behavior (a specific type of bug) executor/k8sapi labels Mar 2, 2021
@alexec alexec self-assigned this Mar 2, 2021
@alexec alexec added executor/k8sapi type/regression Regression from previous behavior (a specific type of bug) and removed executor/k8sapi type/regression Regression from previous behavior (a specific type of bug) labels Mar 2, 2021
alexec added a commit to alexec/argo-workflows that referenced this issue Mar 2, 2021
@alexec
feat(executor): Move exit code capture to controller. See argoproj#5251
3086e0f 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
@alexec alexec mentioned this issue Mar 2, 2021
Closed
alexec added a commit to alexec/argo-workflows that referenced this issue Mar 2, 2021
@alexec
feat(executor): Move exit code capture to controller. See argoproj#5251
a603716 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
@alexec alexec linked a pull request Mar 2, 2021 that will close this issue
feat(executor): Move exit code capture to controller. See #5251 #5254
Closed
alexec added a commit that referenced this issue Mar 2, 2021
@alexec
feat(executor): Move exit code capture to controller. See #5251
e2189b1 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
@alexec alexec mentioned this issue Mar 2, 2021
Closed
alexec added a commit that referenced this issue Mar 2, 2021
@alexec
feat(executor): Move exit code capture to controller. See #5251
af38baa 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
alexec added a commit that referenced this issue Mar 2, 2021
@alexec
feat(executor): Move exit code capture to controller. See #5251
f7fcfa4 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
alexec added a commit that referenced this issue Mar 2, 2021
@alexec
feat(executor): Move exit code capture to controller. See #5251
4e97d2a 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
@caueasantos caueasantos mentioned this issue Mar 2, 2021
Closed
alexec added a commit that referenced this issue Mar 2, 2021
@alexec
feat(executor): Move exit code capture to controller. See #5251
8a06669 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
alexec added a commit that referenced this issue Mar 2, 2021
@alexec
feat(executor): Move exit code capture to controller. See #5251
e13b1ae 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
alexec added a commit that referenced this issue Mar 2, 2021
@alexec
feat(executor): Move exit code capture to controller. See #5251
6122150 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
alexec added a commit that referenced this issue Mar 2, 2021
@alexec
chore(executor): Fix error translation. See #5251
11280af 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
@alexec alexec mentioned this issue Mar 2, 2021
Closed
@alexec alexec linked a pull request Mar 2, 2021 that will close this issue
chore(executor): Fix error translation. See #5251 #5263
Closed
@alexec
Copy link
Contributor

alexec commented Mar 2, 2021

I've created an engineer build for you that has better logging argoproj/argoexec:dev-status. Could I please ask you to run it?

@caueasantos
Copy link
Contributor Author

Let me try that - Give me a few minutes

@caueasantos
Copy link
Contributor Author

I've created an engineer build for you that has better logging argoproj/argoexec:dev-status. Could I please ask you to run it?

What did I do?

Changed executor image (Please let me know if I need to change something else)

    spec:
      containers:
      - args:
        - --configmap
        - workflow-controller-configmap
        - --executor-image
        - argoproj/argoexec:dev-status
        command:
        - workflow-controller
        env:
        - name: LEADER_ELECTION_IDENTITY
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        image: argoproj/workflow-controller:v3.0.0-rc3
        imagePullPolicy: IfNotPresent

Reran test workflow

image

All failed tasks returned the same error message

image

Container main

kubectl logs pod/ci-n7gdl-951919243 -n argo -c main
task2

Container wait

time="2021-03-02T23:37:03.191Z" level=info msg="Starting Workflow Executor" executorType=pns version="{latest+11280af.dirty 2021-03-02T22:41:00Z 11280af18311039859dec6e9f391212803d3dd07 11280af18311039859dec6e9f391212803d3dd07 dirty go1.15.7 gc linux/amd64}"
time="2021-03-02T23:37:03.227Z" level=info msg="Creating PNS executor (namespace: argo, pod: ci-n7gdl-951919243, pid: 6)"
time="2021-03-02T23:37:03.227Z" level=info msg="Creating a K8sAPI executor"
time="2021-03-02T23:37:03.227Z" level=info msg="Executor (version: latest+11280af.dirty, build_date: 2021-03-02T22:41:00Z) initialized (pod: argo/ci-n7gdl-951919243) with template:\n{\"name\":\"print\",\"inputs\":{\"parameters\":[{\"name\":\"message\",\"value\":\"task2\"}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"curlimages/curl:7.75.0\",\"command\":[\"sh\",\"-c\"],\"args\":[\"echo task2\"],\"resources\":{}}}"
time="2021-03-02T23:37:03.227Z" level=info msg="Starting annotations monitor"
time="2021-03-02T23:37:03.229Z" level=info msg="secured root for pid 21 root: runc:[2:INIT]"
time="2021-03-02T23:37:03.237Z" level=info msg="Starting deadline monitor"
time="2021-03-02T23:37:03.229Z" level=info msg="mapped pid 21 to container ID \"023a1cd1afa236a90e182a63fa738f18742bebaf0984bdd607b187c9205e4bdb\""
time="2021-03-02T23:37:03.240Z" level=warning msg="failed to secure root file handle for 21" error="container name not found"
time="2021-03-02T23:37:03.304Z" level=info msg="secured root for pid 21 root: runc:[2:INIT]"
time="2021-03-02T23:37:03.305Z" level=info msg="mapped pid 21 to container ID \"023a1cd1afa236a90e182a63fa738f18742bebaf0984bdd607b187c9205e4bdb\""
time="2021-03-02T23:37:03.305Z" level=warning msg="failed to secure root file handle for 21" error="container name not found"
time="2021-03-02T23:37:03.356Z" level=info msg="secured root for pid 21 root: runc:[2:INIT]"
time="2021-03-02T23:37:03.356Z" level=info msg="mapped pid 21 to container ID \"023a1cd1afa236a90e182a63fa738f18742bebaf0984bdd607b187c9205e4bdb\""
time="2021-03-02T23:37:03.357Z" level=warning msg="failed to secure root file handle for 21" error="container name not found"
time="2021-03-02T23:37:03.408Z" level=info msg="secured root for pid 21 root: runc:[2:INIT]"
time="2021-03-02T23:37:03.409Z" level=info msg="mapped pid 21 to container ID \"023a1cd1afa236a90e182a63fa738f18742bebaf0984bdd607b187c9205e4bdb\""
time="2021-03-02T23:37:03.409Z" level=warning msg="failed to secure root file handle for 21" error="container name not found"
time="2021-03-02T23:37:03.460Z" level=info msg="secured root for pid 21 root: sh"
time="2021-03-02T23:37:03.461Z" level=info msg="mapped pid 21 to container ID \"023a1cd1afa236a90e182a63fa738f18742bebaf0984bdd607b187c9205e4bdb\""
time="2021-03-02T23:37:03.461Z" level=info msg="mapped container name \"main\" to container ID \"023a1cd1afa236a90e182a63fa738f18742bebaf0984bdd607b187c9205e4bdb\" and pid 21"
time="2021-03-02T23:37:06.228Z" level=info msg="Waiting for \"main\" pid 21 to complete"
time="2021-03-02T23:37:06.229Z" level=info msg="\"main\" pid 21 completed"
time="2021-03-02T23:37:06.230Z" level=info msg="Main container completed"
time="2021-03-02T23:37:06.230Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2021-03-02T23:37:06.230Z" level=info msg="Capturing script exit code"
time="2021-03-02T23:37:06.230Z" level=info msg="Getting exit code of main"
time="2021-03-02T23:37:06.247Z" level=info msg="Get pods 200"
time="2021-03-02T23:37:06.259Z" level=error msg="executor error: container \"main\" is not found in the pod ci-n7gdl-951919243\ngithub.com/argoproj/argo-workflows/v3/errors.New\n\t/go/src/github.com/argoproj/argo-workflows/errors/errors.go:49\ngithub.com/argoproj/argo-workflows/v3/workflow/executor/k8sapi.(*k8sAPIClient).GetContainerStatus\n\t/go/src/github.com/argoproj/argo-workflows/workflow/executor/k8sapi/client.go:88\ngithub.com/argoproj/argo-workflows/v3/workflow/executor/k8sapi.(*K8sAPIExecutor).GetExitCode\n\t/go/src/github.com/argoproj/argo-workflows/workflow/executor/k8sapi/k8sapi.go:48\ngithub.com/argoproj/argo-workflows/v3/workflow/executor.(*WorkflowExecutor).CaptureScriptExitCode\n\t/go/src/github.com/argoproj/argo-workflows/workflow/executor/executor.go:719\ngithub.com/argoproj/argo-workflows/v3/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo-workflows/cmd/argoexec/commands/wait.go:55\ngithub.com/argoproj/argo-workflows/v3/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo-workflows/cmd/argoexec/commands/wait.go:18\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:846\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\t/go/src/github.com/argoproj/argo-workflows/cmd/argoexec/main.go:15\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1374"
time="2021-03-02T23:37:06.259Z" level=info msg="Alloc=5975 TotalAlloc=10381 Sys=73041 NumGC=3 Goroutines=9"
time="2021-03-02T23:37:06.260Z" level=fatal msg="container \"main\" is not found in the pod ci-n7gdl-951919243\ngithub.com/argoproj/argo-workflows/v3/errors.New\n\t/go/src/github.com/argoproj/argo-workflows/errors/errors.go:49\ngithub.com/argoproj/argo-workflows/v3/workflow/executor/k8sapi.(*k8sAPIClient).GetContainerStatus\n\t/go/src/github.com/argoproj/argo-workflows/workflow/executor/k8sapi/client.go:88\ngithub.com/argoproj/argo-workflows/v3/workflow/executor/k8sapi.(*K8sAPIExecutor).GetExitCode\n\t/go/src/github.com/argoproj/argo-workflows/workflow/executor/k8sapi/k8sapi.go:48\ngithub.com/argoproj/argo-workflows/v3/workflow/executor.(*WorkflowExecutor).CaptureScriptExitCode\n\t/go/src/github.com/argoproj/argo-workflows/workflow/executor/executor.go:719\ngithub.com/argoproj/argo-workflows/v3/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo-workflows/cmd/argoexec/commands/wait.go:55\ngithub.com/argoproj/argo-workflows/v3/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo-workflows/cmd/argoexec/commands/wait.go:18\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:846\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\t/go/src/github.com/argoproj/argo-workflows/cmd/argoexec/main.go:15\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1374"

Controller

time="2021-03-02T23:37:01.335Z" level=info msg="Pod node ci-n7gdl-951919243 initialized Pending" namespace=argo workflow=ci-n7gdl
time="2021-03-02T23:37:01.360Z" level=info msg="Created pod: ci-n7gdl.task7(0:context:A).task2(4:context:E) (ci-n7gdl-951919243)" namespace=argo workflow=ci-n7gdl
time="2021-03-02T23:37:11.235Z" level=info msg="Updating node ci-n7gdl-951919243 status Pending -> Running" namespace=argo workflow=ci-n7gdl
time="2021-03-02T23:37:21.407Z" level=info msg="Pod failed: Error (exit code 1): container \"main\" is not found in the pod ci-n7gdl-951919243" displayName="task2(4:context:E)" namespace=argo pod=ci-n7gdl-951919243 templateName=print workflow=ci-n7gdl
time="2021-03-02T23:37:21.407Z" level=info msg="Updating node ci-n7gdl-951919243 status Running -> Error" namespace=argo workflow=ci-n7gdl
time="2021-03-02T23:37:21.408Z" level=info msg="Updating node ci-n7gdl-951919243 message: Error (exit code 1): container \"main\" is not found in the pod ci-n7gdl-951919243" namespace=argo workflow=ci-n7gdl
time="2021-03-02T23:37:21.452Z" level=info msg="Outbound nodes of ci-n7gdl-1570139223 set to [ci-n7gdl-1061346077 ci-n7gdl-3717971 ci-n7gdl-1100673809 ci-n7gdl-61149051 ci-n7gdl-219171821 ci-n7gdl-1989509763 ci-n7gdl-419908085 ci-n7gdl-2655658495 ci-n7gdl-475980005 ci-n7gdl-951919243 ci-n7gdl-2208085326 ci-n7gdl-3000436176 ci-n7gdl-2937850994 ci-n7gdl-1544997816 ci-n7gdl-2966924662]" namespace=argo workflow=ci-n7gdl
time="2021-03-02T23:37:21.681Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/ci-n7gdl-951919243/labelPodCompleted

@alexec
Copy link
Contributor

alexec commented Mar 3, 2021

That's progress. Can you add kubectl get pod ci-n7gdl-951919243 -o yaml?

alexec added a commit that referenced this issue Mar 3, 2021
@alexec
feat(executor): Move exit code capture to controller. See #5251
24c5055 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
alexec added a commit that referenced this issue Mar 3, 2021
@alexec
feat(executor): Move exit code capture to controller. See #5251
c1e8be3 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
@alexec
Copy link
Contributor

alexec commented Mar 3, 2021

I've push a new version with the changes to address short-running pods. Can you test please?

@caueasantos
Copy link
Contributor Author

I've push a new version with the changes to address short-running pods. Can you test please?

Should I use dev-k8sapi ?

@alexec
Copy link
Contributor

alexec commented Mar 3, 2021

Yes please.

@caueasantos
Copy link
Contributor Author

@alexec Perfect!

image

image

I know that we're stressing the short running scenario here, but the new release candidate version has issues with DAG tasks even when it's not short running. I just thought was a good idea to share this with you right now, please find below one example

image

The logs are pretty the same

time="2021-03-02T00:24:25.604Z" level=info msg="Starting annotations monitor"
time="2021-03-02T00:24:25.605Z" level=info msg="Starting deadline monitor"
time="2021-03-02T00:24:28.605Z" level=info msg="container PID still unknown (maybe due to short running main container)"
time="2021-03-02T00:24:28.615Z" level=info msg="Watch pods 200"
time="2021-03-02T00:24:28.625Z" level=info msg="mapped container name \"main\" to container ID \"\""
time="2021-03-02T00:24:28.625Z" level=info msg="mapped container name \"mysql\" to container ID \"\""
time="2021-03-02T00:24:28.625Z" level=info msg="mapped container name \"nginx\" to container ID \"\""
time="2021-03-02T00:24:28.625Z" level=info msg="mapped container name \"redis\" to container ID \"\""
time="2021-03-02T00:24:28.625Z" level=info msg="mapped container name \"wait\" to container ID \"\""
time="2021-03-02T00:29:14.587Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T00:29:14.587Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T00:29:14.587Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T00:29:14.587Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T00:29:14.587Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T00:29:25.605Z" level=info msg="Alloc=4635 TotalAlloc=48180 Sys=74833 NumGC=15 Goroutines=11"
time="2021-03-02T00:34:25.605Z" level=info msg="Alloc=4577 TotalAlloc=48203 Sys=74833 NumGC=17 Goroutines=11"
time="2021-03-02T00:36:19.557Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T00:36:19.557Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T00:36:19.557Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T00:36:19.557Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T00:36:19.557Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T00:39:25.605Z" level=info msg="Alloc=4559 TotalAlloc=48457 Sys=74833 NumGC=20 Goroutines=11"
time="2021-03-02T00:44:25.605Z" level=info msg="Alloc=4562 TotalAlloc=48478 Sys=74833 NumGC=22 Goroutines=11"
time="2021-03-02T00:49:25.605Z" level=info msg="Alloc=4560 TotalAlloc=48498 Sys=74833 NumGC=25 Goroutines=11"
time="2021-03-02T00:54:25.605Z" level=info msg="Alloc=4562 TotalAlloc=48518 Sys=74833 NumGC=27 Goroutines=11"
time="2021-03-02T00:59:25.605Z" level=info msg="Alloc=4561 TotalAlloc=48539 Sys=74833 NumGC=30 Goroutines=11"
time="2021-03-02T01:04:25.605Z" level=info msg="Alloc=4563 TotalAlloc=48559 Sys=74833 NumGC=32 Goroutines=11"
time="2021-03-02T01:09:25.605Z" level=info msg="Alloc=4561 TotalAlloc=48580 Sys=74833 NumGC=35 Goroutines=11"
time="2021-03-02T01:09:49.663Z" level=info msg="Watch pods 200"
time="2021-03-02T01:09:49.666Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T01:09:49.666Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T01:09:49.666Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T01:09:49.666Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T01:09:49.666Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T01:14:25.605Z" level=info msg="Alloc=4610 TotalAlloc=49072 Sys=74833 NumGC=37 Goroutines=11"
time="2021-03-02T01:19:25.605Z" level=info msg="Alloc=4610 TotalAlloc=49094 Sys=74833 NumGC=40 Goroutines=11"
time="2021-03-02T01:24:25.605Z" level=info msg="Alloc=4612 TotalAlloc=49114 Sys=74833 NumGC=42 Goroutines=11"
time="2021-03-02T01:29:25.605Z" level=info msg="Alloc=4610 TotalAlloc=49134 Sys=74833 NumGC=45 Goroutines=11"
time="2021-03-02T01:34:25.605Z" level=info msg="Alloc=4612 TotalAlloc=49154 Sys=74833 NumGC=47 Goroutines=11"
time="2021-03-02T01:39:25.605Z" level=info msg="Alloc=4611 TotalAlloc=49175 Sys=74833 NumGC=50 Goroutines=11"
time="2021-03-02T01:44:25.605Z" level=info msg="Alloc=4614 TotalAlloc=49197 Sys=74833 NumGC=52 Goroutines=11"
time="2021-03-02T01:44:31.389Z" level=info msg="Watch pods 200"
time="2021-03-02T01:44:31.393Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T01:44:31.393Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T01:44:31.393Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T01:44:31.393Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T01:44:31.393Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T01:49:25.605Z" level=info msg="Alloc=4612 TotalAlloc=49640 Sys=74833 NumGC=55 Goroutines=11"
time="2021-03-02T01:54:25.605Z" level=info msg="Alloc=4614 TotalAlloc=49660 Sys=74833 NumGC=57 Goroutines=11"
time="2021-03-02T01:59:25.605Z" level=info msg="Alloc=4612 TotalAlloc=49680 Sys=74833 NumGC=60 Goroutines=11"
time="2021-03-02T02:04:25.605Z" level=info msg="Alloc=4615 TotalAlloc=49701 Sys=74833 NumGC=62 Goroutines=11"
time="2021-03-02T02:09:25.605Z" level=info msg="Alloc=4613 TotalAlloc=49721 Sys=74833 NumGC=65 Goroutines=11"
time="2021-03-02T02:14:25.605Z" level=info msg="Alloc=4616 TotalAlloc=49741 Sys=74833 NumGC=67 Goroutines=11"
time="2021-03-02T02:19:25.605Z" level=info msg="Alloc=4613 TotalAlloc=49762 Sys=74833 NumGC=70 Goroutines=11"
time="2021-03-02T02:24:25.605Z" level=info msg="Alloc=4616 TotalAlloc=49782 Sys=74833 NumGC=72 Goroutines=11"
time="2021-03-02T02:29:25.605Z" level=info msg="Alloc=4614 TotalAlloc=49802 Sys=74833 NumGC=75 Goroutines=11"
time="2021-03-02T02:33:21.366Z" level=info msg="Watch pods 200"
time="2021-03-02T02:33:21.369Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T02:33:21.369Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T02:33:21.369Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T02:33:21.369Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T02:33:21.369Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T02:34:25.605Z" level=info msg="Alloc=4674 TotalAlloc=50253 Sys=74833 NumGC=77 Goroutines=11"
time="2021-03-02T02:39:25.605Z" level=info msg="Alloc=4615 TotalAlloc=50273 Sys=74833 NumGC=80 Goroutines=11"
time="2021-03-02T02:44:25.605Z" level=info msg="Alloc=4617 TotalAlloc=50293 Sys=74833 NumGC=82 Goroutines=11"
time="2021-03-02T02:49:25.605Z" level=info msg="Alloc=4615 TotalAlloc=50314 Sys=74833 NumGC=85 Goroutines=11"
time="2021-03-02T02:54:25.605Z" level=info msg="Alloc=4617 TotalAlloc=50334 Sys=74833 NumGC=87 Goroutines=11"
time="2021-03-02T02:59:25.605Z" level=info msg="Alloc=4617 TotalAlloc=50356 Sys=74833 NumGC=90 Goroutines=11"
time="2021-03-02T03:04:25.605Z" level=info msg="Alloc=4620 TotalAlloc=50376 Sys=74833 NumGC=92 Goroutines=11"
time="2021-03-02T03:08:07.624Z" level=info msg="Watch pods 200"
time="2021-03-02T03:08:07.628Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T03:08:07.628Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T03:08:07.628Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T03:08:07.628Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T03:08:07.628Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T03:09:25.605Z" level=info msg="Alloc=4674 TotalAlloc=50826 Sys=74833 NumGC=95 Goroutines=11"
time="2021-03-02T03:14:25.605Z" level=info msg="Alloc=4620 TotalAlloc=50847 Sys=74833 NumGC=97 Goroutines=11"
time="2021-03-02T03:19:25.605Z" level=info msg="Alloc=4618 TotalAlloc=50867 Sys=74833 NumGC=100 Goroutines=11"
time="2021-03-02T03:24:25.605Z" level=info msg="Alloc=4621 TotalAlloc=50887 Sys=74833 NumGC=102 Goroutines=11"
time="2021-03-02T03:29:25.605Z" level=info msg="Alloc=4619 TotalAlloc=50908 Sys=74833 NumGC=105 Goroutines=11"
time="2021-03-02T03:34:25.605Z" level=info msg="Alloc=4622 TotalAlloc=50928 Sys=74833 NumGC=107 Goroutines=11"
time="2021-03-02T03:39:25.605Z" level=info msg="Alloc=4619 TotalAlloc=50949 Sys=74833 NumGC=110 Goroutines=11"
time="2021-03-02T03:44:25.605Z" level=info msg="Alloc=4622 TotalAlloc=50969 Sys=74833 NumGC=112 Goroutines=11"
time="2021-03-02T03:49:25.605Z" level=info msg="Alloc=4624 TotalAlloc=50989 Sys=74833 NumGC=114 Goroutines=11"
time="2021-03-02T03:54:25.605Z" level=info msg="Alloc=4622 TotalAlloc=51009 Sys=74833 NumGC=117 Goroutines=11"
time="2021-03-02T03:59:25.605Z" level=info msg="Alloc=4624 TotalAlloc=51029 Sys=74833 NumGC=119 Goroutines=11"
time="2021-03-02T04:04:25.605Z" level=info msg="Alloc=4622 TotalAlloc=51049 Sys=74833 NumGC=122 Goroutines=11"
time="2021-03-02T04:08:02.098Z" level=info msg="Watch pods 200"
time="2021-03-02T04:08:02.101Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T04:08:02.101Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T04:08:02.101Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T04:08:02.101Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T04:08:02.101Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T04:09:25.605Z" level=info msg="Alloc=5054 TotalAlloc=51499 Sys=74833 NumGC=124 Goroutines=11"
time="2021-03-02T04:14:25.605Z" level=info msg="Alloc=4622 TotalAlloc=51513 Sys=74833 NumGC=127 Goroutines=11"
time="2021-03-02T04:19:25.605Z" level=info msg="Alloc=4625 TotalAlloc=51533 Sys=74833 NumGC=129 Goroutines=11"
time="2021-03-02T04:24:25.605Z" level=info msg="Alloc=4622 TotalAlloc=51553 Sys=74833 NumGC=132 Goroutines=11"
time="2021-03-02T04:29:25.605Z" level=info msg="Alloc=4626 TotalAlloc=51575 Sys=74833 NumGC=134 Goroutines=11"
time="2021-03-02T04:34:25.605Z" level=info msg="Alloc=4623 TotalAlloc=51595 Sys=74833 NumGC=137 Goroutines=11"
time="2021-03-02T04:39:25.605Z" level=info msg="Alloc=4626 TotalAlloc=51615 Sys=74833 NumGC=139 Goroutines=11"
time="2021-03-02T04:44:25.605Z" level=info msg="Alloc=4624 TotalAlloc=51635 Sys=74833 NumGC=142 Goroutines=11"
time="2021-03-02T04:47:41.422Z" level=info msg="Watch pods 200"
time="2021-03-02T04:47:41.425Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T04:47:41.425Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T04:47:41.425Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T04:47:41.425Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T04:47:41.425Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T04:49:25.605Z" level=info msg="Alloc=5058 TotalAlloc=52087 Sys=75089 NumGC=144 Goroutines=11"
time="2021-03-02T04:54:25.605Z" level=info msg="Alloc=4624 TotalAlloc=52101 Sys=75089 NumGC=147 Goroutines=11"
time="2021-03-02T04:59:25.605Z" level=info msg="Alloc=4628 TotalAlloc=52121 Sys=75089 NumGC=149 Goroutines=11"
time="2021-03-02T05:04:25.605Z" level=info msg="Alloc=4625 TotalAlloc=52142 Sys=75089 NumGC=152 Goroutines=11"
time="2021-03-02T05:09:25.605Z" level=info msg="Alloc=4627 TotalAlloc=52162 Sys=75089 NumGC=154 Goroutines=11"
time="2021-03-02T05:14:25.605Z" level=info msg="Alloc=4625 TotalAlloc=52182 Sys=75089 NumGC=157 Goroutines=11"
time="2021-03-02T05:19:25.605Z" level=info msg="Alloc=4628 TotalAlloc=52202 Sys=75089 NumGC=159 Goroutines=11"
time="2021-03-02T05:24:25.605Z" level=info msg="Alloc=4625 TotalAlloc=52222 Sys=75089 NumGC=162 Goroutines=11"
time="2021-03-02T05:29:25.605Z" level=info msg="Alloc=4628 TotalAlloc=52242 Sys=75089 NumGC=164 Goroutines=11"
time="2021-03-02T05:34:25.605Z" level=info msg="Alloc=4625 TotalAlloc=52262 Sys=75089 NumGC=167 Goroutines=11"
time="2021-03-02T05:36:39.431Z" level=info msg="Watch pods 200"
time="2021-03-02T05:36:39.435Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T05:36:39.435Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T05:36:39.435Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T05:36:39.435Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T05:36:39.435Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T05:39:25.605Z" level=info msg="Alloc=4686 TotalAlloc=52712 Sys=75089 NumGC=169 Goroutines=11"
time="2021-03-02T05:44:25.605Z" level=info msg="Alloc=4626 TotalAlloc=52732 Sys=75089 NumGC=172 Goroutines=11"
time="2021-03-02T05:49:25.605Z" level=info msg="Alloc=4629 TotalAlloc=52752 Sys=75089 NumGC=174 Goroutines=11"
time="2021-03-02T05:54:25.605Z" level=info msg="Alloc=4626 TotalAlloc=52772 Sys=75089 NumGC=177 Goroutines=11"
time="2021-03-02T05:59:25.605Z" level=info msg="Alloc=4629 TotalAlloc=52792 Sys=75089 NumGC=179 Goroutines=11"
time="2021-03-02T06:04:25.605Z" level=info msg="Alloc=4626 TotalAlloc=52813 Sys=75089 NumGC=182 Goroutines=11"
time="2021-03-02T06:09:25.605Z" level=info msg="Alloc=4629 TotalAlloc=52833 Sys=75089 NumGC=184 Goroutines=11"
time="2021-03-02T06:14:25.605Z" level=info msg="Alloc=4626 TotalAlloc=52853 Sys=75089 NumGC=187 Goroutines=11"
time="2021-03-02T06:19:25.605Z" level=info msg="Alloc=4629 TotalAlloc=52873 Sys=75089 NumGC=189 Goroutines=11"
time="2021-03-02T06:24:25.605Z" level=info msg="Alloc=4626 TotalAlloc=52893 Sys=75089 NumGC=192 Goroutines=11"
time="2021-03-02T06:24:43.763Z" level=info msg="Watch pods 200"
time="2021-03-02T06:24:43.767Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T06:24:43.767Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T06:24:43.767Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T06:24:43.767Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T06:24:43.767Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T06:29:25.605Z" level=info msg="Alloc=4629 TotalAlloc=53337 Sys=75089 NumGC=194 Goroutines=11"
time="2021-03-02T06:34:25.605Z" level=info msg="Alloc=4627 TotalAlloc=53357 Sys=75089 NumGC=197 Goroutines=11"
time="2021-03-02T06:39:25.604Z" level=info msg="Alloc=4629 TotalAlloc=53377 Sys=75089 NumGC=199 Goroutines=11"
time="2021-03-02T06:44:25.605Z" level=info msg="Alloc=4627 TotalAlloc=53398 Sys=75089 NumGC=202 Goroutines=11"
time="2021-03-02T06:49:25.605Z" level=info msg="Alloc=4630 TotalAlloc=53418 Sys=75089 NumGC=204 Goroutines=11"
time="2021-03-02T06:54:25.605Z" level=info msg="Alloc=4628 TotalAlloc=53439 Sys=75089 NumGC=207 Goroutines=11"
time="2021-03-02T06:59:25.605Z" level=info msg="Alloc=4631 TotalAlloc=53459 Sys=75089 NumGC=209 Goroutines=11"
time="2021-03-02T07:04:25.605Z" level=info msg="Alloc=4628 TotalAlloc=53479 Sys=75089 NumGC=212 Goroutines=11"
time="2021-03-02T07:09:04.667Z" level=info msg="Watch pods 200"
time="2021-03-02T07:09:04.669Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T07:09:04.669Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T07:09:04.669Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T07:09:04.669Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T07:09:04.669Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T07:09:25.605Z" level=info msg="Alloc=5062 TotalAlloc=53930 Sys=75089 NumGC=214 Goroutines=11"
time="2021-03-02T07:14:25.605Z" level=info msg="Alloc=4629 TotalAlloc=53944 Sys=75089 NumGC=217 Goroutines=11"
time="2021-03-02T07:19:25.605Z" level=info msg="Alloc=4632 TotalAlloc=53964 Sys=75089 NumGC=219 Goroutines=11"
time="2021-03-02T07:24:25.605Z" level=info msg="Alloc=4629 TotalAlloc=53984 Sys=75089 NumGC=222 Goroutines=11"
time="2021-03-02T07:29:25.605Z" level=info msg="Alloc=4632 TotalAlloc=54004 Sys=75089 NumGC=224 Goroutines=11"
time="2021-03-02T07:34:25.605Z" level=info msg="Alloc=4629 TotalAlloc=54024 Sys=75089 NumGC=227 Goroutines=11"
time="2021-03-02T07:39:25.605Z" level=info msg="Alloc=4632 TotalAlloc=54044 Sys=75089 NumGC=229 Goroutines=11"
time="2021-03-02T07:44:25.605Z" level=info msg="Alloc=4629 TotalAlloc=54065 Sys=75089 NumGC=232 Goroutines=11"
time="2021-03-02T07:49:25.605Z" level=info msg="Alloc=4632 TotalAlloc=54085 Sys=75089 NumGC=234 Goroutines=11"
time="2021-03-02T07:54:25.605Z" level=info msg="Alloc=4630 TotalAlloc=54105 Sys=75089 NumGC=237 Goroutines=11"
time="2021-03-02T07:59:09.000Z" level=info msg="Watch pods 200"
time="2021-03-02T07:59:09.003Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T07:59:09.003Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T07:59:09.003Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T07:59:09.003Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T07:59:09.003Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T07:59:25.605Z" level=info msg="Alloc=5047 TotalAlloc=54539 Sys=75089 NumGC=239 Goroutines=11"
time="2021-03-02T08:04:25.605Z" level=info msg="Alloc=4630 TotalAlloc=54553 Sys=75089 NumGC=242 Goroutines=11"
time="2021-03-02T08:09:25.605Z" level=info msg="Alloc=4633 TotalAlloc=54573 Sys=75089 NumGC=244 Goroutines=11"
time="2021-03-02T08:14:25.605Z" level=info msg="Alloc=4630 TotalAlloc=54593 Sys=75089 NumGC=247 Goroutines=11"
time="2021-03-02T08:19:25.605Z" level=info msg="Alloc=4633 TotalAlloc=54613 Sys=75089 NumGC=249 Goroutines=11"
time="2021-03-02T08:24:25.605Z" level=info msg="Alloc=4630 TotalAlloc=54633 Sys=75089 NumGC=252 Goroutines=11"
time="2021-03-02T08:29:25.605Z" level=info msg="Alloc=4633 TotalAlloc=54653 Sys=75089 NumGC=254 Goroutines=11"
time="2021-03-02T08:34:25.605Z" level=info msg="Alloc=4630 TotalAlloc=54673 Sys=75089 NumGC=257 Goroutines=11"
time="2021-03-02T08:39:25.605Z" level=info msg="Alloc=4633 TotalAlloc=54693 Sys=75089 NumGC=259 Goroutines=11"
time="2021-03-02T08:44:25.605Z" level=info msg="Alloc=4630 TotalAlloc=54714 Sys=75089 NumGC=262 Goroutines=11"
time="2021-03-02T08:49:25.605Z" level=info msg="Alloc=4633 TotalAlloc=54734 Sys=75089 NumGC=264 Goroutines=11"
time="2021-03-02T08:54:25.605Z" level=info msg="Alloc=4630 TotalAlloc=54754 Sys=75089 NumGC=267 Goroutines=11"
time="2021-03-02T08:55:04.164Z" level=info msg="Watch pods 200"
time="2021-03-02T08:55:04.168Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T08:55:04.168Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T08:55:04.168Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T08:55:04.168Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T08:55:04.168Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T08:59:25.605Z" level=info msg="Alloc=4634 TotalAlloc=55198 Sys=75089 NumGC=269 Goroutines=11"
time="2021-03-02T09:04:25.605Z" level=info msg="Alloc=4632 TotalAlloc=55219 Sys=75089 NumGC=272 Goroutines=11"
time="2021-03-02T09:09:25.605Z" level=info msg="Alloc=4635 TotalAlloc=55239 Sys=75089 NumGC=274 Goroutines=11"
time="2021-03-02T09:14:25.605Z" level=info msg="Alloc=4632 TotalAlloc=55259 Sys=75089 NumGC=277 Goroutines=11"
time="2021-03-02T09:19:25.605Z" level=info msg="Alloc=4635 TotalAlloc=55279 Sys=75089 NumGC=279 Goroutines=11"
time="2021-03-02T09:24:25.605Z" level=info msg="Alloc=4632 TotalAlloc=55299 Sys=75089 NumGC=282 Goroutines=11"
time="2021-03-02T09:29:25.605Z" level=info msg="Alloc=4635 TotalAlloc=55319 Sys=75089 NumGC=284 Goroutines=11"
time="2021-03-02T09:34:25.605Z" level=info msg="Alloc=4633 TotalAlloc=55340 Sys=75089 NumGC=287 Goroutines=11"
time="2021-03-02T09:38:07.820Z" level=info msg="Watch pods 200"
time="2021-03-02T09:38:07.824Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T09:38:07.824Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T09:38:07.824Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T09:38:07.824Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T09:38:07.824Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T09:39:25.605Z" level=info msg="Alloc=5066 TotalAlloc=55790 Sys=75089 NumGC=289 Goroutines=11"
time="2021-03-02T09:44:25.605Z" level=info msg="Alloc=4632 TotalAlloc=55804 Sys=75089 NumGC=292 Goroutines=11"
time="2021-03-02T09:49:25.605Z" level=info msg="Alloc=4635 TotalAlloc=55824 Sys=75089 NumGC=294 Goroutines=11"
time="2021-03-02T09:54:25.605Z" level=info msg="Alloc=4633 TotalAlloc=55844 Sys=75089 NumGC=297 Goroutines=11"
time="2021-03-02T09:59:25.605Z" level=info msg="Alloc=4635 TotalAlloc=55864 Sys=75089 NumGC=299 Goroutines=11"
time="2021-03-02T10:04:25.605Z" level=info msg="Alloc=4633 TotalAlloc=55884 Sys=75089 NumGC=302 Goroutines=11"
time="2021-03-02T10:09:25.605Z" level=info msg="Alloc=4636 TotalAlloc=55905 Sys=75089 NumGC=304 Goroutines=11"
time="2021-03-02T10:14:25.605Z" level=info msg="Alloc=4633 TotalAlloc=55925 Sys=75089 NumGC=307 Goroutines=11"
time="2021-03-02T10:19:25.605Z" level=info msg="Alloc=4636 TotalAlloc=55945 Sys=75089 NumGC=309 Goroutines=11"
time="2021-03-02T10:22:40.535Z" level=info msg="Watch pods 200"
time="2021-03-02T10:22:40.539Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T10:22:40.539Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T10:22:40.539Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T10:22:40.539Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T10:22:40.539Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T10:24:25.605Z" level=info msg="Alloc=4691 TotalAlloc=56395 Sys=75089 NumGC=312 Goroutines=11"
time="2021-03-02T10:29:25.605Z" level=info msg="Alloc=4637 TotalAlloc=56416 Sys=75089 NumGC=314 Goroutines=11"
time="2021-03-02T10:34:25.605Z" level=info msg="Alloc=4635 TotalAlloc=56437 Sys=75089 NumGC=317 Goroutines=11"
time="2021-03-02T10:39:25.605Z" level=info msg="Alloc=4638 TotalAlloc=56457 Sys=75089 NumGC=319 Goroutines=11"
time="2021-03-02T10:44:25.605Z" level=info msg="Alloc=4636 TotalAlloc=56477 Sys=75089 NumGC=322 Goroutines=11"
time="2021-03-02T10:49:25.605Z" level=info msg="Alloc=4638 TotalAlloc=56498 Sys=75089 NumGC=324 Goroutines=11"
time="2021-03-02T10:54:25.605Z" level=info msg="Alloc=4636 TotalAlloc=56518 Sys=75089 NumGC=327 Goroutines=11"
time="2021-03-02T10:59:25.605Z" level=info msg="Alloc=4639 TotalAlloc=56538 Sys=75089 NumGC=329 Goroutines=11"
time="2021-03-02T11:03:16.781Z" level=info msg="Watch pods 200"
time="2021-03-02T11:03:16.785Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T11:03:16.785Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T11:03:16.785Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T11:03:16.785Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T11:03:16.785Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T11:04:25.605Z" level=info msg="Alloc=4694 TotalAlloc=56990 Sys=75089 NumGC=332 Goroutines=11"
time="2021-03-02T11:09:25.605Z" level=info msg="Alloc=4640 TotalAlloc=57010 Sys=75089 NumGC=334 Goroutines=11"
time="2021-03-02T11:14:25.605Z" level=info msg="Alloc=4637 TotalAlloc=57030 Sys=75089 NumGC=337 Goroutines=11"
time="2021-03-02T11:19:25.605Z" level=info msg="Alloc=4640 TotalAlloc=57050 Sys=75089 NumGC=339 Goroutines=11"
time="2021-03-02T11:24:25.605Z" level=info msg="Alloc=4638 TotalAlloc=57070 Sys=75089 NumGC=342 Goroutines=11"
time="2021-03-02T11:29:25.605Z" level=info msg="Alloc=4640 TotalAlloc=57091 Sys=75089 NumGC=344 Goroutines=11"
time="2021-03-02T11:34:25.605Z" level=info msg="Alloc=4638 TotalAlloc=57111 Sys=75089 NumGC=347 Goroutines=11"
time="2021-03-02T11:39:25.605Z" level=info msg="Alloc=4640 TotalAlloc=57131 Sys=75089 NumGC=349 Goroutines=11"
time="2021-03-02T11:44:25.605Z" level=info msg="Alloc=4638 TotalAlloc=57151 Sys=75089 NumGC=352 Goroutines=11"
time="2021-03-02T11:49:25.605Z" level=info msg="Alloc=4640 TotalAlloc=57171 Sys=75089 NumGC=354 Goroutines=11"
time="2021-03-02T11:50:11.528Z" level=info msg="Watch pods 200"
time="2021-03-02T11:50:11.533Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T11:50:11.533Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T11:50:11.533Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T11:50:11.533Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T11:50:11.533Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T11:54:25.605Z" level=info msg="Alloc=4638 TotalAlloc=57621 Sys=75089 NumGC=357 Goroutines=11"
time="2021-03-02T11:59:25.605Z" level=info msg="Alloc=4641 TotalAlloc=57641 Sys=75089 NumGC=359 Goroutines=11"
time="2021-03-02T12:04:25.605Z" level=info msg="Alloc=4638 TotalAlloc=57661 Sys=75089 NumGC=362 Goroutines=11"
time="2021-03-02T12:09:25.605Z" level=info msg="Alloc=4641 TotalAlloc=57682 Sys=75089 NumGC=364 Goroutines=11"
time="2021-03-02T12:14:25.605Z" level=info msg="Alloc=4639 TotalAlloc=57702 Sys=75089 NumGC=367 Goroutines=11"
time="2021-03-02T12:19:25.605Z" level=info msg="Alloc=4641 TotalAlloc=57722 Sys=75089 NumGC=369 Goroutines=11"
time="2021-03-02T12:24:25.605Z" level=info msg="Alloc=4639 TotalAlloc=57742 Sys=75089 NumGC=372 Goroutines=11"
time="2021-03-02T12:29:12.649Z" level=info msg="Watch pods 200"
time="2021-03-02T12:29:12.655Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T12:29:12.655Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T12:29:12.655Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T12:29:12.655Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T12:29:12.655Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T12:29:25.605Z" level=info msg="Alloc=5071 TotalAlloc=58192 Sys=75089 NumGC=374 Goroutines=11"
time="2021-03-02T12:34:25.605Z" level=info msg="Alloc=4639 TotalAlloc=58205 Sys=75089 NumGC=377 Goroutines=11"
time="2021-03-02T12:39:25.605Z" level=info msg="Alloc=4641 TotalAlloc=58225 Sys=75089 NumGC=379 Goroutines=11"
time="2021-03-02T12:44:25.605Z" level=info msg="Alloc=4639 TotalAlloc=58245 Sys=75089 NumGC=382 Goroutines=11"
time="2021-03-02T12:49:25.605Z" level=info msg="Alloc=4642 TotalAlloc=58265 Sys=75089 NumGC=384 Goroutines=11"
time="2021-03-02T12:54:25.605Z" level=info msg="Alloc=4639 TotalAlloc=58286 Sys=75089 NumGC=387 Goroutines=11"
time="2021-03-02T12:59:25.605Z" level=info msg="Alloc=4642 TotalAlloc=58306 Sys=75089 NumGC=389 Goroutines=11"
time="2021-03-02T13:04:25.605Z" level=info msg="Alloc=4639 TotalAlloc=58326 Sys=75089 NumGC=392 Goroutines=11"
time="2021-03-02T13:09:25.605Z" level=info msg="Alloc=4642 TotalAlloc=58346 Sys=75089 NumGC=394 Goroutines=11"
time="2021-03-02T13:14:25.605Z" level=info msg="Alloc=4639 TotalAlloc=58366 Sys=75089 NumGC=397 Goroutines=11"
time="2021-03-02T13:19:25.605Z" level=info msg="Alloc=4642 TotalAlloc=58386 Sys=75089 NumGC=399 Goroutines=11"
time="2021-03-02T13:24:25.605Z" level=info msg="Alloc=4640 TotalAlloc=58407 Sys=75089 NumGC=402 Goroutines=11"
time="2021-03-02T13:25:21.074Z" level=info msg="Watch pods 200"
time="2021-03-02T13:25:21.078Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T13:25:21.078Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T13:25:21.078Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T13:25:21.078Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T13:25:21.078Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T13:29:25.605Z" level=info msg="Alloc=4642 TotalAlloc=58851 Sys=75089 NumGC=404 Goroutines=11"
time="2021-03-02T13:34:25.605Z" level=info msg="Alloc=4640 TotalAlloc=58871 Sys=75089 NumGC=407 Goroutines=11"
time="2021-03-02T13:39:25.605Z" level=info msg="Alloc=4642 TotalAlloc=58891 Sys=75089 NumGC=409 Goroutines=11"
time="2021-03-02T13:44:25.605Z" level=info msg="Alloc=4639 TotalAlloc=58911 Sys=75089 NumGC=412 Goroutines=11"
time="2021-03-02T13:49:25.605Z" level=info msg="Alloc=4642 TotalAlloc=58931 Sys=75089 NumGC=414 Goroutines=11"
time="2021-03-02T13:54:25.605Z" level=info msg="Alloc=4641 TotalAlloc=58952 Sys=75089 NumGC=417 Goroutines=11"
time="2021-03-02T13:59:25.605Z" level=info msg="Alloc=4643 TotalAlloc=58972 Sys=75089 NumGC=419 Goroutines=11"
time="2021-03-02T14:04:25.605Z" level=info msg="Alloc=4641 TotalAlloc=58992 Sys=75089 NumGC=422 Goroutines=11"
time="2021-03-02T14:09:25.605Z" level=info msg="Alloc=4643 TotalAlloc=59012 Sys=75089 NumGC=424 Goroutines=11"
time="2021-03-02T14:13:37.177Z" level=info msg="Watch pods 200"
time="2021-03-02T14:13:37.180Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T14:13:37.180Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T14:13:37.180Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T14:13:37.180Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T14:13:37.180Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T14:14:25.605Z" level=info msg="Alloc=5070 TotalAlloc=59463 Sys=75089 NumGC=427 Goroutines=11"
time="2021-03-02T14:19:25.605Z" level=info msg="Alloc=4643 TotalAlloc=59476 Sys=75089 NumGC=429 Goroutines=11"
time="2021-03-02T14:24:25.605Z" level=info msg="Alloc=4641 TotalAlloc=59497 Sys=75089 NumGC=432 Goroutines=11"
time="2021-03-02T14:29:25.605Z" level=info msg="Alloc=4643 TotalAlloc=59517 Sys=75089 NumGC=434 Goroutines=11"
time="2021-03-02T14:34:25.605Z" level=info msg="Alloc=4641 TotalAlloc=59537 Sys=75345 NumGC=437 Goroutines=11"
time="2021-03-02T14:39:25.605Z" level=info msg="Alloc=4644 TotalAlloc=59557 Sys=75345 NumGC=439 Goroutines=11"
time="2021-03-02T14:44:25.605Z" level=info msg="Alloc=4635 TotalAlloc=59577 Sys=75345 NumGC=442 Goroutines=11"
time="2021-03-02T14:49:25.605Z" level=info msg="Alloc=4638 TotalAlloc=59598 Sys=75345 NumGC=444 Goroutines=11"
time="2021-03-02T14:52:47.346Z" level=info msg="Watch pods 200"
time="2021-03-02T14:52:47.349Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T14:52:47.349Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T14:52:47.349Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T14:52:47.349Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T14:52:47.349Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T14:54:25.605Z" level=info msg="Alloc=4693 TotalAlloc=60049 Sys=75345 NumGC=447 Goroutines=11"
time="2021-03-02T14:59:25.605Z" level=info msg="Alloc=4639 TotalAlloc=60069 Sys=75345 NumGC=449 Goroutines=11"
time="2021-03-02T15:04:25.605Z" level=info msg="Alloc=4637 TotalAlloc=60089 Sys=75345 NumGC=452 Goroutines=11"
time="2021-03-02T15:09:25.605Z" level=info msg="Alloc=4640 TotalAlloc=60109 Sys=75345 NumGC=454 Goroutines=11"
time="2021-03-02T15:14:25.605Z" level=info msg="Alloc=4637 TotalAlloc=60130 Sys=75345 NumGC=457 Goroutines=11"
time="2021-03-02T15:19:25.605Z" level=info msg="Alloc=4640 TotalAlloc=60150 Sys=75345 NumGC=459 Goroutines=11"
time="2021-03-02T15:24:25.605Z" level=info msg="Alloc=4637 TotalAlloc=60170 Sys=75345 NumGC=462 Goroutines=11"
time="2021-03-02T15:29:25.605Z" level=info msg="Alloc=4640 TotalAlloc=60190 Sys=75345 NumGC=464 Goroutines=11"
time="2021-03-02T15:34:25.605Z" level=info msg="Alloc=4637 TotalAlloc=60210 Sys=75345 NumGC=467 Goroutines=11"
time="2021-03-02T15:39:22.841Z" level=info msg="Watch pods 200"
time="2021-03-02T15:39:22.845Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T15:39:22.845Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T15:39:22.845Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T15:39:22.845Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T15:39:22.845Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T15:39:25.605Z" level=info msg="Alloc=5071 TotalAlloc=60661 Sys=75345 NumGC=469 Goroutines=11"
time="2021-03-02T15:44:25.605Z" level=info msg="Alloc=4638 TotalAlloc=60675 Sys=75345 NumGC=472 Goroutines=11"
time="2021-03-02T15:49:25.605Z" level=info msg="Alloc=4641 TotalAlloc=60695 Sys=75345 NumGC=474 Goroutines=11"
time="2021-03-02T15:54:25.605Z" level=info msg="Alloc=4638 TotalAlloc=60715 Sys=75345 NumGC=477 Goroutines=11"
time="2021-03-02T15:59:25.605Z" level=info msg="Alloc=4641 TotalAlloc=60736 Sys=75345 NumGC=479 Goroutines=11"
time="2021-03-02T16:04:25.605Z" level=info msg="Alloc=4638 TotalAlloc=60756 Sys=75345 NumGC=482 Goroutines=11"
time="2021-03-02T16:09:25.605Z" level=info msg="Alloc=4641 TotalAlloc=60776 Sys=75345 NumGC=484 Goroutines=11"
time="2021-03-02T16:14:25.605Z" level=info msg="Alloc=4638 TotalAlloc=60796 Sys=75345 NumGC=487 Goroutines=11"
time="2021-03-02T16:15:15.109Z" level=info msg="Watch pods 200"
time="2021-03-02T16:15:15.113Z" level=info msg="mapped container name \"main\" to container ID \"ea0503fd887ef80fc64d973279b047e0e9cb778f91a670f20b25ef8c5f02f0d8\""
time="2021-03-02T16:15:15.113Z" level=info msg="mapped container name \"mysql\" to container ID \"0878b79f592e315cb2dcae4d7615af4103bb42942810c92a4c47a0ea558373eb\""
time="2021-03-02T16:15:15.113Z" level=info msg="mapped container name \"nginx\" to container ID \"8d6572d12bc7ce39121b8bc51896110490ce6c9c3c4e8221f41e12f35f0cbb4a\""
time="2021-03-02T16:15:15.113Z" level=info msg="mapped container name \"redis\" to container ID \"7f17a0c2237af385217dfe40f6c3d0f0eb16e02157cac4106f88d2bb3ac04179\""
time="2021-03-02T16:15:15.113Z" level=info msg="mapped container name \"wait\" to container ID \"a0e1a74c8a173e4a89f9fa792afe8e36f1bf9cbbf6ef6b7269978b69f9c47dd2\""
time="2021-03-02T16:19:25.605Z" level=info msg="Alloc=4641 TotalAlloc=61240 Sys=75345 NumGC=489 Goroutines=11"
time="2021-03-02T16:24:25.605Z" level=info msg="Alloc=4639 TotalAlloc=61260 Sys=75345 NumGC=492 Goroutines=11"
time="2021-03-02T16:29:25.605Z" level=info msg="Alloc=4641 TotalAlloc=61280 Sys=75345 NumGC=494 Goroutines=11"
time="2021-03-02T16:34:25.605Z" level=info msg="Alloc=4639 TotalAlloc=61300 Sys=75345 NumGC=497 Goroutines=11"

@alexec
Copy link
Contributor

alexec commented Mar 4, 2021

Do you mean dev-k8sapi or v3.0.0-rc4?

@alexec
Copy link
Contributor

alexec commented Mar 4, 2021

The logs look older to me.

@caueasantos
Copy link
Contributor Author

Do you mean dev-k8sapi or v3.0.0-rc4?

dev-k8sapi -> The short running is working fine, your changes apparently fixed the issue.

time="2021-03-04T01:19:01.671Z" level=info msg="Starting annotations monitor"
time="2021-03-04T01:19:01.672Z" level=info msg="Starting deadline monitor"
time="2021-03-04T01:19:01.775Z" level=info msg="secured root for pid 25 root: runc:[2:INIT]"
time="2021-03-04T01:19:01.776Z" level=warning msg="failed to secure root file handle for 25" error="container name not found"
time="2021-03-04T01:19:01.826Z" level=info msg="secured root for pid 25 root: runc:[2:INIT]"
time="2021-03-04T01:19:01.826Z" level=warning msg="failed to secure root file handle for 25" error="container name not found"
time="2021-03-04T01:19:06.768Z" level=info msg="no container PID found for \"main\" - assuming it was short running"
time="2021-03-04T01:19:06.769Z" level=info msg="Main container completed"
time="2021-03-04T01:19:06.770Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2021-03-04T01:19:06.770Z" level=info msg="No output parameters"
time="2021-03-04T01:19:06.770Z" level=info msg="No output artifacts"
time="2021-03-04T01:19:06.770Z" level=info msg="Alloc=4789 TotalAlloc=11723 Sys=73041 NumGC=4 Goroutines=9"

v3.0.0-rc3 && v3.0.0-rc4 -> short running issue + DAG hanging

@caueasantos
Copy link
Contributor Author

The logs look older to me.

Yeah. I'm not running this version on production anymore, just captured a little bit before the downgrade.

alexec added a commit that referenced this issue Mar 4, 2021
@alexec
feat(executor): Move exit code capture to controller. See #5251
fb1583b 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
alexec added a commit that referenced this issue Mar 4, 2021
@alexec
fix(executor): Enhance PNS executor. Resolves #5251
b0b888f 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
@alexec alexec mentioned this issue Mar 4, 2021
Merged
alexec added a commit that referenced this issue Mar 4, 2021
@alexec
fix(executor): Enhance PNS executor. Resolves #5251
f8d6b14 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
alexec added a commit that referenced this issue Mar 4, 2021
@alexec
fix(executor): Enhance PNS executor. Resolves #5251
5f4b802 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
@alexec
Copy link
Contributor

alexec commented Mar 4, 2021

I've another build to test. I fixed a bug, but found a new bug. Can you try argoproj/argoexec:dev-pns?

alexec added a commit that referenced this issue Mar 4, 2021
@alexec
feat(executor): Move exit code capture to controller. See #5251
cc51ed1 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
alexec added a commit that referenced this issue Mar 4, 2021
@alexec
fix(executor): Enhance PNS executor. Resolves #5251
8c8229a 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
alexec added a commit that referenced this issue Mar 4, 2021
@alexec
fix(executor): Enhance PNS executor. Resolves #5251 (#5296)
757e0be
@caueasantos
Copy link
Contributor Author

@alexec Sorry about the late, busy day 😃 .

What did I do?

Pulled the latest version into my environment again v3.0.0-rc4 and used your executor image argoproj/argoexec:dev-pns

image

image

wait container logs

time="2021-03-05T03:56:01.239Z" level=info msg="Starting annotations monitor"
time="2021-03-05T03:56:01.239Z" level=info msg="Starting deadline monitor"
time="2021-03-05T03:56:06.240Z" level=info msg="container PIDs still unknown (maybe short running container, or late starting)"
time="2021-03-05T03:56:06.249Z" level=info msg="Watch pods 200"
time="2021-03-05T04:01:01.240Z" level=info msg="Alloc=4442 TotalAlloc=47817 Sys=74065 NumGC=15 Goroutines=10"
time="2021-03-05T04:06:01.239Z" level=info msg="Alloc=4383 TotalAlloc=47834 Sys=74065 NumGC=17 Goroutines=10"
time="2021-03-05T04:06:14.883Z" level=info msg="Main container completed"
time="2021-03-05T04:06:14.883Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2021-03-05T04:06:14.883Z" level=info msg="Capturing script exit code"
time="2021-03-05T04:06:14.883Z" level=info msg="Getting exit code of main"
time="2021-03-05T04:06:14.888Z" level=info msg="Get pods 200"
time="2021-03-05T04:06:14.889Z" level=info msg="Saving logs"
time="2021-03-05T04:06:14.890Z" level=info msg="Getting output of main"
time="2021-03-05T04:06:14.909Z" level=info msg="List log 200"
time="2021-03-05T04:06:14.912Z" level=info msg="GCS Save path: /tmp/argo/outputs/logs/main.log, key: argo-workflow-logs/2021/03/05/github-20780-6482b31-ztlmf/github-207
80-6482b31-ztlmf-1148270933/main.log"
time="2021-03-05T04:06:15.338Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2021-03-05T04:06:15.338Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2021-03-05T04:06:15.338Z" level=info msg="No output parameters"
time="2021-03-05T04:06:15.338Z" level=info msg="No output artifacts"
time="2021-03-05T04:06:15.338Z" level=info msg="Annotating pod with output"
time="2021-03-05T04:06:15.359Z" level=info msg="Patch pods 200"
time="2021-03-05T04:06:15.361Z" level=info msg="Killing sidecars mysql,redis,nginx"
time="2021-03-05T04:06:15.366Z" level=info msg="Get pods 200"
time="2021-03-05T04:06:15.367Z" level=warning msg="No PID for container named \"nginx\". Process assumed to have completed"
time="2021-03-05T04:06:15.368Z" level=warning msg="No PID for container named \"mysql\". Process assumed to have completed"
time="2021-03-05T04:06:15.368Z" level=warning msg="No PID for container named \"redis\". Process assumed to have completed"
time="2021-03-05T04:06:15.368Z" level=info msg="Alloc=28711 TotalAlloc=72691 Sys=74065 NumGC=18 Goroutines=11"

This was referenced Mar 8, 2021
Closed
Closed
simster7 pushed a commit that referenced this issue Mar 8, 2021
@alexec @simster7
fix(executor): Enhance PNS executor. Resolves #5251 (#5296)
f368c32
@alexec alexec mentioned this issue Mar 8, 2021
Merged
alexec added a commit to alexec/argo-workflows that referenced this issue Mar 8, 2021
@alexec
feat(executor): Move exit code capture to controller. See argoproj#5251
eb44209 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
alexec added a commit that referenced this issue Mar 17, 2021
@alexec
feat(executor): Move exit code capture to controller. See #5251 (#5328)
d717865 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
@simster7 simster7 mentioned this issue Mar 22, 2021
Closed
34 tasks
@simster7 simster7 mentioned this issue Mar 29, 2021
Closed
77 tasks
@simster7 simster7 mentioned this issue Apr 19, 2021
Closed
50 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alexec alexec

Labels
type/bug type/regression Regression from previous behavior (a specific type of bug)
Projects
None yet
Milestone
No milestone
2 participants
@alexec @caueasantos