Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Zombie Workflows #4398

Closed
TekTimmy opened this issue Oct 27, 2020 · 33 comments
Closed

Zombie Workflows #4398

TekTimmy opened this issue Oct 27, 2020 · 33 comments
Assignees
@alexec
Labels
type/bug

Comments

@TekTimmy
Copy link

TekTimmy commented Oct 27, 2020 •
edited
Loading

Summary

In our cluster Workflows occur that seem to be forgotten / ignored by the workflow controller. So we find Workflows in status "Running" that have successfully finished all steps OR have a failed step. This is causing Workflows to be "Running" forever and block resources.
When killing the workflow-controller pod, which then automatically restarts, all workflows get cleaned up accordingly.
I expect the workflow controller to handle those workflows correctly. Could that be caused by not enough resources for the workflow controller? Do you have scaling suggestions for ~800 Running workflows?

I get the Zombie workflows:

NAME                                                   PROGRESS
wgs-alignment-1d500-tn7-v1-lane-h2ckkdsxy-002-l6z7s    6/6
wgs-gvc-annotation-1a1114-0-c1-v1-chr-y-vk4dv          8/8
wgs-gvc-annotation-1a1390-tn8-v1-chr-11-w2nkh          8/8
wgs-gvc-annotation-1a1390-tn8-v1-chr-13-2j9tn          8/8
wgs-gvc-annotation-1a1390-tn8-v1-chr-14-k5w7r          8/8

When running this command:

kubectl get workflows \
  --no-headers \
  --output custom-columns=NAME:".metadata.name",PROGRESS:".status.progress" \
  --selector "workflows.argoproj.io/phase=Running" | \
  awk '{ split($2,a,"/"); if (a[1] == a[2]) { print $0 } }'

Diagnostics

AWS EKS 1.17 with Argo:latest
Workflow Controller Options:

      - args:
        - --configmap
        - workflow-controller-configmap
        - --executor-image
        - argoproj/argoexec:latest
        - --namespaced
        - --pod-workers
        - "64"
        - --workflow-workers
        - "256"
        - --qps
        - "100"
        - --burst
        - "50"

Cleaned YML output:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  creationTimestamp: "2020-10-24T23:38:32Z"
  generateName: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-
  generation: 60
  labels:
    workflows.argoproj.io/creator: system-serviceaccount-v1-0-mocca-cron
    workflows.argoproj.io/phase: Running
    workflows.argoproj.io/workflow-archiving-status: Archived
  name: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh
  namespace: v1-0
  resourceVersion: "44798719"
  selfLink: /apis/argoproj.io/v1alpha1/namespaces/v1-0/workflows/wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh
  uid: 733bd721-ccab-4a4e-96b2-633e6225652d
spec:
  arguments:
    parameters:
    - name: namespace
      value: v1-0
  entrypoint: main
  podGC:
    strategy: OnWorkflowCompletion
  priority: 0
  serviceAccountName: argo
  templates:
  - arguments: {}
    inputs: {}
    metadata: {}
    name: main
    outputs: {}
    retryStrategy:
      limit: 1
      retryPolicy: OnError
    steps:
    - - arguments: {}
        name: create
        templateRef:
          name: s3-copy-templates
          template: create-semaphore
  ttlStrategy:
    secondsAfterSuccess: 120
  volumeClaimTemplates:
  - metadata:
      creationTimestamp: null
      name: transfer
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 100Gi
    status: {}
status:
  conditions:
  - status: "False"
    type: Completed
  finishedAt: null
  nodes:
    wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh:
      children:
      - wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2005753590
      displayName: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh
      finishedAt: "2020-10-27T06:50:41Z"
      id: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh
      message: child 'wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-566086781'
        failed
      name: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh
      phase: Failed
      progress: 4/4
      resourcesDuration:
        cpu: 603211
        memory: 21248202
      startedAt: "2020-10-24T23:38:32Z"
      templateName: main
      templateScope: local/wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh
      type: Retry
    wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-566086781:
      boundaryID: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2005753590
      displayName: samtools-sort-and-index
      finishedAt: "2020-10-27T06:50:39Z"
      hostNodeName: ip-192-160-187-136.eu-central-1.compute.internal
      id: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-566086781
      inputs:
        parameters:
        - name: cpu
          value: "15"
      message: failed with exit code 1
      name: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh(0)[3].samtools-sort-and-index
      outputs:
        artifacts:
        - archiveLogs: true
          name: main-logs
          s3:
            accessKeySecret:
              key: ""
            bucket: test
            endpoint: s3.amazonaws.com
            key: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-566086781/main.log
            secretKeySecret:
              key: ""
        exitCode: "1"
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 9713
        memory: 342470
      startedAt: "2020-10-27T06:40:17Z"
      templateRef:
        name: samtools-templates
        template: samtools-sort-and-index
      templateScope: local/wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh
      type: Pod
    wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-1248541623:
      boundaryID: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2005753590
      children:
      - wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-1278296029
      displayName: s3-download-static-data
      finishedAt: "2020-10-24T23:40:11Z"
      hostNodeName: ip-192-160-159-24.eu-central-1.compute.internal
      id: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-1248541623
      inputs:
        parameters:
        - name: s3-secret-key-ref
          value: aws-access-key
      name: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh(0)[0].s3-download-static-data
      outputs:
        artifacts:
        - archiveLogs: true
          name: main-logs
          s3:
            accessKeySecret:
              key: ""
            bucket: test
            endpoint: s3.amazonaws.com
            key: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-1248541623/main.log
            secretKeySecret:
              key: ""
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 143
        memory: 1526
      startedAt: "2020-10-24T23:38:32Z"
      templateRef:
        name: s3-copy-templates
        template: download-static-data
      templateScope: local/wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh
      type: Pod
    wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-1278296029:
      boundaryID: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2005753590
      children:
      - wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2087606413
      displayName: '[1]'
      finishedAt: "2020-10-24T23:47:00Z"
      id: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-1278296029
      name: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh(0)[1]
      phase: Succeeded
      progress: 3/3
      resourcesDuration:
        cpu: 603068
        memory: 21246676
      startedAt: "2020-10-24T23:40:12Z"
      templateName: main
      templateScope: local/wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh
      type: StepGroup
    wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2005753590:
      children:
      - wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2285100264
      displayName: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh(0)
      finishedAt: "2020-10-27T06:50:41Z"
      id: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2005753590
      message: child 'wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-566086781'
        failed
      name: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh(0)
      outboundNodes:
      - wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-566086781
      phase: Failed
      progress: 4/4
      resourcesDuration:
        cpu: 603211
        memory: 21248202
      startedAt: "2020-10-26T20:18:32Z"
      templateName: main
      templateScope: local/wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh
      type: Steps
    wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2087606413:
      boundaryID: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2005753590
      children:
      - wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2351813382
      displayName: s3-download-processing-data
      finishedAt: "2020-10-24T23:46:58Z"
      hostNodeName: test.eu-central-1.compute.internal
      id: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2087606413
      inputs:
        parameters:
        - name: s3-secret-key-ref
          value: aws-access-key
      name: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh(0)[1].s3-download-processing-data
      outputs:
        artifacts:
        - archiveLogs: true
          name: main-logs
          s3:
            accessKeySecret:
              key: ""
            bucket: test
            endpoint: s3.amazonaws.com
            key: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2087606413/main.log
            secretKeySecret:
              key: ""
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 738
        memory: 7906
      startedAt: "2020-10-24T23:40:12Z"
      templateRef:
        name: s3-copy-templates
        template: copy-processing-data
      templateScope: local/wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh
      type: Pod
    wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2284555811:
      boundaryID: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2005753590
      children:
      - wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-566086781
      displayName: '[3]'
      finishedAt: "2020-10-27T06:50:41Z"
      id: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2284555811
      message: child 'wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-566086781'
        failed
      name: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh(0)[3]
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 9713
        memory: 342470
      startedAt: "2020-10-27T06:40:17Z"
      templateName: main
      templateScope: local/wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh
      type: StepGroup
    wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2285100264:
      boundaryID: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2005753590
      children:
      - wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-1248541623
      displayName: '[0]'
      finishedAt: "2020-10-24T23:40:12Z"
      id: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2285100264
      name: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh(0)[0]
      phase: Succeeded
      progress: 4/4
      resourcesDuration:
        cpu: 603211
        memory: 21248202
      startedAt: "2020-10-24T23:38:32Z"
      templateName: main
      templateScope: local/wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh
      type: StepGroup
    wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2351813382:
      boundaryID: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2005753590
      children:
      - wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-3874162402
      displayName: '[2]'
      finishedAt: "2020-10-27T06:40:17Z"
      id: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2351813382
      name: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh(0)[2]
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 602330
        memory: 21238770
      startedAt: "2020-10-24T23:47:00Z"
      templateName: main
      templateScope: local/wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh
      type: StepGroup
    wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-3874162402:
      boundaryID: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2005753590
      children:
      - wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-2284555811
      displayName: bwa-mem
      finishedAt: "2020-10-27T06:40:16Z"
      hostNodeName: ip-192-160-187-136.eu-central-1.compute.internal
      id: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-3874162402
      inputs:
        parameters:
        - name: aws-ecr
          value: 952545021637.dkr.ecr.eu-central-1.amazonaws.com/
      name: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh(0)[2].bwa-mem
      outputs:
        artifacts:
        - archiveLogs: true
          name: main-logs
          s3:
            accessKeySecret:
              key: ""
            bucket: test
            endpoint: s3.amazonaws.com
            key: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-3874162402/main.log
            secretKeySecret:
              key: ""
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 592617
        memory: 20896300
      startedAt: "2020-10-26T20:18:32Z"
      templateRef:
        name: bwa-templates
        template: bwa-mem
      templateScope: local/wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh
      type: Pod
  persistentVolumeClaims:
  - name: transfer
    persistentVolumeClaim:
      claimName: wgs-alignment-1e329-tn7-v1-lane-h5t23dsxy-001-2t9zh-transfer
  phase: Running
  progress: 4/4
  resourcesDuration:
    cpu: 603211
    memory: 21248202
  startedAt: "2020-10-24T23:38:32Z"
  storedTemplates:
    namespaced/bwa-templates/bwa-mem:
      arguments: {}
      inputs:
        parameters:
        - name: aws-ecr
          value: '{{workflow.parameters.aws-ecr}}'
      metadata: {}
      name: bwa-mem
      outputs: {}
      podSpecPatch: |
        containers:
          - name: main
            resources:
              requests:
                memory: "{{inputs.parameters.memory}}"
                cpu: "{{inputs.parameters.threads}}"
      script:
        command:
        - sh
        - -eux
        - --
        image: 'wgs-utils:latest'
        imagePullPolicy: 'always'
        name: ""
        resources: {}
        source: |
          mkdir --parents "$(dirname "{{inputs.parameters.output-name}}")"
        volumeMounts:
        - mountPath: /transfer
          name: transfer
    namespaced/s3-copy-templates/copy-processing-data:
      arguments: {}
      container:
        args:
        - ' python3  '
        command:
        - sh
        - -ceux
        - --
        env:
        - name: KUBERNETES_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        envFrom:
        - secretRef:
            name: '{{inputs.parameters.s3-secret-key-ref}}'
        image: 'pipe-src:latest'
        imagePullPolicy: 'always'
        name: ""
        resources:
          requests:
            cpu: "1"
            memory: 2Gi
        volumeMounts:
        - mountPath: /transfer
          name: transfer
      inputs:
        parameters:
        - name: s3-secret-key-ref
          value: '{{workflow.parameters.s3-secret-key-ref}}'
      metadata: {}
      name: copy-processing-data
      outputs: {}
    namespaced/s3-copy-templates/create-semaphore:
      arguments: {}
      container:
        args:
        - ' find '
        command:
        - sh
        - -ceux
        - --
        envFrom:
        - secretRef:
            name: '{{inputs.parameters.s3-secret-key-ref}}'
        image: 'pipe-src:latest'
        imagePullPolicy: 'always'
        name: ""
        resources:
          requests:
            cpu: "1"
            memory: 128Mi
      inputs:
        parameters:
        - name: semaphore-file
          value: '{{workflow.parameters.semaphore-file}}'
      metadata: {}
      name: create-semaphore
      outputs: {}
    namespaced/s3-copy-templates/download-static-data:
      arguments: {}
      container:
        args:
        - ' python3 '
        command:
        - sh
        - -ceux
        - --
        envFrom:
        - secretRef:
            name: '{{inputs.parameters.s3-secret-key-ref}}'
        image: 'pipe-src:latest'
        imagePullPolicy: 'always'
        name: ""
        resources:
          requests:
            cpu: "1"
            memory: 2Gi
        volumeMounts:
        - mountPath: /transfer
          name: transfer
      inputs:
        parameters:
        - name: s3-secret-key-ref
          value: '{{workflow.parameters.s3-secret-key-ref}}'
      metadata: {}
      name: download-static-data
      outputs: {}
    namespaced/samtools-templates/samtools-sort-and-index:
      arguments: {}
      inputs:
        parameters:
        - name: aws-ecr
          value: '{{workflow.parameters.aws-ecr}}'
      metadata: {}
      name: samtools-sort-and-index
      outputs: {}
      podSpecPatch: |
        containers:
          - name: main
            resources:
              requests:
                cpu: "{{inputs.parameters.cpu}}"
                memory: "{{inputs.parameters.memory}}"
      script:
        command:
        - sh
        - -eux
        - --
        image: 'wgs-utils:latest'
        imagePullPolicy: 'always'
        name: ""
        resources: {}
        source: |
          find
        volumeMounts:
        - mountPath: /transfer
          name: transfer

time="2020-10-27T17:02:23Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g
time="2020-10-27T17:02:24Z" level=info msg="Updated phase Running -> Succeeded" namespace=v1-0 workflow=wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g
time="2020-10-27T17:02:24Z" level=info msg="Marking workflow completed" namespace=v1-0 workflow=wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g
time="2020-10-27T17:02:24Z" level=info msg="Marking workflow as pending archiving" namespace=v1-0 workflow=wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g
time="2020-10-27T17:02:24Z" level=info msg="Deleting PVC wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g-transfer" namespace=v1-0 workflow=wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g
time="2020-10-27T17:02:24Z" level=info msg="Deleted 1/1 PVCs" namespace=v1-0 workflow=wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g
time="2020-10-27T17:02:24Z" level=info msg="Checking daemoned children of " namespace=v1-0 workflow=wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g
time="2020-10-27T17:02:24Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Succeeded resourceVersion=45510943 workflow=wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g
time="2020-10-27T17:02:32Z" level=info msg="Delete pod for gc successfully" namespace=v1-0 pod=wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g-3272914040
time="2020-10-27T17:02:32Z" level=info msg="archiving workflow" namespace=v1-0 uid=c7907ef1-754d-4246-947e-e6bb3ffbede1 workflow=wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g
time="2020-10-27T17:02:32Z" level=info msg="Delete pod for gc successfully" namespace=v1-0 pod=wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g-3935026858
time="2020-10-27T17:02:32Z" level=info msg="archiving workflow" namespace=v1-0 uid=c7907ef1-754d-4246-947e-e6bb3ffbede1 workflow=wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g
time="2020-10-27T17:02:32Z" level=info msg="Found Workflow v1-0/wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g set expire at 2020-10-27 17:04:24 +0000 UTC (1m51.026279313s from now)"
time="2020-10-27T17:02:32Z" level=info msg="Queueing workflow v1-0/wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g for delete in 1m51.026279313s"
time="2020-10-27T17:02:32Z" level=info msg="Found Workflow v1-0/wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g set expire at 2020-10-27 17:04:24 +0000 UTC (1m51.026269704s from now)"
time="2020-10-27T17:02:32Z" level=info msg="Queueing workflow v1-0/wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g for delete in 1m51.026269704s"
time="2020-10-27T17:02:33Z" level=info msg="Delete pod for gc successfully" namespace=v1-0 pod=wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g-2673349335
time="2020-10-27T17:02:33Z" level=info msg="Delete pod for gc successfully" namespace=v1-0 pod=wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g-884871069
time="2020-10-27T17:02:33Z" level=info msg="Delete pod for gc successfully" namespace=v1-0 pod=wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g-2405450570
time="2020-10-27T17:02:33Z" level=info msg="Delete pod for gc successfully" namespace=v1-0 pod=wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g-1957766120
time="2020-10-27T17:04:24Z" level=info msg="Deleting TTL expired workflow v1-0/wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g"
time="2020-10-27T17:04:24Z" level=info msg="Successfully deleted 'v1-0/wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g'"
time="2020-10-27T17:04:24Z" level=info msg="Found Workflow v1-0/wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g set expire at 2020-10-27 17:04:24 +0000 UTC (-307.655643ms from now)"
time="2020-10-27T17:04:24Z" level=info msg="Queueing workflow v1-0/wgs-alignment-1d106-tn7-v1-lane-hj7ycdsxy-004-dp65g for delete in 0s"

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.
@alexec alexec self-assigned this Oct 27, 2020
@alexec
Copy link
Contributor

alexec commented Oct 27, 2020

There's probably two bugs here:

  • We should always re-sync a workflow every 20m.
  • We should not miss events.

@alexec
Copy link
Contributor

alexec commented Oct 27, 2020

Can you please add the logs from before the restart of the controller?

@alexec
Copy link
Contributor

alexec commented Oct 27, 2020

Can you see if insignificant pod change is ever printed in the logs?

@TekTimmy
Copy link
Author

TekTimmy commented Oct 27, 2020 •
edited
Loading

Can you please add the logs from before the restart of the controller?

Logs with grepping for the workflow name BEFORE the restart when workflow is in Zombie mode:

time="2020-10-27T17:55:39Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:55:39Z" level=info msg="Updating node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 status Pending -> Running"
time="2020-10-27T17:55:39Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:55:39Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:55:39Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:55:39Z" level=info msg="Workflow step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850932395 not yet completed" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:55:39Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45676427 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:55:39Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:55:39Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:55:39Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:55:39Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:55:39Z" level=info msg="Workflow step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850932395 not yet completed" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:34Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:34Z" level=info msg="Setting node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 outputs"
time="2020-10-27T17:56:34Z" level=info msg="Updating node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 status Running -> Succeeded"
time="2020-10-27T17:56:34Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:34Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:34Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:34Z" level=info msg="Step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850932395 successful" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:34Z" level=info msg="node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850932395 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:34Z" level=info msg="node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850932395 finished: 2020-10-27 17:56:34.240017805 +0000 UTC" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:34Z" level=info msg="StepGroup node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850785300 initialized Running" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:34Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:34Z" level=info msg="Pod node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603 initialized Pending" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:34Z" level=info msg="Created pod: wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99(0)[4].dbsnp-annotation (wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603)" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:34Z" level=info msg="Workflow step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850785300 not yet completed" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:34Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45680294 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:35Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:35Z" level=info msg="Updating node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603 message: ContainerCreating"
time="2020-10-27T17:56:35Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:35Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:35Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:35Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:35Z" level=info msg="Skipped pod wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99(0)[4].dbsnp-annotation (wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:35Z" level=info msg="Workflow step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850785300 not yet completed" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:35Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45680355 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:49Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:49Z" level=info msg="Updating node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603 status Pending -> Running"
time="2020-10-27T17:56:49Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:49Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:49Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:49Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:49Z" level=info msg="Skipped pod wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99(0)[4].dbsnp-annotation (wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603) creation: already exists" namespace=v1-0 podPhase=Running workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:49Z" level=info msg="Workflow step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850785300 not yet completed" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:49Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45680837 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:51Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:51Z" level=info msg="Setting node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603 outputs"
time="2020-10-27T17:56:51Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:51Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:51Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:51Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:51Z" level=info msg="Skipped pod wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99(0)[4].dbsnp-annotation (wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603) creation: already exists" namespace=v1-0 podPhase=Running workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:51Z" level=info msg="Workflow step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850785300 not yet completed" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:51Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45680923 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:52Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:52Z" level=info msg="Updating node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603 status Running -> Succeeded"
time="2020-10-27T17:56:52Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:52Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:52Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:52Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:52Z" level=info msg="Step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850785300 successful" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:52Z" level=info msg="node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850785300 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:52Z" level=info msg="node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850785300 finished: 2020-10-27 17:56:52.605138081 +0000 UTC" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:52Z" level=info msg="StepGroup node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1991516297 initialized Running" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:52Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:52Z" level=info msg="Pod node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986 initialized Pending" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:52Z" level=info msg="Created pod: wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99(0)[5].post-process-data (wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986)" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:52Z" level=info msg="Workflow step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1991516297 not yet completed" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:52Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45680967 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:53Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:53Z" level=info msg="Updating node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986 message: PodInitializing"
time="2020-10-27T17:56:53Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:53Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:53Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:53Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:53Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:53Z" level=info msg="Skipped pod wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99(0)[5].post-process-data (wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:53Z" level=info msg="Workflow step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1991516297 not yet completed" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:56:53Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45680989 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:16Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:16Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:16Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:16Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:16Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:16Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:16Z" level=info msg="Skipped pod wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99(0)[5].post-process-data (wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:16Z" level=info msg="Workflow step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1991516297 not yet completed" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:18Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:18Z" level=info msg="Updating node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986 status Pending -> Running"
time="2020-10-27T17:57:18Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:18Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:18Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:18Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:18Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:18Z" level=info msg="Workflow step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1991516297 not yet completed" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:18Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45681648 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:20Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:20Z" level=info msg="Setting node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986 outputs"
time="2020-10-27T17:57:20Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:20Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:20Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:20Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:20Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:20Z" level=info msg="Workflow step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1991516297 not yet completed" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:20Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45681674 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:20Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:20Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:20Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:20Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:20Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:20Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:20Z" level=info msg="Workflow step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1991516297 not yet completed" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:21Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:21Z" level=info msg="Updating node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986 status Running -> Succeeded"
time="2020-10-27T17:57:21Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:21Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:21Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:21Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:21Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:21Z" level=info msg="Step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1991516297 successful" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:21Z" level=info msg="node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1991516297 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:21Z" level=info msg="node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1991516297 finished: 2020-10-27 17:57:21.24739606 +0000 UTC" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:21Z" level=info msg="StepGroup node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-917601586 initialized Running" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:21Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:21Z" level=info msg="Pod node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1891858199 initialized Pending" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:21Z" level=info msg="Created pod: wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99(0)[6].s3-upload-processing-data (wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1891858199)" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:21Z" level=info msg="Workflow step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-917601586 not yet completed" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:21Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45681736 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:22Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:22Z" level=info msg="Updating node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1891858199 message: ContainerCreating"
time="2020-10-27T17:57:22Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:22Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:22Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:22Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:22Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:22Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:22Z" level=info msg="Skipped pod wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99(0)[6].s3-upload-processing-data (wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1891858199) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:22Z" level=info msg="Workflow step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-917601586 not yet completed" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:22Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45681792 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:31Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:31Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:31Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:31Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:31Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:31Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:31Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:31Z" level=info msg="Skipped pod wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99(0)[6].s3-upload-processing-data (wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1891858199) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:31Z" level=info msg="Workflow step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-917601586 not yet completed" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:43Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:43Z" level=info msg="Updating node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1891858199 status Pending -> Running"
time="2020-10-27T17:57:43Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:43Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:43Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:43Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:43Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:43Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:43Z" level=info msg="Skipped pod wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99(0)[6].s3-upload-processing-data (wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1891858199) creation: already exists" namespace=v1-0 podPhase=Running workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:43Z" level=info msg="Workflow step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-917601586 not yet completed" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:43Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45682866 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:45Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:45Z" level=info msg="Setting node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1891858199 outputs"
time="2020-10-27T17:57:45Z" level=info msg="Updating node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1891858199 status Running -> Succeeded"
time="2020-10-27T17:57:45Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:45Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:45Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:45Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:45Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:45Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:45Z" level=info msg="Step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-917601586 successful" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:45Z" level=info msg="node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-917601586 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:45Z" level=info msg="node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-917601586 finished: 2020-10-27 17:57:45.549938271 +0000 UTC" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:45Z" level=info msg="StepGroup node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850344015 initialized Running" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:45Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1891858199 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1891858199]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:45Z" level=info msg="Pod node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-3672635988 initialized Pending" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:45Z" level=info msg="Created pod: wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99(0)[7].create-semaphore (wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-3672635988)" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:45Z" level=info msg="Workflow step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850344015 not yet completed" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:45Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45682951 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:46Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:46Z" level=info msg="Updating node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-3672635988 message: ContainerCreating"
time="2020-10-27T17:57:46Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:46Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:46Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:46Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:46Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:46Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:46Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1891858199 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1891858199]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:46Z" level=info msg="Skipped pod wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99(0)[7].create-semaphore (wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-3672635988) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:46Z" level=info msg="Workflow step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850344015 not yet completed" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:46Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45682984 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:53Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:53Z" level=info msg="Updating node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-3672635988 status Pending -> Running"
time="2020-10-27T17:57:53Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:53Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:53Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:53Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:53Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:53Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:53Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1891858199 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1891858199]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:53Z" level=info msg="Skipped pod wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99(0)[7].create-semaphore (wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-3672635988) creation: already exists" namespace=v1-0 podPhase=Running workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:53Z" level=info msg="Workflow step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850344015 not yet completed" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:53Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45683189 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:54Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:54Z" level=info msg="Setting node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-3672635988 outputs"
time="2020-10-27T17:57:54Z" level=info msg="Updating node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-3672635988 status Running -> Succeeded"
time="2020-10-27T17:57:54Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-662803440]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:54Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1242710242]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:54Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4218559810]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:54Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-142313805]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:54Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4219821603]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:54Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4140846986]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:54Z" level=info msg="SG Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1891858199 are [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-1891858199]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:54Z" level=info msg="Step group node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850344015 successful" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:54Z" level=info msg="node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850344015 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:54Z" level=info msg="node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-850344015 finished: 2020-10-27 17:57:54.256730504 +0000 UTC" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:54Z" level=info msg="Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-3672635988 is [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-3672635988]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:54Z" level=info msg="Outbound nodes of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4209631342 is [wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-3672635988]" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:54Z" level=info msg="node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4209631342 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:54Z" level=info msg="node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4209631342 finished: 2020-10-27 17:57:54.256801161 +0000 UTC" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:54Z" level=info msg="Checking daemoned children of wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99-4209631342" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:54Z" level=info msg="node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:54Z" level=info msg="node wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99 finished: 2020-10-27 17:57:54.25729694 +0000 UTC" namespace=v1-0 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99
time="2020-10-27T17:57:54Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45683247 workflow=wgs-gvc-annotation-1a1390-tp1-v1-chr-y-mlp99

@TekTimmy
Copy link
Author

Can you see if insignificant pod change is ever printed in the logs?

No output from kubectl logs workflow-controller-5b6c7f9b89-v6gst | grep "insignificant pod change" before AND after the restart

alexec added a commit to alexec/argo-workflows that referenced this issue Oct 27, 2020
@alexec
feat(controller): Deprecate pod-workers. Fixes argoproj#4398
772d9db 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
@alexec alexec mentioned this issue Oct 27, 2020
Closed
6 tasks
@alexec
Copy link
Contributor

alexec commented Oct 27, 2020

Engineering build: argoproj/workflow-controller:no-sig

@TekTimmy
Copy link
Author

Engineering build: argoproj/workflow-controller:no-sig

Zombie keeps living:

Name:                wgs-split-by-chr-1a1635-tp1-v1-kfb7w
Namespace:           v1-0
ServiceAccount:      argo
Status:              Running
Created:             Tue Oct 27 15:25:12 +0100 (3 hours ago)
Started:             Tue Oct 27 15:25:12 +0100 (3 hours ago)
Duration:            3 hours 55 minutes
ResourcesDuration:   14h3m13s*(1 cpu),409h58m26s*(100Mi memory)
Parameters:          
  static-data-dir:   /transfer/static-data
  processing-data-dir: /transfer/processing-data

STEP                                        TEMPLATE                                PODNAME                                          DURATION  MESSAGE
 ✔ wgs-split-by-chr-1a1635-tp1-v1-kfb7w(0)  main                                                                                                 
 ├---✔ s3-download-static-data              s3-copy-templates/download-static-data  wgs-split-by-chr-1a1635-tp1-v1-kfb7w-4159657123  26s         
 ├---✔ s3-download-processing-data          s3-copy-templates/copy-processing-data  wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2420392105  20m         
 ├---✔ chunk-wgs-bam-file-by-chromosome     chunk-wgs-bam-file-by-chromosome        wgs-split-by-chr-1a1635-tp1-v1-kfb7w-1847378718  2h          
 ├---✔ index-chr-bam-files                  index-chr-bam-files                     wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3161808260  29m         
 ├---✔ s3-upload-processing-data            s3-copy-templates/copy-processing-data  wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3806904737  18m         
 └---✔ create-semaphore                     s3-copy-templates/create-semaphore      wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2614434590  5s  

time="2020-10-27T18:17:39Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:39Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-4159657123 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-4159657123]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:39Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2420392105 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2420392105]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:39Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-1847378718 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-1847378718]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:39Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3161808260 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3161808260]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:39Z" level=info msg="Skipped pod wgs-split-by-chr-1a1635-tp1-v1-kfb7w(0)[4].s3-upload-processing-data (wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3806904737) creation: already exists" namespace=v1-0 podPhase=Running workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:39Z" level=info msg="Workflow step group node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3571551288 not yet completed" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:47Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:47Z" level=info msg="Setting node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3806904737 outputs"
time="2020-10-27T18:17:47Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-4159657123 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-4159657123]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:47Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2420392105 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2420392105]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:47Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-1847378718 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-1847378718]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:47Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3161808260 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3161808260]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:47Z" level=info msg="Skipped pod wgs-split-by-chr-1a1635-tp1-v1-kfb7w(0)[4].s3-upload-processing-data (wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3806904737) creation: already exists" namespace=v1-0 podPhase=Running workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:47Z" level=info msg="Workflow step group node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3571551288 not yet completed" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:47Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45738485 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:48Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:48Z" level=info msg="Updating node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3806904737 status Running -> Succeeded"
time="2020-10-27T18:17:48Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-4159657123 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-4159657123]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:48Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2420392105 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2420392105]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:48Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-1847378718 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-1847378718]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:48Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3161808260 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3161808260]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:48Z" level=info msg="Step group node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3571551288 successful" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:48Z" level=info msg="node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3571551288 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:48Z" level=info msg="node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3571551288 finished: 2020-10-27 18:17:48.157310459 +0000 UTC" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:48Z" level=info msg="StepGroup node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3638514669 initialized Running" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:48Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3806904737 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3806904737]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:48Z" level=info msg="Pod node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2614434590 initialized Pending" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:48Z" level=info msg="Created pod: wgs-split-by-chr-1a1635-tp1-v1-kfb7w(0)[5].create-semaphore (wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2614434590)" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:48Z" level=info msg="Workflow step group node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3638514669 not yet completed" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:48Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45738530 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:49Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:49Z" level=info msg="Updating node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2614434590 message: ContainerCreating"
time="2020-10-27T18:17:49Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-4159657123 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-4159657123]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:49Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2420392105 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2420392105]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:49Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-1847378718 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-1847378718]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:49Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3161808260 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3161808260]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:49Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3806904737 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3806904737]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:49Z" level=info msg="Skipped pod wgs-split-by-chr-1a1635-tp1-v1-kfb7w(0)[5].create-semaphore (wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2614434590) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:49Z" level=info msg="Workflow step group node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3638514669 not yet completed" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:49Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45738571 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:52Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:52Z" level=info msg="Updating node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2614434590 status Pending -> Running"
time="2020-10-27T18:17:52Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-4159657123 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-4159657123]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:52Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2420392105 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2420392105]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:52Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-1847378718 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-1847378718]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:52Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3161808260 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3161808260]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:52Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3806904737 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3806904737]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:52Z" level=info msg="Skipped pod wgs-split-by-chr-1a1635-tp1-v1-kfb7w(0)[5].create-semaphore (wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2614434590) creation: already exists" namespace=v1-0 podPhase=Running workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:52Z" level=info msg="Workflow step group node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3638514669 not yet completed" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:52Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45738711 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:54Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:54Z" level=info msg="Setting node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2614434590 outputs"
time="2020-10-27T18:17:54Z" level=info msg="Updating node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2614434590 status Running -> Succeeded"
time="2020-10-27T18:17:54Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-4159657123 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-4159657123]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:54Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2420392105 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2420392105]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:54Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-1847378718 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-1847378718]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:54Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3161808260 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3161808260]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:54Z" level=info msg="SG Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3806904737 are [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3806904737]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:54Z" level=info msg="Step group node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3638514669 successful" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:54Z" level=info msg="node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3638514669 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:54Z" level=info msg="node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3638514669 finished: 2020-10-27 18:17:54.740318449 +0000 UTC" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:54Z" level=info msg="Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2614434590 is [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2614434590]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:54Z" level=info msg="Outbound nodes of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3318990570 is [wgs-split-by-chr-1a1635-tp1-v1-kfb7w-2614434590]" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:54Z" level=info msg="node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3318990570 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:54Z" level=info msg="node wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3318990570 finished: 2020-10-27 18:17:54.740457476 +0000 UTC" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:54Z" level=info msg="Checking daemoned children of wgs-split-by-chr-1a1635-tp1-v1-kfb7w-3318990570" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:54Z" level=info msg="node wgs-split-by-chr-1a1635-tp1-v1-kfb7w phase Running -> Succeeded" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:54Z" level=info msg="node wgs-split-by-chr-1a1635-tp1-v1-kfb7w finished: 2020-10-27 18:17:54.741743222 +0000 UTC" namespace=v1-0 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w
time="2020-10-27T18:17:54Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45738770 workflow=wgs-split-by-chr-1a1635-tp1-v1-kfb7w

@alexec
Copy link
Contributor

alexec commented Oct 27, 2020

You'll need to run a new workflow to see if argoproj/workflow-controller:no-sig works.

@TekTimmy
Copy link
Author

You'll need to run a new workflow to see if argoproj/workflow-controller:no-sig works.

Ups okay! That is totally automated! Will provide as soon as one newly submitted is finished... We have looong running workflows ;)

@TekTimmy
Copy link
Author

TekTimmy commented Oct 27, 2020 •
edited
Loading

You'll need to run a new workflow to see if argoproj/workflow-controller:no-sig works.

This one still got a zombie: (grep for workflow)

time="2020-10-27T18:27:00Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="Setting node wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 outputs"
time="2020-10-27T18:27:00Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 status Pending -> Running"
time="2020-10-27T18:27:00Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2190224653 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45764554 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 status Running -> Succeeded"
time="2020-10-27T18:27:00Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="Step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2190224653 successful" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2190224653 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2190224653 finished: 2020-10-27 18:27:00.743633206 +0000 UTC" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="StepGroup node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2123261272 initialized Running" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="Pod node wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 initialized Pending" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="Created pod: wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[5].samtools-sort-and-index-viral-reads-and-mates (wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540)" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2123261272 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:00Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45764594 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 message: PodInitializing"
time="2020-10-27T18:27:01Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[5].samtools-sort-and-index-viral-reads-and-mates (wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2123261272 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45764740 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[5].samtools-sort-and-index-viral-reads-and-mates (wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2123261272 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[5].samtools-sort-and-index-viral-reads-and-mates (wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:01Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2123261272 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:06Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:06Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:06Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:06Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:06Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:06Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:06Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[5].samtools-sort-and-index-viral-reads-and-mates (wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:06Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2123261272 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:09Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:09Z" level=info msg="Setting node wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 outputs"
time="2020-10-27T18:27:09Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 status Pending -> Running"
time="2020-10-27T18:27:09Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:09Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:09Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:09Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:09Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:09Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2123261272 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:10Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45765139 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:11Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:11Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 status Running -> Succeeded"
time="2020-10-27T18:27:11Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:11Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:11Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:11Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:11Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:11Z" level=info msg="Step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2123261272 successful" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:11Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2123261272 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:11Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2123261272 finished: 2020-10-27 18:27:11.013862578 +0000 UTC" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:11Z" level=info msg="StepGroup node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2122716819 initialized Running" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:11Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:11Z" level=info msg="Pod node wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 initialized Pending" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:11Z" level=info msg="Created pod: wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[6].samtools-bam-to-sam-viral-reads-and-mates (wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850)" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:11Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2122716819 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:11Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45765182 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:12Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:12Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 message: PodInitializing"
time="2020-10-27T18:27:12Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:12Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:12Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:12Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:12Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:12Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:12Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[6].samtools-bam-to-sam-viral-reads-and-mates (wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:12Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2122716819 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:12Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45765224 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:24Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:24Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:24Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:24Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:24Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:24Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:24Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:24Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[6].samtools-bam-to-sam-viral-reads-and-mates (wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:24Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2122716819 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:27Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:27Z" level=info msg="Setting node wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 outputs"
time="2020-10-27T18:27:27Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 status Pending -> Running"
time="2020-10-27T18:27:27Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:27Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:27Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:27Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:27Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:27Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:27Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2122716819 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:27Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45765953 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:28Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:28Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 status Running -> Succeeded"
time="2020-10-27T18:27:28Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:28Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:28Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:28Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:28Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:28Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:28Z" level=info msg="Step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2122716819 successful" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:28Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2122716819 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:28Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2122716819 finished: 2020-10-27 18:27:28.160209795 +0000 UTC" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:28Z" level=info msg="StepGroup node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3263742006 initialized Running" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:28Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:28Z" level=info msg="Pod node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 initialized Pending" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:28Z" level=info msg="Created pod: wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[7].restruct-viral-with-mates (wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595)" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:28Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3263742006 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:28Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45766015 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 message: PodInitializing"
time="2020-10-27T18:27:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[7].restruct-viral-with-mates (wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3263742006 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45766055 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[7].restruct-viral-with-mates (wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:29Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3263742006 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:41Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:41Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:41Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:41Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:41Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:41Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:41Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:41Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:41Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[7].restruct-viral-with-mates (wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:41Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3263742006 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:44Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:44Z" level=info msg="Setting node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 outputs"
time="2020-10-27T18:27:44Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 status Pending -> Running"
time="2020-10-27T18:27:44Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:44Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:44Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:44Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:44Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:44Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:44Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:44Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3263742006 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:44Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45766598 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:45Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:45Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 status Running -> Succeeded"
time="2020-10-27T18:27:45Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:45Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:45Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:45Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:45Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:45Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:45Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:45Z" level=info msg="Step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3263742006 successful" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:45Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3263742006 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:45Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3263742006 finished: 2020-10-27 18:27:45.410116717 +0000 UTC" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:45Z" level=info msg="StepGroup node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2190813033 initialized Running" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:45Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:45Z" level=info msg="Pod node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 initialized Pending" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:45Z" level=info msg="Created pod: wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[8].restructure-ensembl-annotation-file (wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175)" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:45Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2190813033 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:45Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45766657 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:46Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:46Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 message: PodInitializing"
time="2020-10-27T18:27:46Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:46Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:46Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:46Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:46Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:46Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:46Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:46Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:46Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[8].restructure-ensembl-annotation-file (wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:46Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2190813033 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:46Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45766705 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:51Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:51Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[8].restructure-ensembl-annotation-file (wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:51Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2190813033 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:52Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:52Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:52Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:52Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:52Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:52Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:52Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:52Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:52Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:52Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[8].restructure-ensembl-annotation-file (wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:52Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2190813033 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:54Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:54Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 status Pending -> Running"
time="2020-10-27T18:27:54Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:54Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:54Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:54Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:54Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:54Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:54Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:54Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:54Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2190813033 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:27:54Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45767104 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:13Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:13Z" level=info msg="Setting node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 outputs"
time="2020-10-27T18:28:13Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 status Running -> Succeeded"
time="2020-10-27T18:28:13Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:13Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:13Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:13Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:13Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:13Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:13Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:13Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:13Z" level=info msg="Step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2190813033 successful" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:13Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2190813033 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:13Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2190813033 finished: 2020-10-27 18:28:13.632942089 +0000 UTC" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:13Z" level=info msg="StepGroup node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3197617268 initialized Running" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:13Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:13Z" level=info msg="Pod node wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 initialized Pending" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:13Z" level=info msg="Created pod: wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[9].intersect-annotated-mates (wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137)" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:13Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3197617268 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:13Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45767939 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:14Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:14Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 message: PodInitializing"
time="2020-10-27T18:28:14Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:14Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:14Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:14Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:14Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:14Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:14Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:14Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:14Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:14Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[9].intersect-annotated-mates (wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:14Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3197617268 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:14Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45767975 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:21Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:21Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:21Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:21Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:21Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:21Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:21Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:21Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:21Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:21Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:21Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[9].intersect-annotated-mates (wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:21Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3197617268 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:22Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:22Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:22Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:22Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:22Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:22Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:22Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:22Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:22Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:22Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:22Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[9].intersect-annotated-mates (wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:22Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3197617268 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:26Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:26Z" level=info msg="Setting node wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 outputs"
time="2020-10-27T18:28:26Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 status Pending -> Running"
time="2020-10-27T18:28:26Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:26Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:26Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:26Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:26Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:26Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:26Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:26Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:26Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:26Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3197617268 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:26Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45768488 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:27Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:27Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 status Running -> Succeeded"
time="2020-10-27T18:28:27Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:27Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:27Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:27Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:27Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:27Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:27Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:27Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:27Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:27Z" level=info msg="Step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3197617268 successful" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:27Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3197617268 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:27Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3197617268 finished: 2020-10-27 18:28:27.828225222 +0000 UTC" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:27Z" level=info msg="StepGroup node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3481716562 initialized Running" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:27Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:27Z" level=info msg="Pod node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 initialized Pending" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:27Z" level=info msg="Created pod: wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[10].export-virus-integration-sites (wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972)" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:27Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3481716562 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:27Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45768546 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:28Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:28Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 message: ContainerCreating"
time="2020-10-27T18:28:28Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:28Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:28Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:28Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:28Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:28Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:28Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:28Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:28Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:28Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:28Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[10].export-virus-integration-sites (wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:28Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3481716562 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:28:28Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45768577 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:06Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:06Z" level=info msg="Setting node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 outputs"
time="2020-10-27T18:29:06Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 status Pending -> Running"
time="2020-10-27T18:29:06Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:06Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:06Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:06Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:06Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:06Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:06Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:06Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:06Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:06Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:06Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[10].export-virus-integration-sites (wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972) creation: already exists" namespace=v1-0 podPhase=Running workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:06Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3481716562 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:06Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45770036 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 status Running -> Succeeded"
time="2020-10-27T18:29:07Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="Step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3481716562 successful" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3481716562 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3481716562 finished: 2020-10-27 18:29:07.214298982 +0000 UTC" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="StepGroup node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3414458991 initialized Running" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="Pod node wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216 initialized Pending" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="Created pod: wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[11].gzip-csv-files (wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216)" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3414458991 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:07Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45770066 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216 message: PodInitializing"
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[11].gzip-csv-files (wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3414458991 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45770095 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[11].gzip-csv-files (wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:08Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3414458991 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:44Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:44Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:44Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:44Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:44Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:44Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:44Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:44Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:44Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:44Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:44Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:44Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:44Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[11].gzip-csv-files (wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:44Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3414458991 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:45Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:45Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:45Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:45Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:45Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:45Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:45Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:45Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:45Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:45Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:45Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:45Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:45Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[11].gzip-csv-files (wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:45Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3414458991 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:49Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:49Z" level=info msg="Setting node wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216 outputs"
time="2020-10-27T18:29:49Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216 status Pending -> Running"
time="2020-10-27T18:29:49Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:49Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:49Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:49Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:49Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:49Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:49Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:49Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:49Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:49Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:49Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:49Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3414458991 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:49Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45771436 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216 status Running -> Succeeded"
time="2020-10-27T18:29:50Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="Step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3414458991 successful" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3414458991 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3414458991 finished: 2020-10-27 18:29:50.338445426 +0000 UTC" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="StepGroup node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3415003444 initialized Running" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="Pod node wgs-viral-hits-1a818-0-c1-v1-cjjpl-564097152 initialized Pending" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="Created pod: wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[12].upload-results (wgs-viral-hits-1a818-0-c1-v1-cjjpl-564097152)" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3415003444 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:50Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45771484 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:51Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:51Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-564097152 message: ContainerCreating"
time="2020-10-27T18:29:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:51Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:51Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[12].upload-results (wgs-viral-hits-1a818-0-c1-v1-cjjpl-564097152) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:51Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3415003444 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:51Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45771519 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:53Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:53Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:53Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:53Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:53Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:53Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:53Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:53Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:53Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:53Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:53Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:53Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:53Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:53Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[12].upload-results (wgs-viral-hits-1a818-0-c1-v1-cjjpl-564097152) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:29:53Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3415003444 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:29Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:29Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-564097152 status Pending -> Running"
time="2020-10-27T18:30:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:29Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:29Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[12].upload-results (wgs-viral-hits-1a818-0-c1-v1-cjjpl-564097152) creation: already exists" namespace=v1-0 podPhase=Running workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:29Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3415003444 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:29Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45772850 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:31Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:31Z" level=info msg="Setting node wgs-viral-hits-1a818-0-c1-v1-cjjpl-564097152 outputs"
time="2020-10-27T18:30:31Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:31Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:31Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:31Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:31Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:31Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:31Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:31Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:31Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:31Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:31Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:31Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:31Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[12].upload-results (wgs-viral-hits-1a818-0-c1-v1-cjjpl-564097152) creation: already exists" namespace=v1-0 podPhase=Running workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:31Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3415003444 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:31Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45772900 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-564097152 status Running -> Succeeded"
time="2020-10-27T18:30:32Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="Step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3415003444 successful" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3415003444 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-3415003444 finished: 2020-10-27 18:30:32.214222445 +0000 UTC" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="StepGroup node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2408199209 initialized Running" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-564097152 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-564097152]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="Pod node wgs-viral-hits-1a818-0-c1-v1-cjjpl-388176554 initialized Pending" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="Created pod: wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[13].create-semaphore (wgs-viral-hits-1a818-0-c1-v1-cjjpl-388176554)" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2408199209 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:32Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45772941 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:33Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:33Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-388176554 message: ContainerCreating"
time="2020-10-27T18:30:33Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:33Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:33Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:33Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:33Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:33Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:33Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:33Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:33Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:33Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:33Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:33Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:33Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-564097152 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-564097152]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:33Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[13].create-semaphore (wgs-viral-hits-1a818-0-c1-v1-cjjpl-388176554) creation: already exists" namespace=v1-0 podPhase=Pending workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:33Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2408199209 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:33Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45772976 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:38Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:38Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-388176554 status Pending -> Running"
time="2020-10-27T18:30:38Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:38Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:38Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:38Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:38Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:38Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:38Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:38Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:38Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:38Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:38Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:38Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:38Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-564097152 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-564097152]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:38Z" level=info msg="Skipped pod wgs-viral-hits-1a818-0-c1-v1-cjjpl(0)[13].create-semaphore (wgs-viral-hits-1a818-0-c1-v1-cjjpl-388176554) creation: already exists" namespace=v1-0 podPhase=Running workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:38Z" level=info msg="Workflow step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2408199209 not yet completed" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:38Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45773150 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="Processing workflow" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="Setting node wgs-viral-hits-1a818-0-c1-v1-cjjpl-388176554 outputs"
time="2020-10-27T18:30:40Z" level=info msg="Updating node wgs-viral-hits-1a818-0-c1-v1-cjjpl-388176554 status Running -> Succeeded"
time="2020-10-27T18:30:40Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1969751806]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4087802134]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3121305946]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2823266272]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-664688444]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-1144152540]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-475336850]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2582305595]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-3647573175]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-101985137]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-2352636972]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-4250709216]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="SG Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-564097152 are [wgs-viral-hits-1a818-0-c1-v1-cjjpl-564097152]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="Step group node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2408199209 successful" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2408199209 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-2408199209 finished: 2020-10-27 18:30:40.15770452 +0000 UTC" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-388176554 is [wgs-viral-hits-1a818-0-c1-v1-cjjpl-388176554]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="Outbound nodes of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1170477805 is [wgs-viral-hits-1a818-0-c1-v1-cjjpl-388176554]" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-1170477805 phase Running -> Succeeded" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl-1170477805 finished: 2020-10-27 18:30:40.157805691 +0000 UTC" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="Checking daemoned children of wgs-viral-hits-1a818-0-c1-v1-cjjpl-1170477805" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl phase Running -> Succeeded" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="node wgs-viral-hits-1a818-0-c1-v1-cjjpl finished: 2020-10-27 18:30:40.159662922 +0000 UTC" namespace=v1-0 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl
time="2020-10-27T18:30:40Z" level=info msg="Workflow update successful" namespace=v1-0 phase=Running resourceVersion=45773180 workflow=wgs-viral-hits-1a818-0-c1-v1-cjjpl

@TekTimmy
Copy link
Author

But it seems like they are cleaned up from time to time!

@alexec
Copy link
Contributor

alexec commented Oct 27, 2020

But it seems like they are cleaned up from time to time!

@TekTimmy I'm assuming that no-sig did not fix your issue. Is this correct? Or did it fix it, but only after some time (maybe 30m)?

@TekTimmy
Copy link
Author

TekTimmy commented Oct 27, 2020 •
edited
Loading

But it seems like they are cleaned up from time to time!

@TekTimmy I'm assuming that no-sig did not fix your issue. Is this correct? Or did it fix it, but only after some time (maybe 30m)?

Correct, there are still zombie workflows marked as "Running". And Yes that 30m sync seems to work but even in that 30m those zombies are blocking our queue. Let me know how I can help, we could also have a call or something

@alexec
Copy link
Contributor

alexec commented Oct 27, 2020

That is good news. no-sig cause workflows to re-sync after 30m and fixes the workflow. However, that is really too long. So now we must understand why they did not reconcile earlier.

I'm going to make another fix and send it to you shortly.

@alexec
Copy link
Contributor

alexec commented Oct 27, 2020

Sidenote: maybe similar to #4048

@alexec
Copy link
Contributor

alexec commented Oct 27, 2020 •
edited
Loading

Sidenote:

Interested in how you want the progress to be shown in the CLI. Do you think this would be a good default? If you want to submit a PR, it would be as easy as adding this one line:

// +kubebuilder:printcolumn:name="Progress",type="string",JSONPath=".status.progress",description="The workflow progress"

to workflow_types.go

@alexec
Copy link
Contributor

alexec commented Oct 27, 2020

Can you please try argoproj/workflow-controller:no-sig again (you'll need to pull the image again).

@TekTimmy
Copy link
Author

TekTimmy commented Oct 27, 2020 •
edited
Loading

Can you please try argoproj/workflow-controller:no-sig again (you'll need to pull the image again).

Have set ImagePullPolicy: Always in the Deployment to update to the latest version.
An error occured:

E1027 19:31:53.199213       1 runtime.go:78] Observed a panic: &errors.errorString{s:"unable to calculate an index entry for key \"v1-0/wgs-cronjob-svc-varscan-1603793880-qj7mc\" on index \"workflow\": object has no workflow label"} (unable to calculate an index entry for key "v1-0/wgs-cronjob-svc-varscan-1603793880-qj7mc" on index "workflow": object has no workflow label)
goroutine 197 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x17bf940, 0xc00c538560)
        /Users/acollins8/go/pkg/mod/k8s.io/apimachinery@v0.17.8/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /Users/acollins8/go/pkg/mod/k8s.io/apimachinery@v0.17.8/pkg/util/runtime/runtime.go:48 +0x82
panic(0x17bf940, 0xc00c538560)
        /usr/local/Cellar/go@1.13/1.13.15/libexec/src/runtime/panic.go:679 +0x1b2
k8s.io/client-go/tools/cache.(*threadSafeMap).updateIndices(0xc000440cf0, 0x0, 0x0, 0x19de0e0, 0xc006e3e380, 0xc007e2c3c0, 0x2d)
        /Users/acollins8/go/pkg/mod/k8s.io/client-go@v0.17.8/tools/cache/thread_safe_store.go:259 +0x4a6
k8s.io/client-go/tools/cache.(*threadSafeMap).Add(0xc000440cf0, 0xc007e2c3c0, 0x2d, 0x19de0e0, 0xc006e3e380)
        /Users/acollins8/go/pkg/mod/k8s.io/client-go@v0.17.8/tools/cache/thread_safe_store.go:73 +0x173
k8s.io/client-go/tools/cache.(*cache).Add(0xc00062f460, 0x19de0e0, 0xc006e3e380, 0x0, 0x0)
        /Users/acollins8/go/pkg/mod/k8s.io/client-go@v0.17.8/tools/cache/store.go:128 +0x102
k8s.io/client-go/tools/cache.(*sharedIndexInformer).HandleDeltas(0xc000664120, 0x17fc600, 0xc007dfe400, 0x0, 0x0)
        /Users/acollins8/go/pkg/mod/k8s.io/client-go@v0.17.8/tools/cache/shared_informer.go:464 +0x211
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc000530000, 0xc0008081e0, 0x0, 0x0, 0x0, 0x0)
        /Users/acollins8/go/pkg/mod/k8s.io/client-go@v0.17.8/tools/cache/delta_fifo.go:422 +0x230
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc000406280)
        /Users/acollins8/go/pkg/mod/k8s.io/client-go@v0.17.8/tools/cache/controller.go:153 +0x40
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000634e00)
        /Users/acollins8/go/pkg/mod/k8s.io/apimachinery@v0.17.8/pkg/util/wait/wait.go:152 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000825e00, 0x3b9aca00, 0x0, 0xc00015a401, 0xc00019ccc0)
        /Users/acollins8/go/pkg/mod/k8s.io/apimachinery@v0.17.8/pkg/util/wait/wait.go:153 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /Users/acollins8/go/pkg/mod/k8s.io/apimachinery@v0.17.8/pkg/util/wait/wait.go:88
k8s.io/client-go/tools/cache.(*controller).Run(0xc000406280, 0xc00019ccc0)
        /Users/acollins8/go/pkg/mod/k8s.io/client-go@v0.17.8/tools/cache/controller.go:125 +0x2fd
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc000664120, 0xc00019ccc0)
        /Users/acollins8/go/pkg/mod/k8s.io/client-go@v0.17.8/tools/cache/shared_informer.go:331 +0x47f
created by github.com/argoproj/argo/workflow/controller.(*WorkflowController).Run
        /Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/controller.go:185 +0x648
E1027 19:31:55.437975       1 shared_informer.go:662] unrecognized notification: <nil>

gonna monitor the zombies now

@TekTimmy
Copy link
Author

Sidenote:

Interested in how you want the progress to be shown in the CLI. Do you think this would be a good default? If you want to submit a PR, it would be as easy as adding this one line:

// +kubebuilder:printcolumn:name="Progress",type="string",JSONPath=".status.progress",description="The workflow progress"

to workflow_types.go

Thanks for the hint! I'm pretty interested to support this project but my go skills are very weak.. have to strengthen them first

@TekTimmy
Copy link
Author

TekTimmy commented Oct 27, 2020 •
edited
Loading

Can you please try argoproj/workflow-controller:no-sig again (you'll need to pull the image again).

workflow controller stopped logging shortly after the error message.
Might running the different workflow-controller versions after the other cause this? I'm working on production here, would be hard to reproduce in test environment I think.

@alexec
Copy link
Contributor

alexec commented Oct 27, 2020

Again with argoproj/workflow-controller:no-sig?

@TekTimmy
Copy link
Author

TekTimmy commented Oct 27, 2020 •
edited
Loading

Again with argoproj/workflow-controller:no-sig?

Error is gone, zombies are still summoned and no log entry at all about those zombie workflows in the workflow-controller logs =/
Its like the workflow-controller does not see them.

@alexec
Copy link
Contributor

alexec commented Oct 27, 2020

Thank you for helping with the testing. I'm really keen to nail down a fix for this.

@alexec
Copy link
Contributor

alexec commented Oct 27, 2020

Can you tell me what labels the zombies have? Do the have a workflows.argoproj.io/completed label?

@TekTimmy
Copy link
Author

Thank you for helping with the testing. I'm really keen to nail down a fix for this.

Thank YOU for developing! I'm happy that I can help.

@TekTimmy
Copy link
Author

TekTimmy commented Oct 27, 2020 •
edited
Loading

Can you tell me what labels the zombies have? Do the have a workflows.argoproj.io/completed label?

These are all the labels

    chromosome: "1"
    sample-dataset-id: 1A-0-C1_v1
    sample-id: 1A-0-C1
    type: wgs
    workflow: gvc-annotation
    workflows.argoproj.io/creator: system-serviceaccount-v1-0-mocca-cron
    workflows.argoproj.io/phase: Running

ouh shit, does our own "workflow" label interfere ?

@TekTimmy
Copy link
Author

TekTimmy commented Oct 27, 2020 •
edited
Loading

We have just moved to wintertime (-1 hour) so its nearly midnight to me in EST time zone... have to get some sleep. Looking forward to test some more tomorrow.

@alexec
Copy link
Contributor

alexec commented Oct 27, 2020

  1. Your workflow label should not interfere.
  2. Can you check for the string conflict in your controller logs?

@TekTimmy
Copy link
Author

TekTimmy commented Oct 28, 2020 •
edited
Loading 

1. Your `workflow` label should not interfere.

2. Can you check for the string `conflict` in your controller logs?
  1. No match, but i switched back to image argoproj/workflow-controller:latest. Let me know if I should check for conflict with image tag no-sig.

@alexec
Copy link
Contributor

alexec commented Oct 28, 2020

This doesn't make sense:

    workflows.argoproj.io/phase: Running
    workflows.argoproj.io/workflow-archiving-status: Archived

No archived workflow should ever be running. They should always be completed.

@alexec
Copy link
Contributor

alexec commented Oct 28, 2020

I'm starting to think something else is interfering. Can you run the workflow, but follow it using kubectl get wf --watch my-wf.

@TekTimmy
Copy link
Author

TekTimmy commented Oct 30, 2020 •
edited
Loading

After discussions with alexec and some tests to solve it zombie workflows still occur from time to time but get cleaned up in an acceptable time. I was not able to reproduce a workflow in phase Running with workflow-archiving-status: Archived anymore. A clear relation / cause was not found as far as I understand.

@alexec
Copy link
Contributor

alexec commented Oct 30, 2020

I'm going to re-open. I think we should actually clean them up a bit quicker.

@alexec alexec reopened this Oct 30, 2020
alexec added a commit to alexec/argo-workflows that referenced this issue Oct 30, 2020
@alexec
feat(controller): Always reconcille every 2m. See argoproj#4398
e3ddcaf 
Signed-off-by: Alex Collins <alex_collins@intuit.com>
@alexec alexec mentioned this issue Oct 30, 2020
Closed
6 tasks
@alexec alexec closed this as completed Nov 19, 2020
@alexec alexec mentioned this issue Nov 19, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alexec alexec

Labels
type/bug
Projects
None yet
Milestone
No milestone
2 participants
@alexec @TekTimmy