Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

semaphore not be released when step is finished #4870

Closed
juchaosong opened this issue Jan 13, 2021 · 9 comments · Fixed by #4847
Closed

semaphore not be released when step is finished #4870

juchaosong opened this issue Jan 13, 2021 · 9 comments · Fixed by #4847
Assignees
Labels

Comments

@juchaosong
Copy link
Contributor

Summary

What happened/what you expected to happen?
semaphore can be released when step is finished

Diagnostics

What Kubernetes provider are you using?
ACK

What version of Argo Workflows are you running?
v2.12.4

Paste a workflow that reproduces the bug, including status:
kubectl get wf -o yaml ${workflow} 
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  annotations:
    workflows.inceptio.ai/failed-channel: workflow-failed-prod
    workflows.inceptio.ai/succeeded-channel: workflow-succeeded-prod
  creationTimestamp: "2021-01-12T06:42:59Z"
  generation: 106
  labels:
    cybertron_commit_id: eeb8b1ccf2f95199cf0faeb52772c7ce8ad55a24
    workflows.argoproj.io/controller-instanceid: kraken-v2.12
    workflows.argoproj.io/phase: Running
    workflows.inceptio.ai/author: cybertron-gitlab
    workflows.inceptio.ai/template: gems-wf-v0-8
  name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
  namespace: argo-hubble
  resourceVersion: "1659280909"
  selfLink: /apis/argoproj.io/v1alpha1/namespaces/argo-hubble/workflows/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
  uid: 45cd47b1-c6d1-4b2e-9b2f-53343062a3b2
spec:
  arguments:
    parameters:
    - name: user
      value: XiaoLing Li
    - name: cybertron_commit_id
      value: eeb8b1ccf2f95199cf0faeb52772c7ce8ad55a24
    - name: cybertron_branch
      value: master
    - name: regression_commit_id
      value: master
    - name: test_case_group
      value: control_tests
  entrypoint: gems-entrypoint
  onExit: exit-handler
  parallelism: 100
  synchronization:
    semaphore:
      configMapKeyRef:
        key: gems_wf_limited
        name: gems-sync
  templates:
  - arguments: {}
    inputs: {}
    metadata: {}
    name: gems-entrypoint
    outputs: {}
    steps:
    - - arguments: {}
        name: build-cybertron-pkg-step
        template: build-cybertron-pkg
      - arguments: {}
        name: build-cybertron-sim-step
        template: build-cybertron-sim
      - arguments: {}
        name: build-cybertron-sim-p1-step
        template: build-cybertron-sim-p1
      - arguments: {}
        name: fetch-regression-repository-step
        template: fetch-regression-repository
    - - arguments: {}
        name: fetch-test-group-step
        template: fetch-test-group
    - - arguments:
          parameters:
          - name: test_case_name
            value: '{{item.test_case_name}}'
          - name: gpu_requirement
            value: '{{item.gpu_requirement}}'
          - name: truck_sim
            value: '{{item.truck_sim}}'
        continueOn:
          failed: true
        name: regression-step
        template: regression
        withParam: '{{steps.fetch-test-group-step.outputs.result}}'
    - - arguments: {}
        continueOn:
          failed: true
        name: aggregation-step
        template: aggregation
    - - arguments: {}
        name: ingest-step
        template: ingest
  - arguments: {}
    inputs: {}
    metadata: {}
    name: exit-handler
    outputs: {}
    steps:
    - - arguments: {}
        name: celebrate
        template: celebrate
        when: '{{workflow.status}} == Succeeded'
      - arguments: {}
        name: cry
        template: cry
        when: '{{workflow.status}} != Succeeded'
  - arguments: {}
    inputs:
      parameters:
      - name: cybertron_commit_timestamp
        value: '{{workflow.outputs.parameters.global_cybertron_commit_timestamp}}'
      - name: cybertron_commit_author
        value: '{{workflow.outputs.parameters.global_cybertron_commit_author}}'
    metadata: {}
    name: celebrate
    outputs: {}
    script:
      command:
      - bash
      env:
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: CYBERTRON_BRANCH
        value: '{{workflow.parameters.cybertron_branch}}'
      - name: CYBERTRON_COMMIT_AUTHOR
        value: '{{inputs.parameters.cybertron_commit_author}}'
      - name: TEST_CASE_GROUP
        value: '{{workflow.parameters.test_case_group}}'
      - name: TRIGGER_USER
        value: '{{workflow.parameters.user}}'
      - name: GEMS_NOTIFY_WECHAT_URL
        valueFrom:
          configMapKeyRef:
            key: gems_wechat_url
            name: cluster-configs
      - name: CHARIOT_WEB_HOST
        valueFrom:
          configMapKeyRef:
            key: cht_web_host
            name: cluster-configs
      - name: GRAFANA_WEB_HOST
        valueFrom:
          configMapKeyRef:
            key: grafana_web_host
            name: cluster-configs
      image: everpeace/curl-jq
      name: ""
      resources: {}
      source: |
        CHARIOT_URL="$CHARIOT_WEB_HOST/workflows/{{workflow.name}}"
        GRAFANA_URL="${GRAFANA_WEB_HOST}/d/R6oDol5Gz/gems-metrics?orgId=1&var-branch=${CYBERTRON_BRANCH}&var-commitId=${CYBERTRON_COMMIT_ID}&var-workflow={{workflow.name}}&from=now-30d&to=now"
        GRAFANA_WF_STATUS_URL="${GRAFANA_WEB_HOST}/d/ouaL2d2Mz/gems-workflows?orgId=1&var-workflow={{workflow.name}}"
        curl -s "${GEMS_NOTIFY_WECHAT_URL}" \
        -H 'Content-Type: application/json' \
        -d '{
                "msgtype": "markdown",
                "markdown": {
                        "content": "<font color=\"info\">Regression test workflow succeed!!!</font>
                        > workflow_id:<font color=\"comment\">'"{{workflow.name}}"'</font>
                        > commit_id:<font color=\"comment\">'"${CYBERTRON_COMMIT_ID:-"UNKNOWN"}"'</font>
                        > branch:<font color=\"comment\">'"${CYBERTRON_BRANCH:-"UNKNOWN"}"'</font>
                        > author:<font color=\"comment\">'"${CYBERTRON_COMMIT_AUTHOR:-"UNKNOWN"}"'</font>
                        > user:<font color=\"comment\">'"${TRIGGER_USER:-"UNKNOWN"}"'</font>
                        > test_group:<font color=\"comment\">'"${TEST_CASE_GROUP:-"UNKNOWN"}"'</font>
                        [Chariot Link]('${CHARIOT_URL}')
                        [Grafana Link]('${GRAFANA_URL}')
                        [Grafana Workflow Status Link]('${GRAFANA_WF_STATUS_URL}')"
                }
            }'
      volumeMounts:
      - mountPath: /tmpdir
        name: tmpdir
  - arguments: {}
    inputs:
      parameters:
      - name: cybertron_commit_timestamp
        value: '{{workflow.outputs.parameters.global_cybertron_commit_timestamp}}'
      - name: cybertron_commit_author
        value: '{{workflow.outputs.parameters.global_cybertron_commit_author}}'
    metadata: {}
    name: cry
    outputs: {}
    script:
      command:
      - bash
      env:
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: CYBERTRON_BRANCH
        value: '{{workflow.parameters.cybertron_branch}}'
      - name: CYBERTRON_COMMIT_AUTHOR
        value: '{{inputs.parameters.cybertron_commit_author}}'
      - name: TRIGGER_USER
        value: '{{workflow.parameters.user}}'
      - name: TEST_CASE_GROUP
        value: '{{workflow.parameters.test_case_group}}'
      - name: GEMS_NOTIFY_WECHAT_URL
        valueFrom:
          configMapKeyRef:
            key: gems_wechat_url
            name: cluster-configs
      - name: CHARIOT_WEB_HOST
        valueFrom:
          configMapKeyRef:
            key: cht_web_host
            name: cluster-configs
      image: everpeace/curl-jq
      name: ""
      resources: {}
      source: |
        CHARIOT_URL="$CHARIOT_WEB_HOST/workflows/{{workflow.name}}"
        GRAFANA_WF_STATUS_URL="${GRAFANA_WEB_HOST}/d/ouaL2d2Mz/gems-workflows?orgId=1&var-workflow={{workflow.name}}"
        curl -s "${GEMS_NOTIFY_WECHAT_URL}" -H 'Content-Type: application/json' \
        -d '{
              "msgtype": "markdown",
              "markdown": {
              "content": "<font color=\"warning\">Regression test workflow failed!!!</font>
                          > workflow_id:<font color=\"comment\">'"{{workflow.name}}"'</font>
                          > commit_id:<font color=\"comment\">'"${CYBERTRON_COMMIT_ID:-"UNKNOWN"}"'</font>
                          > branch:<font color=\"comment\">'"${CYBERTRON_BRANCH:-"UNKNOWN"}"'</font>
                          > author:<font color=\"comment\">'"${CYBERTRON_COMMIT_AUTHOR:-"UNKNOWN"}"'</font>
                          > user:<font color=\"comment\">'"${TRIGGER_USER:-"UNKNOWN"}"'</font>
                          > test_group:<font color=\"comment\">'"${TEST_CASE_GROUP:-"UNKNOWN"}"'</font>
                          Please refer: [Chariot Link]('${CHARIOT_URL}')
                                        [Grafana Workflow Status Link]('${GRAFANA_WF_STATUS_URL}')"
              }
        }'
      volumeMounts:
      - mountPath: /tmpdir
        name: tmpdir
  - arguments: {}
    inputs: {}
    metadata: {}
    name: build-cybertron-pkg
    outputs:
      parameters:
      - globalName: global_cybertron_commit_timestamp
        name: cybertron_commit_timestamp
        valueFrom:
          path: /tmpdir/cybertron_commit_timestamp.txt
      - globalName: global_cybertron_commit_author
        name: cybertron_commit_author
        valueFrom:
          path: /tmpdir/cybertron_commit_author.txt
      - globalName: global_idl_version
        name: idl_version
        valueFrom:
          path: /tmpdir/idl_version.txt
    retryStrategy:
      limit: 2
    script:
      command:
      - bash
      env:
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/sde_cybertron_release:2020_07_03_c12745f7
      name: ""
      resources: {}
      source: |
        set -ex

        ssh-keyscan gitlab.rd.inceptioglobal.ai >> ~/.ssh/known_hosts
        chmod 644 ~/.ssh/known_hosts

        export HTTP_PROXY=http://gfw-proxy.inceptio.tech:8118
        export HTTPS_PROXY=http://gfw-proxy.inceptio.tech:8118
        export NO_PROXY=.cn,.aliyun.com,.aliyuncs.com,.chariot,.artifactory

        mkdir -p /root/cybertron && cd /root/cybertron
        git init
        git remote add origin https://gitlab:$GITLAB_TOKEN@gitlab.rd.inceptioglobal.ai/autonomy/cybertron.git
        git fetch --depth=1 origin $CYBERTRON_COMMIT_ID
        git reset --hard FETCH_HEAD
        echo "\"$(git show -s --format=%ct HEAD)\"" > /tmpdir/cybertron_commit_timestamp.txt
        git show -s --format='%ae' HEAD > /tmpdir/cybertron_commit_author.txt

        # FXIME: install empy when buiding image
        pip3 install empy

        bazel build \
          --define tensorrt=default \
          --remote_cache=grpc://bb-storage.inceptio.tech:8980 \
          --remote_instance_name=cybertron \
          //:bin_pkg_tar //interfaces/tools/gems/pb_converter:all

        # get idl version
        if [ -f irs/common/recorder/idl_record_io.h ];then
          IDL_RECORD_IO=irs/common/recorder/idl_record_io.h
        else
          IDL_RECORD_IO=bazel-cybertron/external/wheeljack_x86_64/irs/common/recorder/idl_record_io.h
        fi
        MAJOR=$(cat $IDL_RECORD_IO | sed -rn 's/^.*kCurrentRepoVersionMajor\ *=\ *([0-9]*);/\1/p')
        MINOR=$(cat $IDL_RECORD_IO | sed -rn 's/^.*kCurrentRepoVersionMinor\ *=\ *([0-9]*);/\1/p')
        IDL=$(cat $IDL_RECORD_IO | sed -rn 's/^.*kCurrentIdlHiddenVersion\ *=\ *([0-9]*);/\1/p')
        echo ${MAJOR}.${MINOR}.${IDL} > /tmpdir/idl_version.txt

        cd ..
        tar --exclude='cybertron/bazel-cybertron' \
          --exclude='cybertron/bazel-out' \
          -h -cf cybertron-pkg.tar cybertron
        ls -alh cybertron-pkg.tar
        cp cybertron-pkg.tar /data
      volumeMounts:
      - mountPath: /root/.jfrog/jfrog-cli.conf
        name: credentials-volume
        subPath: jfrog-cli.conf
      - mountPath: /data
        name: artifacts-volume-claim
      - mountPath: /root/.ssh/id_rsa
        name: gitlab-ssh-key-volume
        subPath: id_rsa
      - mountPath: /tmpdir
        name: tmpdir
  - arguments: {}
    inputs: {}
    metadata: {}
    name: build-cybertron-sim
    outputs: {}
    retryStrategy:
      limit: 2
    script:
      command:
      - bash
      env:
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/sde_cybertron_release:2020_07_03_c12745f7
      name: ""
      resources: {}
      source: |
        set -ex

        ssh-keyscan gitlab.rd.inceptioglobal.ai >> ~/.ssh/known_hosts
        chmod 644 ~/.ssh/known_hosts

        export HTTP_PROXY=http://gfw-proxy.inceptio.tech:8118
        export HTTPS_PROXY=http://gfw-proxy.inceptio.tech:8118
        export NO_PROXY=.cn,.aliyun.com,.aliyuncs.com,.chariot,.artifactory

        mkdir -p /root/cybertron && cd /root/cybertron
        git init
        git remote add origin https://gitlab:$GITLAB_TOKEN@gitlab.rd.inceptioglobal.ai/autonomy/cybertron.git
        git fetch --depth=1 origin $CYBERTRON_COMMIT_ID
        git reset --hard FETCH_HEAD

        # FXIME: install empy when buiding image
        pip3 install empy

        bazel build \
          --define tensorrt=default \
          --remote_cache=grpc://bb-storage.inceptio.tech:8980 \
          --remote_instance_name=cybertron \
          //simulation/launch:simulation_main --config=sim
        bazel build \
          --define tensorrt=default \
          --remote_cache=grpc://bb-storage.inceptio.tech:8980 \
          --remote_instance_name=cybertron \
          //interfaces/tools/gems/pb_converter:all

        cd ..
        tar --exclude='cybertron/bazel-cybertron' \
          --exclude='cybertron/bazel-out' \
          -h -cf cybertron-sim.tar cybertron
        ls -alh cybertron-sim.tar
        cp cybertron-sim.tar /data
      volumeMounts:
      - mountPath: /root/.jfrog/jfrog-cli.conf
        name: credentials-volume
        subPath: jfrog-cli.conf
      - mountPath: /data
        name: artifacts-volume-claim
      - mountPath: /root/.ssh/id_rsa
        name: gitlab-ssh-key-volume
        subPath: id_rsa
  - arguments: {}
    inputs: {}
    metadata: {}
    name: build-cybertron-sim-p1
    outputs: {}
    retryStrategy:
      limit: 2
    script:
      command:
      - bash
      env:
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/sde_cybertron_release:2020_07_03_c12745f7
      name: ""
      resources: {}
      source: |
        set -ex

        ssh-keyscan gitlab.rd.inceptioglobal.ai >> ~/.ssh/known_hosts
        chmod 644 ~/.ssh/known_hosts

        export HTTP_PROXY=http://gfw-proxy.inceptio.tech:8118
        export HTTPS_PROXY=http://gfw-proxy.inceptio.tech:8118
        export NO_PROXY=.cn,.aliyun.com,.aliyuncs.com,.chariot,.artifactory

        mkdir -p /root/cybertron && cd /root/cybertron
        git init
        git remote add origin https://gitlab:$GITLAB_TOKEN@gitlab.rd.inceptioglobal.ai/autonomy/cybertron.git
        git fetch --depth=1 origin $CYBERTRON_COMMIT_ID
        git reset --hard FETCH_HEAD

        # FXIME: install empy when buiding image
        pip3 install empy

        bazel build \
          --define tensorflow_target=p1 \
          --remote_cache=grpc://bb-storage.inceptio.tech:8980 \
          --remote_instance_name=cybertron \
          //simulation/launch:simulation_main --config=sim
        bazel build \
          --define tensorflow_target=p1 \
          --remote_cache=grpc://bb-storage.inceptio.tech:8980 \
          --remote_instance_name=cybertron \
          //interfaces/tools/gems/pb_converter:all

        cd ..
        tar --exclude='cybertron/bazel-cybertron' \
          --exclude='cybertron/bazel-out' \
          -h -cf cybertron-sim-p1.tar cybertron
        ls -alh cybertron-sim-p1.tar
        cp cybertron-sim-p1.tar /data
      volumeMounts:
      - mountPath: /root/.jfrog/jfrog-cli.conf
        name: credentials-volume
        subPath: jfrog-cli.conf
      - mountPath: /data
        name: artifacts-volume-claim
      - mountPath: /root/.ssh/id_rsa
        name: gitlab-ssh-key-volume
        subPath: id_rsa
  - arguments: {}
    inputs: {}
    metadata: {}
    name: fetch-regression-repository
    outputs: {}
    retryStrategy:
      limit: 2
      retryPolicy: Always
    script:
      command:
      - bash
      env:
      - name: REGRESSION_COMMIT_ID
        value: '{{workflow.parameters.regression_commit_id}}'
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/sde_cybertron_release:2020_07_03_c12745f7
      name: ""
      resources: {}
      source: |
        set -ex

        mkdir -p /data/regression && cd /data/regression
        git init
        git remote add origin https://gitlab:$GITLAB_TOKEN@gitlab.rd.inceptioglobal.ai/autonomy/regression.git
        echo "$REGRESSION_COMMIT_ID"
        git fetch --depth=1 origin $REGRESSION_COMMIT_ID
        git reset --hard FETCH_HEAD
      volumeMounts:
      - mountPath: /data
        name: artifacts-volume-claim
  - arguments: {}
    inputs: {}
    metadata: {}
    name: fetch-test-group
    outputs: {}
    script:
      command:
      - bash
      env:
      - name: TEST_CASE_GROUP
        value: '{{workflow.parameters.test_case_group}}'
      - name: REGRESSION_PATH
        value: /data/regression
      image: everpeace/curl-jq
      name: ""
      resources: {}
      source: |
        set -e

        trim_prefix_suffix_slash(){ sed -e 's/^\///' -e 's/\/$//'; }
        cd $REGRESSION_PATH
        mkdir -p /data/test_cases_result/
        ls -F -d ${TEST_CASE_GROUP}/* \
          | grep '/$' | trim_prefix_suffix_slash > /data/test_cases_result/tmp_testcases
        echo '[]' > /data/test_cases_result/test_case_result.json
        for line in `cat /data/test_cases_result/tmp_testcases`
        do
            if cat $line/gems_task_v2.pb.txt | grep "disable" &> /dev/null; then
                continue
            fi
            gpu_requirement=$(cat $line/gems_task_v2.pb.txt | grep num_gpu_required | sed -e 's/ //' -e 's/num_gpu_required://')
            if [ -z "$gpu_requirement" ]; then
              gpu_requirement="0"
            fi
            truck_sim="$(cat $line/gems_task_v2.pb.txt | grep 'use_trucksim: true' || true )"
            if [ -z "$truck_sim" ]; then
              truck_sim="false"
            else
              truck_sim="true"
            fi
            echo "[{ \"gpu_requirement\": \"$gpu_requirement\", \"test_case_name\":\"$line\", \"truck_sim\":\"$truck_sim\" }]" \
              | jq ".+= $(cat /data/test_cases_result/test_case_result.json)" > /data/test_cases_result/test_case_result.json
        done
        cat /data/test_cases_result/test_case_result.json | sed -e 's/ //' -e 's/\n//'
      volumeMounts:
      - mountPath: /data
        name: artifacts-volume-claim
  - arguments: {}
    inputs:
      parameters:
      - name: test_case_name
      - name: gpu_requirement
      - name: truck_sim
    metadata: {}
    name: regression
    outputs: {}
    steps:
    - - arguments:
          parameters:
          - name: test_case_name
            value: '{{inputs.parameters.test_case_name}}'
        name: regenerate-step
        template: regenerate
        when: '{{inputs.parameters.gpu_requirement}} == 0 && {{inputs.parameters.truck_sim}}
          != true'
      - arguments:
          parameters:
          - name: test_case_name
            value: '{{inputs.parameters.test_case_name}}'
        name: regenerate-trucksim-step
        template: regenerate-trucksim
        when: '{{inputs.parameters.gpu_requirement}} == 0 && {{inputs.parameters.truck_sim}}
          == true'
      - arguments:
          parameters:
          - name: test_case_name
            value: '{{inputs.parameters.test_case_name}}'
        name: regenerate-gpu-step
        template: regenerate-gpu
        when: '{{inputs.parameters.gpu_requirement}} != 0 && {{inputs.parameters.truck_sim}}
          != true'
      - arguments:
          parameters:
          - name: test_case_name
            value: '{{inputs.parameters.test_case_name}}'
        name: regenerate-trucksim-gpu-step
        template: regenerate-trucksim-gpu
        when: '{{inputs.parameters.gpu_requirement}} != 0 && {{inputs.parameters.truck_sim}}
          == true'
    - - arguments:
          parameters:
          - name: test_case_name
            value: '{{inputs.parameters.test_case_name}}'
        name: evaluate-step
        template: evaluate
  - arguments: {}
    inputs:
      parameters:
      - name: test_case_name
    metadata:
      annotations:
        k8s.aliyun.com/eci-image-cache: "true"
        k8s.aliyun.com/eci-use-specs: ecs.ic5.large
    name: regenerate
    nodeSelector:
      type: virtual-kubelet
    outputs: {}
    retryStrategy:
      limit: 2
      retryPolicy: Always
    script:
      command:
      - bash
      env:
      - name: TEST_CASE_NAME
        value: '{{inputs.parameters.test_case_name}}'
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: REGENERATED_BUCKET
        valueFrom:
          configMapKeyRef:
            key: moment_regenerated_oss_bucket
            name: cluster-configs
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      - name: CAS_TOKEN
        valueFrom:
          secretKeyRef:
            key: token
            name: cas-service-token
      - name: ENV
        valueFrom:
          configMapKeyRef:
            key: env
            name: cluster-configs
      - name: CAS_WEB_HOST
        valueFrom:
          configMapKeyRef:
            key: cas_web_host
            name: cluster-configs
      - name: CHT_GATEWAY_HOST
        valueFrom:
          configMapKeyRef:
            key: cht_gateway_host
            name: cluster-configs
      - name: RT_CN_WEB_URL
        valueFrom:
          configMapKeyRef:
            key: rt_cn_web_url
            name: cluster-configs
      - name: RT_US_WEB_URL
        valueFrom:
          configMapKeyRef:
            key: rt_us_web_url
            name: cluster-configs
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/infra_regenerate-gems:v20210105-5044b06
      name: ""
      resources: {}
      source: |
        set -ex

        T0=$(date +%s)
        cat_gems_task_pbtxt(){ cat /data/regression/$TEST_CASE_NAME/gems_task_v2.pb.txt; }
        trim_prefix_suffix_quote(){ sed -e 's/^"//' -e 's/"$//'; }
        LAUNCH_NAME=$(cat_gems_task_pbtxt | grep launch_config_rel_path | awk '{print$2}' | trim_prefix_suffix_quote)
        VEHICLE_ID=$(cat_gems_task_pbtxt | grep vehicle_id | awk '{print$2}' | trim_prefix_suffix_quote)
        MAP=$(cat_gems_task_pbtxt | grep map_version | awk '{print$2}' | trim_prefix_suffix_quote)
        TIMEOUT=$(cat_gems_task_pbtxt | grep timeout_sec | awk '{print$2}')
        MOMENT_ID=$(cat_gems_task_pbtxt | grep moment_id | awk '{print$2}' | trim_prefix_suffix_quote)
        SIM_MODE=$(cat_gems_task_pbtxt | grep "sim_mode:" | awk '{print$2}')
        REC_REL_DIR=$(cat_gems_task_pbtxt | grep "rec_rel_dir:" | awk '{print$2}' | trim_prefix_suffix_quote)

        if [[ $SIM_MODE == "IDEAL" || $SIM_MODE == "WORLD" ]]; then
          CYBERTRON_TAR=cybertron-sim.tar
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          CYBERTRON_TAR=cybertron-pkg.tar
        else
          echo "Unsupported sim_mode $SIM_MODE"
          exit 1
        fi

        T1=$(date +%s)
        echo "Parse parameters took $((T1-T0)) seconds"
        cp /data/$CYBERTRON_TAR /root/cybertron.tar && cd /root && tar -xf cybertron.tar && rm cybertron.tar && cd cybertron

        T2=$(date +%s)
        echo "Copy tar took $((T2-T1)) seconds"

        if [[ ! -z "$MOMENT_ID" ]]; then
          dataset client \
            dl moments/$MOMENT_ID \
            --token=$CAS_TOKEN \
            --target=dataset-server.chariot:50050 \
            --dataset-process-target=dataset-server.chariot:50051 \
            --target-path=/root/moments/$REC_REL_DIR \
            --idl-version={{workflow.outputs.parameters.global_idl_version}}
        fi

        T3=$(date +%s)
        echo "Downloading data took $((T3-T2)) seconds"

        # Detect in kube pods
        sed -i "s/docker/kubepods/#g" /root/cybertron/irs/process_manager/setup/ubuntu_x86_64/ipm_u_setup.sh
        # Set chariot env
        #export CHARIOT_ENV=$(echo $ENV | tr 'a-z' 'A-Z')
        export CHARIOT_SITE=cn
        cat <<EOF >> chariot_config.json
        {
          "CAS_URL": "$CAS_WEB_HOST",
          "CHARIOT_SERVER_URL": "$CHT_GATEWAY_HOST",
          "artifacts": {
            "us": {
              "endpoint": "$RT_US_WEB_URL",
              "repo_type": "local"
            },
            "cn": {
              "endpoint": "$RT_CN_WEB_URL",
              "repo_type": "remote"
            }
          }
        }
        EOF
        export CHARIOT_ENV_FILE=chariot_config.json

        chariot -t $CAS_TOKEN -o json config --set disable_auto_update=true
        if [[ $SIM_MODE == "IDEAL" || $SIM_MODE == "WORLD" ]]; then
          if [[ $SIM_MODE == "IDEAL" ]]; then
              # gems-sim use the rec pattern
              mv /root/moments/$TEST_CASE_NAME/${MOMENT_ID:0:2}/$MOMENT_ID/* /root/moments/$TEST_CASE_NAME
          fi

          # gems-sim not untar the packages
          yq delete -i bundle.yaml packages.cybertron.build.tars
          yq delete -i bundle.yaml packages.cybertron.deployScript
          cat bundle.yaml
          chariot -t $CAS_TOKEN deploy -f bundle.yaml
          T4=$(date +%s)
          echo "Chariot deploy took $((T4-T3)) seconds"

          if [ -f /data/regression/$TEST_CASE_NAME/input_command.pb.txt ]; then
              INPUT_COMMAND_LIST_PARAM="-l /data/regression/$TEST_CASE_NAME/input_command.pb.txt"
          fi

          echo "MOMENT_ID: $MOMENT_ID"
          LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/compat:/usr/lib/x86_64-linux-gnu/:$LD_LIBRARY_PATH
          cd /root/cybertron/
          timeout $TIMEOUT bazel-bin/simulation/launch/simulation_main --config_path /data/regression/$TEST_CASE_NAME/gems_task_v2.pb.txt \
            --no_ui \
            --data_path /root/moments/ \
            --output_path /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID
          cd -
          T5=$(date +%s)
          echo "Gems sim took $((T5-T4)) seconds"
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          pip3 install irs/launch/launch irs/launch/launch_ros irs/launch/launch_yaml irs/launch/test_launch_ros irs/launch/osrf_pycommon
          chariot -t $CAS_TOKEN replay \
            --no-check \
            --launch \
            --name=$LAUNCH_NAME \
            --file=bundle.yaml \
            --map=$MAP \
            --vehicle-id=$VEHICLE_ID \
            --timeout=$TIMEOUT \
            --replay-rec-path=/root/moments/$REC_REL_DIR/${MOMENT_ID:0:2}/$MOMENT_ID \
            --output=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID
          T5=$(date +%s)
          echo "time sync replayer took $((T5-T3)) seconds"
        fi

        gems-rec-convert \
          --exec_path=/root/cybertron \
          --input_path=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/rec \
          --output_path=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/pb

        T6=$(date +%s)
        echo "Converter took $((T6-T5)) seconds"

        mkdir -p /data/$TEST_CASE_NAME/regenerated/pb
        cp -a /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/pb /data/$TEST_CASE_NAME/regenerated
        if [[ $SIM_MODE == "IDEAL" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/${MOMENT_ID:0:2}/$MOMENT_ID/$SIM_MODE/$TEST_CASE_NAME/$CYBERTRON_COMMIT_ID"
        elif [[ $SIM_MODE == "WORLD" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/$SIM_MODE/$TEST_CASE_NAME/$CYBERTRON_COMMIT_ID"
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/${MOMENT_ID:0:2}/$MOMENT_ID/$LAUNCH_NAME/$CYBERTRON_COMMIT_ID"
        fi
        # calculate disk storage
        du -hd 1 /root
        ls -l /root/cybertron || true
        du -hd 1 /root/cybertron
        ossutil cp --config-file=/root/.ossutilconfig -r -f \
          /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID \
          $OSS_DEST_PATH

        T7=$(date +%s)
        echo "Oss copy took $((T6-T5)) seconds"
      volumeMounts:
      - mountPath: /root/.jfrog/jfrog-cli.conf
        name: credentials-volume
        subPath: jfrog-cli.conf
      - mountPath: /root/.ossutilconfig
        name: credentials-volume
        subPath: .ossutilconfig
      - mountPath: /data
        name: artifacts-volume-claim
    tolerations:
    - key: virtual-kubelet.io/provider
      operator: Exists
  - arguments: {}
    inputs:
      parameters:
      - name: test_case_name
    metadata:
      annotations:
        k8s.aliyun.com/eci-image-cache: "true"
        k8s.aliyun.com/eci-use-specs: ecs.ic5.large
    name: regenerate-trucksim
    nodeSelector:
      type: virtual-kubelet
    outputs: {}
    retryStrategy:
      limit: 2
      retryPolicy: Always
    script:
      command:
      - bash
      env:
      - name: TEST_CASE_NAME
        value: '{{inputs.parameters.test_case_name}}'
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: REGENERATED_BUCKET
        valueFrom:
          configMapKeyRef:
            key: moment_regenerated_oss_bucket
            name: cluster-configs
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      - name: CAS_TOKEN
        valueFrom:
          secretKeyRef:
            key: token
            name: cas-service-token
      - name: ENV
        valueFrom:
          configMapKeyRef:
            key: env
            name: cluster-configs
      - name: CAS_WEB_HOST
        valueFrom:
          configMapKeyRef:
            key: cas_web_host
            name: cluster-configs
      - name: CHT_GATEWAY_HOST
        valueFrom:
          configMapKeyRef:
            key: cht_gateway_host
            name: cluster-configs
      - name: RT_CN_WEB_URL
        valueFrom:
          configMapKeyRef:
            key: rt_cn_web_url
            name: cluster-configs
      - name: RT_US_WEB_URL
        valueFrom:
          configMapKeyRef:
            key: rt_us_web_url
            name: cluster-configs
      - name: TS_LICENSE_SERVER
        valueFrom:
          configMapKeyRef:
            key: ts_license_server
            name: cluster-configs
      - name: TS_LICENSE_TYPE
        valueFrom:
          configMapKeyRef:
            key: ts_license_type
            name: cluster-configs
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/infra_regenerate-gems:v20210105-5044b06
      name: ""
      resources: {}
      source: |
        set -ex

        T0=$(date +%s)
        cat_gems_task_pbtxt(){ cat /data/regression/$TEST_CASE_NAME/gems_task_v2.pb.txt; }
        trim_prefix_suffix_quote(){ sed -e 's/^"//' -e 's/"$//'; }
        LAUNCH_NAME=$(cat_gems_task_pbtxt | grep launch_config_rel_path | awk '{print$2}' | trim_prefix_suffix_quote)
        VEHICLE_ID=$(cat_gems_task_pbtxt | grep vehicle_id | awk '{print$2}' | trim_prefix_suffix_quote)
        MAP=$(cat_gems_task_pbtxt | grep map_version | awk '{print$2}' | trim_prefix_suffix_quote)
        TIMEOUT=$(cat_gems_task_pbtxt | grep timeout_sec | awk '{print$2}')
        MOMENT_ID=$(cat_gems_task_pbtxt | grep moment_id | awk '{print$2}' | trim_prefix_suffix_quote)
        SIM_MODE=$(cat_gems_task_pbtxt | grep "sim_mode:" | awk '{print$2}')
        REC_REL_DIR=$(cat_gems_task_pbtxt | grep "rec_rel_dir:" | awk '{print$2}' | trim_prefix_suffix_quote)

        T1=$(date +%s)
        echo "Parse parameters took $((T1-T0)) seconds"

        if [[ $SIM_MODE == "IDEAL" || $SIM_MODE == "WORLD" ]]; then
          CYBERTRON_TAR=cybertron-sim-p1.tar
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          CYBERTRON_TAR=cybertron-pkg.tar
        else
          echo "Unsupported sim_mode $SIM_MODE"
          exit 1
        fi
        cp /data/$CYBERTRON_TAR /root/cybertron.tar && cd /root && tar -xf cybertron.tar && rm cybertron.tar && cd cybertron

        T2=$(date +%s)
        echo "copy tar took $((T2-T1)) seconds"

        if [[ ! -z "$MOMENT_ID" ]]; then
          dataset client \
            dl moments/$MOMENT_ID \
            --token=$CAS_TOKEN \
            --target=dns:///dataset-server-headless.chariot:50050 \
            --dataset-process-target=dns:///dataset-server-headless.chariot:50051 \
            --target-path=/root/moments/$REC_REL_DIR \
            --idl-version={{workflow.outputs.parameters.global_idl_version}}
        fi

        T3=$(date +%s)
        echo "Download data took $((T3-T2)) seconds"

        # Detect in kube pods
        sed -i "s/docker/kubepods/#g" /root/cybertron/irs/process_manager/setup/ubuntu_x86_64/ipm_u_setup.sh
        # Set chariot env
        #export CHARIOT_ENV=$(echo $ENV | tr 'a-z' 'A-Z')
        export CHARIOT_SITE=cn
        cat <<EOF >> chariot_config.json
        {
          "CAS_URL": "$CAS_WEB_HOST",
          "CHARIOT_SERVER_URL": "$CHT_GATEWAY_HOST",
          "artifacts": {
            "us": {
              "endpoint": "$RT_US_WEB_URL",
              "repo_type": "local"
            },
            "cn": {
              "endpoint": "$RT_CN_WEB_URL",
              "repo_type": "remote"
            }
          }
        }
        EOF
        export CHARIOT_ENV_FILE=chariot_config.json


        chariot -t $CAS_TOKEN -o json config --set disable_auto_update=true
        if [[ $SIM_MODE == "IDEAL" || $SIM_MODE == "WORLD" ]]; then
          if [[ $SIM_MODE == "IDEAL" ]]; then
              # gems-sim use the rec pattern
              mv /root/moments/$REC_REL_DIR/${MOMENT_ID:0:2}/$MOMENT_ID/* /root/moments/$REC_REL_DIR
          fi

          # gems-sim not untar the packages
          yq delete -i bundle.yaml packages.cybertron.build.tars
          yq delete -i bundle.yaml packages.cybertron.deployScript
          cat bundle.yaml
          chariot -t $CAS_TOKEN deploy -f bundle.yaml

          T4=$(date +%s)
          echo "chariot deploy took $((T4-T3)) seconds"

          if [ -f /data/regression/$TEST_CASE_NAME/input_command.pb.txt ]; then
              INPUT_COMMAND_LIST_PARAM="-l /data/regression/$TEST_CASE_NAME/input_command.pb.txt"
          fi

          #FIXME: force using the 395 calibration for trucksim, for long term will using the gems_task setting or maintain a realtion map between vehicle model and platform(vehicle data/calibration)
          ls /root/cybertron/config/calibration/CN_395
          export IRS_CALIBRATION_PATH=/root/cybertron/config/calibration/CN_395

          echo "MOMENT_ID: $MOMENT_ID"
          LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/compat:/usr/lib/x86_64-linux-gnu/:$LD_LIBRARY_PATH
          cd /root/cybertron/
          # Run truck sim
          # trucksim found this this .so in absolute path
          mkdir -p bazel-cybertron/../../external/trucksim_api/
          cp /opt/trucksim_2020.0/lib64/libtrucksim.so.2020.0 bazel-cybertron/../../external/trucksim_api/libtrucksim.so.2020.0
          T5=$(date +%s)
          echo "Prepare trucksim took $((T5-T4)) seconds"

          timeout $TIMEOUT bazel-bin/simulation/launch/simulation_main --config_path /data/regression/$TEST_CASE_NAME/gems_task_v2.pb.txt \
            --no_ui \
            --data_path /root/moments/ \
            --output_path /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID
          cd -
          T6=$(date +%s)
          echo "Gems-sim took $((T6-T5)) seconds"
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          pip3 install irs/launch/launch irs/launch/launch_ros irs/launch/launch_yaml irs/launch/test_launch_ros irs/launch/osrf_pycommon
          chariot -t $CAS_TOKEN replay \
            --no-check \
            --launch \
            --name=$LAUNCH_NAME \
            --file=bundle.yaml \
            --map=$MAP \
            --vehicle-id=$VEHICLE_ID \
            --timeout=$TIMEOUT \
            --replay-rec-path=/root/moments/$REC_REL_DIR/${MOMENT_ID:0:2}/$MOMENT_ID \
            --output=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID
          T6=$(date +%s)
          echo "chariot replay took $((T6-T4)) seconds"
        fi

        cat /tmp/sim_log.txt || true

        gems-rec-convert \
          --exec_path=/root/cybertron \
          --input_path=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/rec \
          --output_path=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/pb

        T7=$(date +%s)
        echo "converter took $((T7-T6)) seconds"

        mkdir -p /data/$TEST_CASE_NAME/regenerated/pb
        cp -a /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/pb /data/$TEST_CASE_NAME/regenerated
        ls -l /data/$TEST_CASE_NAME/regenerated
        if [[ $SIM_MODE == "IDEAL" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/${MOMENT_ID:0:2}/$MOMENT_ID/$SIM_MODE/$TEST_CASE_NAME/$CYBERTRON_COMMIT_ID"
        elif [[ $SIM_MODE == "WORLD" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/$SIM_MODE/$TEST_CASE_NAME/$CYBERTRON_COMMIT_ID"
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/${MOMENT_ID:0:2}/$MOMENT_ID/$LAUNCH_NAME/$CYBERTRON_COMMIT_ID"
        fi
        # calculate disk storage
        du -hd 1 /root
        ls -l /root/cybertron || true
        du -hd 1 /root/cybertron
        ossutil cp --config-file=/root/.ossutilconfig -r -f \
          /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID \
          $OSS_DEST_PATH

        T8=$(date +%s)
        echo "oss copy took $((T8-T7)) seconds"
      volumeMounts:
      - mountPath: /root/.jfrog/jfrog-cli.conf
        name: credentials-volume
        subPath: jfrog-cli.conf
      - mountPath: /root/.ossutilconfig
        name: credentials-volume
        subPath: .ossutilconfig
      - mountPath: /data
        name: artifacts-volume-claim
    synchronization:
      semaphore:
        configMapKeyRef:
          key: trucksim_regenarate_template
          name: gems-sync
    tolerations:
    - key: virtual-kubelet.io/provider
      operator: Exists
  - arguments: {}
    inputs:
      parameters:
      - name: test_case_name
    metadata:
      annotations:
        k8s.aliyun.com/eci-image-cache: "true"
        k8s.aliyun.com/eci-use-specs: ecs.gn6i-c8g1.2xlarge
    name: regenerate-gpu
    nodeSelector:
      type: virtual-kubelet
    outputs: {}
    retryStrategy:
      backoff:
        duration: 1m
        factor: 1
        maxDuration: 30m
      limit: 2
      retryPolicy: Always
    script:
      command:
      - bash
      env:
      - name: TEST_CASE_NAME
        value: '{{inputs.parameters.test_case_name}}'
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: REGENERATED_BUCKET
        valueFrom:
          configMapKeyRef:
            key: moment_regenerated_oss_bucket
            name: cluster-configs
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      - name: CAS_TOKEN
        valueFrom:
          secretKeyRef:
            key: token
            name: cas-service-token
      - name: ENV
        valueFrom:
          configMapKeyRef:
            key: env
            name: cluster-configs
      - name: CAS_WEB_HOST
        valueFrom:
          configMapKeyRef:
            key: cas_web_host
            name: cluster-configs
      - name: CHT_GATEWAY_HOST
        valueFrom:
          configMapKeyRef:
            key: cht_gateway_host
            name: cluster-configs
      - name: RT_CN_WEB_URL
        valueFrom:
          configMapKeyRef:
            key: rt_cn_web_url
            name: cluster-configs
      - name: RT_US_WEB_URL
        valueFrom:
          configMapKeyRef:
            key: rt_us_web_url
            name: cluster-configs
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/infra_regenerate-gems:v20210105-5044b06
      name: ""
      resources:
        limits:
          nvidia.com/gpu: "1"
      source: |
        set -ex

        cat_gems_task_pbtxt(){ cat /data/regression/$TEST_CASE_NAME/gems_task_v2.pb.txt; }
        trim_prefix_suffix_quote(){ sed -e 's/^"//' -e 's/"$//'; }
        LAUNCH_NAME=$(cat_gems_task_pbtxt | grep launch_config_rel_path | awk '{print$2}' | trim_prefix_suffix_quote)
        VEHICLE_ID=$(cat_gems_task_pbtxt | grep vehicle_id | awk '{print$2}' | trim_prefix_suffix_quote)
        MAP=$(cat_gems_task_pbtxt | grep map_version | awk '{print$2}' | trim_prefix_suffix_quote)
        TIMEOUT=$(cat_gems_task_pbtxt | grep timeout_sec | awk '{print$2}')
        MOMENT_ID=$(cat_gems_task_pbtxt | grep moment_id | awk '{print$2}' | trim_prefix_suffix_quote)
        SIM_MODE=$(cat_gems_task_pbtxt | grep sim_mode | awk '{print$2}')
        REC_REL_DIR=$(cat_gems_task_pbtxt | grep "rec_rel_dir:" | awk '{print$2}' | trim_prefix_suffix_quote)

        if [[ $SIM_MODE == "IDEAL" || $SIM_MODE == "WORLD" ]]; then
          CYBERTRON_TAR=cybertron-sim.tar
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          CYBERTRON_TAR=cybertron-pkg.tar
        else
          echo "Unsupported sim_mode $SIM_MODE"
          exit 1
        fi
        cp /data/$CYBERTRON_TAR /root/cybertron.tar && cd /root && tar -xf cybertron.tar && rm cybertron.tar && cd cybertron

        if [[ ! -z "$MOMENT_ID" ]]; then
          dataset client \
            dl moments/$MOMENT_ID \
            --token=$CAS_TOKEN \
            --target=dns:///dataset-server-headless.chariot:50050 \
            --dataset-process-target=dns:///dataset-server-headless.chariot:50051 \
            --target-path=/root/moments/$REC_REL_DIR \
            --idl-version={{workflow.outputs.parameters.global_idl_version}}
        fi

        # Detect in kube pods
        sed -i "s/docker/kubepods/#g" /root/cybertron/irs/process_manager/setup/ubuntu_x86_64/ipm_u_setup.sh
        # Set chariot env
        # export CHARIOT_ENV=$(echo $ENV | tr 'a-z' 'A-Z')
        export CHARIOT_SITE=cn
        cat <<EOF >> chariot_config.json
        {
          "CAS_URL": "$CAS_WEB_HOST",
          "CHARIOT_SERVER_URL": "$CHT_GATEWAY_HOST",
          "artifacts": {
            "us": {
              "endpoint": "$RT_US_WEB_URL",
              "repo_type": "local"
            },
            "cn": {
              "endpoint": "$RT_CN_WEB_URL",
              "repo_type": "remote"
            }
          }
        }
        EOF
        export CHARIOT_ENV_FILE=chariot_config.json
        chariot -o json config --set disable_auto_update=true
        if [[ $SIM_MODE == "IDEAL" || $SIM_MODE == "WORLD" ]]; then
          if [[ $SIM_MODE == "IDEAL" ]]; then
              # gems-sim use the rec pattern
              mv /root/moments/$REC_REL_DIR/${MOMENT_ID:0:2}/$MOMENT_ID/* /root/moments/$REC_REL_DIR
          fi

          # gems-sim not untar the packages
          yq delete -i bundle.yaml packages.cybertron.build.tars
          yq delete -i bundle.yaml packages.cybertron.deployScript
          cat bundle.yaml
          chariot -t $CAS_TOKEN deploy -f bundle.yaml

          if [ -f /data/regression/$TEST_CASE_NAME/input_command.pb.txt ]; then
              INPUT_COMMAND_LIST_PARAM="-l /data/regression/$TEST_CASE_NAME/input_command.pb.txt"
          fi

          cd /root/cybertron/
          timeout $TIMEOUT bazel-bin/simulation/launch/simulation_main --config_path /data/regression/$TEST_CASE_NAME/gems_task_v2.pb.txt \
            --no_ui \
            --data_path /root/moments/ \
            --output_path /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID
          cd -
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          pip3 install irs/launch/launch irs/launch/launch_ros irs/launch/launch_yaml irs/launch/test_launch_ros irs/launch/osrf_pycommon
          chariot -t $CAS_TOKEN replay \
            --no-check \
            --launch \
            --name=$LAUNCH_NAME \
            --file=bundle.yaml \
            --map=$MAP \
            --vehicle-id=$VEHICLE_ID \
            --timeout=$TIMEOUT \
            --replay-rec-path=/root/moments/$REC_REL_DIR/${MOMENT_ID:0:2}/$MOMENT_ID \
            --output=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID
        fi

        gems-rec-convert \
          --exec_path=/root/cybertron \
          --input_path=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/rec \
          --output_path=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/pb

        mkdir -p /data/$TEST_CASE_NAME/regenerated/pb
        cp -a /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/pb /data/$TEST_CASE_NAME/regenerated
        if [[ $SIM_MODE == "IDEAL" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/${MOMENT_ID:0:2}/$MOMENT_ID/$SIM_MODE/$TEST_CASE_NAME/$CYBERTRON_COMMIT_ID"
        elif [[ $SIM_MODE == "WORLD" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/$SIM_MODE/$TEST_CASE_NAME/$CYBERTRON_COMMIT_ID"
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/${MOMENT_ID:0:2}/$MOMENT_ID/$LAUNCH_NAME/$CYBERTRON_COMMIT_ID"
        fi
        # calculate disk storage
        du -hd 1 /root
        ls -l /root/cybertron || true
        du -hd 1 /root/cybertron
        ossutil cp --config-file=/root/.ossutilconfig -r -f \
          /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID \
          $OSS_DEST_PATH
      volumeMounts:
      - mountPath: /root/.jfrog/jfrog-cli.conf
        name: credentials-volume
        subPath: jfrog-cli.conf
      - mountPath: /root/.ossutilconfig
        name: credentials-volume
        subPath: .ossutilconfig
      - mountPath: /data
        name: artifacts-volume-claim
    tolerations:
    - key: virtual-kubelet.io/provider
      operator: Exists
  - affinity:
      nodeAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - preference:
            matchExpressions:
            - key: gpu_node
              operator: In
              values:
              - "true"
          weight: 100
    arguments: {}
    inputs:
      parameters:
      - name: test_case_name
    metadata:
      annotations:
        k8s.aliyun.com/eci-image-cache: "true"
        k8s.aliyun.com/eci-use-specs: ecs.gn6i-c8g1.2xlarge
    name: regenerate-trucksim-gpu
    outputs: {}
    retryStrategy:
      backoff:
        duration: 1m
        factor: 1
        maxDuration: 30m
      limit: 2
      retryPolicy: Always
    script:
      command:
      - bash
      env:
      - name: TEST_CASE_NAME
        value: '{{inputs.parameters.test_case_name}}'
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: REGENERATED_BUCKET
        valueFrom:
          configMapKeyRef:
            key: moment_regenerated_oss_bucket
            name: cluster-configs
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      - name: CAS_TOKEN
        valueFrom:
          secretKeyRef:
            key: token
            name: cas-service-token
      - name: ENV
        valueFrom:
          configMapKeyRef:
            key: env
            name: cluster-configs
      - name: CAS_WEB_HOST
        valueFrom:
          configMapKeyRef:
            key: cas_web_host
            name: cluster-configs
      - name: CHT_GATEWAY_HOST
        valueFrom:
          configMapKeyRef:
            key: cht_gateway_host
            name: cluster-configs
      - name: RT_CN_WEB_URL
        valueFrom:
          configMapKeyRef:
            key: rt_cn_web_url
            name: cluster-configs
      - name: RT_US_WEB_URL
        valueFrom:
          configMapKeyRef:
            key: rt_us_web_url
            name: cluster-configs
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/infra_regenerate-gems:v20210105-5044b06
      name: ""
      resources:
        limits:
          nvidia.com/gpu: "1"
      source: |
        set -ex

        cat_gems_task_pbtxt(){ cat /data/regression/$TEST_CASE_NAME/gems_task_v2.pb.txt; }
        trim_prefix_suffix_quote(){ sed -e 's/^"//' -e 's/"$//'; }
        LAUNCH_NAME=$(cat_gems_task_pbtxt | grep launch_config_rel_path | awk '{print$2}' | trim_prefix_suffix_quote)
        VEHICLE_ID=$(cat_gems_task_pbtxt | grep vehicle_id | awk '{print$2}' | trim_prefix_suffix_quote)
        MAP=$(cat_gems_task_pbtxt | grep map_version | awk '{print$2}' | trim_prefix_suffix_quote)
        TIMEOUT=$(cat_gems_task_pbtxt | grep timeout_sec | awk '{print$2}')
        MOMENT_ID=$(cat_gems_task_pbtxt | grep moment_id | awk '{print$2}' | trim_prefix_suffix_quote)
        SIM_MODE=$(cat_gems_task_pbtxt | grep sim_mode | awk '{print$2}')
        REC_REL_DIR=$(cat_gems_task_pbtxt | grep "rec_rel_dir:" | awk '{print$2}' | trim_prefix_suffix_quote)

        if [[ $SIM_MODE == "IDEAL" || $SIM_MODE == "WORLD" ]]; then
          CYBERTRON_TAR=cybertron-sim.tar
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          CYBERTRON_TAR=cybertron-pkg.tar
        else
          echo "Unsupported sim_mode $SIM_MODE"
          exit 1
        fi
        cp /data/$CYBERTRON_TAR /root/cybertron.tar && cd /root && tar -xf cybertron.tar && rm cybertron.tar && cd cybertron

        if [[ ! -z "$MOMENT_ID" ]]; then
          dataset client \
            dl moments/$MOMENT_ID \
            --token=$CAS_TOKEN \
            --target=dns:///dataset-server-headless.chariot:50050 \
            --dataset-process-target=dns:///dataset-server-headless.chariot:50051 \
            --target-path=/root/moments/$REC_REL_DIR \
            --idl-version={{workflow.outputs.parameters.global_idl_version}}
        fi

        # Detect in kube pods
        sed -i "s/docker/kubepods/#g" /root/cybertron/irs/process_manager/setup/ubuntu_x86_64/ipm_u_setup.sh
        # Set chariot env
        # export CHARIOT_ENV=$(echo $ENV | tr 'a-z' 'A-Z')
        export CHARIOT_SITE=cn
        cat <<EOF >> chariot_config.json
        {
          "CAS_URL": "$CAS_WEB_HOST",
          "CHARIOT_SERVER_URL": "$CHT_GATEWAY_HOST",
          "artifacts": {
            "us": {
              "endpoint": "$RT_US_WEB_URL",
              "repo_type": "local"
            },
            "cn": {
              "endpoint": "$RT_CN_WEB_URL",
              "repo_type": "remote"
            }
          }
        }
        EOF

        export CHARIOT_ENV_FILE=chariot_config.json
        chariot -t $CAS_TOKEN -o json config --set disable_auto_update=true
        if [[ $SIM_MODE == "IDEAL" || $SIM_MODE == "WORLD" ]]; then
          if [[ $SIM_MODE == "IDEAL" ]]; then
              # gems-sim use the rec pattern
              mv /root/moments/$REC_REL_DIR/${MOMENT_ID:0:2}/$MOMENT_ID/* /root/moments/$REC_REL_DIR
          fi

          # gems-sim not untar the packages
          yq delete -i bundle.yaml packages.cybertron.build.tars
          yq delete -i bundle.yaml packages.cybertron.deployScript
          cat bundle.yaml
          chariot -t $CAS_TOKEN deploy -f bundle.yaml

          if [ -f /data/regression/$TEST_CASE_NAME/input_command.pb.txt ]; then
              INPUT_COMMAND_LIST_PARAM="-l /data/regression/$TEST_CASE_NAME/input_command.pb.txt"
          fi

          #FIXME: force using the 395 calibration for trucksim, for long term will using the gems_task setting or maintain a realtion map between vehicle model and platform(vehicle data/calibration)
          ls /root/cybertron/config/calibration/CN_395
          export IRS_CALIBRATION_PATH=/root/cybertron/config/calibration/CN_395

          cd /root/cybertron/
          # Run truck sim
          # trucksim found this this .so in absolute path
          mkdir -p bazel-cybertron/../../external/trucksim_api/
          cp /opt/trucksim_2020.0/lib64/libtrucksim.so.2020.0 bazel-cybertron/../../external/trucksim_api/libtrucksim.so.2020.0

          timeout $TIMEOUT bazel-bin/simulation/launch/simulation_main --config_path /data/regression/$TEST_CASE_NAME/gems_task_v2.pb.txt \
            --no_ui \
            --data_path /root/moments/ \
            --output_path /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID
          cd -
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          pip3 install irs/launch/launch irs/launch/launch_ros irs/launch/launch_yaml irs/launch/test_launch_ros irs/launch/osrf_pycommon
          chariot -t $CAS_TOKEN replay \
            --no-check \
            --launch \
            --name=$LAUNCH_NAME \
            --file=bundle.yaml \
            --map=$MAP \
            --vehicle-id=$VEHICLE_ID \
            --timeout=$TIMEOUT \
            --replay-rec-path=/root/moments/$TEST_CASE_NAME/${MOMENT_ID:0:2}/$MOMENT_ID \
            --output=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID
        fi

        gems-rec-convert \
          --exec_path=/root/cybertron \
          --input_path=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/rec \
          --output_path=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/pb

        mkdir -p /data/$TEST_CASE_NAME/regenerated/pb
        cp -a /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/pb /data/$TEST_CASE_NAME/regenerated
        if [[ $SIM_MODE == "IDEAL" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/${MOMENT_ID:0:2}/$MOMENT_ID/$SIM_MODE/$TEST_CASE_NAME/$CYBERTRON_COMMIT_ID"
        elif [[ $SIM_MODE == "WORLD" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/$SIM_MODE/$TEST_CASE_NAME/$CYBERTRON_COMMIT_ID"
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/${MOMENT_ID:0:2}/$MOMENT_ID/$LAUNCH_NAME/$CYBERTRON_COMMIT_ID"
        fi
        # calculate disk storage
        du -hd 1 /root
        ls -l /root/cybertron || true
        du -hd 1 /root/cybertron
        ossutil cp --config-file=/root/.ossutilconfig -r -f \
          /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID \
          $OSS_DEST_PATH
      volumeMounts:
      - mountPath: /root/.jfrog/jfrog-cli.conf
        name: credentials-volume
        subPath: jfrog-cli.conf
      - mountPath: /root/.ossutilconfig
        name: credentials-volume
        subPath: .ossutilconfig
      - mountPath: /data
        name: artifacts-volume-claim
    synchronization:
      semaphore:
        configMapKeyRef:
          key: trucksim_regenarate_template
          name: gems-sync
    tolerations:
    - effect: NoSchedule
      key: gpu_node
      operator: Equal
      value: "true"
  - arguments: {}
    inputs:
      parameters:
      - name: test_case_name
    metadata: {}
    name: evaluate
    outputs: {}
    retryStrategy:
      limit: 2
      retryPolicy: OnError
    script:
      command:
      - bash
      env:
      - name: REGRESSION_COMMIT_ID
        value: '{{workflow.parameters.regression_commit_id}}'
      - name: TEST_CASE_NAME
        value: '{{inputs.parameters.test_case_name}}'
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: REGENERATED_BUCKET
        valueFrom:
          configMapKeyRef:
            key: moment_regenerated_oss_bucket
            name: cluster-configs
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/infra_regenerate-gems:v20210105-5044b06
      name: ""
      resources: {}
      source: |
        set -ex

        # download ground truth path
        GROUND_TRUTH_OSS_PATH=oss://inceptio-test-case-management-dev/$TEST_CASE_NAME/groud_truth/*
        mkdir -p /data/$TEST_CASE_NAME/evaluate/ground_truth
        SIZE=$(ossutil ls --config-file=/root/.ossutilconfig -s \
          $GROUND_TRUTH_OSS_PATH \
          | tail -n 3 | head -n 1 | awk '{print$4}')
        if [[ $SIZE -ne 0 && $SIZE -ne 1 ]]; then
          GROUND_TRUTH_LOCAL=/data/$TEST_CASE_NAME/evaluate/ground_truth
          ossutil cp --config-file=/root/.ossutilconfig -r -f \
          $GROUND_TRUTH_OSS_PATH \
          $GROUND_TRUTH_LOCAL
        fi

        cat_gems_task_pbtxt(){ cat /data/regression/$TEST_CASE_NAME/gems_task_v2.pb.txt; }
        trim_prefix_suffix_quote(){ sed -e 's/^"//' -e 's/"$//'; }
        MOMENT_ID=$(cat_gems_task_pbtxt | grep moment_id | awk '{print$2}' | trim_prefix_suffix_quote)
        MAP=$(cat_gems_task_pbtxt | grep map_version | awk '{print$2}' | trim_prefix_suffix_quote)
        TOPICS=$(cat_gems_task_pbtxt | grep evaluation_topics | awk '{print$2}' | trim_prefix_suffix_quote )
        SIM_MODE=$(cat_gems_task_pbtxt | grep "sim_mode:" | awk '{print$2}')
        EVALUATE_FILE_NAME=$(echo ${TEST_CASE_NAME} | sed 's/\//_/g')

        ls /data/
        if [[ $SIM_MODE == "IDEAL" ]]; then
            EVALUATE_OUTPUT_PATH=/data/evaluate_result/${MOMENT_ID}.json
            OSS_DEST_PATH="oss://$REGENERATED_BUCKET/evaluate_result/${MOMENT_ID:0:2}/$MOMENT_ID/$SIM_MODE/$TEST_CASE_NAME/$CYBERTRON_COMMIT_ID/${MOMENT_ID}.json"
        elif [[ $SIM_MODE == "WORLD" ]]; then
            EVALUATE_OUTPUT_PATH=/data/evaluate_result/${EVALUATE_FILE_NAME}.json
            OSS_DEST_PATH="oss://$REGENERATED_BUCKET/evaluate_result/$SIM_MODE/$TEST_CASE_NAME/${TEST_CASE_NAME}.json"
        elif [[ $SIM_MODE == "TIMELY" ]]; then
            EVALUATE_OUTPUT_PATH=/data/evaluate_result/${MOMENT_ID}.json
            OSS_DEST_PATH="oss://$REGENERATED_BUCKET/evaluate_result/${MOMENT_ID:0:2}/$MOMENT_ID/$LAUNCH_NAME/$CYBERTRON_COMMIT_ID/${MOMENT_ID}.json"
        fi

        mkdir -p ${EVALUATE_OUTPUT_PATH%/*}

        cd /data/regression/$TEST_CASE_NAME
        evaluate \
          --lep=local_transformation \
          --srp=/data/$TEST_CASE_NAME/regenerated/pb \
          --topics $TOPICS \
          --output=$EVALUATE_OUTPUT_PATH \
          --map=$MAP \
          --moment=$MOMENT_ID \
          --test-name=$TEST_CASE_NAME \
          --gtp "$GROUND_TRUTH_LOCAL" | tee evaluate.log
        if cat evaluate.log | grep "Evaluation Result:" -A 1 | grep "FAIL"; then
          EXIT_CODE=1
        else
          EXIT_CODE=0
        fi


        ossutil cp --config-file=/root/.ossutilconfig -r -f \
          $EVALUATE_OUTPUT_PATH \
          $OSS_DEST_PATH

        exit $EXIT_CODE
      volumeMounts:
      - mountPath: /data
        name: artifacts-volume-claim
      - mountPath: /root/.ossutilconfig
        name: credentials-volume
        subPath: .ossutilconfig
  - arguments: {}
    inputs: {}
    metadata: {}
    name: aggregation
    outputs: {}
    script:
      command:
      - bash
      env:
      - name: REGRESSION_COMMIT_ID
        value: '{{workflow.parameters.regression_commit_id}}'
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: REGENERATED_BUCKET
        valueFrom:
          configMapKeyRef:
            key: moment_regenerated_oss_bucket
            name: cluster-configs
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/infra_regenerate-gems:v20210105-5044b06
      name: ""
      resources: {}
      source: |
        set -x

        AGGREGATION_PATH=/data/aggregation_result
        mkdir -p $AGGREGATION_PATH

        ls -l $AGGREGATION_PATH
        ls -lR /data/evaluate_result
        aggregate \
          --input=/data/evaluate_result \
          --output=$AGGREGATION_PATH/aggregation_result.json
        RES=$?

        ls -l $AGGREGATION_PATH/aggregation_result.json

        OSS_DEST_PATH="oss://$REGENERATED_BUCKET/aggregation_result/$CYBERTRON_COMMIT_ID/$REGRESSION_COMMIT_ID"

        ossutil cp --config-file=/root/.ossutilconfig -r -f \
          $AGGREGATION_PATH \
          $OSS_DEST_PATH

        exit $RES
      volumeMounts:
      - mountPath: /data
        name: artifacts-volume-claim
      - mountPath: /root/.ossutilconfig
        name: credentials-volume
        subPath: .ossutilconfig
  - arguments: {}
    inputs:
      parameters:
      - name: cybertron_commit_timestamp
        value: '{{workflow.outputs.parameters.global_cybertron_commit_timestamp}}'
      - name: cybertron_commit_author
        value: '{{workflow.outputs.parameters.global_cybertron_commit_author}}'
    metadata: {}
    name: ingest
    outputs: {}
    script:
      command:
      - bash
      env:
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: CYBERTRON_BRANCH
        value: '{{workflow.parameters.cybertron_branch}}'
      - name: CYBERTRON_COMMIT_TIMESTAMP
        value: '{{inputs.parameters.cybertron_commit_timestamp}}'
      - name: CYBERTRON_COMMIT_AUTHOR
        value: '{{inputs.parameters.cybertron_commit_author}}'
      - name: WORKFLOW
        value: '{{workflow.name}}'
      - name: REGRESSION_COMMIT_ID
        value: '{{workflow.parameters.regression_commit_id}}'
      - name: REGENERATED_BUCKET
        valueFrom:
          configMapKeyRef:
            key: moment_regenerated_oss_bucket
            name: cluster-configs
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/infra_metrics:v20201218-ac92e3a
      name: ""
      resources: {}
      source: |
        set -x

        CYBERTRON_COMMIT_TIMESTAMP=$(echo $CYBERTRON_COMMIT_TIMESTAMP | sed -e 's/^"//' -e 's/"$//')
        OSS_DEST_PATH="oss://$REGENERATED_BUCKET/aggregation_result/$CYBERTRON_COMMIT_ID/$REGRESSION_COMMIT_ID"
        metrics ingest gems \
          --cybertron-commit-id=$CYBERTRON_COMMIT_ID \
          --cybertron-branch=$CYBERTRON_BRANCH \
          --cybertron-commit-timestamp=$CYBERTRON_COMMIT_TIMESTAMP \
          --cybertron-commit-author=$CYBERTRON_COMMIT_AUTHOR \
          --workflow=$WORKFLOW \
          --file-url=$OSS_DEST_PATH/aggregation_result.json \
          --merge
      volumeMounts:
      - mountPath: /etc/metrics/mysql-config.yaml
        name: metrics-volume
        subPath: mysql.yaml
      - mountPath: /etc/metrics/oss-config.yaml
        name: credentials-volume
        subPath: config.yaml
      - mountPath: /data
        name: artifacts-volume-claim
  ttlStrategy:
    secondsAfterFailure: 43200
    secondsAfterSuccess: 3600
  volumeClaimTemplates:
  - metadata:
      creationTimestamp: null
      name: artifacts-volume-claim
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: alicloud-flexvolume-nas-sc
    status: {}
  volumes:
  - name: credentials-volume
    secret:
      secretName: argo-hubble-secrets
  - name: gitlab-ssh-key-volume
    secret:
      defaultMode: 384
      secretName: gitlab-ssh-key-secrets
  - name: metrics-volume
    secret:
      secretName: fusion-metrics-secrets
  - emptyDir: {}
    name: tmpdir
status:
  artifactRepositoryRef:
    default: true
  finishedAt: null
  nodes:
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e:
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1059979194
      displayName: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      finishedAt: null
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      phase: Running
      progress: 23/23
      startedAt: "2021-01-12T06:43:00Z"
      templateName: gems-entrypoint
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Steps
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-23833680:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2330008490
      displayName: regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)
      finishedAt: "2021-01-12T07:05:49Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-23833680
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1221
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)
      outboundNodes:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-263337496
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 665
        memory: 665
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Steps
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-25933883:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3908230551
      displayName: regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)
      finishedAt: "2021-01-12T07:05:49Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-25933883
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_south_1131
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)
      outboundNodes:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1036164575
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 671
        memory: 671
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Steps
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-37416442:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-25933883
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2901720506
      displayName: regenerate-trucksim-step(0)
      finishedAt: "2021-01-12T07:05:11Z"
      hostNodeName: virtual-kubelet
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-37416442
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_south_1131
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)[0].regenerate-trucksim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 625
        memory: 625
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-46387983:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3512174579
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2581962514
      displayName: regenerate-trucksim-step
      finishedAt: "2021-01-12T07:06:00Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-46387983
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_2111
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)[0].regenerate-trucksim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 754
        memory: 754
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-87127999:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2066930524
      displayName: fetch-test-group-step
      finishedAt: "2021-01-12T06:59:34Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-87127999
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[1].fetch-test-group-step
      outputs:
        exitCode: "0"
        result: |-
          [
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_south_1141",
             "truck_sim": "true"
           },
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_north_2121",
             "truck_sim": "true"
           },
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_north_1321",
             "truck_sim": "true"
           },
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_north_1221",
             "truck_sim": "true"
           },
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_north_1121",
             "truck_sim": "true"
           },
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_60_south_1131",
             "truck_sim": "true"
           },
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_60_north_2111",
             "truck_sim": "true"
           },
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_60_north_1311",
             "truck_sim": "true"
           },
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_60_north_1211",
             "truck_sim": "true"
           },
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_60_north_1111",
             "truck_sim": "true"
           }
          ]
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 2
        memory: 2
      startedAt: "2021-01-12T06:59:26Z"
      templateName: fetch-test-group
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-183269666:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-690123435
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1212152362
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-183269666
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-195995491:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-892513611
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2714158922
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-195995491
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-222696875:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-302653348
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3593432789
      displayName: '[1]'
      finishedAt: "2021-01-12T07:05:39Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-222696875
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)[1]
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 46
        memory: 46
      startedAt: "2021-01-12T07:05:06Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-228188867:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-690123435
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1212152362
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-228188867
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-263337496:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-23833680
      displayName: evaluate-step(0)
      finishedAt: "2021-01-12T07:05:39Z"
      hostNodeName: cn-hangzhou.10.128.110.72
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-263337496
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1221
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)[1].evaluate-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T07:05:16Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-302653348:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1363722062
      displayName: regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)
      finishedAt: "2021-01-12T07:05:39Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-302653348
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1121
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)
      outboundNodes:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-793091124
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 660
        memory: 660
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Steps
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-307949813:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3903707460
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1786071508
      displayName: evaluate-step
      finishedAt: "2021-01-12T07:05:49Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-307949813
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_2121
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)[1].evaluate-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 47
        memory: 47
      startedAt: "2021-01-12T07:05:16Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-461462639:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-23833680
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2262750919
      displayName: regenerate-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-461462639
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)[0].regenerate-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-495427649:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-23833680
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-263337496
      displayName: evaluate-step
      finishedAt: "2021-01-12T07:05:49Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-495427649
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1221
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)[1].evaluate-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T07:05:16Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-504546219:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1954927711
      displayName: regenerate-trucksim-step
      finishedAt: null
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-504546219
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1111
      message: 'Waiting for argo-hubble/ConfigMap/gems-sync/trucksim_regenarate_template
        lock. Lock status: 0/9 '
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(9:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1111,truck_sim:true)[0].regenerate-trucksim-step
      phase: Pending
      startedAt: "2021-01-12T06:59:36Z"
      synchronizationStatus:
        waiting: argo-hubble/ConfigMap/gems-sync/trucksim_regenarate_template
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-645985476:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-892513611
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3767001135
      displayName: evaluate-step
      finishedAt: "2021-01-12T07:07:01Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-645985476
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1311
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)[1].evaluate-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T07:06:30Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-659249214:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1954927711
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-659249214
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(9:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1111,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-690123435:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1144894791
      displayName: regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)
      finishedAt: "2021-01-12T07:06:10Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-690123435
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1211
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)
      outboundNodes:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-808964239
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 694
        memory: 694
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Steps
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-702756248:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-302653348
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1231178987
      displayName: regenerate-trucksim-step
      finishedAt: "2021-01-12T07:05:06Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-702756248
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1121
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)[0].regenerate-trucksim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 660
        memory: 660
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-793091124:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-302653348
      displayName: evaluate-step(0)
      finishedAt: "2021-01-12T07:05:33Z"
      hostNodeName: cn-hangzhou.10.128.96.238
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-793091124
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1121
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)[1].evaluate-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 46
        memory: 46
      startedAt: "2021-01-12T07:05:06Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-808964239:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-690123435
      displayName: evaluate-step(0)
      finishedAt: "2021-01-12T07:06:00Z"
      hostNodeName: cn-hangzhou.10.128.96.238
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-808964239
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1211
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)[1].evaluate-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T07:05:39Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-853944042:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-892513611
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2714158922
      displayName: regenerate-trucksim-step(0)
      finishedAt: "2021-01-12T07:06:19Z"
      hostNodeName: virtual-kubelet
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-853944042
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1311
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)[0].regenerate-trucksim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 753
        memory: 753
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-892513611:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2646901351
      displayName: regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)
      finishedAt: "2021-01-12T07:07:01Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-892513611
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1311
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)
      outboundNodes:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3767001135
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 791
        memory: 791
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Steps
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-901976510:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-302653348
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-222696875
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-901976510
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 46
        memory: 46
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-944121719:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-892513611
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-853944042
      displayName: regenerate-trucksim-step
      finishedAt: "2021-01-12T07:06:30Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-944121719
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1311
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)[0].regenerate-trucksim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 791
        memory: 791
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1036164575:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-25933883
      displayName: evaluate-step(0)
      finishedAt: "2021-01-12T07:05:43Z"
      hostNodeName: cn-hangzhou.10.128.96.238
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1036164575
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_south_1131
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)[1].evaluate-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 46
        memory: 46
      startedAt: "2021-01-12T07:05:16Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1059979194:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2972486757
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3118598686
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2716065380
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3905507018
      displayName: '[0]'
      finishedAt: "2021-01-12T06:59:26Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1059979194
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[0]
      phase: Succeeded
      progress: 23/23
      resourcesDuration:
        cpu: 11894
        memory: 11894
      startedAt: "2021-01-12T06:43:00Z"
      templateName: gems-entrypoint
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1091565038:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3903707460
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1466575691
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3384778936
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-4065978846
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3461042437
      displayName: '[0]'
      finishedAt: "2021-01-12T07:05:16Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1091565038
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)[0]
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 664
        memory: 664
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1112672011:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3903707460
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2098075083
      displayName: regenerate-trucksim-step(0)
      finishedAt: "2021-01-12T07:05:08Z"
      hostNodeName: virtual-kubelet
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1112672011
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_2121
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)[0].regenerate-trucksim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 617
        memory: 617
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1144894791:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-690123435
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3959449538
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3283133655
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-228188867
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-183269666
      displayName: '[0]'
      finishedAt: "2021-01-12T07:05:39Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1144894791
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)[0]
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 694
        memory: 694
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1212152362:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-690123435
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2677065508
      displayName: '[1]'
      finishedAt: "2021-01-12T07:06:10Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1212152362
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)[1]
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T07:05:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1231178987:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-302653348
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-222696875
      displayName: regenerate-trucksim-step(0)
      finishedAt: "2021-01-12T07:05:05Z"
      hostNodeName: virtual-kubelet
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1231178987
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1121
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)[0].regenerate-trucksim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 614
        memory: 614
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1301740827:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2215544084
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-4151754757
      displayName: '[1]'
      finishedAt: "2021-01-12T07:05:39Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1301740827
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)[1]
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 48
        memory: 48
      startedAt: "2021-01-12T07:05:06Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1363722062:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-302653348
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2094654763
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-702756248
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-901976510
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3468386789
      displayName: '[0]'
      finishedAt: "2021-01-12T07:05:06Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1363722062
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)[0]
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 660
        memory: 660
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1368998398:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2215544084
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2462941339
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2177301544
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3461882798
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-4103516405
      displayName: '[0]'
      finishedAt: "2021-01-12T07:05:06Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1368998398
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)[0]
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 646
        memory: 646
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1382975570:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-25933883
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2901720506
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1382975570
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 46
        memory: 46
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1391739466:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3512174579
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3925502338
      displayName: regenerate-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1391739466
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)[0].regenerate-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1466575691:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3903707460
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2098075083
      displayName: regenerate-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1466575691
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)[0].regenerate-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 47
        memory: 47
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1532059727:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2066489239
      displayName: build-cybertron-sim-p1-step(0)
      finishedAt: "2021-01-12T06:57:29Z"
      hostNodeName: cn-hangzhou.10.128.110.71
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1532059727
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[0].build-cybertron-sim-p1-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 1721
        memory: 1721
      startedAt: "2021-01-12T06:43:00Z"
      templateName: build-cybertron-sim-p1
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1572815716:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2215544084
      displayName: evaluate-step(0)
      finishedAt: "2021-01-12T07:05:35Z"
      hostNodeName: cn-hangzhou.10.128.96.238
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1572815716
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1321
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)[1].evaluate-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 48
        memory: 48
      startedAt: "2021-01-12T07:05:06Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1578161610:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-690123435
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1212152362
      displayName: regenerate-trucksim-step(0)
      finishedAt: "2021-01-12T07:05:29Z"
      hostNodeName: virtual-kubelet
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1578161610
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1211
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)[0].regenerate-trucksim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 656
        memory: 656
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1651362476:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3512174579
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2396221479
      displayName: evaluate-step
      finishedAt: "2021-01-12T07:06:30Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1651362476
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_2111
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)[1].evaluate-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T07:06:00Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1709153968:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3360149644
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2054059219
      displayName: regenerate-trucksim-step
      finishedAt: "2021-01-12T07:05:06Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1709153968
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_south_1141
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)[0].regenerate-trucksim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 664
        memory: 664
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1782319389:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3360149644
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3967631148
      displayName: evaluate-step
      finishedAt: "2021-01-12T07:05:39Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1782319389
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_south_1141
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)[1].evaluate-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 48
        memory: 48
      startedAt: "2021-01-12T07:05:06Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1786071508:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3903707460
      displayName: evaluate-step(0)
      finishedAt: "2021-01-12T07:05:43Z"
      hostNodeName: cn-hangzhou.10.128.96.238
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1786071508
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_2121
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)[1].evaluate-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 47
        memory: 47
      startedAt: "2021-01-12T07:05:16Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1791774067:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3360149644
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2512487043
      displayName: regenerate-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1791774067
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)[0].regenerate-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 48
        memory: 48
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1954927711:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2139304011
      displayName: regression-step(9:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1111,truck_sim:true)
      finishedAt: null
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1954927711
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1111
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(9:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1111,truck_sim:true)
      phase: Running
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Steps
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2054059219:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3360149644
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2512487043
      displayName: regenerate-trucksim-step(0)
      finishedAt: "2021-01-12T07:05:05Z"
      hostNodeName: virtual-kubelet
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2054059219
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_south_1141
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)[0].regenerate-trucksim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 616
        memory: 616
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2058281855:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-23833680
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2262750919
      displayName: regenerate-trucksim-step(0)
      finishedAt: "2021-01-12T07:05:13Z"
      hostNodeName: virtual-kubelet
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2058281855
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1221
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)[0].regenerate-trucksim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 627
        memory: 627
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2066489239:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-87127999
      displayName: '[1]'
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2066489239
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[1]
      phase: Succeeded
      progress: 19/19
      resourcesDuration:
        cpu: 6211
        memory: 6211
      startedAt: "2021-01-12T06:59:26Z"
      templateName: gems-entrypoint
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2066930524:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3360149644
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3903707460
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2215544084
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-23833680
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-302653348
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-25933883
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3512174579
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-892513611
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-690123435
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1954927711
      displayName: '[2]'
      finishedAt: null
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2066930524
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2]
      phase: Running
      progress: 18/18
      startedAt: "2021-01-12T06:59:36Z"
      templateName: gems-entrypoint
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2074540468:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-25933883
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1036164575
      displayName: evaluate-step
      finishedAt: "2021-01-12T07:05:49Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2074540468
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_south_1131
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)[1].evaluate-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 46
        memory: 46
      startedAt: "2021-01-12T07:05:16Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2094654763:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-302653348
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-222696875
      displayName: regenerate-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2094654763
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)[0].regenerate-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 46
        memory: 46
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2098075083:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3903707460
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-307949813
      displayName: '[1]'
      finishedAt: "2021-01-12T07:05:49Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2098075083
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)[1]
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 47
        memory: 47
      startedAt: "2021-01-12T07:05:16Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2139304011:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1954927711
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2675173534
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-504546219
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3886826103
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-659249214
      displayName: '[0]'
      finishedAt: null
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2139304011
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(9:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1111,truck_sim:true)[0]
      phase: Running
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2177301544:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2215544084
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3640068155
      displayName: regenerate-trucksim-step
      finishedAt: "2021-01-12T07:05:06Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2177301544
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1321
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)[0].regenerate-trucksim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 646
        memory: 646
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2215544084:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1368998398
      displayName: regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)
      finishedAt: "2021-01-12T07:05:39Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2215544084
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1321
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)
      outboundNodes:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1572815716
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 646
        memory: 646
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Steps
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2254305746:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-25933883
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2901720506
      displayName: regenerate-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2254305746
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)[0].regenerate-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 46
        memory: 46
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2262750919:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-23833680
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-495427649
      displayName: '[1]'
      finishedAt: "2021-01-12T07:05:49Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2262750919
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)[1]
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T07:05:16Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2330008490:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-23833680
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-461462639
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2939573972
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3938187562
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3883567177
      displayName: '[0]'
      finishedAt: "2021-01-12T07:05:16Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2330008490
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)[0]
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 665
        memory: 665
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2396221479:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3512174579
      displayName: evaluate-step(0)
      finishedAt: "2021-01-12T07:06:21Z"
      hostNodeName: cn-hangzhou.10.128.96.238
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2396221479
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_2111
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)[1].evaluate-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T07:06:00Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2462941339:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2215544084
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1301740827
      displayName: regenerate-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2462941339
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)[0].regenerate-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 48
        memory: 48
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2501369538:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-892513611
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2714158922
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2501369538
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2512487043:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3360149644
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1782319389
      displayName: '[1]'
      finishedAt: "2021-01-12T07:05:39Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2512487043
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)[1]
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 48
        memory: 48
      startedAt: "2021-01-12T07:05:06Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2579744614:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3360149644
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1791774067
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1709153968
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3219618294
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3342516685
      displayName: '[0]'
      finishedAt: "2021-01-12T07:05:06Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2579744614
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)[0]
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 664
        memory: 664
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2581962514:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3512174579
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3925502338
      displayName: regenerate-trucksim-step(0)
      finishedAt: "2021-01-12T07:05:56Z"
      hostNodeName: virtual-kubelet
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2581962514
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_2111
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)[0].regenerate-trucksim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 716
        memory: 716
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2646901351:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-892513611
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2858153314
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-944121719
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-195995491
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2501369538
      displayName: '[0]'
      finishedAt: "2021-01-12T07:06:30Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2646901351
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)[0]
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 791
        memory: 791
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2675173534:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1954927711
      displayName: regenerate-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2675173534
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(9:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1111,truck_sim:true)[0].regenerate-step
      phase: Skipped
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2677065508:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-690123435
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-808964239
      displayName: evaluate-step
      finishedAt: "2021-01-12T07:06:10Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2677065508
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1211
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)[1].evaluate-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T07:05:39Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2714158922:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-892513611
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-645985476
      displayName: '[1]'
      finishedAt: "2021-01-12T07:07:01Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2714158922
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)[1]
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T07:06:30Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2716065380:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1532059727
      displayName: build-cybertron-sim-p1-step
      finishedAt: "2021-01-12T06:57:39Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2716065380
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[0].build-cybertron-sim-p1-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 20/20
      resourcesDuration:
        cpu: 7932
        memory: 7932
      startedAt: "2021-01-12T06:43:00Z"
      templateName: build-cybertron-sim-p1
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2784477151:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3512174579
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1391739466
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-46387983
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-4016896955
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3542058314
      displayName: '[0]'
      finishedAt: "2021-01-12T07:06:00Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2784477151
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)[0]
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 754
        memory: 754
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2858153314:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-892513611
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2714158922
      displayName: regenerate-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2858153314
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)[0].regenerate-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2901720506:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-25933883
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2074540468
      displayName: '[1]'
      finishedAt: "2021-01-12T07:05:49Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2901720506
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)[1]
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 46
        memory: 46
      startedAt: "2021-01-12T07:05:16Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2939573972:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-23833680
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2058281855
      displayName: regenerate-trucksim-step
      finishedAt: "2021-01-12T07:05:16Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2939573972
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1221
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)[0].regenerate-trucksim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 665
        memory: 665
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2972486757:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3353230148
      displayName: build-cybertron-pkg-step
      finishedAt: "2021-01-12T06:59:26Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2972486757
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[0].build-cybertron-pkg-step
      outputs:
        exitCode: "0"
        parameters:
        - globalName: global_cybertron_commit_timestamp
          name: cybertron_commit_timestamp
          value: '"1610430777"'
          valueFrom:
            path: /tmpdir/cybertron_commit_timestamp.txt
        - globalName: global_cybertron_commit_author
          name: cybertron_commit_author
          value: xiaofan.mu@inceptio.ai
          valueFrom:
            path: /tmpdir/cybertron_commit_author.txt
        - globalName: global_idl_version
          name: idl_version
          value: 0.8.1
          valueFrom:
            path: /tmpdir/idl_version.txt
      phase: Succeeded
      progress: 20/20
      resourcesDuration:
        cpu: 8149
        memory: 8149
      startedAt: "2021-01-12T06:43:00Z"
      templateName: build-cybertron-pkg
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3008806611:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-25933883
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2901720506
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3008806611
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 46
        memory: 46
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3118598686:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3428290381
      displayName: build-cybertron-sim-step
      finishedAt: "2021-01-12T06:58:37Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3118598686
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[0].build-cybertron-sim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 20/20
      resourcesDuration:
        cpu: 8049
        memory: 8049
      startedAt: "2021-01-12T06:43:00Z"
      templateName: build-cybertron-sim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3219618294:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3360149644
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2512487043
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3219618294
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 48
        memory: 48
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3283133655:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-690123435
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1578161610
      displayName: regenerate-trucksim-step
      finishedAt: "2021-01-12T07:05:39Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3283133655
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1211
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)[0].regenerate-trucksim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 694
        memory: 694
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3342516685:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3360149644
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2512487043
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3342516685
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 48
        memory: 48
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3353230148:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2066489239
      displayName: build-cybertron-pkg-step(0)
      finishedAt: "2021-01-12T06:59:16Z"
      hostNodeName: cn-hangzhou.10.128.110.71
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3353230148
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[0].build-cybertron-pkg-step(0)
      outputs:
        exitCode: "0"
        parameters:
        - globalName: global_cybertron_commit_timestamp
          name: cybertron_commit_timestamp
          value: '"1610430777"'
          valueFrom:
            path: /tmpdir/cybertron_commit_timestamp.txt
        - globalName: global_cybertron_commit_author
          name: cybertron_commit_author
          value: xiaofan.mu@inceptio.ai
          valueFrom:
            path: /tmpdir/cybertron_commit_author.txt
        - globalName: global_idl_version
          name: idl_version
          value: 0.8.1
          valueFrom:
            path: /tmpdir/idl_version.txt
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 1938
        memory: 1938
      startedAt: "2021-01-12T06:43:00Z"
      templateName: build-cybertron-pkg
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3360149644:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2579744614
      displayName: regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)
      finishedAt: "2021-01-12T07:05:39Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3360149644
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_south_1141
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)
      outboundNodes:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3967631148
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 664
        memory: 664
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Steps
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3384778936:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3903707460
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1112672011
      displayName: regenerate-trucksim-step
      finishedAt: "2021-01-12T07:05:16Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3384778936
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_2121
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)[0].regenerate-trucksim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 664
        memory: 664
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3428290381:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2066489239
      displayName: build-cybertron-sim-step(0)
      finishedAt: "2021-01-12T06:58:27Z"
      hostNodeName: cn-hangzhou.10.128.110.71
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3428290381
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[0].build-cybertron-sim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 1838
        memory: 1838
      startedAt: "2021-01-12T06:43:00Z"
      templateName: build-cybertron-sim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3461042437:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3903707460
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2098075083
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3461042437
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 47
        memory: 47
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3461882798:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2215544084
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1301740827
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3461882798
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 48
        memory: 48
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3468386789:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-302653348
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-222696875
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3468386789
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 46
        memory: 46
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3512174579:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2784477151
      displayName: regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)
      finishedAt: "2021-01-12T07:06:30Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3512174579
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_2111
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)
      outboundNodes:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2396221479
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 754
        memory: 754
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Steps
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3542058314:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3512174579
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3925502338
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3542058314
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3593432789:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-302653348
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-793091124
      displayName: evaluate-step
      finishedAt: "2021-01-12T07:05:39Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3593432789
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1121
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)[1].evaluate-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 46
        memory: 46
      startedAt: "2021-01-12T07:05:06Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3640068155:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2215544084
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1301740827
      displayName: regenerate-trucksim-step(0)
      finishedAt: "2021-01-12T07:04:56Z"
      hostNodeName: virtual-kubelet
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3640068155
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1321
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)[0].regenerate-trucksim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 598
        memory: 598
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3651491441:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2066489239
      displayName: fetch-regression-repository-step(0)
      finishedAt: "2021-01-12T06:44:40Z"
      hostNodeName: cn-hangzhou.10.128.110.71
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3651491441
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[0].fetch-regression-repository-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 186
        memory: 186
      startedAt: "2021-01-12T06:43:00Z"
      templateName: fetch-regression-repository
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3767001135:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-892513611
      displayName: evaluate-step(0)
      finishedAt: "2021-01-12T07:06:51Z"
      hostNodeName: cn-hangzhou.10.128.96.238
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3767001135
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1311
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)[1].evaluate-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T07:06:30Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3883567177:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-23833680
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2262750919
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3883567177
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3886826103:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1954927711
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3886826103
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(9:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1111,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3903707460:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1091565038
      displayName: regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)
      finishedAt: "2021-01-12T07:05:49Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3903707460
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_2121
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)
      outboundNodes:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1786071508
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 664
        memory: 664
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Steps
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3905507018:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3651491441
      displayName: fetch-regression-repository-step
      finishedAt: "2021-01-12T06:44:50Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3905507018
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[0].fetch-regression-repository-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 20/20
      resourcesDuration:
        cpu: 6397
        memory: 6397
      startedAt: "2021-01-12T06:43:00Z"
      templateName: fetch-regression-repository
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3908230551:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-25933883
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2254305746
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-4267979463
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3008806611
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1382975570
      displayName: '[0]'
      finishedAt: "2021-01-12T07:05:16Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3908230551
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)[0]
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 671
        memory: 671
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3925502338:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3512174579
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1651362476
      displayName: '[1]'
      finishedAt: "2021-01-12T07:06:30Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3925502338
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)[1]
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T07:06:00Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: StepGroup
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3938187562:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-23833680
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2262750919
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3938187562
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3959449538:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-690123435
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1212152362
      displayName: regenerate-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3959449538
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)[0].regenerate-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3967631148:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3360149644
      displayName: evaluate-step(0)
      finishedAt: "2021-01-12T07:05:34Z"
      hostNodeName: cn-hangzhou.10.128.96.238
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3967631148
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_south_1141
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)[1].evaluate-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 48
        memory: 48
      startedAt: "2021-01-12T07:05:06Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Pod
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-4016896955:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3512174579
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3925502338
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-4016896955
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 38
        memory: 38
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-4065978846:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3903707460
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2098075083
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-4065978846
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 47
        memory: 47
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-4103516405:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2215544084
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1301740827
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-12T06:59:36Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-4103516405
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 48
        memory: 48
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Skipped
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-4151754757:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2215544084
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1572815716
      displayName: evaluate-step
      finishedAt: "2021-01-12T07:05:39Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-4151754757
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1321
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)[1].evaluate-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 48
        memory: 48
      startedAt: "2021-01-12T07:05:06Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
    gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-4267979463:
      boundaryID: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-25933883
      children:
      - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-37416442
      displayName: regenerate-trucksim-step
      finishedAt: "2021-01-12T07:05:16Z"
      id: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-4267979463
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_south_1131
      name: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)[0].regenerate-trucksim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 671
        memory: 671
      startedAt: "2021-01-12T06:59:36Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
      type: Retry
  outputs:
    parameters:
    - name: global_cybertron_commit_timestamp
      value: '"1610430777"'
    - name: global_cybertron_commit_author
      value: xiaofan.mu@inceptio.ai
    - name: global_idl_version
      value: 0.8.1
  persistentVolumeClaims:
  - name: artifacts-volume-claim
    persistentVolumeClaim:
      claimName: gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-artifacts-volume-claim
  phase: Running
  progress: 23/23
  resourcesDuration:
    cpu: 11894
    memory: 11894
  startedAt: "2021-01-12T06:43:00Z"
  synchronization:
    semaphore:
      holding:
      - holders:
        - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e
        semaphore: argo-hubble/ConfigMap/gems-sync/gems_wf_limited
      - holders:
        - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1709153968
        - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3384778936
        - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2177301544
        - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2939573972
        - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-702756248
        - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-4267979463
        - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-46387983
        - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-944121719
        - gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3283133655
        semaphore: argo-hubble/ConfigMap/gems-sync/trucksim_regenarate_template
      waiting:
      - holders:
        - argo-hubble/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-1709153968
        - argo-hubble/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2939573972
        - argo-hubble/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-4267979463
        - argo-hubble/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3283133655
        - argo-hubble/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-3384778936
        - argo-hubble/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-2177301544
        - argo-hubble/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-702756248
        - argo-hubble/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-46387983
        - argo-hubble/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e/gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-944121719
        semaphore: argo-hubble/ConfigMap/gems-sync/trucksim_regenarate_template
Paste the logs from the workflow controller:
kubectl logs -n argo $(kubectl get pods -l app=workflow-controller -n argo -o name) | grep ${workflow}

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

@juchaosong
Copy link
Contributor Author

The node gems-wf-v0-8-21db1d4b-4a2c-40b0-8ebe-228d2f53d96e-504546219's waiting message is message: 'Waiting for argo-hubble/ConfigMap/gems-sync/trucksim_regenarate_template lock. Lock status: 0/9 '. But other synchronization. semaphore.holding.holders is finished.

@sarabala1979 sarabala1979 self-assigned this Jan 13, 2021
@sarabala1979
Copy link
Member

@juchaosong can you provide the workflow controller log?

@juchaosong
Copy link
Contributor Author

Another wf.yaml

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  annotations:
    workflows.inceptio.ai/failed-channel: workflow-failed-dev
    workflows.inceptio.ai/succeeded-channel: workflow-succeeded-dev
  creationTimestamp: "2021-01-13T12:14:45Z"
  generation: 57
  labels:
    workflows.argoproj.io/controller-instanceid: kraken-v2.12
    workflows.argoproj.io/phase: Running
    workflows.inceptio.ai/author: juchao.song
    workflows.inceptio.ai/template: gems-wf-v0-8
  name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
  namespace: argo-hubble
  resourceVersion: "2466407581"
  selfLink: /apis/argoproj.io/v1alpha1/namespaces/argo-hubble/workflows/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
  uid: 68974504-8407-4ca4-be7b-2bef9d35fb04
spec:
  arguments:
    parameters:
    - name: test_case_group
      value: control_tests
    - name: user
      value: XiaoLing Li
    - name: cybertron_commit_id
      value: eeb8b1ccf2f95199cf0faeb52772c7ce8ad55a24
    - name: cybertron_branch
      value: master
    - name: regression_commit_id
      value: master
  entrypoint: gems-entrypoint
  onExit: exit-handler
  parallelism: 100
  synchronization:
    semaphore:
      configMapKeyRef:
        key: gems_wf_limited
        name: gems-sync
  templates:
  - arguments: {}
    inputs: {}
    metadata: {}
    name: gems-entrypoint
    outputs: {}
    steps:
    - - arguments: {}
        continueOn:
          failed: true
        name: build-cybertron-pkg-step
        template: build-cybertron-pkg
      - arguments: {}
        name: build-cybertron-sim-step
        template: build-cybertron-sim
      - arguments: {}
        name: build-cybertron-sim-p1-step
        template: build-cybertron-sim-p1
      - arguments: {}
        name: fetch-regression-repository-step
        template: fetch-regression-repository
    - - arguments: {}
        name: fetch-test-group-step
        template: fetch-test-group
    - - arguments:
          parameters:
          - name: test_case_name
            value: '{{item.test_case_name}}'
          - name: gpu_requirement
            value: '{{item.gpu_requirement}}'
          - name: truck_sim
            value: '{{item.truck_sim}}'
        continueOn:
          failed: true
        name: regression-step
        template: regression
        withParam: '{{steps.fetch-test-group-step.outputs.result}}'
    - - arguments: {}
        continueOn:
          failed: true
        name: aggregation-step
        template: aggregation
    - - arguments: {}
        name: ingest-step
        template: ingest
  - arguments: {}
    inputs: {}
    metadata: {}
    name: exit-handler
    outputs: {}
    steps:
    - - arguments: {}
        name: celebrate
        template: celebrate
        when: '{{workflow.status}} == Succeeded'
      - arguments: {}
        name: cry
        template: cry
        when: '{{workflow.status}} != Succeeded'
  - arguments: {}
    inputs:
      parameters:
      - name: cybertron_commit_timestamp
        value: '{{workflow.outputs.parameters.global_cybertron_commit_timestamp}}'
      - name: cybertron_commit_author
        value: '{{workflow.outputs.parameters.global_cybertron_commit_author}}'
    metadata: {}
    name: celebrate
    outputs: {}
    script:
      command:
      - bash
      env:
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: CYBERTRON_BRANCH
        value: '{{workflow.parameters.cybertron_branch}}'
      - name: CYBERTRON_COMMIT_AUTHOR
        value: '{{inputs.parameters.cybertron_commit_author}}'
      - name: TEST_CASE_GROUP
        value: '{{workflow.parameters.test_case_group}}'
      - name: TRIGGER_USER
        value: '{{workflow.parameters.user}}'
      - name: GEMS_NOTIFY_WECHAT_URL
        valueFrom:
          configMapKeyRef:
            key: gems_wechat_url
            name: cluster-configs
      - name: CHARIOT_WEB_HOST
        valueFrom:
          configMapKeyRef:
            key: cht_web_host
            name: cluster-configs
      - name: GRAFANA_WEB_HOST
        valueFrom:
          configMapKeyRef:
            key: grafana_web_host
            name: cluster-configs
      image: everpeace/curl-jq
      name: ""
      resources: {}
      source: |
        CHARIOT_URL="$CHARIOT_WEB_HOST/workflows/{{workflow.name}}"
        GRAFANA_URL="${GRAFANA_WEB_HOST}/d/R6oDol5Gz/gems-metrics?orgId=1&var-branch=${CYBERTRON_BRANCH}&var-commitId=${CYBERTRON_COMMIT_ID}&var-workflow={{workflow.name}}&from=now-30d&to=now"
        GRAFANA_WF_STATUS_URL="${GRAFANA_WEB_HOST}/d/ouaL2d2Mz/gems-workflows?orgId=1&var-workflow={{workflow.name}}"
        curl -s "${GEMS_NOTIFY_WECHAT_URL}" \
        -H 'Content-Type: application/json' \
        -d '{
                "msgtype": "markdown",
                "markdown": {
                        "content": "<font color=\"info\">Regression test workflow succeed!!!</font>
                        > workflow_id:<font color=\"comment\">'"{{workflow.name}}"'</font>
                        > commit_id:<font color=\"comment\">'"${CYBERTRON_COMMIT_ID:-"UNKNOWN"}"'</font>
                        > branch:<font color=\"comment\">'"${CYBERTRON_BRANCH:-"UNKNOWN"}"'</font>
                        > author:<font color=\"comment\">'"${CYBERTRON_COMMIT_AUTHOR:-"UNKNOWN"}"'</font>
                        > user:<font color=\"comment\">'"${TRIGGER_USER:-"UNKNOWN"}"'</font>
                        > test_group:<font color=\"comment\">'"${TEST_CASE_GROUP:-"UNKNOWN"}"'</font>
                        [Chariot Link]('${CHARIOT_URL}')
                        [Grafana Link]('${GRAFANA_URL}')
                        [Grafana Workflow Status Link]('${GRAFANA_WF_STATUS_URL}')"
                }
            }'
      volumeMounts:
      - mountPath: /tmpdir
        name: tmpdir
  - arguments: {}
    inputs:
      parameters:
      - name: cybertron_commit_timestamp
        value: '{{workflow.outputs.parameters.global_cybertron_commit_timestamp}}'
      - name: cybertron_commit_author
        value: '{{workflow.outputs.parameters.global_cybertron_commit_author}}'
    metadata: {}
    name: cry
    outputs: {}
    script:
      command:
      - bash
      env:
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: CYBERTRON_BRANCH
        value: '{{workflow.parameters.cybertron_branch}}'
      - name: CYBERTRON_COMMIT_AUTHOR
        value: '{{inputs.parameters.cybertron_commit_author}}'
      - name: TRIGGER_USER
        value: '{{workflow.parameters.user}}'
      - name: TEST_CASE_GROUP
        value: '{{workflow.parameters.test_case_group}}'
      - name: GEMS_NOTIFY_WECHAT_URL
        valueFrom:
          configMapKeyRef:
            key: gems_wechat_url
            name: cluster-configs
      - name: CHARIOT_WEB_HOST
        valueFrom:
          configMapKeyRef:
            key: cht_web_host
            name: cluster-configs
      image: everpeace/curl-jq
      name: ""
      resources: {}
      source: |
        CHARIOT_URL="$CHARIOT_WEB_HOST/workflows/{{workflow.name}}"
        GRAFANA_WF_STATUS_URL="${GRAFANA_WEB_HOST}/d/ouaL2d2Mz/gems-workflows?orgId=1&var-workflow={{workflow.name}}"
        curl -s "${GEMS_NOTIFY_WECHAT_URL}" -H 'Content-Type: application/json' \
        -d '{
              "msgtype": "markdown",
              "markdown": {
              "content": "<font color=\"warning\">Regression test workflow failed!!!</font>
                          > workflow_id:<font color=\"comment\">'"{{workflow.name}}"'</font>
                          > commit_id:<font color=\"comment\">'"${CYBERTRON_COMMIT_ID:-"UNKNOWN"}"'</font>
                          > branch:<font color=\"comment\">'"${CYBERTRON_BRANCH:-"UNKNOWN"}"'</font>
                          > author:<font color=\"comment\">'"${CYBERTRON_COMMIT_AUTHOR:-"UNKNOWN"}"'</font>
                          > user:<font color=\"comment\">'"${TRIGGER_USER:-"UNKNOWN"}"'</font>
                          > test_group:<font color=\"comment\">'"${TEST_CASE_GROUP:-"UNKNOWN"}"'</font>
                          Please refer: [Chariot Link]('${CHARIOT_URL}')
                                        [Grafana Workflow Status Link]('${GRAFANA_WF_STATUS_URL}')"
              }
        }'
      volumeMounts:
      - mountPath: /tmpdir
        name: tmpdir
  - arguments: {}
    inputs: {}
    metadata: {}
    name: build-cybertron-pkg
    outputs:
      parameters:
      - globalName: global_cybertron_commit_timestamp
        name: cybertron_commit_timestamp
        valueFrom:
          path: /tmpdir/cybertron_commit_timestamp.txt
      - globalName: global_cybertron_commit_author
        name: cybertron_commit_author
        valueFrom:
          path: /tmpdir/cybertron_commit_author.txt
      - globalName: global_idl_version
        name: idl_version
        valueFrom:
          path: /tmpdir/idl_version.txt
    retryStrategy:
      limit: 2
    script:
      command:
      - bash
      env:
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/sde_cybertron_release:2020_07_03_c12745f7
      name: ""
      resources: {}
      source: |
        set -ex

        ssh-keyscan gitlab.rd.inceptioglobal.ai >> ~/.ssh/known_hosts
        chmod 644 ~/.ssh/known_hosts

        export HTTP_PROXY=http://gfw-proxy.inceptio.tech:8118
        export HTTPS_PROXY=http://gfw-proxy.inceptio.tech:8118
        export NO_PROXY=.cn,.aliyun.com,.aliyuncs.com,.chariot,.artifactory

        mkdir -p /root/cybertron && cd /root/cybertron
        git init
        git remote add origin https://gitlab:$GITLAB_TOKEN@gitlab.rd.inceptioglobal.ai/autonomy/cybertron.git
        git fetch --depth=1 origin $CYBERTRON_COMMIT_ID
        git reset --hard FETCH_HEAD
        echo "\"$(git show -s --format=%ct HEAD)\"" > /tmpdir/cybertron_commit_timestamp.txt
        git show -s --format='%ae' HEAD > /tmpdir/cybertron_commit_author.txt

        # FXIME: install empy when buiding image
        pip3 install empy

        # bazel build \
        #   --define tensorrt=default \
        #   --remote_cache=grpc://bb-storage.inceptio.tech:8980 \
        #   --remote_instance_name=cybertron \
        #   //:bin_pkg_tar //interfaces/tools/gems/pb_converter:all

        # get idl version
        if [ -f irs/common/recorder/idl_record_io.h ];then
          IDL_RECORD_IO=irs/common/recorder/idl_record_io.h
        else
          IDL_RECORD_IO=bazel-cybertron/external/wheeljack_x86_64/irs/common/recorder/idl_record_io.h
        fi
        MAJOR=$(cat $IDL_RECORD_IO | sed -rn 's/^.*kCurrentRepoVersionMajor\ *=\ *([0-9]*);/\1/p')
        MINOR=$(cat $IDL_RECORD_IO | sed -rn 's/^.*kCurrentRepoVersionMinor\ *=\ *([0-9]*);/\1/p')
        IDL=$(cat $IDL_RECORD_IO | sed -rn 's/^.*kCurrentIdlHiddenVersion\ *=\ *([0-9]*);/\1/p')
        echo ${MAJOR}.${MINOR}.${IDL} > /tmpdir/idl_version.txt

        exit 0

        cd ..
        tar --exclude='cybertron/bazel-cybertron' \
          --exclude='cybertron/bazel-out' \
          -h -cf cybertron-pkg.tar cybertron
        ls -alh cybertron-pkg.tar
        cp cybertron-pkg.tar /data
      volumeMounts:
      - mountPath: /root/.jfrog/jfrog-cli.conf
        name: credentials-volume
        subPath: jfrog-cli.conf
      - mountPath: /data
        name: artifacts-volume-claim
      - mountPath: /root/.ssh/id_rsa
        name: gitlab-ssh-key-volume
        subPath: id_rsa
      - mountPath: /tmpdir
        name: tmpdir
  - arguments: {}
    inputs: {}
    metadata: {}
    name: build-cybertron-sim
    outputs: {}
    retryStrategy:
      limit: 2
    script:
      command:
      - bash
      env:
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/sde_cybertron_release:2020_07_03_c12745f7
      name: ""
      resources: {}
      source: |
        set -ex
        exit 0

        ssh-keyscan gitlab.rd.inceptioglobal.ai >> ~/.ssh/known_hosts
        chmod 644 ~/.ssh/known_hosts

        export HTTP_PROXY=http://gfw-proxy.inceptio.tech:8118
        export HTTPS_PROXY=http://gfw-proxy.inceptio.tech:8118
        export NO_PROXY=.cn,.aliyun.com,.aliyuncs.com,.chariot,.artifactory

        mkdir -p /root/cybertron && cd /root/cybertron
        git init
        git remote add origin https://gitlab:$GITLAB_TOKEN@gitlab.rd.inceptioglobal.ai/autonomy/cybertron.git
        git fetch --depth=1 origin $CYBERTRON_COMMIT_ID
        git reset --hard FETCH_HEAD

        # FXIME: install empy when buiding image
        pip3 install empy

        bazel build \
          --define tensorrt=default \
          --remote_cache=grpc://bb-storage.inceptio.tech:8980 \
          --remote_instance_name=cybertron \
          //simulation/launch:simulation_main --config=sim
        bazel build \
          --define tensorrt=default \
          --remote_cache=grpc://bb-storage.inceptio.tech:8980 \
          --remote_instance_name=cybertron \
          //interfaces/tools/gems/pb_converter:all

        cd ..
        tar --exclude='cybertron/bazel-cybertron' \
          --exclude='cybertron/bazel-out' \
          -h -cf cybertron-sim.tar cybertron
        ls -alh cybertron-sim.tar
        cp cybertron-sim.tar /data
      volumeMounts:
      - mountPath: /root/.jfrog/jfrog-cli.conf
        name: credentials-volume
        subPath: jfrog-cli.conf
      - mountPath: /data
        name: artifacts-volume-claim
      - mountPath: /root/.ssh/id_rsa
        name: gitlab-ssh-key-volume
        subPath: id_rsa
  - arguments: {}
    inputs: {}
    metadata: {}
    name: build-cybertron-sim-p1
    outputs: {}
    retryStrategy:
      limit: 2
    script:
      command:
      - bash
      env:
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/sde_cybertron_release:2020_07_03_c12745f7
      name: ""
      resources: {}
      source: |
        set -ex
        exit 0

        ssh-keyscan gitlab.rd.inceptioglobal.ai >> ~/.ssh/known_hosts
        chmod 644 ~/.ssh/known_hosts

        export HTTP_PROXY=http://gfw-proxy.inceptio.tech:8118
        export HTTPS_PROXY=http://gfw-proxy.inceptio.tech:8118
        export NO_PROXY=.cn,.aliyun.com,.aliyuncs.com,.chariot,.artifactory

        mkdir -p /root/cybertron && cd /root/cybertron
        git init
        git remote add origin https://gitlab:$GITLAB_TOKEN@gitlab.rd.inceptioglobal.ai/autonomy/cybertron.git
        git fetch --depth=1 origin $CYBERTRON_COMMIT_ID
        git reset --hard FETCH_HEAD

        # FXIME: install empy when buiding image
        pip3 install empy

        bazel build \
          --define tensorflow_target=p1 \
          --remote_cache=grpc://bb-storage.inceptio.tech:8980 \
          --remote_instance_name=cybertron \
          //simulation/launch:simulation_main --config=sim
        bazel build \
          --define tensorflow_target=p1 \
          --remote_cache=grpc://bb-storage.inceptio.tech:8980 \
          --remote_instance_name=cybertron \
          //interfaces/tools/gems/pb_converter:all

        cd ..
        tar --exclude='cybertron/bazel-cybertron' \
          --exclude='cybertron/bazel-out' \
          -h -cf cybertron-sim-p1.tar cybertron
        ls -alh cybertron-sim-p1.tar
        cp cybertron-sim-p1.tar /data
      volumeMounts:
      - mountPath: /root/.jfrog/jfrog-cli.conf
        name: credentials-volume
        subPath: jfrog-cli.conf
      - mountPath: /data
        name: artifacts-volume-claim
      - mountPath: /root/.ssh/id_rsa
        name: gitlab-ssh-key-volume
        subPath: id_rsa
  - arguments: {}
    inputs: {}
    metadata: {}
    name: fetch-regression-repository
    outputs: {}
    retryStrategy:
      limit: 2
      retryPolicy: Always
    script:
      command:
      - bash
      env:
      - name: REGRESSION_COMMIT_ID
        value: '{{workflow.parameters.regression_commit_id}}'
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/sde_cybertron_release:2020_07_03_c12745f7
      name: ""
      resources: {}
      source: |
        set -ex

        mkdir -p /data/regression && cd /data/regression
        git init
        git remote add origin https://gitlab:$GITLAB_TOKEN@gitlab.rd.inceptioglobal.ai/autonomy/regression.git
        echo "$REGRESSION_COMMIT_ID"
        git fetch --depth=1 origin $REGRESSION_COMMIT_ID
        git reset --hard FETCH_HEAD
      volumeMounts:
      - mountPath: /data
        name: artifacts-volume-claim
  - arguments: {}
    inputs: {}
    metadata: {}
    name: fetch-test-group
    outputs: {}
    script:
      command:
      - bash
      env:
      - name: TEST_CASE_GROUP
        value: '{{workflow.parameters.test_case_group}}'
      - name: REGRESSION_PATH
        value: /data/regression
      image: everpeace/curl-jq
      name: ""
      resources: {}
      source: |
        set -e

        trim_prefix_suffix_slash(){ sed -e 's/^\///' -e 's/\/$//'; }
        cd $REGRESSION_PATH
        mkdir -p /data/test_cases_result/
        ls -F -d ${TEST_CASE_GROUP}/* \
          | grep '/$' | trim_prefix_suffix_slash > /data/test_cases_result/tmp_testcases
        echo '[]' > /data/test_cases_result/test_case_result.json
        for line in `cat /data/test_cases_result/tmp_testcases`
        do
            if cat $line/gems_task_v2.pb.txt | grep "disable" &> /dev/null; then
                continue
            fi
            gpu_requirement=$(cat $line/gems_task_v2.pb.txt | grep num_gpu_required | sed -e 's/ //' -e 's/num_gpu_required://')
            if [ -z "$gpu_requirement" ]; then
              gpu_requirement="0"
            fi
            truck_sim="$(cat $line/gems_task_v2.pb.txt | grep 'use_trucksim: true' || true )"
            if [ -z "$truck_sim" ]; then
              truck_sim="false"
            else
              truck_sim="true"
            fi
            echo "[{ \"gpu_requirement\": \"$gpu_requirement\", \"test_case_name\":\"$line\", \"truck_sim\":\"$truck_sim\" }]" \
              | jq ".+= $(cat /data/test_cases_result/test_case_result.json)" > /data/test_cases_result/test_case_result.json
        done
        cat /data/test_cases_result/test_case_result.json | sed -e 's/ //' -e 's/\n//'
      volumeMounts:
      - mountPath: /data
        name: artifacts-volume-claim
  - arguments: {}
    inputs:
      parameters:
      - name: test_case_name
      - name: gpu_requirement
      - name: truck_sim
    metadata: {}
    name: regression
    outputs: {}
    steps:
    - - arguments:
          parameters:
          - name: test_case_name
            value: '{{inputs.parameters.test_case_name}}'
        name: regenerate-step
        template: regenerate
        when: '{{inputs.parameters.gpu_requirement}} == 0 && {{inputs.parameters.truck_sim}}
          != true'
      - arguments:
          parameters:
          - name: test_case_name
            value: '{{inputs.parameters.test_case_name}}'
        name: regenerate-trucksim-step
        template: regenerate-trucksim
        when: '{{inputs.parameters.gpu_requirement}} == 0 && {{inputs.parameters.truck_sim}}
          == true'
      - arguments:
          parameters:
          - name: test_case_name
            value: '{{inputs.parameters.test_case_name}}'
        name: regenerate-gpu-step
        template: regenerate-gpu
        when: '{{inputs.parameters.gpu_requirement}} != 0 && {{inputs.parameters.truck_sim}}
          != true'
      - arguments:
          parameters:
          - name: test_case_name
            value: '{{inputs.parameters.test_case_name}}'
        name: regenerate-trucksim-gpu-step
        template: regenerate-trucksim-gpu
        when: '{{inputs.parameters.gpu_requirement}} != 0 && {{inputs.parameters.truck_sim}}
          == true'
    - - arguments:
          parameters:
          - name: test_case_name
            value: '{{inputs.parameters.test_case_name}}'
        name: evaluate-step
        template: evaluate
  - arguments: {}
    inputs:
      parameters:
      - name: test_case_name
    metadata:
      annotations:
        k8s.aliyun.com/eci-image-cache: "true"
        k8s.aliyun.com/eci-use-specs: ecs.ic5.large
    name: regenerate
    nodeSelector:
      type: virtual-kubelet
    outputs: {}
    retryStrategy:
      limit: 2
      retryPolicy: Always
    script:
      command:
      - bash
      env:
      - name: TEST_CASE_NAME
        value: '{{inputs.parameters.test_case_name}}'
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: REGENERATED_BUCKET
        valueFrom:
          configMapKeyRef:
            key: moment_regenerated_oss_bucket
            name: cluster-configs
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      - name: CAS_TOKEN
        valueFrom:
          secretKeyRef:
            key: token
            name: cas-service-token
      - name: ENV
        valueFrom:
          configMapKeyRef:
            key: env
            name: cluster-configs
      - name: CAS_WEB_HOST
        valueFrom:
          configMapKeyRef:
            key: cas_web_host
            name: cluster-configs
      - name: CHT_GATEWAY_HOST
        valueFrom:
          configMapKeyRef:
            key: cht_gateway_host
            name: cluster-configs
      - name: RT_CN_WEB_URL
        valueFrom:
          configMapKeyRef:
            key: rt_cn_web_url
            name: cluster-configs
      - name: RT_US_WEB_URL
        valueFrom:
          configMapKeyRef:
            key: rt_us_web_url
            name: cluster-configs
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/infra_regenerate-gems:v20210105-5044b06
      name: ""
      resources: {}
      source: |
        set -ex

        T0=$(date +%s)
        cat_gems_task_pbtxt(){ cat /data/regression/$TEST_CASE_NAME/gems_task_v2.pb.txt; }
        trim_prefix_suffix_quote(){ sed -e 's/^"//' -e 's/"$//'; }
        LAUNCH_NAME=$(cat_gems_task_pbtxt | grep launch_config_rel_path | awk '{print$2}' | trim_prefix_suffix_quote)
        VEHICLE_ID=$(cat_gems_task_pbtxt | grep vehicle_id | awk '{print$2}' | trim_prefix_suffix_quote)
        MAP=$(cat_gems_task_pbtxt | grep map_version | awk '{print$2}' | trim_prefix_suffix_quote)
        TIMEOUT=$(cat_gems_task_pbtxt | grep timeout_sec | awk '{print$2}')
        MOMENT_ID=$(cat_gems_task_pbtxt | grep moment_id | awk '{print$2}' | trim_prefix_suffix_quote)
        SIM_MODE=$(cat_gems_task_pbtxt | grep "sim_mode:" | awk '{print$2}')
        REC_REL_DIR=$(cat_gems_task_pbtxt | grep "rec_rel_dir:" | awk '{print$2}' | trim_prefix_suffix_quote)

        if [[ $SIM_MODE == "IDEAL" || $SIM_MODE == "WORLD" ]]; then
          CYBERTRON_TAR=cybertron-sim.tar
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          CYBERTRON_TAR=cybertron-pkg.tar
        else
          echo "Unsupported sim_mode $SIM_MODE"
          exit 1
        fi

        T1=$(date +%s)
        echo "Parse parameters took $((T1-T0)) seconds"
        cp /data/$CYBERTRON_TAR /root/cybertron.tar && cd /root && tar -xf cybertron.tar && rm cybertron.tar && cd cybertron

        T2=$(date +%s)
        echo "Copy tar took $((T2-T1)) seconds"

        if [[ ! -z "$MOMENT_ID" ]]; then
          dataset client \
            dl moments/$MOMENT_ID \
            --token=$CAS_TOKEN \
            --target=dataset-server.chariot:50050 \
            --dataset-process-target=dataset-server.chariot:50051 \
            --target-path=/root/moments/$REC_REL_DIR \
            --idl-version={{workflow.outputs.parameters.global_idl_version}}
        fi

        T3=$(date +%s)
        echo "Downloading data took $((T3-T2)) seconds"

        # Detect in kube pods
        sed -i "s/docker/kubepods/#g" /root/cybertron/irs/process_manager/setup/ubuntu_x86_64/ipm_u_setup.sh
        # Set chariot env
        #export CHARIOT_ENV=$(echo $ENV | tr 'a-z' 'A-Z')
        export CHARIOT_SITE=cn
        cat <<EOF >> chariot_config.json
        {
          "CAS_URL": "$CAS_WEB_HOST",
          "CHARIOT_SERVER_URL": "$CHT_GATEWAY_HOST",
          "artifacts": {
            "us": {
              "endpoint": "$RT_US_WEB_URL",
              "repo_type": "local"
            },
            "cn": {
              "endpoint": "$RT_CN_WEB_URL",
              "repo_type": "remote"
            }
          }
        }
        EOF
        export CHARIOT_ENV_FILE=chariot_config.json

        chariot -t $CAS_TOKEN -o json config --set disable_auto_update=true
        if [[ $SIM_MODE == "IDEAL" || $SIM_MODE == "WORLD" ]]; then
          if [[ $SIM_MODE == "IDEAL" ]]; then
              # gems-sim use the rec pattern
              mv /root/moments/$TEST_CASE_NAME/${MOMENT_ID:0:2}/$MOMENT_ID/* /root/moments/$TEST_CASE_NAME
          fi

          # gems-sim not untar the packages
          yq delete -i bundle.yaml packages.cybertron.build.tars
          yq delete -i bundle.yaml packages.cybertron.deployScript
          cat bundle.yaml
          chariot -t $CAS_TOKEN deploy -f bundle.yaml
          T4=$(date +%s)
          echo "Chariot deploy took $((T4-T3)) seconds"

          if [ -f /data/regression/$TEST_CASE_NAME/input_command.pb.txt ]; then
              INPUT_COMMAND_LIST_PARAM="-l /data/regression/$TEST_CASE_NAME/input_command.pb.txt"
          fi

          echo "MOMENT_ID: $MOMENT_ID"
          LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/compat:/usr/lib/x86_64-linux-gnu/:$LD_LIBRARY_PATH
          cd /root/cybertron/
          timeout $TIMEOUT bazel-bin/simulation/launch/simulation_main --config_path /data/regression/$TEST_CASE_NAME/gems_task_v2.pb.txt \
            --no_ui \
            --data_path /root/moments/ \
            --output_path /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID
          cd -
          T5=$(date +%s)
          echo "Gems sim took $((T5-T4)) seconds"
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          pip3 install irs/launch/launch irs/launch/launch_ros irs/launch/launch_yaml irs/launch/test_launch_ros irs/launch/osrf_pycommon
          chariot -t $CAS_TOKEN replay \
            --no-check \
            --launch \
            --name=$LAUNCH_NAME \
            --file=bundle.yaml \
            --map=$MAP \
            --vehicle-id=$VEHICLE_ID \
            --timeout=$TIMEOUT \
            --replay-rec-path=/root/moments/$REC_REL_DIR/${MOMENT_ID:0:2}/$MOMENT_ID \
            --output=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID
          T5=$(date +%s)
          echo "time sync replayer took $((T5-T3)) seconds"
        fi

        gems-rec-convert \
          --exec_path=/root/cybertron \
          --input_path=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/rec \
          --output_path=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/pb

        T6=$(date +%s)
        echo "Converter took $((T6-T5)) seconds"

        mkdir -p /data/$TEST_CASE_NAME/regenerated/pb
        cp -a /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/pb /data/$TEST_CASE_NAME/regenerated
        if [[ $SIM_MODE == "IDEAL" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/${MOMENT_ID:0:2}/$MOMENT_ID/$SIM_MODE/$TEST_CASE_NAME/$CYBERTRON_COMMIT_ID"
        elif [[ $SIM_MODE == "WORLD" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/$SIM_MODE/$TEST_CASE_NAME/$CYBERTRON_COMMIT_ID"
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/${MOMENT_ID:0:2}/$MOMENT_ID/$LAUNCH_NAME/$CYBERTRON_COMMIT_ID"
        fi
        # calculate disk storage
        du -hd 1 /root
        ls -l /root/cybertron || true
        du -hd 1 /root/cybertron
        ossutil cp --config-file=/root/.ossutilconfig -r -f \
          /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID \
          $OSS_DEST_PATH

        T7=$(date +%s)
        echo "Oss copy took $((T6-T5)) seconds"
      volumeMounts:
      - mountPath: /root/.jfrog/jfrog-cli.conf
        name: credentials-volume
        subPath: jfrog-cli.conf
      - mountPath: /root/.ossutilconfig
        name: credentials-volume
        subPath: .ossutilconfig
      - mountPath: /data
        name: artifacts-volume-claim
    tolerations:
    - key: virtual-kubelet.io/provider
      operator: Exists
  - arguments: {}
    inputs:
      parameters:
      - name: test_case_name
    metadata:
      annotations:
        k8s.aliyun.com/eci-image-cache: "true"
        k8s.aliyun.com/eci-use-specs: ecs.ic5.large
    name: regenerate-trucksim
    nodeSelector:
      type: virtual-kubelet
    outputs: {}
    retryStrategy:
      limit: 2
      retryPolicy: Always
    script:
      command:
      - bash
      env:
      - name: TEST_CASE_NAME
        value: '{{inputs.parameters.test_case_name}}'
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: REGENERATED_BUCKET
        valueFrom:
          configMapKeyRef:
            key: moment_regenerated_oss_bucket
            name: cluster-configs
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      - name: CAS_TOKEN
        valueFrom:
          secretKeyRef:
            key: token
            name: cas-service-token
      - name: ENV
        valueFrom:
          configMapKeyRef:
            key: env
            name: cluster-configs
      - name: CAS_WEB_HOST
        valueFrom:
          configMapKeyRef:
            key: cas_web_host
            name: cluster-configs
      - name: CHT_GATEWAY_HOST
        valueFrom:
          configMapKeyRef:
            key: cht_gateway_host
            name: cluster-configs
      - name: RT_CN_WEB_URL
        valueFrom:
          configMapKeyRef:
            key: rt_cn_web_url
            name: cluster-configs
      - name: RT_US_WEB_URL
        valueFrom:
          configMapKeyRef:
            key: rt_us_web_url
            name: cluster-configs
      - name: TS_LICENSE_SERVER
        valueFrom:
          configMapKeyRef:
            key: ts_license_server
            name: cluster-configs
      - name: TS_LICENSE_TYPE
        valueFrom:
          configMapKeyRef:
            key: ts_license_type
            name: cluster-configs
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/infra_regenerate-gems:v20210105-5044b06
      name: ""
      resources: {}
      source: |
        set -ex
        exit 0

        T0=$(date +%s)
        cat_gems_task_pbtxt(){ cat /data/regression/$TEST_CASE_NAME/gems_task_v2.pb.txt; }
        trim_prefix_suffix_quote(){ sed -e 's/^"//' -e 's/"$//'; }
        LAUNCH_NAME=$(cat_gems_task_pbtxt | grep launch_config_rel_path | awk '{print$2}' | trim_prefix_suffix_quote)
        VEHICLE_ID=$(cat_gems_task_pbtxt | grep vehicle_id | awk '{print$2}' | trim_prefix_suffix_quote)
        MAP=$(cat_gems_task_pbtxt | grep map_version | awk '{print$2}' | trim_prefix_suffix_quote)
        TIMEOUT=$(cat_gems_task_pbtxt | grep timeout_sec | awk '{print$2}')
        MOMENT_ID=$(cat_gems_task_pbtxt | grep moment_id | awk '{print$2}' | trim_prefix_suffix_quote)
        SIM_MODE=$(cat_gems_task_pbtxt | grep "sim_mode:" | awk '{print$2}')
        REC_REL_DIR=$(cat_gems_task_pbtxt | grep "rec_rel_dir:" | awk '{print$2}' | trim_prefix_suffix_quote)

        T1=$(date +%s)
        echo "Parse parameters took $((T1-T0)) seconds"

        if [[ $SIM_MODE == "IDEAL" || $SIM_MODE == "WORLD" ]]; then
          CYBERTRON_TAR=cybertron-sim-p1.tar
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          CYBERTRON_TAR=cybertron-pkg.tar
        else
          echo "Unsupported sim_mode $SIM_MODE"
          exit 1
        fi
        cp /data/$CYBERTRON_TAR /root/cybertron.tar && cd /root && tar -xf cybertron.tar && rm cybertron.tar && cd cybertron

        T2=$(date +%s)
        echo "copy tar took $((T2-T1)) seconds"

        if [[ ! -z "$MOMENT_ID" ]]; then
          dataset client \
            dl moments/$MOMENT_ID \
            --token=$CAS_TOKEN \
            --target=dns:///dataset-server-headless.chariot:50050 \
            --dataset-process-target=dns:///dataset-server-headless.chariot:50051 \
            --target-path=/root/moments/$REC_REL_DIR \
            --idl-version={{workflow.outputs.parameters.global_idl_version}}
        fi

        T3=$(date +%s)
        echo "Download data took $((T3-T2)) seconds"

        # Detect in kube pods
        sed -i "s/docker/kubepods/#g" /root/cybertron/irs/process_manager/setup/ubuntu_x86_64/ipm_u_setup.sh
        # Set chariot env
        #export CHARIOT_ENV=$(echo $ENV | tr 'a-z' 'A-Z')
        export CHARIOT_SITE=cn
        cat <<EOF >> chariot_config.json
        {
          "CAS_URL": "$CAS_WEB_HOST",
          "CHARIOT_SERVER_URL": "$CHT_GATEWAY_HOST",
          "artifacts": {
            "us": {
              "endpoint": "$RT_US_WEB_URL",
              "repo_type": "local"
            },
            "cn": {
              "endpoint": "$RT_CN_WEB_URL",
              "repo_type": "remote"
            }
          }
        }
        EOF
        export CHARIOT_ENV_FILE=chariot_config.json


        chariot -t $CAS_TOKEN -o json config --set disable_auto_update=true
        if [[ $SIM_MODE == "IDEAL" || $SIM_MODE == "WORLD" ]]; then
          if [[ $SIM_MODE == "IDEAL" ]]; then
              # gems-sim use the rec pattern
              mv /root/moments/$REC_REL_DIR/${MOMENT_ID:0:2}/$MOMENT_ID/* /root/moments/$REC_REL_DIR
          fi

          # gems-sim not untar the packages
          yq delete -i bundle.yaml packages.cybertron.build.tars
          yq delete -i bundle.yaml packages.cybertron.deployScript
          cat bundle.yaml
          chariot -t $CAS_TOKEN deploy -f bundle.yaml

          T4=$(date +%s)
          echo "chariot deploy took $((T4-T3)) seconds"

          if [ -f /data/regression/$TEST_CASE_NAME/input_command.pb.txt ]; then
              INPUT_COMMAND_LIST_PARAM="-l /data/regression/$TEST_CASE_NAME/input_command.pb.txt"
          fi

          #FIXME: force using the 395 calibration for trucksim, for long term will using the gems_task setting or maintain a realtion map between vehicle model and platform(vehicle data/calibration)
          ls /root/cybertron/config/calibration/CN_395
          export IRS_CALIBRATION_PATH=/root/cybertron/config/calibration/CN_395

          echo "MOMENT_ID: $MOMENT_ID"
          LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/compat:/usr/lib/x86_64-linux-gnu/:$LD_LIBRARY_PATH
          cd /root/cybertron/
          # Run truck sim
          # trucksim found this this .so in absolute path
          mkdir -p bazel-cybertron/../../external/trucksim_api/
          cp /opt/trucksim_2020.0/lib64/libtrucksim.so.2020.0 bazel-cybertron/../../external/trucksim_api/libtrucksim.so.2020.0
          T5=$(date +%s)
          echo "Prepare trucksim took $((T5-T4)) seconds"

          timeout $TIMEOUT bazel-bin/simulation/launch/simulation_main --config_path /data/regression/$TEST_CASE_NAME/gems_task_v2.pb.txt \
            --no_ui \
            --data_path /root/moments/ \
            --output_path /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID
          cd -
          T6=$(date +%s)
          echo "Gems-sim took $((T6-T5)) seconds"
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          pip3 install irs/launch/launch irs/launch/launch_ros irs/launch/launch_yaml irs/launch/test_launch_ros irs/launch/osrf_pycommon
          chariot -t $CAS_TOKEN replay \
            --no-check \
            --launch \
            --name=$LAUNCH_NAME \
            --file=bundle.yaml \
            --map=$MAP \
            --vehicle-id=$VEHICLE_ID \
            --timeout=$TIMEOUT \
            --replay-rec-path=/root/moments/$REC_REL_DIR/${MOMENT_ID:0:2}/$MOMENT_ID \
            --output=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID
          T6=$(date +%s)
          echo "chariot replay took $((T6-T4)) seconds"
        fi

        cat /tmp/sim_log.txt || true

        gems-rec-convert \
          --exec_path=/root/cybertron \
          --input_path=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/rec \
          --output_path=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/pb

        T7=$(date +%s)
        echo "converter took $((T7-T6)) seconds"

        mkdir -p /data/$TEST_CASE_NAME/regenerated/pb
        cp -a /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/pb /data/$TEST_CASE_NAME/regenerated
        ls -l /data/$TEST_CASE_NAME/regenerated
        if [[ $SIM_MODE == "IDEAL" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/${MOMENT_ID:0:2}/$MOMENT_ID/$SIM_MODE/$TEST_CASE_NAME/$CYBERTRON_COMMIT_ID"
        elif [[ $SIM_MODE == "WORLD" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/$SIM_MODE/$TEST_CASE_NAME/$CYBERTRON_COMMIT_ID"
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/${MOMENT_ID:0:2}/$MOMENT_ID/$LAUNCH_NAME/$CYBERTRON_COMMIT_ID"
        fi
        # calculate disk storage
        du -hd 1 /root
        ls -l /root/cybertron || true
        du -hd 1 /root/cybertron
        ossutil cp --config-file=/root/.ossutilconfig -r -f \
          /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID \
          $OSS_DEST_PATH

        T8=$(date +%s)
        echo "oss copy took $((T8-T7)) seconds"
      volumeMounts:
      - mountPath: /root/.jfrog/jfrog-cli.conf
        name: credentials-volume
        subPath: jfrog-cli.conf
      - mountPath: /root/.ossutilconfig
        name: credentials-volume
        subPath: .ossutilconfig
      - mountPath: /data
        name: artifacts-volume-claim
    synchronization:
      semaphore:
        configMapKeyRef:
          key: trucksim_regenarate_template
          name: gems-sync
    tolerations:
    - key: virtual-kubelet.io/provider
      operator: Exists
  - arguments: {}
    inputs:
      parameters:
      - name: test_case_name
    metadata:
      annotations:
        k8s.aliyun.com/eci-image-cache: "true"
        k8s.aliyun.com/eci-use-specs: ecs.gn6i-c8g1.2xlarge
    name: regenerate-gpu
    nodeSelector:
      type: virtual-kubelet
    outputs: {}
    retryStrategy:
      backoff:
        duration: 1m
        factor: 1
        maxDuration: 30m
      limit: 2
      retryPolicy: Always
    script:
      command:
      - bash
      env:
      - name: TEST_CASE_NAME
        value: '{{inputs.parameters.test_case_name}}'
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: REGENERATED_BUCKET
        valueFrom:
          configMapKeyRef:
            key: moment_regenerated_oss_bucket
            name: cluster-configs
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      - name: CAS_TOKEN
        valueFrom:
          secretKeyRef:
            key: token
            name: cas-service-token
      - name: ENV
        valueFrom:
          configMapKeyRef:
            key: env
            name: cluster-configs
      - name: CAS_WEB_HOST
        valueFrom:
          configMapKeyRef:
            key: cas_web_host
            name: cluster-configs
      - name: CHT_GATEWAY_HOST
        valueFrom:
          configMapKeyRef:
            key: cht_gateway_host
            name: cluster-configs
      - name: RT_CN_WEB_URL
        valueFrom:
          configMapKeyRef:
            key: rt_cn_web_url
            name: cluster-configs
      - name: RT_US_WEB_URL
        valueFrom:
          configMapKeyRef:
            key: rt_us_web_url
            name: cluster-configs
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/infra_regenerate-gems:v20210105-5044b06
      name: ""
      resources:
        limits:
          nvidia.com/gpu: "1"
      source: |
        set -ex

        cat_gems_task_pbtxt(){ cat /data/regression/$TEST_CASE_NAME/gems_task_v2.pb.txt; }
        trim_prefix_suffix_quote(){ sed -e 's/^"//' -e 's/"$//'; }
        LAUNCH_NAME=$(cat_gems_task_pbtxt | grep launch_config_rel_path | awk '{print$2}' | trim_prefix_suffix_quote)
        VEHICLE_ID=$(cat_gems_task_pbtxt | grep vehicle_id | awk '{print$2}' | trim_prefix_suffix_quote)
        MAP=$(cat_gems_task_pbtxt | grep map_version | awk '{print$2}' | trim_prefix_suffix_quote)
        TIMEOUT=$(cat_gems_task_pbtxt | grep timeout_sec | awk '{print$2}')
        MOMENT_ID=$(cat_gems_task_pbtxt | grep moment_id | awk '{print$2}' | trim_prefix_suffix_quote)
        SIM_MODE=$(cat_gems_task_pbtxt | grep sim_mode | awk '{print$2}')
        REC_REL_DIR=$(cat_gems_task_pbtxt | grep "rec_rel_dir:" | awk '{print$2}' | trim_prefix_suffix_quote)

        if [[ $SIM_MODE == "IDEAL" || $SIM_MODE == "WORLD" ]]; then
          CYBERTRON_TAR=cybertron-sim.tar
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          CYBERTRON_TAR=cybertron-pkg.tar
        else
          echo "Unsupported sim_mode $SIM_MODE"
          exit 1
        fi
        cp /data/$CYBERTRON_TAR /root/cybertron.tar && cd /root && tar -xf cybertron.tar && rm cybertron.tar && cd cybertron

        if [[ ! -z "$MOMENT_ID" ]]; then
          dataset client \
            dl moments/$MOMENT_ID \
            --token=$CAS_TOKEN \
            --target=dns:///dataset-server-headless.chariot:50050 \
            --dataset-process-target=dns:///dataset-server-headless.chariot:50051 \
            --target-path=/root/moments/$REC_REL_DIR \
            --idl-version={{workflow.outputs.parameters.global_idl_version}}
        fi

        # Detect in kube pods
        sed -i "s/docker/kubepods/#g" /root/cybertron/irs/process_manager/setup/ubuntu_x86_64/ipm_u_setup.sh
        # Set chariot env
        # export CHARIOT_ENV=$(echo $ENV | tr 'a-z' 'A-Z')
        export CHARIOT_SITE=cn
        cat <<EOF >> chariot_config.json
        {
          "CAS_URL": "$CAS_WEB_HOST",
          "CHARIOT_SERVER_URL": "$CHT_GATEWAY_HOST",
          "artifacts": {
            "us": {
              "endpoint": "$RT_US_WEB_URL",
              "repo_type": "local"
            },
            "cn": {
              "endpoint": "$RT_CN_WEB_URL",
              "repo_type": "remote"
            }
          }
        }
        EOF
        export CHARIOT_ENV_FILE=chariot_config.json
        chariot -o json config --set disable_auto_update=true
        if [[ $SIM_MODE == "IDEAL" || $SIM_MODE == "WORLD" ]]; then
          if [[ $SIM_MODE == "IDEAL" ]]; then
              # gems-sim use the rec pattern
              mv /root/moments/$REC_REL_DIR/${MOMENT_ID:0:2}/$MOMENT_ID/* /root/moments/$REC_REL_DIR
          fi

          # gems-sim not untar the packages
          yq delete -i bundle.yaml packages.cybertron.build.tars
          yq delete -i bundle.yaml packages.cybertron.deployScript
          cat bundle.yaml
          chariot -t $CAS_TOKEN deploy -f bundle.yaml

          if [ -f /data/regression/$TEST_CASE_NAME/input_command.pb.txt ]; then
              INPUT_COMMAND_LIST_PARAM="-l /data/regression/$TEST_CASE_NAME/input_command.pb.txt"
          fi

          cd /root/cybertron/
          timeout $TIMEOUT bazel-bin/simulation/launch/simulation_main --config_path /data/regression/$TEST_CASE_NAME/gems_task_v2.pb.txt \
            --no_ui \
            --data_path /root/moments/ \
            --output_path /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID
          cd -
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          pip3 install irs/launch/launch irs/launch/launch_ros irs/launch/launch_yaml irs/launch/test_launch_ros irs/launch/osrf_pycommon
          chariot -t $CAS_TOKEN replay \
            --no-check \
            --launch \
            --name=$LAUNCH_NAME \
            --file=bundle.yaml \
            --map=$MAP \
            --vehicle-id=$VEHICLE_ID \
            --timeout=$TIMEOUT \
            --replay-rec-path=/root/moments/$REC_REL_DIR/${MOMENT_ID:0:2}/$MOMENT_ID \
            --output=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID
        fi

        gems-rec-convert \
          --exec_path=/root/cybertron \
          --input_path=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/rec \
          --output_path=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/pb

        mkdir -p /data/$TEST_CASE_NAME/regenerated/pb
        cp -a /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/pb /data/$TEST_CASE_NAME/regenerated
        if [[ $SIM_MODE == "IDEAL" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/${MOMENT_ID:0:2}/$MOMENT_ID/$SIM_MODE/$TEST_CASE_NAME/$CYBERTRON_COMMIT_ID"
        elif [[ $SIM_MODE == "WORLD" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/$SIM_MODE/$TEST_CASE_NAME/$CYBERTRON_COMMIT_ID"
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/${MOMENT_ID:0:2}/$MOMENT_ID/$LAUNCH_NAME/$CYBERTRON_COMMIT_ID"
        fi
        # calculate disk storage
        du -hd 1 /root
        ls -l /root/cybertron || true
        du -hd 1 /root/cybertron
        ossutil cp --config-file=/root/.ossutilconfig -r -f \
          /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID \
          $OSS_DEST_PATH
      volumeMounts:
      - mountPath: /root/.jfrog/jfrog-cli.conf
        name: credentials-volume
        subPath: jfrog-cli.conf
      - mountPath: /root/.ossutilconfig
        name: credentials-volume
        subPath: .ossutilconfig
      - mountPath: /data
        name: artifacts-volume-claim
    tolerations:
    - key: virtual-kubelet.io/provider
      operator: Exists
  - affinity:
      nodeAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - preference:
            matchExpressions:
            - key: gpu_node
              operator: In
              values:
              - "true"
          weight: 100
    arguments: {}
    inputs:
      parameters:
      - name: test_case_name
    metadata:
      annotations:
        k8s.aliyun.com/eci-image-cache: "true"
        k8s.aliyun.com/eci-use-specs: ecs.gn6i-c8g1.2xlarge
    name: regenerate-trucksim-gpu
    outputs: {}
    retryStrategy:
      backoff:
        duration: 1m
        factor: 1
        maxDuration: 30m
      limit: 2
      retryPolicy: Always
    script:
      command:
      - bash
      env:
      - name: TEST_CASE_NAME
        value: '{{inputs.parameters.test_case_name}}'
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: REGENERATED_BUCKET
        valueFrom:
          configMapKeyRef:
            key: moment_regenerated_oss_bucket
            name: cluster-configs
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      - name: CAS_TOKEN
        valueFrom:
          secretKeyRef:
            key: token
            name: cas-service-token
      - name: ENV
        valueFrom:
          configMapKeyRef:
            key: env
            name: cluster-configs
      - name: CAS_WEB_HOST
        valueFrom:
          configMapKeyRef:
            key: cas_web_host
            name: cluster-configs
      - name: CHT_GATEWAY_HOST
        valueFrom:
          configMapKeyRef:
            key: cht_gateway_host
            name: cluster-configs
      - name: RT_CN_WEB_URL
        valueFrom:
          configMapKeyRef:
            key: rt_cn_web_url
            name: cluster-configs
      - name: RT_US_WEB_URL
        valueFrom:
          configMapKeyRef:
            key: rt_us_web_url
            name: cluster-configs
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/infra_regenerate-gems:v20210105-5044b06
      name: ""
      resources:
        limits:
          nvidia.com/gpu: "1"
      source: |
        set -ex

        cat_gems_task_pbtxt(){ cat /data/regression/$TEST_CASE_NAME/gems_task_v2.pb.txt; }
        trim_prefix_suffix_quote(){ sed -e 's/^"//' -e 's/"$//'; }
        LAUNCH_NAME=$(cat_gems_task_pbtxt | grep launch_config_rel_path | awk '{print$2}' | trim_prefix_suffix_quote)
        VEHICLE_ID=$(cat_gems_task_pbtxt | grep vehicle_id | awk '{print$2}' | trim_prefix_suffix_quote)
        MAP=$(cat_gems_task_pbtxt | grep map_version | awk '{print$2}' | trim_prefix_suffix_quote)
        TIMEOUT=$(cat_gems_task_pbtxt | grep timeout_sec | awk '{print$2}')
        MOMENT_ID=$(cat_gems_task_pbtxt | grep moment_id | awk '{print$2}' | trim_prefix_suffix_quote)
        SIM_MODE=$(cat_gems_task_pbtxt | grep sim_mode | awk '{print$2}')
        REC_REL_DIR=$(cat_gems_task_pbtxt | grep "rec_rel_dir:" | awk '{print$2}' | trim_prefix_suffix_quote)

        if [[ $SIM_MODE == "IDEAL" || $SIM_MODE == "WORLD" ]]; then
          CYBERTRON_TAR=cybertron-sim.tar
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          CYBERTRON_TAR=cybertron-pkg.tar
        else
          echo "Unsupported sim_mode $SIM_MODE"
          exit 1
        fi
        cp /data/$CYBERTRON_TAR /root/cybertron.tar && cd /root && tar -xf cybertron.tar && rm cybertron.tar && cd cybertron

        if [[ ! -z "$MOMENT_ID" ]]; then
          dataset client \
            dl moments/$MOMENT_ID \
            --token=$CAS_TOKEN \
            --target=dns:///dataset-server-headless.chariot:50050 \
            --dataset-process-target=dns:///dataset-server-headless.chariot:50051 \
            --target-path=/root/moments/$REC_REL_DIR \
            --idl-version={{workflow.outputs.parameters.global_idl_version}}
        fi

        # Detect in kube pods
        sed -i "s/docker/kubepods/#g" /root/cybertron/irs/process_manager/setup/ubuntu_x86_64/ipm_u_setup.sh
        # Set chariot env
        # export CHARIOT_ENV=$(echo $ENV | tr 'a-z' 'A-Z')
        export CHARIOT_SITE=cn
        cat <<EOF >> chariot_config.json
        {
          "CAS_URL": "$CAS_WEB_HOST",
          "CHARIOT_SERVER_URL": "$CHT_GATEWAY_HOST",
          "artifacts": {
            "us": {
              "endpoint": "$RT_US_WEB_URL",
              "repo_type": "local"
            },
            "cn": {
              "endpoint": "$RT_CN_WEB_URL",
              "repo_type": "remote"
            }
          }
        }
        EOF

        export CHARIOT_ENV_FILE=chariot_config.json
        chariot -t $CAS_TOKEN -o json config --set disable_auto_update=true
        if [[ $SIM_MODE == "IDEAL" || $SIM_MODE == "WORLD" ]]; then
          if [[ $SIM_MODE == "IDEAL" ]]; then
              # gems-sim use the rec pattern
              mv /root/moments/$REC_REL_DIR/${MOMENT_ID:0:2}/$MOMENT_ID/* /root/moments/$REC_REL_DIR
          fi

          # gems-sim not untar the packages
          yq delete -i bundle.yaml packages.cybertron.build.tars
          yq delete -i bundle.yaml packages.cybertron.deployScript
          cat bundle.yaml
          chariot -t $CAS_TOKEN deploy -f bundle.yaml

          if [ -f /data/regression/$TEST_CASE_NAME/input_command.pb.txt ]; then
              INPUT_COMMAND_LIST_PARAM="-l /data/regression/$TEST_CASE_NAME/input_command.pb.txt"
          fi

          #FIXME: force using the 395 calibration for trucksim, for long term will using the gems_task setting or maintain a realtion map between vehicle model and platform(vehicle data/calibration)
          ls /root/cybertron/config/calibration/CN_395
          export IRS_CALIBRATION_PATH=/root/cybertron/config/calibration/CN_395

          cd /root/cybertron/
          # Run truck sim
          # trucksim found this this .so in absolute path
          mkdir -p bazel-cybertron/../../external/trucksim_api/
          cp /opt/trucksim_2020.0/lib64/libtrucksim.so.2020.0 bazel-cybertron/../../external/trucksim_api/libtrucksim.so.2020.0

          timeout $TIMEOUT bazel-bin/simulation/launch/simulation_main --config_path /data/regression/$TEST_CASE_NAME/gems_task_v2.pb.txt \
            --no_ui \
            --data_path /root/moments/ \
            --output_path /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID
          cd -
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          pip3 install irs/launch/launch irs/launch/launch_ros irs/launch/launch_yaml irs/launch/test_launch_ros irs/launch/osrf_pycommon
          chariot -t $CAS_TOKEN replay \
            --no-check \
            --launch \
            --name=$LAUNCH_NAME \
            --file=bundle.yaml \
            --map=$MAP \
            --vehicle-id=$VEHICLE_ID \
            --timeout=$TIMEOUT \
            --replay-rec-path=/root/moments/$TEST_CASE_NAME/${MOMENT_ID:0:2}/$MOMENT_ID \
            --output=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID
        fi

        gems-rec-convert \
          --exec_path=/root/cybertron \
          --input_path=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/rec \
          --output_path=/root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/pb

        mkdir -p /data/$TEST_CASE_NAME/regenerated/pb
        cp -a /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID/pb /data/$TEST_CASE_NAME/regenerated
        if [[ $SIM_MODE == "IDEAL" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/${MOMENT_ID:0:2}/$MOMENT_ID/$SIM_MODE/$TEST_CASE_NAME/$CYBERTRON_COMMIT_ID"
        elif [[ $SIM_MODE == "WORLD" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/$SIM_MODE/$TEST_CASE_NAME/$CYBERTRON_COMMIT_ID"
        elif [[ $SIM_MODE == "TIMELY" ]]; then
          OSS_DEST_PATH="oss://$REGENERATED_BUCKET/${MOMENT_ID:0:2}/$MOMENT_ID/$LAUNCH_NAME/$CYBERTRON_COMMIT_ID"
        fi
        # calculate disk storage
        du -hd 1 /root
        ls -l /root/cybertron || true
        du -hd 1 /root/cybertron
        ossutil cp --config-file=/root/.ossutilconfig -r -f \
          /root/moments-regenerated/${MOMENT_ID:0:2}/$MOMENT_ID \
          $OSS_DEST_PATH
      volumeMounts:
      - mountPath: /root/.jfrog/jfrog-cli.conf
        name: credentials-volume
        subPath: jfrog-cli.conf
      - mountPath: /root/.ossutilconfig
        name: credentials-volume
        subPath: .ossutilconfig
      - mountPath: /data
        name: artifacts-volume-claim
    synchronization:
      semaphore:
        configMapKeyRef:
          key: trucksim_regenarate_template
          name: gems-sync
    tolerations:
    - effect: NoSchedule
      key: gpu_node
      operator: Equal
      value: "true"
  - arguments: {}
    inputs:
      parameters:
      - name: test_case_name
    metadata: {}
    name: evaluate
    outputs: {}
    retryStrategy:
      limit: 2
      retryPolicy: OnError
    script:
      command:
      - bash
      env:
      - name: REGRESSION_COMMIT_ID
        value: '{{workflow.parameters.regression_commit_id}}'
      - name: TEST_CASE_NAME
        value: '{{inputs.parameters.test_case_name}}'
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: REGENERATED_BUCKET
        valueFrom:
          configMapKeyRef:
            key: moment_regenerated_oss_bucket
            name: cluster-configs
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/infra_regenerate-gems:v20210105-5044b06
      name: ""
      resources: {}
      source: |
        set -ex

        # download ground truth path
        GROUND_TRUTH_OSS_PATH=oss://inceptio-test-case-management-dev/$TEST_CASE_NAME/groud_truth/*
        mkdir -p /data/$TEST_CASE_NAME/evaluate/ground_truth
        SIZE=$(ossutil ls --config-file=/root/.ossutilconfig -s \
          $GROUND_TRUTH_OSS_PATH \
          | tail -n 3 | head -n 1 | awk '{print$4}')
        if [[ $SIZE -ne 0 && $SIZE -ne 1 ]]; then
          GROUND_TRUTH_LOCAL=/data/$TEST_CASE_NAME/evaluate/ground_truth
          ossutil cp --config-file=/root/.ossutilconfig -r -f \
          $GROUND_TRUTH_OSS_PATH \
          $GROUND_TRUTH_LOCAL
        fi

        cat_gems_task_pbtxt(){ cat /data/regression/$TEST_CASE_NAME/gems_task_v2.pb.txt; }
        trim_prefix_suffix_quote(){ sed -e 's/^"//' -e 's/"$//'; }
        MOMENT_ID=$(cat_gems_task_pbtxt | grep moment_id | awk '{print$2}' | trim_prefix_suffix_quote)
        MAP=$(cat_gems_task_pbtxt | grep map_version | awk '{print$2}' | trim_prefix_suffix_quote)
        TOPICS=$(cat_gems_task_pbtxt | grep evaluation_topics | awk '{print$2}' | trim_prefix_suffix_quote )
        SIM_MODE=$(cat_gems_task_pbtxt | grep "sim_mode:" | awk '{print$2}')
        EVALUATE_FILE_NAME=$(echo ${TEST_CASE_NAME} | sed 's/\//_/g')

        ls /data/
        if [[ $SIM_MODE == "IDEAL" ]]; then
            EVALUATE_OUTPUT_PATH=/data/evaluate_result/${MOMENT_ID}.json
            OSS_DEST_PATH="oss://$REGENERATED_BUCKET/evaluate_result/${MOMENT_ID:0:2}/$MOMENT_ID/$SIM_MODE/$TEST_CASE_NAME/$CYBERTRON_COMMIT_ID/${MOMENT_ID}.json"
        elif [[ $SIM_MODE == "WORLD" ]]; then
            EVALUATE_OUTPUT_PATH=/data/evaluate_result/${EVALUATE_FILE_NAME}.json
            OSS_DEST_PATH="oss://$REGENERATED_BUCKET/evaluate_result/$SIM_MODE/$TEST_CASE_NAME/${TEST_CASE_NAME}.json"
        elif [[ $SIM_MODE == "TIMELY" ]]; then
            EVALUATE_OUTPUT_PATH=/data/evaluate_result/${MOMENT_ID}.json
            OSS_DEST_PATH="oss://$REGENERATED_BUCKET/evaluate_result/${MOMENT_ID:0:2}/$MOMENT_ID/$LAUNCH_NAME/$CYBERTRON_COMMIT_ID/${MOMENT_ID}.json"
        fi

        mkdir -p ${EVALUATE_OUTPUT_PATH%/*}

        cd /data/regression/$TEST_CASE_NAME
        evaluate \
          --lep=local_transformation \
          --srp=/data/$TEST_CASE_NAME/regenerated/pb \
          --topics $TOPICS \
          --output=$EVALUATE_OUTPUT_PATH \
          --map=$MAP \
          --moment=$MOMENT_ID \
          --test-name=$TEST_CASE_NAME \
          --gtp "$GROUND_TRUTH_LOCAL" | tee evaluate.log
        if cat evaluate.log | grep "Evaluation Result:" -A 1 | grep "FAIL"; then
          EXIT_CODE=1
        else
          EXIT_CODE=0
        fi


        ossutil cp --config-file=/root/.ossutilconfig -r -f \
          $EVALUATE_OUTPUT_PATH \
          $OSS_DEST_PATH

        exit $EXIT_CODE
      volumeMounts:
      - mountPath: /data
        name: artifacts-volume-claim
      - mountPath: /root/.ossutilconfig
        name: credentials-volume
        subPath: .ossutilconfig
  - arguments: {}
    inputs: {}
    metadata: {}
    name: aggregation
    outputs: {}
    script:
      command:
      - bash
      env:
      - name: REGRESSION_COMMIT_ID
        value: '{{workflow.parameters.regression_commit_id}}'
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: REGENERATED_BUCKET
        valueFrom:
          configMapKeyRef:
            key: moment_regenerated_oss_bucket
            name: cluster-configs
      - name: GITLAB_TOKEN
        valueFrom:
          secretKeyRef:
            key: gitlab-read-repository-token
            name: argo-hubble-secrets
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/infra_regenerate-gems:v20210105-5044b06
      name: ""
      resources: {}
      source: |
        set -x

        AGGREGATION_PATH=/data/aggregation_result
        mkdir -p $AGGREGATION_PATH

        ls -l $AGGREGATION_PATH
        ls -lR /data/evaluate_result
        aggregate \
          --input=/data/evaluate_result \
          --output=$AGGREGATION_PATH/aggregation_result.json
        RES=$?

        ls -l $AGGREGATION_PATH/aggregation_result.json

        OSS_DEST_PATH="oss://$REGENERATED_BUCKET/aggregation_result/$CYBERTRON_COMMIT_ID/$REGRESSION_COMMIT_ID"

        ossutil cp --config-file=/root/.ossutilconfig -r -f \
          $AGGREGATION_PATH \
          $OSS_DEST_PATH

        exit $RES
      volumeMounts:
      - mountPath: /data
        name: artifacts-volume-claim
      - mountPath: /root/.ossutilconfig
        name: credentials-volume
        subPath: .ossutilconfig
  - arguments: {}
    inputs:
      parameters:
      - name: cybertron_commit_timestamp
        value: '{{workflow.outputs.parameters.global_cybertron_commit_timestamp}}'
      - name: cybertron_commit_author
        value: '{{workflow.outputs.parameters.global_cybertron_commit_author}}'
    metadata: {}
    name: ingest
    outputs: {}
    script:
      command:
      - bash
      env:
      - name: CYBERTRON_COMMIT_ID
        value: '{{workflow.parameters.cybertron_commit_id}}'
      - name: CYBERTRON_BRANCH
        value: '{{workflow.parameters.cybertron_branch}}'
      - name: CYBERTRON_COMMIT_TIMESTAMP
        value: '{{inputs.parameters.cybertron_commit_timestamp}}'
      - name: CYBERTRON_COMMIT_AUTHOR
        value: '{{inputs.parameters.cybertron_commit_author}}'
      - name: WORKFLOW
        value: '{{workflow.name}}'
      - name: REGRESSION_COMMIT_ID
        value: '{{workflow.parameters.regression_commit_id}}'
      - name: REGENERATED_BUCKET
        valueFrom:
          configMapKeyRef:
            key: moment_regenerated_oss_bucket
            name: cluster-configs
      image: registry-vpc.cn-hangzhou.aliyuncs.com/inceptio/infra_metrics:v20201218-ac92e3a
      name: ""
      resources: {}
      source: |
        set -x

        CYBERTRON_COMMIT_TIMESTAMP=$(echo $CYBERTRON_COMMIT_TIMESTAMP | sed -e 's/^"//' -e 's/"$//')
        OSS_DEST_PATH="oss://$REGENERATED_BUCKET/aggregation_result/$CYBERTRON_COMMIT_ID/$REGRESSION_COMMIT_ID"
        metrics ingest gems \
          --cybertron-commit-id=$CYBERTRON_COMMIT_ID \
          --cybertron-branch=$CYBERTRON_BRANCH \
          --cybertron-commit-timestamp=$CYBERTRON_COMMIT_TIMESTAMP \
          --cybertron-commit-author=$CYBERTRON_COMMIT_AUTHOR \
          --workflow=$WORKFLOW \
          --file-url=$OSS_DEST_PATH/aggregation_result.json \
          --merge
      volumeMounts:
      - mountPath: /etc/metrics/mysql-config.yaml
        name: metrics-volume
        subPath: mysql.yaml
      - mountPath: /etc/metrics/oss-config.yaml
        name: credentials-volume
        subPath: config.yaml
      - mountPath: /data
        name: artifacts-volume-claim
  ttlStrategy:
    secondsAfterFailure: 43200
    secondsAfterSuccess: 3600
  volumeClaimTemplates:
  - metadata:
      creationTimestamp: null
      name: artifacts-volume-claim
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: alicloud-flexvolume-nas-sc
    status: {}
  volumes:
  - name: credentials-volume
    secret:
      secretName: argo-hubble-secrets
  - name: gitlab-ssh-key-volume
    secret:
      defaultMode: 384
      secretName: gitlab-ssh-key-secrets
  - name: metrics-volume
    secret:
      secretName: fusion-metrics-secrets
  - emptyDir: {}
    name: tmpdir
status:
  artifactRepositoryRef:
    default: true
  finishedAt: null
  nodes:
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7:
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4031031283
      displayName: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      finishedAt: null
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      phase: Running
      progress: 23/23
      startedAt: "2021-01-13T12:14:45Z"
      templateName: gems-entrypoint
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Steps
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-68571808:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1687177185
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3741392784
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-68571808
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 205
        memory: 205
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-127158400:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1687177185
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3741392784
      displayName: regenerate-trucksim-step(0)
      finishedAt: "2021-01-13T12:17:05Z"
      hostNodeName: virtual-kubelet
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-127158400
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_south_1141
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)[0].regenerate-trucksim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-213961484:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3024624406
      displayName: fetch-regression-repository-step(0)
      finishedAt: "2021-01-13T12:16:19Z"
      hostNodeName: cn-hangzhou.10.128.98.235
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-213961484
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[0].fetch-regression-repository-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 176
        memory: 176
      startedAt: "2021-01-13T12:14:45Z"
      templateName: fetch-regression-repository
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-269414011:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2832051700
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3228849915
      displayName: regenerate-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-269414011
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)[0].regenerate-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 188
        memory: 188
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-275563684:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-722146525
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4035425836
      displayName: regenerate-trucksim-step(0)
      finishedAt: "2021-01-13T12:17:01Z"
      hostNodeName: virtual-kubelet
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-275563684
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1221
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)[0].regenerate-trucksim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-291664601:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1687177185
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-127158400
      displayName: regenerate-trucksim-step
      finishedAt: "2021-01-13T12:17:11Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-291664601
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_south_1141
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)[0].regenerate-trucksim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 206
        memory: 206
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-324195964:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1565592669
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4095064236
      displayName: regenerate-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-324195964
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)[0].regenerate-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 184
        memory: 184
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-406695329:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-722146525
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4035425836
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-406695329
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 205
        memory: 205
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-477025405:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3024624406
      displayName: build-cybertron-pkg-step(0)
      finishedAt: "2021-01-13T12:15:58Z"
      hostNodeName: cn-hangzhou.10.128.98.235
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-477025405
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[0].build-cybertron-pkg-step(0)
      outputs:
        exitCode: "0"
        parameters:
        - globalName: global_cybertron_commit_timestamp
          name: cybertron_commit_timestamp
          value: '"1610430777"'
          valueFrom:
            path: /tmpdir/cybertron_commit_timestamp.txt
        - globalName: global_cybertron_commit_author
          name: cybertron_commit_author
          value: xiaofan.mu@inceptio.ai
          valueFrom:
            path: /tmpdir/cybertron_commit_author.txt
        - globalName: global_idl_version
          name: idl_version
          value: ..
          valueFrom:
            path: /tmpdir/idl_version.txt
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 135
        memory: 135
      startedAt: "2021-01-13T12:14:45Z"
      templateName: build-cybertron-pkg
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-558330273:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3602065704
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2336194607
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-558330273
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 203
        memory: 203
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-592867953:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1565592669
      displayName: evaluate-step(0)
      finishedAt: "2021-01-13T12:20:02Z"
      hostNodeName: cn-hangzhou.10.128.98.235
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-592867953
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_2121
      message: failed with exit code 1
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)[1].evaluate-step(0)
      outputs:
        exitCode: "1"
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 184
        memory: 184
      startedAt: "2021-01-13T12:17:11Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-610566068:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-722146525
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4035425836
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-610566068
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 205
        memory: 205
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-722146525:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4102389217
      displayName: regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)
      finishedAt: "2021-01-13T12:20:22Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-722146525
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1221
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      message: child 'gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3385189450'
        failed
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)
      outboundNodes:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2972558065
      phase: Failed
      progress: 2/2
      resourcesDuration:
        cpu: 206
        memory: 206
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Steps
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-729756668:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-722146525
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4035425836
      displayName: regenerate-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-729756668
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)[0].regenerate-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 205
        memory: 205
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-789122789:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1687177185
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3741392784
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-789122789
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 205
        memory: 205
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-801673974:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2282790969
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2926645237
      displayName: evaluate-step
      finishedAt: "2021-01-13T12:20:12Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-801673974
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1121
      message: failed with exit code 1
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)[1].evaluate-step
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 191
        memory: 191
      startedAt: "2021-01-13T12:17:11Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-829637410:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3551042008
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-829637410
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(9:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1111,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-845459564:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1363669702
      displayName: regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)
      finishedAt: "2021-01-13T12:20:12Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-845459564
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_south_1131
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      message: child 'gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3050946813'
        failed
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)
      outboundNodes:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1433739404
      phase: Failed
      progress: 2/2
      resourcesDuration:
        cpu: 190
        memory: 190
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Steps
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-964527601:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3551042008
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-964527601
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(9:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1111,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1019922320:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3331391928
      displayName: evaluate-step(0)
      finishedAt: "2021-01-13T12:20:17Z"
      hostNodeName: cn-hangzhou.10.128.109.137
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1019922320
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1211
      message: failed with exit code 1
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)[1].evaluate-step(0)
      outputs:
        exitCode: "1"
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 203
        memory: 203
      startedAt: "2021-01-13T12:17:12Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1071215084:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3024624406
      displayName: build-cybertron-sim-p1-step(0)
      finishedAt: "2021-01-13T12:14:50Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1071215084
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[0].build-cybertron-sim-p1-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2021-01-13T12:14:45Z"
      templateName: build-cybertron-sim-p1
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1192303368:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2832051700
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1871873947
      displayName: regenerate-trucksim-step
      finishedAt: "2021-01-13T12:17:11Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1192303368
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_2111
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)[0].regenerate-trucksim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 189
        memory: 189
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1211491917:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1687177185
      displayName: evaluate-step(0)
      finishedAt: "2021-01-13T12:20:17Z"
      hostNodeName: cn-hangzhou.10.128.109.137
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1211491917
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_south_1141
      message: failed with exit code 1
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)[1].evaluate-step(0)
      outputs:
        exitCode: "1"
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 205
        memory: 205
      startedAt: "2021-01-13T12:17:11Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1244575150:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2150878129
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3079194717
      displayName: evaluate-step
      finishedAt: "2021-01-13T12:20:12Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1244575150
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1321
      message: failed with exit code 1
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)[1].evaluate-step
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 184
        memory: 184
      startedAt: "2021-01-13T12:17:11Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1250457496:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2282790969
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-801673974
      displayName: '[1]'
      finishedAt: "2021-01-13T12:20:12Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1250457496
      message: child 'gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-801673974'
        failed
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)[1]
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 191
        memory: 191
      startedAt: "2021-01-13T12:17:11Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1296412131:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-845459564
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3050946813
      displayName: '[1]'
      finishedAt: "2021-01-13T12:20:12Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1296412131
      message: child 'gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3050946813'
        failed
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)[1]
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 190
        memory: 190
      startedAt: "2021-01-13T12:17:11Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1299412752:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-845459564
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3529020979
      displayName: regenerate-trucksim-step
      finishedAt: "2021-01-13T12:17:11Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1299412752
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_south_1131
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)[0].regenerate-trucksim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 190
        memory: 190
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1317420877:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2282790969
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3328738768
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3490648065
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3249870445
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2314528024
      displayName: '[0]'
      finishedAt: "2021-01-13T12:17:11Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1317420877
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)[0]
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 192
        memory: 192
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1337420162:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4098539117
      displayName: fetch-test-group-step
      finishedAt: "2021-01-13T12:16:33Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1337420162
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[1].fetch-test-group-step
      outputs:
        exitCode: "0"
        result: |-
          [
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_south_1141",
             "truck_sim": "true"
           },
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_north_2121",
             "truck_sim": "true"
           },
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_north_1321",
             "truck_sim": "true"
           },
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_north_1221",
             "truck_sim": "true"
           },
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_north_1121",
             "truck_sim": "true"
           },
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_60_south_1131",
             "truck_sim": "true"
           },
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_60_north_2111",
             "truck_sim": "true"
           },
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_60_north_1311",
             "truck_sim": "true"
           },
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_60_north_1211",
             "truck_sim": "true"
           },
           {
             "gpu_requirement": "0",
             "test_case_name": "control_tests/jinan_lka_60_north_1111",
             "truck_sim": "true"
           }
          ]
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 3
        memory: 3
      startedAt: "2021-01-13T12:16:29Z"
      templateName: fetch-test-group
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1344779168:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2150878129
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1244575150
      displayName: '[1]'
      finishedAt: "2021-01-13T12:20:12Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1344779168
      message: child 'gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1244575150'
        failed
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)[1]
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 184
        memory: 184
      startedAt: "2021-01-13T12:17:11Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1363669702:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-845459564
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1734346451
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1299412752
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2530900054
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3611099437
      displayName: '[0]'
      finishedAt: "2021-01-13T12:17:11Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1363669702
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)[0]
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 190
        memory: 190
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1411742549:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2150878129
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2652746280
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3405612553
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1965213429
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3764123472
      displayName: '[0]'
      finishedAt: "2021-01-13T12:17:11Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1411742549
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)[0]
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 184
        memory: 184
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1433739404:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-845459564
      displayName: evaluate-step(0)
      finishedAt: "2021-01-13T12:20:06Z"
      hostNodeName: cn-hangzhou.10.128.98.232
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1433739404
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_south_1131
      message: failed with exit code 1
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)[1].evaluate-step(0)
      outputs:
        exitCode: "1"
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 190
        memory: 190
      startedAt: "2021-01-13T12:17:11Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1565592669:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4162027617
      displayName: regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)
      finishedAt: "2021-01-13T12:20:12Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1565592669
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_2121
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      message: child 'gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1729045194'
        failed
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)
      outboundNodes:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-592867953
      phase: Failed
      progress: 2/2
      resourcesDuration:
        cpu: 184
        memory: 184
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Steps
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1687177185:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2734588549
      displayName: regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)
      finishedAt: "2021-01-13T12:20:22Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1687177185
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_south_1141
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      message: child 'gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2557456158'
        failed
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)
      outboundNodes:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1211491917
      phase: Failed
      progress: 2/2
      resourcesDuration:
        cpu: 206
        memory: 206
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Steps
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1694949156:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1565592669
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4095064236
      displayName: regenerate-trucksim-step(0)
      finishedAt: "2021-01-13T12:17:05Z"
      hostNodeName: virtual-kubelet
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1694949156
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_2121
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)[0].regenerate-trucksim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 0
        memory: 0
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1707048177:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3011694024
      displayName: build-cybertron-sim-step
      finishedAt: "2021-01-13T12:15:05Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1707048177
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[0].build-cybertron-sim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 20/20
      resourcesDuration:
        cpu: 1766
        memory: 1766
      startedAt: "2021-01-13T12:14:45Z"
      templateName: build-cybertron-sim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1729045194:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1565592669
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-592867953
      displayName: evaluate-step
      finishedAt: "2021-01-13T12:20:12Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1729045194
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_2121
      message: failed with exit code 1
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)[1].evaluate-step
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 184
        memory: 184
      startedAt: "2021-01-13T12:17:11Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1734346451:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-845459564
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1296412131
      displayName: regenerate-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1734346451
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)[0].regenerate-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 190
        memory: 190
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1743165858:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3331391928
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3494780215
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3699291180
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2218314754
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4073946193
      displayName: '[0]'
      finishedAt: "2021-01-13T12:17:12Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1743165858
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)[0]
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 204
        memory: 204
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1871873947:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2832051700
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3228849915
      displayName: regenerate-trucksim-step(0)
      finishedAt: "2021-01-13T12:17:06Z"
      hostNodeName: virtual-kubelet
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1871873947
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_2111
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)[0].regenerate-trucksim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1945587890:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3602065704
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2336194607
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1945587890
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 203
        memory: 203
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1965213429:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2150878129
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1344779168
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1965213429
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 184
        memory: 184
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1965535357:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-213961484
      displayName: fetch-regression-repository-step
      finishedAt: "2021-01-13T12:16:29Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1965535357
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[0].fetch-regression-repository-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 20/20
      resourcesDuration:
        cpu: 1938
        memory: 1938
      startedAt: "2021-01-13T12:14:45Z"
      templateName: fetch-regression-repository
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1985714517:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2832051700
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3228849915
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1985714517
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 188
        memory: 188
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2150878129:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1411742549
      displayName: regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)
      finishedAt: "2021-01-13T12:20:12Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2150878129
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1321
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      message: child 'gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1244575150'
        failed
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)
      outboundNodes:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3079194717
      phase: Failed
      progress: 2/2
      resourcesDuration:
        cpu: 184
        memory: 184
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Steps
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2152265761:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1565592669
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4095064236
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2152265761
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 184
        memory: 184
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2183692492:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3551042008
      displayName: regenerate-trucksim-step
      finishedAt: null
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2183692492
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1111
      message: 'Waiting for argo-hubble/ConfigMap/gems-sync/trucksim_regenarate_template
        lock. Lock status: 0/9 '
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(9:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1111,truck_sim:true)[0].regenerate-trucksim-step
      phase: Pending
      startedAt: "2021-01-13T12:16:39Z"
      synchronizationStatus:
        waiting: argo-hubble/ConfigMap/gems-sync/trucksim_regenarate_template
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2218314754:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3331391928
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2749675903
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2218314754
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 203
        memory: 203
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2282790969:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1317420877
      displayName: regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)
      finishedAt: "2021-01-13T12:20:12Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2282790969
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1121
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      message: child 'gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-801673974'
        failed
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)
      outboundNodes:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2926645237
      phase: Failed
      progress: 2/2
      resourcesDuration:
        cpu: 192
        memory: 192
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Steps
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2290121541:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1565592669
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1694949156
      displayName: regenerate-trucksim-step
      finishedAt: "2021-01-13T12:17:11Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2290121541
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_2121
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)[0].regenerate-trucksim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 184
        memory: 184
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2314528024:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2282790969
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1250457496
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2314528024
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 191
        memory: 191
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2336194607:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3602065704
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3287310681
      displayName: '[1]'
      finishedAt: "2021-01-13T12:20:22Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2336194607
      message: child 'gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3287310681'
        failed
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)[1]
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 203
        memory: 203
      startedAt: "2021-01-13T12:17:12Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2361211845:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-722146525
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-275563684
      displayName: regenerate-trucksim-step
      finishedAt: "2021-01-13T12:17:11Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2361211845
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1221
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)[0].regenerate-trucksim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 206
        memory: 206
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2403452178:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3602065704
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2633094439
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3048519676
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1945587890
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-558330273
      displayName: '[0]'
      finishedAt: "2021-01-13T12:17:12Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2403452178
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)[0]
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 204
        memory: 204
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2530900054:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-845459564
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1296412131
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2530900054
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 190
        memory: 190
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2557456158:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1687177185
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1211491917
      displayName: evaluate-step
      finishedAt: "2021-01-13T12:20:22Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2557456158
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_south_1141
      message: failed with exit code 1
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)[1].evaluate-step
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 205
        memory: 205
      startedAt: "2021-01-13T12:17:11Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2633094439:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3602065704
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2336194607
      displayName: regenerate-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2633094439
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)[0].regenerate-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 203
        memory: 203
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2652746280:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2150878129
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1344779168
      displayName: regenerate-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2652746280
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)[0].regenerate-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 184
        memory: 184
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2669395086:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2832051700
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3228849915
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2669395086
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 188
        memory: 188
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2701053655:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3551042008
      displayName: regenerate-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2701053655
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(9:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1111,truck_sim:true)[0].regenerate-step
      phase: Skipped
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2734588549:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1687177185
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4060091032
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-291664601
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-789122789
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-68571808
      displayName: '[0]'
      finishedAt: "2021-01-13T12:17:11Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2734588549
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)[0]
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 206
        memory: 206
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2749675903:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3331391928
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3906825385
      displayName: '[1]'
      finishedAt: "2021-01-13T12:20:22Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2749675903
      message: child 'gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3906825385'
        failed
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)[1]
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 203
        memory: 203
      startedAt: "2021-01-13T12:17:12Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2832051700:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3296107486
      displayName: regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)
      finishedAt: "2021-01-13T12:20:12Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2832051700
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_2111
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      message: child 'gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3955556197'
        failed
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)
      outboundNodes:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3706776644
      phase: Failed
      progress: 2/2
      resourcesDuration:
        cpu: 189
        memory: 189
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Steps
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2926645237:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2282790969
      displayName: evaluate-step(0)
      finishedAt: "2021-01-13T12:20:06Z"
      hostNodeName: cn-hangzhou.10.128.98.232
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2926645237
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1121
      message: failed with exit code 1
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)[1].evaluate-step(0)
      outputs:
        exitCode: "1"
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 191
        memory: 191
      startedAt: "2021-01-13T12:17:11Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2972558065:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-722146525
      displayName: evaluate-step(0)
      finishedAt: "2021-01-13T12:20:17Z"
      hostNodeName: cn-hangzhou.10.128.109.137
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2972558065
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1221
      message: failed with exit code 1
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)[1].evaluate-step(0)
      outputs:
        exitCode: "1"
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 205
        memory: 205
      startedAt: "2021-01-13T12:17:11Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3011694024:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3024624406
      displayName: build-cybertron-sim-step(0)
      finishedAt: "2021-01-13T12:14:54Z"
      hostNodeName: cn-hangzhou.10.128.98.235
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3011694024
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[0].build-cybertron-sim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 4
        memory: 4
      startedAt: "2021-01-13T12:14:45Z"
      templateName: build-cybertron-sim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3024624406:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1337420162
      displayName: '[1]'
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3024624406
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[1]
      phase: Succeeded
      progress: 19/19
      resourcesDuration:
        cpu: 1762
        memory: 1762
      startedAt: "2021-01-13T12:16:29Z"
      templateName: gems-entrypoint
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3048519676:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3602065704
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3782999863
      displayName: regenerate-trucksim-step
      finishedAt: "2021-01-13T12:17:12Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3048519676
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1311
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)[0].regenerate-trucksim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 204
        memory: 204
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3050946813:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-845459564
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1433739404
      displayName: evaluate-step
      finishedAt: "2021-01-13T12:20:12Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3050946813
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_south_1131
      message: failed with exit code 1
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)[1].evaluate-step
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 190
        memory: 190
      startedAt: "2021-01-13T12:17:11Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3052868802:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3551042008
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2701053655
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2183692492
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-829637410
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-964527601
      displayName: '[0]'
      finishedAt: null
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3052868802
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(9:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1111,truck_sim:true)[0]
      phase: Running
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3079194717:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2150878129
      displayName: evaluate-step(0)
      finishedAt: "2021-01-13T12:20:02Z"
      hostNodeName: cn-hangzhou.10.128.98.235
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3079194717
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1321
      message: failed with exit code 1
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)[1].evaluate-step(0)
      outputs:
        exitCode: "1"
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 184
        memory: 184
      startedAt: "2021-01-13T12:17:11Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3228849915:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2832051700
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3955556197
      displayName: '[1]'
      finishedAt: "2021-01-13T12:20:12Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3228849915
      message: child 'gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3955556197'
        failed
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)[1]
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 188
        memory: 188
      startedAt: "2021-01-13T12:17:11Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3249870445:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2282790969
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1250457496
      displayName: regenerate-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3249870445
      message: when '0 != 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)[0].regenerate-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 191
        memory: 191
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3287310681:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3602065704
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3689719808
      displayName: evaluate-step
      finishedAt: "2021-01-13T12:20:22Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3287310681
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1311
      message: failed with exit code 1
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)[1].evaluate-step
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 203
        memory: 203
      startedAt: "2021-01-13T12:17:12Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3296107486:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2832051700
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-269414011
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1192303368
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2669395086
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1985714517
      displayName: '[0]'
      finishedAt: "2021-01-13T12:17:11Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3296107486
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)[0]
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 189
        memory: 189
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3328738768:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2282790969
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1250457496
      displayName: regenerate-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3328738768
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)[0].regenerate-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 191
        memory: 191
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3331391928:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1743165858
      displayName: regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)
      finishedAt: "2021-01-13T12:20:22Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3331391928
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1211
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      message: child 'gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3906825385'
        failed
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)
      outboundNodes:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1019922320
      phase: Failed
      progress: 2/2
      resourcesDuration:
        cpu: 204
        memory: 204
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Steps
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3339094222:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-477025405
      displayName: build-cybertron-pkg-step
      finishedAt: "2021-01-13T12:16:08Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3339094222
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[0].build-cybertron-pkg-step
      outputs:
        exitCode: "0"
        parameters:
        - globalName: global_cybertron_commit_timestamp
          name: cybertron_commit_timestamp
          value: '"1610430777"'
          valueFrom:
            path: /tmpdir/cybertron_commit_timestamp.txt
        - globalName: global_cybertron_commit_author
          name: cybertron_commit_author
          value: xiaofan.mu@inceptio.ai
          valueFrom:
            path: /tmpdir/cybertron_commit_author.txt
        - globalName: global_idl_version
          name: idl_version
          value: ..
          valueFrom:
            path: /tmpdir/idl_version.txt
      phase: Succeeded
      progress: 20/20
      resourcesDuration:
        cpu: 1897
        memory: 1897
      startedAt: "2021-01-13T12:14:45Z"
      templateName: build-cybertron-pkg
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3385189450:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-722146525
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2972558065
      displayName: evaluate-step
      finishedAt: "2021-01-13T12:20:22Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3385189450
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1221
      message: failed with exit code 1
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)[1].evaluate-step
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 205
        memory: 205
      startedAt: "2021-01-13T12:17:11Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3405612553:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2150878129
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4080937584
      displayName: regenerate-trucksim-step
      finishedAt: "2021-01-13T12:17:11Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3405612553
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1321
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)[0].regenerate-trucksim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 184
        memory: 184
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3490648065:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2282790969
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4144182104
      displayName: regenerate-trucksim-step
      finishedAt: "2021-01-13T12:17:11Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3490648065
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1121
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)[0].regenerate-trucksim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 192
        memory: 192
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3494780215:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3331391928
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2749675903
      displayName: regenerate-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3494780215
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)[0].regenerate-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 203
        memory: 203
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3529020979:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-845459564
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1296412131
      displayName: regenerate-trucksim-step(0)
      finishedAt: "2021-01-13T12:17:05Z"
      hostNodeName: virtual-kubelet
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3529020979
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_south_1131
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)[0].regenerate-trucksim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 0
        memory: 0
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3539744564:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1565592669
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4095064236
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3539744564
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 184
        memory: 184
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3551042008:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3052868802
      displayName: regression-step(9:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1111,truck_sim:true)
      finishedAt: null
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3551042008
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1111
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(9:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1111,truck_sim:true)
      phase: Running
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Steps
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3602065704:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2403452178
      displayName: regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)
      finishedAt: "2021-01-13T12:20:22Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3602065704
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1311
        - name: gpu_requirement
          value: "0"
        - name: truck_sim
          value: "true"
      message: child 'gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3287310681'
        failed
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)
      outboundNodes:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3689719808
      phase: Failed
      progress: 2/2
      resourcesDuration:
        cpu: 204
        memory: 204
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Steps
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3611099437:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-845459564
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1296412131
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3611099437
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(5:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_south_1131,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 190
        memory: 190
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3689719808:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3602065704
      displayName: evaluate-step(0)
      finishedAt: "2021-01-13T12:20:17Z"
      hostNodeName: cn-hangzhou.10.128.109.137
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3689719808
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1311
      message: failed with exit code 1
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)[1].evaluate-step(0)
      outputs:
        exitCode: "1"
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 203
        memory: 203
      startedAt: "2021-01-13T12:17:12Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3699291180:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3331391928
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3993786535
      displayName: regenerate-trucksim-step
      finishedAt: "2021-01-13T12:17:12Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3699291180
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1211
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)[0].regenerate-trucksim-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 204
        memory: 204
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3706776644:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2832051700
      displayName: evaluate-step(0)
      finishedAt: "2021-01-13T12:20:04Z"
      hostNodeName: cn-hangzhou.10.128.98.232
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3706776644
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_2111
      message: failed with exit code 1
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)[1].evaluate-step(0)
      outputs:
        exitCode: "1"
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 188
        memory: 188
      startedAt: "2021-01-13T12:17:11Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3741392784:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1687177185
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2557456158
      displayName: '[1]'
      finishedAt: "2021-01-13T12:20:22Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3741392784
      message: child 'gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2557456158'
        failed
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)[1]
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 205
        memory: 205
      startedAt: "2021-01-13T12:17:11Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3764123472:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2150878129
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1344779168
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3764123472
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 184
        memory: 184
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3782999863:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3602065704
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2336194607
      displayName: regenerate-trucksim-step(0)
      finishedAt: "2021-01-13T12:17:02Z"
      hostNodeName: virtual-kubelet
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3782999863
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1311
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(7:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1311,truck_sim:true)[0].regenerate-trucksim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3796773213:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1071215084
      displayName: build-cybertron-sim-p1-step
      finishedAt: "2021-01-13T12:14:55Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3796773213
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[0].build-cybertron-sim-p1-step
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 20/20
      resourcesDuration:
        cpu: 1763
        memory: 1763
      startedAt: "2021-01-13T12:14:45Z"
      templateName: build-cybertron-sim-p1
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3906825385:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3331391928
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1019922320
      displayName: evaluate-step
      finishedAt: "2021-01-13T12:20:22Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3906825385
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1211
      message: failed with exit code 1
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)[1].evaluate-step
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 203
        memory: 203
      startedAt: "2021-01-13T12:17:12Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3955556197:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2832051700
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3706776644
      displayName: evaluate-step
      finishedAt: "2021-01-13T12:20:12Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3955556197
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_2111
      message: failed with exit code 1
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(6:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_2111,truck_sim:true)[1].evaluate-step
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 188
        memory: 188
      startedAt: "2021-01-13T12:17:11Z"
      templateName: evaluate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Retry
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3993786535:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3331391928
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2749675903
      displayName: regenerate-trucksim-step(0)
      finishedAt: "2021-01-13T12:17:04Z"
      hostNodeName: virtual-kubelet
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3993786535
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_60_north_1211
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)[0].regenerate-trucksim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4031031283:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3339094222
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1707048177
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3796773213
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1965535357
      displayName: '[0]'
      finishedAt: "2021-01-13T12:16:29Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4031031283
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[0]
      phase: Succeeded
      progress: 23/23
      resourcesDuration:
        cpu: 2078
        memory: 2078
      startedAt: "2021-01-13T12:14:45Z"
      templateName: gems-entrypoint
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4035425836:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-722146525
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3385189450
      displayName: '[1]'
      finishedAt: "2021-01-13T12:20:22Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4035425836
      message: child 'gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3385189450'
        failed
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)[1]
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 205
        memory: 205
      startedAt: "2021-01-13T12:17:11Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4060091032:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1687177185
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3741392784
      displayName: regenerate-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4060091032
      message: when '0 == 0 && true != true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(0:gpu_requirement:0,test_case_name:control_tests/jinan_lka_south_1141,truck_sim:true)[0].regenerate-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 205
        memory: 205
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4073946193:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3331391928
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2749675903
      displayName: regenerate-trucksim-gpu-step
      finishedAt: "2021-01-13T12:16:39Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4073946193
      message: when '0 != 0 && true == true' evaluated false
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(8:gpu_requirement:0,test_case_name:control_tests/jinan_lka_60_north_1211,truck_sim:true)[0].regenerate-trucksim-gpu-step
      phase: Skipped
      progress: 1/1
      resourcesDuration:
        cpu: 203
        memory: 203
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim-gpu
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Skipped
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4080937584:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2150878129
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1344779168
      displayName: regenerate-trucksim-step(0)
      finishedAt: "2021-01-13T12:17:04Z"
      hostNodeName: virtual-kubelet
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4080937584
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1321
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(2:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1321,truck_sim:true)[0].regenerate-trucksim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 0
        memory: 0
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4095064236:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1565592669
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1729045194
      displayName: '[1]'
      finishedAt: "2021-01-13T12:20:12Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4095064236
      message: child 'gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1729045194'
        failed
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)[1]
      phase: Failed
      progress: 1/1
      resourcesDuration:
        cpu: 184
        memory: 184
      startedAt: "2021-01-13T12:17:11Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4098539117:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1687177185
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1565592669
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2150878129
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-722146525
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2282790969
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-845459564
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2832051700
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3602065704
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3331391928
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3551042008
      displayName: '[2]'
      finishedAt: null
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4098539117
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2]
      phase: Running
      progress: 18/18
      startedAt: "2021-01-13T12:16:39Z"
      templateName: gems-entrypoint
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4102389217:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-722146525
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-729756668
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2361211845
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-406695329
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-610566068
      displayName: '[0]'
      finishedAt: "2021-01-13T12:17:11Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4102389217
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(3:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1221,truck_sim:true)[0]
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 206
        memory: 206
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4144182104:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2282790969
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1250457496
      displayName: regenerate-trucksim-step(0)
      finishedAt: "2021-01-13T12:17:02Z"
      hostNodeName: virtual-kubelet
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4144182104
      inputs:
        parameters:
        - name: test_case_name
          value: control_tests/jinan_lka_north_1121
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(4:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_1121,truck_sim:true)[0].regenerate-trucksim-step(0)
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regenerate-trucksim
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: Pod
    gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4162027617:
      boundaryID: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1565592669
      children:
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-324195964
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2290121541
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2152265761
      - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3539744564
      displayName: '[0]'
      finishedAt: "2021-01-13T12:17:11Z"
      id: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-4162027617
      name: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7[2].regression-step(1:gpu_requirement:0,test_case_name:control_tests/jinan_lka_north_2121,truck_sim:true)[0]
      phase: Succeeded
      progress: 2/2
      resourcesDuration:
        cpu: 184
        memory: 184
      startedAt: "2021-01-13T12:16:39Z"
      templateName: regression
      templateScope: local/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
      type: StepGroup
  outputs:
    parameters:
    - name: global_cybertron_commit_timestamp
      value: '"1610430777"'
    - name: global_cybertron_commit_author
      value: xiaofan.mu@inceptio.ai
    - name: global_idl_version
      value: ..
  persistentVolumeClaims:
  - name: artifacts-volume-claim
    persistentVolumeClaim:
      claimName: gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-artifacts-volume-claim
  phase: Running
  progress: 23/23
  resourcesDuration:
    cpu: 2078
    memory: 2078
  startedAt: "2021-01-13T12:14:45Z"
  synchronization:
    semaphore:
      holding:
      - holders:
        - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7
        semaphore: argo-hubble/ConfigMap/gems-sync/gems_wf_limited
      - holders:
        - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-291664601
        - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2290121541
        - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3405612553
        - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2361211845
        - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3490648065
        - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1299412752
        - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1192303368
        - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3048519676
        - gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3699291180
        semaphore: argo-hubble/ConfigMap/gems-sync/trucksim_regenarate_template
      waiting:
      - holders:
        - argo-hubble/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2290121541
        - argo-hubble/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-2361211845
        - argo-hubble/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3699291180
        - argo-hubble/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-291664601
        - argo-hubble/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3405612553
        - argo-hubble/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3490648065
        - argo-hubble/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1299412752
        - argo-hubble/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-1192303368
        - argo-hubble/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7/gems-wf-v0-8-60d146aa-e5ad-4b62-b7cb-a319ce581ee7-3048519676
        semaphore: argo-hubble/ConfigMap/gems-sync/trucksim_regenarate_template

@juchaosong
Copy link
Contributor Author

juchaosong commented Jan 14, 2021

The controller log https://gist.github.com/juchaosong/e606ea193078d56aeeb9df71566983f1.
@sarabala1979 The bug is same.

@sarabala1979
Copy link
Member

@juchaosong I was working with another Semaphore issue #4832. I found a few edge case lock is not being released. I fixed and build a eng build.
Is it possible to try on your env to verify whether issue is fixed or not?
sarabala1979/workflow-controller:mutex

@juchaosong
Copy link
Contributor Author

juchaosong commented Jan 14, 2021

@sarabala1979 I saw the issue #4832, and I've tried the sarabala1979/workflow-controller:mutex image, but it' seems not working.

@sarabala1979
Copy link
Member

@juchaosong I pushed the new image few minutes before. Can you pull the new image and try it?

@juchaosong
Copy link
Contributor Author

@sarabala1979 Okay, I'm try it.

@juchaosong
Copy link
Contributor Author

juchaosong commented Jan 14, 2021

@sarabala1979 The latest sarabala1979/workflow-controller:mutex image is working, the semaphore has been released after pod is finished.
Thx a lot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants