Cronworkflow still trigger twice after upgraded to v2.12.0-rc1

[monsterxx03]

Summary

Hi, In #4445 , I reported two issues, suspend/resume cronwf will 100% trigger two workflows, it's fixed after v2.12.0-rc1. Another issue is cronwf will randomly trigger two workflows at scheduled time. I still come to it 3 or 4 times everyday, I have batch of cronwf, scheduled at different intervals(every X mins, daily, hourly...), trigger twice issue happens to random one. I did some protection in application code level, double triggered workflow will failed, I will get a notification.

I'm sure I only have one argo workflow controller in cluster.

BTW, my comment in #4445 (comment) says I thought it's caused by I use kubectl apply -f xxx.yaml to update cronwf spec. Seems it's not root case. It still happen after I recreate every cronwf

Diagnostics

eks 1.16. argo v2.12.0-rc1

A recent failed cronwf spec

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: misc-cal-hot-topics
  namespace: glow
spec:
  schedule: "3-59/10 * * * *"
  successfulJobsHistoryLimit: 1
  timezone: UTC
  failedJobsHistoryLimit: 1
  concurrencyPolicy: Forbid
  workflowMetadata:
    labels:
      # workflows.argoproj.io/archive-strategy: always
      workflow-interval: misc
  workflowSpec:
    entrypoint: data-infra
    arguments:
      parameters:
        - name: script
          value: "cal_hot.py"
        - name: timeout
          value: "300"
        - name: cpuReq
          value: 500m
        - name: memoryReq
          value: 500Mi
    workflowTemplateRef:
      name: base-etl

wftemplate base-etl, just some container image definitions, I think it's not very useful, a simplified version:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: base-etl
  namespace: glow
spec:
  templates:
    - name: data-infra
      inputs:
        parameters:
          - name: script
          - name: args
            default: ""
          - name: timeout
            default: "600"
          - name: cpuReq
            default: 100m
          - name: memoryReq
            default: 200Mi
          - name: cpuLimit
            default: 1000m
          - name: memoryLimit
            default: 2048Mi
          - name: tag
            default: latest
      podSpecPatch: |
        activeDeadlineSeconds: {{inputs.parameters.timeout}}
        containers:
          - name: main
            resources:
              requests:
                cpu: {{inputs.parameters.cpuReq}}
                memory: {{inputs.parameters.memoryReq}}
              limits:
                cpu: {{inputs.parameters.cpuLimit}}
                memory: {{inputs.parameters.memoryLimit}}
      nodeSelector:
        scope: "{{inputs.parameters.node_scope}}"
      container:
        image: "xxxx/data_infra:{{inputs.parameters.tag}}"
        imagePullPolicy: Always
        command: [ "bash" ]
        workingDir: "/mnt/repos"
        env:
          - name: NODE_IP
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
        args: ["-c", "python3 {{inputs.parameters.script}} {{inputs.parameters.args}}"]

Logs:

time="2020-11-19T00:23:00Z" level=info msg="Running misc-cal-hot-topics" namespace=glow workflow=misc-cal-hot-topics
time="2020-11-19T00:23:00Z" level=info msg="Processing workflow" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:00Z" level=info msg="Updated phase -> Running" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:00Z" level=info msg="Processing glow/misc-cal-hot-topics" cronWorkflow=glow/misc-cal-hot-topics
time="2020-11-19T00:23:00Z" level=info msg="Pod node misc-cal-hot-topics-27klm initialized Pending" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:00Z" level=info msg="Created pod: misc-cal-hot-topics-27klm (misc-cal-hot-topics-27klm)" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:01Z" level=info msg="Workflow update successful" namespace=glow phase=Running resourceVersion=280938332 workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:01Z" level=info msg="CronWorkflow glow/misc-cal-hot-topics added" cronWorkflow=glow/misc-cal-hot-topics
time="2020-11-19T00:23:01Z" level=info msg="Enforcing history limit for 'misc-cal-hot-topics'" namespace=glow workflow=misc-cal-hot-topics
time="2020-11-19T00:23:01Z" level=info msg="Processing glow/misc-cal-hot-topics" cronWorkflow=glow/misc-cal-hot-topics
time="2020-11-19T00:23:01Z" level=info msg="misc-cal-hot-topics missed an execution at Thu Nov 19 00:23:00 2020 and is within StartingDeadline" namespace=glow workflow=misc-cal-hot-topics
time="2020-11-19T00:23:01Z" level=info msg="Running misc-cal-hot-topics" namespace=glow workflow=misc-cal-hot-topics
time="2020-11-19T00:23:01Z" level=info msg="Processing workflow" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:01Z" level=info msg="Updated phase -> Running" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:01Z" level=info msg="Processing glow/misc-cal-hot-topics" cronWorkflow=glow/misc-cal-hot-topics
time="2020-11-19T00:23:01Z" level=info msg="Processing workflow" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:01Z" level=info msg="Updating node misc-cal-hot-topics-27klm message: ContainerCreating"
time="2020-11-19T00:23:01Z" level=info msg="Skipped pod misc-cal-hot-topics-27klm (misc-cal-hot-topics-27klm) creation: already exists" namespace=glow podPhase=Pending workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:02Z" level=info msg="Workflow update successful" namespace=glow phase=Running resourceVersion=280938344 workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:02Z" level=info msg="Processing workflow" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:02Z" level=info msg="Skipped pod misc-cal-hot-topics-27klm (misc-cal-hot-topics-27klm) creation: already exists" namespace=glow podPhase=Pending workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:02Z" level=info msg="Pod node misc-cal-hot-topics-lbfdj initialized Pending" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:02Z" level=info msg="Created pod: misc-cal-hot-topics-lbfdj (misc-cal-hot-topics-lbfdj)" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:02Z" level=info msg="Workflow update successful" namespace=glow phase=Running resourceVersion=280938351 workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:02Z" level=info msg="CronWorkflow glow/misc-cal-hot-topics added" cronWorkflow=glow/misc-cal-hot-topics
time="2020-11-19T00:23:03Z" level=info msg="Processing workflow" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:03Z" level=info msg="Updating node misc-cal-hot-topics-lbfdj message: ContainerCreating"
time="2020-11-19T00:23:03Z" level=info msg="Skipped pod misc-cal-hot-topics-lbfdj (misc-cal-hot-topics-lbfdj) creation: already exists" namespace=glow podPhase=Pending workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:03Z" level=info msg="Workflow update successful" namespace=glow phase=Running resourceVersion=280938367 workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:04Z" level=info msg="Processing workflow" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:04Z" level=info msg="Updating node misc-cal-hot-topics-27klm status Pending -> Running"
time="2020-11-19T00:23:04Z" level=info msg="Skipped pod misc-cal-hot-topics-27klm (misc-cal-hot-topics-27klm) creation: already exists" namespace=glow podPhase=Running workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:04Z" level=info msg="Workflow update successful" namespace=glow phase=Running resourceVersion=280938382 workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:05Z" level=info msg="Processing workflow" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:05Z" level=info msg="Updating node misc-cal-hot-topics-lbfdj status Pending -> Running"
time="2020-11-19T00:23:05Z" level=info msg="Skipped pod misc-cal-hot-topics-lbfdj (misc-cal-hot-topics-lbfdj) creation: already exists" namespace=glow podPhase=Running workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:05Z" level=info msg="Workflow update successful" namespace=glow phase=Running resourceVersion=280938391 workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:08Z" level=info msg="Processing workflow" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:08Z" level=info msg="Pod failed" displayName=misc-cal-hot-topics-lbfdj namespace=glow pod=misc-cal-hot-topics-lbfdj templateName= workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:08Z" level=info msg="Setting node misc-cal-hot-topics-lbfdj outputs"
time="2020-11-19T00:23:08Z" level=info msg="Updating node misc-cal-hot-topics-lbfdj status Running -> Failed"
time="2020-11-19T00:23:08Z" level=info msg="Updating node misc-cal-hot-topics-lbfdj message: failed with exit code 1"
time="2020-11-19T00:23:08Z" level=info msg="Running OnExit handler: exit-handler" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:08Z" level=info msg="Steps node misc-cal-hot-topics-lbfdj-1547581229 initialized Running" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:08Z" level=info msg="StepGroup node misc-cal-hot-topics-lbfdj-3296476977 initialized Running" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:08Z" level=info msg="Pod node misc-cal-hot-topics-lbfdj-659440499 initialized Pending" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:08Z" level=info msg="Created pod: misc-cal-hot-topics-lbfdj.onExit[0].failure-notification (misc-cal-hot-topics-lbfdj-659440499)" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:08Z" level=info msg="Workflow step group node misc-cal-hot-topics-lbfdj-3296476977 not yet completed" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:08Z" level=info msg="Workflow update successful" namespace=glow phase=Running resourceVersion=280938415 workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:08Z" level=info msg="Processing workflow" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:08Z" level=info msg="Pod failed" displayName=misc-cal-hot-topics-lbfdj namespace=glow pod=misc-cal-hot-topics-lbfdj templateName= workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:08Z" level=info msg="Running OnExit handler: exit-handler" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:08Z" level=info msg="Skipped pod misc-cal-hot-topics-lbfdj.onExit[0].failure-notification (misc-cal-hot-topics-lbfdj-659440499) creation: already exists" namespace=glow podPhase=Pending workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:08Z" level=info msg="Workflow step group node misc-cal-hot-topics-lbfdj-3296476977 not yet completed" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:08Z" level=info msg="Labeled pod completed" namespace=glow pod=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:08Z" level=info msg="Workflow update successful" namespace=glow phase=Running resourceVersion=280938419 workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:08Z" level=info msg="Labeled pod completed" namespace=glow pod=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:09Z" level=info msg="Processing workflow" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:09Z" level=info msg="Updating node misc-cal-hot-topics-lbfdj-659440499 message: PodInitializing"
time="2020-11-19T00:23:09Z" level=info msg="Running OnExit handler: exit-handler" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:09Z" level=info msg="Skipped pod misc-cal-hot-topics-lbfdj.onExit[0].failure-notification (misc-cal-hot-topics-lbfdj-659440499) creation: already exists" namespace=glow podPhase=Pending workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:09Z" level=info msg="Workflow step group node misc-cal-hot-topics-lbfdj-3296476977 not yet completed" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:09Z" level=info msg="Workflow update successful" namespace=glow phase=Running resourceVersion=280938430 workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:10Z" level=info msg="Processing workflow" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:10Z" level=info msg="Running OnExit handler: exit-handler" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:10Z" level=info msg="Skipped pod misc-cal-hot-topics-lbfdj.onExit[0].failure-notification (misc-cal-hot-topics-lbfdj-659440499) creation: already exists" namespace=glow podPhase=Pending workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:10Z" level=info msg="Workflow step group node misc-cal-hot-topics-lbfdj-3296476977 not yet completed" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="Processing workflow" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="Setting node misc-cal-hot-topics-lbfdj-659440499 outputs"
time="2020-11-19T00:23:12Z" level=info msg="Updating node misc-cal-hot-topics-lbfdj-659440499 status Pending -> Running"
time="2020-11-19T00:23:12Z" level=info msg="Running OnExit handler: exit-handler" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="Workflow step group node misc-cal-hot-topics-lbfdj-3296476977 not yet completed" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="Workflow update successful" namespace=glow phase=Running resourceVersion=280938459 workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="Processing workflow" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="Updating node misc-cal-hot-topics-lbfdj-659440499 status Running -> Succeeded"
time="2020-11-19T00:23:12Z" level=info msg="Running OnExit handler: exit-handler" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="Step group node misc-cal-hot-topics-lbfdj-3296476977 successful" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="node misc-cal-hot-topics-lbfdj-3296476977 phase Running -> Succeeded" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="node misc-cal-hot-topics-lbfdj-3296476977 finished: 2020-11-19 00:23:12.091680811 +0000 UTC" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="Outbound nodes of misc-cal-hot-topics-lbfdj-659440499 is [misc-cal-hot-topics-lbfdj-659440499]" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="Outbound nodes of misc-cal-hot-topics-lbfdj-1547581229 is [misc-cal-hot-topics-lbfdj-659440499]" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="node misc-cal-hot-topics-lbfdj-1547581229 phase Running -> Succeeded" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="node misc-cal-hot-topics-lbfdj-1547581229 finished: 2020-11-19 00:23:12.09176305 +0000 UTC" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="Checking daemoned children of misc-cal-hot-topics-lbfdj-1547581229" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="Updated phase Running -> Failed" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="Updated message -> failed with exit code 1" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="Marking workflow completed" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="Doesn't match with archive label selector. Skipping Archive" namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="Checking daemoned children of " namespace=glow workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="Enforcing history limit for 'misc-cal-hot-topics'" namespace=glow workflow=misc-cal-hot-topics
time="2020-11-19T00:23:12Z" level=info msg="Workflow update successful" namespace=glow phase=Failed resourceVersion=280938464 workflow=misc-cal-hot-topics-lbfdj
time="2020-11-19T00:23:12Z" level=info msg="Labeled pod completed" namespace=glow pod=misc-cal-hot-topics-lbfdj-659440499
time="2020-11-19T00:23:16Z" level=info msg="Processing workflow" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:16Z" level=info msg="Setting node misc-cal-hot-topics-27klm outputs"
time="2020-11-19T00:23:16Z" level=info msg="Updating node misc-cal-hot-topics-27klm status Running -> Succeeded"
time="2020-11-19T00:23:16Z" level=info msg="Running OnExit handler: exit-handler" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:16Z" level=info msg="Steps node misc-cal-hot-topics-27klm-932959126 initialized Running" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:16Z" level=info msg="StepGroup node misc-cal-hot-topics-27klm-1651962632 initialized Running" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:16Z" level=info msg="Skipping misc-cal-hot-topics-27klm.onExit[0].failure-notification: when 'Succeeded != Succeeded' evaluated false" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:16Z" level=info msg="Skipped node misc-cal-hot-topics-27klm-2381304280 initialized Skipped (message: when 'Succeeded != Succeeded' evaluated false)" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:16Z" level=info msg="Step group node misc-cal-hot-topics-27klm-1651962632 successful" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:16Z" level=info msg="node misc-cal-hot-topics-27klm-1651962632 phase Running -> Succeeded" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:16Z" level=info msg="node misc-cal-hot-topics-27klm-1651962632 finished: 2020-11-19 00:23:16.820231856 +0000 UTC" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:16Z" level=info msg="Outbound nodes of misc-cal-hot-topics-27klm-2381304280 is [misc-cal-hot-topics-27klm-2381304280]" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:16Z" level=info msg="Outbound nodes of misc-cal-hot-topics-27klm-932959126 is [misc-cal-hot-topics-27klm-2381304280]" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:16Z" level=info msg="node misc-cal-hot-topics-27klm-932959126 phase Running -> Succeeded" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:16Z" level=info msg="node misc-cal-hot-topics-27klm-932959126 finished: 2020-11-19 00:23:16.820308051 +0000 UTC" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:16Z" level=info msg="Checking daemoned children of misc-cal-hot-topics-27klm-932959126" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:16Z" level=info msg="Updated phase Running -> Succeeded" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:16Z" level=info msg="Marking workflow completed" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:16Z" level=info msg="Doesn't match with archive label selector. Skipping Archive" namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:16Z" level=info msg="Checking daemoned children of " namespace=glow workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:16Z" level=info msg="Workflow update successful" namespace=glow phase=Succeeded resourceVersion=280938501 workflow=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:16Z" level=info msg="Labeled pod completed" namespace=glow pod=misc-cal-hot-topics-27klm
time="2020-11-19T00:23:23Z" level=info msg="Enforcing history limit for 'misc-cal-hot-topics'" namespace=glow workflow=misc-cal-hot-topics
time="2020-11-19T00:23:23Z" level=info msg="Deleted Workflow 'misc-cal-hot-topics-h94vs' due to CronWorkflow 'misc-cal-hot-topics' history limit" namespace=glow workflow=misc-cal-hot-topics
time="2020-11-19T00:23:23Z" level=info msg="Deleted Workflow 'misc-cal-hot-topics-7d4bj' due to CronWorkflow 'misc-cal-hot-topics' history limit" namespace=glow workflow=misc-cal-hot-topics
time="2020-11-19T00:23:23Z" level=info msg="Processing glow/misc-cal-hot-topics" cronWorkflow=glow/misc-cal-hot-topics
time="2020-11-19T00:23:23Z" level=info msg="CronWorkflow glow/misc-cal-hot-topics added" cronWorkflow=glow/misc-cal-hot-topics
time="2020-11-19T00:23:36Z" level=info msg="Enforcing history limit for 'misc-cal-hot-topics'" namespace=glow workflow=misc-cal-hot-topics

At 00:23, cronwf misc-cal-hot-topics triggered two wf, misc-cal-hot-topics-27klm and misc-cal-hot-topics-lbfdj, lbfdj failed by my protection code.

useful log is:
time="2020-11-19T00:23:01Z" level=info msg="misc-cal-hot-topics missed an execution at Thu Nov 19 00:23:00 2020 and is within StartingDeadline" namespace=glow workflow=misc-cal-hot-topics, seems this check passed because scheduled wf is not ready.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[simster7]

Will take a look

[simster7]

@monsterxx03 I'm having a difficult time reproducing this. My initial suspicion is that our syncAllWorkflows function -- which runs ever 10 seconds -- will sometimes run at exactly the same time as the scheduled Run function (which is what actually triggers the CronWorkflow).

My hypothesis is that syncAllWorkflows will pick up a stale CronWorkflow from the informer and persist it.

I will likely make a development build with a solution to this problem as well as debugging logs and have you test it on your cluster for a day or so.

[monsterxx03]

@simster7 It's okay, can you share a docker image to me? I can run it to collect logs

[simster7]

Will work on this today

[rmgpinto]

We're having the same issue.

[alexec]

@simster7 I'm going to push using the timestamp to create a deterministic name solution again, this is what CronJob does:

https://github.com/kubernetes/kubernetes/blob/0b1d94a849e3e31d83b8e29454bf6698b93a8f11/pkg/controller/cronjob/cronjob_controllerv2.go#L594

[simster7]

Sure, will take a look

[alexec]

1. Using an opaque but deterministic name will prevent users from trying to parse the name.
2. I'm not sure if/what we do about corn workflows that schedule more often than once per minute.
3. We should use the scheduled time, not time.Now() obviously.

[simster7]

I'm not sure if/what we do about corn workflows that schedule more often than once per minute.

It is impossible to schedule cron workflows that run more frequently than every minute

[alexec]

What it two different Goroutine create one?