Workflow PodDisruptionBudget support is broken in 3.4.7 #10942

kulinskyvs · 2023-04-18T17:23:04Z

Pre-requisites

I have double-checked my configuration
I can confirm the issues exists when I tested with :latest
I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

Looks like "podDisruptionBudget" support has been broken in Argo Workflows (tested in version 3.4.7)

Steps to reproduce (using the workflow definition listed below)

Check no PDBs exist before test

$ kubectl get pdb
No resources found in usr-p-vku namespace.

Create the workflow

$ kubectl apply -f  pdb-issue.yaml
workflow.argoproj.io/pdb-wf-1 created

Check workflow details

kubectl describe workflow pdb-wf-1

Name:         pdb-wf-1
....
Status:
  ....
  Conditions:
    Status:     True
    Type:       Completed
  Finished At:  2023-04-18T17:15:48Z
  Message:      Unable to create PDB resource for workflow, pdb-wf-1 error: poddisruptionbudgets.policy "pdb-wf-1" already exists
  Phase:        Failed
  Progress:     0/0
Events:
  Type     Reason           Age   From                 Message
  ----     ------           ----  ----                 -------
  Normal   WorkflowRunning  57s   workflow-controller  Workflow Running
  Normal   WorkflowRunning  57s   workflow-controller  Workflow Running
  Warning  WorkflowFailed   57s   workflow-controller  Unable to create PDB resource for workflow, pdb-wf-1 error: poddisruptionbudgets.policy "pdb-wf-1" already exists

Check not PDBs exist after workflows failure

$ kubectl get pdb
No resources found in usr-p-vku namespace.

Version

v3.4.7

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: pdb-wf-1
spec:
  entrypoint: main
  podDisruptionBudget:
    minAvailable: 9999
  templates:
    - name: main
      container:
        image: docker/whalesay:latest
        command: [cowsay]
        args: ["test"]

Logs from the workflow controller

time="2023-04-18T17:15:48.421Z" level=info msg="Processing workflow" namespace=usr-p-vku workflow=pdb-wf-1
time="2023-04-18T17:15:48.430Z" level=info msg="Updated phase  -> Running" namespace=usr-p-vku workflow=pdb-wf-1
time="2023-04-18T17:15:48.447Z" level=info msg="Created PDB resource for workflow." namespace=usr-p-vku workflow=pdb-wf-1
time="2023-04-18T17:15:48.447Z" level=info msg="Pod node pdb-wf-1 initialized Pending" namespace=usr-p-vku workflow=pdb-wf-1
time="2023-04-18T17:15:48.480Z" level=info msg="Created pod: pdb-wf-1 (pdb-wf-1)" namespace=usr-p-vku workflow=pdb-wf-1
time="2023-04-18T17:15:48.481Z" level=info msg="TaskSet Reconciliation" namespace=usr-p-vku workflow=pdb-wf-1
time="2023-04-18T17:15:48.481Z" level=info msg=reconcileAgentPod namespace=usr-p-vku workflow=pdb-wf-1
time="2023-04-18T17:15:48.487Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"pdb-wf-1\": the object has been modified; please apply your changes to the latest version and try again Conflict" namespace=usr-p-vku workflow=pdb-wf-1
time="2023-04-18T17:15:48.488Z" level=info msg="Re-applying updates on latest version and retrying update" namespace=usr-p-vku workflow=pdb-wf-1
time="2023-04-18T17:15:48.494Z" level=info msg="Failed to re-apply update" error="must never update completed workflows" namespace=usr-p-vku workflow=pdb-wf-1

Logs from in your workflow's wait container

time="2023-04-18T17:15:51.528Z" level=info msg="Starting Workflow Executor" version=v3.4.7
time="2023-04-18T17:15:51.551Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2023-04-18T17:15:51.552Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=usr-p-vku podName=pdb-wf-1 template="{\"name\":\"main\",\"inputs\":{},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker/whalesay:latest\",\"command\":[\"cowsay\"],\"args\":[\"test\"],\"resources\":{}}}" version="&Version{Version:v3.4.7,BuildDate:2023-04-11T16:19:29Z,GitCommit:f2292647c5a6be2f888447a1fef71445cc05b8fd,GitTag:v3.4.7,GitTreeState:clean,GoVersion:go1.19.8,Compiler:gc,Platform:linux/amd64,}"
time="2023-04-18T17:15:51.552Z" level=info msg="Starting deadline monitor"
time="2023-04-18T17:15:53.558Z" level=info msg="Main container completed" error="<nil>"
time="2023-04-18T17:15:53.558Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2023-04-18T17:15:53.558Z" level=info msg="No output parameters"
time="2023-04-18T17:15:53.558Z" level=info msg="No output artifacts"
time="2023-04-18T17:15:53.558Z" level=info msg="Alloc=6927 TotalAlloc=14842 Sys=29037 NumGC=5 Goroutines=7"

The text was updated successfully, but these errors were encountered:

terrytangyuan · 2023-04-18T17:59:43Z

I could not reproduce this. What's your Kubernetes version?

kulinskyvs · 2023-04-18T18:16:00Z

I could not reproduce this. What's your Kubernetes version?

AWS EKS 1.22

kubectl version -o yaml

clientVersion:
  buildDate: "2022-12-08T19:58:30Z"
  compiler: gc
  gitCommit: b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d
  gitTreeState: clean
  gitVersion: v1.26.0
  goVersion: go1.19.4
  major: "1"
  minor: "26"
  platform: linux/amd64
kustomizeVersion: v4.5.7
serverVersion:
  buildDate: "2023-01-24T09:34:06Z"
  compiler: gc
  gitCommit: 47b89ea2caa1f7958bc6539d6865820c86b4bf60
  gitTreeState: clean
  gitVersion: v1.22.17-eks-48e63af
  goVersion: go1.16.15
  major: "1"
  minor: 22+
  platform: linux/amd64

kulinskyvs · 2023-04-18T18:35:06Z

An update, the problem is not reproduced with workflow templates.
The workflow below is OK.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: pdb-wft
spec:
  podDisruptionBudget:
    maxUnavailable: 0
  templates:
    - name: main
      container:
        image: docker/whalesay:latest
        command: [cowsay]
        args: ["test"]
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: pdb-wf
spec:
  entrypoint: main
  templates:
    - name: main
      steps:
        - - name: call
            templateRef:
              name: pdb-wft
              template: main

terrytangyuan · 2023-04-18T19:02:52Z

I could not reproduce it. I even tried creating an existing PDB before submitting the workflow and everything seems to work fine.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: pdb-wf-1
spec:
  minAvailable: 2
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: pdb-wf-1
spec:
  entrypoint: main
  podDisruptionBudget:
    minAvailable: 9999
  templates:
    - name: main
      container:
        image: docker/whalesay:latest
        command: [cowsay]
        args: ["test"]

kulinskyvs · 2023-04-18T19:05:30Z

@terrytangyuan ,

Thank you for your effort.
I've noticed that a WARN message with an actual Error is present in Argo Workflows Controller logs

time="2023-04-18T17:15:48.487Z" level=warning msg="Error updating workflow: Operation cannot be fulfilled on workflows.argoproj.io \"pdb-wf-1\": the object has been modified; please apply your changes to the latest version and try again Conflict" namespace=usr-p-vku workflow=pdb-wf-1

Not sure if this can be related.

Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>

terrytangyuan · 2023-04-18T19:36:57Z

I submitted a potential fix in #10944. Would you like to try it out? It should be available in an image tag dev-fix-pdb as soon as the release builds finish https://github.com/argoproj/argo-workflows/actions/runs/4736276262/jobs/8407593700?pr=10944.

kulinskyvs · 2023-04-18T20:33:08Z

I submitted a potential fix in #10944. Would you like to try it out? It should be available in an image tag dev-fix-pdb as soon as the release builds finish https://github.com/argoproj/argo-workflows/actions/runs/4736276262/jobs/8407593700?pr=10944.

Awesome!
I do confirm that the fix actually resolves the problem - tested with the dev-fix-pdb workflows controller tag.
Thank you for your reactivity!

terrytangyuan · 2023-04-18T21:29:26Z

Great to hear!

Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>

argoproj#10944) Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>

argoproj#10944) Signed-off-by: Yuan Tang <terrytangyuan@gmail.com> Signed-off-by: Dillen Padhiar <dillen_padhiar@intuit.com>

kulinskyvs added the type/bug label Apr 18, 2023

kulinskyvs changed the title ~~PodDisruptionBudget support is broken in 3.4.7~~ Workflow PodDisruptionBudget support is broken in 3.4.7 Apr 18, 2023

terrytangyuan added a commit that referenced this issue Apr 18, 2023

fix: Check AlreadyExists error when creating PDB. Fixes #10942

2811825

Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>

terrytangyuan mentioned this issue Apr 18, 2023

fix: Check AlreadyExists error when creating PDB. Fixes #10942 #10944

Merged

terrytangyuan added a commit that referenced this issue Apr 19, 2023

fix: Check AlreadyExists error when creating PDB. Fixes #10942

2b4c1c1

Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>

terrytangyuan added a commit that referenced this issue Apr 19, 2023

fix: Check AlreadyExists error when creating PDB. Fixes #10942

84077a5

Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>

terrytangyuan added a commit that referenced this issue Apr 19, 2023

fix: Check AlreadyExists error when creating PDB. Fixes #10942

80b6aa2

Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>

terrytangyuan self-assigned this Apr 19, 2023

terrytangyuan closed this as completed in #10944 Apr 19, 2023

terrytangyuan added a commit that referenced this issue Apr 19, 2023

fix: Check AlreadyExists error when creating PDB. Fixes #10942 (#10944)

1617db0

Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>

terrytangyuan mentioned this issue May 22, 2023

Release v3.4.8 cherry-pick candidates #11118

Closed

44 tasks

terrytangyuan added a commit that referenced this issue May 25, 2023

fix: Check AlreadyExists error when creating PDB. Fixes #10942 (#10944)

131a854

Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>

JPZ13 pushed a commit to pipekit/argo-workflows that referenced this issue Jul 4, 2023

fix: Check AlreadyExists error when creating PDB. Fixes argoproj#10942 (

282d056

argoproj#10944) Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>

agilgur5 added the area/controller Controller issues, panics label Apr 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Workflow PodDisruptionBudget support is broken in 3.4.7 #10942

Workflow PodDisruptionBudget support is broken in 3.4.7 #10942

kulinskyvs commented Apr 18, 2023 •

edited

terrytangyuan commented Apr 18, 2023

kulinskyvs commented Apr 18, 2023

kulinskyvs commented Apr 18, 2023 •

edited by agilgur5

terrytangyuan commented Apr 18, 2023 •

edited by agilgur5

kulinskyvs commented Apr 18, 2023

terrytangyuan commented Apr 18, 2023

kulinskyvs commented Apr 18, 2023

terrytangyuan commented Apr 18, 2023

Workflow PodDisruptionBudget support is broken in 3.4.7 #10942

Workflow PodDisruptionBudget support is broken in 3.4.7 #10942

Comments

kulinskyvs commented Apr 18, 2023 • edited

Pre-requisites

What happened/what you expected to happen?

Version

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

Logs from the workflow controller

Logs from in your workflow's wait container

terrytangyuan commented Apr 18, 2023

kulinskyvs commented Apr 18, 2023

kulinskyvs commented Apr 18, 2023 • edited by agilgur5

terrytangyuan commented Apr 18, 2023 • edited by agilgur5

kulinskyvs commented Apr 18, 2023

terrytangyuan commented Apr 18, 2023

kulinskyvs commented Apr 18, 2023

terrytangyuan commented Apr 18, 2023

kulinskyvs commented Apr 18, 2023 •

edited

kulinskyvs commented Apr 18, 2023 •

edited by agilgur5

terrytangyuan commented Apr 18, 2023 •

edited by agilgur5