Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

deletePDBResource panics if workflowRef not found #3121

Closed
alexec opened this issue May 28, 2020 · 4 comments
Closed

deletePDBResource panics if workflowRef not found #3121

alexec opened this issue May 28, 2020 · 4 comments
Assignees
@sarabala1979
Labels
type/bug

Comments

@alexec
Copy link
Contributor

alexec commented May 28, 2020

Workflow:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: ui-workflow-error
  namespace: argo
spec:
  entrypoint: main
  templates:
    - name: noop
  workflowTemplateRef:
    name: not-exists

Logs:

controller | time="2020-05-28T10:23:59-07:00" level=info msg="Marking workflow completed" namespace=argo workflow=ui-workflow-error
 controller | time="2020-05-28T10:23:59-07:00" level=info msg="Updated message workflowtemplate.argoproj.io \"not-exists\" not found -> runtime error: invalid memory address or nil pointer dereference" namespace=argo workflow=ui-workflow-error
 controller | time="2020-05-28T10:23:59-07:00" level=info msg="Marking workflow completed" namespace=argo workflow=ui-workflow-error
 controller | time="2020-05-28T10:23:59-07:00" level=info msg="Checking daemoned children of " namespace=argo workflow=ui-workflow-error
 controller | time="2020-05-28T10:23:59-07:00" level=info msg="Workflow update successful" namespace=argo phase=Error resourceVersion=110454 workflow=ui-workflow-error
 controller | E0528 10:24:00.414888   86681 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
 controller | goroutine 238 [running]:
 controller | k8s.io/apimachinery/pkg/util/runtime.logPanic(0x22a5cc0, 0x3333450)
 controller | 	/Users/acollins8/go/pkg/mod/k8s.io/apimachinery@v0.16.7-beta.0/pkg/util/runtime/runtime.go:74 +0xa3
 controller | k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
 controller | 	/Users/acollins8/go/pkg/mod/k8s.io/apimachinery@v0.16.7-beta.0/pkg/util/runtime/runtime.go:48 +0x82
 controller | panic(0x22a5cc0, 0x3333450)
 controller | 	/usr/local/Cellar/go/1.13.4/libexec/src/runtime/panic.go:679 +0x1b2
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).persistUpdates(0xc00074e3c0)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:520 +0x461
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).operate.func1(0xc00074e3c0)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:157 +0xa7
 controller | panic(0x22a5cc0, 0x3333450)
 controller | 	/usr/local/Cellar/go/1.13.4/libexec/src/runtime/panic.go:679 +0x1b2
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).deletePDBResource(0xc00074e3c0, 0x24a4f4f, 0x9)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:2619 +0x41
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).markWorkflowPhase(0xc00074e3c0, 0x24a1c7e, 0x5, 0xc000b61901, 0xc000b61938, 0x1, 0x1)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:1619 +0x639
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).markWorkflowError(0xc00074e3c0, 0x26e68c0, 0x3333450, 0x26e6801)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:1657 +0x99
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).operate.func2(0xc00074e3c0)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:162 +0xae
 controller | panic(0x22a5cc0, 0x3333450)
 controller | 	/usr/local/Cellar/go/1.13.4/libexec/src/runtime/panic.go:679 +0x1b2
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).deletePDBResource(0xc00074e3c0, 0x24a4f4f, 0x9)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:2619 +0x41
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).markWorkflowPhase(0xc00074e3c0, 0x24a1c7e, 0x5, 0x1, 0xc000b62210, 0x1, 0x1)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:1619 +0x639
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).markWorkflowError(0xc00074e3c0, 0x26e42a0, 0xc00074e460, 0x1)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:1657 +0x99
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).operate(0xc00074e3c0)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:176 +0x3502
 controller | github.com/argoproj/argo/workflow/controller.(*WorkflowController).processNextItem(0xc0000cc000, 0x0)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/controller.go:439 +0x8c2
 controller | github.com/argoproj/argo/workflow/controller.(*WorkflowController).runWorker(0xc0000cc000)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/controller.go:365 +0x2b
 controller | k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000a898c0)
 controller | 	/Users/acollins8/go/pkg/mod/k8s.io/apimachinery@v0.16.7-beta.0/pkg/util/wait/wait.go:152 +0x5e
 controller | k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000a898c0, 0x3b9aca00, 0x0, 0x1, 0xc000130660)
 controller | 	/Users/acollins8/go/pkg/mod/k8s.io/apimachinery@v0.16.7-beta.0/pkg/util/wait/wait.go:153 +0xf8
 controller | k8s.io/apimachinery/pkg/util/wait.Until(0xc000a898c0, 0x3b9aca00, 0xc000130660)
 controller | 	/Users/acollins8/go/pkg/mod/k8s.io/apimachinery@v0.16.7-beta.0/pkg/util/wait/wait.go:88 +0x4d
 controller | created by github.com/argoproj/argo/workflow/controller.(*WorkflowController).Run
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/controller.go:178 +0x9cd
 controller | panic: runtime error: invalid memory address or nil pointer dereference [recovered]
 controller | 	panic: runtime error: invalid memory address or nil pointer dereference
 controller | 	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
 controller | 	panic: runtime error: invalid memory address or nil pointer dereference
 controller | [signal SIGSEGV: segmentation violation code=0x1 addr=0x158 pc=0x2100671]
 controller | goroutine 238 [running]:
 controller | k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
 controller | 	/Users/acollins8/go/pkg/mod/k8s.io/apimachinery@v0.16.7-beta.0/pkg/util/runtime/runtime.go:55 +0x105
 controller | panic(0x22a5cc0, 0x3333450)
 controller | 	/usr/local/Cellar/go/1.13.4/libexec/src/runtime/panic.go:679 +0x1b2
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).persistUpdates(0xc00074e3c0)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:520 +0x461
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).operate.func1(0xc00074e3c0)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:157 +0xa7
 controller | panic(0x22a5cc0, 0x3333450)
 controller | 	/usr/local/Cellar/go/1.13.4/libexec/src/runtime/panic.go:679 +0x1b2
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).deletePDBResource(0xc00074e3c0, 0x24a4f4f, 0x9)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:2619 +0x41
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).markWorkflowPhase(0xc00074e3c0, 0x24a1c7e, 0x5, 0xc000b61901, 0xc000b61938, 0x1, 0x1)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:1619 +0x639
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).markWorkflowError(0xc00074e3c0, 0x26e68c0, 0x3333450, 0x26e6801)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:1657 +0x99
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).operate.func2(0xc00074e3c0)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:162 +0xae
 controller | panic(0x22a5cc0, 0x3333450)
 controller | 	/usr/local/Cellar/go/1.13.4/libexec/src/runtime/panic.go:679 +0x1b2
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).deletePDBResource(0xc00074e3c0, 0x24a4f4f, 0x9)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:2619 +0x41
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).markWorkflowPhase(0xc00074e3c0, 0x24a1c7e, 0x5, 0x1, 0xc000b62210, 0x1, 0x1)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:1619 +0x639
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).markWorkflowError(0xc00074e3c0, 0x26e42a0, 0xc00074e460, 0x1)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:1657 +0x99
 controller | github.com/argoproj/argo/workflow/controller.(*wfOperationCtx).operate(0xc00074e3c0)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/operator.go:176 +0x3502
 controller | github.com/argoproj/argo/workflow/controller.(*WorkflowController).processNextItem(0xc0000cc000, 0x0)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/controller.go:439 +0x8c2
 controller | github.com/argoproj/argo/workflow/controller.(*WorkflowController).runWorker(0xc0000cc000)
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/controller.go:365 +0x2b
 controller | k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000a898c0)
 controller | 	/Users/acollins8/go/pkg/mod/k8s.io/apimachinery@v0.16.7-beta.0/pkg/util/wait/wait.go:152 +0x5e
 controller | k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000a898c0, 0x3b9aca00, 0x0, 0x1, 0xc000130660)
 controller | 	/Users/acollins8/go/pkg/mod/k8s.io/apimachinery@v0.16.7-beta.0/pkg/util/wait/wait.go:153 +0xf8
 controller | k8s.io/apimachinery/pkg/util/wait.Until(0xc000a898c0, 0x3b9aca00, 0xc000130660)
 controller | 	/Users/acollins8/go/pkg/mod/k8s.io/apimachinery@v0.16.7-beta.0/pkg/util/wait/wait.go:88 +0x4d
 controller | created by github.com/argoproj/argo/workflow/controller.(*WorkflowController).Run
 controller | 	/Users/acollins8/go/src/github.com/argoproj/argo/workflow/controller/controller.go:178 +0x9cd
 controller | Terminating controller
@alexec alexec changed the title deletePDBResource panics ii workflowRef not found deletePDBResource panics if workflowRef not found May 28, 2020
@alexec
Copy link
Contributor Author

alexec commented May 28, 2020

@sarabala1979 not sure which line this is...

@alexec
Copy link
Contributor Author

alexec commented May 28, 2020

This line:

if woc.wfSpec.PodDisruptionBudget == nil {

So woc.wfSpec must be nil.

@sarabala1979
Copy link
Member

sarabala1979 commented May 28, 2020 •
edited

Above workflow definition is invalid using ArgoCL. But if user uses kubectl, we have this issue. I am fixing it.

sbalasubram@INTUL1732f3e11 ~/go/src/github.com/argoproj/argo (issue-3121) $ dist/argo submit wf.yaml 
2020/05/28 12:00:01 Failed to submit workflow: Templates is invalid field in spec if workflow referred WorkflowTemplate reference
sbalasubram@INTUL1732f3e11 ~/go/src/github.com/argoproj/argo (issue-3121) $ dist/argo submit wf.yaml 
2020/05/28 12:00:41 Failed to submit workflow: workflowtemplates.argoproj.io "not-exists" not found

@sarabala1979 sarabala1979 mentioned this issue May 28, 2020
Merged
6 tasks
@alexec alexec reopened this Jun 3, 2020
@alexec alexec mentioned this issue Jun 3, 2020
Closed
@alexec
Copy link
Contributor Author

alexec commented Jun 3, 2020

@sarabala1979 this bug still happens.

deletePDBResource needs to be robust to wf.wfSpec being nil.

alexec added a commit to alexec/argo-workflows that referenced this issue Jun 3, 2020
@alexec
fix(controller): Maybe bug with nil wf.wfSpec. Fixes argoproj#3121
cec25e8
@alexec alexec closed this as completed in c6565d7 Jun 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sarabala1979 sarabala1979

Labels
type/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@alexec @sarabala1979