Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rancher server panics and crashes when deleting a node when in provisioning state #30430

Closed
sowmyav27 opened this issue Dec 10, 2020 · 1 comment
Closed

Rancher server panics and crashes when deleting a node when in provisioning state #30430

sowmyav27 opened this issue Dec 10, 2020 · 1 comment
Assignees
@pennyscissors @sowmyav27
Labels
kind/bug-qa Issues that have not yet hit a real release. Bugs introduced by a new feature or enhancement
Milestone
v2.5.4

Comments

@sowmyav27
Copy link
Contributor

What kind of request is this (question/bug/enhancement/feature request): bug

Steps to reproduce (least amount of steps as possible):

  • Deploy a node driver cluster - 1 etcd/ 1 control and 1 worker
  • When the node is in provisioning state(I gave it a minute before deleting the node), delete the worker node.
2020/12/10 21:13:50 [ERROR] error syncing 'c-db5qm/m-b9vqg': handler node-controller: an empty namespace may not be set when a resource name is provided, requeuing
2020/12/10 21:13:50 [INFO] Removing node sowmya-pni-wrk-1
E1210 21:13:50.427267       8 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 6612 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x3c3df00, 0x73c0820)
	/go/pkg/mod/github.com/rancher/apimachinery@v0.19.0-rancher1/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/github.com/rancher/apimachinery@v0.19.0-rancher1/pkg/util/runtime/runtime.go:48 +0x82
panic(0x3c3df00, 0x73c0820)
	/usr/local/go/src/runtime/panic.go:969 +0x166
github.com/rancher/rancher/pkg/node.IsEtcd(0x0, 0x0)
	/go/src/github.com/rancher/rancher/pkg/node/node.go:197 +0x5
github.com/rancher/rancher/pkg/node.DrainBeforeDelete(...)
	/go/src/github.com/rancher/rancher/pkg/node/node.go:184
github.com/rancher/rancher/pkg/controllers/management/node.(*Lifecycle).drainNode(0xc00264fbc0, 0xc00aff8480, 0x4514bca, 0x10)
	/go/src/github.com/rancher/rancher/pkg/controllers/management/node/controller.go:787 +0x12f
github.com/rancher/rancher/pkg/controllers/management/node.(*Lifecycle).Remove.func1(0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/rancher/rancher/pkg/controllers/management/node/controller.go:262 +0x3ed
github.com/rancher/norman/condition.Cond.doInternal(0x44f712d, 0x7, 0x4e57c01, 0x4e57c40, 0xc00aff8480, 0xc008d5d628, 0x197, 0x44f4f5f, 0x6, 0x44fe38d)
	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20200930000340-693d65aaffe3/condition/condition.go:212 +0x97
github.com/rancher/norman/condition.Cond.do2(0x44f712d, 0x7, 0x44f7101, 0x4e57c40, 0xc00aff8480, 0xc008d5d628, 0x4b6, 0xc00ed750c0, 0xc00ed75120, 0xc00c7466a0, ...)
	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20200930000340-693d65aaffe3/condition/condition.go:181 +0x1d8
github.com/rancher/norman/condition.Cond.do(...)
	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20200930000340-693d65aaffe3/condition/condition.go:170
github.com/rancher/norman/condition.Cond.DoUntilTrue(0x44f712d, 0x7, 0x4e57c40, 0xc00aff8480, 0xc008d5d628, 0x2, 0x2, 0x6, 0x4)
	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20200930000340-693d65aaffe3/condition/condition.go:136 +0xb3
github.com/rancher/rancher/pkg/controllers/management/node.(*Lifecycle).Remove(0xc00264fbc0, 0xc00aff8480, 0xc008d5d6d0, 0x136cd18, 0xc005031d28, 0xc00aff86a8)
	/go/src/github.com/rancher/rancher/pkg/controllers/management/node/controller.go:223 +0x97
github.com/rancher/rancher/pkg/generated/norman/management.cattle.io/v3.(*nodeLifecycleAdapter).Finalize(0xc0008b08b0, 0x4e57c40, 0xc00aff8480, 0x0, 0x0, 0x8000104, 0x0)
	/go/src/github.com/rancher/rancher/pkg/generated/norman/management.cattle.io/v3/zz_generated_node_lifecycle_adapter.go:39 +0x52
github.com/rancher/norman/lifecycle.checkNil(0x4e57c40, 0xc00aff8480, 0xc008d5d8a8, 0xc005031b00, 0x0, 0x0, 0xc0018d2ff0)
	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20200930000340-693d65aaffe3/lifecycle/object.go:191 +0x3e
github.com/rancher/norman/lifecycle.(*objectLifecycleAdapter).record(0xc0041ce360, 0x4e57c40, 0xc005031b00, 0xc008d5d8a8, 0x24, 0xc0018d2f01, 0x24, 0x4c4b40)
	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20200930000340-693d65aaffe3/lifecycle/object.go:181 +0xd0
github.com/rancher/norman/lifecycle.(*objectLifecycleAdapter).finalize(0xc0041ce360, 0x4e57c40, 0xc005031b00, 0x4e57c40, 0xc005031b00, 0xc00a617d01, 0x8, 0xc008d5d940)
	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20200930000340-693d65aaffe3/lifecycle/object.go:106 +0x226
github.com/rancher/norman/lifecycle.(*objectLifecycleAdapter).sync(0xc0041ce360, 0xc00612ac00, 0xf, 0x44742e0, 0xc005031b00, 0x40ca16, 0xc00a2878b0, 0x10, 0x10)
	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20200930000340-693d65aaffe3/lifecycle/object.go:58 +0x133
github.com/rancher/rancher/pkg/generated/norman/management.cattle.io/v3.NewNodeLifecycleAdapter.func1(0xc00612ac00, 0xf, 0xc005031b00, 0x10, 0x3931d80, 0x1, 0xc00a2878b0)
	/go/src/github.com/rancher/rancher/pkg/generated/norman/management.cattle.io/v3/zz_generated_node_lifecycle_adapter.go:61 +0x53
github.com/rancher/rancher/pkg/generated/norman/management.cattle.io/v3.(*nodeController).AddHandler.func1(0xc00612ac00, 0xf, 0x44742e0, 0xc005031b00, 0xc008d5db20, 0x3, 0x3, 0x0)
	/go/src/github.com/rancher/rancher/pkg/generated/norman/management.cattle.io/v3/zz_generated_node_controller.go:156 +0x63
github.com/rancher/norman/controller.(*genericController).AddHandler.func1(0xc00612ac00, 0xf, 0x4e57c40, 0xc005031b00, 0x0, 0x0, 0x0, 0x0)
	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20200930000340-693d65aaffe3/controller/generic_controller.go:62 +0x221
github.com/rancher/lasso/pkg/controller.SharedControllerHandlerFunc.OnChange(0xc0041ce390, 0xc00612ac00, 0xf, 0x4e57c40, 0xc005031b00, 0x0, 0x0, 0x0, 0x0)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20200905045615-7fcb07d6a20b/pkg/controller/sharedcontroller.go:29 +0x4e
github.com/rancher/lasso/pkg/controller.(*sharedHandler).OnChange(0xc001ab6bd0, 0xc00612ac00, 0xf, 0x4e57c40, 0xc005031b00, 0x474c01, 0x0)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20200905045615-7fcb07d6a20b/pkg/controller/sharedhandler.go:65 +0x14a
github.com/rancher/lasso/pkg/controller.(*controller).syncHandler(0xc0003dee70, 0xc00612ac00, 0xf, 0xc0045e6a50, 0xc008d5dd50)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20200905045615-7fcb07d6a20b/pkg/controller/controller.go:210 +0xd1
github.com/rancher/lasso/pkg/controller.(*controller).processSingleItem(0xc0003dee70, 0x3931fc0, 0xc005c1e140, 0x0, 0x0)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20200905045615-7fcb07d6a20b/pkg/controller/controller.go:192 +0xd7
github.com/rancher/lasso/pkg/controller.(*controller).processNextWorkItem(0xc0003dee70, 0x203001)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20200905045615-7fcb07d6a20b/pkg/controller/controller.go:169 +0x51
github.com/rancher/lasso/pkg/controller.(*controller).runWorker(0xc0003dee70)
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20200905045615-7fcb07d6a20b/pkg/controller/controller.go:158 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0023ce600)
	/go/pkg/mod/github.com/rancher/apimachinery@v0.19.0-rancher1/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0023ce600, 0x4e2c600, 0xc0045e7f20, 0xc0032e9c01, 0xc001e44420)
	/go/pkg/mod/github.com/rancher/apimachinery@v0.19.0-rancher1/pkg/util/wait/wait.go:156 +0xa3
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0023ce600, 0x3b9aca00, 0x0, 0x1, 0xc001e44420)
	/go/pkg/mod/github.com/rancher/apimachinery@v0.19.0-rancher1/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc0023ce600, 0x3b9aca00, 0xc001e44420)
	/go/pkg/mod/github.com/rancher/apimachinery@v0.19.0-rancher1/pkg/util/wait/wait.go:90 +0x4d
created by github.com/rancher/lasso/pkg/controller.(*controller).run
	/go/pkg/mod/github.com/rancher/lasso@v0.0.0-20200905045615-7fcb07d6a20b/pkg/controller/controller.go:129 +0x31f
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0x2879985]

Result:

Other details that may be helpful:

Environment information

  • Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): 2.5-head commit id: 3463f5a8a
  • Installation option (single install/HA): single node

Cluster information

  • Cluster type (Hosted/Infrastructure Provider/Custom/Imported): rke do
@sowmyav27 sowmyav27 added the kind/bug-qa Issues that have not yet hit a real release. Bugs introduced by a new feature or enhancement label Dec 10, 2020
@sowmyav27 sowmyav27 added this to the v2.5.4 milestone Dec 10, 2020
This was referenced Dec 11, 2020
Merged
Merged
Merged
@zube zube bot removed the [zube]: Next Up label Dec 12, 2020
@sowmyav27
Copy link
Contributor Author

On 2.5-head - commit id: e3f52647a, master-head - commit id: e971e0735 and 2.4-head - commit id: 0584e1798

  • Deploy a node driver cluster - 1 etcd/ 1 control and 1 worker
  • When the node is in provisioning state(I gave it a minute before deleting the node), delete the worker node.
  • No panic seen, node gets removed and re provisions

@superseb superseb mentioned this issue Dec 28, 2020
Closed
@zube zube bot removed the [zube]: Done label Mar 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pennyscissors pennyscissors

@sowmyav27 sowmyav27

Labels
kind/bug-qa Issues that have not yet hit a real release. Bugs introduced by a new feature or enhancement
Projects
None yet
Milestone
v2.5.4
Development

No branches or pull requests
3 participants
@maggieliu @pennyscissors @sowmyav27