Cloud controller crashes when using load balancer with certificates #116

pavle-j4nk · 2020-11-11T10:41:48Z

cloud-controller-manager worked fine without load-balancer.hetzner.cloud/http-certificates: my-cert annotation on load balancer. After it got annotated it keeps crashing.

cloud-controller-manager logs:

I1111 10:31:55.442401       1 load_balancers.go:81] "ensure Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" service="ingress-nginx-controller" nodes=[worker1]
I1111 10:31:55.442765       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"810f96cf-7ff1-4244-8ee9-9ea34c8d8a6a", APIVersion:"v1", ResourceVersion:"1609817", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
I1111 10:31:55.695614       1 load_balancer.go:439] "update service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=80 loadBalancerID=127243
E1111 10:31:55.695862       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 188 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x192fb40, 0x2912e60)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.8/pkg/util/runtime/runtime.go:74 +0xa6
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.8/pkg/util/runtime/runtime.go:48 +0x89
panic(0x192fb40, 0x2912e60)
	/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/hetznercloud/hcloud-go/hcloud.loadBalancerUpdateServiceOptsToSchema(0x1b8089e, 0x3, 0xc0006e2610, 0x0, 0xc0009662c0, 0xc000966300, 0x0, 0x0, 0x0, 0x0, ...)
	/go/pkg/mod/github.com/hetznercloud/hcloud-go@v1.22.0/hcloud/schema.go:860 +0x1a6
github.com/hetznercloud/hcloud-go/hcloud.(*LoadBalancerClient).UpdateService(0xc00054e288, 0x1dd4aa0, 0xc000046030, 0xc0006afb00, 0x50, 0x1b8089e, 0x3, 0xc0006e2610, 0x0, 0xc0009662c0, ...)
	/go/pkg/mod/github.com/hetznercloud/hcloud-go@v1.22.0/hcloud/load_balancer.go:708 +0x5d
github.com/hetznercloud/hcloud-cloud-controller-manager/internal/hcops.(*LoadBalancerOps).ReconcileHCLBServices(0xc000b1ecc0, 0x1dd4aa0, 0xc000046030, 0xc0006afb00, 0xc000021720, 0xc0003ef3c8, 0x1, 0x1)
	/maschine-controller/src/internal/hcops/load_balancer.go:445 +0x5ce
github.com/hetznercloud/hcloud-cloud-controller-manager/hcloud.(*loadBalancers).EnsureLoadBalancer(0xc000b5bda0, 0x1dd4aa0, 0xc000046030, 0x1b88653, 0xa, 0xc000021720, 0xc0003ef3c8, 0x1, 0x1, 0x1b82877, ...)
	/maschine-controller/src/hcloud/load_balancers.go:110 +0x5e2
k8s.io/kubernetes/pkg/controller/service.(*Controller).ensureLoadBalancer(0xc000b3ee00, 0xc000021720, 0x0, 0x0, 0x6)
	/go/pkg/mod/k8s.io/kubernetes@v1.18.3/pkg/controller/service/controller.go:389 +0xde
k8s.io/kubernetes/pkg/controller/service.(*Controller).syncLoadBalancerIfNeeded(0xc000b3ee00, 0xc000021720, 0xc0008d1ce0, 0x26, 0xc00073fcb0, 0x1526313, 0x1df0360)
	/go/pkg/mod/k8s.io/kubernetes@v1.18.3/pkg/controller/service/controller.go:344 +0x889
k8s.io/kubernetes/pkg/controller/service.(*Controller).processServiceCreateOrUpdate(0xc000b3ee00, 0xc000021720, 0xc0008d1ce0, 0x26, 0x0, 0x0)
	/go/pkg/mod/k8s.io/kubernetes@v1.18.3/pkg/controller/service/controller.go:280 +0xd9
k8s.io/kubernetes/pkg/controller/service.(*Controller).syncService(0xc000b3ee00, 0xc0008d1ce0, 0x26, 0x0, 0x0)
	/go/pkg/mod/k8s.io/kubernetes@v1.18.3/pkg/controller/service/controller.go:759 +0x31a
k8s.io/kubernetes/pkg/controller/service.(*Controller).processNextWorkItem(0xc000b3ee00, 0x203000)
	/go/pkg/mod/k8s.io/kubernetes@v1.18.3/pkg/controller/service/controller.go:238 +0xf5
k8s.io/kubernetes/pkg/controller/service.(*Controller).worker(0xc000b3ee00)
	/go/pkg/mod/k8s.io/kubernetes@v1.18.3/pkg/controller/service/controller.go:227 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000842c00)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.8/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000842c00, 0x1d8e4c0, 0xc0007d2240, 0x1, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.8/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000842c00, 0x3b9aca00, 0x0, 0x1, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.8/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc000842c00, 0x3b9aca00, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.8/pkg/util/wait/wait.go:90 +0x4d
created by k8s.io/kubernetes/pkg/controller/service.(*Controller).Run
	/go/pkg/mod/k8s.io/kubernetes@v1.18.3/pkg/controller/service/controller.go:216 +0x20e
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1755c06]

goroutine 188 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.8/pkg/util/runtime/runtime.go:55 +0x10c
panic(0x192fb40, 0x2912e60)
	/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/hetznercloud/hcloud-go/hcloud.loadBalancerUpdateServiceOptsToSchema(0x1b8089e, 0x3, 0xc0006e2610, 0x0, 0xc0009662c0, 0xc000966300, 0x0, 0x0, 0x0, 0x0, ...)
	/go/pkg/mod/github.com/hetznercloud/hcloud-go@v1.22.0/hcloud/schema.go:860 +0x1a6
github.com/hetznercloud/hcloud-go/hcloud.(*LoadBalancerClient).UpdateService(0xc00054e288, 0x1dd4aa0, 0xc000046030, 0xc0006afb00, 0x50, 0x1b8089e, 0x3, 0xc0006e2610, 0x0, 0xc0009662c0, ...)
	/go/pkg/mod/github.com/hetznercloud/hcloud-go@v1.22.0/hcloud/load_balancer.go:708 +0x5d
github.com/hetznercloud/hcloud-cloud-controller-manager/internal/hcops.(*LoadBalancerOps).ReconcileHCLBServices(0xc000b1ecc0, 0x1dd4aa0, 0xc000046030, 0xc0006afb00, 0xc000021720, 0xc0003ef3c8, 0x1, 0x1)
	/maschine-controller/src/internal/hcops/load_balancer.go:445 +0x5ce
github.com/hetznercloud/hcloud-cloud-controller-manager/hcloud.(*loadBalancers).EnsureLoadBalancer(0xc000b5bda0, 0x1dd4aa0, 0xc000046030, 0x1b88653, 0xa, 0xc000021720, 0xc0003ef3c8, 0x1, 0x1, 0x1b82877, ...)
	/maschine-controller/src/hcloud/load_balancers.go:110 +0x5e2
k8s.io/kubernetes/pkg/controller/service.(*Controller).ensureLoadBalancer(0xc000b3ee00, 0xc000021720, 0x0, 0x0, 0x6)
	/go/pkg/mod/k8s.io/kubernetes@v1.18.3/pkg/controller/service/controller.go:389 +0xde
k8s.io/kubernetes/pkg/controller/service.(*Controller).syncLoadBalancerIfNeeded(0xc000b3ee00, 0xc000021720, 0xc0008d1ce0, 0x26, 0xc00073fcb0, 0x1526313, 0x1df0360)
	/go/pkg/mod/k8s.io/kubernetes@v1.18.3/pkg/controller/service/controller.go:344 +0x889
k8s.io/kubernetes/pkg/controller/service.(*Controller).processServiceCreateOrUpdate(0xc000b3ee00, 0xc000021720, 0xc0008d1ce0, 0x26, 0x0, 0x0)
	/go/pkg/mod/k8s.io/kubernetes@v1.18.3/pkg/controller/service/controller.go:280 +0xd9
k8s.io/kubernetes/pkg/controller/service.(*Controller).syncService(0xc000b3ee00, 0xc0008d1ce0, 0x26, 0x0, 0x0)
	/go/pkg/mod/k8s.io/kubernetes@v1.18.3/pkg/controller/service/controller.go:759 +0x31a
k8s.io/kubernetes/pkg/controller/service.(*Controller).processNextWorkItem(0xc000b3ee00, 0x203000)
	/go/pkg/mod/k8s.io/kubernetes@v1.18.3/pkg/controller/service/controller.go:238 +0xf5
k8s.io/kubernetes/pkg/controller/service.(*Controller).worker(0xc000b3ee00)
	/go/pkg/mod/k8s.io/kubernetes@v1.18.3/pkg/controller/service/controller.go:227 +0x2b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000842c00)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.8/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000842c00, 0x1d8e4c0, 0xc0007d2240, 0x1, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.8/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000842c00, 0x3b9aca00, 0x0, 0x1, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.8/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(0xc000842c00, 0x3b9aca00, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.18.8/pkg/util/wait/wait.go:90 +0x4d
created by k8s.io/kubernetes/pkg/controller/service.(*Controller).Run
	/go/pkg/mod/k8s.io/kubernetes@v1.18.3/pkg/controller/service/controller.go:216 +0x20e

Load Balancer:

Name:                     ingress-nginx-controller
Namespace:                ingress-nginx
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/version=0.40.2
                          helm.sh/chart=ingress-nginx-3.4.1
Annotations:              load-balancer.hetzner.cloud/http-certificates: my-cert
                          load-balancer.hetzner.cloud/location: fsn1
                          load-balancer.hetzner.cloud/name: my-lb
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP:                       10.109.150.119
LoadBalancer Ingress:     49.12.18.110
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  30060/TCP
Endpoints:                10.244.1.46:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  31128/TCP
Endpoints:                10.244.1.46:443
Session Affinity:         None
External Traffic Policy:  Cluster

The text was updated successfully, but these errors were encountered:

ByteAlex · 2020-11-11T14:20:56Z

Is that my-cert available as a secret in that namespace?

pavle-j4nk · 2020-11-11T14:42:43Z

Yes, my-cert secret of type kubernetes.io/tls exists in that namespace. I also have tried creating that secret in both cloud-controller-manager's namespace and in load balancer's namespace.

Name:         my-cert
Namespace:    ingress-nginx
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
tls.crt:  1915 bytes
tls.key:  1704 bytes

LKaemmerling · 2020-11-11T18:49:10Z

Is the certificate with that name created in the Cloud console? The cloud controller can‘t access your secrets and therefore it can not create the certificate. You need to add a already existing certificate. That it crashes is indeed not nice and I will look into why tomorrow :)

LKaemmerling · 2020-11-12T06:10:53Z

@fhofherr could you please look into this? It looks like we only accept the cert IDs and if someone enters something others this exception is raised.

pavle-j4nk · 2020-11-12T06:30:32Z

I have added existing certificate named my-cert to cloud console and still, controller keeps crashing.

LKaemmerling · 2020-11-12T06:32:06Z

I have added existing certificate named my-cert to cloud console and still, controller keeps crashing.

@pavle-j4nk Could you try to use the ID of the cert? It looks like the code only supports using the ID at the moment.

pavle-j4nk · 2020-11-12T06:40:09Z

Works fine with ID. Thank you very much!

LKaemmerling assigned fhofherr Nov 12, 2020

LKaemmerling added the bug Something isn't working label Nov 12, 2020

fhofherr mentioned this issue Nov 23, 2020

Reference certificates by name #124

Merged

fhofherr closed this as completed in #124 Nov 23, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cloud controller crashes when using load balancer with certificates #116

Cloud controller crashes when using load balancer with certificates #116

pavle-j4nk commented Nov 11, 2020

ByteAlex commented Nov 11, 2020

pavle-j4nk commented Nov 11, 2020

LKaemmerling commented Nov 11, 2020

LKaemmerling commented Nov 12, 2020

pavle-j4nk commented Nov 12, 2020

LKaemmerling commented Nov 12, 2020

pavle-j4nk commented Nov 12, 2020 •

edited

Cloud controller crashes when using load balancer with certificates #116

Cloud controller crashes when using load balancer with certificates #116

Comments

pavle-j4nk commented Nov 11, 2020

ByteAlex commented Nov 11, 2020

pavle-j4nk commented Nov 11, 2020

LKaemmerling commented Nov 11, 2020

LKaemmerling commented Nov 12, 2020

pavle-j4nk commented Nov 12, 2020

LKaemmerling commented Nov 12, 2020

pavle-j4nk commented Nov 12, 2020 • edited

pavle-j4nk commented Nov 12, 2020 •

edited