Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Changing app-protocol from HTTP2 to other protocols will cause glbc to hit nil pointer #675

Closed
freehan opened this issue Mar 11, 2019 · 3 comments
Closed

Changing app-protocol from HTTP2 to other protocols will cause glbc to hit nil pointer #675

freehan opened this issue Mar 11, 2019 · 3 comments
Assignees
@freehan
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness.

Comments

@freehan
Copy link
Contributor

freehan commented Mar 11, 2019 •
edited
Loading

Symptom: Create an ingress with HTTP2 app protocol. For instance: https://github.com/kubernetes/kubernetes/tree/master/test/e2e/testing-manifests/ingress/http2
Manually change the app protocol on the spec of service from HTTP2 to something like HTTPS,HTTP. This will cause glbc to hit nil pointer and crash with following stacktrace in ingress-gce 1.2:

/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:573
/usr/local/go/src/runtime/panic.go:502
/usr/local/go/src/runtime/panic.go:63
/usr/local/go/src/runtime/signal_unix.go:388
/go/src/k8s.io/ingress-gce/pkg/healthchecks/healthchecks.go:332
/go/src/k8s.io/ingress-gce/pkg/healthchecks/healthchecks.go:244
/go/src/k8s.io/ingress-gce/pkg/healthchecks/healthchecks.go:106
/go/src/k8s.io/ingress-gce/pkg/backends/backends.go:220
/go/src/k8s.io/ingress-gce/pkg/backends/backends.go:283
/go/src/k8s.io/ingress-gce/pkg/backends/backends.go:252
/go/src/k8s.io/ingress-gce/pkg/controller/cluster_manager.go:98
/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:349
/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:299
/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:111
/go/src/k8s.io/ingress-gce/pkg/utils/taskqueue.go:84
/go/src/k8s.io/ingress-gce/pkg/utils/taskqueue.go:54
/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/go/src/k8s.io/ingress-gce/pkg/utils/taskqueue.go:54
/usr/local/go/src/runtime/asm_amd64.s:2361
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1674e63]

goroutine 99 [running]:
k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107
panic(0x19a12a0, 0x29aa020)
	/usr/local/go/src/runtime/panic.go:502 +0x229
k8s.io/ingress-gce/pkg/healthchecks.NewHealthCheck(...)
	/go/src/k8s.io/ingress-gce/pkg/healthchecks/healthchecks.go:332
k8s.io/ingress-gce/pkg/healthchecks.(*HealthChecks).Get(0xc42007bec0, 0xc420a8e3e0, 0x1e, 0x1cda028, 0x2, 0x0, 0x0, 0x0)
	/go/src/k8s.io/ingress-gce/pkg/healthchecks/healthchecks.go:244 +0x313
k8s.io/ingress-gce/pkg/healthchecks.(*HealthChecks).Sync(0xc42007bec0, 0xc42068b880, 0xc, 0xc420518c10, 0xb, 0x0)
	/go/src/k8s.io/ingress-gce/pkg/healthchecks/healthchecks.go:106 +0xae
k8s.io/ingress-gce/pkg/backends.(*Backends).ensureHealthCheck(0xc420303960, 0xc420518b90, 0xc, 0xc420518c10, 0xb, 0x0, 0x1bb, 0x0, 0x0, 0x7f21, ...)
	/go/src/k8s.io/ingress-gce/pkg/backends/backends.go:220 +0xc0
k8s.io/ingress-gce/pkg/backends.(*Backends).ensureBackendService(0xc420303960, 0xc420518b90, 0xc, 0xc420518c10, 0xb, 0x0, 0x1bb, 0x0, 0x0, 0x7f21, ...)
	/go/src/k8s.io/ingress-gce/pkg/backends/backends.go:283 +0x233
k8s.io/ingress-gce/pkg/backends.(*Backends).Ensure(0xc420303960, 0xc4208ab980, 0x1, 0x1, 0xc4203d1b30, 0x1, 0x1, 0x1670ad8, 0x184a100)
	/go/src/k8s.io/ingress-gce/pkg/backends/backends.go:252 +0x1fd
k8s.io/ingress-gce/pkg/controller.(*ClusterManager).EnsureLoadBalancer(0xc4208f7680, 0xc4203f9b20, 0xc4208aa380, 0x1, 0x1, 0xc4203d1b30, 0x1, 0x1, 0x1, 0x1)
	/go/src/k8s.io/ingress-gce/pkg/controller/cluster_manager.go:98 +0x1ed
k8s.io/ingress-gce/pkg/controller.(*LoadBalancerController).ensureIngress(0xc4204405a0, 0xc4208b7a20, 0xc420771c00, 0x3, 0x4, 0xc4208aa280, 0x1, 0x1, 0x0, 0x0)
	/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:349 +0x4b6
k8s.io/ingress-gce/pkg/controller.(*LoadBalancerController).sync(0xc4204405a0, 0xc42039dd80, 0x14, 0xc420699b01, 0xc420472d00)
	/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:299 +0x3e5
k8s.io/ingress-gce/pkg/controller.(*LoadBalancerController).(k8s.io/ingress-gce/pkg/controller.sync)-fm(0xc42039dd80, 0x14, 0xf, 0xc4208dddd8)
	/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:111 +0x3e
k8s.io/ingress-gce/pkg/utils.(*PeriodicTaskQueue).worker(0xc420400240)
	/go/src/k8s.io/ingress-gce/pkg/utils/taskqueue.go:84 +0x17a
k8s.io/ingress-gce/pkg/utils.(*PeriodicTaskQueue).(k8s.io/ingress-gce/pkg/utils.worker)-fm()
	/go/src/k8s.io/ingress-gce/pkg/utils/taskqueue.go:54 +0x2a
k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc42072dfa8)
	/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc420a1ffa8, 0x3b9aca00, 0x0, 0x1, 0xc4200848a0)
	/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd
k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc42072dfa8, 0x3b9aca00, 0xc4200848a0)
	/go/src/k8s.io/ingress-gce/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
k8s.io/ingress-gce/pkg/utils.(*PeriodicTaskQueue).Run(0xc420400240, 0x3b9aca00, 0xc4200848a0)
	/go/src/k8s.io/ingress-gce/pkg/utils/taskqueue.go:54 +0x55
created by k8s.io/ingress-gce/pkg/controller.(*LoadBalancerController).Run
	/go/src/k8s.io/ingress-gce/pkg/controller/controller.go:231 +0x86

Root Cause:

  1. GCE GetHealthCheck V1 API used to return the health check type to "HTTP" while the healthcheck object is create beta API using "HTTP2".

  2. Now HTTP2 is promoted to GA on GCE. GCE GetHealthCheck V1 API returns health check type to "HTTP2".

  3. The GCE go client used by ingress-gce does not have the new HTTP2 fields in the V1 Healthcheck API.

  4. In case 1, this path is hit https://github.com/kubernetes/ingress-gce/blob/release-1.2/pkg/healthchecks/healthchecks.go#L328

  5. With 2, this path is hit https://github.com/kubernetes/ingress-gce/blob/release-1.2/pkg/healthchecks/healthchecks.go#L332 hc.Http2HealthCheck is nil because the V1 healthcheck object does not contain HTTP2HealthCheck fields due to 3

Impacted Versions: 1.2, 1.3, 1.4, 1.5, HEAD
@freehan freehan mentioned this issue Mar 11, 2019
Closed
@rramkumar1 rramkumar1 added the kind/bug Categorizes issue or PR as related to a bug. label Mar 11, 2019
@freehan freehan mentioned this issue Mar 12, 2019
Merged
@MrHohn MrHohn mentioned this issue Mar 12, 2019
Closed
@freehan freehan mentioned this issue Mar 15, 2019
Merged
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 12, 2019
@bowei
Copy link
Member

bowei commented Jun 12, 2019

/lifecycle frozen

@k8s-ci-robot k8s-ci-robot added lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jun 12, 2019
@rramkumar1
Copy link
Contributor

Closing since this issue is fully mitigated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@freehan freehan

Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@bowei @freehan @k8s-ci-robot @rramkumar1 @fejta-bot