Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

daemon: fix panic when starting Cilium #12101

Merged
merged 1 commit into from Jun 16, 2020

Conversation

aanm
Copy link
Member

@aanm aanm commented Jun 16, 2020

It seems the we try to validate an endpoint without setting its
allocator which causes Cilium to panic while restoring old endpoints:

goroutine 259 [running]:
github.com/cilium/cilium/pkg/endpoint.(*Endpoint).identityLabelsChanged(0xc000b122c0, 0x280e920, 0xc0009d2840, 0x1, 0x0, 0x0, 0x0)
        /go/src/github.com/cilium/cilium/pkg/endpoint/endpoint.go:1818 +0x45c
github.com/cilium/cilium/pkg/endpoint.(*Endpoint).runIdentityResolver(0xc000b122c0, 0x280e920, 0xc0009d2840, 0x1, 0x1, 0xc000cb2d80)
        /go/src/github.com/cilium/cilium/pkg/endpoint/endpoint.go:1748 +0x3db
github.com/cilium/cilium/pkg/endpoint.(*Endpoint).UpdateLabels(0xc000b122c0, 0x280e920, 0xc0009d2840, 0xc000d2ca50, 0xc000d2cab0, 0x1, 0x6)
        /go/src/github.com/cilium/cilium/pkg/endpoint/endpoint.go:1709 +0x3d0
github.com/cilium/cilium/pkg/endpoint.(*Endpoint).RunMetadataResolver.func2(0x280e920, 0xc0009d2840, 0x3b4b200, 0xc000ce31b8)
        /go/src/github.com/cilium/cilium/pkg/endpoint/endpoint.go:1621 +0x46a
github.com/cilium/cilium/pkg/controller.(*Controller).runController(0xc0000d6800)
        /go/src/github.com/cilium/cilium/pkg/controller/controller.go:205 +0xa2a
created by github.com/cilium/cilium/pkg/controller.(*Manager).updateController
        /go/src/github.com/cilium/cilium/pkg/controller/manager.go:120 +0xb09

With this change we set the allocator in the endpoint before we validate
the endpoint, avoiding it to panic.

Signed-off-by: André Martins andre@cilium.io

It seems the we try to validate an endpoint without setting its
allocator which causes Cilium to panic while restoring old endpoints:

```
goroutine 259 [running]:
github.com/cilium/cilium/pkg/endpoint.(*Endpoint).identityLabelsChanged(0xc000b122c0, 0x280e920, 0xc0009d2840, 0x1, 0x0, 0x0, 0x0)
        /go/src/github.com/cilium/cilium/pkg/endpoint/endpoint.go:1818 +0x45c
github.com/cilium/cilium/pkg/endpoint.(*Endpoint).runIdentityResolver(0xc000b122c0, 0x280e920, 0xc0009d2840, 0x1, 0x1, 0xc000cb2d80)
        /go/src/github.com/cilium/cilium/pkg/endpoint/endpoint.go:1748 +0x3db
github.com/cilium/cilium/pkg/endpoint.(*Endpoint).UpdateLabels(0xc000b122c0, 0x280e920, 0xc0009d2840, 0xc000d2ca50, 0xc000d2cab0, 0x1, 0x6)
        /go/src/github.com/cilium/cilium/pkg/endpoint/endpoint.go:1709 +0x3d0
github.com/cilium/cilium/pkg/endpoint.(*Endpoint).RunMetadataResolver.func2(0x280e920, 0xc0009d2840, 0x3b4b200, 0xc000ce31b8)
        /go/src/github.com/cilium/cilium/pkg/endpoint/endpoint.go:1621 +0x46a
github.com/cilium/cilium/pkg/controller.(*Controller).runController(0xc0000d6800)
        /go/src/github.com/cilium/cilium/pkg/controller/controller.go:205 +0xa2a
created by github.com/cilium/cilium/pkg/controller.(*Manager).updateController
        /go/src/github.com/cilium/cilium/pkg/controller/manager.go:120 +0xb09
```

With this change we set the allocator in the endpoint before we validate
the endpoint, avoiding it to panic.

Signed-off-by: André Martins <andre@cilium.io>
@aanm aanm added priority/release-blocker release-note/bug This PR fixes an issue in a previous release of Cilium. kind/regression This functionality worked fine before, but was broken in a newer release of Cilium. labels Jun 16, 2020
@aanm aanm requested a review from a team as a code owner June 16, 2020 13:09
@maintainer-s-little-helper maintainer-s-little-helper bot added this to In progress in 1.8.0 Jun 16, 2020
@aanm
Copy link
Member Author

aanm commented Jun 16, 2020

test-me-please

@aanm aanm mentioned this pull request Jun 16, 2020
20 tasks
@aanm
Copy link
Member Author

aanm commented Jun 16, 2020

retest-4.19

Copy link
Member

@joestringer joestringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a reliable failure? Just wondering when we broke it and whether it could affect earlier releases.

@maintainer-s-little-helper maintainer-s-little-helper bot added this to Needs backport from master in 1.8.0 Jun 16, 2020
@aanm
Copy link
Member Author

aanm commented Jun 16, 2020

@joestringer I hit this by running #11903 (comment) and restarting Cilium pod. Not sure if it affects older versions. I have a feeling that #11714 might have introduced this regression.

@joestringer joestringer merged commit df97ca1 into cilium:master Jun 16, 2020
1.8.0 automation moved this from In progress to Merged Jun 16, 2020
@aanm aanm deleted the pr/fix-allocator-panic branch June 17, 2020 11:55
@maintainer-s-little-helper maintainer-s-little-helper bot moved this from Needs backport from master to Backport pending to v1.8 in 1.8.0 Jun 18, 2020
@maintainer-s-little-helper maintainer-s-little-helper bot moved this from Needs backport from master to Backport pending to v1.8 in 1.8.0 Jun 18, 2020
@maintainer-s-little-helper maintainer-s-little-helper bot moved this from Backport pending to v1.8 to Backport done to v1.8 in 1.8.0 Jun 18, 2020
@maintainer-s-little-helper maintainer-s-little-helper bot moved this from Backport pending to v1.8 to Backport done to v1.8 in 1.8.0 Jun 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/regression This functionality worked fine before, but was broken in a newer release of Cilium. release-note/bug This PR fixes an issue in a previous release of Cilium.
Projects
No open projects
1.8.0
  
Merged
1.8.0
Backport done to v1.8
Development

Successfully merging this pull request may close these issues.

None yet

5 participants