Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Beta mTLS segfault in Pilot #21816

Closed
howardjohn opened this issue Mar 4, 2020 · 8 comments
Closed

Beta mTLS segfault in Pilot #21816

howardjohn opened this issue Mar 4, 2020 · 8 comments
Labels
Milestone

Comments

@howardjohn
Copy link
Member

@howardjohn howardjohn commented Mar 4, 2020

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x17b791c]

goroutine 735 [running]:
istio.io/istio/pilot/pkg/model.(*AuthenticationPolicies).GetNamespaceMutualTLSMode(...)
        istio.io/istio/pilot/pkg/model/authentication.go:191
istio.io/istio/pilot/pkg/model.(*PushContext).BestEffortInferServiceMTLSMode(0xc000e07e00, 0xc0010160f0, 0xc000e44420, 0x25fd5e8)
        istio.io/istio/pilot/pkg/model/push_context.go:1832 +0x4c
istio.io/istio/pilot/pkg/networking/core/v1alpha3.(*ConfigGeneratorImpl).buildOutboundClusters(0xc000595760, 0xc001766580, 0xc000e07e00, 0x408dda0, 0xd, 0x25fd9c8)
        istio.io/istio/pilot/pkg/networking/core/v1alpha3/cluster.go:205 +0x587
istio.io/istio/pilot/pkg/networking/core/v1alpha3.(*ConfigGeneratorImpl).BuildClusters(0xc000595760, 0xc001766580, 0xc000e07e00, 0x13, 0x25fd9d8, 0x8)
        istio.io/istio/pilot/pkg/networking/core/v1alpha3/cluster.go:102 +0xaf
istio.io/istio/pilot/pkg/proxy/envoy/v2.(*DiscoveryServer).generateRawClusters(0xc000916f00, 0xc001766580, 0xc000e07e00, 0x2603b23, 0xc, 0x2612bb0)
        istio.io/istio/pilot/pkg/proxy/envoy/v2/cds.go:73 +0x63
istio.io/istio/pilot/pkg/proxy/envoy/v2.(*DiscoveryServer).pushCds(0xc000916f00, 0xc001492480, 0xc000e07e00, 0xc001145880, 0x16, 0x1, 0xc0004fd528)
        istio.io/istio/pilot/pkg/proxy/envoy/v2/cds.go:51 +0x83
istio.io/istio/pilot/pkg/proxy/envoy/v2.(*DiscoveryServer).pushConnection(0xc000916f00, 0xc001492480, 0xc00156a4b0, 0x1, 0x1)
        istio.io/istio/pilot/pkg/proxy/envoy/v2/ads.go:544 +0x5e7
istio.io/istio/pilot/pkg/proxy/envoy/v2.(*DiscoveryServer).StreamAggregatedResources(0xc000916f00, 0x2a21b00, 0xc000f783a0, 0x0, 0x0)
        istio.io/istio/pilot/pkg/proxy/envoy/v2/ads.go:382 +0x268d
github.com/envoyproxy/go-control-plane/envoy/service/discovery/v2._AggregatedDiscoveryService_StreamAggregatedResources_Handler(0x25eafe0, 0xc000916f00, 0x2a17000, 0xc00176a180, 0x408dda0, 0xc00177a000)
        github.com/envoyproxy/go-control-plane@v0.9.4/envoy/service/discovery/v2/ads.pb.go:194 +0xad
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00023ec00, 0x2a273e0, 0xc001492300, 0xc00177a000, 0xc0007ad590, 0x4033920, 0xc00172ea80, 0x0, 0x0)
        google.golang.org/grpc@v1.26.0/server.go:1237 +0xcd1
google.golang.org/grpc.(*Server).handleStream(0xc00023ec00, 0x2a273e0, 0xc001492300, 0xc00177a000, 0xc00172ea80)
        google.golang.org/grpc@v1.26.0/server.go:1317 +0xcd6
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00175a6c0, 0xc00023ec00, 0x2a273e0, 0xc001492300, 0xc00177a000)
        google.golang.org/grpc@v1.26.0/server.go:722 +0xa1
created by google.golang.org/grpc.(*Server).serveStreams.func1
        google.golang.org/grpc@v1.26.0/server.go:720 +0xa1

Running from master

Looking into this now, will see if I can reproduce and if it applies to 1.5

cc @fpesce @diemtvu

@howardjohn

This comment has been minimized.

Copy link
Member Author

@howardjohn howardjohn commented Mar 4, 2020

I can reproduce. I am developing so there are some tiny changes i have made to pilot, but it seems unrelated

@diemtvu

This comment has been minimized.

Copy link
Contributor

@diemtvu diemtvu commented Mar 4, 2020

@howardjohn

This comment has been minimized.

Copy link
Member Author

@howardjohn howardjohn commented Mar 4, 2020

I have no policies. I am trying to figure out the root cause here, not sure what is triggering it. I am adding a new integration test and for whatever reason it fails immediately after the test exits. It could be an issue on my end

@howardjohn

This comment has been minimized.

Copy link
Member Author

@howardjohn howardjohn commented Mar 4, 2020

found the issue: #21818

@diemtvu

This comment has been minimized.

Copy link
Contributor

@diemtvu diemtvu commented Mar 4, 2020

Cool, I was about to dig in to see when the policy is not initialize properly within push_context.

@howardjohn

This comment has been minimized.

Copy link
Member Author

@howardjohn howardjohn commented Mar 4, 2020

The impact here is that any error in push context initialization such as env.List(VirtualService), which may fail due to networking errors, etc, will lead to a panic

@diemtvu

This comment has been minimized.

Copy link
Contributor

@diemtvu diemtvu commented Mar 4, 2020

@howardjohn

This comment has been minimized.

Copy link
Member Author

@howardjohn howardjohn commented Mar 4, 2020

Well right now it crashes. The intent is that it will skip the update

@dgn dgn added this to the 1.5 milestone Mar 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

3 participants
You can’t perform that action at this time.