Beta mTLS segfault in Pilot

[howardjohn]

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x17b791c]

goroutine 735 [running]:
istio.io/istio/pilot/pkg/model.(*AuthenticationPolicies).GetNamespaceMutualTLSMode(...)
        istio.io/istio/pilot/pkg/model/authentication.go:191
istio.io/istio/pilot/pkg/model.(*PushContext).BestEffortInferServiceMTLSMode(0xc000e07e00, 0xc0010160f0, 0xc000e44420, 0x25fd5e8)
        istio.io/istio/pilot/pkg/model/push_context.go:1832 +0x4c
istio.io/istio/pilot/pkg/networking/core/v1alpha3.(*ConfigGeneratorImpl).buildOutboundClusters(0xc000595760, 0xc001766580, 0xc000e07e00, 0x408dda0, 0xd, 0x25fd9c8)
        istio.io/istio/pilot/pkg/networking/core/v1alpha3/cluster.go:205 +0x587
istio.io/istio/pilot/pkg/networking/core/v1alpha3.(*ConfigGeneratorImpl).BuildClusters(0xc000595760, 0xc001766580, 0xc000e07e00, 0x13, 0x25fd9d8, 0x8)
        istio.io/istio/pilot/pkg/networking/core/v1alpha3/cluster.go:102 +0xaf
istio.io/istio/pilot/pkg/proxy/envoy/v2.(*DiscoveryServer).generateRawClusters(0xc000916f00, 0xc001766580, 0xc000e07e00, 0x2603b23, 0xc, 0x2612bb0)
        istio.io/istio/pilot/pkg/proxy/envoy/v2/cds.go:73 +0x63
istio.io/istio/pilot/pkg/proxy/envoy/v2.(*DiscoveryServer).pushCds(0xc000916f00, 0xc001492480, 0xc000e07e00, 0xc001145880, 0x16, 0x1, 0xc0004fd528)
        istio.io/istio/pilot/pkg/proxy/envoy/v2/cds.go:51 +0x83
istio.io/istio/pilot/pkg/proxy/envoy/v2.(*DiscoveryServer).pushConnection(0xc000916f00, 0xc001492480, 0xc00156a4b0, 0x1, 0x1)
        istio.io/istio/pilot/pkg/proxy/envoy/v2/ads.go:544 +0x5e7
istio.io/istio/pilot/pkg/proxy/envoy/v2.(*DiscoveryServer).StreamAggregatedResources(0xc000916f00, 0x2a21b00, 0xc000f783a0, 0x0, 0x0)
        istio.io/istio/pilot/pkg/proxy/envoy/v2/ads.go:382 +0x268d
github.com/envoyproxy/go-control-plane/envoy/service/discovery/v2._AggregatedDiscoveryService_StreamAggregatedResources_Handler(0x25eafe0, 0xc000916f00, 0x2a17000, 0xc00176a180, 0x408dda0, 0xc00177a000)
        github.com/envoyproxy/go-control-plane@v0.9.4/envoy/service/discovery/v2/ads.pb.go:194 +0xad
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00023ec00, 0x2a273e0, 0xc001492300, 0xc00177a000, 0xc0007ad590, 0x4033920, 0xc00172ea80, 0x0, 0x0)
        google.golang.org/grpc@v1.26.0/server.go:1237 +0xcd1
google.golang.org/grpc.(*Server).handleStream(0xc00023ec00, 0x2a273e0, 0xc001492300, 0xc00177a000, 0xc00172ea80)
        google.golang.org/grpc@v1.26.0/server.go:1317 +0xcd6
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00175a6c0, 0xc00023ec00, 0x2a273e0, 0xc001492300, 0xc00177a000)
        google.golang.org/grpc@v1.26.0/server.go:722 +0xa1
created by google.golang.org/grpc.(*Server).serveStreams.func1
        google.golang.org/grpc@v1.26.0/server.go:720 +0xa1

Running from master

Looking into this now, will see if I can reproduce and if it applies to 1.5

cc @fpesce @diemtvu

[howardjohn]

I can reproduce. I am developing so there are some tiny changes i have made to pilot, but it seems unrelated

[diemtvu]

Thanks for the report. What is the setup and policies you have in the system? Thanks.

…

On Wed, Mar 4, 2020 at 9:45 AM John Howard ***@***.***> wrote: I can reproduce. I am developing so there are some tiny changes i have made to pilot, but it seems unrelated — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21816?email_source=notifications&email_token=AF7X24KUPXLZRKQVXH7VJBTRF2HTHA5CNFSM4LBOVKF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENZFQ4A#issuecomment-594696304>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF7X24KCRYSGCMHDISTQZY3RF2HTHANCNFSM4LBOVKFQ> .

-- Diem Vu | Software Engineer | diemvu@google.com | +1 408-215-8127

[howardjohn]

I have no policies. I am trying to figure out the root cause here, not sure what is triggering it. I am adding a new integration test and for whatever reason it fails immediately after the test exits. It could be an issue on my end

[howardjohn]

found the issue: #21818

[diemtvu]

Cool, I was about to dig in to see when the policy is not initialize properly within push_context.

[howardjohn]

The impact here is that any error in push context initialization such as env.List(VirtualService), which may fail due to networking errors, etc, will lead to a panic

[diemtvu]

If the push context return err, does it gracefully skip that config snapshot then retry a few minutes later?

…

On Wed, Mar 4, 2020 at 10:31 AM John Howard ***@***.***> wrote: The impact here is that any error in push context initialization such as env.List(VirtualService), which may fail due to networking errors, etc, will lead to a panic — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21816?email_source=notifications&email_token=AF7X24OAKNU5KGSPOIJYV6DRF2NARA5CNFSM4LBOVKF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENZORFA#issuecomment-594733204>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF7X24NLDA7DEUQP2Y3IJW3RF2NARANCNFSM4LBOVKFQ> .

-- Diem Vu | Software Engineer | diemvu@google.com | +1 408-215-8127

[howardjohn]

Well right now it crashes. The intent is that it will skip the update