Enabling MTLS between Nginx Ingress Controller and Istio

[robincher]

Bug description
This is similar to Issue #18176

To illustrate what i am trying to achieve, you can check out a simple context diagram i have drawn.

As you can see, i am able to enable and successfully encrypt traffics within the mesh, but interestingly, i will get some ssl error between nginx controller to the service after enforcing mTLS STRICT mode on namespace : alpha.

This may not be technically a bug, but what i observe is that Nginx controller pod, despite being injected with Envoy's side car, is unable to forward "X-Forwarded-Client-Cert" header to the upstream service.

Is there some work around for this?
I was attempting this based on https://www.tetrate.io/blog/using-istio-with-other-ingress-proxies/ too, but i wonder if its still applicable for Istio 1.6.x and onwards.

[X] Configuration Infrastructure
[X] Docs
[ ] Installation
[X] Networking
[ ] Performance and Scalability
[X] Policies and Telemetry
[X] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Expected behavior

1. Nginx Controller pod should auto inject the client certificate towards the upstream service.

Steps to reproduce the bug

1. Create a EKS cluster

2. Install Nginx-ingress controller and Istio 1.6.1

3. Install sleep/httpbin test services into foo namespace

4. Install sleep into ingress-nginx namespace

5. Apply Peer Authentication Policy to foo namespace and set MODE as STRICT

6. TCP Dump on httpbin service
   kubectl exec -nfoo "$(kubectl get pod -nfoo -lapp=httpbin -ojsonpath={.items..metadata.name})" -c istio-proxy -it -- sudo tcpdump dst port 80 -A

7. Testing the service

a. Make a request from external browser (You may need to set up AWS ALB and nodeports based on your test env)
b. Make a request from sleep service in foo namespace to httpbin service
c. Make a request from sleep service in ingress-nginx namespace httpbin service

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

Kubernetes Version : 1.16
Istio Version : 1.6.1
nginx-ingress-controller: 0.30.0

How was Istio installed?

istioctl install --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY --set meshConfig.enableAutoMtls=true   --set values.global.proxy.privileged=true --set profile=default

Environment where bug was observed (cloud vendor, OS, etc)

1. AWS EKS

dump.txt

[satishmane]

@robincher
Hi,
I am facing similar issue. I too have nginx pod inside istio mesh which receives external traffic, but fails with 503 error to forward to request to another service inside mesh which is in MTLS strict mode. My nginx has proxy.conf to upgrade connection, modify header, to rewrite uri etc. The url in nginx proxy.conf points to kubernetes external name type service which has destination service full url. I applied destination rule with istio-mutual to this external name type service, but no luck

[satishmane]

@robincher

If I change proxy_set header as below, then it works. Got clue from here - #14450 (comment)

current rule
proxy_set_header Host $http_host;

changed to
proxy_set_header Host clusterIP service name : port number; e.g. foo-service:8080

[robincher]

Hi @satishmane thanks!

You set this at the ingress controller i presumed? Or at individual ingress object.

[satishmane]

I am not using kubernetes nginx ingress controller or ingress object. I have nginx docker image which is deployed as pod with istio sidecar with rules defined in proxy.conf

[wapdat]

I have/had exactly the same problem - getting mTLS to work from ingress-nginx with Istio. I added these ingress-nginx annotations and it seems to have fixed it (my main use case is getting authorisation rules to work for segregation)

nginx.ingress.kubernetes.io/service-upstream: "true"
nginx.ingress.kubernetes.io/upstream-vhost: your-app.the-namespace.svc.cluster.local

[howardjohn]

Seems this is resolved based on the comments above #14450 (comment).

There is a feature tracked that would make this work OOTB but its not likely to be implemented soon. Tracking in #23494