Nginx reverse proxy - upstream name must match the service or it fails to route

[MrBlaise]

If I have a service called foo-bar and I create an upstream block like this:

upstream foo {
    zone foo_service 64k;
    server foo-bar:8080;
}

Then I proxy the request to this upstream like this:

location ~ ^/v1/bar$ {
    limit_except put delete options {
        deny all;
    }
    set $upstream foo;
    rewrite ^/v1/(.*)$ /_foo/$1 last;
}


location ~ ^/_foo.*$ {
    internal;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;

    rewrite ^/_foo/(.*)$ /$1 break;
    proxy_pass http://$upstream;
}

It will cause an infinite loop with istio, but if I change the name of the upstream to foo-bar instead of foo so it matches the name of the service it works fine.

This is quite a headache when I am using third party solutions like cloudendpoints esp proxy.

EDIT:

It seems like it is not an infinite loop (maybe just some retry logic?). I believe the problem is that istio-proxy intercepts the local calls and decides the upstream that I configured with nginx must be out of istio's scope so it tries to redirect it outside of the mesh.

[linsun]

sorry but how is this related to istio?

[MrBlaise]

@linsun this problem occurs when istio-proxy is injected, normally this flow is totally fine but when the envoy proxy is present I get this behaviour.

[MrBlaise]

[2019-05-31T07:13:55.641Z] "GET /v1/namespaces HTTP/1.1" 503 UC "-" 0 95 2 - "127.0.0.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" "a447d68d-83f9-4cd8-8fab-d687f9da5f8f" "localhost" "10.0.91.42:8080" PassthroughCluster - 10.0.91.42:8080 127.0.0.1:0 -
[2019-05-31T07:13:56.016Z] "- - -" 0 - "-" 2801 4207 369 - "-" "-" "-" "-" "74.125.133.95:443" PassthroughCluster 10.0.93.77:35412 74.125.133.95:443 10.0.93.77:35410 -
[2019-05-31T07:13:57.020Z] "- - -" 0 - "-" 3841 4255 13 - "-" "-" "-" "-" "108.177.15.95:443" PassthroughCluster 10.0.93.77:56156 108.177.15.95:443 10.0.93.77:56154 -

I get a bunch of PassthroughCluster in istio-proxy logs when I try to visit an endpoint that is proxied to another service. (This is the third party cloudendpoints esp with predefined nginx config)

The same thing was happening in my own nginx reverse proxy, but when I changed the upstream name to match the server name of the service I try to proxy to, I see this: outbound_.8080_.v1_.foo-bar.namespace.svc.cluster.local

[MrBlaise]

My guess is that nginx registers the upstream with the name I provide and when I call is using proxy_pass it would normally go to the server I specified but istio-proxy intercepts the local call as well and decides it is out of cluster. I am not that familiar with the inner working of the proxy but this is my guess.

[howardjohn]

Might be related to #14443

[Stono]

Hey,
This is not an istio issue, getting nginx to work with istio is a little bit difficult. The issue is because fundamentally nginx is making an outbound request to an ip that is has resolved from your hostname foo-bar. This won't work as envoy doesn't know what cluster ip belongs to, so it fails.

I'd suggest using the ingress-nginx kubernetes project and in turn using the following value in your Ingress configuration:

  annotations:
    nginx.ingress.kubernetes.io/service-upstream: "true"

What this does is ensure that nginx doesn't resolve the upstream address to an ip, and maintains the correct Host header which the sidecar uses in order to route to your destination.

I recommend using this project because I use it, with Istio, with a 240 odd service deployment.

If you're not using ingress-nginx, I think you can set proxy_ssl_server_name on; or another thing you could try is forcefully setting the Host header on the outbound request to the internal fqdn of the service so:

proxy_set_header Host foo-bar;

Hope this helps but as I say, it's an nginx configuration rather than an istio problem.

[MrBlaise]

@Stono Thanks I will investigate these options. I do use ingress-nginx and I want to change all my ingresses to istio gateways+virtualservices, the reason I use an nginx is for my api-gateway.

ESP proxy (Google Cloud Endpoints) -> Nginx API-gateway -> My Application

[Stono]

@Mr-Linus yeah gateways would be one way to do it but then you're managing certs on envoy.
I personally use ingress-nginx with the Deployment annotation traffic.sidecar.istio.io/includeInboundPorts: "" which means istio-proxy won't intercept inbound traffic.

Then you have:

[google cloud load balancer] -> [ingress-nginx -> envoy] -> [envoy -> some-service]

And ingress-nginx can do your tls termination, SNI routing etc. It's basically an edge.

Can you close this issue of as it's not an istio issue, it is a great topic of conversation however so probably more one for the istio-users groups.

I'll write up a blog post over over the next few days detailing our setup too.

[MrBlaise]

Yes I'll close it and thanks!

[choldrim]

Hey, This is not an istio issue, getting nginx to work with istio is a little bit difficult. The issue is because fundamentally nginx is making an outbound request to an ip that is has resolved from your hostname foo-bar. This won't work as envoy doesn't know what cluster ip belongs to, so it fails.

I'd suggest using the ingress-nginx kubernetes project and in turn using the following value in your Ingress configuration:

  annotations:
    nginx.ingress.kubernetes.io/service-upstream: "true"

What this does is ensure that nginx doesn't resolve the upstream address to an ip, and maintains the correct Host header which the sidecar uses in order to route to your destination.

I recommend using this project because I use it, with Istio, with a 240 odd service deployment.

If you're not using ingress-nginx, I think you can set proxy_ssl_server_name on; or another thing you could try is forcefully setting the Host header on the outbound request to the internal fqdn of the service so:

proxy_set_header Host foo-bar;

Hope this helps but as I say, it's an nginx configuration rather than an istio problem.

hello @Stono
I met the same problem, and the way you mentioned proxy_ssl_server_name on; and proxy_set_header Host foo-bar; doesn't help for me, the only way working fine is setting the upstream name to k8s service name(I think this way just ignores the upstream and uses the k8s service name directly).

Actually, I don't want to change all my nginx conf by the way, it's unfriendly for user, would you tell me why it conflicts with Istio? Is there any blog or docs to explain that?

Sorry, my fault, I found the true reason is that there are some health checkers failed after injected sidecar, and finally I have already fixed them.