Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

how does one do SSL passthrough with source ip preservation #57

Closed
sandys opened this Issue Oct 19, 2016 · 9 comments

Comments

Projects
None yet
4 participants
@sandys
Copy link

sandys commented Oct 19, 2016

hi guys,
this has been a big question on the k8s slack group.. but nobody is a big enough expert in nginx to figure out.

the requirement is very simply - i dont want to terminate my ssl at the ingress controller: I want to terminate them on the nginx pods that i have running inside. how do I do this ?

also - nginx ingress controller does not preserve source ip. we need the original client source-ip for audit purposes.

now i figured out that one must use "stream" and "proxy-protocol" to somehow configure this, but we are just not able to figure this out on the k8s slack. An example would be truly awesome!

@pleshakov

This comment has been minimized.

Copy link
Collaborator

pleshakov commented Oct 20, 2016

@sandys
It is possible with the ngx_stream_ssl_preread_module, which was added recently in 1.11.5 release, combined with proxy protocol and stream module.

For example, this can be your load balancer configuration:

...
stream {
  map $ssl_preread_server_name $name {
      backend.example.com      backend;
      default                  default-backend;
  }

  upstream backend {
      server 192.168.0.1:443;
  }

  upstream default-backend {
      server 192.168.0.2:443;
  }

  proxy_protocol on;

  server {
      listen      443;
      proxy_pass  $name;
      ssl_preread on;
  }
}
...

You nginx web server configuration can lool like this:

server {
    listen 443 ssl proxy_protocol;

    ssl_certificate my.crt;
    ssl_certificate_key my.key;

    set_real_ip_from <address or CIDR  see http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from >;
    real_ip_header proxy_protocol;
    ...
}

However, it's not supported by the Ingress controller. You have to configure NGINX without the Ingress controller.

@aledbf

This comment has been minimized.

Copy link

aledbf commented Oct 20, 2016

@pleshakov thanks for this awesome example. Is valid to use the stream feature to listen in port 443 and for the default case in the map send traffic to the same nginx instance in a different port (like 442) with multiple servers (all this servers provide SSL)?

@pleshakov

This comment has been minimized.

Copy link
Collaborator

pleshakov commented Oct 20, 2016

@aledbf yep, that should work

@aledbf

This comment has been minimized.

Copy link

aledbf commented Oct 20, 2016

@pleshakov awesome. Thanks!

@aledbf

This comment has been minimized.

Copy link

aledbf commented Oct 26, 2016

@pleshakov

stream {
    # map FQDN that requires SSL passthrough
    map $ssl_preread_server_name $stream_upstream {
        foo-898.bar.com                 default-echoheaders-x-80-0-pt;
        foo-997.bar.com                 default-echoheaders-x-80-1-pt;
        kubernetes.foo-bar.com          default-echoheaders-x-80-2-pt;
        # send SSL traffic to this nginx in a different port
        default                         nginx-ssl-backend;
    }

    log_format log_stream '$remote_addr [$time_local] $protocol [$ssl_preread_server_name] [$stream_upstream] '
        '$status $bytes_sent $bytes_received $session_time';

    access_log /var/log/nginx/access.log log_stream;
    error_log  /var/log/nginx/error.log;

    # configure default backend for SSL
    upstream nginx-ssl-backend {
        server 127.0.0.1:442;
    }

    upstream default-echoheaders-x-80-0-pt {
        server 10.2.18.8:8080;
    }
    upstream default-echoheaders-x-80-1-pt {
        server 10.2.18.8:8080;
    }
    upstream default-echoheaders-x-80-2-pt {
        server 10.2.18.8:8080;
    }

    server {
        listen                  443;
        proxy_pass              $stream_upstream;
        ssl_preread             on;
    }
}

using

curl -v -H 'Host: kubernetes.foo-bar.com' https://172.17.4.99 -k

I see this in the logs:

10.2.18.1 [26/Oct/2016:15:11:25 +0000] TCP [] [nginx-ssl-backend] 200 2100 273 0.022

(no $ssl_preread_server_name)
How I can debug the issue?
Thanks

@pleshakov

This comment has been minimized.

Copy link
Collaborator

pleshakov commented Oct 27, 2016

@aledbf might be the case that curl isn't sending the SNI information

@aledbf

This comment has been minimized.

Copy link

aledbf commented Oct 27, 2016

@pleshakov yes, that's the issue. It's working fine in a browser.
Thanks!

@pleshakov pleshakov closed this Nov 21, 2016

@sandys

This comment has been minimized.

Copy link
Author

sandys commented Jan 23, 2017

@pleshakov sorry to comment on an old bug, but is the ssl_preread_server_name going to be surfaced into the ingress configuration ?

@kedlav

This comment has been minimized.

Copy link

kedlav commented May 2, 2018

use curl --resolve, not curl -H 'Host:...'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.