Multiple certificates in same Gateway HTTPs Listener

[pvillaverde]

Hey there,

I'm migrating from Ingress Nginx to Nginx Gateway Fabric and I've run into a issue that might be my lack of understanding. I'm running version 2.6.7 and I have configured a gateway with 2 listeners (http and https).

  listeners:
  - name: http-listener
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
  - name: https-listener
    protocol: HTTPS
    allowedRoutes:
      namespaces:
        from: All 
    port: 443
    tls:
      certificateRefs:
      - kind: Secret
        name: domain1.com-wildcard-tls
      - kind: Secret
        name: domain2.com-wildcard-tls
      - kind: Secret
        name: domain3.com-wildcard-tls

For the HTTP I have configured a single HTTPRoute that redirects all traffic to https. Then my idea was to have HTTPRoutes for each different site using the https-listener, i.e.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: mysite.domain1.com
spec:
  parentRefs:
  - name: nginx-gateway
    namespace: kube-system
    sectionName: https-listener
  hostnames:
  - "mysite.domain.1com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: myservice
      port: 80

However, it seems that the gateway is not able to serve the certificate according to the SNI and is always returning the last certificate (domain3). I've read the documentation and haven't found exactly how am I supposed to do it. In fact all examples I see set the hostname in the listener, which doesn't make sense for me, unless the expected is to have a listener for each hostname (or domain).

Edited:

I've tried adding several listeners on the same ports but its quite messy and it feels overkillingly repeating, specially if you want to serve both root and sub domains, i.e.

  listeners:
  - name: http-listener
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
  - name: https-listener
    protocol: HTTPS
    allowedRoutes:
      namespaces:
        from: All 
    port: 443
    tls:
      certificateRefs:
      - kind: Secret
        name: domain1.com-wildcard-tls
    hostname: "*.domain1.com"
  - name: https-listener-domain2
    protocol: HTTPS
    allowedRoutes:
      namespaces:
        from: All 
    port: 443
    tls:
      certificateRefs:
      - kind: Secret
        name: domain2.com-wildcard-tls
    hostname: "domain2.com"
  - name: https-listener-domain2-wildcard
    protocol: HTTPS
    allowedRoutes:
      namespaces:
        from: All 
    port: 443
    tls:
      certificateRefs:
      - kind: Secret
        name: domain2.com-wildcard-tls
    hostname: "*.domain2.com"
  - name: https-listener-domain3
    protocol: HTTPS
    allowedRoutes:
      namespaces:
        from: All 
    port: 443
    tls:
      certificateRefs:
      - kind: Secret
        name: domain3.com-wildcard-tls
    hostname: "domain3.com"
  - name: https-listener-domain3-wildcard
    protocol: HTTPS
    allowedRoutes:
      namespaces:
        from: All 
    port: 443
    tls:
      certificateRefs:
      - kind: Secret
        name: domain3.com-wildcard-tls
    hostname: "*.domain3.com"

So this makes me think there has to be a better way. And what is the point of specifying multiple certificateRefs in the same listener if its just going to pick the last one instead of choose based on the SNI? It it possible that it got implemented here #3645 but it doesnt work when it has to decide which cert to serve the client request?

[sjberman]

Hi @pvillaverde, are you able to share the nginx conf that gets generated for the HTTPS listener/routes?

[pvillaverde]

Hi @pvillaverde, are you able to share the nginx conf that gets generated for the HTTPS listener/routes?

Hey there @sjberman ! Sure, I've anonimez domains and ommited most of the hosts, but all are configured the same way (changing the server_name and the upstream). Every request is being served with the domain3 certificate

http2 on;
# Set $gw_api_compliant_host variable to the value of $http_host unless $http_host is empty, then set it to the value
# of $host. We prefer $http_host because it contains the original value of the host header, which is required by the
# Gateway API. However, in an HTTP/1.0 request, it's possible that $http_host can be empty. In this case, we will use
# the value of $host. See http://nginx.org/en/docs/http/ngx_http_core_module.html#var_host.
map $http_host $gw_api_compliant_host {
    '' $host;
    default $http_host;
}

# Understanding the Connection header behavior:
# For normal HTTP proxying with keepAlive disabled, we set Connection header to close. This tells the upstream to close the connection after the response.
# When upgrading the connection to WebSocket, we set Connection header to upgrade to inform the upstream to switch protocols.
# For normal HTTP proxying with keepAlive enabled, we leave the Connection header empty. This allows NGINX to manage persistent connections with the upstream.

# Set $connection_header variable to upgrade when the $http_upgrade header is set, otherwise, set it to close. This
# allows support for websocket connections. See https://nginx.org/en/docs/http/websocket.html.
map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

# Set $connection_keepalive variable to upgrade when the $http_upgrade header is set, otherwise, set it to empty for when
# keepAlive is enabled.
map $http_upgrade $connection_keepalive {
    default upgrade;
    '' '';
}

## Returns just the path from the original request URI.
map $request_uri $request_uri_path {
  "~^(?P<path>[^?]*)(\?.*)?$"  $path;
}

# NGINX health check server block.
server {
    listen 8081;
    listen [::]:8081;

    location = /readyz {
        access_log off;
        return 200;
    }
}

server_tokens off;

js_preload_object matches from /etc/nginx/conf.d/matches.json;
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    default_type text/html;
    return 404;
}

server {
    listen 80;
    listen [::]:80;

    server_name ~^;

        
    location / {
        

        

        
        return 301 "https://$host$request_uri";

        

        proxy_http_version 1.1;
    }
}
server {
    listen 443 ssl default_server;
    listen [::]:443 ssl default_server;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_kube-system_domain-1.pem;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_kube-system_domain-2.pem;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_kube-system_domain-3.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_kube-system_domain-1.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_kube-system_domain-2.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_kube-system_domain-3.pem;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_kube-system_domain-1.pem;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_kube-system_domain-2.pem;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_kube-system_domain-3.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_kube-system_domain-1.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_kube-system_domain-2.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_kube-system_domain-3.pem;
    if ($sni_listener_id_443 != $host_listener_id_443) {
        return 421;
    }

    server_name app1.domain1.com;

        
    location / {
        

        

        

        

        proxy_http_version 1.1;
        proxy_set_header Host "$gw_api_compliant_host";
        proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for";
        proxy_set_header X-Real-IP "$remote_addr";
        proxy_set_header X-Forwarded-Proto "$scheme";
        proxy_set_header X-Forwarded-Host "$host";
        proxy_set_header X-Forwarded-Port "$server_port";
        proxy_set_header Upgrade "$http_upgrade";
        proxy_set_header Connection "$connection_keepalive";
        proxy_pass http://domain1_app1_80;
            
            
            
    }
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_kube-system_domain-1.pem;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_kube-system_domain-2.pem;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_kube-system_domain-3.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_kube-system_domain-1.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_kube-system_domain-2.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_kube-system_domain-3.pem;
    if ($sni_listener_id_443 != $host_listener_id_443) {
        return 421;
    }

    server_name app3.domain3.com;

        
    location / {
        

        

        

        

        proxy_http_version 1.1;
        proxy_set_header Host "$gw_api_compliant_host";
        proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for";
        proxy_set_header X-Real-IP "$remote_addr";
        proxy_set_header X-Forwarded-Proto "$scheme";
        proxy_set_header X-Forwarded-Host "$host";
        proxy_set_header X-Forwarded-Port "$server_port";
        proxy_set_header Upgrade "$http_upgrade";
        proxy_set_header Connection "$connection_keepalive";
        proxy_pass http://domain3_app3_80;
            
            
            
    }
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_kube-system_domain-1.pem;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_kube-system_domain-2.pem;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_kube-system_domain-3.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_kube-system_domain-1.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_kube-system_domain-2.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_kube-system_domain-3.pem;
    if ($sni_listener_id_443 != $host_listener_id_443) {
        return 421;
    }

    server_name sub.domain2.com;

        
    location / {
        

        

        

        

        proxy_http_version 1.1;
        proxy_set_header Host "$gw_api_compliant_host";
        proxy_set_header X-Forwarded-For "$proxy_add_x_forwarded_for";
        proxy_set_header X-Real-IP "$remote_addr";
        proxy_set_header X-Forwarded-Proto "$scheme";
        proxy_set_header X-Forwarded-Host "$host";
        proxy_set_header X-Forwarded-Port "$server_port";
        proxy_set_header Upgrade "$http_upgrade";
        proxy_set_header Connection "$connection_keepalive";
        proxy_pass http://domain2_app2_80;
            
            
            
    }
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_kube-system_domain-1.pem;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_kube-system_domain-2.pem;
    ssl_certificate /etc/nginx/secrets/ssl_keypair_kube-system_domain-3.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_kube-system_domain-1.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_kube-system_domain-2.pem;
    ssl_certificate_key /etc/nginx/secrets/ssl_keypair_kube-system_domain-3.pem;
    if ($sni_listener_id_443 != $host_listener_id_443) {
        return 421;
    }

    server_name ~^;

        
    location = / {
        

        

        
        return 404 "";

        

        proxy_http_version 1.1;
    }
}

server {
    listen unix:/var/run/nginx/nginx-503-server.sock;
    access_log off;

    return 503;
}

server {
    listen unix:/var/run/nginx/nginx-500-server.sock;
    access_log off;

    return 500;
}


upstream domain1_app1_80 {
    random two least_conn;
    zone domain1_app1_80 512k;
    
        
    server 10.42.1.90:8123;
    keepalive 16;
    
    
    
}
upstream domain3_app3_80 {
    random two least_conn;
    zone domain3_app3_80 512k;
    
        
    server 10.42.0.236:80;
    keepalive 16;
    
    
    
}


upstream domain2_app2_80 {
    random two least_conn;
    zone domain2_app2_80 512k;
    
        
    server 10.42.0.237:8096;
    keepalive 16;
    
    
    
}
upstream invalid-backend-ref {
    
    
        
    server unix:/var/run/nginx/nginx-500-server.sock;
    
    
    
    
}




map $ssl_server_name $sni_listener_id_443 {
        hostnames;


        default 0;

}
map $host $host_listener_id_443 {
        hostnames;


        default 0;

[sjberman]

Technically your workaround is the correct way to define what you intend, per the Gateway API spec. One other option (if it's feasible) is to use a wildcard cert that can cover the different domains. Then you can keep the single Listener.

The spec states:

The association of SNIs to Certificate defined in ListenerTLSConfig is defined based on the Hostname field for this listener.

and

These certificates are used to establish a TLS handshake for requests that match the hostname of the associated listener.

This means that the certs that are attached to a Listener are all available for that hostname (which in your case results in the empty, default server block; meaning all hostnames), but aren't mapped individually to each hostname defined in each HTTPRoute that is attached to that Listener. Our controller does not inspect the certs directly to determine each cert's SNI so we can map them to each HTTPRoute hostname attached to the empty Listener.

When multiple certs are defined in the nginx config in a single server block, nginx negotiates the best cipher out of those certs, but does not pick the right one by SNI. SNI-based virtual server selection happens by matching the server_name.

Also, since there's a limit on the number of Listeners in a Gateway, it may be better to use a ListenerSet if you find yourself having to define many.

[pvillaverde]

Thanks for the clarification! That makes sense with that implementation in mind. However I still don't see the use case for the certificateRefs. It would only work with same hostnames certificates... for rotation basically I guess.

Could it be improved in the future to infer the mapping in the future?

In the meantime, I've opened a small PR here to add some clarification. Not sure is the best place, but at least I hope it will avoid some hours to the next person who runs into this nginx/documentation#2176

[sjberman]

Right now the use case would be allowing for certs of different types (like RSA vs ECDSA) for the same domain and having nginx pick the right algorithm based on the connecting client's TLS stack.

We could look into improving this though by having our controller inspect the certificates and apply to the correct servers for each hostname. We'd need to understand the implications of this as well.