Ingress does not support several TLS certificates

[gedimin45]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): no.

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): ingress tls.

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST

Kubernetes version (use kubectl version): 1.4

Environment:

• Cloud provider or hardware configuration: GKE
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened: Currently Ingress resources can only have one TLS certificate configured:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test
spec:
  tls:
    - secretName: testsecret
  rules:
  - host: foo.bar.com
    http:
      paths:
      - backend:
          serviceName: s1
          servicePort: 80
  - host: bar.foo.com
    http:
      paths:
      - backend:
          serviceName: s2
          servicePort: 80

What you expected to happen: Would it make sense to have a TLS certificate per route rule. Something like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test
spec:
  rules:
  - host: foo.bar.com
    tls:
      - secretName: testsecret
    http:
      paths:
      - backend:
          serviceName: s1
          servicePort: 80
  - host: bar.foo.com
    tls:
      - secretName: testsecret
    http:
      paths:
      - backend:
          serviceName: s2
          servicePort: 80

I am not aware of the design decisions behind the current implementation so this might not make sense at all.
The reason I propose this is because I see Ingress'es as public IP addresses. You might want to have several public IPs pointing to your cluster but also only one IP and then have several hosts point to that IP.

Initially submitted at kubernetes-retired/contrib#2058, but per request of @aledbf, submitting here.

[bprashanth]

Cloudproviders don't really support muxing multiple certs onto a single lb, so your current options are:

1. Create 2 ingresses with 2 certs (works everywhere)
2. Create 1 ingress with a wildcard or san (works everywhere)
3. Create 1 ingress with a list of certs with different hostnames, one per host (depends on backend implementation, cloud provider will simply pick the first cert and assume it's a san throwing a warning event)

In theory a backend can parse the hostnames out of the list of certs and use the right one with the right Host header.

We don't put the cert in with the HTTP rules because encryption is handled at Layer 5. So:

1. We'd need to repeat the same structs in a bunch of places for non-http protocols (dtls, just tcp + ssl etc)
2. Keeping the certs in a separate section also makes centralized validation and renewal easier, i.e I can write a letsencrypt controller that doesn't even care about the rules

Non-http ingress is being discussed here: #23291
More explicity ingress sharing is being discussed here: #30151

[alexferl]

@bprashanth There's an example on how to use multiple TLS certificates here that @aledbf wrote. It actually lets you create the Ingress but it does not work as it uses the first certificate (on GKE anyways) only as you mention in 3.

When you describe the Ingress you actually see:

TLS:
  somedomain.io-cert terminates somedomain.io,www.somedomain.io
  api.somedomain.io-cert terminates api.somedomain.io
  otherdomain.com-cert terminates otherdomain.com,www.otherdomain.com

So I guess this is a GKE issue?

[bprashanth]

What controller are you running? If you're runnign the nginx controller, it has nothing to do with gke or gce, it functions the same everywhere. If you're running the gce ingress controller, gce doesn't accept multiple certs. By default you get the gce controller on gce, if you want to deploy the nginx controller instead, you need to do: https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx#running-on-cloudproviders, eg https://github.com/kubernetes/contrib/blob/master/ingress/controllers/gce/BETA_LIMITATIONS.md#disabling-glbc

[bprashanth]

To clarify, we don't validate against a single cert, because it's 100% valid to use a list of certs and look them up in the backend. If you're interested in submitting a patch to the nginx backend, i'm sure @aledbf would be more than happy to guide you (assuming the nginx controller doesn't already handle this).

You just need to read out the hostname from the provided certs and write the appropriate rules, eg

const (
	keyFile  = "/tmp/nginx.key"
	certFile = "/tmp/nginx.crt"
	hostname = "nginxsvc"
)

func main() {
	cmd := fmt.Sprintf("openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout %v -out %v -subj \"/CN=%v/O=%v\"",
		keyFile, certFile, hostname)
	exec.Command("sh", "-c", cmd).Run()
	defer func() {
		exec.Command("rm", "-f", certFile, keyFile).Run()
	}()
	rawCert, _ := ioutil.ReadFile(certFile)
	cert, _ := pem.Decode(rawCert)
	x509Cert, _ := x509.ParseCertificate(cert.Bytes)
	if err := x509Cert.VerifyHostname(hostname); err != nil {
		log.Fatal(err)
	}
	log.Printf("successfully verified hostname %v in cert %v", x509Cert.Subject.CommonName, certFile)
}

The "write the appropriate rules" part won't work when using a cloudprovider lb (eg: GCLB, ELB) but you can still use something like nginx on a cloudprovider.

[gedimin45]

Cloudproviders don't really support muxing multiple certs onto a single lb, so your current options are:

Create 2 ingresses with 2 certs (works everywhere)
Create 1 ingress with a wildcard or san (works everywhere)
Create 1 ingress with a list of certs with different hostnames, one per host (depends on backend implementation, cloud provider will simply pick the first cert and assume it's a san throwing a warning event)

@bprashanth I am lost now - which of these options are supported on GKE? :)

[bprashanth]

If you're running the nginx controller on GKE, you can do all of them
If you're using the normal GCE ingress controller on GKE, then options that say "(works everywhere)"

[k8s-github-robot]

@Ged15 There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
(2) specifying the label manually: /sig <label>

Note: method (1) will trigger a notification to the team. You can find the team list here.

[cmluciano]

/sig network

[cmluciano]

Is this still an issue?

[gedimin45]

Sorry for the lack of responsiveness. I've not worked with GKE and K8s Ingress integration with Google load balancer for a while so do not know whether this is still an issue. Will close.

[c4b4d4]

This is still a issue.

[noname85]

@Ged15 This is still an issue.
Please re-open this issue

[miquelbar]

@Ged15 can you reopen this ticket? This is still an issue

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[pkdetlefsen]

Is this something that might get supported eventually? It would be a big plus to be able to put all your services, hosted on different domains using multiple certificates, behind a single Google Cloud Load Balancer using an Ingress.

Currently I can work around the issue by managing the additional certs manually, as it looks like the GCLB does support multiple certs and SNI. For now I will probably move towards using a SAN cert for most of the domains to avoid managing them manually, but that's not possible for all of them in my case unfortunately.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/ingress.class: "gce"
spec:
  tls:
  - hosts: # First cert are managed automatically
    - my.domain.com
    - domain.com
    secretName: domain-tls
  - hosts: # Additional certs have to be added manually to the load balancer
    - other-domain.com
    secretName: other-domain-tls
  rules: ...
  backend: ...

[pkdetlefsen]

/remove-lifecycle stale

[Aaron3]

Is there a technical reason this is not supported on GCE?

P.S. Issue for the gce ingress is here: kubernetes/ingress-gce#46

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[anderspetersson]

@ianatha GCE ingress added multiple cert support in 1.1.0, that will be shipped in kubernetes 1.11.

[pkdetlefsen]

Multiple TLS certs are now working for me as of k8s ver. 1.10.2-gke.1 in GCP! It works by adding multiple hosts in the tls section like in my previous example.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[rramkumar1]

As stated above, ingress-gce does support multiple TLS certs now.

/close

[gsf]

This is now documented in the following section, prefaced by "Specifying multiple secrets...":

https://github.com/kubernetes/ingress-gce/blob/63bd8b9e8a7e0013a50ef728aabf9e5076d5ae65/README.md#secret