How can I apply multiple certificates in istio-ingressgateway

[exiaohao]

--what the problem is--
I have several TLS certificates for different domains on ingress-gateway,
But it seems support only one certificate which was istio-ingressgateway-certs from secret/tls (from https://istio.io/docs/tasks/traffic-management/secure-ingress/)

How can I apply more than one certificate for different domains?

ps.
Why not use Kubernetes' secret to specify key & certificates instead of loading files like this:

tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key

[vadimeisenbergibm]

You can either edit the ingress gateway deployment manually and add more volume mounts, or use helm and set secretVolumes array with your certificates, see the original value in https://github.com/istio/istio/blob/master/install/kubernetes/helm/istio/values.yaml#L237.

Why not use Kubernetes' secret to specify key & certificates

Because Istio is designed to run also in non-Kubernetes environments, so it's configuration strives not to contain Kubernetes constructs.

[dunjut]

@vadimeisenbergibm So essentially, to get https traffic working, a user only has to specify the certificate path in Gateway and make sure it exists in the ingressgateway container, no matter the certificate is mounted from kubernetes Secret or any other means, right?

If so, to dynamically adding more tls certificates to ingressgateway without re-deployment, we may need to develop a tool which defines how users upload their certs (using k8s Secret for example) and automatically place those certs into user-specified paths inside ingressgateway container when uploading events happen?

[vadimeisenbergibm]

@dunjut That's correct.

[dunjut]

@vadimeisenbergibm Plenty of users run istio in kubernetes (istio is designed for multi kind of environments though), to make dynamic tls certificate support available to users like us, is it acceptable that we develop such a tool and make it optional in helm chart like kiali, certmanager and so?

[vadimeisenbergibm]

@dunjut My understanding is that the Istio community welcomes the ecosystem of tools for a service mesh on top of istio.io. Contribution of tools, plugins on top of istio.io for various platforms,and especially for Kubernetes, documentation and code, is very welcome.

[dunjut]

@incfly replied to this istio-dev thread and said you're working on integrating Envoy SDS to let the workload fetch key cert in the runtime. Will that work solve this issue? cc @vadimeisenbergibm

[vadimeisenbergibm]

integrating Envoy SDS to let the workload fetch key cert in the runtime. Will that work solve this issue?

@dunjut Yes, it seems so.

[myidpt]

@dunjut
Now Istio supports multiple external certificates on gateway.
https://istio.io/docs/reference/config/istio.networking.v1alpha3/#Gateway
I haven't tried how to configure it yet (sorry, just came back from long vacation). I will update the "secure ingress" task soon after I verify the configuration.

If you dynamically add domains:

1. You need to define/redefine the gateway, and gateway Envoy can pick up the new config automatically.
2. If you need to use new key/cert for the domain, you need to create a new secret for them, and you may need to modify the ingress deployment to mount the secret. Mount path should be under this dir so that pilot agent file watcher will auto-restart Envoy when the key/cert are changed.

P.S., In the future, I believe we should get rid of the file watcher, and use SDS to update the key/cert for external certs as well.

[andersjanmyr]

@myidpt I don't see how Istio supports multiple external certificates from the documentation on gateway that you referenced. As far as I can tell it only supports one certificate per server.

 - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
    hosts:
    - "*"

We would also need to specify for what host a certificate is served for.
Envoy supports multiple certificates through filter_chains, https://www.envoyproxy.io/docs/envoy/latest/faq/sni, but as far as I can tell this is not generated by Istio at the moment.

Do I misunderstand?

[dunjut]

I just spend some time writing a tool to automate the synchronization between kubernetes secrets and istio-ingressgateway certificates.

Folks who suffer from this issue can have a try on it (a very early version though).

cc @vadimeisenbergibm @myidpt @andraxylia @andersjanmyr

[vadimeisenbergibm]

@exiaohao Here is how you can do it https://preliminary.istio.io/docs/tasks/traffic-management/secure-ingress/#configure-a-tls-ingress-gateway-for-multiple-hosts

[ZhiminXiang]

@vadimeisenbergibm I am not sure if the Gateway could still serve the traffic well during the process of the redeployment?

[exiaohao]

@vadimeisenbergibm Thanks for your help, it seem that new certificates will take effect after redeploy istio-ingressgateway. But in this way I'm not sure Gateway can upgrade without downtime.
if so, it's not really a good idea for users I think, we'd reduce any downtime and have a failover plan for incorrect config.
Is there any official solutions like @dunjut 's way, I think it's a easy way without downtime.

[sudermanjr]

@exiaohao With the ingressgateway being scalable now, I think you should be able to rolling-update the ingressgateway with your new cert without downtime.

[dunjut]

@sudermanjr Scaling and rolling-updating implies some kind of loadbalancer in front of ingressgateway, which is common in a cloud environment, but it's not always true in bare-metals.

In our scenario, we deploy ingressgateway as DaemonSet on a set of dedicated physical machines, facing directly to users, acting as edge layer 7 gateway.

[andraxylia]

@myidpt @wattli No downtime is a must and we need a solution that allows the Gateway to function as an edge proxy.

cc @costinm

[myidpt]

I think the best solution here is to have ingress gateway use SDS to load these certs from node agent (which can watch certs from k8s secrets or external API). This can be implemented after node agent work is done on master (hopefully soon). The work is not very complex when node agent is ready, and dunjut is also willing to help :) For anyone having a timing concern, please try dunjut's tool.

[ZhiminXiang]

My understanding about dunjut's tool is that it places certificates into the volumes that have been created in the istio-ingressgateway pods.
But if all of volumes are already taken and we still want to add a new domain and its certificate into Gateway, we have to redeploy istio-ingressgateway pods to add more volumes, right? Is there any solution to avoid redeployment of istio-ingressgateway in this case?

[exiaohao]

@ZhiminXiang https://github.com/dunjut/cert-sync use <certdir>/<namespace>/<name> as path for certificates.
By default setting, certificates has load to /etc/istio/ingressgateway-certs/<namespace>/<name>.{key, crt}
It takes only one directory for all certificates and no need to redeploy.

FYI. https://github.com/dunjut/cert-sync#use-tls-secrets-in-istio-gateway

[vadimeisenbergibm]

@exiaohao Is this issue still relevant? What else is required to be done?

[annismckenzie]

This shouldn't have been closed until @myidpt's suggestion is either implemented or documented to be working?

[mmckane]

Can someone Please Re-Open this? This is not implemented or documented from what I can tell. Envoy appears to have implemented the Secret Discovery Service, and the Node Agent appears to have some implementation of the envoy secret discovery service but I am guessing that is for mtls cert refresh. If this does work for the Istio Gateway does anyone have details on how this works or how to set it up?

[myidpt]

Please see #9030. It should solve the multiple-tls-cert issue here.

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 90 days. It will be closed in the next 30 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had activity in the last month and a half. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.