Private DNS management with private clusters does not give full control over API server address

[vcariven]

Summary

Private DNS management with private clusters (--enable-private-cluster) does not give full control over API server address.
We have several options, each one coming with its limitations.

In our case, we setup and operate an Hub and Spoke architecture for different clients with DNS forwarder in the Hub zones.
So we setup our on-prem DNS servers to forward a client-related list of private DNS zones to the client HUB DNS Forwarders.

When creating a private AKS cluster, we do not have full control on API Server address name.

Current available options and limitations

1. Using --private-dns-zone system. A private DNS zone is created in the AKS resource group looking like <uid>.privatelink.<region>.azmk8s.io and Azure registers the API server address inside it (ex. <aks-prefix><random>.<uid>.privatelink.<region>.azmk8s.io)

In that case, for each AKS cluster, we could configure the on-prem DNS forwarding to the correct DNS forwarders regarding the client.

Limitations

• updating on-prem DNS for each created/updated cluster is not easy
• we have no control on the subzone naming <uid>.privatelink.<region>.azmk8s.io

2. Using --private-dns-zone <private-zone-uid> (and optionnaly --fqdn-subdomain my-custom-aks-name)

In that case, we have no choice for the zone name, it must be privatelink.<region>.azmk8s.io and Azure registers the API server address inside it (ex. my-custom-aks-name.privatelink.<region>.azmk8s.io)

Limitations

• In a multi-client environment, we cannot configure DNS forwarding on on-prem DNS servers for multiple clients as the private zone name is unique (privatelink.<region>.azmk8s.io)

3. Using --enable-public-fqdn and --private-dns-zone none

Fits most of the needs, but...

Limitations

• Currently in Preview
• No much control over naming
• private network info visible on public DNS

Feature request

Though the case 3 fits most of the needs, it is still in preview and does not give full control.
The case 2 is the most promising but is missing some features:

• At least allow to define a subzone of privatelink.<region>.azmk8s.io (ex. client1.privatelink.<region>.azmk8s.io)
  • This could allow configuring correct DNS forwarding on on-prem DNS regarding clients
    • client1.privatelink.<region>.azmk8s.io forwarding to HUB-CLIENT-1 DNS forwarders
    • client2.privatelink.<region>.azmk8s.io forwarding to HUB-CLIENT-2 DNS forwarders
• Allow any private DNS zone name (ex. client1.azr.corp)
  • client1.azr.corp forwarding to HUB-CLIENT-1 DNS forwarders
  • client2.azr.corp forwarding to HUB-CLIENT-2 DNS forwarders
• Add an option to add some subject alternative names (SAN) in the generated API Server certificate

In that case, even with the current options, we could also automatically maintain some aliases in managed private DNS zones

Example

• my-custom-aks-name.privatelink.<region>.azmk8s.io A 10.100.10.5
• Auto register a record in another managed private DNS zone : my-custom-aks-name.client1.azr.corp A 10.100.10.5
• If my-custom-aks-name.client1.azr.corp can be configured at cluster creation as an alternative SAN, we could access the cluster with kubectl with the server name: https://my-custom-aks-name.client1.azr.corp:443 without TLS Handshake failures.

Unable to connect to the server: x509: certificate is valid for localhost, hcp-kubernetes, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, hcp-kubernetes.610b84d06967ea0001ba3cd9.svc.cluster.local, my-custom-aks-name.privatelink.westeurope.azmk8s.io, not my-custom-aks-name.client1.azr.corp:443)

It could be an enhancement of the API and the Azure CLI. The underlying aks-engine seems to handle multiple SAN (https://github.com/Azure/aks-engine/blob/master/docs/topics/clusterdefinitions.md#masterprofile)

Regards,

[ghost]

Hi vcariven, AKS bot here 👋
Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such:

1. If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster.
2. Please abide by the AKS repo Guidelines and Code of Conduct.
3. If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics?
4. Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS.
5. Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue.
6. If you have a question, do take a look at our AKS FAQ. We place the most common ones there!

[vcariven]

Maybe linked to #1508

[ghost]

Triage required from @Azure/aks-pm

[miwithro]

@feiskyer another feature-request for subzone on private dns.

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[miwithro]

This is supported now.

https://docs.microsoft.com/en-us/azure/aks/private-clusters

Specifically,

"CUSTOM_PRIVATE_DNS_ZONE_RESOURCE_ID", which requires you to create a Private DNS Zone in this format for Azure global cloud: privatelink..azmk8s.io or .privatelink..azmk8s.io. You will need the Resource ID of that Private DNS Zone going forward. Additionally, you will need a user assigned identity or service principal with at least the private dns zone contributor and network contributor roles.

[ghost]

Thanks for reaching out. I'm closing this issue as it was marked with "Answer Provided" and it hasn't had activity for 2 days.