[Feature Request] AKS support of BYO user-assigned-identity for Managed Identity support

[jluk]

What happened:
In the current Managed Identity model, only AKS created identities are supported.

This blocks enterprise scenarios where a dedicated networking team provides network permissions, but can't assign permissions to an identity that can be passed an app team prior to creating the cluster. This requirement expands to any needed permissions which should be granted to a cluster identity prior to cluster creation.

The goal of this issue is to enable a user to bring their own user assigned identity which must have all necessary permissions to be used in the cluster, similar to bringing your own SP today.

Addresses:
#1557
#1542

[Tbohunek]

Hello @jluk , thanks for a super quick response.
This is an urgent issue for us, we were using MSI on pre-Prod deployments on a scenario which is now considered BYO RT, and our future deployments are now blocked. The issue is we should go Prod in matter of days, and we cannot deploy a functioning cluster now. Falling back to SPs has its own issues and would require us to redeploy the cluster once MSI is supported again.

In other words, release of BYO RT broke previously working functionality.

[TomGeske]

@Tbohunek: current ETA is that we are planning to release public preview for BYO control plane MI this month.

[jluk]

@Tbohunek the BYO RT functionality should have only impacted net-new scenarios in which you brought an existing subnet/rt to a cluster and AKS will now use the existing subnet.

Can you clarify what is breaking? If you are deploying MSI clusters today I presume you are manually adding routes to the route table AKS creates on your behalf, which should continue to work.

[Tbohunek]

@jluk We have a subnet subnet with Route Table rt1 assigned. We deploy the AKS cluster cluster1, this created Route Table aks-agentpool-xxx-rt in AKS RG mc_RG_xxx which we then script-assigned to the subnet1.

Right now this is seen as BYO RT scenario and AKS no longer creates aks-agentpool-xxx Route table.

[Tbohunek]

What I see is that you need to update the AKS deployment chain of events so that the Managed Identity is created first and given permissions on the subnet and on the BYO RT before AKS begins to create routes in the BYO RT.

That doesn't sound too hard, but I know there are also issues that AKS deployment did not create Role assignments for the SP nor did it assign the AKS-managed RT to the subnet afterwards, even though it should have. I need to verify that this actually works now. #400

[Tbohunek]

Hey,
So we did a workaround (un-peer vnet from hub to stop BGP routing to avoid using custom RT, deploy, then peer again). This seems to work to deploy with Managed Identities.
However @jluk , we noticed a defect where we are not able to provision istio ingress gateways due to missing permissions for the Managed Identity to read the subnet.

LoadBalancer    65s (x5 over 2m20s)  service-controller  Ensuring load balancer
  Warning  SyncLoadBalancerFailed  65s (x5 over 2m20s)  service-controller  Error syncing load balancer: failed to ensure load balancer: network.SubnetsClient#Get: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client 'xxx' with object id 'xxx' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/subnets/read' over scope '/subscriptions/xxx/resourceGroups/rg-xxx/providers/Microsoft.Network/virtualNetworks/vnet-xxxx/subnets/sn-xxx' or the scope is invalid. If access was recently granted, please refresh your credentials."

Is this already known? Manually adding role assignment works for the moment.

[jluk]

@Tbohunek thanks for raising, we're looking into the missing permission for managed identity cc @TomGeske

[Tbohunek]

@jluk Cool, thanks. Do you also plan to include the required permission on BYO RT and Subnet, or are those two planned to be kept manual?

[kenans]

@TomGeske Hey, I wonder if there's any update about the BYO control plane MI public preview so far?

[TomGeske]

Yes, Preview will be available very soon. We are currently wrapping up final items like cli and docs.

[kenans]

@TomGeske Thank you for the information! Also I wonder if there's any plan to allow user assigned MI for Kubelet? I saw it was still "Not currently supported" from the document

[TomGeske]

@kenans: Yes, that's definitely in our pipeline. Plan is to validate first bring your own control plane MI, once we are good we will others like Kubelet for bring your own.

[TomGeske]

We just shipped preview for bring your own control plane managed identity: https://docs.microsoft.com/en-us/azure/aks/use-managed-identity#bring-your-own-control-plane-mi-preview. Give it a try and let us know how it goes.

[Tbohunek]

Hi @TomGeske Sadly we were unable to deploy the preview version due to different reasons. Do you have feedback from other customers? Does it look good with GA soon?

Thanks

[TomGeske]

@Tbohunek: technical reason?

[Tbohunek]

@TomGeske nope, operational reason - the team is entering production so there is limited time for experiments. I'll let you know next week.

[jastis77]

Adding an existing app gateway on on an existing cluster using az aks enable-addons cli command creates the Ingress controller managed identity and assigns it the contributor role on the App gateway resource group. This is the desired behavior. However using an ARM template to create a AKS cluster with an App gateway add on using an existing App gateway resource Id does not do the same role assignment. This impacts our effort in deploying AKS with app gateway using CI CD where our service principal cannot be given User access admin role.

[TomGeske]

Bring your own support for add-on MIs like the one from AppGW are planned in future.

[stijnv1]

We are trying to use BYO user-assigned identity for AKS in combination with kubenet as network plugin and custom UDR.
Following article mentions that managed identity in combination with custom UDR and kubenet is not supported.

Following article though mentions in a note that both service principal or managed identity can be used for custom UDR in combination with kubenet.

We tested the deployment with a BYO user-managed identity, kubenet and custom UDR and get an error telling us managed identities in combination with custom udr is not supported.
Tested this with both ARM template deployment and Azure CLI using latest aks-preview extension.

Will this become a supported scenario in the future (BYO user-assigned-identity icw kubenet and custom UDR)?

[TomGeske]

@stijnv1: thanks for your feedback. That might be a validation that need to be removed. let me double check.
Do you have the exact error message?

[stijnv1]

@TomGeske

Exact message we get when deploying with Azure CLI or ARM:

"code": "CustomRouteTableWithMSINotSupported",
"message": "Clusters using managed identity do not support bringing your own route table. Please see https://aka.ms/aks/customrt for more information"

[noose]

Can you please a look

msrest.http_logger :     'x-ms-correlation-request-id': '842ae2a8-adcd-46f4-be73-b6c64be44057'
msrest.http_logger :     'x-ms-routing-request-id': 'FRANCESOUTH:20200909T115339Z:842ae2a8-adcd-46f4-be73-b6c64be44057'

?
We also expiring this problem

Deployment failed. Correlation ID: 842ae2a8-adcd-46f4-be73-b6c64be44057. {
  "status": "Failed",
  "error": {
    "code": "ResourceDeploymentFailure",
    "message": "The resource operation completed with terminal provisioning state 'Failed'.",
    "details": [
      {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.",
        "details": [
          {
            "code": "BadRequest",
            "message": "{\r\n  \"code\": \"CustomRouteTableWithMSINotSupported\",\r\n  \"message\": \"Clusters using managed identity do not support bringing your own route table. Please see https://aka.ms/aks/customrt for more information\"\r\n}"
          }
        ]
      }
    ]
  }
}

[TomGeske]

Thanks for your feedback. That's still not ready. We are working on enabling bring your own route table for managed identity. Current ETA is October.

[denniszielke]

Since we now have October - can you outline when MSI+ Kubenet + Outboundtype will be working?

[miwithro]

Before the end of October. I can provide a better date next week.

[adhodgson1]

Hi guys,

I need a little assistance with creating an ARM template that uses the user assigned identity. Here are relevant portions of the template, the user assigned identity is being created in the same resource group as the AKS cluster:

  "variables": {
    "managedIdentityId": "[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', parameters('resourceName'))]"
  },
  "resources": [
    {
      "apiVersion": "2020-09-01",
      "type": "Microsoft.ContainerService/managedClusters",
      "location": "[resourceGroup().location]",
      "name": "[parameters('resourceName')]",
      "tags": "[parameters('tags')]",
      "identity": {
        "type": "UserAssigned",
        "userAssignedIdentities" : {
          "[variables('managedIdentityId')]": {
            "clientId": "45d7d4ee-9a58-4518-9723-4b9aa796e667",
            "principalId": "9f121e4b-3e54-4573-82ff-e8737d7c7374"
          }
        }
      },

I have also tried with the ID of the identity instead of using a variable, though the end goal will be to have all these parameterised.

Here is the error I am getting:

"code": "InvalidIdentityValues",
"message": "Invalid value for the identities '/subscriptions/<my-sub>/resourceGroups/<my-rg>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<my-identity-name>'. The 'UserAssignedIdentities' property keys should only be empty json objects, null or the resource exisiting property."

[adhodgson1]

Fixed this, for those getting the same issue the correct syntax is:
"identity": {
"type": "UserAssigned",
"userAssignedIdentities" : {
"[variables('managedIdentityId')]": {}
}
}

[cdunford]

This feature appears to be in preview now; is there any timetable for GA?

[palma21]

It is now GA

[ThorstenHans]

Thanks @palma21 for sharing 🚀

[hello-woof]

@TomGeske and @miwithro, I've been trying to do similar configuration to #1591 (comment), and I'm still seeing the same error. Is this still in progress?

resource "azurerm_kubernetes_cluster" "aks" {
  name                    = "myaks"
  location                = azurerm_resource_group.kube.location
  kubernetes_version      = var.kube_version
  resource_group_name     = azurerm_resource_group.kube.name
  dns_prefix              = "myaks"
  private_cluster_enabled = true

  default_node_pool {
    name           = "default"
    node_count     = var.nodepool_nodes_count
    vm_size        = var.nodepool_vm_size
    vnet_subnet_id = module.knet.subnet_ids["aks"]
  }

  identity {
    type = "SystemAssigned"
  }

  network_profile {
    docker_bridge_cidr = var.network_docker_bridge_cidr
    dns_service_ip     = var.network_dns_service_ip
    network_plugin     = "kubenet"
    outbound_type      = "userDefinedRouting"
    service_cidr       = var.network_service_cidr
  }

  depends_on = [module.routetable]
}

[TomGeske]

You would need bring your own identity in that case. @miwithro couldn't find an example in Terraform how to define bring your own identity for control plane. Can you help?

identity {
  type = "SystemAssigned"
}