Support for AKS cluster with API Server VNet Integration

[EppO]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

API Server VNET Integration for AKS cluster is in preview for both private and public clusters. It would be sweet to be able to provision it through Terraform. There are 2 modes for this feature:

• Managed VNet: when no VNet is provided, AKS will create the VNET and multiple subnets including one for the API server and configure them.
• BYO VNet: subnet for the API server needs to be created first and delegated to Microsoft.ContainerService/managedClusters (not sure about the corresponding actions in my example though) and passed to the AKS cluster. We need also that the AKS cluster Managed Identity to have Network Contributor roles on both AKS cluster and API server subnets.

New or Affected Resource(s)/Data Source(s)

azurerm_kubernetes_cluster

Potential Terraform Configuration

resource "azurerm_resource_group" "example" {
  name     = "example-resources"
  location = "West Europe"
}

resource "azurerm_virtual_network" "example" {
  name                = "example-vnet"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
}

resource "azurerm_subnet" "node_subnet" {
  name                 = "node-subnet"
  resource_group_name  = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["10.0.1.0/24"]
}

resource "azurerm_subnet" "apiserver_subnet" {
  name                 = "apiserver-subnet"
  resource_group_name  = azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.example.name
  address_prefixes     = ["10.0.2.0/28"]

  delegation {
    name = "delegation"

    service_delegation {
      name    = "Microsoft.ContainerService/managedClusters"
      actions = ["Microsoft.Network/virtualNetworks/subnets/join/action", "Microsoft.Network/virtualNetworks/subnets/prepareNetworkPolicies/action"]
    }
  }
}

resource "azurerm_user_assigned_identity" "example" {
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location

  name = "aks"
}

resource "azurerm_role_assignment" "aks_subnet_role" {
  scope                = azurerm_subnet.node_subnet.id
  role_definition_name = "Network Contributor""
  principal_id         = azurerm_user_assigned_identity.example.id
}

resource "azurerm_role_assignment" "apiserver_subnet_role" {
  scope                = azurerm_subnet.apiserver_subnet.id
  role_definition_name = "Network Contributor""
  principal_id         = azurerm_user_assigned_identity.example.id
}

resource "azurerm_kubernetes_cluster" "example" {
  name                               = "example-aks1"
  location                           = azurerm_resource_group.example.location
  resource_group_name                = azurerm_resource_group.example.name
  dns_prefix                         = "exampleaks1"
  apiserver_vnet_integration_enabled = true
  apiserver_subnet_id                = azurerm_subnet.apiserver_subnet.id
  vnet_subnet_id                     = azurerm_subnet.node_subnet.id

  default_node_pool {
    name       = "default"
    node_count = 1
    vm_size    = "Standard_D2_v2"
  }

  identity {
    type         = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.example.id]
  }

  tags = {
    Environment = "Production"
  }

  depends_on = [
    azurerm_role_assignment.aks_subnet_role,
    azurerm_role_assignment.apiserver_subnet_role
  ]
}

References

https://learn.microsoft.com/en-us/azure/aks/api-server-vnet-integration

[aristosvo]

This is dependent on a newer Go SDK:

1. Or hashicorp/go-azure-sdk
2. Or Track 1 Azure SDK (which is not generated anymore..

Option 1 seems the most logical, but would require rework of the AKS resource.

[EppO]

I guess #18705 is needed to move forward and be able to expose a new setting for that feature.

[aristosvo]

I guess #18705 is needed to move forward and be able to expose a new setting for that feature.

Indeed! Working on it at the moment, but it takes a bit of time unfortunately.

[EppO]

Sure! Just wanted to keep track of what's needed to be done, didn't mean that in a bad way, sorry. I really appreciate you are working on this, thanks a lot!

[jennifer-klemisch-seagen]

@aristosvo is there an update on enabling the aks with vnet integration within TF?
thank you.

[aristosvo]

@jennifer-klemisch-seagen @EppO First implementation is available as PR

[aristosvo]

@MarkKharitonov To answer your question in Azure/AKS#1200 (comment) and to update the status of this issue, it is almost there. Tests are passing, it might be necessary to change a few things based on reviews, but the biggest work has been done.

[MarkKharitonov]

Guys, does it mean the next version of the azurerm provider will have this feature?

[aristosvo]

Guys, does it mean the next version of the azurerm provider will have this feature?

@MarkKharitonov Yes it does!🎉🎉 Release 3.39 is due by January 12, 2023, so I'd expect there to be a release this week.

[MarkKharitonov]

@aristosvo - 3 weeks from now?

[EppO]

Released in v3.39.0

[github-actions]

This functionality has been released in v3.39.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.