Update azurerm_kubernetes_cluster for GA #1389

holmesb · 2018-06-13T21:46:00Z

AKS has gone GA. Main functionality now lacking is the basic\advanced network setting and the various parameters if advanced is chosen:

VNet
Subnet
Service address range
DNS IP
Docker bridge IP

torresdal · 2018-06-14T07:45:41Z

I would gladly contribute to this, but looks like we are dependant on azure-sdk-for-go to include the new package from autorest containerservices (package-2018-03)?

Currently azure-sdk-for-go only has the 2017-09-30 package.

I haven't contributed to this project before, so I'm new to how the dependency to the azure-sdk-for-go-project and azure-rest-api-specs-project works. Anyone know how/when a new autorest package will be generated?

robinsk · 2018-06-14T09:05:48Z

Right, this seems relevant for us. We require advanced networking, and face problems when creating an AKS cluster from Terraform.

When creating the cluster from the Azure Portal, there are 31 "connected devices" on the vnet per NIC (i.e. per host/vm in the cluster), which makes sense with maximum 30 pods per host.

When we create the cluster from Terraform, specifing vnet_subnet_id, there's only one connected device per host, so none of our pods have any networking.

twem · 2018-06-15T11:46:12Z

Also agent_pool_profile, os_type=windows has a dependency on windowsProfile but that is not documented or in the code.

Error code InvalidParameter Message Required parameter windowsProfile is missing (null).

torresdal · 2018-06-15T17:53:02Z

@twem I think Windows agent is still in private preview. At least according to this: https://docs.microsoft.com/en-us/azure/aks/faq

To run Windows Server containers, you need to run Windows Server-based nodes. Windows Server-based nodes are currently in private preview. If you need to run Windows Server containers on Kubernetes in Azure outside of the preview, please see the documentation for acs-engine.

I did try anyway of course :-), using an ARM template with additional agent pool using os_type=windows and windowsProfile defined. It bootstrapped OK, but I got some issues with failing pods in the running cluster:

NAMESPACE     NAME                                    READY     STATUS             RESTARTS   AGE
kube-system   azureproxy-899885bfb-ckhkd              0/1       ImagePullBackOff   0          3h
kube-system   heapster-56c6f9566f-dvrxx               2/2       Running            0          3h
kube-system   kube-dns-v20-7c556f89c5-bbfgw           3/3       Running            0          3h
kube-system   kube-dns-v20-7c556f89c5-spbp2           3/3       Running            0          3h
kube-system   kube-proxy-ksphx                        1/1       Running            0          3h
kube-system   kube-proxy-npkvt                        1/1       Running            0          3h
kube-system   kube-svc-redirect-6cksx                 0/1       CrashLoopBackOff   28         3h
kube-system   kube-svc-redirect-nthhr                 0/1       CrashLoopBackOff   28         3h
kube-system   kubernetes-dashboard-5ffc5c5558-6ht5g   0/1       CrashLoopBackOff   40         3h
kube-system   tunnelfront-7bcb487d9f-kns4q            1/1       Running            0          3h

It could be failing for a different reason though.

dstrebel · 2018-06-15T19:30:58Z

Windows is not available in AKS yet.

tombuildsstuff · 2018-06-20T06:58:19Z

The new/GA API version should be available in v17.4 of the Azure SDK for Go: Azure/azure-sdk-for-go#2069 - once that's merged we'll look into updating to that version and then it should be possible to implement this.

tombuildsstuff · 2018-06-20T07:09:05Z

Upgrading to the GA API also means the Private Networking Route Table bug (detailed in the docs) is resolved: #1197 (comment)

koalalorenzo · 2018-06-20T09:36:46Z

Any ETA for this? :D
What about Azure Active Directory support?

lfshr · 2018-06-21T20:34:27Z

Made a start on this. So far have the containerservices upgraded to 2018-03-31 - This is my first attempt at anything GO, so some guidance would be much appreciated! Feel free to start again and overtake me if you must.

The plan is to fully upgrade to v16.2.1 before I even attempt extra implementations. So far I've only managed containerservices. Will probably open a PR tomorrow for the upgrade standalone if nobody has issues with that.

Package Name	API Version	Status
containerservices	2018-03-31	Done
documentdb	2015-04-08	Was listed in updates - will check
network	2018-05-01	To Do

https://github.com/lfshr/terraform-provider-azurerm/tree/azure-sdk-for-go/v16.2.1

hafizullah · 2018-06-22T08:36:15Z

looks like the sdk fix is released, any plans when this will be merged?

tombuildsstuff · 2018-06-22T11:22:29Z

@lfshr thanks for looking into this. We upgrade the entire SDK holistically; which we're doing in #1418 - that said the changes in your branch otherwise look good. Would you be able to remove the vendoring from your branch and then we should be able to use that once #1418 has been merged :)

Once the SDK's upgrades been merged and we've switched over to the GA version (as in your branch) - it should be possible to add the new functionality :)

Thanks!

lfshr · 2018-06-22T13:46:58Z

@tombuildsstuff no problem :) I'm actually really digging go so far! I'm still coming to grips with how this is all glued together. Shall I leave the inclusion of the containerservices 2018-03-31 vendor in, will that be included in #1418, or shall that go in as a separate PR altogether? I notice it's not currently included in #1418

lfshr · 2018-06-22T14:12:48Z

@tombuildsstuff I've removed the autorest upgrade and the removal of containerservices 2017-09-30 for now. I can't remove the addition of 2018-03-31 as there were a few breaking changes between the two versions. I'm happy for this to go in separately with another cleanup PR to remove 2017-09-30. I'm easy either way.

tombuildsstuff · 2018-06-22T14:19:48Z

@lfshr sounds good. Once the SDK upgrades merged, we can rebase that branch on top and then remove the old API version :)

lfshr · 2018-06-25T12:13:40Z

Hey @tombuildsstuff I see that PR got merged. I've rebased to the latest version of master and stuck my code in a more suitable named branch. Let us know what your plan is.
https://github.com/lfshr/terraform-provider-azurerm/tree/containerservices/2018-03-31

luisdavim · 2018-07-01T16:57:49Z

Any updates on this?

tombuildsstuff · 2018-07-02T06:18:43Z

@lfshr sorry for the delayed response here; would you mind sending a PR for that? Looking at the code, other than fixing the vendoring that branch otherwise looks good to me 👍

lfshr · 2018-07-02T07:42:06Z

@tombuildsstuff should we pull this into a separate branch? This is just the update itself with no new implementations. Looking at the vendor stuff just now.

tombuildsstuff · 2018-07-02T07:47:10Z

@lfshr I'd suggest we upgrade to the new API and then we can add the new fields since they're related, but not dependent tasks (and that'll allow us to fix the routing table bug mentioned above in the interim) - what do you think?

lfshr · 2018-07-02T08:31:31Z

#1474 opened

lfshr · 2018-07-02T18:35:08Z

Started work on advanced networking implementation. Active feedback would be appreciated 😄 PS. I have zero pride. Don't worry about hurting it! 😛
https://github.com/lfshr/terraform-provider-azurerm/tree/containerservices-advancednetworking

Currently most of the params are not flagged as ForceNew. I assume this to be needed for all of them, but I want to test scenarios before jumping to my assumption.

lfshr · 2018-07-02T20:19:26Z

NetworkProfile implementation seems to be finished.. Just tests to add.
https://github.com/lfshr/terraform-provider-azurerm/tree/containerservices-advancednetworking

tombuildsstuff · 2018-07-03T07:39:14Z

Closing this in favour of #1434 since the GA upgrade has been merged in - thanks @lfshr :)

tombuildsstuff · 2018-12-12T08:58:16Z

Since the original issue here's been resolved I'm going to lock this issue for the moment.

katbyte added service/kubernetes-cluster enhancement upstream/microsoft Indicates that there's an upstream issue blocking this issue/PR labels Jun 14, 2018

lfshr mentioned this issue Jun 15, 2018

Phantom routetable created when vnet_subnet_id is provided in azurerm_kubernetes_cluster #1394

Closed

tombuildsstuff mentioned this issue Jun 19, 2018

Container Service API 2018-03-01 Azure/azure-sdk-for-go#2069

Closed

tombuildsstuff removed the upstream/microsoft Indicates that there's an upstream issue blocking this issue/PR label Jun 22, 2018

tombuildsstuff mentioned this issue Jun 22, 2018

Support rbac and azure ad integration for Azure Kubernetes Service #1429

Closed

tombuildsstuff mentioned this issue Jun 25, 2018

Feature Request : addonProfiles and networkProfile for Azure Kubernetes Service (AKS) #1434

Closed

luisdavim mentioned this issue Jun 25, 2018

Terraform resource "azurerm_kubernetes_cluster" does not support aks rbac option MicrosoftDocs/azure-docs#10544

Closed

subesokun mentioned this issue Jun 26, 2018

Setting the vnet_subnet_id causes AKS cluster creation to fail #1444

Closed

lfshr mentioned this issue Jul 3, 2018

AKS: NetworkProfile / Advanced Networking #1479

Merged

tombuildsstuff closed this as completed Jul 3, 2018

This comment has been minimized.

Sign in to view

hashicorp locked as resolved and limited conversation to collaborators Dec 12, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update azurerm_kubernetes_cluster for GA #1389

Update azurerm_kubernetes_cluster for GA #1389

holmesb commented Jun 13, 2018

torresdal commented Jun 14, 2018

robinsk commented Jun 14, 2018

twem commented Jun 15, 2018

torresdal commented Jun 15, 2018

dstrebel commented Jun 15, 2018

tombuildsstuff commented Jun 20, 2018

tombuildsstuff commented Jun 20, 2018 •

edited

Loading

koalalorenzo commented Jun 20, 2018 •

edited

Loading

lfshr commented Jun 21, 2018 •

edited

Loading

hafizullah commented Jun 22, 2018

tombuildsstuff commented Jun 22, 2018

lfshr commented Jun 22, 2018

lfshr commented Jun 22, 2018

tombuildsstuff commented Jun 22, 2018

lfshr commented Jun 25, 2018

luisdavim commented Jul 1, 2018

tombuildsstuff commented Jul 2, 2018

lfshr commented Jul 2, 2018

tombuildsstuff commented Jul 2, 2018

lfshr commented Jul 2, 2018

lfshr commented Jul 2, 2018 •

edited

Loading

lfshr commented Jul 2, 2018

tombuildsstuff commented Jul 3, 2018 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

tombuildsstuff commented Dec 12, 2018

Update azurerm_kubernetes_cluster for GA #1389

Update azurerm_kubernetes_cluster for GA #1389

Comments

holmesb commented Jun 13, 2018

torresdal commented Jun 14, 2018

robinsk commented Jun 14, 2018

twem commented Jun 15, 2018

torresdal commented Jun 15, 2018

dstrebel commented Jun 15, 2018

tombuildsstuff commented Jun 20, 2018

tombuildsstuff commented Jun 20, 2018 • edited Loading

koalalorenzo commented Jun 20, 2018 • edited Loading

lfshr commented Jun 21, 2018 • edited Loading

hafizullah commented Jun 22, 2018

tombuildsstuff commented Jun 22, 2018

lfshr commented Jun 22, 2018

lfshr commented Jun 22, 2018

tombuildsstuff commented Jun 22, 2018

lfshr commented Jun 25, 2018

luisdavim commented Jul 1, 2018

tombuildsstuff commented Jul 2, 2018

lfshr commented Jul 2, 2018

tombuildsstuff commented Jul 2, 2018

lfshr commented Jul 2, 2018

lfshr commented Jul 2, 2018 • edited Loading

lfshr commented Jul 2, 2018

tombuildsstuff commented Jul 3, 2018 • edited Loading

This comment has been minimized.

This comment has been minimized.

tombuildsstuff commented Dec 12, 2018

tombuildsstuff commented Jun 20, 2018 •

edited

Loading

koalalorenzo commented Jun 20, 2018 •

edited

Loading

lfshr commented Jun 21, 2018 •

edited

Loading

lfshr commented Jul 2, 2018 •

edited

Loading

tombuildsstuff commented Jul 3, 2018 •

edited

Loading