Azure clusters in existing resource groups

[rfranzke]

Some time back we have disabled deployments into existing Azure resource groups and existing Azure VNets because the Azure cloud provider implementation did not clean up self-created resources properly (tested with version 1.7.6).

Has the Azure cloud provider been improved in that regards so that we can re-enable it again?

Status:

• Enable deploying shoots into existing vNETs.
  ref: Azure: Allow deployments into existing vNets gardener#1558 Azure: Allow to deploy cluster in existing vNets gardener-attic/gardener-extensions#371
• Enable deploying shoots into existing resourceGroups

[rfranzke]

cc @dkistner @AndreasBurger

[dkistner]

Currently not. It seems, that there are still issues with orphaned iaas resources after LB service deletion exists. See here: kubernetes/kubernetes#59255
They are already working on a fix for this issue, but I'm not sure if this is the only remaining issue. Further investigation is required.

[rfranzke]

FYI: https://github.com/kubernetes/community/blob/master/keps/sig-azure/0025-20180809-cross-resource-group-nodes.md

[rfranzke]

@dkistner any updates here for Kubernetes 1.{9,10,11,12}?

[rfranzke]

kubernetes/kubernetes#67604 Maybe relevant?

[rfranzke]

@dkistner any update here?

[vlerenc]

Ping... @dkistner.

[dkistner]

I think there is no update as we still see also with latest k8s versions (v1.13, 1.14) left overs in Azure, we would not get rid of them when we delete the cluster. Currently we manage the ResourceGroup and drop it when the Shoot gets deleted. All left overs will be deleted during this process. This would not be the case for existing ResourceGroups because then we would not delete them.

My advise: Wait until we see no left overs anymore and when we have enough trust that the clean up works reliable.

[rfranzke]

@dkistner what I understand from the Kubernetes 1.15 release notes this scenario should become possible, right?

[dkistner]

We are currently testing if the infrastructure left overs are gone with the changes in k8s v1.15. I would recommend to re-enable this only if we are sure that we are really not experience any leaked resources. Let's wait for our test results.

[dkistner]

We will enable Azure Shoot deployments into existing vNets, which are located in a different resource group as the Shoot resources.

Current status:

• MCM adaption: Azure: Allow machines assigned to vnet in a different resource group machine-controller-manager#344 (done)
• Gardener extension adaption: Azure: Allow to deploy cluster in existing vNets gardener-attic/gardener-extensions#371 (done)
• Gardener: Azure: Allow deployments into existing vNets gardener#1558 (done)

In general I see with this change no reason why we should re-enable deployments in existing resource groups. We had that only because the Azure Kubernetes provider supported only deployment in existing vNets which are in the same resource group as the cluster resources and we disabled it due to the orphan resource issues (which partly still exists). Meanwhile the Azure Kubernetes provider support also deployments in existing vNets which are in other resource groups (this scenario is now implemented).
Therefore I would suggest to remove the inactive validation logic in the Gardener for the existing resource group case and close this issue afterwards. WDYT? @rfranzke, @AndreasBurger, @vlerenc

[vpnachev]

@dkistner can we close this issue ?

[dkistner]

No only the deployment into existing vnets is implemented. The second part deployment into existing resource groups is still in discussion.

[vpnachev]

ok. I have updated the description of this issue and marked the vNETs as implemented.

[muenchdo]

We would be interested in this as we are currently in discussions with a customer about deploying a shoot into their Azure subscription. Being able to create resource groups seems to require quite broad permissions which we would like to avoid.

[dkistner]

As far a I recall correctly the only argument against Shoot deployment into existing resource groups are the left overs which could remain on the infrastructure when the cluster is deleted. Unfortunately the Azure cloud provider have still issues with that.
So yes, I think we can discuss to enable that.
Btw: PRs for that are warmly welcome :)

[muenchdo]

So you would say it could be enabled again and there would be a disclaimer somewhere that it can result in orphaned resources?

[dkistner]

Yes, something like this I could image. Users have to be aware that some left overs can remain if the cluster is gone.

[rfranzke]

@muenchdo wanna give it a try? If you need help just ping, happy to guide you