Creating AKS clusters with custom vnet breaks the aks overlay network

[aemneina]

Rancher versions:
rancher/server or rancher/rancher: 2.0.7

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
Azure AKS

Steps to Reproduce:
I tried 4 permutations of this, but basically

1. create vnet (can be within the same resource group or not, doesnt seem to make a difference)
2. create an azure cluster within rancher UI
3. open the advanced options for nodes and define the desired VNET
4. provision cluster

Results:
After the cluster comes up, things seem to work. If you try to get pods to communicate with each other across nodes, they'll fail. I did a similar task within the Azure portal and that created a working cluster.

[nathan-jenan-rancher]

To fix this issue we need to update to the latest version of the azure go sdk and make sure the following fields are included in the API call:

networkPlugin
serviceCidr
dnsServiceIP
dockerBridgeCidr

networkPlugin can be safely defaulted to azure when a vnet is supplied, but the remaining options must be restricted based on the virtual network selected.

The serviceCidr must not fall within the address range of the selected virtual network. Azure defaults this value to 10.0.0.0/16, although it seems possible that this would overlap with the virtual network selected, so we cannot just default this field.

The dnsServiceIP field must be within the serviceCidr range. So this also cannot be hardcoded. This field defaults to 10.0.0.10 on the azure ui.

dockerBridgeCidr defaults to 172.17.0.1/16 on the azure UI. It cannot overlap with the serviceCidr or the address range of the virtual network.

[nathan-jenan-rancher]

Backend API changes are in and can be tested but still needs UI work.

[nathan-jenan-rancher]

The following fields are now exposed as part of the AKS driver config:

serviceCidr
dnsServiceIp
dockerBridgeCidr

We need some validation on the UI.
serviceCidr cannot overlap the virtual network address range
dnsServiceIp must fall within the serviceCidr
dockerBridgeCidr must not overlap the virtual network address range or the serviceCidr

[westlywright]

ui changes are in latest

[soumyalj]

Tested with v2.1.0-rc1. Created a AKS cluster with a custom virtual network and providing advanced options below:

1. Kubernetes service address range
2. Kubernetes DNS service IP address
3. Docker bridge address range

Cluster got created successfully.
Created few pods and pod communication across nodes was successful

root@wk1-6c8fb499d4-6swq4:/# curl http://11.0.0.11/name.html
wk2-5746cbf558-6lp2k
root@wk1-6c8fb499d4-6swq4:/# curl http://11.0.0.19/name.html
wk1-6c8fb499d4-s69ww
root@wk1-6c8fb499d4-6swq4:/# curl http://11.0.0.46/name.html
wk2-5746cbf558-7sqzf
root@wk1-6c8fb499d4-6swq4:/#
root@wk2-5746cbf558-7sqzf:/# curl http://11.0.0.86/name.html
wk1-6c8fb499d4-6swq4

DNS resolution was also fine

root@wk2-5746cbf558-7sqzf:/# dig extdns.default.svc.cluster.local +short
www.google.com.
172.217.10.228
root@wk2-5746cbf558-7sqzf:/# dig wk1dns.default.svc.cluster.local +short
11.0.0.19
11.0.0.86
root@wk2-5746cbf558-7sqzf:/#