[DualStack] Support NSG and clean LBs #3898

lzhecheng · 2023-05-18T10:55:50Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Support NSG and clean LBs for dualstack
Support related UTs for dualstack
Refactor

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

* Support NSG and clean LBs for dualstack
* Support related UTs for dualstack
* Refactor

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

netlify · 2023-05-18T10:55:57Z

✅ Deploy Preview for kubernetes-sigs-cloud-provide-azure canceled.

Name	Link
🔨 Latest commit	`051a677`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-cloud-provide-azure/deploys/64913ac7b24fdc0008b605dd

k8s-ci-robot · 2023-05-18T10:56:05Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lzhecheng

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [lzhecheng]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

lzhecheng · 2023-05-25T02:50:37Z

/retest

lzhecheng · 2023-06-06T09:38:39Z

/test pull-cloud-provider-azure-e2e-ccm-ipv6-capz
/test pull-cloud-provider-azure-e2e-ccm-ipv6-vmss-capz

lzhecheng · 2023-06-07T02:28:51Z

/test pull-cloud-provider-azure-e2e-ccm-ipv6-capz
/test pull-cloud-provider-azure-e2e-ccm-ipv6-vmss-capz

lzhecheng · 2023-06-07T05:45:37Z

/retest

coveralls · 2023-06-07T12:55:40Z

coverage: 79.914% (+0.03%) from 79.885% when pulling 98a237d on lzhecheng:dualstack-nsg into e5a3523 on kubernetes-sigs:master.

lzhecheng · 2023-06-08T06:19:20Z

/retest

jwtty · 2023-06-09T03:50:06Z

tests/e2e/network/node.go

@@ -197,7 +197,7 @@ var _ = Describe("Azure node resources", Label(utils.TestSuiteLabelNode), func()

 		utils.Logf("getting all NICs of VMSS VMs")
 		for _, vmssVM := range vmssVMs {
-			utils.Logf("Checking %d VMSS VM %q", vmssVM.Name)
+			utils.Logf("Checking %d VMSS VM %q", *vmssVM.Name)


jwtty · 2023-06-09T04:23:21Z

pkg/provider/azure_vmssflex.go

@@ -989,7 +989,9 @@ func (fs *FlexScaleSet) ensureBackendPoolDeletedFromNode(vmssFlexVMNameMap map[s
 			nics[nicName] = nic
 		}
 	}
+	nicUpdaterLock := "nicUpdaterLock"


+1 atomic.Store() may be better than lock?

Will update in the other PR and rebase.

jwtty · 2023-06-09T05:03:07Z

pkg/provider/azure_privatelinkservice.go

+			klog.V(2).Infof("IPv6 is not supported for private link service, skip reconcilePrivateLinkService for service(%s)", serviceName)
+			return nil
+		}
+		if !isDualStack {


We don't need this if at all?

MartinForReal · 2023-06-12T02:38:34Z

pkg/provider/azure_loadbalancer_backendpool.go

@@ -135,10 +135,10 @@ func (bc *backendPoolTypeNodeIPConfig) CleanupVMSetFromBackendPoolByCondition(sl
 			})
 		}
 		if v4Enabled {
-			findBackendpoolToBeDeleted(false)
+			findBackendpoolToBeDeleted(IsIPv4)


can we call findBackendpoolToBeDeleted directly?

MartinForReal · 2023-06-12T03:29:45Z

pkg/provider/azure_standard.go

@@ -296,8 +296,8 @@ func getBackendPoolName(clusterName string, isIPv6 bool) string {
 // getBackendPoolNames returns the IPv4 and IPv6 backend pool names.
 func getBackendPoolNames(clusterName string) map[bool]string {
 	return map[bool]string{
-		false: getBackendPoolName(clusterName, false),
-		true:  getBackendPoolName(clusterName, true),
+		IsIPv4: getBackendPoolName(clusterName, IsIPv4),


can we define a enum type for IsIPv4 and IsIPv6? I'm not sure these two variables are mutually exclusive.

IsIPv4 is actually NotIPv6, so they are mutually exclusive.

lzhecheng · 2023-06-12T07:18:53Z

/test pull-cloud-provider-azure-e2e-ccm-ipv6-capz
/test pull-cloud-provider-azure-e2e-ccm-ipv6-vmss-capz
/test pull-cloud-provider-azure-e2e-ccm-dualstack-capz
/test pull-cloud-provider-azure-e2e-ccm-dualstack-vmss-capz

lzhecheng · 2023-06-13T02:06:43Z

/test pull-cloud-provider-azure-e2e-ccm-capz

MartinForReal · 2023-06-15T02:54:48Z

tests/e2e/network/node.go

@@ -197,7 +197,7 @@ var _ = Describe("Azure node resources", Label(utils.TestSuiteLabelNode), func()

 		utils.Logf("getting all NICs of VMSS VMs")
 		for _, vmssVM := range vmssVMs {
-			utils.Logf("Checking %d VMSS VM %q", vmssVM.Name)
+			utils.Logf("Checking VMSS VM %q", *vmssVM.Name)


shall we safely refer to this variable securely by invoking to. String?

lzhecheng · 2023-06-15T10:10:18Z

/retest

nilo19 · 2023-06-19T06:45:26Z

pkg/provider/azure_loadbalancer.go

@@ -68,10 +68,10 @@ func (az *Cloud) existsPip(clusterName string, service *v1.Service) bool {
 		return existingPip
 	}

-	if v4Enabled && !existsPipSingleStack(false) {
+	if v4Enabled && !existsPipSingleStack(IsIPv4) {


I think a better name would be IPVersionIPv4. IsIPv4 sounds like a variable, not a constant.

nilo19 · 2023-06-19T07:12:53Z

pkg/provider/azure_loadbalancer.go

+// Before DualStack support, old logic takes the first ingress IP as non-additional one
+// and the second one as additional one. With DualStack support, the second IP may be
+// the IP of another IP family so the new logic returns two variables.
+func (az *Cloud) getServiceLoadBalancerStatus(service *v1.Service, lb *network.LoadBalancer) (status *v1.LoadBalancerStatus, lbIPsNoAdditionalPIPs []string, fipConfigs []*network.FrontendIPConfiguration, err error) {


is noAdditional means primary?

It means compared to ingress IPs in status, lbIPsNoAdditionalPIPs doesn't include those IPs from service.beta.kubernetes.io/azure-additional-public-ips annotation.

Will it be better to replace to primary?

Yes, NoAdditional seems too straightforward. Replaced.

nilo19 · 2023-06-19T07:30:55Z

pkg/provider/azure_utils.go

@@ -34,6 +34,11 @@ import (
 	"sigs.k8s.io/cloud-provider-azure/pkg/consts"
 )

+const (
+	IsIPv6 bool = true


move to consts.go?

Signed-off-by: Zhecheng Li <zhechengli@microsoft.com>

* Support NSG and clean LBs for dualstack * Support related UTs for dualstack * Refactor Signed-off-by: Zhecheng Li <zhechengli@microsoft.com>

nilo19 · 2023-06-21T06:29:19Z

/lgtm

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels May 18, 2023

k8s-ci-robot requested a review from jwtty May 18, 2023 10:56

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label May 18, 2023

k8s-ci-robot requested a review from nilo19 May 18, 2023 10:56

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 18, 2023

lzhecheng force-pushed the dualstack-nsg branch from acc8ea5 to c53a811 Compare May 23, 2023 09:53

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 25, 2023

lzhecheng force-pushed the dualstack-nsg branch from c53a811 to 141670b Compare May 25, 2023 02:52

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 25, 2023

lzhecheng force-pushed the dualstack-nsg branch 3 times, most recently from a81bbfd to 7217c25 Compare May 30, 2023 09:04

lzhecheng force-pushed the dualstack-nsg branch 2 times, most recently from 669012f to ed6e470 Compare June 6, 2023 09:35

lzhecheng force-pushed the dualstack-nsg branch 2 times, most recently from d814ca9 to 59de6d3 Compare June 7, 2023 01:54

lzhecheng force-pushed the dualstack-nsg branch 2 times, most recently from 4192a01 to 98a237d Compare June 7, 2023 12:03

lzhecheng force-pushed the dualstack-nsg branch 2 times, most recently from e02069b to 4ca9b1e Compare June 8, 2023 13:03

lzhecheng force-pushed the dualstack-nsg branch from 4ca9b1e to 24050fb Compare June 8, 2023 14:51

jwtty reviewed Jun 9, 2023

View reviewed changes

lzhecheng force-pushed the dualstack-nsg branch from 24050fb to 07ac09f Compare June 9, 2023 06:03

MartinForReal reviewed Jun 12, 2023

View reviewed changes

lzhecheng force-pushed the dualstack-nsg branch from 07ac09f to d352258 Compare June 12, 2023 06:38

lzhecheng force-pushed the dualstack-nsg branch from d352258 to d5ebc38 Compare June 14, 2023 08:29

MartinForReal reviewed Jun 15, 2023

View reviewed changes

lzhecheng force-pushed the dualstack-nsg branch from d5ebc38 to 27e816a Compare June 15, 2023 05:10

nilo19 reviewed Jun 19, 2023

View reviewed changes

lzhecheng force-pushed the dualstack-nsg branch 2 times, most recently from 53d35e2 to 813fd2f Compare June 19, 2023 10:01

lzhecheng added 2 commits June 20, 2023 05:35

Use const vars to define isIPv6: false & true

1fb762e

Signed-off-by: Zhecheng Li <zhechengli@microsoft.com>

[DualStack] Support NSG and clean LBs

051a677

* Support NSG and clean LBs for dualstack * Support related UTs for dualstack * Refactor Signed-off-by: Zhecheng Li <zhechengli@microsoft.com>

lzhecheng force-pushed the dualstack-nsg branch from 813fd2f to 051a677 Compare June 20, 2023 05:36

k8s-ci-robot assigned nilo19 Jun 21, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 21, 2023

k8s-ci-robot merged commit 12b808f into kubernetes-sigs:master Jun 21, 2023
21 checks passed

lzhecheng deleted the dualstack-nsg branch June 21, 2023 13:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DualStack] Support NSG and clean LBs #3898

[DualStack] Support NSG and clean LBs #3898

lzhecheng commented May 18, 2023

netlify bot commented May 18, 2023 •

edited

k8s-ci-robot commented May 18, 2023

lzhecheng commented May 25, 2023

lzhecheng commented Jun 6, 2023

lzhecheng commented Jun 7, 2023

lzhecheng commented Jun 7, 2023

coveralls commented Jun 7, 2023

lzhecheng commented Jun 8, 2023

jwtty Jun 9, 2023

lzhecheng Jun 9, 2023

jwtty Jun 9, 2023

lzhecheng Jun 9, 2023

jwtty Jun 9, 2023

lzhecheng Jun 9, 2023

MartinForReal Jun 12, 2023

MartinForReal Jun 12, 2023

lzhecheng Jun 12, 2023

lzhecheng commented Jun 12, 2023

lzhecheng commented Jun 13, 2023

MartinForReal Jun 15, 2023

lzhecheng Jun 15, 2023

lzhecheng commented Jun 15, 2023

nilo19 Jun 19, 2023

lzhecheng Jun 19, 2023

nilo19 Jun 19, 2023

lzhecheng Jun 19, 2023

nilo19 Jun 19, 2023

lzhecheng Jun 20, 2023

nilo19 Jun 19, 2023

lzhecheng Jun 19, 2023

nilo19 commented Jun 21, 2023

[DualStack] Support NSG and clean LBs #3898

[DualStack] Support NSG and clean LBs #3898

Conversation

lzhecheng commented May 18, 2023

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

netlify bot commented May 18, 2023 • edited

✅ Deploy Preview for kubernetes-sigs-cloud-provide-azure canceled.

k8s-ci-robot commented May 18, 2023

lzhecheng commented May 25, 2023

lzhecheng commented Jun 6, 2023

lzhecheng commented Jun 7, 2023

lzhecheng commented Jun 7, 2023

coveralls commented Jun 7, 2023

lzhecheng commented Jun 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lzhecheng commented Jun 12, 2023

lzhecheng commented Jun 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lzhecheng commented Jun 15, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nilo19 commented Jun 21, 2023

netlify bot commented May 18, 2023 •

edited