Ensure Azure load balancer cleaned up on 404 or 403

[feiskyer]

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:

When deleting LoadBalancer services, Azure may return 404 or 403. This is usually caused by wrong annotations configured on the service spec.

Currently, an error is reported in EnsureLoadBalancerDeleted(), so that the service controller to retry deleting it again. However, then 404 or 403 is reported, this retry won't succeed. And if you create a new service with the same name, it will always fail.

This PR fixes the issue by checking the response codes, and reports nil on 404 and 403.

Which issue(s) this PR fixes:

Fixes #75198

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Ensure Azure load balancer cleaned up on 404 or 403 when deleting LoadBalancer services.

/sig azure
/kind bug
/priority critical-urgent
/milestone v1.14
/cc @andyzhangx @khenidak @weinong

[k8s-ci-robot]

@feiskyer: GitHub didn't allow me to request PR reviews from the following users: weinong.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:

When deleting LoadBalancer services, Azure may return 404 or 403. This is usually caused by wrong annotations configured on the service spec.

Currently, an error is reported in EnsureLoadBalancerDeleted(), so that the service controller to retry deleting it again. However, then 404 or 403 is reported, this retry won't succeed. And if you create a new service with the same name, it will always fail.

This PR fixes the issue by checking the response codes, and reports nil on 404 and 403.

Which issue(s) this PR fixes:

Fixes #75198

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Ensure Azure load balancer cleaned up on 404 or 403 when deleting LoadBalancer services.

/sig azure
/kind bug
/priority critical-urgent
/milestone v1.14
/cc @andyzhangx @khenidak @weinong

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[feiskyer]

/test pull-kubernetes-e2e-aks-engine-azure

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: feiskyer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/cloudprovider/providers/azure/OWNERS [feiskyer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[feiskyer]

@andyzhangx Could you help to take a look?

[andyzhangx]

/lgtm

[djsly]

thanks @feiskyer, we just discovered that issue on 1.13.4. Some users set the wrong idle timeout (3 which isn't a valid value) and another user set a wrong resource group name. Both case, the service creation failed and the service got stuck in Deleting state with the same root cause. Users tried to delete recreate the same service (name + resourcegroup name == same) and the controller manager was ignoring the creation request since it was stuck trying to delete.

After wiping the kube-controller-manager's memory (restart service) all was good.

I'm guessing those PR/Cherry Pick are there to fix that exact issue ?

[feiskyer]

@djsly Yep, cherry picking the fixes to all stable releases.