New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure orphan public IPs deleted #70463

Merged
merged 1 commit into from Oct 31, 2018

Conversation

@feiskyer
Member

feiskyer commented Oct 31, 2018

What type of PR is this?

Uncomment only one, leave it on its own line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:

After kube-controller-manager restarted, it may get stuck in ensure load balancer as it's trying to delete a public IP that's referenced by an orphan frontend configuration which is no longer referenced by any k8s service.

This PR removed referenced frontend IP configurations and load balancer rules before deleting the public IP.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #69352

Special notes for your reviewer:

Note: this is a workaround for #69352, it only deletes orphan public IPs when service recreated with the same name. To complete resolve orphaned load balancer issue, service finalizer should be added (probably v1.14).

Does this PR introduce a user-facing change?:

Ensure orphan public IPs on Azure deleted when service recreated with the same name.

/sig azure
/kind bug
/milestone v1.13
/cc @brendandburns @khenidak @andyzhangx

@k8s-ci-robot

This comment has been minimized.

Contributor

k8s-ci-robot commented Oct 31, 2018

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: feiskyer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@feiskyer

This comment has been minimized.

Member

feiskyer commented Oct 31, 2018

/retest

@andyzhangx

LGTM overall, leave some minor comments

return err
}
}
glog.V(10).Infof("DeletePublicIPWithRetry(%s, %q): end", pipResourceGroup, pipName) // response not read yet...

This comment has been minimized.

@andyzhangx

andyzhangx Oct 31, 2018

Member

read --> ready?

This comment has been minimized.

@feiskyer

feiskyer Oct 31, 2018

Member

The comment is actually misleading, let me remove it.

frontendIPConfigUpdated := false
loadBalancerRuleUpdated := false
// Check whether there are still frontend IP configurations referring it.

This comment has been minimized.

@andyzhangx

andyzhangx Oct 31, 2018

Member

referring to

This comment has been minimized.

@feiskyer

feiskyer Oct 31, 2018

Member

Updated.

@yastij

yastij approved these changes Oct 31, 2018

/lgtm

@feiskyer

This comment has been minimized.

Member

feiskyer commented Oct 31, 2018

/test pull-kubernetes-integration

@yastij

This comment has been minimized.

Member

yastij commented Oct 31, 2018

@feiskyer - I'm also able to reproduce another edge case: Delete a loadbalancer service then recreate it before the cleanup. The service will still a pending state the front-end IP being not cleaned up. (this is not a blocker for this one though)

@k8s-ci-robot k8s-ci-robot merged commit dc23974 into kubernetes:master Oct 31, 2018

18 checks passed

cla/linuxfoundation feiskyer authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gke Skipped
pull-kubernetes-e2e-kops-aws Job succeeded.
Details
pull-kubernetes-e2e-kubeadm-gce Skipped
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped
pull-kubernetes-local-e2e-containerized Skipped
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
tide In merge pool.
Details

@feiskyer feiskyer deleted the feiskyer:pip-safe-clean branch Nov 1, 2018

k8s-ci-robot added a commit that referenced this pull request Nov 2, 2018

Merge pull request #70517 from feiskyer/automated-cherry-pick-of-#70463
…-upstream-release-1.12

Automated cherry pick of #70463: Ensure orphan public IPs deleted

k8s-ci-robot added a commit that referenced this pull request Nov 5, 2018

Merge pull request #70519 from feiskyer/automated-cherry-pick-of-#70463
…-upstream-release-1.10

Automated cherry pick of #70463: Ensure orphan public IPs deleted
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment