Remove kube-proxy autocleanup for inactive modes

[vllry]

What type of PR is this?
/kind bug

What this PR does / why we need it:
kube-proxy attempts to clean up network rules for other modes (EG cleaning up iptables rules when running in IPVS mode). The complexity of this code is prone to bugs, and can delay kube-proxy's readiness between restarts.

This PR see KEP aims to remove auto-cleanup logic for non-current kube-proxy modes. In other words, kube-proxy will only automatically clean up rules relevant to its current mode. Users should use --cleanup or restart the node when switching between kube-proxy modes.

Which issue(s) this PR fixes:
Fixes #75408 (tracking issue)
Fixes #75360 (bug caused by auto-cleanup)

Special notes for your reviewer:
Worth discussing if we gate this behavior with a flag (EG --only-clean-current-mode), or outright GA. Outright GA is currently the plan.

Does this PR introduce a user-facing change?:

kube-proxy no longer automatically cleans up network rules created by running kube-proxy in other modes. If you are switching the mode that kube-proxy is in running in (EG: iptables to IPVS), you will need to run `kube-proxy --cleanup`, or restart the worker node (recommended) before restarting kube-proxy.

If you are not switching kube-proxy between different modes, this change should not require any action.

[vllry]

/priority important-soon
/sig network
/assign @thockin

[thockin]

I keep looking for something missing here, but I can't find it.

We should fix the description in cmd/kube-proxy/app/server.go on the cleanup-ipvs flag: "cleanup ipvs rules before running" is no longer true, right?

Otherwise LGTM

[vllry]

We should fix the description in cmd/kube-proxy/app/server.go on the cleanup-ipvs flag: "cleanup ipvs rules before running" is no longer true, right?

Correct, I'll fix that.

We also need docs changes, which I have no idea how to coordinate.

[thockin]

Docs are in a different repo, so there's no coordination per se. You make a doc PR, it gets targetted at the 1.15 release.

…

On Wed, Apr 3, 2019 at 4:52 PM Kubernetes Prow Robot < ***@***.***> wrote: @vllry <https://github.com/vllry>: The following test *failed*, say /retest to rerun them all: Test name Commit Details Rerun command pull-kubernetes-e2e-gce cdce2d0 <cdce2d0> link <https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/76109/pull-kubernetes-e2e-gce/1113572795524059139/> /test pull-kubernetes-e2e-gce Full PR test history <https://prow.k8s.io/pr-history?org=kubernetes&repo=kubernetes&pr=76109>. Your PR dashboard <https://gubernator.k8s.io/pr/vllry>. Please help us cut down on flakes by linking to <https://git.k8s.io/community/contributors/devel/flaky-tests.md#filing-issues-for-flaky-tests> an open issue <https://github.com/kubernetes/kubernetes/issues?q=is:issue+is:open> when you hit one in your PR. Instructions for interacting with me using PR comments are available here <https://git.k8s.io/community/contributors/guide/pull-requests.md>. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra <https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:> repository. I understand the commands that are listed here <https://go.k8s.io/bot-commands>. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#76109 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFVgVLgOEUpQgUR_f4s5o2_Au6BvPxEwks5vdT6xgaJpZM4cblMc> .

[vllry]

/test pull-kubernetes-e2e-gce

[thockin]

/lgtm
/approve

[andrewsykim]

Even though it's implied, I think the release note should say something about --cleanup-ipvs being deprecated and no longer having any effect. Should we also mark the release note for this as "action required" since we are being more explicit about users either setting --cleanup or rebooting nodes during a proxy mode switch? cc v1.14 patch release team @aleksandra-malinowska @spiffxp @tpepper since this is going into v1.14.1

[thockin]

`--cleanup-ipvs` still has meaning. In IPVS mode `--cleanup` means "do low-impact cleanup and exit" while `--cleanup --cleanup-ipvs` means "do full cleanup and exit"

…

On Thu, Apr 4, 2019 at 3:17 PM Andrew Sy Kim ***@***.***> wrote: Even though it's implied, I think the release note should say something about --cleanup-ipvs being deprecated and no longer having any effect. Should we also mark the release note for this as "action required" since we are being more explicit about users either setting --cleanup or rebooting nodes during a proxy mode switch? cc v1.14 patch release team @aleksandra-malinowska @spiffxp @tpepper — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.

[andrewsykim]

In IPVS mode --cleanup means "do low-impact cleanup and exit" while --cleanup --cleanup-ipvs means "do full cleanup and exit"

Sorry if I'm missing something, but I'm not seeing the changes in this PR reflect this 🤔 The current changes indicate --cleanup-ipvs has no effect and --cleanup will always do full clean up. Fwiw I prefer the current changes where we just deprecate --cleanup-ipvs but would like to make sure we're on the same page

/hold

[vllry]

Gotcha. I'll push up a change shortly, have to run to a meeting.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: thockin, vllry

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cmd/OWNERS [thockin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[andrewsykim]

/lgtm

[andrewsykim]

/hold cancel

[vllry]

@thockin @andrewsykim issues resolved + tests finally passed, we're good to go.

[ravilr]

@vllry @andrewsykim can this be cherry-picked to release-1.13 also please?