Allow switching kube-proxy mode for k8s cluster > 1.16

[DockToFuture]

What this PR does / why we need it:
This PR lifts the restrictions on kube-proxy to allow switching of kube-proxy mode (IPTables, IPVS) for k8s cluster > 1.16. Further the iptables and ipvs rules of the previous mode are cleaned up in an init container before running kube-proxy in the new mode.

PR kubernetes/kubernetes#78775 fixed the cleanup error in v1.16 which led in the past to the restrictions.

Which issue(s) this PR fixes:
Fixes #2225

Special notes for your reviewer:

Release note:

Restrictions on kube-proxy are lifted to allow switching of kube-proxy mode (IPTables, IPVS) for k8s cluster > 1.16. 

[ialidzhikov]

@DockToFuture, can you update the PR description with the corresponding k/k issue and provide details about it?

[DockToFuture]

Will update it, please wait with merging.

[rfranzke]

/kind/enhancement

[rfranzke]

@DockToFuture what's the status of this PR?

[DockToFuture]

Before we switch the proxy-mode we have to run kube-proxy with the --cleanup flag and wait until it successfully terminates otherwise the ipvs rules or iptables rules won't be properly purged. It will not work after the mode is switched. Therefore I'm looking for a proper way how to implement it.

[vpnachev]

An init container with kube-proxy --cleanup should do the job, what do you think?

[DockToFuture]

This cleans up the current mode not the one from which kube-proxy has switched.

[zanetworker]

would a prestop hook to do the cleanup help here?

[rfranzke]

A prestop hook would be executed whenever the kube-proxy pod stops, so I don't think that's what we want. Instead, would it be viable to go with @vpnachev's idea? Could we add an init container that reads the configured mode from the kube-proxy config and then decides whether to cleanup?

Basically: When the mode is iptables then check if there are IPVS artifacts on the node, and if yes then run the --cleanup job, if not simply exit 0. If the mode is IPVS then check if there are iptable artifacts on the node, and if yes then run the --cleanup job, if not simply exit 0.

If that's not possible then we would have to "remember" the old mode and then decide in code when to spawn the --cleanup job. However, this would be more complex. So can we achieve above suggestion?

[vpnachev]

When the mode is iptables then check if there are IPVS artifacts on the node, and if yes then run the --cleanup job, if not simply exit 0. If the mode is IPVS then check if there are iptable artifacts on the node, and if yes then run the --cleanup job, if not simply exit 0.

This was exactly what I was thinking last night. I am just not sure how to properly detect existing artifacts, so maybe the init container can dump the current active mode in a file on the host, e.g. /kube-proxy-mode.

OLD_KUBE_PROXY_MODE=$(cat /kube-proxy-mode)
[[ ${OLD_KUBE_PROXY_MODE} == ${KUBE_PROXY_MODE} ]] && exit 0
kube-proxy --clean-up || exit $?
echo ${KUBE_PROXY_MODE} >/kube-proxy-mode

The KUBE_PROXY_MODE variable can be set by gardener in the daemonset, or can be read from /var/lib/kube-proxy-config/config.yaml file.

[rfranzke]

+💯 @vpnachev

[zanetworker]

@rfranzke the --cleanup flag cleans up both IPVS and IPtables according to the documentation, do we need the check here? Or is that not the case @DockToFuture

--cleanup
  If true cleanup iptables and ipvs rules and exit.

[rfranzke]

You need the check because you only want to run the cleanup if you switch the mode. Otherwise, we end up with the same problem like the pre-stop hook - it would be executed everytime when kube-proxy starts. 😁

[rfranzke]

Tested it out, works great, just a few left-over cosmetics, otherwise lgtm.

[vpnachev]

Thank you!

It looks very well now, I have just a few minor change requests.

[vpnachev]

/lgtm