Update Calico config for eBPF mode

[hakman]

Based on discussion with @fasaxc in Slack I changed the Calico manifest to use an optional ConfigMap to set the k8s endpoints.
Additionally I added an option for the cleanup of iptables rules (in case it's need for migration) and updated the version to 3.16.4.

/cc @seh @olemarkus

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hakman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [hakman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[olemarkus]

/lgtm

[seh]

Thank you for this one, @hakman. Is it correct that for now, even when Calico's eBPF mode is enabled, we still run kube-proxy, or do we automatically disable those static pods when we're in that mode?

[hakman]

For now kube-proxy is always enabled unless spec.kubeProxy.enabled: false and rolling-update is done.

[seh]

Do you envision ever tying these settings together in kops automatically? That is, if Calico is in eBPF mode, then the default value for "spec.kubeProxy.enabled" would become false?

[hakman]

Can't say never, but you probably understand the complications of doing this.
Cilium can coexist with kube-proxy, so there is no problem there. Not sure if Calico can do that also in the future.