IC/ICNI: Remove the need for k8s.ovn.org/external-gw-pod-ips annotation in

[tssurya]

- What this PR does and why is it needed
Currently in legacy ICNI, we use the k8s.ovn.org/external-gw-pod-ips as a way for communicating the change in an ICNI gateway pod's status (add/delete/update) from the ovnkube-controller (ovnkube-master in nonIC) to ovnkube-node. In IC multi-zone since the ovnkube-controller and ovnkube-node run as the same process, this is not needed. We can remove dependeny on the extra annotation update we do for IC multizone deployments. We flush the conntrack on ovnkube-controller directly.

On nonIC and IC with single zone we still use the annotation approach for flushing.

- Special notes for reviewers
Review the namespace update bits carefully.

- How to verify it
E2Es present for signalling issues. Ran v6 ones locally since CI does not run it. Need to dust off #3512 because some of the v6 ones are failing on the setup parts :/ so TODO to fix that in future, but as of now v4 ones pass and v6 should follow the same logic as far as code is concerned.

- Description for the changelog
Remove the need for k8s.ovn.org/external-gw-pod-ips annotation in IC/ICNI

[coveralls]

Changes unknown when pulling fd2435a on tssurya:skeleton into ** on ovn-org:master**.

[trozet]

why is this conditionalized to single node zone IC? Isn't this something generic you are fixing?

[tssurya]

its conditionalized to multizone IC (not single zone).
because for nonIC and single zone IC we want to use the tranditional legacy method of annotation flush. The annotation k8s.ovn.org/external-gw-pod-ips must only contain ips of the nsInfo.routingExternalPodGWs NOT nsInfo.routingExternalGWs (this is already present as part of k8s.ovn.org/routing-external-gws so we don't want to duplicate it.) So TLDR; this condition is to compliment the

} else {
		// flush here since we know we have added an egressgw pod and we also know the full list of existing gatewayIPs

below this block.

[tssurya]

Summarizing 2 Failures:

[Fail] Services when a nodePort service targeting a pod with hostNetwork:false is created when tests are run towards the agnhost echo service [It] queries to the nodePort service shall work for TCP 
/home/runner/work/ovn-kubernetes/ovn-kubernetes/test/e2e/service.go:254

[Fail] Services when a nodePort service targeting a pod with hostNetwork:false is created when tests are run towards the agnhost echo service [It] queries to the nodePort service shall work for TCP 
/home/runner/work/ovn-kubernetes/ovn-kubernetes/test/e2e/service.go:254

I think we broke something in these tests recently, they are flaking way more than they use to.

[openshift-merge-robot]

Fix included in accepted release 4.15.0-0.nightly-2023-09-29-095642