Do not apply Egress to traffic destined for ServiceCIDRs

[tnqn]

When AntreaProxy is asked to skip some Services or is not running at all, Pod-to-Service traffic would be forwarded to Egress Node and be load-balanced remotely, as opposed to locally, which could incur performance issue and unexpected behaviors.

This patch installs flows to prevent traffic destined for ServiceCIDRs from being SNAT'd.

Fixes #5494

[tnqn]

/test-all
/test-ipv6-all
/test-ipv6-only-all

[antoninbas]

I am fine with this change.

However, I wonder if instead, we should install flows for skipServices to the AntreaProxy LB table in OVS, and send the packets directly to the host network (same behavior as when AntreaProxy is disabled). This would avoid adding Service flows to the SNAT table. I am being a bit vague here, since I am not super familiar with the current version of the pipeline. This approach may not make sense or be possible.

[tnqn]

However, I wonder if instead, we should install flows for skipServices to the AntreaProxy LB table in OVS, and send the packets directly to the host network (same behavior as when AntreaProxy is disabled). This would avoid adding Service flows to the SNAT table. I am being a bit vague here, since I am not super familiar with the current version of the pipeline. This approach may not make sense or be possible.

Thanks for the comment. I thought it over, and tend to keep the current approach for a few reasons, please let me know whether they make sense to you or not:

1. skipServices is not the only case that can bypass AntreaProxy, services with service.kubernetes.io/service-proxy-name label can also bypass it, we would need to consider both. Excluding all ServiceCIDRs from Egress may be simpler to cover all cases regardless of how AntreaProxy is configured and whether it runs.
2. We have used the way of exception for Node IPs and user configured exceptCIDRs, it may be easier to understand to keep them in the same place.
3. Bypassing most stages for specific traffic flows generally adds complexity, needing careful consideration and could cause some problems. For example, it would be impossible to enforce Egress NetworkPolicy or enable logging for Services skipped via skipServices config, service-proxy-name label.
4. I'm even thinking if it really makes sense to have the current behavior of disabled AntreaProxy. It has several problems: in noEncap/policyOnly mode, Egress NetworkPolicy is not enforced when the Pod accesses ANY destination via Service; in encap mode, Egress NetworkPolicy is not enforced when the Pod accesses external destination/hostNetwork Pod (e.g. kube-apiserver) via Service. If we change to make it go over the pipeline, at least user could configure the service IP or leverage toService peer to enforce Egress NetworkPolicy, though it would be an extra configuration compared with enabled AntreaProxy. We can document it clearly: when AntreaProxy is not enabled or a Service is skipped by AntreaProxy, the Egress rule should use Service-oriented peer instead of Endpoint-oriented peer for Pod-to-Service traffic.

[hongliangl]

LGTM overall, just a small question posted in the following.

[hongliangl]

A basic question: why consume the first tick? Is it for testing?

[tnqn]

The timer is used for retry. We don't want it to trigger InstallSNATBypassServiceFlows before receiving a service CIDR update, which is destined to fail, so consume the first tick to avoid it.

[antoninbas]

@tnqn thanks for thinking it over, it makes sense to me.
If we ever need to exclude more Service traffic for features in the future, maybe it will make sense at that point to consider a classifier and register mark early in the pipeline

[antoninbas]

LGTM

[tnqn]

@tnqn thanks for thinking it over, it makes sense to me. If we ever need to exclude more Service traffic for features in the future, maybe it will make sense at that point to consider a classifier and register mark early in the pipeline

Yes, makes sense.

[tnqn]

/test-e2e

[tnqn]

/test-e2e