How egress IPs are visible to the external network

[Edward033]

Hi,

I’m analyzing the egress IP feature (https://antrea.io/docs/v1.13.1/docs/egress/) as a mechanism to enforce more granular controls on our web proxies.

I’m curious how egress IPs are “visible” or “expose” to the external network. If my egresses IPs are not interface nodes, do I need to add them to my load balancer pool? (In my environment MetallLB on L2 mode) or the egress IP has the ability to respond to external ARP requests without additional configuration on the Kube side? I need to configure the external network accordingly of course. I’ll be testing this feature in my lab environment, but wanted to clarify the expectations before testing.

Thanks,
Edward.

[tnqn]

Antrea takes care of responding external ARP requests as well as advertising Egress IPs, you don't need additional configuration other than configuring ExternalIPPool and Egress.

If you just allocate Egress IPs from the same subnet as the Node IPs, you don't need configure external network either. In other cases, you need to configure external network to route traffic towards the Egress IP's subnet to the Node's network.

[Edward033]

Hey @tnqn,

Thanks for the helpful response. Given certain limitations in my environment I've configured a separate network from the worker nodes for the egress IPs. As you mentioned, I had to configure the external network to route the traffic to the egress IPs.

I found a solution using a static route with an interface as the next-hop. This interface is the VLAN interface of the Kubernetes workers. With this setup, when routing happens, the router ARPs directly for the destination IP (Egress IP) instead of ARPing for the IP of the next-hop. This is a working solution functioning on our lab environment. However, my Network team won't accept this solution in Production because we don't have this type of static routing anywhere in our network.

Since each Kube worker is configured with a different IP, I think I can't use a static route pointing directly to a Kube worker (static route with IP as the next-hop). Someone also mentioned the possibility of advertising the route to the external network via BGP from Kubernetes. This idea was quickly dismissed by the team that manages the Kubernetes clusters given the bigger lift.

So I'm out of ideas about how to make this feature work on our environment. Is there anything I can leverage on the Antrea side to route the traffic to a single IP maybe? Or other ideas come to your mind to facilitate routing from the external network towards the egress IPs?

Thanks!

[tnqn]

@Edward033 I'm not sure "a static route with an interface as the next-hop" looks like. By configuring external network to route traffic towards the Egress IP's subnet to the Node's network, I meant a single route to route the whole subnet of egress IPs. I think either of the following approachs should work if you haven't tried and if it's acceptable in your network:

1. Assign an secondary IP to the router interface which connects to the K8s node subnet. For example, if 10.0.0.0/24 is your node network subnet, 10.0.0.1 is the router interface's primary IP, and you want to use 10.0.1.0/24 as the egress IP subnet, then you could assign 10.0.1.1/24 as the router interface's secondary IP.
2. If configuring secondary IP is not possible, you could also configure a single route on the router, I don't know how to configure your router but it would be ip route add 10.0.1.0/24 dev <INTERFACE> on linux, which will make it treat 10.0.1.0/24 as a local subnet even the router interface doesn't actually have any IP in the subnet.

[Edward033]

2. If configuring secondary IP is not possible, you could also configure a single route on the router, I don't know how to configure your router but it would be ip route add 10.0.1.0/24 dev <INTERFACE> on linux, which will make it treat 10.0.1.0/24 as a local subnet even the router interface doesn't actually have any IP in the subnet.

Hi @tnqn ,

A static route with an interface as the next-hop is the same as the option 2 that you stated. Your option 1 (secondary ip address on the VLAN) is very interesting, and don't see why it wouldn't work.

I'll be reaching out to our Network team, so they can consider it as an alternative. Thanks!