How Can I Control Source IP Addresses With BGP And Cilium

[TheDarkula]

Stemming from this conversation, I have an email server (stalwart) running behind traefik.
I need to be able to control the source address (SNAT) so that I can route traffic correctly.
Currently, everything is coming from the kubernetes node address (192.168.1.122).
With Cilium, what is the best way to be able to specify the origin IP address?

[joestringer]

This might help: https://docs.cilium.io/en/stable/network/egress-gateway/egress-gateway/

[TheDarkula]

@joestringer I am tracking down what the issue is.

As far as the curl syntax, from the manual, --interface <name>:

Perform the operation using a specified interface. You can enter interface name, IP address or hostname.

It seems to be a mismatch/conflict with the route announced by BGP, and I think it is causing asymmetric routing.

I configured my node/host networking as follows:

sudo nmcli connection add type ethernet ifname enp41s0 con-name primary ipv4.method auto
sudo nmcli connection modify primary +ipv4.addresses 192.168.20.3/24
sudo nmcli connection up primary

In my mikrotik router, if I do this:

/ip/address add address=192.168.20.1/24 interface=bridge1

And I also disable the BGP announcement/kubernetes service running on 192.168.20.3, this succeeds: curl --interface 192.168.20.3 http://ifconfig.me.

This allows normal egress:

[admin@MikroTik] /ip/route> print
Flags: D - DYNAMIC; I - INACTIVE, A - ACTIVE; c - CONNECT, s - STATIC, b - BGP, d - DHCP
Columns: DST-ADDRESS, GATEWAY, ROUTING-TABLE, DISTANCE
#     DST-ADDRESS       GATEWAY         ROUTING-TABLE   DISTANCE
  DAc 192.168.20.0/24   bridge1         main                   0

This does not:

[admin@MikroTik] /ip/route> print
Flags: D - DYNAMIC; I - INACTIVE, A - ACTIVE; c - CONNECT, s - STATIC, b - BGP, d - DHCP
Columns: DST-ADDRESS, GATEWAY, ROUTING-TABLE, DISTANCE
#     DST-ADDRESS       GATEWAY         ROUTING-TABLE   DISTANCE
  DAb 192.168.20.3/32   192.168.1.122   main                  20

The difference seems to be in which gateway is used. When I manually add the IP block/route to my bridge1, it works properly.
With BGP, it uses 192.168.1.122 as the gateway, which breaks the return route.

[TheDarkula]

@joestringer Is there something you see that I am missing?
I am trying to sort out the "correct" way to do this, but I am not certain what that is.
On paper, if there is a way to modify the BGP route to specify the bridge1 gateway, that reads like it would remedy the problem.
What is the prescribed way to do this with cilium?

[joestringer]

Ah. The egress gateway policy only affects how traffic source address is selected. The feature only controls the policy for how traffic egresses from the Kubernetes cluster, it doesn't affect the advertising of routes or incoming traffic. I'm not sure if the BGP support right now has a way to choose multiple gateways or dynamically select the gateway for the routes.

[TheDarkula]

@joestringer I am not sure if this is the "clean" way to do this, but I have everything working!

I created an IP address block in my mikrotik router that is separate from any IP pools controlled via cilium/BGP:

3   192.168.22.1/24     192.168.22.0   bridge1    main

This automatically creates a route as well:

 DAc 192.168.22.0/24   bridge1         main                   0

Now, mikrotik owns a block of IP addresses.

With NetworkManager, this works for a 5 IP block:

sudo nmcli connection add type ethernet ifname enp41s0 con-name primary ipv4.method manual ipv4.addresses "192.168.1.122/24"
sudo nmcli connection modify primary ipv4.gateway "192.168.1.1"
sudo nmcli connection modify primary +ipv4.addresses "192.168.22.2/24"
sudo nmcli connection modify primary +ipv4.routes "192.168.22.0/24 192.168.22.1"
sudo nmcli connection modify primary +ipv4.addresses "192.168.32.2/24"
sudo nmcli connection modify primary +ipv4.routes "192.168.32.0/24 192.168.32.1"
sudo nmcli connection modify primary +ipv4.addresses "192.168.42.2/24"
sudo nmcli connection modify primary +ipv4.routes "192.168.42.0/24 192.168.42.1"
sudo nmcli connection modify primary +ipv4.addresses "192.168.52.2/24"
sudo nmcli connection modify primary +ipv4.routes "192.168.52.0/24 192.168.52.1"
sudo nmcli connection modify primary +ipv4.addresses "192.168.62.2/24"
sudo nmcli connection modify primary +ipv4.routes "192.168.62.0/24 192.168.62.1"
sudo nmcli connection up primary

I have my stalwart email server behind traefik with IngressRouteTCPs.

In order to have SNAT match up, I needed to apply the same CiliumEgressGatewayPolicy to both stalwart and my traefik instance, otherwise outbound email would fail.