Skip to content

Commit

Permalink
Merge pull request #4040 from martinkennelly/various-eip-multi-fixes
Browse files Browse the repository at this point in the history 
EIP multi NIC IPv6 support, copy routes, fixes and E2E tests
  • Loading branch information
@trozet
trozet committed Feb 2, 2024
2 parents 7f99281 + 3abe49d commit cbff639
Show file tree
Hide file tree
Showing 53 changed files with 4,207 additions and 2,221 deletions.
12 changes: 6 additions & 6 deletions docs/egress-ip.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ It specifies to use `172.18.0.33` or `172.18.0.44` egressIP for pods that are la
Both selectors use the [generic kubernetes label selectors](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors).

## Traffic flows
If the Egress IP(s) are hosted on the OVN managed primary network then the implementation is redirecting the POD traffic
If the Egress IP(s) are hosted on the OVN primary network then the implementation is redirecting the POD traffic
to an egress node where it is SNATed and sent out.

Using the example EgressIP and a matching pod with `10.244.1.3` IP, the following logical router policies are configured in `ovn_cluster_router`:
Expand Down Expand Up @@ -71,8 +71,8 @@ TYPE GATEWAY_PORT EXTERNAL_IP EXTERNAL_PORT LOGIC
snat 172.18.0.44 10.244.1.3
```

Lets now imagine the Egress IP(s) mentioned previously, are not hosted by the OVN managed primary network and is hosted
in a non-OVN managed network interface, a redirect to the egress-able node management port IP address:
Lets now imagine the Egress IP(s) mentioned previously, are not hosted by the OVN primary network and is hosted
by a secondary host network which is assigned to a standard linux interface, a redirect to the egress-able node management port IP address:
```shell
Routing Policies
1004 inport == "rtos-ovn-control-plane" && ip4.dst == 172.18.0.4 /* ovn-control-plane */ reroute 10.244.0.2
Expand Down Expand Up @@ -131,7 +131,7 @@ sh-5.2# ip route show table 1008
default dev dummy
```

No NAT is required on the OVN managed primary network gateway router.
No NAT is required on the OVN primary network gateway router.
OVN-Kubernetes (ovnkube-node) takes care of adding a rule to the rule table with src IP of the pod and routed towards a
new routing table specifically created to route the traffic out the correct interface. IPTables is also altered and an entry
is created within the chain `OVN-KUBE-EGRESS-IP-Multi-NIC` for each pod to allow SNAT to occur when a src IP is match
Expand Down Expand Up @@ -173,8 +173,8 @@ priority=104,ip,in_port=2,nw_src=<clusterSubnet> actions=drop
priority=100,ip,in_port=2 actions=ct(commit,zone=64000,exec(set_field:0x1->ct_mark)),output:1
```

## Special considerations for non-OVN managed Egress IPs
If you wish to assign an Egress IP to a non-OVN managed network, then the following is required:
## Special considerations for Egress IPs hosted by standard linux interfaces
If you wish to assign an Egress IP to a standard linux interface (non OVS type), then the following is required:
* Link is up
* IP address must have scope universe / global
* Links and their addresses must not be removed during runtime after an egress IP is assigned to it. If you wish to remove the link, first
Expand Down
2 changes: 1 addition & 1 deletion go-controller/go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ require (
github.com/spf13/afero v1.9.5
github.com/stretchr/testify v1.8.2
github.com/urfave/cli/v2 v2.2.0
github.com/vishvananda/netlink v1.2.1-beta.2.0.20230420174744-55c8b9515a01
github.com/vishvananda/netlink v1.2.1-beta.2.0.20231024175852-77df5d35f725
golang.org/x/exp v0.0.0-20230811145659-89c5cff77bcb
golang.org/x/net v0.17.0
golang.org/x/sync v0.2.0
Expand Down
6 changes: 3 additions & 3 deletions go-controller/go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -748,8 +748,8 @@ github.com/urfave/cli/v2 v2.2.0/go.mod h1:SE9GqnLQmjVa0iPEY0f1w3ygNIYcIJ0OKPMoW2
github.com/vishvananda/netlink v0.0.0-20181108222139-023a6dafdcdf/go.mod h1:+SR5DhBJrl6ZM7CoCKvpw5BKroDKQ+PJqOg65H/2ktk=
github.com/vishvananda/netlink v1.1.0/go.mod h1:cTgwzPIzzgDAYoQrMm0EdrjRUBkTqKYppBueQtXaqoE=
github.com/vishvananda/netlink v1.1.1-0.20201029203352-d40f9887b852/go.mod h1:twkDnbuQxJYemMlGd4JFIcuhgX83tXhKS2B/PRMpOho=
github.com/vishvananda/netlink v1.2.1-beta.2.0.20230420174744-55c8b9515a01 h1:F9xjJm4IH8VjcqG4ujciOF+GIM4mjPkHhWLLzOghPtM=
github.com/vishvananda/netlink v1.2.1-beta.2.0.20230420174744-55c8b9515a01/go.mod h1:cAAsePK2e15YDAMJNyOpGYEWNe4sIghTY7gpz4cX/Ik=
github.com/vishvananda/netlink v1.2.1-beta.2.0.20231024175852-77df5d35f725 h1:jXQwcl0Ga5tfyej8AdhADlg1R4OcbpSVRWrpn4ihl8M=
github.com/vishvananda/netlink v1.2.1-beta.2.0.20231024175852-77df5d35f725/go.mod h1:whJevzBpTrid75eZy99s3DqCmy05NfibNaF2Ol5Ox5A=
github.com/vishvananda/netns v0.0.0-20180720170159-13995c7128cc/go.mod h1:ZjcWmFBXmLKZu9Nxj3WKYEafiSqer2rnvPr0en9UNpI=
github.com/vishvananda/netns v0.0.0-20191106174202-0a2b9b5464df/go.mod h1:JP3t17pCcGlemwknint6hfoeCVQrEMVwxRLRjXpq+BU=
github.com/vishvananda/netns v0.0.0-20200728191858-db3c7e526aae/go.mod h1:DD4vA1DwXk04H54A1oHXtwZmA0grkVMdPxx/VGLCah0=
Expand Down Expand Up @@ -1019,9 +1019,9 @@ golang.org/x/sys v0.0.0-20211019181941-9d821ace8654/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220209214540-3681064d5158/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220804214406-8e32c043e418/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.13.0 h1:Af8nKPmuFypiUBjVoU9V20FiaFXOcuZI21p0ycVYYGE=
golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
Expand Down
2 changes: 1 addition & 1 deletion go-controller/hack/test-go.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ function testrun {
}

# These packages requires root for network namespace manipulation in unit tests
root_pkgs=("github.com/ovn-org/ovn-kubernetes/go-controller/pkg/node" "github.com/ovn-org/ovn-kubernetes/go-controller/pkg/node/iptables" "github.com/ovn-org/ovn-kubernetes/go-controller/pkg/node/rulemanager" "github.com/ovn-org/ovn-kubernetes/go-controller/pkg/node/routemanager")
root_pkgs=("github.com/ovn-org/ovn-kubernetes/go-controller/pkg/node" "github.com/ovn-org/ovn-kubernetes/go-controller/pkg/node/iptables" "github.com/ovn-org/ovn-kubernetes/go-controller/pkg/node/rulemanager" "github.com/ovn-org/ovn-kubernetes/go-controller/pkg/node/routemanager" "github.com/ovn-org/ovn-kubernetes/go-controller/pkg/node/controllers/egressip")

# These packages are big and require more than the 10m default to run the unit tests
big_pkgs=("github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn")
Expand Down
38 changes: 19 additions & 19 deletions go-controller/pkg/clustermanager/egressip_controller.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -690,10 +690,10 @@ func (eIPC *egressIPClusterController) setNodeEgressReachable(nodeName string, i
}
}

// reconcileNonOVNNetworkEIPs is used to reconsider existing assigned EIPs that are assigned to non-OVN managed
// reconcileSecondaryHostNetworkEIPs is used to reconsider existing assigned EIPs that are assigned to secondary host
// networks and will send a 'synthetic' reconcile for any EIPs which are hosted by an invalid network which is determined
// from the nodes host-cidrs annotation
func (eIPC *egressIPClusterController) reconcileNonOVNNetworkEIPs(node *v1.Node) error {
func (eIPC *egressIPClusterController) reconcileSecondaryHostNetworkEIPs(node *v1.Node) error {
var errorAggregate []error
egressIPs, err := eIPC.kube.GetEgressIPs()
if err != nil {
Expand All @@ -714,14 +714,14 @@ func (eIPC *egressIPClusterController) reconcileNonOVNNetworkEIPs(node *v1.Node)
reconcileEgressIPs = append(reconcileEgressIPs, egressIP.DeepCopy())
continue
}
// do not reconcile if EIP is hosted by OVN managed network or if network is what we expect
if util.IsOVNManagedNetwork(eNode.egressIPConfig, egressIPIP) {
// do not reconcile if EIP is hosted by OVN network or if network is what we expect
if util.IsOVNNetwork(eNode.egressIPConfig, egressIPIP) {
continue
}
network, err := util.GetNonOVNNetworkContainingIP(node, egressIPIP)
network, err := util.GetSecondaryHostNetworkContainingIP(node, egressIPIP)
if err != nil {
errorAggregate = append(errorAggregate, fmt.Errorf("failed to determine if egress IP %s IP %s "+
"is hosted by non-OVN managed network for node %s: %w", egressIP.Name, egressIPIP.String(), node.Name, err))
"is hosted by secondary host network for node %s: %w", egressIP.Name, egressIPIP.String(), node.Name, err))
continue
}
if network == "" {
Expand All @@ -734,7 +734,7 @@ func (eIPC *egressIPClusterController) reconcileNonOVNNetworkEIPs(node *v1.Node)
for _, egressIP := range reconcileEgressIPs {
if err := eIPC.reconcileEgressIP(nil, egressIP); err != nil {
errorAggregate = append(errorAggregate, fmt.Errorf("re-assignment for EgressIP %s hosted by a "+
"non-OVN managed network failed, unable to update object, err: %v", egressIP.Name, err))
"secondary host network failed, unable to update object, err: %v", egressIP.Name, err))
}
}
if len(errorAggregate) > 0 {
Expand Down Expand Up @@ -1169,7 +1169,7 @@ func (eIPC *egressIPClusterController) getCloudPrivateIPConfigMap(objs []interfa
// ascending order following their existing amount of allocations, and trying to
// assign the egress IP to the node with the lowest amount of allocations every
// time, this does not guarantee complete balance, but mostly complete.
// For Egress IPs that are hosted by non-OVN managed networks, there must be at least
// For Egress IPs that are hosted by secondary host networks, there must be at least
// one node that hosts the network and exposed via the nodes host-cidrs annotation.
func (eIPC *egressIPClusterController) assignEgressIPs(name string, egressIPs []string) []egressipv1.EgressIPStatusItem {
eIPC.allocator.Lock()
Expand Down Expand Up @@ -1258,7 +1258,7 @@ func (eIPC *egressIPClusterController) assignEgressIPs(name string, egressIPs []
return assignments
}
}
// Egress IP for non-OVN managed networks is only available on baremetal environments
// Egress IP for secondary host networks is only available on baremetal environments
if !util.PlatformTypeIsEgressIPCloudProvider() {
assignableNodesWithSecondaryNet := make([]*egressNode, 0)
for _, eNode := range assignableNodes {
Expand All @@ -1268,12 +1268,12 @@ func (eIPC *egressIPClusterController) assignEgressIPs(name string, egressIPs []
name, eNode.name, eIP.String(), err)
continue
}
if util.IsOVNManagedNetwork(eNode.egressIPConfig, eIP) {
if util.IsOVNNetwork(eNode.egressIPConfig, eIP) {
continue
}
network, err := util.GetNonOVNNetworkContainingIP(node, eIP)
network, err := util.GetSecondaryHostNetworkContainingIP(node, eIP)
if err != nil {
klog.Warningf("Failed to determine if egress IP is hosted by a non-OVN managed networks for node %s: %v",
klog.Warningf("Failed to determine if egress IP is hosted by a secondary host network for node %s: %v",
eIP.String(), node.Name, err)
continue
}
Expand All @@ -1283,11 +1283,11 @@ func (eIPC *egressIPClusterController) assignEgressIPs(name string, egressIPs []
assignableNodesWithSecondaryNet = append(assignableNodesWithSecondaryNet, eNode)

}
// if the EIP is hosted by a non OVN managed network, then restrict the assignable nodes to the set of nodes that
// may host the non-OVN managed network
// if the EIP is hosted by a secondary host network, then limit the assignable nodes to the set of nodes
// that connect to this network.
if len(assignableNodesWithSecondaryNet) > 0 {
klog.V(5).Infof("Restricting the number of assignable nodes from %d to %d because EgressIP %s IP %s "+
"is going to be hosted by a non-OVN managed network", len(assignableNodes), len(assignableNodesWithSecondaryNet), name, eIP.String())
"is going to be hosted by a secondary host network", len(assignableNodes), len(assignableNodesWithSecondaryNet), name, eIP.String())
assignableNodes = assignableNodesWithSecondaryNet
}
}
Expand Down Expand Up @@ -1455,13 +1455,13 @@ func (eIPC *egressIPClusterController) validateEgressIPStatus(name string, items
klog.Errorf("Allocator error: failed to validate and will not consider node %s for egress IP %s: %v",
eNode.name, name, err)
}
isOVNManaged := util.IsOVNManagedNetwork(eNode.egressIPConfig, ip)
isSecondaryNetwork, err := util.IsNonOVNManagedNetworkContainingIP(node, ip)
isOVNNetwork := util.IsOVNNetwork(eNode.egressIPConfig, ip)
isSecondaryHostNetwork, err := util.IsSecondaryHostNetworkContainingIP(node, ip)
if err != nil {
klog.Errorf("Allocator error: failed to determine if Egress IP %q is to be hosted by a non-OVN managed "+
klog.Errorf("Allocator error: failed to determine if Egress IP %q is to be hosted by a secondary host "+
"network for egress IP %s: %v", eIPStatus.EgressIP, name, err)
}
if !isOVNManaged && !isSecondaryNetwork {
if !isOVNNetwork && !isSecondaryHostNetwork {
klog.Errorf("Allocator error: failed to assign Egress IP %s IP %q", name, eIPStatus.EgressIP)
validAssignment = false
}
Expand Down

0 comments on commit cbff639

Please sign in to comment.