Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions _topic_maps/_topic_map.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1627,8 +1627,6 @@ Topics:
File: configuring-secondary-external-gateway
- Name: Configuring an egress IP address
File: configuring-egress-ips-ovn
- Name: Assigning an egress IP address
File: assigning-egress-ips-ovn
- Name: Configuring an egress service
File: configuring-egress-traffic-for-vrf-loadbalancer-services
- Name: Considerations for the use of an egress router pod
Expand Down
102 changes: 5 additions & 97 deletions modules/nw-egress-ips-about.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,13 @@
//
// * networking/ovn_kubernetes_network_provider/configuring-egress-ips-ovn.adoc

ifeval::["{context}" == "configuring-egress-ips-ovn"]
:ovn:
endif::[]

:_mod-docs-content-type: REFERENCE
[id="nw-egress-ips-about_{context}"]
= Egress IP address architectural design and implementation

The {product-title} egress IP address functionality allows you to ensure that the traffic from one or more pods in one or more namespaces has a consistent source IP address for services outside the cluster network.

For example, you might have a pod that periodically queries a database that is hosted on a server outside of your cluster. To enforce access requirements for the server, a packet filtering device is configured to allow traffic only from specific IP addresses.
To ensure that you can reliably allow access to the server from only that specific pod, you can configure a specific egress IP address for the pod that makes the requests to the server.
By using the {product-title} egress IP address functionality, you can ensure that the traffic from one or more pods in one or more namespaces has a consistent source IP address for services outside the cluster network.

For example, you might have a pod that periodically queries a database that is hosted on a server outside of your cluster. To enforce access requirements for the server, a packet filtering device is configured to allow traffic only from specific IP addresses. To ensure that you can reliably allow access to the server from only that specific pod, you can configure a specific egress IP address for the pod that makes the requests to the server.

An egress IP address assigned to a namespace is different from an egress router, which is used to send traffic to specific destinations.

Expand Down Expand Up @@ -160,80 +155,8 @@ On Azure, the following capacity limits exist for IP address assignment:

For more information, see link:https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits?toc=/azure/virtual-network/toc.json#networking-limits[Networking limits].

ifdef::ovn[]
[id="nw-egress-ips-multi-nic-considerations_{context}"]
== Considerations for using an egress IP on additional network interfaces

In {product-title}, egress IPs provide administrators a way to control network traffic. Egress IPs can be used with a `br-ex` Open vSwitch (OVS) bridge interface and any physical interface that has IP connectivity enabled.

You can inspect your network interface type by running the following command:

[source,terminal]
----
$ ip -details link show
----

The primary network interface is assigned a node IP address which also contains a subnet mask. Information for this node IP address can be retrieved from the Kubernetes node object for each node within your cluster by inspecting the `k8s.ovn.org/node-primary-ifaddr` annotation. In an IPv4 cluster, this annotation is similar to the following example: `"k8s.ovn.org/node-primary-ifaddr: {"ipv4":"192.168.111.23/24"}"`.

If the egress IP is not within the subnet of the primary network interface subnet, you can use an egress IP on another Linux network interface that is not of the primary network interface type. By doing so, {product-title} administrators are provided with a greater level of control over networking aspects such as routing, addressing, segmentation, and security policies. This feature provides users with the option to route workload traffic over specific network interfaces for purposes such as traffic segmentation or meeting specialized requirements.

If the egress IP is not within the subnet of the primary network interface, then the selection of another network interface for egress traffic might occur if they are present on a node.

You can determine which other network interfaces might support egress IPs by inspecting the `k8s.ovn.org/host-cidrs` Kubernetes node annotation. This annotation contains the addresses and subnet mask found for the primary network interface. It also contains additional network interface addresses and subnet mask information. These addresses and subnet masks are assigned to network interfaces that use the link:https://networklessons.com/cisco/ccna-200-301/longest-prefix-match-routing[longest prefix match routing] mechanism to determine which network interface supports the egress IP.

[NOTE]
====
OVN-Kubernetes provides a mechanism to control and direct outbound network traffic from specific namespaces and pods. This ensures that it exits the cluster through a particular network interface and with a specific egress IP address.
====


[id="nw-egress-ips-multi-nic-requirements_{context}"]
=== Requirements for assigning an egress IP to a network interface that is not the primary network interface

For users who want an egress IP and traffic to be routed over a particular interface that is not the primary network interface, the following conditions must be met:

* {product-title} is installed on a bare-metal cluster. This feature is disabled within a cloud or a hypervisor environment.

* Your {product-title} pods are not configured as _host-networked_.

* If a network interface is removed or if the IP address and subnet mask which allows the egress IP to be hosted on the interface is removed, the egress IP is reconfigured. Consequently, the egress IP could be assigned to another node and interface.

* If you use an Egress IP address on a secondary network interface card (NIC), you must use the Node Tuning Operator to enable IP forwarding on the secondary NIC.
endif::ovn[]
endif::openshift-rosa[]

ifdef::ovn[]
[id="nw-egress-ips-considerations_{context}"]
== Assignment of egress IPs to pods

To assign one or more egress IPs to a namespace or specific pods in a namespace, the following conditions must be satisfied:

- At least one node in your cluster must have the `k8s.ovn.org/egress-assignable: ""` label.
- An `EgressIP` object exists that defines one or more egress IP addresses to use as the source IP address for traffic leaving the cluster from pods in a namespace.

[IMPORTANT]
====
If you create `EgressIP` objects prior to labeling any nodes in your cluster for egress IP assignment, {product-title} might assign every egress IP address to the first node with the `k8s.ovn.org/egress-assignable: ""` label.

To ensure that egress IP addresses are widely distributed across nodes in the cluster, always apply the label to the nodes you intent to host the egress IP addresses before creating any `EgressIP` objects.
====

[id="nw-egress-ips-node-assignment_{context}"]
== Assignment of egress IPs to nodes

When creating an `EgressIP` object, the following conditions apply to nodes that are labeled with the `k8s.ovn.org/egress-assignable: ""` label:

- An egress IP address is never assigned to more than one node at a time.
- An egress IP address is equally balanced between available nodes that can host the egress IP address.
- If the `spec.EgressIPs` array in an `EgressIP` object specifies more than one IP address, the following conditions apply:
* No node will ever host more than one of the specified IP addresses.
* Traffic is balanced roughly equally between the specified IP addresses for a given namespace.
- If a node becomes unavailable, any egress IP addresses assigned to it are automatically reassigned, subject to the previously described conditions.

When a pod matches the selector for multiple `EgressIP` objects, there is no guarantee which of the egress IP addresses that are specified in the `EgressIP` objects is assigned as the egress IP address for the pod.

Additionally, if an `EgressIP` object specifies multiple egress IP addresses, there is no guarantee which of the egress IP addresses might be used. For example, if a pod matches a selector for an `EgressIP` object with two egress IP addresses, `10.10.20.1` and `10.10.20.2`, either might be used for each TCP connection or UDP conversation.

[id="nw-egress-ips-node-architecture_{context}"]
== Architectural diagram of an egress IP address configuration

Expand All @@ -246,12 +169,7 @@ Both Node 1 and Node 3 are labeled with `k8s.ovn.org/egress-assignable: ""` and

The dashed lines in the diagram depict the traffic flow from pod1, pod2, and pod3 traveling through the pod network to egress the cluster from Node 1 and Node 3. When an external service receives traffic from any of the pods selected by the example `EgressIP` object, the source IP address is either `192.168.126.10` or `192.168.126.102`. The traffic is balanced roughly equally between these two nodes.

The following resources from the diagram are illustrated in detail:

`Namespace` objects::
+
--
The namespaces are defined in the following manifest:
Based on the diagram, the following manifest file defines namespaces:

.Namespace objects
[source,yaml]
Expand All @@ -270,12 +188,8 @@ metadata:
labels:
env: prod
----
--

`EgressIP` object::
+
--
The following `EgressIP` object describes a configuration that selects all pods in any namespace with the `env` label set to `prod`. The egress IP addresses for the selected pods are `192.168.126.10` and `192.168.126.102`.
Based on the diagram, the following `EgressIP` object describes a configuration that selects all pods in any namespace with the `env` label set to `prod`. The egress IP addresses for the selected pods are `192.168.126.10` and `192.168.126.102`.

.`EgressIP` object
[source,yaml]
Expand All @@ -300,9 +214,3 @@ status:
----

For the configuration in the previous example, {product-title} assigns both egress IP addresses to the available nodes. The `status` field reflects whether and where the egress IP addresses are assigned.
--
endif::ovn[]

ifdef::ovn[]
:!ovn:
endif::ovn[]
24 changes: 18 additions & 6 deletions modules/nw-egress-ips-assign.adoc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Module included in the following assemblies:
//
// * networking/ovn_kubernetes_network_provider/assigning-egress-ips-ovn.adoc
// * networking/ovn_kubernetes_network_provider/configuring-egress-ips-ovn.adoc

:_mod-docs-content-type: PROCEDURE
[id="nw-egress-ips-assign_{context}"]
Expand All @@ -16,8 +16,10 @@ You can assign one or more egress IP addresses to a namespace or to specific pod

.Procedure

. Create an `EgressIP` object:
. Create an `EgressIP` object.
+
.. Create a `<egressips_name>.yaml` file where `<egressips_name>` is the name of the object.
+
.. In the file that you created, define an `EgressIP` object, as in the following example:
+
[source,yaml]
Expand All @@ -39,9 +41,14 @@ spec:
+
[source,terminal]
----
$ oc apply -f <egressips_name>.yaml <1>
$ oc apply -f <egressips_name>.yaml
----
<1> Replace `<egressips_name>` with the name of the object.
+
--
where:

`<egressips_name>`:: Replace `<egressips_name>` with the name of the object.
--
+
.Example output
[source,terminal]
Expand All @@ -55,9 +62,14 @@ egressips.k8s.ovn.org/<egressips_name> created
+
[source,terminal]
----
$ oc label ns <namespace> env=qa <1>
$ oc label ns <namespace> env=qa
----
<1> Replace `<namespace>` with the namespace that requires egress IP addresses.
+
--
where:

`<namespace>`:: Replace `<namespace>` with the namespace that requires egress IP addresses.
--

.Verification

Expand Down
13 changes: 10 additions & 3 deletions modules/nw-egress-ips-config-object.adoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
// Module included in the following assemblies:
//
// * networking/ovn_kubernetes_network_provider/assigning-egress-ips-ovn.adoc
// * networking/ovn_kubernetes_network_provider/configuring-egress-ips-ovn.adoc

:_mod-docs-content-type: CONCEPT
[id="nw-egress-ips-config-object_{context}"]
= The egressIPConfig object

Expand Down Expand Up @@ -31,5 +32,11 @@ spec:
routingViaHost: false
genevePort: 6081
----
<1> The `egressIPConfig` holds the configurations for the options of the `EgressIP` object. By changing these configurations, you can extend the `EgressIP` object.
<2> The value for `reachabilityTotalTimeoutSeconds` accepts integer values from `0` to `60`. A value of `0` disables the reachability check of the egressIP node. Setting a value from `1` to `60` corresponds to the timeout in seconds for a probe to send the reachability check to the node.

--
where:

`<egressIPConfig>`:: The `egressIPConfig` holds the configurations for the options of the `EgressIP` object. By changing these configurations, you can extend the `EgressIP` object.

`<reachabilityTotalTimeoutSeconds>`:: The value for `reachabilityTotalTimeoutSeconds` accepts integer values from `0` to `60`. A value of `0` disables the reachability check of the egressIP node. Setting a value from `1` to `60` corresponds to the timeout in seconds for a probe to send the reachability check to the node.
--
32 changes: 32 additions & 0 deletions modules/nw-egress-ips-considerations.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
// Module included in the following assemblies:
//
// * networking/ovn_kubernetes_network_provider/configuring-egress-ips-ovn.adoc

:_mod-docs-content-type: REFERENCE
[id="nw-egress-ips-considerations_{context}"]
= Assignment of egress IPs to a namespace, nodes, and pods

To assign one or more egress IPs to a namespace or specific pods in a namespace, the following conditions must be satisfied:

- At least one node in your cluster must have the `k8s.ovn.org/egress-assignable: ""` label.
- An `EgressIP` object exists that defines one or more egress IP addresses to use as the source IP address for traffic leaving the cluster from pods in a namespace.

[IMPORTANT]
====
If you create `EgressIP` objects prior to labeling any nodes in your cluster for egress IP assignment, {product-title} might assign every egress IP address to the first node with the `k8s.ovn.org/egress-assignable: ""` label.

To ensure that egress IP addresses are widely distributed across nodes in the cluster, always apply the label to the nodes you intent to host the egress IP addresses before creating any `EgressIP` objects.
====

When creating an `EgressIP` object, the following conditions apply to nodes that are labeled with the `k8s.ovn.org/egress-assignable: ""` label:

- An egress IP address is never assigned to more than one node at a time.
- An egress IP address is equally balanced between available nodes that can host the egress IP address.
- If the `spec.EgressIPs` array in an `EgressIP` object specifies more than one IP address, the following conditions apply:
* No node will ever host more than one of the specified IP addresses.
* Traffic is balanced roughly equally between the specified IP addresses for a given namespace.
- If a node becomes unavailable, any egress IP addresses assigned to it are automatically reassigned, subject to the previously described conditions.

When a pod matches the selector for multiple `EgressIP` objects, there is no guarantee which of the egress IP addresses that are specified in the `EgressIP` objects is assigned as the egress IP address for the pod.

Additionally, if an `EgressIP` object specifies multiple egress IP addresses, there is no guarantee which of the egress IP addresses might be used. For example, if a pod matches a selector for an `EgressIP` object with two egress IP addresses, `10.10.20.1` and `10.10.20.2`, either might be used for each TCP connection or UDP conversation.
39 changes: 39 additions & 0 deletions modules/nw-egress-ips-multi-nic-considerations.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
// Module included in the following assemblies:
//
// * networking/ovn_kubernetes_network_provider/configuring-egress-ips-ovn.adoc

:_mod-docs-content-type: CONCEPT
[id="nw-egress-ips-multi-nic-considerations_{context}"]
= Considerations for using an egress IP on additional network interfaces

In {product-title}, egress IPs provide administrators a way to control network traffic. Egress IPs can be used with a `br-ex` Open vSwitch (OVS) bridge interface and any physical interface that has IP connectivity enabled.

You can inspect your network interface type by running the following command:

[source,terminal]
----
$ ip -details link show
----

The primary network interface is assigned a node IP address which also contains a subnet mask. Information for this node IP address can be retrieved from the Kubernetes node object for each node within your cluster by inspecting the `k8s.ovn.org/node-primary-ifaddr` annotation. In an IPv4 cluster, this annotation is similar to the following example: `"k8s.ovn.org/node-primary-ifaddr: {"ipv4":"192.168.111.23/24"}"`.

If the egress IP is not within the subnet of the primary network interface subnet, you can use an egress IP on another Linux network interface that is not of the primary network interface type. By doing so, {product-title} administrators are provided with a greater level of control over networking aspects such as routing, addressing, segmentation, and security policies. This feature provides users with the option to route workload traffic over specific network interfaces for purposes such as traffic segmentation or meeting specialized requirements.

If the egress IP is not within the subnet of the primary network interface, then the selection of another network interface for egress traffic might occur if they are present on a node.

You can determine which other network interfaces might support egress IPs by inspecting the `k8s.ovn.org/host-cidrs` Kubernetes node annotation. This annotation contains the addresses and subnet mask found for the primary network interface. It also contains additional network interface addresses and subnet mask information. These addresses and subnet masks are assigned to network interfaces that use the link:https://networklessons.com/cisco/ccna-200-301/longest-prefix-match-routing[longest prefix match routing] mechanism to determine which network interface supports the egress IP.

[NOTE]
====
OVN-Kubernetes provides a mechanism to control and direct outbound network traffic from specific namespaces and pods. This ensures that it exits the cluster through a particular network interface and with a specific egress IP address.
====

For users who want an egress IP and traffic to be routed over a particular interface that is not the primary network interface, the following conditions must be met:

* {product-title} is installed on a bare-metal cluster. This feature is disabled within a cloud or a hypervisor environment.

* Your {product-title} pods are not configured as _host-networked_.

* If a network interface is removed or if the IP address and subnet mask which allows the egress IP to be hosted on the interface is removed, the egress IP is reconfigured. Consequently, the egress IP could be assigned to another node and interface.

* If you use an Egress IP address on a secondary network interface card (NIC), you must use the Node Tuning Operator to enable IP forwarding on the secondary NIC.
Loading