-
Notifications
You must be signed in to change notification settings - Fork 349
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Signed-off-by: Surya Seetharaman <suryaseetharaman.9@gmail.com>
- Loading branch information
Showing
66 changed files
with
1,066 additions
and
270 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
# Blog | ||
|
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,210 @@ | ||
# OVN Kubernetes Architecture | ||
|
||
There are two deployment modes for ovn kubernetes depending on | ||
which the architecture is drastically different: | ||
|
||
* default mode (centralized control plane architecture) | ||
* interconnect mode (distributed control plane architecture) | ||
|
||
End users can pick either of these modes depending on their use | ||
cases and what suits them well. Let's look at both these modes | ||
in depth so that you are empowered to make your choice between | ||
these two modes of deployment. | ||
|
||
## OVN Kubernetes Components - Default Mode | ||
|
||
<p align="center"> | ||
<img src=/../images/ovnkube-centralized-components.png alt="ovn-kubernetes-centralized-components"/> | ||
</p> | ||
|
||
|
||
The control plane has the `ovnkube-master` pod in the `ovn-kubernetes` namespace | ||
which are running only on the control plane nodes in your cluster: | ||
|
||
* ovnkube-master pod | ||
* ovnkube-master container: | ||
* OVN-Kubernetes component | ||
* Watches K8s API for objects - namespaces, pods, services, endpoints, | ||
network policies, CRs | ||
* Translates K8s objects into OVN logical entities | ||
* Stores OVN entities in NorthBound Database (NBDB) | ||
* Manages pod subnet allocation to nodes (pod IPAM) | ||
* nbdb container: | ||
* Native OVN component | ||
* Runs the OVN NBDB database | ||
* Stores the logical elements created by ovnkube-master | ||
* 3 replicas across control plane nodes running using RAFT leadership | ||
algorithm in HA mode | ||
* northd container: | ||
* Native OVN component | ||
* Converts the OVN logical elements from NBDB to OVN logical flows in SBDB | ||
* sbdb container: | ||
* Native OVN component | ||
* Stores the logical flows created by northd | ||
* 3 replicas across control plane nodes running using RAFT leadership algorithm | ||
in HA mode | ||
|
||
The data plane includes the `ovnkube-node` and `ovs-node` pods in the `ovn-kubernetes` | ||
namespace which are running on all your nodes in the cluster. | ||
|
||
* ovnkube-node pod | ||
* ovnkube-node container: | ||
* OVN Kubernetes component | ||
* Runs the CNI executable (CNI ADD/DEL) | ||
* Digests the IPAM annotation set on pod by ovnkube-master | ||
* Creates the veth pair for the pod | ||
* Creates the ovs port on bridge | ||
* Programs the necessary iptables and gateway service flows on a per node basis. | ||
* ovn-controller container: | ||
* Native OVN component | ||
* Connects to SBDB running in control plane using TLS | ||
* Converts SBDB logical flows into openflows | ||
* Write them to OVS | ||
* ovs-node pod | ||
* ovs-daemons container: | ||
* OVS Native component | ||
* OVS daemon and database running as a container | ||
* virtual switch that pushes the network plumbing to the edge on the node | ||
|
||
## Default Mode Architecture | ||
|
||
Now that we know the pods and components running in the default mode, let's tie up | ||
loose ends and show how these components run on a standard HA Kubernetes cluster. | ||
|
||
### Control Plane Nodes: | ||
|
||
<p align="center"> | ||
<img src=/../images/ovnkube-centralized-arch-cp.png alt="ovn-kubernetes-centralized-components-control-plane"/> | ||
</p> | ||
|
||
### Worker Nodes: | ||
|
||
<p align="center"> | ||
<img src=/../images/ovnkube-centralized-arch-dp.png alt="ovn-kubernetes-centralized-components-data-plane"/> | ||
</p> | ||
|
||
## OVN Kubernetes Components - Interconnect mode | ||
|
||
<p align="center"> | ||
<img src=/../images/ovnkube-distributed-components.png alt="ovn-kubernetes-distributed-components"/> | ||
</p> | ||
|
||
The control plane has the `ovnkube-control-plane` pod in the `ovn-kubernetes` namespace | ||
which is super light weight and running only on the control plane nodes in your cluster: | ||
|
||
* ovnkube-control-plane pod | ||
* ovnkube-cluster-manager container: | ||
* OVN-Kubernetes component | ||
* Watches K8s API for objects - nodes mainly | ||
* Allocates pod subnet to each node | ||
* Allocates join subnet IP to each node | ||
* Allocates transit subnet IP to each node | ||
* Consolidates zone statuses across all nodes for features like EgressFirewall | ||
and EgressQoS | ||
|
||
The data plane includes the `ovnkube-node` and `ovs-node` pods in the `ovn-kubernetes` | ||
namespace running on all your nodes in the cluster making this architecture localized | ||
and more distributed. | ||
|
||
* ovnkube-node pod | ||
* ovnkube-controller container: | ||
* OVN-Kubernetes component | ||
* Allocates podIP from the podSubnet to each pod in its zone (IPAM) | ||
* Watches K8s API for objects - nodes, namespaces, pods, services, endpoints, | ||
network policies, CRs | ||
* Translates K8s objects into OVN logical entities - stores them in OVN databases | ||
* Stores OVN entities in NorthBound Database (NBDB) | ||
* Manages pod subnet allocation to nodes (pod IPAM) | ||
* Runs the CNI executable (CNI ADD/DEL) | ||
* Digests the IPAM annotation set on pod | ||
* Creates the veth pair for the pod | ||
* Creates the ovs port on bridge | ||
* Programs the necessary iptables and gateway service flows on a | ||
* nbdb container: | ||
* Native OVN component | ||
* Runs the OVN NBDB database | ||
* Stores the logical elements created by ovnkube-controller | ||
* runs only 1 replica, contains information local to this node | ||
* northd container: | ||
* Native OVN component | ||
* Converts the OVN logical elements from NBDB to OVN logical flows in SBDB | ||
* sbdb container: | ||
* Native OVN component | ||
* Stores the logical flows created by northd | ||
* runs only 1 replica, contains information local to this node | ||
* ovn-controller container: | ||
* Native OVN component | ||
* Connects to SBDB running in control plane using TLS | ||
* Converts SBDB logical flows into openflows | ||
* Write them to OVS | ||
* ovs-node pod | ||
* ovs-daemons container: | ||
* OVS Native component | ||
* OVS daemon and database running as a container | ||
* virtual switch that pushes the network plumbing to the edge on the node | ||
|
||
Thus as we can see, the databases, northd and ovn kubernetes controller components | ||
now run per zone rather than only on the control-plane. | ||
|
||
## Interconnect Mode Architecture | ||
|
||
### What is Interconnect? | ||
|
||
[OVN Interconnection](https://docs.ovn.org/en/latest/tutorials/ovn-interconnection.html) | ||
is a feature that allows connecting multiple OVN deployments with OVN managed GENEVE tunnels. | ||
Native ovn-ic feature allows for an `ovn-ic`, OVN interconnection controller, that is a | ||
centralized daemon which communicates with global interaction databases (IC_NB/IC_SB) to | ||
configure and exchange data with local NB/SB databases for interconnecting with other OVN | ||
deployments. See [this](https://man7.org/linux/man-pages/man8/ovn-ic.8.html) for more details. | ||
|
||
### Adopting OVN-Interconnect into OVN-Kubernetes | ||
|
||
In order to effectively adapt the capabilities of the interconnect feature in the kubernetes | ||
world, ovn-kubernetes components will replace `ovn-ic` daemon. Also note that the term `zone` | ||
which will be used heavily in these docs just refers to a single OVN deployment. Now that we | ||
know the pods and components running in the interconnect mode, let's tie up loose ends and | ||
show how these components run on a standard HA Kubernetes cluster. By default each node in | ||
the cluster is a `zone`, so each `zone` contains 1 node. There is no more RAFT since each | ||
node has its own database. | ||
|
||
### Control Plane Nodes: | ||
|
||
<p align="center"> | ||
<img src=/../images/ovnkube-distributed-arch-cp.png alt="ovn-kubernetes-distributed-components-control-plane"/> | ||
</p> | ||
|
||
### Worker Nodes: | ||
|
||
<p align="center"> | ||
<img src=/../images/ovnkube-distributed-arch-dp.png alt="ovn-kubernetes-distributed-components-data-plane"/> | ||
</p> | ||
|
||
### Why do we need Interconnect mode in OVN-Kubernetes? | ||
|
||
This architecture brings about several improvements: | ||
|
||
* Stability: The OVN Northbound and Southbound databases are local to each node. Since | ||
they are running in the standalone mode, that eliminates the need for RAFT, thus avoiding | ||
all the “split-brain” issues. If one of the databases goes down, the impact is now isolated | ||
to only that node. This has led to improved stability of the OVN Kubernetes stack and | ||
simpler customer escalation resolution. | ||
* Scale: As seen in the above diagram, the ovn-controller container connects to the | ||
local Southbound database for logical flow information. On large clusters with N nodes, | ||
this means each Southbound database is handling only one connection from its own local | ||
ovn-controller. This has removed the scale bottlenecks that were present in the centralized | ||
model helping us to scale horizontally with node count. | ||
* Performance: The OVN Kubernetes brain is now local to each node in the cluster and it is | ||
storing and processing changes to only those Kubernetes pods, services, endpoints objects | ||
that are relevant for that node (note: some features like NetworkPolicies need to process | ||
pods running on other nodes). This in turn means the OVN stack is also processing less data | ||
thus leading to improved operational latency. Another benefit is that the control plane | ||
stack is now lighter-weight. | ||
* Security: Since the infrastructure network traffic between ovn-controller and OVN Southbound | ||
database is now contained within each node, overall cross-node and cross-cluster | ||
(HostedControlPlane, ManagedSaaS) chatter is decreased and traffic security can be increased. | ||
|
||
## Default Mode versus Interconnect Mode | ||
|
||
* When you want your databases to stay centralized and don't mind much about linear scaling of number of nodes in your cluster, choose the default mode | ||
* Note that there is no different to OVS between the two deployment modes. | ||
* FIXME: This section needs to be written well |
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
File renamed without changes.
Oops, something went wrong.