Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Update multi-cluster user guide and templates
Add a quick start guide. Update the user guide with Multi-cluster Gateway. Signed-off-by: Lan Luo <luola@vmware.com> Signed-off-by: Jianjun Shen <shenj@vmware.com>
- Loading branch information
1 parent
87918f6
commit f27e3d1
Showing
6 changed files
with
446 additions
and
178 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,168 @@ | ||
| # Antrea Multi-cluster Quick Start | ||
|
|
||
| In this quick start guide, we will set up a Antrea Multi-cluster ClusterSet with | ||
| two clusters. One cluster will serve as both the leader and a member cluster, | ||
| while another cluster will be a member only. The diagram belows shows the two | ||
| clusters and the ClusterSet to be created. | ||
|
|
||
| ## Preparation | ||
|
|
||
| We assume a Antrea version >= v1.7.0 is used in this guide, and the Antrea | ||
| version is set to an environment variable `TAG`. For example, the following | ||
| command sets the Antrea version to v1.7.0. | ||
|
|
||
| ```bash | ||
| export TAG=v1.7.0 | ||
| ``` | ||
|
|
||
| To use the latest version of Antrea Multi-cluster from the Antrea main branch, | ||
| you can change the YAML manifest path to: https://github.com/antrea-io/antrea/tree/main/multicluster/build/yamls/. | ||
|
|
||
| Antrea must be deployed in both cluster A and cluster B. To configure Antrea | ||
| Multi-cluster Gateways, `antrea-agent` must be deployed with the `Multicluster` | ||
| feature enabled in both cluster A and B. Multi-cluster Gateways are required for | ||
| routing multi-cluster Service traffic through tunnels across the member | ||
| clusters. Set the following options in `antrea-agent.conf` of the Antrea | ||
| deployment manifest to enable the `Multicluster` feature: | ||
|
|
||
| ```yaml | ||
| antrea-agent.conf: | | ||
| ... | ||
| featureGates: | ||
| ... | ||
| Multicluster: true | ||
| ... | ||
| multicluster: | ||
| enable: true | ||
| namespace: "" | ||
| ``` | ||
|
|
||
| ## Set up Leader and Member in Cluster A | ||
|
|
||
| ### Step 1 - deploy Antrea Multi-cluster Controllers for leader and member | ||
|
|
||
| Run the following commands to deploy Multi-cluster Controller for the leader | ||
| into Namespace `antrea-multicluster` (Namespace `antrea-multicluster` will be | ||
| created by the commands), and Multi-cluster Controller for the member into | ||
| Namepsace `kube-system`. | ||
|
|
||
| ```bash | ||
| $kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-leader-global.yml | ||
| $kubectl create ns antrea-multicluster | ||
| $curl -L https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-leader-namespaced.yml > antrea-multicluster-leader-namespaced.yml | ||
| $sed 's/changeme/antrea-multicluster/g' antrea-multicluster-leader-namespaced.yml | kubectl apply -f - | ||
| $kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-member.yml | ||
| ``` | ||
|
|
||
| ### Step 2 - initialize ClusterSet | ||
|
|
||
| We provide several template files to set up a ClusterSet quicker. You can run | ||
| the following commands to create a ClusterSet named `test-clusterset` and obtain | ||
| a ServiceAccount token for the member clusters (both cluster A and B in this | ||
| guide) to access the leader cluster (cluster A in this guide). | ||
|
|
||
| ```bash | ||
| $kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/multicluster_clusterset_template.yaml | ||
| $kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/multicluster_leader_access_token_template.yaml | ||
| $kubectl get secret leader-access-token -n antrea-multicluster -o yaml | grep -w -e '^apiVersion' -e '^data' -e '^metadata' -e '^ *name:' -e '^kind' -e ' ca.crt' -e ' token:' -e '^type' -e ' namespace' | sed -e 's/kubernetes.io\/service-account-token/Opaque/g' -e 's/antrea-multicluster/kube-system/g' > leader-access-token.yml | ||
| ``` | ||
|
|
||
| The last command saves the ServiceAccount token to `leader-access-token.yml` | ||
| which will be needed for member clusters to join the ClusterSet. Note, in this | ||
| guide, we use a pre-defined ServiceAccount `antrea-mc-member-access-sa` for all | ||
| member clusters. If you want to create a separate ServiceAccount for each member | ||
| cluster for security considerations, you can follow the instructions in the | ||
| [Multi-cluster User Guide](user-guide.md#set-up-access-to-leader-cluster). | ||
|
|
||
| Next, run the following commands to make cluster A join the ClusterSet also as a | ||
| member: | ||
|
|
||
| ```bash | ||
| $kubectl apply -f leader-access-token.yml | ||
| $curl -L https://raw.githubusercontent.com/antrea-io/antrea/v1.7.0/multicluster/config/samples/clusterset_init/multicluster_membercluster_template.yaml > multicluster_membercluster.yaml | ||
| $sed -e 's/test-cluster-member/test-cluster-leader/g' -e 's/<LEADER_CLUSTER_IP>/172.10.0.11/g' multicluster_membercluster.yaml | kubectl apply -f - | ||
| ``` | ||
|
|
||
| Here, `172.10.0.11` is the `kube-apiserver` IP of cluster A. You should replace | ||
| it with the `kube-apiserver` IP of your leader cluster. | ||
|
|
||
| ### Step 3 - specify Multi-cluster Gateway Node | ||
|
|
||
| Last, you need to choose a Node in cluster A to serve as the Multi-cluster | ||
| Gateway. The Node should have an IP that is reachable from the cluster B's | ||
| Gateway Node, so a tunnel can be created between the two Gateways. For more | ||
| information about Multi-cluster Gatweay, please refer to the [Multi-cluster | ||
| User Guide](user-guide.md#multi-cluster-gateway-configuration). | ||
|
|
||
| Assuming K8s Node `node-a` is chosen to be the Multi-cluster Gateway, run | ||
| the following command to annotate the Node with | ||
| `multicluster.antrea.io/gateway=true` (so `antrea-agent` can know it is the | ||
| Gateway Node from the annotation): | ||
|
|
||
| ```bash | ||
| $kubectl annotate node node-a multicluster.antrea.io/gateway=true | ||
| ``` | ||
|
|
||
| ## Set up Cluster B | ||
|
|
||
| Let us switch to cluster B. All the `kubectl` commands in the following steps | ||
| should be run with the `kubeconfig` for cluster B. | ||
|
|
||
| ### Step 1 - deploy Antrea Multi-cluster Controllers for member | ||
|
|
||
| Run the following command to deploy the member Antrea Multi-cluster Controller | ||
| into Namespace `kube-system`. | ||
|
|
||
| ```bash | ||
| $kubectl apply -f https://github.com/antrea-io/antrea/releases/download/$TAG/antrea-multicluster-member.yml | ||
| ``` | ||
|
|
||
| ### Step 2 - initialize ClusterSet | ||
|
|
||
| Run the following commands to make cluster B join the ClusterSet: | ||
|
|
||
| ```bash | ||
| $kubectl apply -f leader-access-token.yml | ||
| $curl -L https://raw.githubusercontent.com/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/multicluster_membercluster_template.yaml > multicluster_membercluster.yaml | ||
| $sed -e 's/<LEADER_CLUSTER_IP>/172.10.0.11/g' multicluster_membercluster.yaml | kubectl apply -f - | ||
| ``` | ||
|
|
||
| `leader-access-token.yml` saves the leader cluster ServiceAccount token which | ||
| was generated when initializing the ClusterSet in cluster A. | ||
|
|
||
| ### Step 3 - specify Multi-cluster Gateway Node | ||
|
|
||
| Assuming K8s Node `node-b` is chosen to be the Multi-cluster Gateway for cluster | ||
| B, run the following command to annotate the Node: | ||
|
|
||
| ```bash | ||
| $kubectl annotate node node-b multicluster.antrea.io/gateway=true | ||
| ``` | ||
|
|
||
| ## What is Next | ||
|
|
||
| So far, we set up a Antrea Multi-cluster ClusterSet with two clusters following | ||
| the above sections of this guide. Next, you can start to consume the Antrea | ||
| Multi-cluster features with the ClusterSet, including [Multi-cluster Services](user-guide.md#multi-cluster-service) | ||
| and [Multi-cluster ClusterNetworkPolicy Replication](user-guide.md#multi-cluster-clusternetworkpolicy-replication). | ||
| Check the relevant Antrea Multi-cluster User Guide sections to learn more. | ||
|
|
||
| If you want to add a new member cluster to your ClusterSet, you can follow the | ||
| steps for cluster B to do so. But note, you will need the following two changes: | ||
|
|
||
| 1. You need to add the new mumber cluster to the ClusterSet in the leader | ||
| cluster (cluster A). You can do that by adding the cluster ID of the new member | ||
| to `multicluster_clusterset_template.yaml` and re-applying the manifest in | ||
| cluster A. | ||
|
|
||
| 2. You need to update the member cluster ID in | ||
| `multicluster_membercluster_template.yaml` to the cluster ID of the new member | ||
| cluster. For example, in the step 2 of initialize ClusterSet, you can change the | ||
| commands to the following (assuming the new member cluster ID is | ||
| `test-cluster-member2`): | ||
|
|
||
| ```bash | ||
| $kubectl apply -f leader-access-token.yml | ||
| $curl -L https://raw.githubusercontent.com/antrea-io/antrea/$TAG/multicluster/config/samples/clusterset_init/multicluster_membercluster_template.yaml > multicluster_membercluster.yaml | ||
| $sed -e 's/<LEADER_CLUSTER_IP>/172.10.0.11/g' -e 's/test-cluster-member/test-cluster-member2/g' multicluster_membercluster.yaml | kubectl apply -f - | ||
| ``` |
Oops, something went wrong.