From ada7fd063c31c075a2482abe7a9a168b8d7a6240 Mon Sep 17 00:00:00 2001 From: Ze Gan Date: Thu, 9 May 2024 04:40:35 +0800 Subject: [PATCH] [dash]: Add DPU kvm topology guidance (#1676) --- doc/dash/dash-sonic-kvm.md | 93 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) diff --git a/doc/dash/dash-sonic-kvm.md b/doc/dash/dash-sonic-kvm.md index bd5237060b9..dee316b6b23 100644 --- a/doc/dash/dash-sonic-kvm.md +++ b/doc/dash/dash-sonic-kvm.md @@ -16,6 +16,12 @@ - [4 Dataflow](#4-dataflow) - [4.1 Data plane](#41-data-plane) - [4.2 Control plane](#42-control-plane) +- [5 KVM topology](#5-kvm-topology) + - [5.1 Single device mode testbed](#51-single-device-mode-testbed) + - [5.1.1 Create DPU testbed](#511-create-dpu-testbed) + - [5.1.2 Connect to the DPU](#512-connect-to-the-dpu) + - [5.1.3 gnmi client access](#513-gnmi-client-access) + - [5.2 DPU with VPP NPU testbed](#52-dpu-with-vpp-npu-testbed) # 1 Motivation @@ -87,3 +93,90 @@ graph TD In the physical SmartSwitch, configuration is forwarded via GNMI in the NPU. So, in the virtual SONiC environment, the SWSS module is capable of receiving configuration from an external GNMI service through the management port, eth-midplane. However, in the single device mode, the GNMI service can also be run within the KVM and directly forward the configuration to the local SWSS module. +# 5 KVM topology + +This section describes the steps to setup a virtual DPU testbed, please check a [pre-learning document](https://github.com/sonic-net/sonic-mgmt/blob/master/docs/testbed/README.testbed.VsSetup.md) to setup your environment and download the EOS container. Beginning the following steps, I assume you have got the cEOS image with the specific version(https://github.com/sonic-net/sonic-mgmt/blob/master/ansible/group_vars/all/ceos.yml) and sonic-mgmt containers. + + +- The normal SONiC KVM image without DPU dataplane can be downloaded from: https://sonic-build.azurewebsites.net/api/sonic/artifacts?branchName=master&platform=vs&target=target/sonic-vs.img.gz +- The DPU SONiC KVM image with dataplane will be released at the next stage + +## 5.1 Single device mode testbed + +## 5.1.1 Create DPU testbed + +```shell + +# Login to your sonic-mgmt container +ssh -i ~/.ssh/id_rsa_docker_sonic_mgmt zegan@172.17.0.2 + +# CD to the ansible directory +cd /data/sonic-mgmt/ansible + +# Build the DPU testbed +./testbed-cli.sh -t vtestbed.yaml -m veos_vtb add-topo vms-kvm-dpu password.txt +./testbed-cli.sh -t vtestbed.yaml -m veos_vtb deploy-mg vms-kvm-dpu veos_vtb password.txt + +``` + +If you are lucky, your testbed should pass all testcases via following commands + +``` shell +# Login to your sonic-mgmt container +ssh -i ~/.ssh/id_rsa_docker_sonic_mgmt zegan@172.17.0.2 + +# CD to the tests directory +cd /data/sonic-mgmt/tests + +# Run testcases +./run_tests.sh -u -n vms-kvm-dpu -d vlab-01 -m individual -e --skip_sanity -e --disable_loganalyzer -c dash/test_dash_vnet.py -f vtestbed.yaml -i ../ansible/veos_vtb -e "--skip_dataplane_checking" +``` + +## 5.1.2 Connect to the DPU + +``` shell +# Login to the DPU, the default password is `password` and the default mgmt IP address is `10.250.0.101` +sshpass -p 'password' ssh -o TCPKeepAlive=yes -o ServerAliveInterval=30 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no admin@10.250.0.101 + +``` + +## 5.1.3 gnmi client access + +Once dash test cases are completed, we can find required certificates in sonic-mgmt container. + +``` +# The certificates are in /tmp directory +user@mgmt:~$ ls -l /tmp/e55648be-5a9c-49da-bcf7-645b1a926fb6 +total 40 +-rw-r--r-- 1 user user 99 Apr 12 05:52 extfile.cnf +-rw------- 1 user user 1675 Apr 12 05:52 gnmiCA.key +-rw-r--r-- 1 user user 1131 Apr 12 05:52 gnmiCA.pem +-rw-r--r-- 1 user user 41 Apr 12 05:52 gnmiCA.srl +-rw-r--r-- 1 user user 1017 Apr 12 05:52 gnmiclient.crt +-rw-r--r-- 1 user user 907 Apr 12 05:52 gnmiclient.csr +-rw------- 1 user user 1679 Apr 12 05:52 gnmiclient.key +-rw-r--r-- 1 user user 1070 Apr 12 05:52 gnmiserver.crt +-rw-r--r-- 1 user user 907 Apr 12 05:52 gnmiserver.csr +-rw------- 1 user user 1675 Apr 12 05:52 gnmiserver.key +``` + +Please install gnmi_cli_py from https://github.com/lguohan/gnxi/tree/master/gnmi_cli_py. + +Restart GNMI server with server certificates: + +``` +# Run below commands on DUT +docker exec gnmi supervisorctl stop gnmi-native +docker exec gnmi bash -c "/usr/bin/nohup /usr/sbin/telemetry -logtostderr --port 50052 --server_crt /etc/sonic/telemetry/gnmiserver.crt --server_key /etc/sonic/telemetry/gnmiserver.key --ca_crt /etc/sonic/telemetry/gnmiCA.pem -gnmi_native_write=true -v=10 >/root/gnmi.log 2>&1 &" +``` + +Follow below example to run GNMI client: + +``` +# /root/update1 and /root/update2 are protobuf value for DASH table +python2 /root/gnxi/gnmi_cli_py/py_gnmicli.py --timeout 30 -t 10.0.0.88 -p 50052 -xo sonic-db -rcert /root/gnmiCA.pem -pkey /root/gnmiclient.key -cchain /root/gnmiclient.crt -m set-update --xpath /APPL_DB/localhost/DASH_APPLIANCE_TABLE[key=123] /APPL_DB/localhost/DASH_VNET_TABLE[key=Vnet1] --value $/root/update1 $/root/update2 +``` + +## 5.2 DPU with VPP NPU testbed + +TBD