-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[dash]: Add DPU kvm topology guidance #1676
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -16,6 +16,12 @@ | |
- [4 Dataflow](#4-dataflow) | ||
- [4.1 Data plane](#41-data-plane) | ||
- [4.2 Control plane](#42-control-plane) | ||
- [5 KVM topology](#5-kvm-topology) | ||
- [5.1 Single device mode testbed](#51-single-device-mode-testbed) | ||
- [5.1.1 Create DPU testbed](#511-create-dpu-testbed) | ||
- [5.1.2 Connect to the DPU](#512-connect-to-the-dpu) | ||
- [5.1.3 gnmi client access](#513-gnmi-client-access) | ||
- [5.2 DPU with VPP NPU testbed](#52-dpu-with-vpp-npu-testbed) | ||
|
||
# 1 Motivation | ||
|
||
|
@@ -87,3 +93,90 @@ graph TD | |
|
||
In the physical SmartSwitch, configuration is forwarded via GNMI in the NPU. So, in the virtual SONiC environment, the SWSS module is capable of receiving configuration from an external GNMI service through the management port, eth-midplane. However, in the single device mode, the GNMI service can also be run within the KVM and directly forward the configuration to the local SWSS module. | ||
|
||
# 5 KVM topology | ||
|
||
This section describes the steps to setup a virtual DPU testbed, please check a [pre-learning document](https://github.com/sonic-net/sonic-mgmt/blob/master/docs/testbed/README.testbed.VsSetup.md) to setup your environment and download the EOS container. Beginning the following steps, I assume you have got the cEOS image with the specific version(https://github.com/sonic-net/sonic-mgmt/blob/master/ansible/group_vars/all/ceos.yml) and sonic-mgmt containers. | ||
|
||
|
||
- The normal SONiC KVM image without DPU dataplane can be downloaded from: https://sonic-build.azurewebsites.net/api/sonic/artifacts?branchName=master&platform=vs&target=target/sonic-vs.img.gz | ||
- The DPU SONiC KVM image with dataplane will be released at the next stage | ||
|
||
## 5.1 Single device mode testbed | ||
|
||
## 5.1.1 Create DPU testbed | ||
|
||
```shell | ||
|
||
# Login to your sonic-mgmt container | ||
ssh -i ~/.ssh/id_rsa_docker_sonic_mgmt zegan@172.17.0.2 | ||
|
||
# CD to the ansible directory | ||
cd /data/sonic-mgmt/ansible | ||
|
||
# Build the DPU testbed | ||
./testbed-cli.sh -t vtestbed.yaml -m veos_vtb add-topo vms-kvm-dpu password.txt | ||
./testbed-cli.sh -t vtestbed.yaml -m veos_vtb deploy-mg vms-kvm-dpu veos_vtb password.txt | ||
|
||
``` | ||
|
||
If you are lucky, your testbed should pass all testcases via following commands | ||
|
||
``` shell | ||
# Login to your sonic-mgmt container | ||
ssh -i ~/.ssh/id_rsa_docker_sonic_mgmt zegan@172.17.0.2 | ||
|
||
# CD to the tests directory | ||
cd /data/sonic-mgmt/tests | ||
|
||
# Run testcases | ||
./run_tests.sh -u -n vms-kvm-dpu -d vlab-01 -m individual -e --skip_sanity -e --disable_loganalyzer -c dash/test_dash_vnet.py -f vtestbed.yaml -i ../ansible/veos_vtb -e "--skip_dataplane_checking" | ||
``` | ||
|
||
## 5.1.2 Connect to the DPU | ||
|
||
``` shell | ||
# Login to the DPU, the default password is `password` and the default mgmt IP address is `10.250.0.101` | ||
sshpass -p 'password' ssh -o TCPKeepAlive=yes -o ServerAliveInterval=30 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no admin@10.250.0.101 | ||
|
||
``` | ||
|
||
## 5.1.3 gnmi client access | ||
|
||
Once dash test cases are completed, we can find required certificates in sonic-mgmt container. | ||
|
||
``` | ||
# The certificates are in /tmp directory | ||
user@mgmt:~$ ls -l /tmp/e55648be-5a9c-49da-bcf7-645b1a926fb6 | ||
total 40 | ||
-rw-r--r-- 1 user user 99 Apr 12 05:52 extfile.cnf | ||
-rw------- 1 user user 1675 Apr 12 05:52 gnmiCA.key | ||
-rw-r--r-- 1 user user 1131 Apr 12 05:52 gnmiCA.pem | ||
-rw-r--r-- 1 user user 41 Apr 12 05:52 gnmiCA.srl | ||
-rw-r--r-- 1 user user 1017 Apr 12 05:52 gnmiclient.crt | ||
-rw-r--r-- 1 user user 907 Apr 12 05:52 gnmiclient.csr | ||
-rw------- 1 user user 1679 Apr 12 05:52 gnmiclient.key | ||
-rw-r--r-- 1 user user 1070 Apr 12 05:52 gnmiserver.crt | ||
-rw-r--r-- 1 user user 907 Apr 12 05:52 gnmiserver.csr | ||
-rw------- 1 user user 1675 Apr 12 05:52 gnmiserver.key | ||
``` | ||
|
||
Please install gnmi_cli_py from https://github.com/lguohan/gnxi/tree/master/gnmi_cli_py. | ||
|
||
Restart GNMI server with server certificates: | ||
|
||
``` | ||
# Run below commands on DUT | ||
docker exec gnmi supervisorctl stop gnmi-native | ||
docker exec gnmi bash -c "/usr/bin/nohup /usr/sbin/telemetry -logtostderr --port 50052 --server_crt /etc/sonic/telemetry/gnmiserver.crt --server_key /etc/sonic/telemetry/gnmiserver.key --ca_crt /etc/sonic/telemetry/gnmiCA.pem -gnmi_native_write=true -v=10 >/root/gnmi.log 2>&1 &" | ||
``` | ||
|
||
Follow below example to run GNMI client: | ||
|
||
``` | ||
# /root/update1 and /root/update2 are protobuf value for DASH table | ||
python2 /root/gnxi/gnmi_cli_py/py_gnmicli.py --timeout 30 -t 10.0.0.88 -p 50052 -xo sonic-db -rcert /root/gnmiCA.pem -pkey /root/gnmiclient.key -cchain /root/gnmiclient.crt -m set-update --xpath /APPL_DB/localhost/DASH_APPLIANCE_TABLE[key=123] /APPL_DB/localhost/DASH_VNET_TABLE[key=Vnet1] --value $/root/update1 $/root/update2 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. py_gnmicli.py is using python2, need to rewrite this tool. |
||
``` | ||
|
||
## 5.2 DPU with VPP NPU testbed | ||
|
||
TBD |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/tmp will be cleaned after reboot. Does it matter? #Pending
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ganglyu
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now we generate new certificate for dash test, if there's any requirement, we can use fixed certificate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's be frank and warn the reader if it has possibility to happen. Another option is to use fixed certificate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Feel free to use another PR to resolve this comment.