Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dash]: Add DPU kvm topology guidance #1676

Merged
merged 2 commits into from
May 8, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
93 changes: 93 additions & 0 deletions doc/dash/dash-sonic-kvm.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@
- [4 Dataflow](#4-dataflow)
- [4.1 Data plane](#41-data-plane)
- [4.2 Control plane](#42-control-plane)
- [5 KVM topology](#5-kvm-topology)
- [5.1 Single device mode testbed](#51-single-device-mode-testbed)
- [5.1.1 Create DPU testbed](#511-create-dpu-testbed)
- [5.1.2 Connect to the DPU](#512-connect-to-the-dpu)
- [5.1.3 gnmi client access](#513-gnmi-client-access)
- [5.2 DPU with VPP NPU testbed](#52-dpu-with-vpp-npu-testbed)

# 1 Motivation

Expand Down Expand Up @@ -87,3 +93,90 @@ graph TD

In the physical SmartSwitch, configuration is forwarded via GNMI in the NPU. So, in the virtual SONiC environment, the SWSS module is capable of receiving configuration from an external GNMI service through the management port, eth-midplane. However, in the single device mode, the GNMI service can also be run within the KVM and directly forward the configuration to the local SWSS module.

# 5 KVM topology

This section describes the steps to setup a virtual DPU testbed, please check a [pre-learning document](https://github.com/sonic-net/sonic-mgmt/blob/master/docs/testbed/README.testbed.VsSetup.md) to setup your environment and download the EOS container. Beginning the following steps, I assume you have got the cEOS image with the specific version(https://github.com/sonic-net/sonic-mgmt/blob/master/ansible/group_vars/all/ceos.yml) and sonic-mgmt containers.


- The normal SONiC KVM image without DPU dataplane can be downloaded from: https://sonic-build.azurewebsites.net/api/sonic/artifacts?branchName=master&platform=vs&target=target/sonic-vs.img.gz
- The DPU SONiC KVM image with dataplane will be released at the next stage

## 5.1 Single device mode testbed

## 5.1.1 Create DPU testbed

```shell

# Login to your sonic-mgmt container
ssh -i ~/.ssh/id_rsa_docker_sonic_mgmt zegan@172.17.0.2

# CD to the ansible directory
cd /data/sonic-mgmt/ansible

# Build the DPU testbed
./testbed-cli.sh -t vtestbed.yaml -m veos_vtb add-topo vms-kvm-dpu password.txt
./testbed-cli.sh -t vtestbed.yaml -m veos_vtb deploy-mg vms-kvm-dpu veos_vtb password.txt

```

If you are lucky, your testbed should pass all testcases via following commands

``` shell
# Login to your sonic-mgmt container
ssh -i ~/.ssh/id_rsa_docker_sonic_mgmt zegan@172.17.0.2

# CD to the tests directory
cd /data/sonic-mgmt/tests

# Run testcases
./run_tests.sh -u -n vms-kvm-dpu -d vlab-01 -m individual -e --skip_sanity -e --disable_loganalyzer -c dash/test_dash_vnet.py -f vtestbed.yaml -i ../ansible/veos_vtb -e "--skip_dataplane_checking"
```

## 5.1.2 Connect to the DPU

``` shell
# Login to the DPU, the default password is `password` and the default mgmt IP address is `10.250.0.101`
sshpass -p 'password' ssh -o TCPKeepAlive=yes -o ServerAliveInterval=30 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no admin@10.250.0.101

```

## 5.1.3 gnmi client access

Once dash test cases are completed, we can find required certificates in sonic-mgmt container.

```
# The certificates are in /tmp directory
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tmp

/tmp will be cleaned after reboot. Does it matter? #Pending

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now we generate new certificate for dash test, if there's any requirement, we can use fixed certificate.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's be frank and warn the reader if it has possibility to happen. Another option is to use fixed certificate.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to use another PR to resolve this comment.

user@mgmt:~$ ls -l /tmp/e55648be-5a9c-49da-bcf7-645b1a926fb6
total 40
-rw-r--r-- 1 user user 99 Apr 12 05:52 extfile.cnf
-rw------- 1 user user 1675 Apr 12 05:52 gnmiCA.key
-rw-r--r-- 1 user user 1131 Apr 12 05:52 gnmiCA.pem
-rw-r--r-- 1 user user 41 Apr 12 05:52 gnmiCA.srl
-rw-r--r-- 1 user user 1017 Apr 12 05:52 gnmiclient.crt
-rw-r--r-- 1 user user 907 Apr 12 05:52 gnmiclient.csr
-rw------- 1 user user 1679 Apr 12 05:52 gnmiclient.key
-rw-r--r-- 1 user user 1070 Apr 12 05:52 gnmiserver.crt
-rw-r--r-- 1 user user 907 Apr 12 05:52 gnmiserver.csr
-rw------- 1 user user 1675 Apr 12 05:52 gnmiserver.key
```

Please install gnmi_cli_py from https://github.com/lguohan/gnxi/tree/master/gnmi_cli_py.

Restart GNMI server with server certificates:

```
# Run below commands on DUT
docker exec gnmi supervisorctl stop gnmi-native
docker exec gnmi bash -c "/usr/bin/nohup /usr/sbin/telemetry -logtostderr --port 50052 --server_crt /etc/sonic/telemetry/gnmiserver.crt --server_key /etc/sonic/telemetry/gnmiserver.key --ca_crt /etc/sonic/telemetry/gnmiCA.pem -gnmi_native_write=true -v=10 >/root/gnmi.log 2>&1 &"
```

Follow below example to run GNMI client:

```
# /root/update1 and /root/update2 are protobuf value for DASH table
python2 /root/gnxi/gnmi_cli_py/py_gnmicli.py --timeout 30 -t 10.0.0.88 -p 50052 -xo sonic-db -rcert /root/gnmiCA.pem -pkey /root/gnmiclient.key -cchain /root/gnmiclient.crt -m set-update --xpath /APPL_DB/localhost/DASH_APPLIANCE_TABLE[key=123] /APPL_DB/localhost/DASH_VNET_TABLE[key=Vnet1] --value $/root/update1 $/root/update2
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

python2

Possible to use python3? python2 is deprecated. #Closed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

py_gnmicli.py is using python2, need to rewrite this tool.

```

## 5.2 DPU with VPP NPU testbed

TBD