Install & Configure OvS with DPDK

Steps

1. Change to the patched-ovs directory within the ovs-dpdk-with-tso parent directory.

   root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso# cd patched-ovs/
   root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# 
   

2. Disable Apparmor profile for libvirtd to avoid further problems while using vhostuser port. Changes will reflect only after a reboot (Step 5).

   root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# sudo ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/usr.sbin.libvirtd
   

3. In /etc/libvirt/qemu.conf uncomment/add these lines:

   user = "root"
   group = "root"
   

4. Through Grub settings, change the default Hugepagesize to 1GB. Also, set Intel I/O Memory Management parameters as stated below.

   root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# gedit /etc/default/grub 
   GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G iommu=pt intel_iommu=on"
   

5. Now reboot the machine.

   root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# reboot

6. After reboot, allocate the required number of Hugepages and mount them as a file system.

   root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# echo 6 > /proc/sys/vm/nr_hugepages
   root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# mount -t hugetlbfs -o pagesize=1G none /dev/hugepages
   

7. Restart the libvirtd service.

   root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# systemctl restart libvirtd.service

8. Execute the following commands to configure, build and install OvS with DPDK.

   root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# ./boot.sh 
   root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# ./configure --with-dpdk=/usr/local/src/ovs-dpdk-with-tso/dpdk-stable-18.11.2/x86_64-native-linuxapp-gcc DESTDIR=install  
   root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# make
   root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# make install
   

9. Create a database that can be used by the ovsdb-server. Make sure that you are in the patched-ovs directory while performing this step.

   root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# mkdir -p /usr/local/etc/openvswitch
   root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# ovsdb-tool create /usr/local/etc/openvswitch/conf.db \
   vswitchd/vswitch.ovsschema
   

10. Configure ovsdb-server to use the database created above.

root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# mkdir -p /usr/local/var/run/openvswitch
root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \
--remote=db:Open_vSwitch,Open_vSwitch,manager_options \
--pidfile --detach

11. Initialize the database (Only needed when the database is created for the first time).

root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# ovs-vsctl --no-wait init

12. Start the Open vSwitch Daemon.

root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# ovs-vswitchd unix:/usr/local/var/run/openvswitch/db.sock --pidfile --detach
2019-09-06T10:52:03Z|00001|ovs_numa|INFO|Discovered 8 CPU cores on NUMA node 0
2019-09-06T10:52:03Z|00002|ovs_numa|INFO|Discovered 1 NUMA nodes and 8 CPU cores
2019-09-06T10:52:03Z|00003|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock: connecting...
2019-09-06T10:52:03Z|00004|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock: connected
2019-09-06T10:52:03Z|00005|dpdk|INFO|DPDK Disabled - Use other_config:dpdk-init to enable

13. Now initialize the dpdk datapath and set all the required dpdk parameters. Please note that Multi Segment mbuf feature needs to be set true in order to enable TSO support.

root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# ovs-vsctl set Open_vSwitch . other_config:dpdk-init=true
root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x7F
root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# ovs-vsctl set Open_vSwitch . other_config:dpdk-lcore-mask=0x80
root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# ovs-vsctl set Open_vSwitch . other_config:dpdk-socket-mem="1024"
root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# ovs-vsctl set Open_vSwitch . other_config:dpdk-multi-seg-mbufs=true

14. Kill the ovs-vswitchd and ovsdb-server daemons.

root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# killall ovs-vswitchd ovsdb-server

15. Execute these steps to start OvS again. When OvS starts you will be able to see the changed values of DPDK parameters.

root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# ovsdb-tool create /usr/local/etc/openvswitch/conf.db \
vswitchd/vswitch.ovsschema

root@SDCLAB04:/usr/local/src/ovs-dpdk-with-tso/patched-ovs# ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \
--remote=db:Open_vSwitch,Open_vSwitch,manager_options \
--pidfile --detach

Note: Whenever any change is done to DPDK parameters, Step 14 & 15 needs to be executed. Only after executing those steps, changes will appear in OvS.

Now move on the installation steps at Create OvS-DPDK bridge with vhostuser ports page.