Deploy Packet Generator via provisioning code

[lixuna]

Provisioning a packet generator machine via code

Packet.net machine provisioning

• update terraform code for provisioning a Packet.net machine for the packet generator

Host Configuration Tasks:

• Ensure SSH keys for CNF dev team are installed

• Install the Mellanox drivers on Packet Generator host - refer to https://github.com/cncf/cnfs/blob/master/comparison/cnf_edge_throughput/packet_generator/README.md

• Add mellanox init service restart and make sure it works

• Ensure that Virtual Functions (VFs) are setup for the the Mellanox NIC

mst status
  - Note down the MST Device path, e.g. '/dev/mst/mt4117_pciconf0'
mlxconfig -d /dev/mst/mt4117_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=2
  - Use device path collected during the previous step

• Update GRUB GRUB_CMDLINE_LINUX in /etc/default/grub to include the following settings
  numa_balancing=disable intel_pstate=disable intel_iommu=on iommu=pt isolcpus=2-55 nohz_full=2-55 rcu_nocbs=2-55 hugepagesz=2M hugepages=8096
• Once the GRUB file has been updated, run update-grub2
• Update Linux host networking configuration to remove the default bonded interface
• reboot system

DEFERRED tasks:

• verify that the machine is set up for layer 2 networking
  • Deferring until after [Edge Throughput] Add code to provision layer 2 networking in Linux host for Packet.net #67 and Setup layer 2 networking using Packet API (from #67) #90 are complete

NFVbench tasks:

• Install docker prereqs - reference ehttps://github.com/cncf/cnfs/blob/master/comparison/cnf_edge_throughput/install_docker_prereqs.sh
• Install NFVbench reference https://github.com/cncf/cnfs/blob/master/comparison/cnf_edge_throughput/packet_generator/run_container.sh#L16
• Update configuration to support hugepages in NFVbench https://github.com/cncf/cnfs/blob/master/comparison/cnf_edge_throughput/packet_generator/run_container.sh#L16
• Modify the traffic server to use less memory https://github.com/cncf/cnfs/blob/master/comparison/cnf_edge_throughput/packet_generator/run_container.sh#L28
• To simplify use of NFVbench, add the following line to /root/.bashrc alias nfvbench="sudo docker exec -it nfvbench nfvbench -c /tmp/nfvbench/nfvbench_config.cfg"
• Create template file for the nfvbench_config.cfg based on https://github.com/cncf/cnfs/blob/master/comparison/cnf_edge_throughput/packet_generator/nfvbench_config.cfg
• In nfvbench_config.cfg, update the "generator_profile" interfaces to match the PCI IDs for the virtual network functions
  • Find the PCI devices that will be used for traffic. eg. lspci | grep Eth
  • Note the IDs of the virtual functions
• Update the "vlans" in nfvbench_config.cfg to match the network configuration between the servers. eg. vlans: [1070, 1064]

Testing:

nfvbench status

Verify that NFVbench is running using nfvbench --status. Output should be similar to what can be seen below

INFO Loading configuration file: /tmp/nfvbench/nfvbench_config.cfg
INFO EXT chain with OpenStack mode disabled
INFO Version: 0.0.0
INFO Status: idle

What's left as of Oct 24 2018:

• reboot system

• create a script to run terraform code for provisioning a Packet.net machine for the packet generator

• document how to run terraform code for provisioning a Packet.net machine for the packet generator

• document how to add SSH keys for CNF dev team are installed / reference packet.net help https://help.packet.net/article/31-ssh-access

[taylor]

@michaelspedersen please add a more minimal nfvbench config to this ticket or in the repo

[denverwilliams]

It looks like the --upstream-libs flag in the docs is not valid, going to skip using that flag for now.
https://github.com/cncf/cnfs/blob/master/comparison/cnf_edge_throughput/packet_generator/README.md

root@cnfdev07:/tmp/MLNX_OFED_LINUX-4.1-1.0.2.0-ubuntu16.04-x86_64# ./mlnxofedinstall --dpdk --upstream-libs --force

Unsupported installation option: '--upstream-libs'
To see list of supported options, run: ./mlnxofedinstall --help

[michaelspedersen]

@denverwilliams Just looked through the bash history on the generator machine, and it looks like I ended up using the following "./mlnxofedinstall --all --force"

I guess my initial messing with the installation caused me to miss that difference - Good thing you noticed.

[denverwilliams]

@michaelspedersen Awesome, thanks.

[taylor]

Virtual functions work as expected

root@cnfauto:/tmp/MLNX_OFED_LINUX-4.1-1.0.2.0-ubuntu16.04-x86_64# mst status
MST modules:
------------
    MST PCI module loaded
    MST PCI configuration module loaded

MST devices:
------------
/dev/mst/mt4099_pciconf0         - PCI configuration cycles access.
                                   domain:bus:dev.fn=0000:02:00.0 addr.reg=88 data.reg=92
                                   Chip revision is: 01
/dev/mst/mt4099_pci_cr0          - PCI direct access.
                                   domain:bus:dev.fn=0000:02:00.0 bar=0xc7500000 size=0x100000
                                   Chip revision is: 01

and mlxconfig -d /dev/mst/mt4117_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=2

[denverwilliams]

@taylor @lixuna Packet generator is now starting

root@cnfauto:~/cnfs/comparison/cnf_edge_throughput/packet_generator/ansible# nfvbench 
2018-10-23 19:51:31,660 INFO Loading configuration file: /tmp/nfvbench/nfvbench_config.cfg
2018-10-23 19:51:31,687 INFO EXT chain with OpenStack mode disabled
2018-10-23 19:51:31,687 INFO -c /tmp/nfvbench/nfvbench_config.cfg
2018-10-23 19:51:31,692 INFO ServiceChain initialized.
2018-10-23 19:51:31,692 INFO ChainRunner initialized.
2018-10-23 19:51:31,692 INFO Adjusting frame size 64 Bytes to minimum size 68 Bytes due to traffic generator restriction
2018-10-23 19:51:31,693 INFO Connecting to TRex...
2018-10-23 19:51:36,701 INFO TRex server is running...
nohup: ignoring input and appending output to 'nohup.out'

[lixuna]

What's left as of Oct 24 2018:

• reboot system

• create a script to run terraform code for provisioning a Packet.net machine for the packet generator

• document how to run terraform code for provisioning a Packet.net machine for the packet generator

• document how to add SSH keys for CNF dev team are installed / reference packet.net help https://help.packet.net/article/31-ssh-access

[lixuna]

https://github.com/cncf/cnfs/blob/master/comparison/cnf_edge_throughput/README-layer2-network.md