cloudlab.md

Demikernel in Cloudlab

Requesting resources on Cloudlab

• Look at the node IDs which have which have Mellanox ConnectX-4 or above in the hardware allocation in the hardware list (https://docs.cloudlab.us/hardware.html).
  • Some of the ones that work: Utah (c6525-100g, c6525-25g, xl170, d6515), Clemson (r7525)
• Starting a new experiment
  • Choose Profile: small-lan
  • Parameterize
    • No of nodes: 2+
    • OS Image: Ubuntu 20.04
    • Optional Physical Node Type [REALLY IMPORTANT]: Put one of the IDSs mentioned above. Check availability here: https://www.cloudlab.us/resinfo.php?embedded=true
• Finalize
• Schedule

Pre Requisites

Run the following scripts on all the nodes.

Installing Mellanox OFED (mlx5)

wget https://content.mellanox.com/ofed/MLNX_OFED-5.5-1.0.3.2/MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.tgz --no-check-certificate
tar -xzvf MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64.tgz 
cd MLNX_OFED_LINUX-5.5-1.0.3.2-ubuntu20.04-x86_64
sudo ./mlnxofedinstall --upstream-libs --dpdk

Installing Rust Cargo

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Setup for Demikernel

Pull the code from repository

Fetch the code from the original repository and use dev branch for latest updates

git clone --recursive https://github.com/demikernel/demikernel.git
cd demikernel
git checkout dev 

Running setup scripts

Execute these scripts within the demikernel repository to setup pre-requisite for Ubuntu, Rust DPDK, and Hugepages

./scripts/setup/debian.sh
./scripts/setup/dpdk.sh
./scripts/setup/hugepages.sh

Running Demikernel

Run make all to make sure everything compiles.

Configuration files

Edit the config files in scripts/config/default.yaml using the following steps.

Node 0 → Server

• Get information about the Mellanox NICs using lshw

  user@node0:~/demikernel/scripts/config$ sudo lshw | grep -i Mellanox -A 10
   vendor: Mellanox Technologies
                   physical id: 0.1
                   bus info: pci@0000:03:00.1
                   logical name: ens1f1
                   version: 00
                   serial: 9c:dc:71:5e:2f:71
                   capacity: 25Gbit/s
                   width: 64 bits
                   clock: 33MHz
                   capabilities: pciexpress vpd msix pm bus_master cap_list ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd autonegotiation
                   configuration: autonegotiation=off broadcast=yes driver=mlx5_core driverversion=5.5-1.0.3 duplex=full firmware=14.18.2030 (HP_2420110034) ip=10.10.1.1 latency=0 link=yes multicast=yes
  

From the above output extract the following entities required for config file:

1. my_interface_name → logical name (ens1f1)
2. my_link_address → serial (9c:dc:71:5e:2f:71)

If there are multiple interfaces pick one of the interface. For this example, our interface name is ens1f1.

• Finding out the IP of the system using ifconfig

    user@node0:~/demikernel/scripts/config$ ifconfig ens1f1
    ens1f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
            inet 10.10.1.1  netmask 255.255.255.0  broadcast 10.10.1.255
            inet6 fe80::9edc:71ff:fe5e:2f71  prefixlen 64  scopeid 0x20<link>
            ether 9c:dc:71:5e:2f:71  txqueuelen 1000  (Ethernet)
            RX packets 8  bytes 670 (670.0 B)
            RX errors 0  dropped 1  overruns 0  frame 0
            TX packets 19  bytes 1895 (1.8 KB)
            TX errors 1  dropped 0 overruns 0  carrier 1  collisions 0

IP: 10.10.1.1

• Finding the PCI address for the eal_init section using lshw

    user@node0:~/demikernel/scripts/config$ sudo lshw -c network -businfo
    Bus info          Device     Class          Description
    =======================================================
    pci@0000:03:00.1  ens1f1     network        MT27710 Family [ConnectX-4 Lx]

From the above example, the PCI address will be 03:00.1

Node 1 → Client

Apply the same commands to get the values for Node 1 which will be our client.

• Get information about the NIC using lshw

    user@node1:~/demikernel/scripts/config$ sudo lshw | grep -i Mellanox -A 10
                    vendor: Mellanox Technologies
                    physical id: 0.1
                    bus info: pci@0000:03:00.1
                    logical name: ens1f1
                    version: 00
                    serial: 9c:dc:71:5d:41:31
                    capacity: 25Gbit/s
                    width: 64 bits
                    clock: 33MHz
                    capabilities: pciexpress vpd msix pm bus_master cap_list ethernet physical 1000bt-fd 10000bt-fd 25000bt-fd autonegotiation
                    configuration: autonegotiation=off broadcast=yes driver=mlx5_core driverversion=5.5-1.0.3 duplex=full firmware=14.18.2030 (HP_2420110034) ip=10.10.1.2 latency=0 link=yes multicast=yes

From the above output extract the following entities required for config file

1. my_interface_name → logical name (ens1f1)
2. my_link_address → serial (9c:dc:71:5d:41:31)

If there are multiple interfaces pick one of the interface. For this example, our interface name is ens1f1.

• Get information about IP using ifconfig

    user@node1:~/demikernel$ ifconfig ens1f1
    ens1f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
            inet 10.10.1.2  netmask 255.255.255.0  broadcast 10.10.1.255
            inet6 fe80::9edc:71ff:fe5d:4131  prefixlen 64  scopeid 0x20<link>
            ether 9c:dc:71:5d:41:31  txqueuelen 1000  (Ethernet)
            RX packets 14  bytes 1293 (1.2 KB)
            RX errors 0  dropped 1  overruns 0  frame 0
            TX packets 19  bytes 1895 (1.8 KB)
            TX errors 1  dropped 0 overruns 0  carrier 1  collisions 0
  

IP → 10.10.1.2

• Finding the PCI address for the eal_init section using lshw

  
    user@node1:~/demikernel$ sudo lshw -c network -businfo
    Bus info          Device     Class          Description
    =======================================================
    pci@0000:03:00.1  ens1f1     network        MT27710 Family [ConnectX-4 Lx]
  
  

From the above example, the PCI address will be 03:00.1

Putting together config file for Node 0

    # Copyright (c) Microsoft Corporation.
    # Licensed under the MIT license.
    
    server:
      bind:
        host: 10.10.1.1 (IP of Node 0)
        port: 12345
      client:
        host: 10.10.1.2 (IP of Node 1)
        port: 12345
    catnip:
      my_ipv4_addr: 10.10.1.1 (IP of Node 0)
      my_link_addr: "9c:dc:71:5e:2f:71" (Serial of ens1f1 in Node 0)
      my_interface_name: "ens1f1" 
      arp_table:
        <Link_addr_Node_x> : <IP_Node_x>
        "9c:dc:71:5e:2f:71": 10.10.1.1 
        "9c:dc:71:5d:41:31": 10.10.1.2
    dpdk:
            eal_init: ["-c", "0xff", "-n", "4", "-w", "03:00.1","--proc-type=auto"]
                                                          ||
                                                          \/
                                                      PCI address of ens1f1 in Node 0
    # vim: set tabstop=2 shiftwidth=2

Putting together config file for Node 1

    user@node1:~/demikernel$ cat scripts/config/cl_node_1.yaml 
    # Copyright (c) Microsoft Corporation.
    # Licensed under the MIT license.
    
    client:
      connect_to:
        host: 10.10.1.1 (IP of Node 0)
        port: 12345
      client:
        host: 10.10.1.2 (IP of Node 1)
        port: 12345
    catnip:
      my_ipv4_addr: 10.10.1.2 (IP of Node 1)
      my_link_addr: "9c:dc:71:5d:41:31" (Serial of ens1f1 in Node 1)
      my_interface_name: "ens1f1"
      arp_disable: true
    dpdk:
            eal_init: ["-c", "0xff", "-n", "4", "-w", "03:00.1","--proc-type=auto"]
                                                          ||
                                                          \/
                                                      PCI address of ens1f1 in Node 1
    # vim: set tabstop=2 shiftwidth=2

Running the code

Running test files:

Edit the CONFIG_PATH in Makefile with the absolute path of your config file:

Node 0 →

    export CONFIG_PATH ?= /users/user/demikernel/scripts/config/cl_node_0.yaml

Node 1 →

    export CONFIG_PATH ?= /users/user/demikernel/scripts/config/cl_node_1.yaml

Run the following test scripts to make sure everything works

Node 0 (Running Server) →

    user@node0:~/demikernel$ PEER=server TIMEOUT=300 TEST=udp_ping_pong sudo -E make test-catnip

Node 1 (Running Client) →

    user@node1:~/demikernel$ PEER=client TIMEOUT=300 TEST=udp_ping_pong sudo -E make test-catnip