Donard: A PCIe Peer-2-Peer kernel patch and library that builds on top of NVM. Express. Also see https://github.com/sbates130272/linux-donard for the linux kernel source used with the user-space code and modules.
Python Shell
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
LICENSES
donard_rdma @ ce648ef
donard_tools @ 872b730
io_peer_mem @ 22c0f98
libargconfig @ 4e0fcaa
libdonard @ 068bc48
likwid @ 9cc6507
mbw @ 1ff382d
mmap_dd @ 4d1f962
mmap_test @ 7967b90
nvme_donard @ 98e259b
nvram_bench @ fa9fce4
papers
perform
perftest @ 8d59cb2
.gitignore
.gitmodules
README.md
linux-donard.pdf
lsmod.list
packages.list

README.md

Donard

Summary

Donard: A PCIe Peer-2-Peer Library that builds on top of NVM Express

Code Structure

The Donard repo is really just a container for a multiple of smaller git repos. See of them in turn for their README and licensing information.

Installation

  1. Start from your Linux distro of choice on a bare-metal machine. You could try running this inside a VM but since we need to access bare-metal I am not sure why you would. We used Debian Wheezy as a start point but we have made many mods from there.

  2. Install the packages required to install a updated Linux kernel. At a minimum you are going to need the following packages. kernel-package, libncurses5-dev, fakeroot, bzip2 and bc.

  3. Clone the Donard version of the linux kernel from the relevant GitHub repo. i.e. git clone https://github.com/sbates130272/linux-donard.git. For a view of how this kernel is constructed see linux-donard.pdf.

  4. Build and install this version of the linux kernel. cd linux-donard make-kpkg clean fakeroot make-kpkg --initrd --append-to-version=-docker-donard kernel_image kernel_headers cd .. dpkg -i linux-image-3.16.3-docker-donard+_3.16.3-docker-donard+-10.00.Custom_amd64.deb dpkg -i linux-headers-3.16.3-docker-donard+_3.16.3-docker-donard+-10.00.Custom_amd64.deb

  5. Now pull the rest of the donard project code: git clone https://github.com/sbates130272/donard.git

  6. Install the Nvidia driver. We used the instructions at https://wiki.debian.org/NvidiaGraphicsDrivers. Note that several people have has issues with this step and tieing the Nvidia code into the nvme_donard module. So we outline a more complete procedure in the next section.

Installation - Nvidia code and nvme_donard (thanks Jack ;-))

  1. Build ubuntu 14.04.3 server on the system

  2. apt-get install git kernel-package, libncurses5-dev fakerootbzip2 bc

  3. git clone https://github.com/sbates130272/linux-donard.git.

  4. apt-get update followed by apt–get upgrade

  5. cd to linux-donard directory

  6. make-kpkg clean

  7. fakeroot make-kpkg —initrd --append-to-version=-docker-donard kernel_image kernel_headers (Accept defaults from pmem and DAX).

  8. cd ..

  9. (As root) dpkg -I donard.deb

  10. Reboot

  11. uname -r to verify kernel loaded

  12. Check soft links

  13. /lib/modules/3.19.1-docker-donard+/build and /lib/modules/3.19.1-docker-donard+/source should point to /usr/src/linux-headers-3.19.1-docker-donard+

  14. Load latest cuda package from nvidia

  15. As root

  16. Copy .deb file from https://developer.nvidia.com/cuda-downloads

  17. dpkg –I cuda*.deb

  18. apt-get update

  19. Apt-get install cuda

  20. Export PATH and library variables

  21. Driver will be installed in /usr/src/nvidia-352-352.39 for cuda 7.5

  22. cd to nvidia src directory

  23. As root, not sudo

  24. kernelver=$(uname –r)

  25. kernel_source_dir=/lib/modules/$kernelver/build

  26. make module KERNDIR=/lib/modules/$kernelver
    IGNORE_XEN_PRESENCE=1 IGNORE_CC_MISMATCH=1
    SYSSRC=$kernel_source_dir LD=/usr/bin/ld.bfd

  27. Verify Module.symvers is built in nvidia src directory

  28. Build Donard components

  29. cd to home

  30. git clone —recursive https://github.com/sbates130272/donard.git

  31. cd to donard/nvme_donard directory

  32. Edit Makefile to point to correct nvidia src directory

  33. Run make install as root

  34. lsmod |grep nvme_donard

  35. If module not visible then

  36. modprobe nvme_donard

  37. Build and run tests

  38. May have to make modules followed by make install

  39. cd donard/libargconfig

  40. Execute ./waf , may have to ./waf install

  41. cd to donard/libdonard

  42. apt-get install libfftw3-dev libmagickwand-dev

  43. ./waf

  44. modprobe donard_nv_pinbuf

  45. mkdir /temp

  46. mkfs.ext4 /dev/nvme0n1

  47. mount /dev/nvme0n1 /temp

  48. dd if=/dev/zero of=/temp/test1.dat bs=1K count=100K

  49. cd donard/libdonard/build/speed

  50. ./nvme2gpu_read -b 128M -D /temp/test1.dat

  51. ./nvme2gpu_read -b 128M /temp/test1.dat

Example output

~/donard/libdonard/build/speed# ./nvme2gpu_read -b 128M /temp/test1.dat Total CPU Time: 0.0s user, 0.9s system Page Faults: 9 Copied 104.86MB in 348.6 ms 300.78MB/s

~/donard/libdonard/build/speed# ./nvme2gpu_read -b 128M -D /temp/test1.dat Total CPU Time: 0.1s user, 0.9s system Page Faults: 1607

Copied 104.86MB in 80.9 ms 1.30GB/s

Quick Start - NVMe<->GPU

  1. cd /nvme_donard
  2. make install (this loads the nvme_donard kernel module and blacklists the defult nvme one).
  3. lsmod | grep nvme should return nvme_donard (and not nvme).
  4. cd /libdonard
  5. ./waf
  6. cd /libdonard/build/speed
  7. dd if=/dev/zero of=/<nvme_drive>/test1.dat bs=1K count=100K (to create a test file, for now keep it at 128MB or less)
  8. ./nvme2gpu_read -b 128M -D /<nvme_drive>/test1.dat
  9. ./nvme2gpu_read -b 128M =/<nvme_drive>/test1.dat

If all of this runs you should see the –D mode has WAY more page faults the without the –D (the non –D is p2p, with –D the transfer goes via DRAM). Depending on your system the non –D option may be faster too. This is nvme->gpu transfer. There is a similar executable in the same fodler to go the other way. You can use likwid-perfctr to get better memory and CPU measurements too.

Quick Start - perform.sh

A simple bash script resides in the perform folder. When run it checks for certain files on the path and then executes a client/server perftest test whilst tracking DRAM bandwidth on the server.

Quick Start - NVDIMMs

We have some code in here to test NVDIMMs and the IOMEM exposed in the PMC Flashtec NVRAM drive. You can also use it to test any memory region really. Anyway here are some steps for the NVDIMM...

  1. git clone --recursive https://github.com/sbates130272/donard.git pulls the code.
  2. cd donard/libargconfig
  3. sudo ./waf install
  4. cd ../..
  5. cd donard/nvram_bench
  6. ./waf (builds the code, should be no errors).
  7. Since our kernel has the PMEM+DAX patches we can setup the NVDIMM but adding the following line to /etc/modules: pmem pmem_start_gb=8 pmem_size_gb=8
  8. The NVDIMM appears as a /dev/pmem<num> and we can mount it using the following in /etc/fstab: /dev/pmem<num> /mnt/nvdimm ext4 dax,noatime 0 0
  9. You can now run the nvdimm.sh script on the nvdimm.

Quick Start - RDMA

For these tests you will need two machines (a server and a client) and the donard kernel and OFED drivers on both machines. Each machine will also need a OFED compliant NIC installed. We've done some testing on both the Chelsio T540-CR iWARP card and the Mellanox MT27600 IB card.

Server

  1. git clone --recursive https://github.com/sbates130272/donard.git pulls the code.
  2. cd donard/donard_rdma
  3. ./waf
  4. Run the server (three modes available): i. ./build/donard_rdma_server (runs in main memory) ii. ./build/donard_rdma_server -g (runs in a bar on GPU) iii. ./build/donard_rdma_server -m (runs in a mmap of the specified file. If that file is on a NVRAM card it will use that and do p2p).
  5. Run the client (see below). When the client runs you should see somethng like the following on the server:

Buffer Type: CPU Listening on port 11935

Buffer Created: 0x7fa8040a6010 length 1024kB Accepting Client Connection: 172.16.0.2 Testing Send/Recv Send Completed Successfully. Recv Completed Succesfully. Got Seed 220246839, length 32768, use_zeros 0 Buffer Matches Random Seed. Client Disconnected.

Client

  1. git clone --recursive https://github.com/sbates130272/donard.git pulls the code.
  2. cd donard/donard_rdma
  3. ./waf
  4. ./build/donard_rdma_client -a -w (note address will be system specific and in our case is donard-rdma)
  5. If things work as expected you should get something like:

batesste@cgy1-flash:/donard/donard_rdma$ donard_rdma_client -a donard-rdma Seed: 1422379394 rdma_connect: Connection refused batesste@cgy1-flash:/donard/donard_rdma$ donard_rdma_client -a donard-rdma -w Seed: 1422379414 Remote Buffer: 0x7fa8040a6010 : length 1024KiB : bs = 32768B Testing Send/Recv Recv Completed Succesfully. Send Completed Successfully.

Testing Writes

Wrote: 8MiB Transfered: 8.42MB in 0.0s 281.21MB/s

References

A good place to get started is the Flash Memory Summit 2014 paper (which can be found in the papers subfolder) that discusses Donard. Another reference is the article on PMC-Sierra's blog

Licensing

This code is licensed under Apache Version 2.0 and, where required, GPL Version 2.0