This repository cointains the source code for fastswap (drivers), the far memory server (farmemserver) and the Linux kernel patches that fastswap needs (kernel). The kernel patch must be applied on top of Linux kernel 4.11.0.
We assume Ubuntu 16.04 is being used, but other distributions might work as well. Further, we assume this directory is cloned in your home directory as 'fastswap'.
We only support Mellanox NICs. The best supported NIC is ConnectX-3. However, if you find issues on ConnectX-4 or ConnectX-5, please let us know and we will try to address the issues.
First you need a copy of the source for kernel 4.11 with SHA a351e9b9fc24e982ec2f0e76379a49826036da12. We outline the high level steps here.
cd ~
wget https://github.com/torvalds/linux/archive/a351e9b9fc24e982ec2f0e76379a49826036da12.zip
mv a351e9b9fc24e982ec2f0e76379a49826036da12.zip linux-4.11.zip
unzip linux-4.11.zip
cd linux-4.11
git init .
git add .
git commit -m "first commit"
Now use the provided patch and apply it against your copy of linux-4.11, and use the generic Ubuntu config file for kernel 4.11. You can get the config file from internet, or you can use the one we provide.
git apply ~/fastswap/kernel/kernel.patch
cp ~/fastswap/kernel/config-4.11.0-041100-generic ~/linux-4.11/.config
Make sure you have necessary prerequisites to compile the kernel, and compile it:
sudo apt-get install git build-essential kernel-package fakeroot libncurses5-dev libssl-dev ccache bison flex
make -j `getconf _NPROCESSORS_ONLN` deb-pkg LOCALVERSION=-fastswap
Once it's done, your deb packages should be one directory above, you can simply install them all:
cd ..
sudo dpkg -i *.deb
We only support Mellanox OFED drivers, and we recommend version 4.1, 4.2 or 4.3. You must have an OFED installation working before proceeding. A good test before continuing is being able to run both ib_read_lat and ib_write_lat between your far memory node and your client. We assume the far memory node has ip $farmemip and the client has ip $clientip. Note these ips must be bound to RDMA NICs.
To build and run the far memory server do:
cd farmemserver
make
./rmserver 50000
You should see a message saying "listening on port 50000".
In order to activate the swap system in Linux, you must have a swap device
pre-registered. This swap device won't receive any data traffic, we only need
it to activate swap code paths in the kernel. Further, the amount of far memory
the client will be able to access is determined by the swap device size. So if
you want 32GB (default value) of far memory available, your swap device must
be exactly 32GB or less. When you type free in the terminal you must see
that Swap has 32GB of space available in total column.
To build the drivers:
cd drivers
make BACKEND=RDMA
If you have a custom installation path for the Mellanox OFED drivers, modify the OFA_DIR variable in the Makefile accordingly. The compilation will only succeed if you booted on a fastswap kernel.
Now we will load the fastswap drivers.
sudo insmod fastswap_rdma.ko sport=50000 sip="$farmemip" cip="$clientip" nq=8
sudo insmod fastswap.ko
sport is the port where the far memory server is running, sip is the far memory node ip, cip is this node ip (client) and nq must be set to the number of cpus available in the system. If you type dmesg and you see "ctrl is ready for reqs" then the connection was successful!
A good next step would be to try out our CFM framework: https://github.com/clusterfarmem/cfm
You can use the DRAM backend for experimentation. Compile and load as follows:
cd drivers
make BACKEND=DRAM
sudo insmod fastswap_dram.ko
sudo insmod fastswap.ko
You still need to have swap device enabled, but data won't flow there. By default the DRAM backend will allocate 32GB of memory.
For more information, please refer to our paper accepted at EUROSYS 2020
For additional questions please contact us at amaro a@t berkeley.edu