libccp

libccp is an implementation of the core functionality necsesary for a datapath to communicate with a CCP process. The datapath is responsible for providing a few callback functions for modifying state internal to the datapath (e.g. congestion window or packet pacing rate) and a few utility functions and libccp handles everything else. The instructions below detail all of the steps necessary to make a datapath CCP compatible.

If you do not need to create a new datapath and just want to get started using CCP, please see our guide.

Build

The simplest way to build is to simply run make. By default, we build both a shared and static version of the library.

cmake

If you want to use cmake, run cmake <path_to_libccp> from within a separate build directory, then run make and (optionally) make install (which copies the libraries, public header files, and target lists to an install prefix). If you do not specify anything, it will use a system default install prefix, such as /usr/lib and /usr/include. To use a different install prefix, add -DCMAKE_INSTALL_PREFIX=... when you run cmake (but then run make and make install as normal).

For example, you could do (from within the libccp root directory):

mkdir build/
cd build/
cmake -DCMAKE_INSTALL_PREFIX=~/test_install ../ # runs cmake on the libccp root directory but places the build files in the current libccp/build directory
make # builds the libraries
make install # copies all the files to ~/test_install/...

To do an equivalent "clean" of all cmake-generated files, simply delete the build directory.

Supporting New Datapaths

Follow these steps to add ccp support to a new datapath.

Implementation

0 | Include ccp.h in all relevant files

In C source files:

#include "libccp/ccp.h"

In C++ source files:

extern "C" {
#include "libccp/ccp.h"
}

1 | Initialization Global (ccp_init and ccp_free)

In an initialize / register function called once (not per connection), we need to provide pointers to 6 callback/utility functions for libccp to invoke (we'll come back to the implementation of them later). All of the functions must be implemented or libccp will throw an error. There is a single optional impl field, which is a void* that can be used to retain a reference to global datapath state, which will be passed as a parameter to each of the callback functions. For example, in the mtcp datapath, impl is a pointer to the global mtcp context structure, which holds the unix sockets necessary for communicating with the CCP. The send_msg function is a callback invoked by the ccp that must have access to these sockets.

(Note in this example e.g. _set_cwnd is the datapath's implementation of the set_cwnd function.)

struct ccp_datapath dp = {
	.set_cwnd = &_set_cwnd,
        .set_rate_abs = &_set_rate_abs,
        .now = &_now,
        .after_usecs = &_after_usecs
        .send_msg = &_send_msg,
        .impl = // pointer to anything
};

ok = ccp_init(&dp);
if (ok < 0) {
	return -1;
}

Be sure to call ccp_free in a destructor as well:

ccp_free();

2 | Initialize Connection (ccp_connection_start and ccp_connection_free)

When a new connection is created, the datapath must call ccp_connection_start, This function again takes a void* which can be used to store datapath-specific per-connection state (e.g. the linux kernel datapath uses this field to store a reference to the corresponding struct sock) and returns a pointer to a struct ccp_connection. This should be stored somewhere for later access. It will be necessary for accessing the impl and the deconstructor at the end.

On connection start:

struct sock sk;
...
conn = ccp_connection_start((void *) sk);
if (conn == NULL) {
  // connection failed
} else {
  // connection successful, has index dp->index
}
// save reference to conn somewhere

When connection ends:

// need reference to conn from above

if (conn != NULL) {
	ccp_connection_free(conn->index);
} else {
	// already freed
}

Given a reference to the struct ccp_connection, the impl field can be accessed and casted like so (e.g. unboxing a struct sock *)

struct sock *sk;
*sk = (struct sock *) ccp_get_impl(dp);

3 | Implement control functions

Now it's time to implement the functions from Step 1. The function signatures and relevant details can all be found in libccp/ccp.h. Check out the ccp-kernel or ccp-mtcp repositories for specific examples of how these functions might be implemented.

Make sure the names of these functions match what you provided to ccp_init.

4 | Implement measurement

On each ACK received, you must set all of the fields in conn->prims accordingly with the measurements for the ACK, and then call ccp_invoke(conn). Libccp will use this to update its internal state and occasionally send this to the ccp. As the datapath, you don't need to be concerned with when this happens, as libccp handles all of this using the send_msg function from (3).

struct ccp_connection *conn;
// ... get access to conn for this connection
struct ccp_primitives *mmt = &conn->prims

mmt->bytes_acked =      // ...
mmt->bytes_misordered = // ... 
...

ccp_invoke(conn);
conn->prims.was_timeout = false;

prims.was_timeout is by default set to false. Whenver the datapath suspects there has been a drop, this field should be set to true. Just be sure to set it back to false again after calling ccp_invoke (as above) to make sure that the same signal is not handled twice. Again libccp is responsible for when it communicates this information to the ccp.

Putting it all together

Now you should be ready to build everything. The following is for userspace datapaths; for kernel datapaths, see https://github.mit.edu/nebula/ccp-kernel.

0 | Build libccp

Simply run make in the top level of this repository. This will produce the shared library libccp.so. You can leave it here, or move it to a more standard location (e.g. /usr/lib). Either way, be sure to note the path, it will be used as LIBCCP in the following step.

Important Note: If you intend to link libccp with C++ code, you must build libccp with a C++ compiler (e.g. g++) rather than gcc. Change the first line of the Makefile to CC=g++ and recompile.

1 | Link libccp

Add the following to your project Makefile. This will link against libccp and ensure the compiler knows where to find the necessary header files.

• be sure to set /path/to/libccp appropriately
• if LIBS and INC are already defined, change = to +=

LIBCCP = /path/to/libccp
LIBS = -L$(LIBCCP) -lccp
INC = -I$(LIBCCP)

If LIBS and INC are not already included, be sure to add them to your compile comands, e.g...

file.o : file.c
    $(CC) -c file.c $(INC)
exe : file.o
	$(CC) file.o $(INC) $(LIBS) -o exe

2 | Build your application

Just run make.

Before running your application, you need to ensure that the path to libccp is included in the LD_LIBRARY_PATH environment variable so that your application knows where to find the library at run time.

For example, if its stored at /home/ubuntu/libs/libccp.so, you can append as follows

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/ubuntu/libs