TRIMS (MXNet with Persistent GPU Memory)

Reference

Dakkak, A., Li, C., De Gonzalo, S. G., Xiong, J., & Hwu, W-M. W. (2019). TrIMS: Transparent and isolated model sharing for low latency deep learning inference in function-as-a-service. In E. Bertino, C. K. Chang, P. Chen, E. Damiani, M. Goul, & K. Oyama (Eds.), Proceedings - 2019 IEEE International Conference on Cloud Computing, CLOUD 2019 - Part of the 2019 IEEE World Congress on Services (pp. 372-382). [8814494] (IEEE International Conference on Cloud Computing, CLOUD; Vol. 2019-July). IEEE Computer Society. https://doi.org/10.1109/CLOUD.2019.00067

Installation

Requirements

Setup your target location (make sure that LD_LIBRARY_PATH and PATH has been updated to include this)

export UPR_INSTALL_PREFIX=$HOME/.usr
export UPR_BASE_DIR=$HOME/carml/data/mxnet

make sure that both directories exist

mkdir -p $UPR_INSTALL_PREFIX
mkdir -p $UPR_BASE_DIR

Remember to update your PKG_CONFIG_PATH

export PKG_CONFIG_PATH=$UPR_INSTALL_PREFIX/lib/pkgconfig/:$PKG_CONFIG_PATH

C-ARES

wget https://c-ares.haxx.se/download/c-ares-1.13.0.tar.gz
tar -xf c-ares-1.13.0.tar.gz
cd c-ares-1.13.0
./buildconf
./configure --prefix=$UPR_INSTALL_PREFIX --disable-dependency-tracking --disable-debug
make install

Protobuf

wget https://github.com/google/protobuf/archive/v3.5.1.tar.gz
tar -xf v3.5.1.tar.gz
cd protobuf-3.5.1
./autogen.sh
./configure --prefix=$UPR_INSTALL_PREFIX --disable-dependency-tracking --disable-debug --with-zlib
make
make install

Protobuf-c

wget https://github.com/protobuf-c/protobuf-c/releases/download/v1.3.0/protobuf-c-1.3.0.tar.gz
tar -xf protobuf-c-1.3.0.tar.gz
cd protobuf-c-1.3.0
./configure --prefix=$UPR_INSTALL_PREFIX --disable-dependency-tracking --disable-debug
make install

GRPC

wget https://github.com/grpc/grpc/archive/v1.9.1.tar.gz
tar -xf v1.9.1
cd v1.9.1
make install prefix=$UPR_INSTALL_PREFIX
make install-plugins prefix=$UPR_INSTALL_PREFIX

OpenCV (Optional)

wget https://github.com/opencv/opencv/archive/3.4.1.tar.gz
tar -xf 3.4.1.tar.gz
cd opencv-3.4.1/
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=$UPR_INSTALL_PREFIX -DWITH_CUDA=OFF -DWITH_OPENCL=OFF ..
make
make install

From APT-GET

Other requirements can be installed using APT . The base requirements are listed in the (MXNet installation guide)[https://mxnet.apache.org/install/index.html].

Regenerate GRPC Code

It is not recommended to regenerate the protofile, since it's dependent on the protobuf version. The only time you need to regenerate it is after updating it.

cd src/c_api
make

Build Server

The server is part of the MXNet build process.

Downloading Models

Running

Server

Client

Profiling

Environment Variables

Name	Description	Default Value
UPR_ENABLED		true
UPR_PROFILE_IO	only makes sense if UPR_ENABLED=false	true
UPR_RUN_ID		[undefined]
git		build_git_sha
UPR_GIT_BRANCH		build_git_branch
UPR_GIT_TIME		build_git_time
UPR_CLIENT
UPR_BASE_DIR
UPR_MODEL_NAME
UPR_PROFILE_TARGET		profile.json
UPR_INITIALIZE_EAGER		false
UPR_INITIALIZE_EAGER_ASYNC		false
UPR_INPUT_CHANNELS		3
UPR_INPUT_WIDTH		224
UPR_INPUT_HEIGHT		224
UPR_INPUT_MEAN_R		0
UPR_INPUT_MEAN_G		0
UPR_INPUT_MEAN_B		0
UPR_ENABLE_MEMORY_PROFILE		false
UPR_ENABLE_CUDA_FREE		false
UPR_SHARING_GRANULARITY		model
--------------------------	-----------	-------------
UPRD_EVICTION_POLICY		LRU
UPRD_ESTIMATION_RATE		1.0
UPRD_MEMORY_PERCENTAGE		0.8
UPRD_PERSIST_CPU		true
UPRD_PERSIST_ONLY_CPU	only persist on cpu memory	false
UPRD_WRITE_PROFILE	write server profile file	false
UPRD_ESTIMATE_WITH_INTERNAL_MEMORY	use internal memory info for estimate	true