RED: Improving I/O Efficiency of Model Training Through File Redirection.

Install prerequisites

• Python 3.8
• Pytorch 1.7.0
• CUDA 11.2
• CMAKE 3.18.0
• GCC 7.5.0
• libconfig
• gRPC

Compile the system

Compile the server of RED

 mkdir libRED/server/build/
 cd libRED/server/build/
 cmake ../
 make

Compile the client of RED

mkdir libRED/client/build/
cd libRED/client/build/
cmake ../
make

Run the System

Setting Up Configuration Information

Before running the system, you'll need to set up the configuration parameters according to your training requirements. Follow these steps:

cd  libRED/configure/
vim grpc_configure.cfg

Model Training

Based on RED, training a model is not much different from training a model using native PyTorch. Simply execute the following command:

source run.sh

This script will initiate the system and run it according to the specified training configuration, dataset, and model information. You can set these parameters in the run.sh file, and then simply execute the above command to start running the system.

It's worth noting that RED has two special parameters:

• chunk-size: Specifies the size of the data chunk used for this training, representing the number of training data contained in each data chunk.
• cache-ratio: Specifies the ratio of dataset to be cached in memory for this training.