Guide to GPUDrive setup on NYU HPC

🧱 Installation (first time only)

Step 1: Clone the repository

Clone the gpudrive repository into your /home/$USER directory (info on HPC directories and data management).

git clone --recursive https://github.com/Emerge-Lab/gpudrive.git

Step 2: Navigate to repository

Move into the cloned repository folder:

cd gpudrive

Step 3: Set up overlay image

• Create a directory for overlay files:

mkdir -p /home/$USER/gpudrive/integrations/hpc
cd /home/$USER/gpudrive/integrations/hpc

• Copy and decompress the overlay image:

cp /scratch/work/public/overlay-fs-ext3/overlay-10GB-400K.ext3.gz .
gunzip overlay-10GB-400K.ext3.gz

This may take a couple of minutes, depending on the size of the overlay image.

Note: Other overlay images should work fine as well.

• 3. Verify the decompressed overlay image exists:

ls /home/$USER/gpudrive/integrations/hpc

Step 4: Launch Singularity container

Run the following to start the container with GPU support and the overlay image:

cd ../..

singularity exec --nv --overlay /home/$USER/gpudrive/integrations/hpc/overlay-10GB-400K.ext3:rw \
/scratch/work/public/singularity/cuda12.2.2-cudnn8.9.4-devel-ubuntu22.04.3.sif /bin/bash

you will see:

Singularity> 

---

Details on Sinularity and overlay images on NYU HPC here.

---

Step 5: Request a GPU

srun --nodes=1 --tasks-per-node=1 --cpus-per-task=1 --mem=10GB --gres=gpu:1 \
--time=1:00:00 --account=<ASK> --pty /bin/bash

Ouput:

>>> srun: job 55367179 queued and waiting for resources
>>> srun: job 55367179 has been allocated resources

You will see something like:

[08:33:52 Wed Dec 25 2024] username@ga021.hpc.nyu.edu ~/gpudrive

Ask Eugene for your account code.

Step 6: Set up Python environment

• Inside the Singularity container, create a virtual environment:

• One-off step: Create conda environment with Python 3.10

conda create --name gpudrive python=3.10

Why use conda? Because currently, the only way to use a Python version > 3.8.6 on the cluster without Docker is with conda.

• Activate conda environment conda activate gpudrive

Now you will see:

(/scratch/username/.conda/gpudrive) Singularity>

Step 7: Set up GPUDrive

We use the manual install option to set up GPUDrive, see the readme for details.

• Build C++ codebase: Inside the gpudrive folder, run:

mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j 32 # cores to build with, e.g. 32
cd ..

if successful, you'll see

[100%] Linking CXX executable my_tests
[100%] Built target my_tests

• Setup Python components

pip install -e . 

Step 8: Verify installation

# Launch Python
python3
# Check GPUDrive import
>>> import gpudrive

If there are no errors, the installation is successful!

⚡️ Usage | Interactive node

1. Request an interactive compute node, e.g:

srun --nodes=1 --tasks-per-node=1 --cpus-per-task=1 --mem=10GB --gres=gpu:1 \
--time=1:00:00 --account=<account_number> --pty /bin/bash

Replace <account_number> with the your project number.

2. Navigate to repository:

cd /home/$USER/gpudrive

3. Launch the Singularity image:

singularity exec --nv --overlay /home/$USER/gpudrive/integrations/hpc/overlay-10GB-400K.ext3:rw \
/scratch/work/public/singularity/cuda12.2.2-cudnn8.9.4-devel-ubuntu22.04.3.sif /bin/bash

4. Activate the virtual environment:

conda activate gpudrive 

5. Run experiments!

e.g.,

python baselines/ippo/ippo_pufferlib.py

🚀 Usage | sbatch

1. Define run configurations and hyperparameters to sweep over in generate_sbatch.py and run it. This stores an sbatch script for you in LOG_FOLDER.

2. Submit the jobs using

sbatch hpc/run_scripts/<your_file>.sh