Skip to content

Guide to GPUDrive setup on NYU HPC

Jump to bottom
Daphne Cornelisse edited this page Dec 25, 2024 · 59 revisions

🧱 Installation (first time only)

Step 1: Clone the repository

Clone the gpudrive repository into your /home/$USER directory (info on HPC directories and data management).

git clone --recursive https://github.com/Emerge-Lab/gpudrive.git

Step 2: Navigate to repository

Move into the cloned repository folder:

cd gpudrive

Step 3: Set up overlay image

  • Create a directory for overlay files:
mkdir -p /home/$USER/gpudrive/integrations/hpc
cd /home/$USER/gpudrive/integrations/hpc
  • Copy and decompress the overlay image:
cp /scratch/work/public/overlay-fs-ext3/overlay-10GB-400K.ext3.gz .
gunzip overlay-10GB-400K.ext3.gz

This may take a couple of minutes, depending on the size of the overlay image.

Note: Other overlay images should work fine as well.

    1. Verify the decompressed overlay image exists:
ls /home/$USER/gpudrive/integrations/hpc

Step 4: Launch Singularity container

Run the following to start the container with GPU support and the overlay image:

cd ../..
singularity exec --nv --overlay /home/$USER/gpudrive/integrations/hpc/overlay-10GB-400K.ext3:rw \
/scratch/work/public/singularity/cuda12.2.2-cudnn8.9.4-devel-ubuntu22.04.3.sif /bin/bash

you will see:

Singularity>

Details on Sinularity and overlay images on NYU HPC here.

Step 5: Request a GPU

srun --nodes=1 --tasks-per-node=1 --cpus-per-task=1 --mem=10GB --gres=gpu:1 \
--time=1:00:00 --account=<ASK> --pty /bin/bash

Ouput:

>>> srun: job 55367179 queued and waiting for resources
>>> srun: job 55367179 has been allocated resources

You will see something like:

[08:33:52 Wed Dec 25 2024] username@ga021.hpc.nyu.edu ~/gpudrive

Ask Eugene for your account code.

Step 6: Set up Python environment

  • Inside the Singularity container, create a virtual environment:

  • One-off step: Create conda environment with Python 3.10

conda create --name gpudrive python=3.10
  • Activate conda environment conda activate gpudrive

Now you will see:

(/scratch/username/.conda/gpudrive) Singularity>

Step 7: Set up GPUDrive

For details, see the GPUDrive README. Here we use the manual install option.

  • Build C++ codebase: Inside the gpudrive folder, run:
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j 32 # cores to build with, e.g. 32
cd ..

if successful, you'll see

[100%] Linking CXX executable my_tests
[100%] Built target my_tests
  • Setup Python components
pip install -e .

Step 8: Verify installation

# Launch Python
python3
# Check GPUDrive import
>>> import gpudrive

If there are no errors, the installation is successful.

⚡️ Usage

  1. Request an interactive compute node, e.g:
srun --nodes=1 --tasks-per-node=1 --cpus-per-task=1 --mem=10GB --gres=gpu:1 \
--time=1:00:00 --account=<account_number> --pty /bin/bash

Replace <account_number> with the your project number.

  1. Navigate to repository:
cd /home/$USER/gpudrive
  1. Launch the Singularity image:
singularity exec --nv --overlay /home/$USER/gpudrive/integrations/hpc/overlay-10GB-400K.ext3:rw \
/scratch/work/public/singularity/cuda12.2.2-cudnn8.9.4-devel-ubuntu22.04.3.sif /bin/bash
  1. Activate the virtual environment:
conda activate gpudrive
  1. Run experiments!
python baselines/ippo/ippo_pufferlib.py

Clone this wiki locally