Skip to content

Guide to GPUDrive setup on NYU HPC

Jump to bottom
Daphne Cornelisse edited this page Dec 25, 2024 · 59 revisions

🧱 Installation (first time only)

Step 1: Clone the repository

Clone the gpudrive repository into your /home/$USER directory (info on HPC directories and data management).

git clone --recursive https://github.com/Emerge-Lab/gpudrive.git

Step 2: Navigate to repository

Move into the cloned repository folder:

cd gpudrive

Step 3: Set up overlay image

    1. Create a directory for overlay files:
mkdir -p /home/$USER/gpudrive/integrations/hpc
cd /home/$USER/gpudrive/integrations/hpc
    1. Copy and decompress the overlay image:
cp /scratch/work/public/overlay-fs-ext3/overlay-10GB-400K.ext3.gz .
gunzip overlay-10GB-400K.ext3.gz

This may take a couple of minutes, depending on the size of the overlay image.

Note: There are other overlay images that should work fine as well.

    1. Verify the decompressed overlay image exists:
ls /home/$USER/gpudrive/integrations/hpc

Step 4: Launch Singularity container

Run the following to start the container with GPU support and the overlay image:

cd ../..
singularity exec --nv --overlay /home/$USER/gpudrive/integrations/hpc/overlay-10GB-400K.ext3:rw \
/scratch/work/public/singularity/cuda12.2.2-cudnn8.9.4-devel-ubuntu22.04.3.sif /bin/bash

you will see:

Singularity>

Details on Sinularity and overlay images on NYU HPC here.

Step 5: Set up Python environment

  1. Inside the Singularity container, create a virtual environment:
  • One-off step: Create conda environment with Python 3.10
conda create --name gpudrive python=3.10
  • Activate conda environment conda activate gpudrive

Now you will see:

(/scratch/username/.conda/gpudrive) Singularity>

Step 6: Build GPUDrive

For detailed instructions, see the GPUDrive README. Here we use the manual install option.

  1. Build C++ codebase: Inside the gpudrive folder, do:
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j 32 # cores to build with, e.g. 32
cd ..

Step 7: Verify installation

# Launch Python
python
# Check GPUDrive import
>>> import gpudrive

If there are no errors, the installation is successful.

⚡️ Usage

  1. Request an interactive compute node:

    srun --nodes=1 --tasks-per-node=1 --cpus-per-task=1 --mem=10GB --gres=gpu:1 \
--time=1:00:00 --partition=<gpu_partition> --pty /bin/bash

    Replace <gpu_partition> with the appropriate partition (e.g., short, gpu).

  2. Navigate to the repository:

    cd /home/$USER/gpudrive

  3. Launch the Singularity image:

    ...

  4. Activate the virtual environment:

    source .venv/bin/activate

Clone this wiki locally