Guide to GPUDrive setup on NYU HPC

⚡️ Usage

TODO

🧱 Installation (first time only)

Step 1: Clone the repository

Clone the gpudrive repository into your /home/$USER directory (info on HPC directories and data management).

git clone --recursive https://github.com/Emerge-Lab/gpudrive.git

Step 2: Navigate to repository

Move into the cloned repository folder:

cd gpudrive

Step 3: Set up overlay image

• 1. Create a directory for overlay files:

mkdir -p /home/$USER/gpudrive/integrations/hpc
cd /home/$USER/gpudrive/integrations/hpc

• 2. Copy and decompress the overlay image:

cp /scratch/work/public/overlay-fs-ext3/overlay-10GB-400K.ext3.gz .
gunzip overlay-10GB-400K.ext3.gz

This may take a couple of minutes, depending on the size of the overlay image.

Note: There are other overlay images that should work fine as well.

• 3. Verify the decompressed overlay image exists:

ls /home/$USER/gpudrive/integrations/hpc

Step 4: Launch Singularity container

Run the following to start the container with GPU support and the overlay image:

cd ../..

singularity exec --nv --overlay /home/$USER/gpudrive/integrations/hpc/overlay-10GB-400K.ext3:rw \
/scratch/work/public/singularity/cuda12.2.2-cudnn8.9.4-devel-ubuntu22.04.3.sif /bin/bash

you will see:

Singularity> 

---

Details on Sinularity and overlay images on NYU HPC here.

---

Step 5: Set up Python environment

1. Inside the Singularity container, create a virtual environment:

python3 -m venv .venv

2. Activate the virtual environment:

source .venv/bin/activate

Step 6: Build GPUDrive

For detailed instructions, see the GPUDrive README. Here we use the manual install option.

1. Build C++ codebase: Inside the gpudrive folder, do:

mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j 32 # cores to build with, e.g. 32
cd ..

error

-- Including X11 support
-- Found X11: /usr/include   
-- Using the multi-header code from /home/dc4971/gpudrive/external/json/include/
CMake Error at src/CMakeLists.txt:65 (madrona_python_module):
  Unknown CMake command "madrona_python_module".


-- Configuring incomplete, errors occurred!
See also "/home/dc4971/gpudrive/build/CMakeFiles/CMakeOutput.log".

Step 7: Verify Installation

1. Exit the Singularity container:

exit

2. Relaunch the container using the same command as in Step 4.

3. Inside the container, activate the virtual environment:

source .venv/bin/activate

4. Verify the GPUDrive installation:

# Launch Python
python
# Check GPUDrive import
>>> import gpudrive

If there are no errors, the installation is successful.

Usage

1. Request an interactive compute node:

   srun --nodes=1 --tasks-per-node=1 --cpus-per-task=1 --mem=10GB --gres=gpu:1 \
   --time=1:00:00 --partition=<gpu_partition> --pty /bin/bash

   Replace <gpu_partition> with the appropriate partition (e.g., short, gpu).

2. Navigate to the repository:

   cd /scratch/$USER/gpudrive

3. Launch the Singularity image:

   bash ./scripts/hpc/launch_image.sh

4. Activate the virtual environment:

   source .venv/bin/activate

---

Navigate to the repo:

cd /scratch/$USER/gpudrive

Launch the Singularity image:

bash ./scripts/hpc/launch_image.sh

Activate the Python virtual environment:

source .venv/bin/activate