Guide to GPUDrive setup on NYU HPC

Installation (First Time Only)

Step 1: Clone the repository

Clone the gpudrive repository into your /scratch/$USER directory.

git clone --recursive https://github.com/Emerge-Lab/gpudrive.git

Step 2: Navigate to repository

Move into the cloned repository folder:

cd gpudrive

Step 3: Set up overlay image

• 1. Create a directory for overlay files:

mkdir -p /scratch/$USER/image
cd /scratch/$USER/image

• 2. Copy and decompress the overlay image:

cp /scratch/work/public/overlay-fs-ext3/overlay-50G-10M.ext3.gz .
gunzip overlay-50G-10M.ext3.gz

Note: There are other overlay images that should work fine as well.

• 3. Verify the decompressed overlay image exists:

ls /scratch/$USER/image

Step 4: Launch Singularity container

Run the following to start the container with GPU support and the overlay image:

singularity exec --nv --overlay /scratch/$USER/image/overlay-50G-10M.ext3:rw \
/scratch/work/public/singularity/cuda12.2.2-cudnn8.9.4-devel-ubuntu22.04.3.sif /bin/bash

---

More details about steps 3 and 4 here.

---

Step 5: Set up Python environment

1. Inside the Singularity container, create a virtual environment:

python3 -m venv .venv

2. Activate the virtual environment:

source .venv/bin/activate

Step 6: Install GPUDrive

For detailed instructions, see the GPUDrive README. Here we use the manual install option.

1. Build C++ codebase:

mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j # cores to build with, e.g. 32
cd ..

error

(.venv) Singularity> cmake .. -DCMAKE_BUILD_TYPE=Release
CMake Error at CMakeLists.txt:4 (include):
  include could not find requested file:

    /home/dc4971/gpudrive/external/madrona/cmake/madrona_init.cmake


-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:8 (include):
  include could not find requested file:

    setup


CMake Error at CMakeLists.txt:9 (include):
  include could not find requested file:

    dependencies






CMake Error at external/CMakeLists.txt:1 (set):
  Maximum recursion depth of 1000 exceeded


-- Configuring incomplete, errors occurred!
See also "/home/dc4971/gpudrive/build/CMakeFiles/CMakeOutput.log".

Step 7: Verify Installation

1. Exit the Singularity container:

exit

2. Relaunch the container using the same command as in Step 4.

3. Inside the container, activate the virtual environment:

source .venv/bin/activate

4. Verify the GPUDrive installation:

# Launch Python
python
# Check GPUDrive import
>>> import gpudrive

If there are no errors, the installation is successful.

Usage

1. Request an interactive compute node:

   srun --nodes=1 --tasks-per-node=1 --cpus-per-task=1 --mem=10GB --gres=gpu:1 \
   --time=1:00:00 --partition=<gpu_partition> --pty /bin/bash

   Replace <gpu_partition> with the appropriate partition (e.g., short, gpu).

2. Navigate to the repository:

   cd /scratch/$USER/gpudrive

3. Launch the Singularity image:

   bash ./scripts/hpc/launch_image.sh

4. Activate the virtual environment:

   source .venv/bin/activate

---

Navigate to the repo:

cd /scratch/$USER/gpudrive

Launch the Singularity image:

bash ./scripts/hpc/launch_image.sh

Activate the Python virtual environment:

source .venv/bin/activate