Guide to GPUDrive setup on NYU HPC

🧱 Installation (first time only)

Step 1: Clone the repository

Clone the gpudrive repository into your /home/$USER directory (info on HPC directories and data management).

git clone --recursive https://github.com/Emerge-Lab/gpudrive.git

Step 2: Navigate to repository

Move into the cloned repository folder:

cd gpudrive

Step 3: Set up overlay image

• 1. Create a directory for overlay files:

mkdir -p /home/$USER/gpudrive/integrations/hpc
cd /home/$USER/gpudrive/integrations/hpc

• 2. Copy and decompress the overlay image:

cp /scratch/work/public/overlay-fs-ext3/overlay-10GB-400K.ext3.gz .
gunzip overlay-10GB-400K.ext3.gz

This may take a couple of minutes, depending on the size of the overlay image.

Note: There are other overlay images that should work fine as well.

• 3. Verify the decompressed overlay image exists:

ls /home/$USER/gpudrive/integrations/hpc

Step 4: Launch Singularity container

Run the following to start the container with GPU support and the overlay image:

cd ../..

singularity exec --nv --overlay /home/$USER/gpudrive/integrations/hpc/overlay-10GB-400K.ext3:rw \
/scratch/work/public/singularity/cuda12.2.2-cudnn8.9.4-devel-ubuntu22.04.3.sif /bin/bash

you will see:

Singularity> 

---

Details on Sinularity and overlay images on NYU HPC here.

---

Step 5: Request a GPU

srun --nodes=1 --tasks-per-node=1 --cpus-per-task=1 --mem=10GB --gres=gpu:1 \
--time=1:00:00 --account=<ASK> --pty /bin/bash

>>> srun: job 55367179 queued and waiting for resources
>>> srun: job 55367179 has been allocated resources

You will see something like:

[08:33:52 Wed Dec 25 2024] username@ga021.hpc.nyu.edu ~/gpudrive

Ask Eugene for your account code.

Step 6: Set up Python environment

1. Inside the Singularity container, create a virtual environment:

• One-off step: Create conda environment with Python 3.10

conda create --name gpudrive python=3.10

• Activate conda environment conda activate gpudrive

Now you will see:

(/scratch/username/.conda/gpudrive) Singularity>

Step 7: Setup GPUDrive

For details, see the GPUDrive README. Here we use the manual install option.

1. Build C++ codebase: Inside the gpudrive folder, do:

mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j 32 # cores to build with, e.g. 32
cd ..

if successful, you'll see

[100%] Linking CXX executable my_tests
[100%] Built target my_tests

2. Setup Python components

pip install -e . # Add -Cpackages.madrona_escape_room.ext-out-dir=PATH_TO_YOUR_BUILD_DIR on Windows

Step 8: Verify installation

# Launch Python
python
# Check GPUDrive import
>>> import gpudrive

If there are no errors, the installation is successful.

⚡️ Usage

1. Request an interactive compute node:

   srun --nodes=1 --tasks-per-node=1 --cpus-per-task=1 --mem=10GB --gres=gpu:1 \
   --time=1:00:00 --partition=<gpu_partition> --pty /bin/bash

   Replace <gpu_partition> with the appropriate partition (e.g., short, gpu).

2. Navigate to the repository:

   cd /home/$USER/gpudrive

3. Launch the Singularity image:

   ...

4. Activate the virtual environment:

   source .venv/bin/activate