Multi-GPU inference

[MihinP]

Add Multi-GPU Inference Support

Overview

Enables multi-GPU inference for improved throughput when multiple CUDA devices are available.

Changes

• New BatchedLoadImages: "Batched" (distributed GPU) image loader to replace single image loading
• New MultiGPUModel: Wrapper class that distributes inference across multiple GPUs
• Enhanced select_multi_device(): Improved device selection with multi-GPU support. Previously would only return the first available GPU
• Updated detect() function: Modified to use new multi-GPU infrastructure

Key Features

• Automatic detection and utilization of all available GPUs
• Thread pool executor for parallel inference across GPU devices
• Preserved original functionality for single GPU/CPU usage

Limitations

• Batch processing limitation: Current YOLO model requires single image inference due to tensor boolean operations in forward pass
• Sequential processing: Despite batching infrastructure, images are processed individually per GPU
• Future work needed: YOLO model forward method needs updates to support true batched inference

Performance

• Linear speedup based on number of available GPUs
• Added timing metrics for batch processing

Praise Claude

[edoyango]

Good work! Do you need me to help with anything here? It looks like it should work, but I don't have the infrastructure to test it :/

[MihinP]

Should be all good, I'll test on Milton once deployed but currently works on my branch. I'll merge the request. Also the manuscript is on its way, chuck me an email at perera.m@wehi.edu.au and I'll share it.