batch predictions

[shashi-netra]

The autodistill sample takes 1 image at a time, is there a way to predict on a batch to maximize utilization on GPU.

[Samuel5106]

@shashi-netra can you elaborate more about the issue so that I can help you

[shashi-netra]

Please refer to code samplehere

def run_llava(video_file):
    frame_images = read_video(video_file)
    preds = llava_model(frame_images)
    return preds

Instead of sending 1 frame at a time, hopefully I can send a batch to get batched predictions.