Is DetectionDataset intended behavior to store all images?

[David-rn]

Search before asking

• I have searched the Supervision issues and found no similar feature requests.

Question

Hi! 👋

I found that supervision was using >40GB of RAM in my system when using DectectionDataset to save a relatively large dataset as_yolo. I realized that the DectectionDataset uses a dictionary to map the image name to the image, which is stored in memory.

I wanted to ask if this is the expected behavior since it could lead to take all the available RAM when dealing with some datasets.
If it is not, maybe it could store a reference path to the original image that is only opened when exporting the dataset.

As an example of this behavior I used this skeleton script:

for image_name in list_images:
    image = cv2.imread(join(images_path, image_name))
    detection = get_detection_from_file(gt_file_path)

    images[image_name] = image
    annotations[image_name] = detection

dataset = sv.DetectionDataset(
        classes=classes,
        images=images,
        annotations=annotations,
    )
dataset.as_yolo(annotations_directory_path=output_gt_path, images_directory_path=output_images_path)

Additional

No response

[LinasKo]

Hi @David-rn 👋

You've hit on something very high in our priorities list.
In the next supervision release we'd like to start implementing a more efficient data loader.

You can find some context here if it's something that catches your interest: #316

[SkalskiP]

@LinasKo and @David-rn, I'm closing this issue and marking it as duplicate.

[David-rn]

Perfect! I didn't find that issue, thanks @LinasKo for the clarification. If you need anything, glad to help!

[SkalskiP]

For now, we will work with @LinasKo, as we know, to design the proper API. Once we will create the first dataset loader (probably for YOLO format), you can pick the next one (COCO or PASCAL).