Changed Mask class to store RLE encoded masks to save RAM.

[ChristofferEdlund]

Changed Mask class to store RLE encoded masks to save RAM usage on large images. (Work in progress)

When doing inference on large images (5300, 3600) pixel with around 100 instances, the sahi library uses around 14 GB of RAM.
I introduced internal conversion to RLE encoded masks in the Mask class to save RAM memory and got it down to around 2GB RAM usage for our use-case (see plots).

Since pycocotools are a bit tricky to install on windows, the code are designed to utilize RLE encoding if pycocotools are available.

The inference time does not seem to increase by these changes, but only ran a single test so take the results with a grain of salt.

[fcakyon]

Wow @ChristofferEdlund this is an amazing contribution!

[fcakyon]

@ChristofferEdlund you should follow these steps for proper style formatting:

pip install sahi[dev]

python -m scripts.run_code_style format

[fcakyon]

You should not reformat with the latest version of black, you should you the versions installed with pip install sahi[dev].

black version is important:

sahi/setup.py

Line 40 in 5f095d5

"dev": ["black==21.7b0", "flake8==3.9.2", "isort==5.9.2", "jupyterlab==3.0.14"],

[fcakyon]

@ChristofferEdlund ram usage and inference benchmark plots seem really nice. How did you generate them?

Would you like to open a separate PR adding scripts generating such benchmark plots?

[devrimcavusoglu]

since pycocotools is moved to requirments.txt as a direct dependency, I think we change this import procedure, I think the following would be sufficient

from pycocotools import mask as mask_utils

but @ChristofferEdlund got a point that installing pycoctools on Windows is not straightforward, should we make this a soft dependency rather ? @fcakyon

[ChristofferEdlund]

As @fcakyon pointed out, it seems like one can actually install pycocotools on windows right now. But it is still this hurdle to get over:

https://stackoverflow.com/questions/64261546/how-to-solve-error-microsoft-visual-c-14-0-or-greater-is-required-when-inst

[fcakyon]

then it might be better to make pycocotools optional, c ompiler might cause issues to some users.

[fcakyon]

i will handle the required updates 👍

[ChristofferEdlund]

Sounds great!

@fcakyon, regarding an other PR for the plots, maybe.. Those plots where generated by monitoring the RAM usage of a docker container running on internal models and data. Let me think if there is a good way to do something similar with default SAHI and some good open source images.

[fcakyon]

maybe we can create a dockerfile for sahi (can also push the image to dockerhub)?

[fcakyon]

LGTM

[lioneldaniel]

@fcakyon similarly, would you accept a soft dependency to opencv in addition to pycocotools in coco.py to support json file with coco format containing RLE annotations?

Here would be my patch that automatically converts RLE segmentations to Polygon segmentations:

• add these lines into the import section of coco.py :

try:
    from pycocotools import mask as mask_utils
    import cv2

    use_rle = True
except ImportError:
    use_rle = False

• replace the first lines of method from_coco_annotation_dict in coco.py by the following ones:

    @classmethod
    def from_coco_annotation_dict(cls, annotation_dict: Dict, category_name: Optional[str] = None):
        """
        Creates CocoAnnotation object from category name and COCO formatted
        annotation dict (with fields "bbox", "segmentation", "category_id").

        Args:
            category_name: str
                Category name of the annotation
            annotation_dict: dict
                COCO formatted annotation dict (with fields "bbox", "segmentation", "category_id")
        """

        has_rle_segmentation = annotation_dict.__contains__("segmentation") and not isinstance(annotation_dict["segmentation"], list)

        if use_rle and has_rle_segmentation:
            # convert RLE to polygons: depend on opencv and pycocotools
            binary_mask = mask_utils.decode(annotation_dict["segmentation"])
            contours, _ = cv2.findContours(binary_mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
            poly_segmentation = []
            for contour in contours:
                if contour.size >= 2*3:  # Valid polygon has at least 3 points
                    poly_segmentation.append(contour.flatten().tolist())
            annotation_dict["segmentation"] = poly_segmentation
            has_rle_segmentation = False
        else:
            logger.warning(
                f"Segmentation annotation for id {annotation_dict['id']} is skipped since RLE segmentation format is not supported without pycocotools and opencv"
            )