[BUG] 'Slide' inference mode does not perform as expected

[androbaza]

Describe the bug
I trained a model that has an extremely small image features (around 5-20px in size), while having a 6000px sized images. Overall the situation is similar to satellite image segmentation. The bug is that test_cfg mode='slide' performs inadequately during inference -- the performance is worse than mode='whole' almost all the time. I tested difference crop_size and stride combinations, but it does not change the situation much.

I used 256px crops in training, taking random crops from the 6000px images. It would be logical that test_cfg slide with crop_size of 256 and stride around 150-200 would perform the best on similar 6000px test images. But it completely loses the details and performs worse than 'whole' method.

If I crop the test inference image to around 1000-1500px size, and use 'whole' method -- the segmentation is perfect, almost 100% accuracy (so the model is indeed trained correctly). On the other hand, with the same crop, if i use 'slide' method, it still performs worse than 'whole' method.

Do you have any idea what is wrong? Have you experienced this during training the iSAID dataset, or other satellite datasets? The main question is - why doesn't the 'slide' method perform better at big images?

I've added examples of what i described above:
6000px_slide

6000px_whole

1000px_whole

1000px_slide. See how the inference with this method missed marking 'blobs' with blue in the left bottom corner, while it worked with 'whole' method.

[MengzhangLI]

Hi, sorry for late reply.

From your description and your four visualization results, although I do not know details right now, it is probably caused by slide test mode. Because in many cases mIoU of slide is better than whole, when difference between crop size and image original size is much smaller than your case.

And from figures above, 256 px in training and 1000 px whole inference gets good results, whose cropping size and test mode are frequently used in MMSegmentation users. So we would try to figure out whether and why the slide mode cause bad prediction (like 1000px_slide example above) and let you know when we get conclusion.

As for iSAID, we just cropping the original dataset off-line in advance. The training and testing processes are both based on smaller images. Defaultly, the shape of image is (896x896) with 384 overlap slide, for all train/va/test dataset.

https://github.com/open-mmlab/mmsegmentation/blob/master/tools/convert_datasets/isaid.py#L151-L164

[androbaza]

Hi, sorry for late reply.

From your description and your four visualization results, although I do not know details right now, it is probably caused by slide test mode. Because in many cases mIoU of slide is better than whole, when difference between crop size and image original size is much smaller than your case.

And from figures above, 256 px in training and 1000 px whole inference gets good results, whose cropping size and test mode are frequently used in MMSegmentation users. So we would try to figure out whether and why the slide mode cause bad prediction (like 1000px_slide example above) and let you know when we get conclusion.

As for iSAID, we just cropping the original dataset off-line in advance. The training and testing processes are both based on smaller images. Defaultly, the shape of image is (896x896) with 384 overlap slide, for all train/va/test dataset.

https://github.com/open-mmlab/mmsegmentation/blob/master/tools/convert_datasets/isaid.py#L151-L164

Thank you a lot. I tried to understand myself what was wrong with the slide method, but have not seen anything off.

I also started thinking about cropping the images for inference and putting them back together, but putting tiles back together is quite cumbersome task and takes quite a lot of time to figure out. I will wait for your conclusion.

[MengzhangLI]

Hi, sorry for late reply.
From your description and your four visualization results, although I do not know details right now, it is probably caused by slide test mode. Because in many cases mIoU of slide is better than whole, when difference between crop size and image original size is much smaller than your case.
And from figures above, 256 px in training and 1000 px whole inference gets good results, whose cropping size and test mode are frequently used in MMSegmentation users. So we would try to figure out whether and why the slide mode cause bad prediction (like 1000px_slide example above) and let you know when we get conclusion.
As for iSAID, we just cropping the original dataset off-line in advance. The training and testing processes are both based on smaller images. Defaultly, the shape of image is (896x896) with 384 overlap slide, for all train/va/test dataset.
https://github.com/open-mmlab/mmsegmentation/blob/master/tools/convert_datasets/isaid.py#L151-L164

Thank you a lot. I tried to understand myself what was wrong with the slide method, but have not seen anything off.

I also started thinking about cropping the images for inference and putting them back together, but putting tiles back together is quite cumbersome task and takes quite a lot of time to figure out. I will wait for your conclusion.

OK, we would try to figure it out in this and next month (sorry about that since we are busy developing other tasks). I suggest you try to use similar script like iSAID. For example, in training set, it could crop each images to 896x896 with 50% overlap (overlap_area=384), in validation or test set, set overlap_area=0 to ensure pixels of your validation or test set are the same with your original validation or test set, whose mIoU value is also the same. Then concatenating tiles of each original images.

It is my own suggestion, perhaps you could have a try.

Best,

[androbaza]

Hi @MengzhangLI. I have trained another model using an older mmsegmentation release and another model, and now the slide method works perfectly. Because of change of both release and model, I could not say for sure what was the issue exactly, but my guess would be that it is the model.

I would also say this proves that it is not needed to do the tiling step for training, nor for testing. The pipeline works perfectly with 6000px images for both processes.