Mosaic Augmentation for Object Detection

[innat]

Mosaic augmentation for object detection is used in Yolo-V4 literature, FR. I'm not sure if it's used for classification tasks there. I was wondering if it's possible to do so for the image classification task. Here are two issues:

• References. Not sure if it's ever used for classification only in any literature.
• Creation of class labels.

For creating class labels, here is one possible solution, described HERE; using Dirichlet distribution.

# for 2 images. Equivalent to λ and (1-λ)
>>> np.random.dirichlet((1, 1), 1)  
array([[0.92870347, 0.07129653]])  

>>> np.random.dirichlet((1, 1, 1), 1)  # for 3 images.
array([[0.38712673, 0.46132787, 0.1515454 ]])

>>> np.random.dirichlet((1, 1, 1, 1), 1)  # for 4 images.
array([[0.59482542, 0.0185333 , 0.33322484, 0.05341645]])

As mosaic takes 4 images.

---

update:

Mosaic augmentation for classification did use in literature. Please check: #250 (comment)

[kartik4949]

this definitely useful layer

[innat]

If anyone is interested, you can follow up on the conversation here.
AlexeyAB/darknet#7088 (comment)

[artu1999]

Would be happy to take this one.

[innat]

@artu1999 Great.
But note that, you may need to validate the creation of mosaic labels for the classification task. I'm not sure if there are any references (paper-work). The above approach (dirichlet) is just a pointer.

[artu1999]

@innat I can dig deeper into it, but it looks like yolov4 is the only one that references it for image classification. Maybe we can try and ask them again to point us at the label creation they used for image classification?

Also, just to clarify, with validation do you mean verifying that the probabilities you’d get from the dirichlet distribution are proportionate to the “sub-images” sizes in the mosaic?

[innat]

Also, just to clarify, with validation do you mean verifying that the probabilities you’d get from the dirichlet distribution are proportionate to the “sub-images” sizes in the mosaic?

I mean, whatever the approach would be (the label creation)), should be useful for the classification task (performance like cutmix and mixup).

[innat]

cc. @AlexeyAB
Mentioning Alex, to give some advice here. It would be really helpful.

[AlexeyAB]

We set the ground truth probability in proportion to the area occupied by each sample: https://github.com/AlexeyAB/darknet/blob/4ee3be7e68fb9c7eda5cc390e47e59f01e40dded/src/data.c#L1913-L1942
So if areas for: Car=10%, Table=20%, Cat=30%, Dog = 40%, then we set labels: Car=0.1, Table=0.2, Cat=0.3, Dog=0.4.

We have not tried using it in any other way.

Mosaic for Classifier reduces Top5-error from 6.0% to 5.5% (-10 relative %) on Imagenet-1k, and it works better than CutMix/MixUp, Tables 2 and 3: https://arxiv.org/abs/2004.10934

---

We used Mosaic data augmentation:

• for Classifier: YOLOv4
• for Detector: YOLOv4, Scaled-YOLOv4, YOLOv5, YOLOR: https://github.com/WongKinYiu/yolor

We didn't use Mosaic for Classifier in (Scaled-YOLOv4, YOLOv5, YOLOR), just because they don't need Classification or any pre-trained weights. We train these Detector from scratch to achieve the best accuracy/speed ratio.

It was originally introduced in the YOLOv4 paper: https://arxiv.org/abs/2004.10934

Some later work has used Mosaic:

• https://arxiv.org/abs/2004.12432
• https://arxiv.org/abs/2004.12178
• https://openaccess.thecvf.com/content/CVPR2021/html/Wang_Scaled-YOLOv4_Scaling_Cross_Stage_Partial_Network_CVPR_2021_paper.html
• https://arxiv.org/abs/2105.04206
  ...

[artu1999]

@innat @LukeWood I'm working on an implementation based on what Alex suggested. I managed to get the values for the labels right but I am having a few issues with creating the mosaic. Looking at CutMix as a reference, I am using fill_rectangles to draw the mosaic images over the input. This works fine when I fill only a single one of the sub-images, however when I try to do it for all three images I get a mixed up result (see the notebook for reference). Any idea why this happens?

[innat]

Not sure but some suspicious feelings about the creation of mosaic_x/y, s1-s4, center_x, center_y. Can you check this and this implementation, it might give you some pointers.

[LukeWood]

mentioning #21 to de-duplicate them.

[LukeWood]

@innat @LukeWood I'm working on an implementation based on what Alex suggested. I managed to get the values for the labels right but I am having a few issues with creating the mosaic. Looking at CutMix as a reference, I am using fill_rectangles to draw the mosaic images over the input. This works fine when I fill only a single one of the sub-images, however when I try to do it for all three images I get a mixed up result (see the notebook for reference). Any idea why this happens?

hey @artu1999 any progress here? Sorry I missed your comment.

[artu1999]

@innat @LukeWood I'm working on an implementation based on what Alex suggested. I managed to get the values for the labels right but I am having a few issues with creating the mosaic. Looking at CutMix as a reference, I am using fill_rectangles to draw the mosaic images over the input. This works fine when I fill only a single one of the sub-images, however when I try to do it for all three images I get a mixed up result (see the notebook for reference). Any idea why this happens?

hey @artu1999 any progress here? Sorry I missed your comment.

Hey @LukeWood, sorry for the inactivity recently, I’ve been travelling in the last few weeks and didn’t really have time to sit down properly and wrap it up. At the moment, I’ve got the preprocessing function, but before pushing I would like to test it on a pre-trained model to see if it brings any benefits for classification. I am going back in London in two days, will push some updates in the next few days.

[LukeWood]

Don't sweat it @artu1999 ! Enjoy your trip

[LukeWood]

@quantumalaviya @AdityaKane2001 any interest in working on this?