Why the argumentation transform of the orginal image contains colorjitter

[HelenMao]

Hi, thanks for sharing your code.
I have several questions about your design choice and looking forward to your reply.

1. Data augmentation:
   I find the augmentation operation contains

transforms.ColorJitter(0.4, 0.4, 0.4, 0.125) operation.

Since the style information always includes the color, why you involve the ColorJitter operation and regard this transformation sample as the positive sample of the original image?
Will that influence the final results?

2. queue samples use the transformed image
   You use
   x_k = data[1]

   in def initialize_queue(model_k, device, train_loader, feat_size=128)
   This means you use a transformed image to extract style vector rather than the original image, why?

Looking forward to your reply, thanks!

[FriedRonaldo]

1. Yes, it might influence performance because of the difficulty of the negative samples. Contrastive learning requires hard negative samples to learn more meaningful representation. It means that the color jittering operation is used only for the negative samples, not for the query image.
2. As you can see in https://github.com/clovaai/tunit/blob/master/train/train_unsupervised.py#L121 (and https://github.com/clovaai/tunit/blob/master/validation/validation.py#L74), we do not use the transformed image as a reference image but we use the original image as a reference. So, the style vector does not include the information related to the color jittered sample.

---

EDIT

The queue contains the key features for calculating the contrastive loss, not for the style vectors.

The first element of a batch is the resized, randomly flipped, and normalized image.
You can see it in datasetgetter function. https://github.com/clovaai/tunit/blob/master/datasets/datasetgetter.py#L35

[HelenMao]

Got it.
From my initial understanding, contrastive learning and the method iic help to extract the style vector, therefore the positive sample includes the color jittering seems strange from the I2I translation perspective. (since positive samples should share similar texture /style like features)

So you mean using 1) coloring jittering and 2) transformed samples to construct negative samples queue helps to learn the representation?

[FriedRonaldo]

You are right that we construct the positive and negative samples by using color jittering and transformation.
I misused "negative samples" in "It means that the color jittering operation is used only for the negative samples, not for the query image." (I should change "negative samples" to "key samples" - In EDIT, I changed it but I did not find out that the sentence above contains misused word at that time.)

I got your question related to color jittering. I adopted color jittering as a way to generate hard negative samples for contrastive learning and did not take into account the style vector view. However, I think that randomness of the parameters makes the color jittering less harmful to the style space in terms of the generator. (The color jittering operation applied with randomly selected parameters in (0.4, 0.4, 0.125) so that the effect of color jittering might not be harmful to the style vector extraction.)

[HelenMao]

Got it. Thanks!
Another question is that if we do not use the color jittering, will that influence the performance of the final results since it cannot construct hard negative samples?

[FriedRonaldo]

Yes, I think so. But I also think that part might be compensated with random augmentation without the transformations not related to the color.

[HelenMao]

Thanks for your reply :-)!