Self-penalization loss

[hankyul2]

Hi, thank you for sharing work!

I am a bit confusing about why top_prob_sd is used instead of top_prob_td in below code.

FixBi/trainer/fixbi_trainer.py

Line 67 in 3c98694

sp_mask_td = torch.lt(top_prob_sd, threshold_td)

Thank you!

[shreejalt]

@hankyul2
I also thought the same thing...
If we refer the diagram given in the paper it should be top_prob_td instead of top_prob_sd.

Did you tried by changing it to top_prob_td?

[hankyul2]

@shreejalt
Yes, I tried but failed to reproduce same results :(
If you success in reproducing same results, let me know.

[shreejalt]

@hankyul2
Did you used Imagenet pretrained weights? or the DANN pretrained weights?

Actually in the paper they have mentioned that the pseudo labels should be the output from the baseline model such as DANN

in the load_net function?

[hankyul2]

@shreejalt
I used DANN trained model. But whenever I try to add self-penalty loss, model does not converge at all.

For your information, I did not use this source code, but reimplemented version, which can be reason for not converging model issue.

[shreejalt]

@hankyul2
Can you share the DANN model weights? If it is possible for you?

Let me try with the changes I made and will let you know?

[hankyul2]

@shreejalt
I upload weight on my repo.

To use pretrained weights, use code below. (I write it on phone, sorry for skipping details)
If you find it hard to apply, let me know. I will update it tommorrow.

from torch import nn
from torchvision import models


class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.backbone = models.resnet50()
        self.backbone.fc = nn.Identity()
        self.bottleneck = nn.Sequential( nn.Linear(2048, 256), nn.BatchNorm1d(256), nn.Tanh(), nn.Dropout(0.1) )
        self.fc = nn.Linear(2048, 31)

    def forward(self, x):
        return self.fc(self.bottleneck(self.backbone(x)))

[shreejalt]

@hankyul2
Thanks a lot for your snippet.
I saw your repo and your number seems pretty less than the numbers given in the paper.

You used DANN model from your repo itself? a

Also did you tried by suing pretrained ImageNet weights? of the feature extractor?

[hankyul2]

@shreejalt

You used DANN model from your repo itself?

Yes

Also did you tried by suing pretrained ImageNet weights?

Yes

But difference from this repo is that I could not apply self attention that drop accuracy, which is because of my wrong implementation.

[shreejalt]

@hankyul2
I am unable to produce office 31 results from this repo even if I use Imagenet pretrainer weights.

I am getting 4-5% accuracy on the target data. Were you getting this much less accuracy or did you changed anything in this repo?

My FixMix loss of TDM is not decreasing at all. But my accuracy of SDM is increasing gradually after the epochs.

Also I haven’t changed anything in the code and I run directly using the command they have given.

[hankyul2]

@shreejalt

I am getting 4-5% accuracy on the target data. Were you getting this much less accuracy or did you changed anything in this repo?

Yes I remove self-penalization loss. And did you try with my weight?

My FixMix loss of TDM is not decreasing at all. But my accuracy of SDM is increasing gradually after the epochs.

I could not answer about it, because I did not try to run this repo.
But I add my source domain & target domain learning graph. (webcam -> amazon)

[shreejalt]

@hankyul2
Thanks for your reference.
When you load pretrained Imagenet weights, do you face the issue in which the accuracy of SDM is increasing and accuracy of TDM does not increase at all?

In your repo?

[hankyul2]

@shreejalt
I have never used imagenet weight for training fixbi directly. You should use pretrained weight like DANN, MSTN, etc.

[NaJaeMin92]

Hi @hankyul2 @shreejalt
Thank you for the discussions.

As @hankyul2 mentioned, top_prob_sd should be replaced with top_prob_td .
I've just fixed the code. Thank you for the report.

Basically, FixBi needs to load pre-trained baseline such as DANN, MSTN.

Actually, I've never experienced divergence issues when training FixBi with the settings I've published.
I recommend you to change the hyperparameters (e.g., sp_params or threshold) according to the performance of the baseline. Moreover, I used 0.3 and 0.7 for fixed ratio, but you can use other fixed ratios depending on your situations.

I hope my answer is helpful.