Loss function used to train baseline classifier g

[hssandriss]

Hello,

Great work guys,

I have a small question regarding the baseline classifier g.

Which loss are you using to train it? Is it Vanilla CE Loss? Because when analyzing accuracy per class on the unbalanced training data using the baseline classifier g with the weights that you provided, I found these already good accuracy values for unbalanced training: [99.8200, 99.8332, 99.2205, 97.8644, 98.6047, 97.1576, 97.8448, 99.2806, 97.5904, 96.0000].

Does it make sense to train it on more sophisticated losses like Weighted CE or so, to allow the model pick the features related to the minority samples better? Because on my dateset, the baseline classifier trained with CE performs really badly with a low accuracy near 5% on the minority sample of the training set. i.e. How well should the baseline classifier perform on the minority samples of the training data?

It would be great if you share the training procedure for reproducing the baseline classifier g (schedule, sampling methods, loss..).

I look forward to hearing from.

Many many thanks
Best,
Hssan

[bbuing9]

Hi, thanks for interest in our work!

As we have noted in the paper, we used a vanilla classifier, i.e., trained with vanilla CE loss without mitigating imbalance as g. The training script is already given in our "run.sh" file.

In our experiments, more sophisticated losses do not improve the performance of our algorithm, M2m.
Because capturing the features of the majority class is be more important for the baseline classifier g as one can see in Table 3.
Here, it is verified that the diversity of seed samples (majority class) directly affects the improvement from M2m.

But, if vanilla CE can't learn the training dataset with high accuracy, then I recommend 1) to increase the capacity of the network or 2) try sophisticated losses. Although the later one was not effective in our case, it can be different in your case.

Best,
Jaehyung.