some questions about experiment setting and discriminator

[freeman-1995]

HI~ @szq0214

I'm highly intersted in your work!
Here is a question, I hope you can give your thoughts about it.

1. in experiment setting, why set weight_decay to 0, in general, weight_decay is important factor to the final performance, usually have 1% validation accuracy difference on ILSVRC2012 imagenet.

2. about the discriminator, It contains three convolution operations, its inputs is the logits of student and combined logits of teachers, but the target for discriminator is not right, in code that is as following:

target = torch.FloatTensor([[1, 0] for _ in range(batch_size//2)] + [[0, 1] for _ in range(batch_size//2)])

I think the target should be [1,0] through the whole batch_size, so that is weird. are there any considerations? if so, the influence of discriminator loss is to make logit of students away from teachers, something like regularization？

[szq0214]

Hi @alxer, thanks for your interest in our work!

1. weight_decay is not always used on ImageNet, for example, in some cases of training the binary neural networks, we also choose to use 0. I will run a comparison and let you know the results.
2. The discriminator is used to distinguish the teacher ensemble or student, acting as a regularizer. We want it can prevent the student's output from being identical to the teacher ensemble. If you only want them to be similar, I think KL loss is enough to achieve this purpose.

[wuwuwuxxx]

I'm curious about this target too. You didn't permute the input, so the discriminator output the same result across all training steps, does this operation cause a overfitting problem?

[szq0214]

Hi @wuwuwuxxx, it seems the discriminator converges very fast and tends to be overfitting, I'm not sure whether shuffling the input can alleviate this, as I think the feature patterns from teacher ensemble and student have a clear difference and the discriminator is easy to distinguish them, but I will try it later.

[normster]

@szq0214 I don't think shuffling the input would make a difference. The model isn't aware of the batch index of each input so it shouldn't be able to overfit to the ordering of the logits in the batch.

[szq0214]

@Freeman1937, see #4 for the comparison of using weight decay and without it.

[zimenglan-sysu-512]

I'm curious about this target too. You didn't permute the input, so the discriminator output the same result across all training steps, does this operation cause a overfitting problem?

i also confuse this settings