About DAFL

[KarhouTam]

Efficient-Computing/Data-Efficient-Model-Compression/DAFL/DAFL-train.py

Lines 64 to 68 in c43e133

	self.conv_blocks1 = nn.Sequential(
	nn.Conv2d(128, 128, 3, stride=1, padding=1),
	nn.BatchNorm2d(128, 0.8),
	nn.LeakyReLU(0.2, inplace=True),
	)

Hi. your work is inspired. I have some questions about a claim in the paper and the code, which are not big deals.

First, in the code. I found that the eps argument of torch.nn.BatchNorm2d is set to 0.8, which is quite bigger than the default value 1e-5. I wanna know whether this setting is marginal, or it helps generator training.

Second, in the paper, Section 3.1 claimed that Since filters in the teacher DNNs have been trained to extract intrinsic patterns in training data, feature maps tend to receive higher activation value if input images are real rather than some random vectors. I am confuse about the connection between the number of L1-norm of $f_T^i$ and the authenticity of input images. Can you explain why or lead me to some references?

That's all. Thanks for your excellent work again and look forward to your reply!

[HantingChen]

Thank you for your interest in our work and for these thoughtful questions.

Firstly, regarding the eps parameter setting in Batch Normalization, we indeed set it to 0.8 which is notably larger than the default value of 1e-5 in PyTorch. This setting is not arbitrary but directly borrowed from traditional GANs. You can find related code in this link: https://github.com/eriklindernoren/PyTorch-GAN/blob/master/implementations/dcgan/dcgan.py. This setting has been empirically found to be beneficial for the training of the generator.

As for your question about the correlation between the L1 norm and the authenticity of input images, our idea is that filters in the DNNs have been trained to extract intrinsic patterns in training data, and therefore feature maps tend to receive higher activation values if the input images are real rather than some random vectors. This notion is based on the paper "Interpretable convolutional neural networks" which suggests that activations in the neural network represent useful information in the input images. Therefore, we use the L1 norm as a way to measure the amount of useful information in the input images, and indirectly, their authenticity.

I hope these answers help. If you have any further questions, please don't hesitate to ask. Thank you again for your interest and support in our work!

[KarhouTam]

Wow! Thanks for your rapid answer. These info definitely solve my questions. Thanks a lot!