-
Notifications
You must be signed in to change notification settings - Fork 67
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
questions about The two Tf-KD methods #2
Comments
Hi, the first one is a good question, but DML is not self-training, it is a totally different method. For the second question, please make the equation of LSR=(1-alpha)p(k)+alpha*u be clear, then I can understand what did you mean. If you read our paper, we have illustrated the difference between our method of LSR. Of course, you can view our second method as a special LSR, because the standard KD also is a special LSR (regularization term is from a teacher rather than a pre-designed distribution), as stated in our paper. |
Thanks for your detailed explanation. LSR=(1-alpha)p(k)+alphau is the eq (1) in your paper. LSR= q'(k)=(1-alpha)q(k)+alphau(k) (sorry for wrong typing q as p). Here q(k) is hard label, which means q(k)=1 if k=c, q(k)=0 if k=others. eq(8) is your manually designed distribution. If let a=(1-alpha)+alpha/K, u=1/K in eq(8), then eq(8) equals eq(1). |
(1). Eq(8) will not be equal to Eq(1) unless temperature is 1. |
@yuanli2333 @pecanjk |
Hi, the simplest way to check the difference between LSR and Tf-KD_{reg} is rewrite the eq2 or eq9, check if they can be rewrote as other one. |
the first Tf-KD is just born again network right? |
The first Tf-KD method is self-training, which is quite similar to the "Deep Mutual Learning" paper.
The second Tf-KD method actually equals to the LSR.let a=(1-alpha)+alpha/K, u=1/K, your manually designed distribution equals to LSR=(1-alpha)p(k)+alpha*u, where p(k) is hard labels
The text was updated successfully, but these errors were encountered: