How does the attention loss work

[zeabin]

Hi, thanks for sharing the code.

I notice that detach() is called before backward() for attention loss in train_step and the back propagation should not go through attention loss. So how can the attention loss work?

NAD/main.py

Lines 30 to 34 in d61e4d7

	cls_loss = criterionCls(output_s, target)
	at3_loss = criterionAT(activation3_s, activation3_t).detach() * opt.beta3
	at2_loss = criterionAT(activation2_s, activation2_t).detach() * opt.beta2
	at1_loss = criterionAT(activation1_s, activation1_t).detach() * opt.beta1
	at_loss = at1_loss + at2_loss + at3_loss + cls_loss

[bboylyg]

Hi, thank you very much for your reminder. This error has been fixed now.