Question about implementation of calculating second-order derivative

[kdwonn]

First of all, thank you for the great PyTorch implementation.

But I can't get how the optimization procedure of the teacher network (eq.3 of the paper) is implemented.
The code should calculate the 2nd-order derivative during training, which is missing in the current version of the code.
Would you check the code again and let me know whether there is something I'm missing?

Thank you!

[kekmodel]

First of all, thank you for the great PyTorch implementation.

But I can't get how the optimization procedure of the teacher network (eq.3 of the paper) is implemented.
The code should calculate the 2nd-order derivative during training, which is missing in the current version of the code.
Would you check the code again and let me know whether there is something I'm missing?

Thank you!

In the paper, eq.3 is eventually updated to eq.12. The dot product term of eq.12 is approximated by Taylor expansion according to the author's code. Please see the derivation of t_loss_mpl in my code.
https://github.com/google-research/google-research/blob/d4d85f5977ede30c77c96fdbf171027538b91bfc/meta_pseudo_labels/training_utils.py#L417-426
https://github.com/google-research/google-research/blob/d4d85f5977ede30c77c96fdbf171027538b91bfc/meta_pseudo_labels/training_utils.py#L473-496

[kdwonn]

That is a point I have missed. Thank you!