Prior p(w)

[ZeWang95]

Dear authors,

According to your ICLR paper Empirical Bayes Transductive Meta-Learning with Synthetic Gradients, it appears p(w) is adjustable and is trained to achieve empirical bayes.
However, in your code, sib.py line 29, I belive the p(w) is fixed as a zero-mean gaussian.
Please correct me if I were wrong, but how is this implementation achieving empirical bayes?

Thank you in advance!

[hushell]

Hi Ze,

Sorry for the late reply! Yes, the prior is fixed in the implementation: we tried to optimize the Gaussian mean but didn't yield a significant difference in terms of the performance. This however makes sense because the parameterization has to match the aggregated posterior q(w | t) as shown in our Theorem 1. Therefore, simply making p(w) a Gaussian is sub-optimal. We stopped exploring more powerful parameterization for p(w) because that wasn't the emphasis of the paper. We are investigating this for future work. For example, letting p(w) to be an auto-regressive model sounds a good idea...

[adamian]

Thanks for the reply Shell. Ze, I'm closing the issue but feel free to reopen if there are more questions.

[ZeWang95]

Thanks for the answer! There indeed should be a lot of work we can do on p(w).
Just want to say that according to the current implementation, the generated w is actually an output of a network, so it is not covered by model.parameters(), thus is not covered by weight decay.
But the scaling parameter in SIB layer can be covered by the weight decay, and applying weight decay to the scaling parameter is kind of approximating applying weight decay to w. So it makes sense, however, it'd better to note these in the code.
I was very confusing when first looking at the code, and it took me a long time to figure out the above explaination.
Thanks!
Very nice work!

[hushell]

Hi @ZeWang95, sorry for my slow responses!

the generated w is actually an output of a network, so it is not covered by model.parameters(), thus is not covered by weight decay.

You're right! What I wrote in the comment was incorrect. I'll add the log p(w) term back to the synthetic gradient updates. That will make the implementation precisely an empirical Bayes.

Thanks for pointing this out!