Regarding the dimension of query and key

[netw0rkf10w]

Hi,

I observed in the code that the query's and key's dimensions are haft of the value's (out_planes // 2, group_planes // 2). Is there a specific reason for that (apart making it faster)?

Thanks.

[csrhddlam]

No, the "half" here is just an architectural hyper-parameter, which controls efficiency-accuracy trade-off. Empirically, we found "half" is a good trade-off in our cases, but "one" or "quarter" might also work well.

[netw0rkf10w]

@csrhddlam Thanks. I guess "one" is supposed to give the best accuracy? Do you have an estimate of how better it is compared to "half"?

[csrhddlam]

Yes, more channels usually leads to better accuracy, but we did not study much about it. Personally, I won't expect much improvement by switching from "half" to "one".

[netw0rkf10w]

Thanks for the answer.