Beit why no k bias in attention?

[cqliheping]

The k bias is always zero in code. Is there any reason for this? This is different from the normal implement.

unilm/beit/modeling_finetune.py

Line 124 in 421cffe

qkv_bias = torch.cat((self.q_bias, torch.zeros_like(self.v_bias, requires_grad=False), self.v_bias))

In my test. when finetune, k bias has little affect on performance. But I do not have a test on pretrain.

[donglixp]

Both (i.e., with or without key.bias) are equivalent in terms of calculation results. They are canceled by the softmax function.

Softmax(q,k) = exp(q.weight * key.weight + q.bias * key.weight + q.weight * key.bias + q.bias * key.bias) / Z

Because the query is the same over all the keys, so the term (q.weight * key.bias + q.bias * key.bias) remains the same across all the keys, which in turn can be cancelled without affecting the softmax results.

exp(a)/(exp(a)+ exp(b)) == exp(a+C)/(exp(a+C)+ exp(b+C))