why "split" to get multi-head?

[LifangD]

as the paper said or in some other implementation:
self.w_qs = nn.Linear(d_model, n_head * d_k)
the data size is larger.
but in this project, it is

       Q_ = tf.concat(tf.split(Q, num_heads, axis=2), axis=0) # (h*N, T_q, C/h) 
       K_ = tf.concat(tf.split(K, num_heads, axis=2), axis=0) # (h*N, T_k, C/h) 
       V_ = tf.concat(tf.split(V, num_heads, axis=2), axis=0) # (h*N, T_k, C/h)

it's like using partial of Q/K/V to form one head.
Can anyone help to explain why it uses "split" and "concat" to get multi-head?

Thanks!

[ty5491003]

I noted you closed this issue, do you have the answer? I have the same question.
@LifangD thx.

[LifangD]

Because of the parametric way, I think multiple projection to lower dimension is equivalent to "first project then split" , e.g: 100d ->100d->split(25d * 4), the number of parameters is 100 * 100; multiple projection(4times): 100d->25d, the number of parameters is 100*25 *4. And "split and concat " looks more elegant. By the way, I also tried such two methods in my own experiments and found no difference on final performance. @ty5491003

[ty5491003]

I got it, thx /:laughing:/