You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I was reading through the code and noticed that the self attention module is implemented like this :
class CausalSelfAttention(nn.Module):
def __init__(self, config):
super().__init__()
assert config.n_embd % config.n_head == 0
# key, query, value projections for all heads, but in a batch
self.c_attn = nn.Linear(config.n_embd, 3 * config.n_embd)
# output projection
self.c_proj = nn.Linear(config.n_embd, config.n_embd)
...
nn.Linear has bias=True by default so this means the self.c_attn has biases active.
I don't think that is part of the multiheaded attention implementation, and just adds some extra parameters and computation.
Maybe the author did it, just because OpenAI did it.
But then why did OpenAI do it in the first place ? To me it just seems like a small oversight
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
I was reading through the code and noticed that the self attention module is implemented like this :
class CausalSelfAttention(nn.Module):
...
nn.Linear has bias=True by default so this means the self.c_attn has biases active.
I don't think that is part of the multiheaded attention implementation, and just adds some extra parameters and computation.
Maybe the author did it, just because OpenAI did it.
But then why did OpenAI do it in the first place ? To me it just seems like a small oversight
All reactions