Is there ambiguity between 2 embedding layers of 50,527 and 768 components

[richlevinson-design-2]

Hi Sebastian,

I am still working my way thru your book. Just starting ch 5, but have reviewed ahead and App A for neural networks etc.
I think the book is terrific and it is my anchor for learning about LLM's.
One question I have is about the embedding vectors and if there are actually 2 classes of vectors that are used in the book.
To clarify: on p 95, where the frequently used configuration, GPT_CONFIG_124M, is defined, there are 2 values:
"vocab_size": 50257
"emb_dim": 768
These 2 values keep reappearing in the book and I am finding some ambiguity between them.
On p95, "emb_dim" is described as "embedding size, transforming each token into a 768-dimensional vector".
That's ok, but then, on p99, we have: model outputs: logits, followed by explanation:
"Each text sample consists of four tokens; each token is a 50,257-dimensional vector, which matches the size of the tokenizer's vocabulary."
It goes on: "The embedding has 50,257 dimensions because ...".
So, we appear to have 2 classes of embeddings: 768 dim and 50257 dim corresponding to the two config variables above.

What I am taking from all this is the 768 seems to apply to the token embeddings used in the neural network part of the architecture, and the 50,257 seems to apply primarily when the input is first encoded from text to tokens then embeddings, and when the output is finally decoded from embeddings to tokens then text.

Where the ambiguity comes in for me is I don't see any clear distinction where the 50,257 initial encoding switches to the 768 for neural net processing, and on the output side where the 768 dim vectors get transitioned back to the 50,257 vectors that are appended to the output string.

I can see the 50,257 has value because it represents the probability that the next token will be one of the 50,257 words or subwords in the vocab: i.e. all the probs add up to 1 and the largest of these is a good candidate for the output token to add to the response.

So, generally, my question is, for example, could someplace like p94, fig 4.4, be used to show a box where 768-dim tokens get transformed to 50,257 on output, and another box where the 50,257-dim tokens get transitioned to 768 for input to the neural net stack?

Thanks,
Rich

[sueun-dev]

The 50,257-dim thing isn’t really a second embedding layer. It’s the logits vector over the vocab.

In the current source at ff0b3d9, tok_emb = nn.Embedding(vocab_size, emb_dim) maps token ids to [batch, tokens, 768], and the transformer stack stays in that 768-wide hidden space. The switch back to vocab size happens at out_head = nn.Linear(emb_dim, vocab_size), so the model output is [batch, tokens, 50257].

Then generation usually takes only the last token’s logits, chooses/samples one vocab index, and appends that integer token id. So I’d label the output box as “LM head / vocab projection,” not as another embedding step.

[richlevinson-design-2]

Ok, so it sounds like:
The nn.Embedding(50257,768) provides the input transition: 50257->768.
And the nn.Linear(768,50257) provides the output transition: 768->50257.
Hope I've got that right.
Thanks.

[rasbt]

Yes, exactly. Fun fact, nn.Embedding is essentially the same as nn.Linear, but more efficient in this direction. I've written about this in the bonus notebook here: https://github.com/rasbt/LLMs-from-scratch/blob/main/ch02/03_bonus_embedding-vs-matmul/embeddings-and-linear-layers.ipynb

[rasbt]

Hi there,

to provide a bit more information to the excellent answer above.

"embedding size, transforming each token into a 768-dimensional vector".

The 768-dim are the internal representations that the model is working it. You can think of the 50,257-dimensions as the (input and output) interface between the model and the tokenizer.

Maybe it's easier to see in this figure from chapter 6:

For the input, it's similar but inverted

I hope this helps!

[richlevinson-design-2]

Hi again,
The diagram from p 184, fig 6.9 is really helpful. I had not yet come across that until you showed it in your reply.
Also your updated version of the diagram w the upside down mapping showing the 50257->768 token embedding layer is helpful as well.
I assume the two mappings are consistent w the nn.Embedding and nn.Linear described in the reply from sueun-dev as well.

One remaining question is if the GPT model has weight matrices for these input and output transitions?
I am guessing it does but have not had time yet to dig further to uncover that.

Thanks, this is really interesting to me.