Question on output dim in Figure 4.4

[bowenxieoregon]

Hi all,
I have a question about the output dim in Figure 4.4. It says 'The LLM returns one 768-dimensional output vector for each 768-dimensional input token embedding'. However, I think the output dim should be the vocab_size, which is 50257. But here in the figure, the text says output dim is 768.

[rasbt]

Thanks for the note. You are right. Looks like a bad typo in the figure. The next page is correct:

The output tensor has two rows corresponding to the two text samples. Each text sam-
ple consists of four tokens; each token is a 50,257-dimensional vector, which matches
the size of the tokenizer’s vocabulary.
The embedding has 50,257 dimensions because each of these dimensions refers to
a unique token in the vocabulary.