From 4b3ff759e40ec91db5e5a0ffda396cb45ae8a51d Mon Sep 17 00:00:00 2001 From: Adithya Ganesh Date: Sun, 17 Sep 2023 22:10:23 -0700 Subject: [PATCH] impl --- docs/index.html | 10 ++ docs/index.xml | 12 +- .../2019-07-13-polya-burnside/index.html | 2 + docs/posts/index.html | 18 +++ docs/posts/index.xml | 12 +- docs/posts/transformers/index.html | 152 ++++++++++++------ docs/sitemap.xml | 9 +- site/public/index.html | 10 ++ site/public/index.xml | 12 +- .../2019-07-13-polya-burnside/index.html | 2 + site/public/posts/index.html | 18 +++ site/public/posts/index.xml | 12 +- site/public/posts/transformers/index.html | 152 ++++++++++++------ site/public/sitemap.xml | 9 +- 14 files changed, 318 insertions(+), 112 deletions(-) diff --git a/docs/index.html b/docs/index.html index 4fbf703..94c1f2e 100644 --- a/docs/index.html +++ b/docs/index.html @@ -73,6 +73,16 @@

Adi Ganesh

+

+ + GPT in words and code + +

+ + +

Notes on transformers / LLMs from the ground up.

+ +

Pólya-Burnside enumeration in combinatorics diff --git a/docs/index.xml b/docs/index.xml index cf5fc39..79297ba 100644 --- a/docs/index.xml +++ b/docs/index.xml @@ -6,7 +6,17 @@ Recent content on Adi Ganesh Hugo -- gohugo.io en-us - Sat, 13 Jul 2019 00:00:00 +0000 + Sun, 20 Aug 2023 00:00:00 +0000 + + GPT in words and code + https://acganesh.github.io/posts/transformers/ + Sun, 20 Aug 2023 00:00:00 +0000 + + https://acganesh.github.io/posts/transformers/ + I find that the best way to understand how machine learning papers work is to write the code for a model forward pass. If you can load the weights from a pre-trained model and get the same outputs from a single model inference, you can be pretty confident that you’ve re-implemented all of the details from a model. The advantages of doing this are: +Does not require any training, which can be time-consuming and expensive. + + Pólya-Burnside enumeration in combinatorics https://acganesh.github.io/posts/2019-07-13-polya-burnside/ diff --git a/docs/posts/2019-07-13-polya-burnside/index.html b/docs/posts/2019-07-13-polya-burnside/index.html index 8bc010e..0da4f1b 100644 --- a/docs/posts/2019-07-13-polya-burnside/index.html +++ b/docs/posts/2019-07-13-polya-burnside/index.html @@ -324,6 +324,8 @@

    diff --git a/docs/posts/index.html b/docs/posts/index.html index 139264b..551822e 100644 --- a/docs/posts/index.html +++ b/docs/posts/index.html @@ -75,6 +75,24 @@

    Posts

    + +
    • 2019 (3)
      • diff --git a/docs/posts/index.xml b/docs/posts/index.xml index 22d137b..262e6c1 100644 --- a/docs/posts/index.xml +++ b/docs/posts/index.xml @@ -6,7 +6,17 @@ Recent content in Posts on Adi Ganesh Hugo -- gohugo.io en-us - Sat, 13 Jul 2019 00:00:00 +0000 + Sun, 20 Aug 2023 00:00:00 +0000 + + GPT in words and code + https://acganesh.github.io/posts/transformers/ + Sun, 20 Aug 2023 00:00:00 +0000 + + https://acganesh.github.io/posts/transformers/ + I find that the best way to understand how machine learning papers work is to write the code for a model forward pass. If you can load the weights from a pre-trained model and get the same outputs from a single model inference, you can be pretty confident that you’ve re-implemented all of the details from a model. The advantages of doing this are: +Does not require any training, which can be time-consuming and expensive. + + Pólya-Burnside enumeration in combinatorics https://acganesh.github.io/posts/2019-07-13-polya-burnside/ diff --git a/docs/posts/transformers/index.html b/docs/posts/transformers/index.html index 70e8126..4154117 100644 --- a/docs/posts/transformers/index.html +++ b/docs/posts/transformers/index.html @@ -57,7 +57,7 @@

      GPT in words and code

    • Does not require any training, which can be time-consuming and expensive.
    • Can test model outputs layer-by-layer to validate the correctness of different components.
      • -
    • Get a satisfying payoff at the end with a working model.
      • +
    • Get a satisfying payoff at the end with a working model, and develop an understanding of the model that is more detailed than what is found in the paper.

    This is the strategy adopted in Jay Mody’s picoGPT and Andrej Karpathy’s nanoGPT.

    For a good exercise to replicate GPT-inference, I would recommend reimplementing the gpt2 function in the picoGPT repo above, located here. picoGPT makes this especially easy because the weight loading code is already written and you can just write the forward pass in NumPy. A link to my implementation can be found here.

    @@ -65,76 +65,126 @@

    Model architecture #

    -

    Here I will break down the GPT architecture into its components. Here I will focus on GPT-3, since the architecture has been described in the 2020 paper.

    -

    Given $n_{\text{ctx}} = 2048$ tokens, GPT-3 will output a probability distribution over its vocabulary size of 50257 tokens. Decoding the next token can be done by grabbing the argmax over this distribution.

    -

    GPT-3 has the following architectural parameters:

    -
      -
    • $n_{\text{params}} = 175B$
      • -
    • $n_{\text{layers}} = 96$
      • -
    • $d_{\text{model}} = 12288$
      • -
    • $n_{\text{heads}} = 96$
      • -
    • $d_{\text{head}} = 128$
      • -
    +

    Here I will break down the GPT architecture into its components. Here I will focus on GPT-2, since the weights are publicly available. GPT-3 has a very similar architecture but is massively scaled up.

    +

    GPT2 can be implemented in the following pseudocode:

    +
    def gpt(input_string):
+  input_tokens = tokenize(input_string)
+  x = wte[input_tokens] + wpe[range(len(input_tokens))]
+  for transformer_block in transformer_blocks:
+    x = transformer_block(x) 
+  x = layer_norm(x)
+  return x @ wte.T
+

    In the following sections we will break down each piece.

    1) Byte-Pair Embedding Tokenizer #

    -

    $n_{ctx} = 2048$ tokens of text $\to$ one-hot tensor of shape $n_{ctx} \times n_{vocab} = 2048 \times 50257$.

    +

    $\text{String} -> \text{List[Integer]}$

    The first step is to convert words to numbers using a tokenizer. GPT uses byte-pair encoding (BPE). In BPE, the most common words are mapped to single tokens while less common words will be broken down into chunks and mapped to multiple tokens.

    OpenAI’s Tokenizer tool shows how different sentences will be broken into tokens under BPE. In this example, most words get mapped to a single token, except for “Sylvester,” which gets chunked into three tokens: Sy, lves, and ter.

    TODO: Fix image. Tokenizer

    -

    -2A) Word Embeddings -# +

    +2.1) Word Embeddings +#

    -

    $n_{ctx} \times n_{vocab} \to n_{ctx} \times d_{\text{model}}$

    -

    Now we convert the sparse one-hot tokens tensor into a dense embedding matrix. This is done by a linear layer.

    -

    -2B) Positional Encodings -# +

    We start by embedding each token which is done with a lookup

    +
    wte[input_tokens].
+

    This gives us a tensor of shape $n_{tokens} \times n_{embed}$. For GPT-2, $n_{embed} = 1600$.

    +

    +2.2) Positional Encodings +#

    -

    $n_{\text{ctx}} \times 1 \to n_{\text{ctx}} \times d_{\text{model}}$

    -

    Transformers are invariant to the order of inputs, so we need to tell the model which position each word is in. In GPT-3, this is done with (EQUATION).

    -

    -2C) Sum -# +

    Transformers are invariant to the order of inputs, so we need to tell the model which position each word is in. We grab positional embeddings with a similar lookup:

    +
    wpe[range(len(inputs))]
+

    This gives us another tensor of shape $n_{tokens} \times n_{embed}$.

    +

    +2.3) Sum +#

    -

    At this step, we sum up the Word Embeddings and Positional Embedding to aggregate them into a single embedding of shape n_ctx x d_model.

    -

    -3) Multi-Head Attention -# +

    Now we simply sum of the two tensors from before to get a single tensor of shape $n_{tokens} \times n_{embed}$.

    +
    x = wte[inputs] + wpe[range(len(inputs))]
+

    +3) Transformer Block +#

    -

    To explain how multi-head attention works in GPT-3, we will start with single-head attention.

    -

    We define the attention operation as follows: +

    The transformer block can be expressed as the following operation:

    +
    def transformer_block(x):
+  x = x + MultiHeadAttention(LayerNorm(x))
+  x = x = FFN(LayerNorm(x))
+  return x
+

    +3.1) Attention +# +

    +

    We will start by discussing single-head attention. We define the attention operation as follows: $$ \text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax} \left ( \frac{\mathbf{QK^T}}{\sqrt{d_k}} \right ) \mathbf{V}. $$

    +

    Here, $\mathbf{Q}, \mathbf{K}, \mathbf{V}$ are obtained from a linear layer on the input tensor.

    In code, this looks like this:

    -
    def softmax(x):
-    exp_x = np.exp(x - np.max(x, axis=-1, keepdims=True))
-    return exp_x / np.sum(exp_x, axis=-1, keepdims=True)
-
-def attention(q, k, v, mask):  # [n_q, d_k], [n_k, d_k], [n_k, d_v], [n_q, n_k] -> [n_q, d_v]
-    return softmax(q @ k.T / np.sqrt(q.shape[-1]) + mask) @ v
-

    GPT-3 uses multi-head attention, which means we do the following:

    -

    +
    causal_mask = (1 - np.tri(x.shape[0], dtype=x.dtype)) * -1e10
+
+def attention(q, k, v, mask):
+    # [n_q, d_k], [n_k, d_k], [n_k, d_v], [n_q, n_k] -> [n_q, d_v]
+    return softmax(q @ k.T / np.sqrt(q.shape[-1]) + mask) @ v
+

    Here, the causal mask prevents a tokens from attending to future tokens - in the context of language modeling, this is necessary since we will see each word stream in one by one.

    +

    Intuitively, $\mathbf{Q} \mathbf{K}^T$ will result in an “importance” matrix that returns the importance of each token to each other token. We then divide this by $\sqrt{d_k}$ and then pass this through a softmax. Finally, we multiply this by the $\mathbf{V}$ matrix, which represents the importance of each token.

    +

    +3.2) MultiHeadAttention +# +

    +

    MultiHeadAttention is a simple extension of single-head attention. Here, we will just redo the above operation several times with a different learnd $\mathbf{Q}, \mathbf{K}$ and $\mathbf{V}$` matrices. We will then concatenate the result of each attention head together, which is then multiplied by a linear projection.

    +

    In code, this looks like this:

    +
    def multi_head_attention(x, c_attn, c_proj, n_head):
+    x = linear(x,
+               w=c_attn['w'],
+               b=c_attn['b'])
+
+    qkv = np.split(x, 3, axis=-1)
+
+    qkv_heads = []
+    for elt in qkv:
+        qkv_head_split = np.split(elt, n_head, axis=-1)
+        qkv_heads.append(qkv_head_split)
+
+    causal_mask = (1 - np.tri(x.shape[0], dtype=x.dtype)) * -1e10
+
+    out_heads = []
+    for q, k, v in zip(*qkv_heads):
+        x = attention(q, k, v, causal_mask)
+        out_heads.append(x)
+    
+    x = np.hstack(out_heads)
+
+    x = linear(x,
+               w=c_proj['w'],
+               b=c_proj['b'])
+
+    return x
+

    4) FFN #

    -

    This is just a linear layer.

    -

    -5) Add and Norm -# -

    -

    We use a residual connection and then run layer norm.

    -

    -5) Decode -# +

    The rest of the transformer block is quite simple. The FFN block looks like this:

    +
    def ffn(x):
+  x = linear(x) # project up
+  x = gelu(x)
+  x = linear(x) # project down
+

    The GELU is an activation function defined in this paper, defined as $x \Phi(x)$, where $\Phi(x)$ is the standard Gaussian CDF function.

    +

    We will call the FFN block in the transformer block with a skip connection as follows:

    +
    x = x + ffn(x)
+

    +5) LayerNorm and Decode +#

    -

    At the end, we have a big word embedding matrix. We decode by multiplying by the transpose of $W_E$.

    -

    -Demo: +

    Before decoding words, we will run a last iteration of LayerNorm as follows:

    +
    x = x + layer_norm(x)
+

    At the end, we have a big word embedding matrix. We decode by multiplying by the transpose of $W_E$ to get back to tokens:

    +
    x = x @ wte.T
+

    +Demo #

    +

    And that’s it! For a demo, check out my repo! https://github.com/acganesh/tinyGPT/

    diff --git a/docs/sitemap.xml b/docs/sitemap.xml index 4f98ce3..129ca3e 100644 --- a/docs/sitemap.xml +++ b/docs/sitemap.xml @@ -3,12 +3,15 @@ xmlns:xhtml="http://www.w3.org/1999/xhtml"> https://acganesh.github.io/ - 2019-07-13T00:00:00+00:00 + 2023-08-20T00:00:00+00:00 - https://acganesh.github.io/posts/2019-07-13-polya-burnside/ - 2019-07-13T00:00:00+00:00 + https://acganesh.github.io/posts/transformers/ + 2023-08-20T00:00:00+00:00 https://acganesh.github.io/posts/ + 2023-08-20T00:00:00+00:00 + + https://acganesh.github.io/posts/2019-07-13-polya-burnside/ 2019-07-13T00:00:00+00:00 https://acganesh.github.io/posts/2019-07-13-modern-algorithmic-toolbox/ diff --git a/site/public/index.html b/site/public/index.html index 4fbf703..94c1f2e 100644 --- a/site/public/index.html +++ b/site/public/index.html @@ -73,6 +73,16 @@

    Adi Ganesh

    +

    + + GPT in words and code + +

    + + +

    Notes on transformers / LLMs from the ground up.

    + +

    Pólya-Burnside enumeration in combinatorics diff --git a/site/public/index.xml b/site/public/index.xml index cf5fc39..79297ba 100644 --- a/site/public/index.xml +++ b/site/public/index.xml @@ -6,7 +6,17 @@ Recent content on Adi Ganesh Hugo -- gohugo.io en-us - Sat, 13 Jul 2019 00:00:00 +0000 + Sun, 20 Aug 2023 00:00:00 +0000 + + GPT in words and code + https://acganesh.github.io/posts/transformers/ + Sun, 20 Aug 2023 00:00:00 +0000 + + https://acganesh.github.io/posts/transformers/ + I find that the best way to understand how machine learning papers work is to write the code for a model forward pass. If you can load the weights from a pre-trained model and get the same outputs from a single model inference, you can be pretty confident that you’ve re-implemented all of the details from a model. The advantages of doing this are: +Does not require any training, which can be time-consuming and expensive. + + Pólya-Burnside enumeration in combinatorics https://acganesh.github.io/posts/2019-07-13-polya-burnside/ diff --git a/site/public/posts/2019-07-13-polya-burnside/index.html b/site/public/posts/2019-07-13-polya-burnside/index.html index 8bc010e..0da4f1b 100644 --- a/site/public/posts/2019-07-13-polya-burnside/index.html +++ b/site/public/posts/2019-07-13-polya-burnside/index.html @@ -324,6 +324,8 @@

      diff --git a/site/public/posts/index.html b/site/public/posts/index.html index 139264b..551822e 100644 --- a/site/public/posts/index.html +++ b/site/public/posts/index.html @@ -75,6 +75,24 @@

      Posts

    + +
    • 2019 (3)
      • diff --git a/site/public/posts/index.xml b/site/public/posts/index.xml index 22d137b..262e6c1 100644 --- a/site/public/posts/index.xml +++ b/site/public/posts/index.xml @@ -6,7 +6,17 @@ Recent content in Posts on Adi Ganesh Hugo -- gohugo.io en-us - Sat, 13 Jul 2019 00:00:00 +0000 + Sun, 20 Aug 2023 00:00:00 +0000 + + GPT in words and code + https://acganesh.github.io/posts/transformers/ + Sun, 20 Aug 2023 00:00:00 +0000 + + https://acganesh.github.io/posts/transformers/ + I find that the best way to understand how machine learning papers work is to write the code for a model forward pass. If you can load the weights from a pre-trained model and get the same outputs from a single model inference, you can be pretty confident that you’ve re-implemented all of the details from a model. The advantages of doing this are: +Does not require any training, which can be time-consuming and expensive. + + Pólya-Burnside enumeration in combinatorics https://acganesh.github.io/posts/2019-07-13-polya-burnside/ diff --git a/site/public/posts/transformers/index.html b/site/public/posts/transformers/index.html index 70e8126..4154117 100644 --- a/site/public/posts/transformers/index.html +++ b/site/public/posts/transformers/index.html @@ -57,7 +57,7 @@

      GPT in words and code

    • Does not require any training, which can be time-consuming and expensive.
    • Can test model outputs layer-by-layer to validate the correctness of different components.
      • -
    • Get a satisfying payoff at the end with a working model.
      • +
    • Get a satisfying payoff at the end with a working model, and develop an understanding of the model that is more detailed than what is found in the paper.

    This is the strategy adopted in Jay Mody’s picoGPT and Andrej Karpathy’s nanoGPT.

    For a good exercise to replicate GPT-inference, I would recommend reimplementing the gpt2 function in the picoGPT repo above, located here. picoGPT makes this especially easy because the weight loading code is already written and you can just write the forward pass in NumPy. A link to my implementation can be found here.

    @@ -65,76 +65,126 @@

    Model architecture #

    -

    Here I will break down the GPT architecture into its components. Here I will focus on GPT-3, since the architecture has been described in the 2020 paper.

    -

    Given $n_{\text{ctx}} = 2048$ tokens, GPT-3 will output a probability distribution over its vocabulary size of 50257 tokens. Decoding the next token can be done by grabbing the argmax over this distribution.

    -

    GPT-3 has the following architectural parameters:

    -
      -
    • $n_{\text{params}} = 175B$
      • -
    • $n_{\text{layers}} = 96$
      • -
    • $d_{\text{model}} = 12288$
      • -
    • $n_{\text{heads}} = 96$
      • -
    • $d_{\text{head}} = 128$
      • -
    +

    Here I will break down the GPT architecture into its components. Here I will focus on GPT-2, since the weights are publicly available. GPT-3 has a very similar architecture but is massively scaled up.

    +

    GPT2 can be implemented in the following pseudocode:

    +
    def gpt(input_string):
+  input_tokens = tokenize(input_string)
+  x = wte[input_tokens] + wpe[range(len(input_tokens))]
+  for transformer_block in transformer_blocks:
+    x = transformer_block(x) 
+  x = layer_norm(x)
+  return x @ wte.T
+

    In the following sections we will break down each piece.

    1) Byte-Pair Embedding Tokenizer #

    -

    $n_{ctx} = 2048$ tokens of text $\to$ one-hot tensor of shape $n_{ctx} \times n_{vocab} = 2048 \times 50257$.

    +

    $\text{String} -> \text{List[Integer]}$

    The first step is to convert words to numbers using a tokenizer. GPT uses byte-pair encoding (BPE). In BPE, the most common words are mapped to single tokens while less common words will be broken down into chunks and mapped to multiple tokens.

    OpenAI’s Tokenizer tool shows how different sentences will be broken into tokens under BPE. In this example, most words get mapped to a single token, except for “Sylvester,” which gets chunked into three tokens: Sy, lves, and ter.

    TODO: Fix image. Tokenizer

    -

    -2A) Word Embeddings -# +

    +2.1) Word Embeddings +#

    -

    $n_{ctx} \times n_{vocab} \to n_{ctx} \times d_{\text{model}}$

    -

    Now we convert the sparse one-hot tokens tensor into a dense embedding matrix. This is done by a linear layer.

    -

    -2B) Positional Encodings -# +

    We start by embedding each token which is done with a lookup

    +
    wte[input_tokens].
+

    This gives us a tensor of shape $n_{tokens} \times n_{embed}$. For GPT-2, $n_{embed} = 1600$.

    +

    +2.2) Positional Encodings +#

    -

    $n_{\text{ctx}} \times 1 \to n_{\text{ctx}} \times d_{\text{model}}$

    -

    Transformers are invariant to the order of inputs, so we need to tell the model which position each word is in. In GPT-3, this is done with (EQUATION).

    -

    -2C) Sum -# +

    Transformers are invariant to the order of inputs, so we need to tell the model which position each word is in. We grab positional embeddings with a similar lookup:

    +
    wpe[range(len(inputs))]
+

    This gives us another tensor of shape $n_{tokens} \times n_{embed}$.

    +

    +2.3) Sum +#

    -

    At this step, we sum up the Word Embeddings and Positional Embedding to aggregate them into a single embedding of shape n_ctx x d_model.

    -

    -3) Multi-Head Attention -# +

    Now we simply sum of the two tensors from before to get a single tensor of shape $n_{tokens} \times n_{embed}$.

    +
    x = wte[inputs] + wpe[range(len(inputs))]
+

    +3) Transformer Block +#

    -

    To explain how multi-head attention works in GPT-3, we will start with single-head attention.

    -

    We define the attention operation as follows: +

    The transformer block can be expressed as the following operation:

    +
    def transformer_block(x):
+  x = x + MultiHeadAttention(LayerNorm(x))
+  x = x = FFN(LayerNorm(x))
+  return x
+

    +3.1) Attention +# +

    +

    We will start by discussing single-head attention. We define the attention operation as follows: $$ \text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax} \left ( \frac{\mathbf{QK^T}}{\sqrt{d_k}} \right ) \mathbf{V}. $$

    +

    Here, $\mathbf{Q}, \mathbf{K}, \mathbf{V}$ are obtained from a linear layer on the input tensor.

    In code, this looks like this:

    -
    def softmax(x):
-    exp_x = np.exp(x - np.max(x, axis=-1, keepdims=True))
-    return exp_x / np.sum(exp_x, axis=-1, keepdims=True)
-
-def attention(q, k, v, mask):  # [n_q, d_k], [n_k, d_k], [n_k, d_v], [n_q, n_k] -> [n_q, d_v]
-    return softmax(q @ k.T / np.sqrt(q.shape[-1]) + mask) @ v
-

    GPT-3 uses multi-head attention, which means we do the following:

    -

    +
    causal_mask = (1 - np.tri(x.shape[0], dtype=x.dtype)) * -1e10
+
+def attention(q, k, v, mask):
+    # [n_q, d_k], [n_k, d_k], [n_k, d_v], [n_q, n_k] -> [n_q, d_v]
+    return softmax(q @ k.T / np.sqrt(q.shape[-1]) + mask) @ v
+

    Here, the causal mask prevents a tokens from attending to future tokens - in the context of language modeling, this is necessary since we will see each word stream in one by one.

    +

    Intuitively, $\mathbf{Q} \mathbf{K}^T$ will result in an “importance” matrix that returns the importance of each token to each other token. We then divide this by $\sqrt{d_k}$ and then pass this through a softmax. Finally, we multiply this by the $\mathbf{V}$ matrix, which represents the importance of each token.

    +

    +3.2) MultiHeadAttention +# +

    +

    MultiHeadAttention is a simple extension of single-head attention. Here, we will just redo the above operation several times with a different learnd $\mathbf{Q}, \mathbf{K}$ and $\mathbf{V}$` matrices. We will then concatenate the result of each attention head together, which is then multiplied by a linear projection.

    +

    In code, this looks like this:

    +
    def multi_head_attention(x, c_attn, c_proj, n_head):
+    x = linear(x,
+               w=c_attn['w'],
+               b=c_attn['b'])
+
+    qkv = np.split(x, 3, axis=-1)
+
+    qkv_heads = []
+    for elt in qkv:
+        qkv_head_split = np.split(elt, n_head, axis=-1)
+        qkv_heads.append(qkv_head_split)
+
+    causal_mask = (1 - np.tri(x.shape[0], dtype=x.dtype)) * -1e10
+
+    out_heads = []
+    for q, k, v in zip(*qkv_heads):
+        x = attention(q, k, v, causal_mask)
+        out_heads.append(x)
+    
+    x = np.hstack(out_heads)
+
+    x = linear(x,
+               w=c_proj['w'],
+               b=c_proj['b'])
+
+    return x
+

    4) FFN #

    -

    This is just a linear layer.

    -

    -5) Add and Norm -# -

    -

    We use a residual connection and then run layer norm.

    -

    -5) Decode -# +

    The rest of the transformer block is quite simple. The FFN block looks like this:

    +
    def ffn(x):
+  x = linear(x) # project up
+  x = gelu(x)
+  x = linear(x) # project down
+

    The GELU is an activation function defined in this paper, defined as $x \Phi(x)$, where $\Phi(x)$ is the standard Gaussian CDF function.

    +

    We will call the FFN block in the transformer block with a skip connection as follows:

    +
    x = x + ffn(x)
+

    +5) LayerNorm and Decode +#

    -

    At the end, we have a big word embedding matrix. We decode by multiplying by the transpose of $W_E$.

    -

    -Demo: +

    Before decoding words, we will run a last iteration of LayerNorm as follows:

    +
    x = x + layer_norm(x)
+

    At the end, we have a big word embedding matrix. We decode by multiplying by the transpose of $W_E$ to get back to tokens:

    +
    x = x @ wte.T
+

    +Demo #

    +

    And that’s it! For a demo, check out my repo! https://github.com/acganesh/tinyGPT/