transfomer-source-code-and-learning-note.ipynb

{"cells":[{"source":"<a href=\"https://www.kaggle.com/code/yaaangzhou/transfomer-pytorch-code-and-learning-note?scriptVersionId=148517602\" target=\"_blank\"><img align=\"left\" alt=\"Kaggle\" title=\"Open in Kaggle\" src=\"https://kaggle.com/static/images/open-in-kaggle.svg\"></a>","metadata":{},"cell_type":"markdown"},{"cell_type":"markdown","id":"fdd5296b","metadata":{"papermill":{"duration":0.011824,"end_time":"2023-10-30T02:33:15.812433","exception":false,"start_time":"2023-10-30T02:33:15.800609","status":"completed"},"tags":[]},"source":["**Created by Yang Zhou**\n","\n","**Source Url: https://cloud.tencent.com/developer/article/1885525**\n","\n","**[中][Transfomer]Source code and Learning Note**\n","\n","**1 Sep 2023**"]},{"cell_type":"markdown","id":"cfe8a2aa","metadata":{"papermill":{"duration":0.010927,"end_time":"2023-10-30T02:33:15.8348","exception":false,"start_time":"2023-10-30T02:33:15.823873","status":"completed"},"tags":[]},"source":["# <center style=\"font-family: consolas; font-size: 32px; font-weight: bold;\">[中][Transfomer]Source code and Learning Note</center>\n","***\n","\n","Transformer是一种重要的自然语言处理（NLP）架构，它在NLP任务中取得了巨大的成功，并且也被应用在其他领域，如计算机视觉和语音识别。Transformer架构的出现标志着一种重大的方法ological转变，它取代了传统的循环神经网络（RNN）和卷积神经网络（CNN），成为当前NLP领域的主要架构。\n","\n","以下是Transformer的一些背景信息和关键技术：\n","\n","1. **背景：** 在NLP领域，传统的方法主要依赖于循环神经网络（RNN）和卷积神经网络（CNN）。然而，RNN在处理长序列时存在梯度消失和梯度爆炸的问题，而CNN通常需要固定大小的输入。这些限制限制了它们在处理长文本和捕捉文本中的长距离依赖性方面的性能。\n","\n","2. **注意力机制：** Transformer引入了一种称为“自注意力（Self-Attention）”机制，它允许模型在输入序列中的不同位置之间建立关联，而不受距离的限制。自注意力机制允许模型根据输入的不同部分来计算注意力权重，这使得模型能够同时关注不同位置的信息，从而更好地捕捉长距离依赖性。\n","\n","3. **多头自注意力：** Transformer中的自注意力机制通常采用多头注意力的形式，即模型可以学习多个不同的注意力权重，每个头关注输入的不同方面。这有助于模型更好地捕捉多尺度和多粒度的信息。\n","\n","4. **位置编码：** 由于Transformer不包括RNN或CNN中的位置信息，它引入了一种位置编码来表示输入序列中的位置信息。这使得模型能够理解输入序列中不同位置的相对位置。\n","\n","5. **残差连接和层归一化：** Transformer中的残差连接和层归一化技术有助于训练更深的神经网络，减轻了梯度消失问题。\n","\n","6. **Transformer架构：** Transformer由编码器和解码器组成。编码器用于处理输入序列，而解码器用于生成输出序列。这种架构被广泛用于机器翻译（如Google的BERT和OpenAI的GPT系列）等任务。\n","\n"]},{"cell_type":"markdown","id":"b679768c","metadata":{"papermill":{"duration":0.010935,"end_time":"2023-10-30T02:33:15.857157","exception":false,"start_time":"2023-10-30T02:33:15.846222","status":"completed"},"tags":[]},"source":["# <center style=\"font-family: consolas; font-size: 32px; font-weight: bold;\">基本架构</center>\n","\n","简单来说，Transformers的架构做的事就是在Encoder中不断对输入进行编码，随后将编码后的结果一一输入至解码器。"]},{"cell_type":"markdown","id":"ab348c4a","metadata":{"papermill":{"duration":0.010852,"end_time":"2023-10-30T02:33:15.879326","exception":false,"start_time":"2023-10-30T02:33:15.868474","status":"completed"},"tags":[]},"source":["\n","+ 编码器：\n","    - 以下是Google论文原文中的示例图，可以看到编码器由N个相同的层堆叠在一起，每一层有两个子层。\n","    - 编码器的第一个子层是多头注意力层，第二个则是一个全连接层。\n","    - 每个子层都添加了残差连接和层归一化。\n","        + 残差连接是何凯明大神提出的为了解决梯度消失的方案，通过不断拟合残差以达到在更深层网络时性能不会下降的问题。\n","        + 层归一化则区别于批归一化，批归一化相当于对整个batch进行归一化，它会考虑全局的分布。而层归一化则是对独立的每个输入（整句话）进行归一化，也就是它不会考虑其他输入句子的分布。\n","\n","+ 解码器：\n","    - 解码器的架构大体上与编码器差不多，区别在于解码器中加入了掩码多头注意力。\n","    - 掩码多头注意力在这里的作用是遮挡了未来的信息。因为编码器输出了全局信息，而在生成解码器的输出时，如果提前看到了全局信息，可能会造成语法的不连贯。\n","    - 这也是Transformers的一种妥协。因为RNN本身是可以满足时间序列上逐个输入的效果，但是它无法并行计算，并且对长序列处理不佳。Transformers通过注意力机制解决了长序列的问题，但是它不是顺序输入，而是会看到整个输入。因此需要加入掩码来有意识的遮挡未来的输入。\n","    \n","+ 模型输出：\n","    - 模型的输出是解码器的输出简单加上softmax。输出的是多分类的概率。\n","    "]},{"cell_type":"code","execution_count":1,"id":"12b0911e","metadata":{"execution":{"iopub.execute_input":"2023-10-30T02:33:15.903656Z","iopub.status.busy":"2023-10-30T02:33:15.90326Z","iopub.status.idle":"2023-10-30T02:33:19.252477Z","shell.execute_reply":"2023-10-30T02:33:19.251495Z"},"papermill":{"duration":3.36453,"end_time":"2023-10-30T02:33:19.255139","exception":false,"start_time":"2023-10-30T02:33:15.890609","status":"completed"},"tags":[]},"outputs":[],"source":["import torch.nn as nn\n","import math"]},{"cell_type":"markdown","id":"c815098c","metadata":{"papermill":{"duration":0.011282,"end_time":"2023-10-30T02:33:19.279184","exception":false,"start_time":"2023-10-30T02:33:19.267902","status":"completed"},"tags":[]},"source":["## Embedding\n","\n","Embedding层将文本转变为向量表示，来描述原始数据所包含的信息。\n","\n","Embedding的初始化函数中要求输入词嵌入维度的大小和词表的大小。举个例子，Bert中的词嵌入大小是768维，词表大小则是30万。\n","\n","**先通过nn.Embedding将词表中的词汇转变成向量。随后乘以词嵌入维度的开根号。这是为了缩放词嵌入的数值范围。**"]},{"cell_type":"markdown","id":"de18ebf4","metadata":{"papermill":{"duration":0.011584,"end_time":"2023-10-30T02:33:19.302395","exception":false,"start_time":"2023-10-30T02:33:19.290811","status":"completed"},"tags":[]},"source":["_在前向传播中，输入文本的索引张量，输出对应文本的词嵌入向量。_\n","\n","_例如，如果 x 是一个形状为 (batch_size, sequence_length) 的张量，那么前向传播的输出将是一个形状为 (batch_size, sequence_length, d_model) 的张量，其中 d_model 表示词嵌入的维度，每个单词对应一个 d_model 维的词嵌入向量。_"]},{"cell_type":"code","execution_count":2,"id":"b77e6064","metadata":{"execution":{"iopub.execute_input":"2023-10-30T02:33:19.327175Z","iopub.status.busy":"2023-10-30T02:33:19.326668Z","iopub.status.idle":"2023-10-30T02:33:19.334764Z","shell.execute_reply":"2023-10-30T02:33:19.333838Z"},"papermill":{"duration":0.0229,"end_time":"2023-10-30T02:33:19.336803","exception":false,"start_time":"2023-10-30T02:33:19.313903","status":"completed"},"tags":[]},"outputs":[],"source":["class Embeddings(nn.Module):\n","    def __init__(self, d_model, vocab):\n","        \"\"\"\n","        类的初始化函数\n","        d_model：指词嵌入的维度\n","        vocab:指词表的大小\n","        \"\"\"\n","        super(Embeddings, self).__init__()\n","        #之后就是调用nn中的预定义层Embedding，获得一个词嵌入对象self.lut\n","        self.lut = nn.Embedding(vocab, d_model)\n","        #最后就是将d_model传入类中\n","        self.d_model =d_model\n","    def forward(self, x):\n","        \"\"\"\n","        Embedding层的前向传播逻辑\n","        参数x：这里代表输入给模型的单词文本通过词表映射后的one-hot向量\n","        将x传给self.lut并与根号下self.d_model相乘作为结果返回\n","        \"\"\"\n","        embedds = self.lut(x)\n","        return embedds * math.sqrt(self.d_model)"]},{"cell_type":"markdown","id":"9c699e9d","metadata":{"papermill":{"duration":0.01124,"end_time":"2023-10-30T02:33:19.359551","exception":false,"start_time":"2023-10-30T02:33:19.348311","status":"completed"},"tags":[]},"source":["# Positional Encoding 位置编码\n","\n","位置编码主要是给每个token分配位置，这里并没有直接使用绝对的index作为位置编码，而是通过了正余弦函数进行计算。其中Sin函数代表奇数位置，Cos函数代表偶数位置。\n","\n","计算后，每个token的位置编码将是一个长度为512的向量，这也是为了可以和embedding相加。\n","\n","初始化函数中，要求输入词嵌入维度，dropout的比例和句子最大长度。\n","\n","**创建了位置编码的张量，大小是(max_len, d_model)。这代表着每个token的位置编码。**"]},{"cell_type":"markdown","id":"3b00f276","metadata":{"papermill":{"duration":0.011371,"end_time":"2023-10-30T02:33:19.382379","exception":false,"start_time":"2023-10-30T02:33:19.371008","status":"completed"},"tags":[]},"source":["_前向传播中，接受词嵌入的输入维度(batch_size, sequence_length, d_model)。在计算出位置编码的512维向量后，对应相加，最后同样输出维度为(batch_size, sequence_length, d_model)的张量。与输入的区别只在于添加了位置信息。_"]},{"cell_type":"code","execution_count":3,"id":"ffa09cd7","metadata":{"execution":{"iopub.execute_input":"2023-10-30T02:33:19.407494Z","iopub.status.busy":"2023-10-30T02:33:19.406506Z","iopub.status.idle":"2023-10-30T02:33:19.416643Z","shell.execute_reply":"2023-10-30T02:33:19.415772Z"},"papermill":{"duration":0.025322,"end_time":"2023-10-30T02:33:19.419113","exception":false,"start_time":"2023-10-30T02:33:19.393791","status":"completed"},"tags":[]},"outputs":[],"source":["class PositionalEncoding(nn.Module):\n","    def __init__(self, d_model, dropout, max_len=5000):\n","        \"\"\"\n","        位置编码器类的初始化函数\n","        \n","        共有三个参数，分别是\n","        d_model：词嵌入维度\n","        dropout: dropout触发比率\n","        max_len：每个句子的最大长度\n","        \"\"\"\n","        super(PositionalEncoding, self).__init__()\n","        self.dropout = nn.Dropout(p=dropout)\n","        \n","        # Compute the positional encodings\n","        pe = torch.zeros(max_len, d_model) # 构建位置编码的张量\n","        position = torch.arange(0, max_len).unsqueeze(1) # 将每个token的位置（即从0到max_len-1）变为列向量\n","        div_term = torch.exp(torch.arange(0, d_model, 2) *\n","                             -(math.log(10000.0) / d_model)) # 计算\n","        pe[:, 0::2] = torch.sin(position * div_term) # 使用了广播机制分配pe的不同列。这代表着从第0行开始每隔两列取一列。返回所有偶数位置的编码。\n","        pe[:, 1::2] = torch.cos(position * div_term)\n","        pe = pe.unsqueeze(0) # 最后增加了一个维度，也就是batch的维度。\n","        self.register_buffer('pe', pe) #将位置编码张量存储在模型中。这是因为位置编码不需要进行训练，因此可以将其视为模型的一个固定部分。\n","        \n","    def forward(self, x):\n","        x = x + Variable(self.pe[:, :x.size(1)], requires_grad=False)\n","        return self.dropout(x)"]},{"cell_type":"markdown","id":"0b1d1e1f","metadata":{"papermill":{"duration":0.011389,"end_time":"2023-10-30T02:33:19.442296","exception":false,"start_time":"2023-10-30T02:33:19.430907","status":"completed"},"tags":[]},"source":["# Inputs of Encoders and Decoders\n","\n","Encoder和Decoder都包含输入模块。其中，编码器只进行一次推理，用于将Input的文本推理为向量表示。\n","\n","而Decoder则是类似于RNN一样逐步输入。这也是为什么之前提到需要用掩码遮掉未来的信息。因为需要用Encoder的输出，一步步输入Decoder。例如机器翻译任务中，输入如果是**我来自中国**，那么得到向量表示后，解码器将根据预测的第一个单词**I**输入解码器，然后根据下一个预测逐步完成整个句子的翻译。"]},{"cell_type":"markdown","id":"68f6d902","metadata":{"papermill":{"duration":0.011204,"end_time":"2023-10-30T02:33:19.464985","exception":false,"start_time":"2023-10-30T02:33:19.453781","status":"completed"},"tags":[]},"source":["_简单说，解码器的输入包括：编码器的完整输出，上一个时刻编码器的输出。_"]},{"cell_type":"markdown","id":"37ec7bdd","metadata":{"papermill":{"duration":0.011048,"end_time":"2023-10-30T02:33:19.487675","exception":false,"start_time":"2023-10-30T02:33:19.476627","status":"completed"},"tags":[]},"source":["# Encoder 编码器\n","\n","编码器由多个编码器层组成。\n","\n","以下函数中，clones用于复制给定的模块 module 并创建一个包含 N 个相同模块的模块列表（nn.ModuleList）。\n","\n","_Encoder是编码器的类，其中接受的参数是编码器层layer和层数。self.layers 是一个列表，其中包含了多个相同的编码器层（Encoder Layer）。每个元素都是一个编码器层的实例。_\n","\n","_前向传播时，编码器将输入张量 x 和一个掩码张量 mask 传递给每个编码器层。它会依次遍历 self.layers 中的每个编码器层，将输入 x 传递给每一层，并将每一层的输出作为下一层的输入。最终，返回编码器层处理后的输出，并通过层归一化 self.norm 进行最终的归一化操作。_"]},{"cell_type":"code","execution_count":4,"id":"c1e85933","metadata":{"execution":{"iopub.execute_input":"2023-10-30T02:33:19.512435Z","iopub.status.busy":"2023-10-30T02:33:19.512035Z","iopub.status.idle":"2023-10-30T02:33:19.521102Z","shell.execute_reply":"2023-10-30T02:33:19.520008Z"},"papermill":{"duration":0.024225,"end_time":"2023-10-30T02:33:19.523292","exception":false,"start_time":"2023-10-30T02:33:19.499067","status":"completed"},"tags":[]},"outputs":[],"source":["# 定义一个clones函数，来更方便的将某个结构复制若干份\n","def clones(module, N):\n","    \"Produce N identical layers.\"\n","    return nn.ModuleList([copy.deepcopy(module) for _ in range(N)])\n","\n","\n","class Encoder(nn.Module):\n","    \"\"\"\n","    Encoder\n","    The encoder is composed of a stack of N=6 identical layers.\n","    \"\"\"\n","    def __init__(self, layer, N):\n","        super(Encoder, self).__init__()\n","        # 调用时会将编码器层传进来，我们简单克隆N分，叠加在一起，组成完整的Encoder\n","        self.layers = clones(layer, N)\n","        self.norm = LayerNorm(layer.size)\n","        \n","    def forward(self, x, mask):\n","        \"Pass the input (and mask) through each layer in turn.\"\n","        for layer in self.layers:\n","            x = layer(x, mask)\n","        return self.norm(x)"]},{"cell_type":"markdown","id":"ac81a233","metadata":{"papermill":{"duration":0.01141,"end_time":"2023-10-30T02:33:19.546022","exception":false,"start_time":"2023-10-30T02:33:19.534612","status":"completed"},"tags":[]},"source":["# 编码器层\n","\n","具体看一下编码器层：每个编码器层都有两个子层：多头注意力和全连接层。其中每个子层都使用了残差连接。\n"]},{"cell_type":"markdown","id":"50e37ad2","metadata":{"papermill":{"duration":0.011064,"end_time":"2023-10-30T02:33:19.568503","exception":false,"start_time":"2023-10-30T02:33:19.557439","status":"completed"},"tags":[]},"source":["以下是一个子层残差连接的结构。"]},{"cell_type":"code","execution_count":5,"id":"1ff74d4a","metadata":{"execution":{"iopub.execute_input":"2023-10-30T02:33:19.593432Z","iopub.status.busy":"2023-10-30T02:33:19.592448Z","iopub.status.idle":"2023-10-30T02:33:19.599901Z","shell.execute_reply":"2023-10-30T02:33:19.599006Z"},"papermill":{"duration":0.022337,"end_time":"2023-10-30T02:33:19.602086","exception":false,"start_time":"2023-10-30T02:33:19.579749","status":"completed"},"tags":[]},"outputs":[],"source":["class SublayerConnection(nn.Module):\n","    \"\"\" \n","    实现子层连接结构的类\n","    \"\"\"\n","    def __init__(self, size, dropout):\n","        super(SublayerConnection, self).__init__()\n","        self.norm = LayerNorm(size)\n","        self.dropout = nn.Dropout(dropout)\n","\n","    def forward(self, x, sublayer):\n","        sublayer_out = sublayer(x)\n","        x_norm = self.norm(x + self.dropout(sublayer_out))\n","\n","        return x_norm"]},{"cell_type":"markdown","id":"6b2dbb1e","metadata":{"papermill":{"duration":0.011033,"end_time":"2023-10-30T02:33:19.624893","exception":false,"start_time":"2023-10-30T02:33:19.61386","status":"completed"},"tags":[]},"source":["现在就可以实现编码器层的结构。解码器接受四个参数，分别是：\n","+ size：编码器的输出维度\n","+ self_attn：自注意力机制模块\n","+ feed_forward：全连接前馈模块\n","+ drop_out：drop out的比例\n","\n","_前向传播中，编码器接受之前得到的Embedding+位置编码的张量x，以及一个掩码张量mask。x和mask先输入自注意层，随后将结果输入前馈神经网络。_"]},{"cell_type":"code","execution_count":6,"id":"2526a3a2","metadata":{"execution":{"iopub.execute_input":"2023-10-30T02:33:19.649482Z","iopub.status.busy":"2023-10-30T02:33:19.649013Z","iopub.status.idle":"2023-10-30T02:33:19.657321Z","shell.execute_reply":"2023-10-30T02:33:19.655995Z"},"papermill":{"duration":0.023454,"end_time":"2023-10-30T02:33:19.659754","exception":false,"start_time":"2023-10-30T02:33:19.6363","status":"completed"},"tags":[]},"outputs":[],"source":["class EncoderLayer(nn.Module):\n","    \"EncoderLayer is made up of two sublayer: self-attn and feed forward\"                                                                                                         \n","    def __init__(self, size, self_attn, feed_forward, dropout):\n","        super(EncoderLayer, self).__init__()\n","        self.self_attn = self_attn\n","        self.feed_forward = feed_forward\n","        self.sublayer = clones(SublayerConnection(size, dropout), 2)\n","        self.size = size   # embedding's dimention of model, 默认512\n","\n","    def forward(self, x, mask):\n","        # attention sub layer\n","        x = self.sublayer[0](x, lambda x: self.self_attn(x, x, x, mask))\n","        # feed forward sub layer\n","        z = self.sublayer[1](x, self.feed_forward)\n","        return z"]},{"attachments":{"f2f93c55-50d9-4814-ae4b-556732252e64.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAArgAAACxCAIAAACgMlmZAAAgAElEQVR4Ae2dd1QT2fv/ZyaFJhCKomJbXVHXLta1Ai42dMXeGxbUXV0bq65iWXvvZdW1u/Yu+rX3fuyNgyBypHloP4ST5JPMye9MQkibSSYYBOI7f0Ayc+eW173PzHtueS6hwgcEQAAEQAAEQAAEOAgQHMdxGARAAARAAARAAARUEApoBCAAAiAAAiAAApwEIBQ40eAECIAACIAACIAAhALaAAiAAAiAAAiAACcBCAVONDgBAiAAAiAAAiAAoYA2AAIgAAIgAAIgwEkAQoETDU6AAAiAAAiAAAhAKKANgAAIgAAIgAAIcBKAUOBEgxMgAAIgAAIgAAIQCmgDIAACIAACIAACnAQgFDjR4AQIgAAIgAAIgACEAtoACIAACIAACIAAJwEIBU40OAECIAACIAACIAChgDYAAiAAAiAAAiDASQBCgRMNToAACIAACIAACEAooA2AAAiAAAiAAAhwEoBQ4ESDEyAAAiAAAiAAAhAKaAMgAAIgAAIgAAKcBCAUONHgBAiAAAiAAAiAAIQC2gAIgAAIgAAIgAAnAQgFTjQ4AQIgAAIgAAIgAKGANgACIAACIAACIMBJAEKBEw1OgAAIgAAIgAAIQCigDYAACIAACIAACHASgFDgRIMTIAACIAACIAACEApoAyAAAiAAAiAAApwEIBQ40eAECIAACIAACIAAhALaAAiAAAiAAAiAACcBCAVONDgBAiAAAiAAAiAAoYA2AAIgAAIgAAIgwEkAQoETDU6AAAiAAAiAAAhAKKANgAAIgAAIgAAIcBKAUOBEgxMgAAIgAAIgAAIQCmgDIAACIAACIAACnAQgFDjR4AQIgAAIgAAIgACEAtoACIAACIAACIAAJwEIBU40OAECIAACIAACIAChgDYAAiAAAiAAAiDASQBCgRMNToAACIAACIAACEAooA2AAAiAAAiAAAhwEoBQ4ESDEyAAAiAAAiAAAhAKaAMgAAIgAAIgAAKcBCAUONHgBAiAAAiAAAiAAIQC2gAIgAAIgAAIgAAnAQgFTjQ4AQIgAAIgAAIgAKGANgACIAACIAACIMBJAEKBEw1OgAAIgAAIgAAIQCigDYAACIAACIAACHASgFDgRIMTIAACIAACIAACEApoAyAAAiAAAiAAApwEIBQ40eAECIAACIAACIAAhALaAAiAAAiAAAiAACcBCAVONDgBAiAAAiAAAiAAoYA2AAIgAAIgAAIgwEkAQoETDU6AAAiAAAiAAAhAKKANgAAIgAAIgAAIcBKAUOBEgxMgAAIgAAIgAAIQCmgDIAACIAACIAACnAQgFDjR4AQIgAAIgMA3IaD4cOPArp38Prv2nHiU+k1yhUTyCEAooCmAAAiAAAgUKYGsY4N8KMr1h5ahw3+bMn3W7MjIOZF/hvqJSIJyb9B36qxI5jN75pQxvVtUcCSFfpNuFWl2v7vEIRS+uypHgUEABECgWBHIONi3TNkOa19I9XKleDSzjoigJL3/y9I7qlK8XtC0VPuNyfrH8L2wCUAoFDZhxA8CIAACIGCGQNq+XrX6H0yl9YPQH1a1EZOkY9D6TwbHVbkH+9YcfV6mHxbfC5sAhEJhE0b8IAACIAACnATo1J2hLabflxsGyNgT6kYSokZzXigMT6Rv+zVg6Xul4UH8KlwCEAqFyxexgwAIgEBxJSBLenh4ZcTogaEdWjepX6euf8ug0LC/1h9/+pnXczg3OebV67cxsfGfklLTM7OyMtNTkz7Fx8a8e/06Ls3oua9SydPi3ryJfv8hITHlc0ZGWnJCXEz0pyxapZJe+HPYhrdGKUqjwnwFhKDK+KtGfQfK2C0jZ1wyOlhc+dpNviAU7KYqURAQAAEQ4EeAzniyc3LXej6Sym2G/b33ytPYNCmtlKa+ubprdo9a7iLP+v1X3Uoz7PI3jpj+uKNfzSqVfH0kDgKS0HxISuxWxrdSFb+ghQ+NlELOmbHV3UTagARBCp29/frv/MieiOLB9NpCgvLseyjbOF38LgICEApFAB1JggAIgEBREch+sW9SgK+DuELwnAsJRo9zJk9fHq/uVE5AutQeeeSj0Xs+a5blD6bXFhEEQTr6z7yTyf7gV19If1jZRkyQLtU6TNgQ9TrdaEhBP25l3IrWYpJ0+mVjopn49K/A90IlAKFQqHgROQiAAAgUHwLKT2cmNZFQVOl2kZeSOFUAnXJ0cEUBQXkErHjBoiSMipO+61dXkiBIcavlsZxRqlTKj8dHNagePPNETK5RBKY/03d3ZyYo+M97aUZNmF6GI4VFAEKhsMgiXhAAARAoTgSkr7b3+dGRJEs1nHo53fybuuLF/MZikqC8Q3cnmw+pkp4dUY4iCEL405/GExLzC0+n3ljco12vZTdTzSiJ/NAq6dmw8hQhqPr7NcxF0FEpym8QCkVJH2mDAAiAwDchQCccHvqjmCSFlfrujbP8uFY8nV1fRBCkw89L3pkNLb//509CgiCociPO6rtByC8UnXZ3ZZ92oQuuJpuNJz+8SqWJkvLqf+SL3lF8LUICEApFCB9JgwAIgMC3ICB/vSbIkyIIQcXBRw39FXClnnukv4QZUHD6ZZO5eQLK2OWtxEw4124700ziojMfrh/YLiTywie+IkGlUmmiJJ2CNydZ6MwwSQ8HCokAhEIhgUW0IAACIFA8CCjerGznRhIEJemw+QPPR7bsyvgqAoJgViheMTMAkD9BoeUyY98G2U+3DmvXafrpeMvTHAwwpe3s5koSosZ/v+aZVYOr8aMwCEAoFAZVxAkCIAACxYQAnbQ71Fs9i6Da+Cs5fDMluzimIiMUSPc+h7hnH2pmExCEsFaEwQSFnJe7RgYETzr63ozG4MiJ9Awz50FYbcINKwUGR3w4bAMCEAo2gIgoQAAEQKCYElA8+qse47+AFDWMfMZ/EcGX/T2YtQzmhYL8Qd4EhbLDz+RPUMh9uz/c37vutDu8RYk+ufSzI38QEKTzr7sNtnjQD4Lv35wAhMI3R44EQQAEQOBbEZBeHqseQiBFTazpy1c8n9OIcY5ACH74jXPtgTJ2hdEEBVnMkQlNPQUkQTr+vMTY26KZImedmdSqUSN//wY1Kpbx8i5durR3mQrV69Sv36BRYORN9CyYIfdtTkEofBvOSAUEQAAEvj0B2bXffmBGEAhR/dlP+fcnqDQTBZjJjMGbOOcUatwdEKTo56UxSpX8w8mILsGjF41v6sR0RQj9Jt3M72X49gVHirYkAKFgS5qICwRAAASKEQHFy3n+mo6BimMuWjFfQOscwazTI+m5sPKMCBHWnHYrLmpW18AR255lq+iU3aHMAguC8ul/OL0YsUBWCk4AQqHg7HAlCIAACBRrArlH+jGLHAnSscNWfssi1cXJvTCmknomo9h/nvHujfnlZVw3Mx4USOd67TsGDFj/ICNvNWPOxXDNaIdLwFpzvhrzY8KX4k4AQqG41xDyBwIgAAIFJJC6OVisnmlQZZy5RY5GseecZbZuZJZThmz7xOXLQLMfA+O6uVLI0luf9YMpns1ppHauIKozw3hzKKOk8LNEEIBQKBHVhEyCAAiAgPUE0rZ2cFDPUPCfy9kzYBwr/Wl7Vw+KWSbx4/jL3L4RtRMUxG1Wmnh6pBM2d2QcNxAC3+GnsXrBmHDJ+w2hUPLqDDkGARAAAV4EpMcHqucLiBrPf8XTfZHs/oy6IpIgBZWGnTAzWiE9p+l1YDoNWCZJZp4Yqp6/QLp33srZKcGrCAhUDAhAKBSDSkAWQAAEQKAwCChjlv7MDALw3mBJ/nJZGzeSIEU/DDtmbjsoRd4EBYHvyCjWtQ3yexE/qd03iP3nPmdREoVRWsRZWAQgFAqLLOIFARAAgaImoHgW2ZDpH3BotyZeN41AmXRt7e99OwcGdg1bcVVPD8heLG3rThGUpPWCB2bdJSnjVrZWz0JwD92bwV5G5ftVbZ3Vww9Vwi+ZjYz9ehwtRgQgFIpRZSArIAACIGBbAnSCesYB6RKwOm8zBmXc7oFtey67HJ9DK1Jvzg3puT5aPSqhjD84pLqYpCRNp0Zxuk7Iy1z+BIW2qz7o9IdhzunPB3qrXUdTnqF7UrhCGV6DX8WTAIRC8awXu8qVIvnp04/ofLRpnebEPHmVjnuvTZmW4MjMmpjsxapAT4oUVh5+klmbkBMVHjDhcra2tNmHBgXNfaGQRh8Iq+NCOVQKWXE/U3uO83/umREaDwq1pz8w4zcxJ2qUZr8IxxaLvs0OT7ALzjr7qhP2JxTo9JhnsZm4hX5Vs7DhxVn3l3fx/3XTazO3Exum9t1E9eX+og7NBmx/xTo8/N1QQEEZApZNjE46M8lfQlGlA+bfTIlZHjLsqK7dyG9Pqd+05yB/L5F73YHr7hosc2Tjq5RmxF1fFFyacalEiJrOfvBZxn63pXMT7yzrqAlHUKWD5l2IyeAIypZOwY7BLgrGzdJVdicUMo8O9BH5hp3lXtVjCQnO246A7MWa4B8aR1zTemKxHLMs6eHhlRGjB4Z2aN2kfp26/i2DQsP+Wn/86WeeU7Ytp5AXIjc55tXrtzGx8Z+SUtMzs7Iy01OTPsXHxrx7/TouzUTVyNPi3ryJfv8hITHlc0ZGWnJCXEz0pyz2GyRXFnKSot/GfEhISk1Lz8jMTP+cnJjwITbm7Zt3ieyNVZn58e3b6Nj4hKTU9Iz01MSPsdFvP6Tn98zQqVG/NajRdy/fbYO5slXyj+e8O75obJ9OQUEdOoYMmn3oVf7LcskvmuUS8Dax7Oe7JwRWcXH0bepf9cc+m+++e//64bXjWyJHtKno4FS18/Rd95Pz25ZKRSc/e/bJyObS9vbwFAhEYkcXV4mHl3dpby9PicStlLOjSOhQZZyB30dl7NpAN5FI7Ohcyo0J6+0pcXV2EAnEVcZdssI/pOXim4bgZRc2tkXTXNjbEXsTCun7e3lSBOUz+Ph3dbfIa5a5b/77a2i/YbMOvdW9MBRdi6XTLvxep/wv697p3X+4c0NnPNk5uWs9H0nlNsP+3nvlaWyalFZKU99c3TW7Ry13kWf9/qtupVn3ZOZOTEV/3NGvZpVKvj4SBwEz34r5kJTYrYxvpSp+QQuNncTknBlb3Y2Zwp33IYXO3n79d360Ij/K6A3d/Sr7li6lH43AUVK2cr0xR9kc3cpv/tXExzk/dwQpcJD4Bix8rEdT9nxxK9/mkXe/x7aeV7lfHq4JrVbKq/m0Ux9yM69HtvASCMr3P2RmVZ+ZRlHyTllnYiqVSpr46OTmub8P7xfaJTi4c+igsTOXbZoa4ObafY/hcMOX8+HNws5w7y5dvFFZsgub2WLxxmDL3NmZUPi8J1Si9jJeZsCR787NhzJ6eWv1biykc+BazhlGtmw95uKikw/29/UMXPNW78HGFT77xb5JAb4O4grBcy4kmLzNq1RfHq/uVE5AutQeeeSj0VsOV5R8jzNuaBlf+KSj/8w75kas6A8r24gJ0qVahwkbol7rXuz5JqQNl3V+jGaTHmHVIQfeW5gMTmfeiWggIilJgxEbb8SzBJY/X9hCUmPCle9UKmRGjakmosr02JNIq1Ty6xOqMR6FhdUmXGdpRNoKsJ//1pgYd6mzd3Z1FFQOv6RTBXTalcmNqg07VYIbFT+7sK0tciO2gzN2JRTo1J3d3DWvfZR330OGGpm7smRXJ7cO1+2mzh6QVyD2S2131HwmlK8XNnNgik86tlmRN8HZdmlbGVP25d/8nGr/ec9iP6Py05lJTZjh03aRl5I4VQCdcnRwRQFBeQSseGHTh0D6rl9dScYNbavl5rzSKz8eH9WgevDMEzG6+6mVRLTB07d3cWRqSdQg8plFEcVs9uveYNp17racfWFM1VL+kY8sgtambz//6U+bgl1IgW/YWU2l5D7ZOKh149aDNz7VdKiZN5cSz4G3iZktKZ20IUhMUF5Byx4zI2DyxFubw/wl4toRd0t2i+JlFza2RbOgS/ZJexIKdPL2kLItOrTW9Cl49jrA1qHLUl3Ze3tUHnzCQmc9r0Askdv0kKVMZD3aOq57x+7jtz0p4pcB+dN5TZw9um635JJN+mp7nx8dSbJUw6mXLczhV7yY31hMEpR36G69dd9fi1e7SZ7wpz/vcwkQOvXG4h7tei27mcqpY6zJhuzSWPV+O1TpISctNDr6485Qv8AVL83espXRy1o6uQeu1SxxsyYnJT2s9PggL4oQB6zjaGaWzKVEl5+viVksZPq2TprXC5GbT3kvZyHjncm3/0FL6yMtxlvUAfjYha1tsajLXHjp25FQoBO3dCrX5Z83+3tr9jiVhO5N4wNOfmdKTR9LQoFXID6pfU2YYpEJPgXIODLAR1hm4FHzoz90wuGhP4pJUlip714TZ/GmySiezq4vIgjS4ecl72zyxGben+7/+RPTW02VG3GW9aFNp91d2add6IKrybZKUsX4ylNv/CsOWJtgbpIDnXxoQI22S19a6nagE//p7Cas9tvVr+7sMIVerI9kbu/sQBAO3XaxTwYtMeZSEMj8TIxPzPI7U2owNqD9kI5+I0+UeJmgUql42IXNbZEP8BIZxn6EAp2wMbhct39T6MzD/dVLckjJr7t4TGqS34uoJfK2IBR4BSrsBlAsMsGjkHTSP53dBOWHn2a/f+fFIH+9JohRdIKKg4/yqCaVSpV7pD+zYy7p9MsmZkjaBh9l7PJWavdyrt12mopKOvPh+oHtQiIvGM/+/sqUcw71UQ+QCatPusXVi6FS0Wmnwmo1m/PYbG+CJid0yrYupQQWhdlXZrv4XZ6xrZNaKOxmmbyhUpUUcykIV34mxjNmZdzeAdXUo2EE6VgxcOqxWO5WyTPK4hGMh13Y3BaLR8ltnwu7EQp0/Lqg8qG7mVXAWccG+TAzGkm3rjss+QNTxm3rWpqizAsFXoFsXzeGMRaLTBhmif0XnbS1YylK0nM/97i6SqV4s7Ids7scJemwme/6PtmV8epd7gVVxluxYy57JjVH8ycotFxmPKcj++nWYe06TT8db/ubJvMEY17hyFKh+zi1VObF8XX9/7zN+gg0KRIdv6adI+Xd/7D5LhyT60r4AXNCocSYS0HqgJeJWRVxbsLjy1GX7r1LZe1XsyqqYhTYsl3Y3haLUfFtmRV7EQrKuNUB5Xvu07wWfjk1rJxaKbh2/sdcH1r26wPhDd0ogjAnFHgF0lUJnRVzbf/6xZEzZkQuWrvn0psM4y5rZW5GUtzbZ/dvXjp75oH2XVWa8vbB5ZNHjp+7/jyJzVT5ZIKWZ6d+jHn56PbVqFM3orm6oekvcTcPbV4+76/psxas2nH6aQrrG2uBMpkHIXN/TwllvludTtodqvbtKqw2/gq/Z6FKpZJdHKNx8+be5xBX8XQVweOb9GxYeaadCGtFGExQyHm5a2RA8KSj71nZ8IjYfBA6ZVN7MdPRK2qygMNd3ZebUxvWm3CF93NfvUOPwHfkebbGYz433+KsPPX5+T3rFkXOmDZ15vx1B2/H59JZj/duv8w22YRfA1XnmlMo8DEXlbk2fuzsjVepRhJRlvLi+ukjh4+dvf4s0XzzkyY8OL1307I5s+au3HH0ypOPX2zT/6WrKR4mpgv8PX+zaBeFYIv2ydtOhILy/fI2vn3+0+5OknNWswMqWarDZvaJTpmnJzWv5qld0k6SApFD/sepypjz6kcEr0C6dqFMurq0bx2Je43Q2TvPXr108t95A+p7Smr3W3sv399Q7uF+TO+55qORJ1+e7hjXLbj7sEnzli/7a0jTMmJJnX7rHugeEfwyoXg5v7G2NATB/piWxZ2b3+On8n4dflu+58yVaxeObo0c3LRixVbh/zw0mEhYgEzqIKhkl8dWFgh++O0a91NW8eivekxmSVFDHhP/8+P+sr8Hs0CBIG0kFOQP8iYolB2uW/OS+3Z/uL933Wl3eOuX/Pzx/iI9G6YWsoLyYVFslKQPZjf5aeQZg0qxEPmXg70lpKhB5FNLExosxGPz03TqtQXdGjXvF7nj7N0X0e/fPb99avPU0LZNaniK/CbfNnwW822g0tNhvk6MuYoYLxN6tluq2d93TvAwbJXKchv3cfJuOHCTeiWAKuvx1lHBAV2HMjY6a1jzsmLJT71X3c23aj1oirjjU4Mbtxsye9OhqKtXTu9ZOqqFj1MZ/+GbHhouvVW+WdLW26tMWd8KFStWrOBbtoy3p8Sjxd8vNNUnPTG8oodXmXK+FSpU8C1b2lPi0XTOE13N8jAxvSx9118t2kUh2KJdArcPoaB8t6Slb//Dur7u3Atj1DPLSZf2G9i94ijl0tzc3NyMA30kJEF5DTicwfxUf6RSed4rAK9AmnahjPtvqJ+ToHTw6me6RwydcXVSXQeBV8DK53m3RKU0Oz3xzcXIthKK6cc4/GjjoB4Rx9/nvwhmX/mtupAU/jA6Kl8q8MsELc/NSo29ubQD86bOIhS+PFrV2VfsHbTkgcEdSx5/JKyGo1v98GPxuq6PAmQy3zjo+FVtxYS4/Ua2t0VNKOnlseohBFLU5G+ON+r86PS+MCsF1ZMAzasQvSvMf1XGrjCaoCCLOTKhqaeAJEjHn5e81fEwH4/1Z5Wv5jdW+25wCN5sOj1D/nxRqxqDj1gaNDNMVj3Tk3Tpvie/2RieL6JfytitnctUHnbCyC+wMm57tzIiP4MpGlY1ULmUsd6kTR0cCELUfMHLL3mWK1PQ/MxFxbONVx0TlfJkRc8uk4/G5Nvol+sTawhJYaURp7QvJlq6ubci6vu2+/uu7kakUinidvfyFVBu/jOu69cNLc/Nzkh+e2lV7+qaxbJ1xp/9kKPtepB9OjO2ppAgHXxbj1r63/WX8WlS7SmVioeJaTP03f+3aBeFYIt2Cd0uhILyzcLmFQYd07dD6ZXxarc2pFPAWr3dVU3qUKqeIGdu6IFxZ2YxkPz50jYSiiwVsMp4kVr64f4+FCXpuFk/F8rXC5qICMqjXfd+Y4zWIcmuMgPxVJkhJ3R6Q51ry5lQqVTKuBWtRaZCgU4+MaKaiPLstoOlfyXn5uSaIkrSevGT/FuhOsGCZJLxevN7VYHZlX+ya79pPA6J6s+25gU4bWc3TYeCU/AmcwNKJnXMcUC7/53o56UxSpX8w8mILsGjF41vqvZZJfSbdNMQB0csBTqcvSfUhekbYVmUqXi7Osiv5x6zyyHY0vy8pYMDKfKfZ3GNRN7FyrhtfWv4+viUKdDHp1wl/ykXLQJSvJznL3IJ3WfUlJlhpBsT/GrqhEKBGqhKM/SgqT9jJrzMRWW5jUtaBHUeus/QbOQ3Jv4oJCjPvocMVyFLTw0tQxEE6dp2pb7MVL5b3EJMkg4tFr9hEZ/yd1tCygoIQdmu/8ToOg0UL+b6ezWezLqJo2UTM4bxHf+2aBeFYIv2iNsehILi5fymFYeeNJwXJrsx4Uf1jDHHtqvMrL3jdT+xFIhO3tvDmyLIUl22mb4HZh/q60mR4hb6a/qY2fYigiRFdf96pLs5qNuX8vXfzNumqPVKI9+KljKhvppOWh8oNhEKudcn+olIqszAY4Y3trwGrXyzsKmImVa4yWBaYUEyqVJl7+nuTAprTr1r2Kussx31w4MZexFUHGPgHl4XhPWb1uMBYcXTkDWivIPSc2Ga/e9qTrsVFzWra+CIbc+yVXTK7lDN6lqf/od5+uEwlwr7Oc2ThiBISd/DBsPdytgtXfy6bDHn+ok9RpX05BBvinOVJ8tFsveXd25Ys2Z1gT5r1m05/MC0M8Q4GVnUSF+BsNro88av3ipV5s7uDabeyWskBWugthAKFg2RTcwp3y1uztho04X6ekClyjnS30M958XINaRmASLpwNHNRqeeHsXYp1vzeQ80kir3yeLARkMOcahFiyZmXA3f82+LdlEItmiPvO1AKCiez/GvPOKM8VuL/PZkP41SaL3ceEq7riZ5PX4tBKITN/3iTBKEqPH8V6avDJrHMFlK35u65hlMCCqMvmA8Rp13StRobt54pTavFjKhCcYqFDIP9vVixiParWYfhVHJLoUzAzVGMwYKkkkVnbguQEyI/I1zry0Fs8hRM02DdOyw1fKjJv+6/MEksf88IzL5Yaz5wrhuVjcP53rtOwYMWP9AO+KcczFcMzDiErC2AA9sXnmgP65up5nOaLDcgk7Y3cMvaDUfp9cmyahnepIuPfYbCA+TYN/2gDJ2VVsnkqQk9ftF7roWk6VnHXT6q8cxedremgaqXwAb9CiohYI5Q6TKhZ0z7jnJqz5RnRkPDWW+PPpoZNiQ8KUXDHsglO8WNWOEhYlN5xfmy93IJqVIUlxj7LnPytSzY5u0nXPP8MUnPygPE9OFxTeLdlEItmiP1Eu+UFA8mVXfteHwpSZvRivGtVT7aDTroYfX49dCIOnRAcyLBCluM+/GY9PPtciWzF1C/5GgVQMsk890p4z8+1rIhKZxsgkFWZT61Zl07X2Q4yGieDXfn3FlJG69Ik7XyHU5MRkg0J0yyqRKGbOkhYgQNV/M6RMpdXOw+hkpqDLOikWO2umplCRkm+FdWJdha74xYzRqDwriSiFLbxkMoSuezWmkPsU8CLj6RaxJiyWsxqcgQQgqj7uslYp0ypFBNVot0s5mYbnK3CH5zT+qCwmHTttMX97NXVbY5zIu/VFHPZbDTEIVuVesF9BrzOx1J55rZRmTvlUNVD/DNhMKbLNAtW28zkzjTj9V3rPFeLGMftYYfz85n17cPHd0zz/rl0d0VgtxFnPPv0bxdkPH0hQpKBfQK6BBjx3crzY8TEwb6Y4dO36x909UVJS2uKz/LduF7W2RNSMl/GCJFwqKhzPq+TT6ddBglk/v5urOZVLcbCHb6CBTdbwev+YD0Ynq7n6CdKrTY9JUjs+0v7bd0/X7a29CLEPK2lMmNxXzmchrhmxCIe/JTHkNOm78apR3lfL9MkbLEIKqv1/TtWdtTqzJpEoZu7yliBA1W868hNYAABngSURBVMQpFNK2MjPQCLO9DrpMaL7Rn7Z3VWsx0Y/jL3O9aRlfZPa3doKCuM1Kk4EpOmFzR8bHAyHwHX5af+KL2RitOsmoW/V0RqcuOzRPdjr9zMhaTWY/4Kgii7HLbzLj5g5ddhROhi2mzxlA+enCvF/93HW7YDKSwaFCwOSDb/N0q1UNVD8d2wkFq9q4TihMu8ciJKUfrm6dPqD1j15l/Nr2Hhe5esfBE/+OYjYeY5MjesWhEw8PriIkCWGtKXfMtQLLJqaN9PTp07/b++fu3bva4rL+t2wXtrdF1oyU8IMlXSjI70XU+XHcZXbDUjz+q676dswMChh2Emqrjdfj10KgtH86Ms7SSUk/wwFnbRqm/wvyDLaQCU0iLEKBTt6oXrVPeQ48xo4p36kw836ry2xBMqmikzcGiQlRoznP2XmrVNLjA9WTANgHanTJ632T3Z9RV0QSpKDSsBNWjFboxWD8VXpOs37WtPdYHTLzxFCNxHTvvNUW/RfGyatU6ds7a3eGUvfXZF3+vW7DqTcLLoLUw0eka6//OHqNTLPwLY/QmTHXD22cP3lEaEB9XxdKvchV9OOoM8y+4dY1UP1cFzuhQKffWdnDz4USlW0z5cCL/F4TZbRmToOJ+NcvjEqljN0a4iMSkKRzg2kGKyQMg1k2McPw3/cvHnZhc1u0R+IlXCjIb0+uVWPiDW3vrXENKZ5FNtQs2Od6crE9ftP2jei/RX9eH3cgdYKyC6MrCAiCEAdvTjXOAfvvgjyDuTOhP/LLMplRemqYei62U7dduj4Ng3wpHs+qxwgqUfPF0boTBcmkSpVzsI87aW6jX8a9OtOzz3RfcNWbLhPMfgwvl7VxIwlS9MOwY9xrLg0usfRD7YdF3WUwMopVPMnvRfykbjhi/7mcksdSKubO500LyXP1lXMrolGd8Rf1V9WZu5jtnPRofwllxfxQOv3GuglhQ4ewdMTxODRk2KgZB15zacH8/Ck/nNt93nhjcGXas4MzgnxFJOnUekWMUqWyqoHmx61SWT2ZUWPY+jEUrI3nDz0Y9ijIni5t60ExO5wue2Yg13gJhZwH89s2HHbg6ppgb4oUVRl8mFOhWjQx/RJ+79952IXNbdEemZdsoSC7MbFGbe4J9iqVeoo98/7CtRSP5fFLp2zq3I5ZMZf/4Q6kCZJ9bBDzKBb8MP4qx6OPzox9r7f3YEHuT9yZ0MspS4+CSpW+rwczWUPUMJLjmZc3W9vo9bogmWQ87E+rJSQl/Y4Y3CrzUapUqjz1Rjq0W6O3ZFSZdG3t7307BwZ2DVtxVU8PyF4sbetOEZSk9YK8KeH6cRXsuzJupWaCgnvoXo4hfeX7VW2ZCaqEoEr4JeN5sgVL1eAqvd1o1rx/ENm05ohTzNt1gT8aD3OM0tNrDeZio1Oi5g34tUuXzgX6dAkJHbXlCUu/u2GS0lNDfVstZcuS2hcB5THgKKPTrGmg+glY2aOQZ9j6MRSsjbMKBTr5325qM6s703hmi2YRpnboQRn974QFRjcKOuXU6KbtFz/OVanoTwcHVBKSlHfwutfsgC2bmH4Rv+/vfOzC1rZol8RLtFCQXhnvV2/6A3MvNsrXC5oy768Ey0pEpkJl50f5CgyXqdEfVgYFG+zrxx0or01I702vIyIJYY0pt1mVguzWFP9ft+t6zQtyf+LOhN4DhlUoqLIvj68uJEnnwHVGSy412c+9GF5ZQJBugWv11REz3aAVM7Jq5fhtyuZgR9Jg6qax4dAJ6hkHpEvA6rw5W8q43QPb9lx2OT6HVqTenBvSc73m4aKMPzikupikJE2nsq4oz4v5y9vzO7bsuRbLqU2McpA/QaHtKlYgTHD684Heai/TlGfoHtNFrwYxWps+c7H6tZBxpVAlpFcLvwEH9bSRQdw8f8jvRzD6rO/hQhA1PLPAEkx6ckhp54A1bJBl58LKC73y9mIrWAM126PAbS76+SxgG9csWhHW0u9RyOsXYVvJlPfKmjdHQfFoZrM+BotTFO82d/Xvvz9/NWTGpYm1HUnSxX/mLbY+QNqyiemX8bv+zssubGyLdgm8BAsFZeymDhKnDls/m60YzepEjUN/lqlHyvfLWzuQBmsScs+NbjLawLWu5UB0+qU/6jqRlHeXraZL6hTRG7v4jz6r17GsjGbWBhivR1SXQ3vKZDzTciaYx1vCGmbdnbit4UpIOuXUKD8xKao9yXTsU/l+c0cvivJou8TY4VIBMsmMFNybVlNIlQtj37ZZU1eyF6sCPSlSWHn4SWbBQU5UeMCEy/n3xOxDg4LmvlBIow+E1XGhHCqFrLivx864tunkfb3UQytC3/7/8XLElHtmhMaDQu3pD9jf2dRp5ESN0mwt4dhikRkHktanr45d/VqoduVNlem+k2PdqnFZuX9n7vrVhRS1WGIg9biDf6MzjFCgxPVm3DMd30nb9au7W8ctef3rBWqgqs9bgtWeGdlmzvIyF5XW2lhciWtPmbo60bo1MxQKeX7EKO+Bx4xmmmRfmtCkrITSvqvIr09sMlQ3r1j2bs/gGmU6bcmXCUzd5D6a39yVJEXVhh9nadJ8TOwb1TCTjDzx/87fu3ijcD+pqTyHdQ0LzssubGyLhjmwk18lUCjQ8qyktzf2Rf7KbI1KOlTvtfLC65Qcth5XWpoed3f7oOrMenmCoEoHzbsYk67nC5WpQ2X89q7eFOnYZO5TdWcA/WlXz8bh5w1tnU8gefTeYT+VErg1CD+k8/aqojMebBjQpHXEZT3H/fLMt9t7lhMQBOXVZf3LDK2/aCY3ulOewcsfpuTql8pyJhRZMXsHMA83QZUhB95n6Ueskr7c3t/PWVSp2+r7+lmJPznRXyL0ahFh/Mauy4lVmWRucqeHlaXMrBlXWw6ddGaSv4SiSgfMv5kSszxkmLoPWmNU8ttT6jftOcjfS+Red+C6uwZrFzUh9P8qHs6ow0ywYPafbr/B4HarH0zzXSnNiLu+KFi9DTkhajr7wWeZXn+MXng6N/HOso6acOqWcyEmgz2oVenrUtBN4euw0cxaON0FZr/Jr0+oJjTn5srs1YV2Ui0UmOVA4ScT9Pv96NSLE+t7NpyuP3nT2gaqyI47PLyqkCAon5DV9z9lG7R2fobNq417dFzzIl0/ckVW9K6+voz5ugcsupeUf+eRP17Qwo0ZJAtYqjdFITf6QHjo1BP7h/kKSJfWy98q5C8X/tJtUzxNyz5H3/pv4ZAmZUQk6dpuyfNsXUukc+OPhan9xVGSZhP23o/P1M8ATxMrtFo1jFjxYmFzsVtjtQUW4p+9e/capsvrFz+7sK0t8spYiQtU8oRC7qF+no7Oru4SDy9vb29PD4mbi6PYqd0qvfFudS3k7Al1dSrlJvH08vYuzXy8vTw9JBJXZ0cH35H69UQnnZ/dqZqbY4U2w6ZMG9PZ/+cJp037gfkFyniyZ2rnmh7ulVr1CZ82OzJiTI9WDVsNXXcn/1mXe6ivROzgXMpdnS1vT4m7q7OjyIWZZig9NtCb45Q2t2YyIT05tKzYgcHiyVCRuJVychA5d92pv1iOTnv076SONcqUbxASNmnmX9PG9Wtb1bt84/4LTscYdNp/XSZVKrX/HB5+lrOf754QWMXF0bepf9Uf+2y+++7964fXjm+JHNGmooNT1c7Td91P1nu60MnPnmk329QCUf9XRu8aXNfbo0rzQP+fp9zk6iJI29vDUyAQiR1dXJmWw7QGicStlLOjSOhQZZyBi0hl7NpAN5FI7Ohcyk3dyjwlrs4OIoG4yrhLLCNL/NI3yDLzQ70bDeXeboXlOYEm1xofUDvCEPzwm9HIt3Gwb/5bempYWZ9fVly9tKxfqzY9fpuzYsOmjSvnTezz80+Nevx9MVFfBjN549tAZRdGVxI7MMbN1GTp0t5MZbo6O/oadmOZMRdVwdp4npUZma9zl+15S1xT76wbFeAnEbvV7Dp+zvLVS/4a1T2o27RjcQqVMv7wqDqulKBUeb8WI3bFKKQnhvg4OLq4SbwYa3Uv5eTcYNZjbWuXX59Q3UFzl/Bm7lseElenUkFrE3T1x9fEdFcU1reM40N8RdUnXEn9f4X7+d///md9EfjahS1t0fpclogrSp5QKBysyow3V47s2XXgzKNElmdBXpq8AqlUsuQX104f2rv34Kmrz5O5YytgQXhmgjt2WfKL62cO791z4Pj5e+8zjW/W3NfxP5N1dKCPwC1kh4WhfXWE0sRHJzfP/X14v9AuwcGdQweNnbls09QAN1d9R5ZMwC/nw5uFnTEQNEYZkt+f2Y1bKBgFLoSfVqZPp56M6DHm3zdcysaKHCpjlrdyEPpNumXzxmZFJtiC0qlPbr/WDCnRmTG3Tu3fuXPPwZNXnyeZq0fGfmzXQL/aXNjKZf6YNOnFjahjB/YePHPjZbLeoIvyS8LLZ+/TtXLAfCQWzlpjYhai+prTyrfLWrm4/7JRf43Y18Rn22t524UNbdG2JSg+sUEoFJ+6sJ+cSG/84Sdy7fwPy/gqj0Jm7+zqKKgcfkn3NKHTrkxuVG3YqfxpDGyxZP83sKu5bT3YrrHlsSJLXz1h17HZQhb/4bYsH+IqTgS+zsRsVJLs86OriH4Iv2g4UMsaOZ0Z9+j6+eMH/t28eun82XN2G02IYr3mKw/CLr4SoP7lEAr6NPDdRgSUH/4J8XJsPL8guzLQSRuCxATlFbTsMXMDkife2hzmLxHXjrhr9oU5+8SIwJlGvvdtVBpe0RRZ+rk3JvqJq4w8y7HQk1fmEajEEfgaE7NNYZVx64PcSrVZwemEVS8ZxeNF7atXKu/hqPbQybY0RC+wbb7CLmzDURMLhIItaSKufAKKl0taulUYeqIAOzCmb+vEeLokSJGbT3kvZyHjncm3v9F23PkJab7QH7f37LSQ7x7LRhfb4GeRpU9/3NrZo/Sv/1qYxWmDIiKKYkbgK0zMFiVhtvwUVxh+ysyKJJNkpFGMR1TSvcc+a64yiYbHAdgFD0hWBIFQsAIWglpDIPfuzIbutf64Zna8gC1Cza68uhnUpKPfyBNmRzHoxCPDA8adK7J36iJLn045OriSV8AqG0yIZKsJHCveBApsYl9fLPrT9hAPp6YLrNLm9Me1AQ4k6dButfHU86/PkEEMsAsDHDb4AaFgA4iIgp2A7NW6DhXq/XFFbz0me0Cjo8q4vQOYta9Mr4JjxcCpx2LNzvhTPF8Z0mH2basFiVGqBf5ZZOnTSUcGV6s28CB6EwpcdyX9woKa2NeWW3Z/eh2Hsv0O5q/n4hVh5r6eEpLxJ5G/wIPXZdYGgl1YS8xyeAgFy4wQosAE6NRzvzeoGrLJ+rn9uQmPL0dduvcuVW/aOEc26LS4uPwNeDjCFObhIko/59HidtVaz79bZAKpMJkibr4ECm5ifFMwDUen7O9V2qH+rEdmBbzJdRpvmQLfUectG7XJxbwPwC54o7IiIISCFbAQtAAElIlR09u3jbiuW8NQgEhwiREBOvXEmLbdl90pSoFklCX8LCoC39rEFM/m+jt6he4yOxxoCkPxaGYdEUFyb7BieonVR2AXViPjdwGEAj9OCPU1BJS5uWZXLHxN3N/rtVJpYb6Wfa9US2y5v6GJZRwfXF5cc+ptK9ufMm5lGzFJis1ssGIL+rALW1A0iQNCwQQJDoAACIAACLASUL5Z2tJF0nGL5e1JFOnvH145dfzs7ZhMxju1Zis2UZ2Zj2zicoo1czhYWAQgFAqLLOIFARAAATsjkB01qrKo2vgrZncpVabeWhvWpmol/96T5i9dMDG0defIKwmnR5SjCIHvyEKdoGBntItPcSAUik9dICcgAAIgUIwJKGPXBbm5Bqw2t0lp9oOVIZUcXOqOPfExr+uATjw0qGmLZpWFBOkWuqfIFjEXY67FP2sQCsW/jpBDEAABECh6ArnXJ1QXVQo7q7/VnGGuZM/XdCgroLw6bzHYElXxdHZ9EUGQ4jarPug2yTS8FL+KMwEIheJcO8gbCIAACBQ2gaxXURdfWVyWRCdsC/FwbrH4NedGcjl3Z/k7k6Rz62VvjcKkb+/iSBKiOjNMnKzTn84vmzhqSN/QLr/8Mnp3nNF1hV1yxM+TAIQCT1AIBgIgAAJ2RyDr6dbBP7lSzm1WRJt/SMvu/1nHodzAI5xO2ZWvl/zsTBKUR+juVKNuA+n5Ub4CQuAbFmWyVIJOf3H+4LY/WrpRlGffQ3AKUkwbGIRCMa0YZAsEQAAECpVA9rNtw1s19PMREAQh9Jt4w+QprkudTtnXq7Rjo8innEsWvpwbWVFAEJSk5z7jaQiKhzMYDwpu3Xezqww6fnU7B9Kx/YZEI4GhSx/fipYAhELR8kfqIAACIFAEBBRPlnQJHLvvTfa75a2dSYKgvHvuM+4JyM+W2sdS6Z57Uzif5LlnRpQXEATp0umfZKNAytgVrRgPCq1XcIwsZO7vISFFjeYUZLPZ/CziS2ESgFAoTLqIGwRAAASKKQGlUvNETzvUz4di9lVpvohj/kH6scHlxbUj7nG7TZPfmuQnJAhC1GyRybbTaTu7uZKEsPb0B+zdEbILoysIBJXHXeGOvpgS/H6yBaHw/dQ1SgoCIAACpgRk9yJ+EjG7uVcdd5nFQYLyzZKfXTy6bDO3+diXPd2dmBgqjb1k/LjPPT2sLEUIyoedYx/aUDyZVU9EefQ5yL2YwjTLOPJtCUAofFveSA0EQAAEihkBZdz69q7M8INHt39NdnBQ+1iqPuGa2WURWTu7OpIE6fDLJuPRCfmdKX6MBwXtBIX0A0N7r9fbZpr+uCbAgXQMXJ+nQ3JeHZg+uEfPXj2DgwZsfmndtlPFDKsdZQdCwY4qE0UBARAAgYIQyDo13JeZYiD2n/vcYIBAGbs20M0taF2shTURl8IrCghS0u+IkZ5QPItsICJIcavlmhi+HBvS5Perer0Omft7SkhRg8hnCpVKFntsWt+wNXdSZe82hZQTwd9zQaqyUK6BUCgUrIgUBEAABEoQAcWTyIZi9eDByHN6ixRzr/1eXVx59Hm9Q+yFyjw2qCxl0qOQeSuyhURAEsze0mpxkHVieOPR5/TEhOzimIoCQaXwS9LM+ysHhEacTmB0ivzmn3W9fFvNuorhCHbc3/oohMK3Jo70QAAEQKDYEaA/bQ+RMHMa3XU7PtEJ/3SRuLRcauw/iS3z8meLWrpR7p3/+ZS36EGZeCmyS+DY3ev7lqEYJSBTqZQx6zs1n3Zbrz9B8WR2fRHpGjDx75GhYRseQhewoS0GxyAUikElIAsgAAIgUNQEvlwMr8IMP4jqz3qsnhsgu/dnbQffIceN/SJwZFT56UxEm/Jl/AfPWb128eS+Ae36L7maRKuUHw6MqCXxbT9lwfTeLdpOvZimt3qS/rg2wIEUOLo4S/yCBk1c+O/VWL3eBo50cPjbE4BQ+PbMkSIIgAAIFD8CylcLmzkyww++Q09kquiUvT1LOzWeZzhnwVKucz/eP/vf7t1Hrr5N15vrIP30+Pzh/47fjTda+KCZoOA/94UsK+bqljFNJALXBlMvZ1pKBOe/NQEIhW9NHOmBAAiAQLEkQKfu6+nNDD+4Bq2LfjynkWOZPgc4nTDZoAQyZgqk4IfxeZMb1c4YKM9Bx9VyIufB/kPWiRQb5AhRsBOAUGDngqMgAAIg8N0RkN78g3GcRIqqNWtSxqHuzAeFuT5Rvakk5ZG/xYPsUnglgbD6HzeZRDMPD25vxmP0d1czRVtgCIWi5Y/UQQAEQKD4EFDGrGrrQhIEQVCe3f4t1M0X6IR1jAeFoA3a6Y/ye9NqCUU/L41RqhRvVnTtvt5gr+riw+g7zAmEwndY6SgyCIAACHAQSD86sBxFkMIak24azSjguKCgh2UXRvsKDVw00InHx9YvX79/xMSewUO2vy3M3oyCZvo7vQ5C4TuteBQbBEAABNgIyB/Obujk2eWfePM+ltgute4Ynf7ucXS63iII5nJF+vtH959/YnElbV3kCG1LAhAKtqSJuEAABECgxBNQZKWkYZliia9GGxYAQsGGMBEVCIAACIAACNgbAQgFe6tRlAcEQAAEQAAEbEgAQsGGMBEVCIAACIAACNgbAQgFe6tRlAcEQAAEQAAEbEgAQsGGMBEVCIAACIAACNgbAQgFe6tRlAcEQAAEQAAEbEgAQsGGMBEVCIAACIAACNgbAQgFe6tRlAcEQAAEQAAEbEgAQsGGMBEVCIAACIAACNgbAQgFe6tRlAcEQAAEQAAEbEgAQsGGMBEVCIAACIAACNgbAQgFe6tRlAcEQAAEQAAEbEgAQsGGMBEVCIAACIAACNgbAQgFe6tRlAcEQAAEQAAEbEgAQsGGMBEVCIAACIAACNgbAQgFe6tRlAcEQAAEQAAEbEgAQsGGMBEVCIAACIAACNgbAQgFe6tRlAcEQAAEQAAEbEgAQsGGMBEVCIAACIAACNgbAQgFe6tRlAcEQAAEQAAEbEgAQsGGMBEVCIAACIAACNgbAQgFe6tRlAcEQAAEQAAEbEgAQsGGMBEVCIAACIAACNgbAQgFe6tRlAcEQAAEQAAEbEgAQsGGMBEVCIAACIAACNgbAQgFe6tRlAcEQAAEQAAEbEgAQsGGMBEVCIAACIAACNgbAQgFe6tRlAcEQAAEQAAEbEgAQsGGMBEVCIAACIAACNgbAQgFe6tRlAcEQAAEQAAEbEgAQsGGMBEVCIAACIAACNgbAQgFe6tRlAcEQAAEQAAEbEgAQsGGMBEVCIAACIAACNgbAQgFe6tRlAcEQAAEQAAEbEgAQsGGMBEVCIAACIAACNgbAQgFe6tRlAcEQAAEQAAEbEgAQsGGMBEVCIAACIAACNgbgf8Py+fHx7PSrQYAAAAASUVORK5CYII="}},"cell_type":"markdown","id":"55877bc1","metadata":{"papermill":{"duration":0.012589,"end_time":"2023-10-30T02:33:19.683655","exception":false,"start_time":"2023-10-30T02:33:19.671066","status":"completed"},"tags":[]},"source":["# Attention 注意力机制\n","\n","注意力机制就不完整介绍了。简单来说就是输入Query,Key,Value,得到每个Query对应Key的注意力值。\n","\n","在自注意力机制中，一句话的每个token输入将会对句子中的每个输入有一个注意力值。这个注意力值是由softmax产生的概率值，表示当前token对整句话中所有token的注意力（或是权重）。而这种情况下，Q,K,V都是同样的矩阵，大小为(Length,Hidden State)。\n","\n","![image.png](attachment:f2f93c55-50d9-4814-ae4b-556732252e64.png)\n"]},{"cell_type":"markdown","id":"af1ddb81","metadata":{"papermill":{"duration":0.011026,"end_time":"2023-10-30T02:33:19.706175","exception":false,"start_time":"2023-10-30T02:33:19.695149","status":"completed"},"tags":[]},"source":["attention函数接受几个参数：Q,K,V即输入的文本向量。\n","\n","d_k是Query最后一维的大小，也就是最后一个token的长度，意思是词嵌入的维度。\n","\n","scores是得到的注意力分数矩阵。\n","\n","**值得注意的一点，key矩阵的大小为(batch_size, sequence_length, embedding_dim)，key.transpose(-2, -1)的意思就是将后面两个维度进行交换，这样才能进行矩阵乘法。**\n","\n","接下来，查看是否使用了掩码mask，如果使用了，就将注意力分数矩阵中相应位置为0的位置替换为一个较大的负值。这样在softmax计算中，该token的权重会被下降。\n","\n","然后使用 F.softmax 函数对注意力分数矩阵 scores 进行softmax操作，以获得最终的注意力权重张量 p_attn。这个张量表示了每个查询位置对应的注意力权重。"]},{"cell_type":"markdown","id":"82f81090","metadata":{"papermill":{"duration":0.010906,"end_time":"2023-10-30T02:33:19.728461","exception":false,"start_time":"2023-10-30T02:33:19.717555","status":"completed"},"tags":[]},"source":["_attention函数的输出是两个部分：_\n","\n","_自注意力输出（Self-Attention Output）： 这是通过将注意力权重矩阵 p_attn 与值张量 value 相乘得到的。它表示了每个查询位置对应的自注意力加权值，即模型在处理输入序列时，为每个位置分配的权重。这个自注意力输出包含了位置间的关联性信息。_\n","\n","_注意力权重张量（Attention Weights Tensor）： 这是由 softmax 操作得到的注意力权重张量 p_attn。它表示了每个查询位置对应的注意力权重分布，即模型关注输入序列中的哪些位置。这个注意力权重张量用于可视化或进一步的分析。_\n","\n","_注意，其中的自注意输出将被输入下一层。_"]},{"cell_type":"code","execution_count":7,"id":"f5dfe398","metadata":{"execution":{"iopub.execute_input":"2023-10-30T02:33:19.752622Z","iopub.status.busy":"2023-10-30T02:33:19.752244Z","iopub.status.idle":"2023-10-30T02:33:19.759988Z","shell.execute_reply":"2023-10-30T02:33:19.759164Z"},"papermill":{"duration":0.022437,"end_time":"2023-10-30T02:33:19.762109","exception":false,"start_time":"2023-10-30T02:33:19.739672","status":"completed"},"tags":[]},"outputs":[],"source":["def attention(query, key, value, mask=None, dropout=None):\n","    \"Compute 'Scaled Dot Product Attention'\"\n","\n","    #首先取query的最后一维的大小，对应词嵌入维度\n","    d_k = query.size(-1)\n","    #按照注意力公式，将query与key的转置相乘，这里面key是将最后两个维度进行转置，再除以缩放系数得到注意力得分张量scores\n","    scores = torch.matmul(query, key.transpose(-2, -1)) / math.sqrt(d_k)\n","    \n","    #接着判断是否使用掩码张量\n","    if mask is not None:\n","        #使用tensor的masked_fill方法，将掩码张量和scores张量每个位置一一比较，如果掩码张量则对应的scores张量用-1e9这个置来替换\n","        scores = scores.masked_fill(mask == 0, -1e9)\n","        \n","    #对scores的最后一维进行softmax操作，使用F.softmax方法，这样获得最终的注意力张量\n","    p_attn = F.softmax(scores, dim = -1)\n","    \n","    #之后判断是否使用dropout进行随机置0\n","    if dropout is not None:\n","        p_attn = dropout(p_attn)\n","    \n","    #最后，根据公式将p_attn与value张量相乘获得最终的query注意力表示，同时返回注意力张量\n","    return torch.matmul(p_attn, value), p_attn"]},{"cell_type":"markdown","id":"bd584e2b","metadata":{"papermill":{"duration":0.010976,"end_time":"2023-10-30T02:33:19.784464","exception":false,"start_time":"2023-10-30T02:33:19.773488","status":"completed"},"tags":[]},"source":["# Multi-Headed Attention 多头注意力机制\n","\n","多头注意力有点像CV中的卷积核，它将词汇的Embedding根据不同的注意力头分成不同的部分，并分别计算每个注意力头的注意力值。这相当于根据不同语义的主题分别计算注意力。\n","\n","在计算时，Q,K,V被均分成数个矩阵，随后进入线性层实例化。\n","\n","_前向传播时，一样是输入Q,K,V矩阵，最终输出是经过线性映射后的张量，其大小是 (batch_size, sequence_length, d_model)。_\n"]},{"cell_type":"code","execution_count":8,"id":"2b00c963","metadata":{"execution":{"iopub.execute_input":"2023-10-30T02:33:19.808726Z","iopub.status.busy":"2023-10-30T02:33:19.808343Z","iopub.status.idle":"2023-10-30T02:33:19.821478Z","shell.execute_reply":"2023-10-30T02:33:19.820305Z"},"papermill":{"duration":0.028104,"end_time":"2023-10-30T02:33:19.82381","exception":false,"start_time":"2023-10-30T02:33:19.795706","status":"completed"},"tags":[]},"outputs":[],"source":["class MultiHeadedAttention(nn.Module):\n","    def __init__(self, h, d_model, dropout=0.1):\n","        #在类的初始化时，会传入三个参数，h代表头数，d_model代表词嵌入的维度，dropout代表进行dropout操作时置0比率，默认是0.1\n","        super(MultiHeadedAttention, self).__init__()\n","        #在函数中，首先使用了一个测试中常用的assert语句，判断h是否能被d_model整除，这是因为我们之后要给每个头分配等量的词特征，也就是embedding_dim/head个\n","        assert d_model % h == 0\n","        #得到每个头获得的分割词向量维度d_k\n","        self.d_k = d_model // h\n","        #传入头数h\n","        self.h = h\n","        \n","        #创建linear层，通过nn的Linear实例化，它的内部变换矩阵是embedding_dim x embedding_dim，然后使用，为什么是四个呢，这是因为在多头注意力中，Q,K,V各需要一个，最后拼接的矩阵还需要一个，因此一共是四个\n","        self.linears = clones(nn.Linear(d_model, d_model), 4)\n","        #self.attn为None，它代表最后得到的注意力张量，现在还没有结果所以为None\n","        self.attn = None\n","        self.dropout = nn.Dropout(p=dropout)\n","        \n","    def forward(self, query, key, value, mask=None):\n","        #前向逻辑函数，它输入参数有四个，前三个就是注意力机制需要的Q,K,V，最后一个是注意力机制中可能需要的mask掩码张量，默认是None\n","        if mask is not None:\n","            # Same mask applied to all h heads.\n","            #使用unsqueeze扩展维度，代表多头中的第n头\n","            mask = mask.unsqueeze(1)\n","        #接着，我们获得一个batch_size的变量，他是query尺寸的第1个数字，代表有多少条样本\n","        nbatches = query.size(0)\n","        \n","        # 1) Do all the linear projections in batch from d_model => h x d_k \n","        # 首先利用zip将输入QKV与三个线性层组到一起，然后利用for循环，将输入QKV分别传到线性层中，做完线性变换后，开始为每个头分割输入，这里使用view方法对线性变换的结构进行维度重塑，多加了一个维度h代表头，这样就意味着每个头可以获得一部分词特征组成的句子，其中的-1代表自适应维度，计算机会根据这种变换自动计算这里的值，然后对第二维和第三维进行转置操作，为了让代表句子长度维度和词向量维度能够相邻，这样注意力机制才能找到词义与句子位置的关系，从attention函数中可以看到，利用的是原始输入的倒数第一和第二维，这样我们就得到了每个头的输入\n","        query, key, value = \\\n","            [l(x).view(nbatches, -1, self.h, self.d_k).transpose(1, 2)\n","             for l, x in zip(self.linears, (query, key, value))]\n","\n","        # 2) Apply attention on all the projected vectors in batch. \n","        # 得到每个头的输入后，接下来就是将他们传入到attention中，这里直接调用我们之前实现的attention函数，同时也将mask和dropout传入其中\n","        x, self.attn = attention(query, key, value, mask=mask, \n","                                 dropout=self.dropout)\n","\n","        # 3) \"Concat\" using a view and apply a final linear. \n","        # 通过多头注意力计算后，我们就得到了每个头计算结果组成的4维张量，我们需要将其转换为输入的形状以方便后续的计算，因此这里开始进行第一步处理环节的逆操作，先对第二和第三维进行转置，然后使用contiguous方法。这个方法的作用就是能够让转置后的张量应用view方法，否则将无法直接使用，所以，下一步就是使用view重塑形状，变成和输入形状相同。  \n","        x = x.transpose(1, 2).contiguous() \\\n","             .view(nbatches, -1, self.h * self.d_k)\n","        #最后使用线性层列表中的最后一个线性变换得到最终的多头注意力结构的输出\n","        return self.linears[-1](x)"]},{"attachments":{"9d096ba6-5ad8-4697-907a-e25dedf7b525.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAZEAAABDCAIAAADSwq+QAAATlUlEQVR4Ae2ceVQT1xfHCSEBRQMFd1z5Kait2qpYN1yqtmoVRcUWF1xQUXEBQZQtSnFXVER/dUfEBTfUIuJRUUFBRI/IIssBAT2s/SlwWE6Sk8yZ35msM5lJMomxJvTmH2fevOXez7vznffuDJqg8AMCQAAIGA8BE+MxFSwFAkAACKCgWRAEQAAIGBMB0Cxjmi2wFQgAAdAsiAEgAASMiQBoljHNFtgKBIAAaBbEABAAAsZEADTLmGYLbAUCQAA0C2IACAABYyIAmmVMswW2AgEgAJoFMQAEgIAxEQDNMqbZAluBABAAzYIYAAJAwJgIgGYZ02yBrUAACIBmQQwAASBgTARAs4xptsBWIAAEQLMgBoAAEDAmAqBZxjRbYCsQAAKgWRADQAAIGBMB0Cxjmi2wFQgAAdAsiAEgAASMiQBoljHNFtgKBIAAaBbEABAAAsZEADTLmGYLbAUCQAA0C2IACAABYyIAmmVMswW2AgEgAJoFMQAEgIAxEQDNMqbZAluBABAAzYIYAAJAwJgIgGYZ02yBrUAACIBmKWJAVJF8/Um14lz9UX16/N1Sgfo6reaqBA3Savz55x35V4XLl8X7VTWruba8rFTdr6yqQYiiKN16SN27rNevxL/X2WX1hFsMqSvLzSsoKiktLS0pys8rqGwigkU+XN/gzn38idCIWEXprPnlwaWrY4oxA1v3T3s0uvMQfnyXk/u2oLjkXUlRQV5OdlENH+sMqSvLyckrKH6HFb/NzS6oapGOwasuzMnNLywuKS58m1f6SaT70PRb8quyHyfeuHT2xNnkMtoDqgqX1usxfaDa1fyamiV8HR2w3st9lB2LYcLsNMJ97UYf8W+D96olcycO7MBimA3c8kKA0q2HCnOv7AgN9HUbwjE1MbUavy8HtwwSZl8IXuvm1JVlamk/0cNnTwIh2vhZ+2bOjsjTUn9E5TGLfw19pqR+2k2AwdemQiN8/zDS33tTCDfIx9svMrlCS26qfUZqU05w10/tw2IwWN3GePhyj6fUIphkpZ/Z5jvL0YJhYtrRaYEvN+pBhUQsRMXx25aN78k2bWs/0SMgJksmZaqH+PwrSOXDo1zvST2YrKHbcrRwnTJcWrXHn8+aqoevqVkSewTp/o5mJuYzzynd+sin1KCR3Rdck0Uh3XqoICNkgauLPcuUM2Z3tvgprXC8OWHVdG6WcpyJiiKnjQvJxCmcooX6I1FhxC8Twl4rjaK+jVFdpUAjehc9/9sxwSn1Ykfqn3HHDZx79h3t9YZm9/mPvHszmb3WPCRiFWQEDDAz7bTkNk+pC/7Tra6bHnykv0JWao+dingtxNEo6uCLBM8392f3WpusVSNUVbi0Xo/xzPR1bACalbFloJmJuUtMM8mn5gSv3w+VSWNRQLMeKsgIXXskJ3GNA8u03ahwop4InoesPfpBKbrr4pc4zjxVpVRKsoa6oO7qQodZZyp1a0zdpQGVktEgVRfmde254q5itpoTPXt0dDmjTFV3L4RvuD+wGNbu8seVpCvhq5DBLEbb2bHEhxtSHu0T+qBB9+Gwls0XfxvLfaP8LFPdp6j0gLP5N/PjtB6WOlxas8eqIep6xaA1C/37zLotT6TLHzWaRagn1qyo96L6B+v7s00tR2x/iXsuU2gWUn3GpadrzEddCTbf9Og1bn+xHtcZulqi93YUaJDyyIltLaaf+oQbrDFmVlsL5wMl+kKAfIicyDYxn3oCPydIbfymYV2YDPaEwwR1/JQQ5H9Jx+eNwofGaNdhW1/Q16xPMbM5Fj9FESxR9KbuiDJcWrXH6mjodM0QNas+ft9RSZ5AWHAz/rVKzaKuJ9UsBEUbHvkMZDPaDgvNkO0vUQrN+vvMDNufjlBHn6D+fUFhVbN0EcX/WFr4vp4U2J/OzbJxPqDPzRF5JkUtdVVlJZWNmCXChorCt8U1cp9QhPfxXX5hRQOFZojq3mU+Srqfnl/dosNKkAJN42U3a2ZH4v6Mf3eFHZPjGivZLJKNVy7RSLU51rUtgzVyV6HCo+anOzYfj3K3MWV9j9/b857v3HAUV015KLrnWmoW745nN/b32MKMV1te0agwk8Z4lOHSqj2mAUWrKgaoWcLMkPnBGaTkEmmdpaKeQrNQtDHFf5A5o80PQWnSrQxZs3gJy+wcfFJJwyHVDyP8AnYdP3Nw5eRJAXdLUiP9Ag+fi/QaPdTjYjkBsfANd6iNa2wdoVCPJ/wnO2aO6mvNNO287GbBrT2B2yNOnj8dOmPg4AWn8lpqHh0JCtl9LObc3oU//Gfitic43Wh6eSJg067TCanpj+LC542a6H0+R7qrEqQdWjpn8tC+Pey6dXOYfgjL7zUl+fzQw657D/sBTr9wkyUrUwo0osJdP7KYfdY9wudxBE99HcxYw8JySXJOokCPKu+OZ1dTM0f/dNmsCPOO+B3Kbnq83p7J7O0tTyKJCo9t3JmBW0aTxqNboJ1mYaHI7jZ7647A7ZFnz0V4TpkWeJd2doAyXFq1x3QngW49Q9Es1pBlB6Oioo4c3Ok/b7Dtt1tUaZbmenjNQtGmZ1uGWDAshgQ8Fd+vJM0SFe8d3e6nKOWIE+b/d33IPeyVFYr+ffznth37ukUVi/gPVvdkmvX1SSXSbTjnwhkU9JJYKD9DqhLDPT08Fmv6eSzbdF7li0vMTHNb5wXbL5RIhAEpPzTenDPGbUtkplSOG+Lm21iMlW9SRSUR4217/bznhURdmlN8HC36rk9WrM5EFTc8HcxtXU6/F3spyNg6atjK83mKPBUVGvGDw6zvxhSZmmBuCtL9+5uZOfg+xRfK/ccd0KUqSN/c38y0y/I7EjlCPpzfEv6kCRVmbxuKy3MhVZc3B9/Rz6NCK80Slewbw2Z2mRqZLaFZf3HuN3YeN3GPC5zP5ENpuBAEvnV7TGbwWSUGo1mjt95OSkpKvHX52OZJPQep1iyN9YiahaLN6cFD2zDMv9v0pBEl7w35j9fZW7vFKW5lMc2mBP+1sdWS3RS2tGBzZp3DkitNb+/G3StQTrzy7njadfK4pWoehLVv01JSnmj6pTzN+qAQDKXOkOpjk9j4JQbKi19ow7SaHSNPLAnS/B1ZdiuTpCsgUdnpOXbWDuvvS/vk3V7aifVtYCb+Vml+ETbSqsOkQ7l8fu7R1f7xFYT9IxUa8ctbkmY939zfjGm/XpZ5VLJdfkqbqqhw90gs235e/Jypu8sNvlGDffFQcQSX52pM5vrFSPRWPgJ2wH9/78Afl4q02q+hWmnWx2iX9hbjD5bKhuAnrbRj9VzzQLb21GCBNFwIy0NdPW4quH2Iu8V33Yat+6++0eLbQlR/HutuAmHatDgxFM3CvTcU5u7wCFG1ztJcT1mzULTlBXd4WwZ7wMbkBr7ye0NewvKunRbfJAQQigprKmtkN3f9hTlW5s4R8gglw+U/8u7Tfs5F8gX9lYg1iz0xUpF2493y6MjCbZ9QbAnE6rz0LyVXUGFd0dO/Lp2N2uD8DUtpgYSiwuKTM7u0GzR70Yr9svWa3GgqNMKXwYNYzD7exL3hs00OZmaO/mka1ln0qX48OdXchD3h8HsE5b3YF3BKmt5vvuBqKc1zCbIObDxM3IwiHx9F+vkFbHUb1OY/6vQTqU7cobzwdXfu1WHwjEWEtbDHMr/zb2VRIKeCoi0Jy7qwB4e8ll1Cak9MbcPs6nmHR88CabgQH5O6eNzwONyLe7uoQYQKqpJDnTt1dzleIDMKZy+KauUxoaXkRLXHtE2g6FXnIgPULLTpwelYcl6VlM9SUY+sWSjKexX2YztTlqN34v1g4rcOWP7Yer7yOguHk39vVXf2t2rfKfEfrull7RaHa6T3Q4lmTTomXfuhKCrWLLxdYs3qtFTx9ZKo8tEBzynOUz3DziZlFpff8LQjaxaKIpWnZ1qz+iyNV3QttZ4KDfL+0Hi2aTfPRNmaAqvLT/buzWSN3K3F2kYDVd41d2sGlm3nFR3ffOCVbDBeojTPxSs9vemPVOpVacvleVZqNQsVVuekJCc/xP8SAsc5LDx+D1/0MPlxZhnxswoxGOHLoO/Ydp535c8GXqKnHZM9ao/cfQ0WSMOFqFk6eFwX49p3TOhD6ZaUn+LTD7/YI4Tgl/L4f/RNINjzeSeGqFnUHpE1i7oelWahKD9r5+j2pqw+YyYsJnyfhX0baDmN8Oqe0K0wM/A7djdPaWYFRfl1n0i3Cu/m4k7dVyYR2ilOkOqk3auXL1+m6bd8xZaL+ZSPSexvV7C9IVsbzaq7t2GgZadpR/OlNzz/vld3qWYhuC1gY9pOz4AIvxFWXX49XkQcnRIN/75XD2Ybl3P4/XFLnBuH2dHjFvEuVBAgH2miKvvGMu5CcNhDxVCC55sHmJl2WXbhalBgvCTZSO5bg2KQG2AlWuwNWy7Pa2/56xn5dxiCVJ9+ZhYjd72VbRVRDRZIw0UmxBKDdPCYl77XbW74Yyke0dvw4Syr+VfoToJePP48E6inQnNp69SsNVHkRIcge48zx9SEsMFCUaTq6KR2P+Lfqov/ui3vQeKrGhGKior2jGK3m3lWmjUSFRxcHf6SeGujSO2fU7AuVMEWkB/rhOe59ETFY13Sq7aahdScnGZp1hv3nTYvfpGtWLN4twKDpaYilTc2rztZIECFhX9O72w1gvscv66gRIM23FzSlT10O+5vVkRFu0eybeZdlGXWkMby/NI6+R0sx4LU0aYqfMP9nsWwdJzgH4f/9AobicVg9RvlfUr1pyUaFENuDuFAmzs4br51d3naEG24u6oPu7fnbdxn+OotkIULEdDneIx5IiqKGN++m/sVVUpO8BY70avH4t61NoFkE80CA9AsLBtiYq5mrSNxRUCzHlJz4bdR65Pxd5+UhSD3wARrc3xSSPLSa0A34vdGTTcWdTRlOYW/FfJfh03o2F4WoUjFNX//S4qUkqzblI0OvVbdIz43aeKnWw35cHgCm/A5Je/6gm9Yjn6KHBLGh2UrTc0h1Sent2PaLU+QYkCqb2517snu4XWf33g5kIsNyy+/7Tt6yq58yb2DVMTM6Wzeb/n1D4p7SZDur4wGA/YyzMna6Q95LklUtN/Zemjwc+kDXlSw39mSwXb0eaz0xNeGqthdBmfCQaUUAZb1YVgM2ybfLVLwU68YFA2wIi3uYKT8v790m3VOrNBI7V8rHWyGBSQT0t/qLRBQhsvneIyi/JyDk3sP90uirVh69hgLJ+1NUDEXGou/pmYJ0g57LnKfObyHFYdjZdtvrOuCxYuXRpDfmNOthwqe7f996gh7Ww7Hqkv/sTO8Tit/OyDMj5zx25/EzE1LwvIeg4PwiydBZrjzwGnbYi9F+G/6M/1FlOvoeTsv34jevcn3cCohOMV0RcV7x3Rxu6yfd+5U88VP3b/IdZyDDYdj6+DsunDvk8acU2vdpgzqyuFY93Sa6R5ys+JDfPDvM4b3tOJwug7+ed7a07lCtCX75JJhPQe6bou+cSPmIDf0ZFr+laX97JzcPH1PPvoryGWMY2eOZVubYSHPsMy5qODkHHtOW8v2Nn3HzFoQfk+SJCGjEdsnyI/2GDF+7cWs2sa/s6/4TnT6/YT0rT+2Rq28tMjexs5+IvcF0RmtqDbFuloPCXwuzxlJu+Jdc+/Qb819xW6ROIT4TL1iUDTAirTQLBTlZR1xm7biyNW4yA0zfloY8exv3F4b60ytBarCRXePBcXnl0+efyijTskMFa5Ki/XqsW4mqDdQ9dWvqVmqrfpHr/CSvR2cwnCbHewerit5mZFTIf0CvrkyNyMjr0r5DhJbiZRFTuqzOF7tbfSPuoMfTFhXmpX+LLOwVroIFNbXfFRa/uCrk46p0Egq8Soybpw4uC/i+LX0DxRcmq9t25FB6o4+VVFpcmJWI6kDpDI18QVuF0aqoEkxqBpgZVrdwVgDXk1+5ovcStmfSBD6VadZKsNFR48FRbEblm6/V4WtjoV5Vy7T/kN//XmsswkEaFqcgGahqDBn19hhm9N02twJc8LHOgXp5VtsLWbtH6uqIxrB813ca3S/sdSzMy2X5lrZr3+s3WwK0iJCr5B2/TpapsYC/YYLvyDa08XrWII4I/ogMXrtwu2vlJKtKl3Qk8efY4JK2zRcAM3CADWn+o+bc7pcq8U11g6pubJo9Eo9fYutYaa+0mUd0IjKYvzCH5Ner35xB4RvYrlb/VdO7mfdYfCsdQGBe26pztR/GWMoLMAPpN9waUj0smcxTBQ/RnvXC+S1Kd4AfR9/HRNAsyTz2Ji23XXZecL/A6hxgpGq614u/g/JKS6NLY2qgpZokIqroWF3lXM8RuXxlzH2XxIuXwYevlfQLBkN0ftbW1fuSqWdS29+eXjlhtgiDd9+y3o36n+1RWPUzn4h4/9F4fKFCMq7Bc2So8D2iB/pp6gFnz6J/2sYfPtWfKwNmlaMQWfX/mXhojMnGg1Bs2hAgipAAAgYDAHQLIOZCjAECAABGgRAs2hAgipAAAgYDAHQLIOZCjAECAABGgRAs2hAgipAAAgYDAHQLIOZCjAECAABGgRAs2hAgipAAAgYDAHQLIOZCjAECAABGgRAs2hAgipAAAgYDAHQLIOZCjAECAABGgRAs2hAgipAAAgYDAHQLIOZCjAECAABGgRAs2hAgipAAAgYDAHQLIOZCjAECAABGgRAs2hAgipAAAgYDIH/AyJCJyN8GSjxAAAAAElFTkSuQmCC"}},"cell_type":"markdown","id":"25d4ecae","metadata":{"papermill":{"duration":0.011058,"end_time":"2023-10-30T02:33:19.84615","exception":false,"start_time":"2023-10-30T02:33:19.835092","status":"completed"},"tags":[]},"source":["# Feed Forward Layer 前馈全连接层\n","\n","前馈全连接层就是简单的两个全连接层，在进行了Attention操作之后，encoder和decoder中的每一层都包含了一个全连接前向网络，对每个position的向量分别进行相同的操作，包括两个线性变换和一个ReLU激活输出。\n","\n","![image.png](attachment:9d096ba6-5ad8-4697-907a-e25dedf7b525.png)"]},{"cell_type":"code","execution_count":9,"id":"9ae57952","metadata":{"execution":{"iopub.execute_input":"2023-10-30T02:33:19.870282Z","iopub.status.busy":"2023-10-30T02:33:19.869876Z","iopub.status.idle":"2023-10-30T02:33:19.877665Z","shell.execute_reply":"2023-10-30T02:33:19.876451Z"},"papermill":{"duration":0.022729,"end_time":"2023-10-30T02:33:19.880164","exception":false,"start_time":"2023-10-30T02:33:19.857435","status":"completed"},"tags":[]},"outputs":[],"source":["class PositionwiseFeedForward(nn.Module):\n","    def __init__(self, d_model, d_ff, dropout=0.1):\n","        #初始化函数有三个输入参数分别是d_model，d_ff，和dropout=0.1，第一个是线性层的输入维度也是第二个线性层的输出维度，因为我们希望输入通过前馈全连接层后输入和输出的维度不变，第二个参数d_ff就是第二个线性层的输入维度和第一个线性层的输出，最后一个是dropout置0比率。\n","        super(PositionwiseFeedForward, self).__init__()\n","        self.w_1 = nn.Linear(d_model, d_ff)\n","        self.w_2 = nn.Linear(d_ff, d_model)\n","        self.dropout = nn.Dropout(dropout)\n","\n","    def forward(self, x):\n","        #输入参数为x，代表来自上一层的输出，首先经过第一个线性层，然后使用F中的relu函数进行激活，之后再使用dropout进行随机置0，最后通过第二个线性层w2，返回最终结果\n","        return self.w_2(self.dropout(F.relu(self.w_1(x))))"]},{"cell_type":"markdown","id":"312605d5","metadata":{"papermill":{"duration":0.010972,"end_time":"2023-10-30T02:33:19.902719","exception":false,"start_time":"2023-10-30T02:33:19.891747","status":"completed"},"tags":[]},"source":["# Layer Normalization 层归一化\n","\n","这个没什么好解释的，就是对每个句子的输入进行归一化。"]},{"cell_type":"code","execution_count":10,"id":"7540dcf6","metadata":{"execution":{"iopub.execute_input":"2023-10-30T02:33:19.926996Z","iopub.status.busy":"2023-10-30T02:33:19.926588Z","iopub.status.idle":"2023-10-30T02:33:19.935492Z","shell.execute_reply":"2023-10-30T02:33:19.934307Z"},"papermill":{"duration":0.023909,"end_time":"2023-10-30T02:33:19.937899","exception":false,"start_time":"2023-10-30T02:33:19.91399","status":"completed"},"tags":[]},"outputs":[],"source":["class LayerNorm(nn.Module):\n","    \"Construct a layernorm module (See citation for details).\"\n","    def __init__(self, feature_size, eps=1e-6):\n","        #初始化函数有两个参数，一个是features,表示词嵌入的维度,另一个是eps它是一个足够小的数，在规范化公式的分母中出现,防止分母为0，默认是1e-6。\n","        super(LayerNorm, self).__init__()\n","        #根据features的形状初始化两个参数张量a2，和b2，第一初始化为1张量，也就是里面的元素都是1，第二个初始化为0张量，也就是里面的元素都是0，这两个张量就是规范化层的参数。因为直接对上一层得到的结果做规范化公式计算，将改变结果的正常表征，因此就需要有参数作为调节因子，使其即能满足规范化要求，又能不改变针对目标的表征，最后使用nn.parameter封装，代表他们是模型的参数\n","        self.a_2 = nn.Parameter(torch.ones(feature_size))\n","        self.b_2 = nn.Parameter(torch.zeros(feature_size))\n","        #把eps传到类中\n","        self.eps = eps\n","\n","    def forward(self, x):\n","    #输入参数x代表来自上一层的输出，在函数中，首先对输入变量x求其最后一个维度的均值，并保持输出维度与输入维度一致，接着再求最后一个维度的标准差，然后就是根据规范化公式，用x减去均值除以标准差获得规范化的结果。\n","    #最后对结果乘以我们的缩放参数，即a2,*号代表同型点乘，即对应位置进行乘法操作，加上位移参b2，返回即可\n","        mean = x.mean(-1, keepdim=True)\n","        std = x.std(-1, keepdim=True)\n","        return self.a_2 * (x - mean) / (std + self.eps) + self.b_2"]},{"cell_type":"markdown","id":"d69f6620","metadata":{"papermill":{"duration":0.01158,"end_time":"2023-10-30T02:33:19.961404","exception":false,"start_time":"2023-10-30T02:33:19.949824","status":"completed"},"tags":[]},"source":["# Mask 掩码\n","\n","掩码的作用：在transformer中，掩码主要的作用有两个，一个是屏蔽掉无效的padding区域，一个是屏蔽掉来自“未来”的信息。Encoder中的掩码主要是起到第一个作用，Decoder中的掩码则同时发挥着两种作用。\n","\n","在Encoder中，Mask只是一个上三角矩阵，size是输入语句的大小，下三角部分被设置为0，意思是将未来的信息都设置为0。\n"]},{"cell_type":"code","execution_count":11,"id":"3785986c","metadata":{"execution":{"iopub.execute_input":"2023-10-30T02:33:19.986878Z","iopub.status.busy":"2023-10-30T02:33:19.98583Z","iopub.status.idle":"2023-10-30T02:33:19.992367Z","shell.execute_reply":"2023-10-30T02:33:19.991421Z"},"papermill":{"duration":0.021384,"end_time":"2023-10-30T02:33:19.994577","exception":false,"start_time":"2023-10-30T02:33:19.973193","status":"completed"},"tags":[]},"outputs":[],"source":["def subsequent_mask(size):\n","    #生成向后遮掩的掩码张量，参数size是掩码张量最后两个维度的大小，它最后两维形成一个方阵\n","\n","    \"Mask out subsequent positions.\"\n","    attn_shape = (1, size, size)\n","    \n","    #然后使用np.ones方法向这个形状中添加1元素，形成上三角阵\n","    subsequent_mask = np.triu(np.ones(attn_shape), k=1).astype('uint8')\n","\n","    #最后将numpy类型转化为torch中的tensor，内部做一个1- 的操作。这个其实是做了一个三角阵的反转，subsequent_mask中的每个元素都会被1减。\n","    #如果是0，subsequent_mask中的该位置由0变成1\n","    #如果是1，subsequect_mask中的该位置由1变成0\n","    return torch.from_numpy(subsequent_mask) == 0"]},{"cell_type":"markdown","id":"8f90742a","metadata":{"papermill":{"duration":0.068298,"end_time":"2023-10-30T02:33:20.074207","exception":false,"start_time":"2023-10-30T02:33:20.005909","status":"completed"},"tags":[]},"source":["# Decoder 解码器\n","\n","解码器和编码器结构差不多，不过前向传播中参数多了编码层的输入以及源数据的掩码和目标数据的掩码。\n","\n","它的作用就是根据编码器的结果以及上一次预测的结果，输出序列的下一个结果。"]},{"cell_type":"code","execution_count":12,"id":"5b2a3d4c","metadata":{"execution":{"iopub.execute_input":"2023-10-30T02:33:20.099281Z","iopub.status.busy":"2023-10-30T02:33:20.098577Z","iopub.status.idle":"2023-10-30T02:33:20.10626Z","shell.execute_reply":"2023-10-30T02:33:20.105182Z"},"papermill":{"duration":0.022954,"end_time":"2023-10-30T02:33:20.108661","exception":false,"start_time":"2023-10-30T02:33:20.085707","status":"completed"},"tags":[]},"outputs":[],"source":["#使用类Decoder来实现解码器\n","class Decoder(nn.Module):\n","    \"Generic N layer decoder with masking.\"\n","    def __init__(self, layer, N):\n","        #初始化函数的参数有两个，第一个就是解码器层layer，第二个是解码器层的个数N\n","        super(Decoder, self).__init__()\n","        #首先使用clones方法克隆了N个layer，然后实例化一个规范化层，因为数据走过了所有的解码器层后最后要做规范化处理。\n","        self.layers = clones(layer, N)\n","        self.norm = LayerNorm(layer.size)\n","        \n","    def forward(self, x, memory, src_mask, tgt_mask):\n","        #forward函数中的参数有4个，x代表目标数据的嵌入表示，memory是编码器层的输出，source_mask，target_mask代表源数据和目标数据的掩码张量，然后就是对每个层进行循环，当然这个循环就是变量x通过每一个层的处理，得出最后的结果，再进行一次规范化返回即可。\n","        for layer in self.layers:\n","            x = layer(x, memory, src_mask, tgt_mask)\n","        return self.norm(x)"]},{"cell_type":"markdown","id":"f0ae02f3","metadata":{"papermill":{"duration":0.011226,"end_time":"2023-10-30T02:33:20.131461","exception":false,"start_time":"2023-10-30T02:33:20.120235","status":"completed"},"tags":[]},"source":["# Decoder Layer 解码器层"]},{"cell_type":"code","execution_count":13,"id":"b7514eb1","metadata":{"execution":{"iopub.execute_input":"2023-10-30T02:33:20.156236Z","iopub.status.busy":"2023-10-30T02:33:20.155822Z","iopub.status.idle":"2023-10-30T02:33:20.166131Z","shell.execute_reply":"2023-10-30T02:33:20.165037Z"},"papermill":{"duration":0.025609,"end_time":"2023-10-30T02:33:20.168482","exception":false,"start_time":"2023-10-30T02:33:20.142873","status":"completed"},"tags":[]},"outputs":[],"source":["#使用DecoderLayer的类实现解码器层\n","class DecoderLayer(nn.Module):\n","    \"Decoder is made of self-attn, src-attn, and feed forward (defined below)\"\n","    def __init__(self, size, self_attn, src_attn, feed_forward, dropout):\n","        #初始化函数的参数有5个，分别是size，代表词嵌入的维度大小，同时也代表解码器的尺寸，第二个是self_attn，多头自注意力对象，也就是说这个注意力机制需要Q=K=V，第三个是src_attn,多头注意力对象，这里Q!=K=V，第四个是前馈全连接层对象，最后就是dropout置0比率\n","        super(DecoderLayer, self).__init__()\n","        self.size = size\n","        self.self_attn = self_attn\n","        self.src_attn = src_attn\n","        self.feed_forward = feed_forward\n","        #按照结构图使用clones函数克隆三个子层连接对象\n","        self.sublayer = clones(SublayerConnection(size, dropout), 3)\n"," \n","    def forward(self, x, memory, src_mask, tgt_mask):\n","        #forward函数中的参数有4个，分别是来自上一层的输入x，来自编码器层的语义存储变量memory，以及源数据掩码张量和目标数据掩码张量，将memory表示成m之后方便使用。\n","        \"Follow Figure 1 (right) for connections.\"\n","        m = memory\n","        #将x传入第一个子层结构，第一个子层结构的输入分别是x和self-attn函数，因为是自注意力机制，所以Q,K,V都是x，最后一个参数时目标数据掩码张量，这时要对目标数据进行遮掩，因为此时模型可能还没有生成任何目标数据。\n","        #比如在解码器准备生成第一个字符或词汇时，我们其实已经传入了第一个字符以便计算损失，但是我们不希望在生成第一个字符时模型能利用这个信息，因此我们会将其遮掩，同样生成第二个字符或词汇时，模型只能使用第一个字符或词汇信息，第二个字符以及之后的信息都不允许被模型使用。\n","        x = self.sublayer[0](x, lambda x: self.self_attn(x, x, x, tgt_mask))\n","        #接着进入第二个子层，这个子层中常规的注意力机制，q是输入x;k,v是编码层输出memory，同样也传入source_mask，但是进行源数据遮掩的原因并非是抑制信息泄露，而是遮蔽掉对结果没有意义的padding。\n","        x = self.sublayer[1](x, lambda x: self.src_attn(x, m, m, src_mask))\n","        \n","        #最后一个子层就是前馈全连接子层，经过它的处理后就可以返回结果，这就是我们的解码器结构\n","        return self.sublayer[2](x, self.feed_forward)"]},{"cell_type":"markdown","id":"4e3e5f1d","metadata":{"papermill":{"duration":0.012105,"end_time":"2023-10-30T02:33:20.192129","exception":false,"start_time":"2023-10-30T02:33:20.180024","status":"completed"},"tags":[]},"source":["# Output of model 模型输出\n","\n","模型的输出是对每一个时间的输出都加上一个线性层和一个softmax层，以输出预测的token。"]},{"cell_type":"code","execution_count":14,"id":"dda9d523","metadata":{"execution":{"iopub.execute_input":"2023-10-30T02:33:20.217184Z","iopub.status.busy":"2023-10-30T02:33:20.216782Z","iopub.status.idle":"2023-10-30T02:33:20.223795Z","shell.execute_reply":"2023-10-30T02:33:20.222751Z"},"papermill":{"duration":0.022309,"end_time":"2023-10-30T02:33:20.226112","exception":false,"start_time":"2023-10-30T02:33:20.203803","status":"completed"},"tags":[]},"outputs":[],"source":["#将线性层和softmax计算层一起实现，因为二者的共同目标是生成最后的结构\n","#因此把类的名字叫做Generator，生成器类\n","class Generator(nn.Module):\n","    \"Define standard linear + softmax generation step.\"\n","    def __init__(self, d_model, vocab):\n","        #初始化函数的输入参数有两个，d_model代表词嵌入维度，vocab.size代表词表大小\n","        super(Generator, self).__init__()\n","        #首先就是使用nn中的预定义线性层进行实例化，得到一个对象self.proj等待使用\n","        #这个线性层的参数有两个，就是初始化函数传进来的两个参数：d_model，vocab_size\n","        self.proj = nn.Linear(d_model, vocab)\n","\n","    def forward(self, x):\n","        #前向逻辑函数中输入是上一层的输出张量x,在函数中，首先使用上一步得到的self.proj对x进行线性变化,然后使用F中已经实现的log_softmax进行softmax处理。\n","        return F.log_softmax(self.proj(x), dim=-1)"]},{"cell_type":"markdown","id":"cfca4966","metadata":{"papermill":{"duration":0.011337,"end_time":"2023-10-30T02:33:20.249031","exception":false,"start_time":"2023-10-30T02:33:20.237694","status":"completed"},"tags":[]},"source":["# Model Architecture 模型构建"]},{"cell_type":"code","execution_count":15,"id":"04cadf17","metadata":{"execution":{"iopub.execute_input":"2023-10-30T02:33:20.27406Z","iopub.status.busy":"2023-10-30T02:33:20.273608Z","iopub.status.idle":"2023-10-30T02:33:20.289385Z","shell.execute_reply":"2023-10-30T02:33:20.288245Z"},"papermill":{"duration":0.031166,"end_time":"2023-10-30T02:33:20.291807","exception":false,"start_time":"2023-10-30T02:33:20.260641","status":"completed"},"tags":[]},"outputs":[],"source":["# Model Architecture\n","#使用EncoderDecoder类来实现编码器-解码器结构\n","class EncoderDecoder(nn.Module):\n","    \"\"\"\n","    A standard Encoder-Decoder architecture. \n","    Base for this and many other models.\n","    \"\"\"\n","    def __init__(self, encoder, decoder, src_embed, tgt_embed, generator):\n","        #初始化函数中有5个参数，分别是编码器对象，解码器对象,源数据嵌入函数，目标数据嵌入函数，以及输出部分的类别生成器对象.\n","        super(EncoderDecoder, self).__init__()\n","        self.encoder = encoder\n","        self.decoder = decoder\n","        self.src_embed = src_embed    # input embedding module(input embedding + positional encode)\n","        self.tgt_embed = tgt_embed    # ouput embedding module\n","        self.generator = generator    # output generation module\n","        \n","    def forward(self, src, tgt, src_mask, tgt_mask):\n","        \"Take in and process masked src and target sequences.\"\n","        #在forward函数中，有四个参数，source代表源数据，target代表目标数据,source_mask和target_mask代表对应的掩码张量,在函数中，将source source_mask传入编码函数，得到结果后与source_mask target 和target_mask一同传给解码函数\n","        memory = self.encode(src, src_mask)\n","        res = self.decode(memory, src_mask, tgt, tgt_mask)\n","        return res\n","    \n","    def encode(self, src, src_mask):\n","        #编码函数，以source和source_mask为参数,使用src_embed对source做处理，然后和source_mask一起传给self.encoder\n","        src_embedds = self.src_embed(src)\n","        return self.encoder(src_embedds, src_mask)\n","    \n","    def decode(self, memory, src_mask, tgt, tgt_mask):\n","        #解码函数，以memory即编码器的输出，source_mask target target_mask为参数,使用tgt_embed对target做处理，然后和source_mask,target_mask,memory一起传给self.decoder\n","        target_embedds = self.tgt_embed(tgt)\n","        return self.decoder(target_embedds, memory, src_mask, tgt_mask)\n","\n","\n","# Full Model\n","def make_model(src_vocab, tgt_vocab, N=6, d_model=512, d_ff=2048, h=8, dropout=0.1):\n","    \"\"\"\n","    构建模型\n","    params:\n","        src_vocab:\n","        tgt_vocab:\n","        N: 编码器和解码器堆叠基础模块的个数\n","        d_model: 模型中embedding的size，默认512\n","        d_ff: FeedForward Layer层中embedding的size，默认2048\n","        h: MultiHeadAttention中多头的个数，必须被d_model整除\n","        dropout:\n","    \"\"\"\n","    c = copy.deepcopy\n","    attn = MultiHeadedAttention(h, d_model)\n","    ff = PositionwiseFeedForward(d_model, d_ff, dropout)\n","    position = PositionalEncoding(d_model, dropout)\n","    model = EncoderDecoder(\n","        Encoder(EncoderLayer(d_model, c(attn), c(ff), dropout), N),\n","        Decoder(DecoderLayer(d_model, c(attn), c(attn), c(ff), dropout), N),\n","        nn.Sequential(Embeddings(d_model, src_vocab), c(position)),\n","        nn.Sequential(Embeddings(d_model, tgt_vocab), c(position)),\n","        Generator(d_model, tgt_vocab))\n","    \n","    # This was important from their code. \n","    # Initialize parameters with Glorot / fan_avg.\n","    for p in model.parameters():\n","        if p.dim() > 1:\n","            nn.init.xavier_uniform_(p)\n","    return model"]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.12"},"papermill":{"default_parameters":{},"duration":9.021907,"end_time":"2023-10-30T02:33:21.125463","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2023-10-30T02:33:12.103556","version":"2.4.0"}},"nbformat":4,"nbformat_minor":5}