model_sample.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# まずはライブラリをインポート"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import transformers\n",
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 最大文字数を指定\n",
    "文ごとに文字数が変わるが、学習・推論する時には一定サイズのベクトルを渡す必要がある。  \n",
    "そのため、最大文字数を決めておく必要がある。  \n",
    "今回は192"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LENGTH = 192"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# エンコードする文字ベクトルを作成\n",
    "詳しくは\n",
    "[ここ](https://huggingface.co/transformers/model_doc/roberta.html)  \n",
    "「This is a pen.」と「I am a man」を1つの文章として、トークンIDへ変換。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "[0, 713, 16, 10, 7670, 4, 2, 2, 100, 524, 10, 313, 2]"
     },
     "metadata": {},
     "execution_count": 3
    }
   ],
   "source": [
    "tokenizer = transformers.AutoTokenizer.from_pretrained(\"roberta-base\")\n",
    "text = \"This is a pen.\"\n",
    "text2 = \"I am a man\"\n",
    "\n",
    "ids = tokenizer.encode(text)\n",
    "ids2 = tokenizer.encode(text2)\n",
    "\n",
    "token_ids = tokenizer.build_inputs_with_special_tokens(ids, ids2)\n",
    "token_ids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# パディング＆Attention Maskの作成\n",
    "文字数の最大値を指定したが、たいていのケースで最大文字数に達しない。  \n",
    "そのため、使わない領域として、***特別な文字***([参照](https://qiita.com/ishikawa-takumi/items/5fc45ddd121b23db5de9))で埋めてあげる必要がある。  \n",
    "また、有効な領域だとモデル示すために、Attention maskというものも用意する。  \n",
    "具体的には、有効な文字があるところが「１」、パティングしたところが「０」となる。  \n",
    "maskのサイズも文字最大数と同じ。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "mask = [1] * len(token_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "padding_length = MAX_LENGTH - len(token_ids)\n",
    "if padding_length > 0:\n",
    "    token_ids = token_ids + ([1] * padding_length)\n",
    "    mask = mask + ([0] * padding_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modelの生成\n",
    "RoBERTaを利用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "RobertaModel(\n  (embeddings): RobertaEmbeddings(\n    (word_embeddings): Embedding(50265, 768, padding_idx=1)\n    (position_embeddings): Embedding(514, 768, padding_idx=1)\n    (token_type_embeddings): Embedding(1, 768)\n    (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n    (dropout): Dropout(p=0.1, inplace=False)\n  )\n  (encoder): BertEncoder(\n    (layer): ModuleList(\n      (0): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=768, out_features=768, bias=True)\n            (key): Linear(in_features=768, out_features=768, bias=True)\n            (value): Linear(in_features=768, out_features=768, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=768, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=768, out_features=3072, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=3072, out_features=768, bias=True)\n          (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (1): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=768, out_features=768, bias=True)\n            (key): Linear(in_features=768, out_features=768, bias=True)\n            (value): Linear(in_features=768, out_features=768, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=768, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=768, out_features=3072, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=3072, out_features=768, bias=True)\n          (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (2): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=768, out_features=768, bias=True)\n            (key): Linear(in_features=768, out_features=768, bias=True)\n            (value): Linear(in_features=768, out_features=768, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=768, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=768, out_features=3072, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=3072, out_features=768, bias=True)\n          (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (3): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=768, out_features=768, bias=True)\n            (key): Linear(in_features=768, out_features=768, bias=True)\n            (value): Linear(in_features=768, out_features=768, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=768, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=768, out_features=3072, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=3072, out_features=768, bias=True)\n          (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (4): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=768, out_features=768, bias=True)\n            (key): Linear(in_features=768, out_features=768, bias=True)\n            (value): Linear(in_features=768, out_features=768, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=768, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=768, out_features=3072, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=3072, out_features=768, bias=True)\n          (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (5): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=768, out_features=768, bias=True)\n            (key): Linear(in_features=768, out_features=768, bias=True)\n            (value): Linear(in_features=768, out_features=768, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=768, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=768, out_features=3072, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=3072, out_features=768, bias=True)\n          (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (6): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=768, out_features=768, bias=True)\n            (key): Linear(in_features=768, out_features=768, bias=True)\n            (value): Linear(in_features=768, out_features=768, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=768, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=768, out_features=3072, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=3072, out_features=768, bias=True)\n          (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (7): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=768, out_features=768, bias=True)\n            (key): Linear(in_features=768, out_features=768, bias=True)\n            (value): Linear(in_features=768, out_features=768, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=768, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=768, out_features=3072, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=3072, out_features=768, bias=True)\n          (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (8): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=768, out_features=768, bias=True)\n            (key): Linear(in_features=768, out_features=768, bias=True)\n            (value): Linear(in_features=768, out_features=768, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=768, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=768, out_features=3072, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=3072, out_features=768, bias=True)\n          (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (9): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=768, out_features=768, bias=True)\n            (key): Linear(in_features=768, out_features=768, bias=True)\n            (value): Linear(in_features=768, out_features=768, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=768, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=768, out_features=3072, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=3072, out_features=768, bias=True)\n          (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (10): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=768, out_features=768, bias=True)\n            (key): Linear(in_features=768, out_features=768, bias=True)\n            (value): Linear(in_features=768, out_features=768, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=768, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=768, out_features=3072, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=3072, out_features=768, bias=True)\n          (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n      (11): BertLayer(\n        (attention): BertAttention(\n          (self): BertSelfAttention(\n            (query): Linear(in_features=768, out_features=768, bias=True)\n            (key): Linear(in_features=768, out_features=768, bias=True)\n            (value): Linear(in_features=768, out_features=768, bias=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n          (output): BertSelfOutput(\n            (dense): Linear(in_features=768, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (intermediate): BertIntermediate(\n          (dense): Linear(in_features=768, out_features=3072, bias=True)\n        )\n        (output): BertOutput(\n          (dense): Linear(in_features=3072, out_features=768, bias=True)\n          (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n          (dropout): Dropout(p=0.1, inplace=False)\n        )\n      )\n    )\n  )\n  (pooler): BertPooler(\n    (dense): Linear(in_features=768, out_features=768, bias=True)\n    (activation): Tanh()\n  )\n)"
     },
     "metadata": {},
     "execution_count": 6
    }
   ],
   "source": [
    "model = transformers.AutoModel.from_pretrained(\"roberta-base\")\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modelを使ってエンコード\n",
    "## まずはIDとmaskをmodelに入力できる型に変換"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "token_ids_tensor = torch.tensor([token_ids], dtype=torch.long)\n",
    "mask_tensor = torch.tensor([mask], dtype=torch.long)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modelで処理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "(tensor([[[-0.0977,  0.0892, -0.0099,  ..., -0.0847, -0.0366, -0.0534],\n          [ 0.0274,  0.2033,  0.1038,  ...,  0.2127, -0.0719, -0.0050],\n          [ 0.1462,  0.1765,  0.1418,  ..., -0.2564, -0.0259,  0.0907],\n          ...,\n          [-0.0685,  0.0695,  0.0897,  ...,  0.0282, -0.0272,  0.0324],\n          [-0.0511,  0.0756,  0.0510,  ...,  0.0246, -0.0135,  0.0227],\n          [-0.0344,  0.0882,  0.0317,  ...,  0.0632, -0.0208,  0.0009]]],\n        grad_fn=<NativeLayerNormBackward>),\n tensor([[-1.0725e-02, -2.2089e-01, -2.0918e-01, -6.8649e-02,  1.2540e-01,\n           2.1667e-01,  2.7022e-01, -7.1740e-02, -9.9609e-02, -1.6666e-01,\n           2.6844e-01, -1.6258e-02, -1.1163e-01,  1.3448e-01, -1.3481e-01,\n           4.8152e-01,  2.2144e-01, -5.0482e-01,  6.7738e-02, -7.9790e-03,\n          -2.9393e-01,  6.0414e-02,  4.7367e-01,  3.3544e-01,  1.1456e-01,\n           5.4199e-02, -1.6571e-01, -1.8260e-02,  1.6509e-01,  2.5049e-01,\n           2.9651e-01,  4.9777e-02,  1.2795e-01,  2.4528e-01, -2.5723e-01,\n           5.8533e-02, -3.2723e-01,  1.4764e-02,  2.7094e-01, -1.6650e-01,\n          -7.8529e-02,  1.7682e-01,  2.1336e-01, -1.5592e-01, -9.8193e-02,\n           4.1889e-01,  2.6822e-01,  3.5542e-02, -1.4788e-01, -1.0083e-01,\n          -3.3928e-01,  3.5642e-01,  3.1385e-01,  1.9669e-01, -5.6032e-02,\n           4.0987e-02, -1.1498e-01,  2.7192e-01, -8.1781e-02, -8.6911e-02,\n          -1.1628e-01, -2.2389e-01,  1.3456e-03, -6.4918e-02,  2.7829e-02,\n          -1.2634e-01,  1.0480e-01, -1.5357e-01, -1.3970e-01,  6.0855e-02,\n          -1.1290e-01,  1.5020e-01,  1.7302e-01, -2.9930e-01, -2.9682e-01,\n           7.0488e-02, -6.1358e-01, -1.0293e-01,  3.4108e-01,  4.4635e-01,\n          -9.0948e-02,  2.2626e-01,  4.0433e-02,  2.2871e-01, -1.6427e-02,\n          -5.5887e-02, -3.2147e-02, -1.4223e-01,  1.6809e-01,  2.7001e-01,\n          -2.0783e-01, -4.2115e-01,  5.5891e-02,  3.7251e-02, -8.8319e-02,\n           2.1002e-02, -4.8602e-03, -7.7421e-02, -1.8222e-01, -1.8588e-01,\n           8.1507e-02, -2.5709e-01, -1.5080e-01,  2.6354e-01, -8.7523e-03,\n          -1.8510e-01, -8.0861e-03,  3.0705e-01,  5.9894e-02, -9.4996e-02,\n          -2.0243e-01,  4.5008e-01,  3.4034e-01, -1.0336e-02, -1.8405e-04,\n           1.7736e-01,  1.5349e-01, -2.8624e-01,  4.4453e-01, -3.1194e-01,\n          -7.9183e-03, -8.8667e-02,  1.1373e-01,  1.9431e-01, -2.1270e-01,\n           2.8221e-01,  1.5266e-01,  2.9722e-01,  2.0833e-01,  1.1711e-01,\n          -4.8050e-02,  1.2027e-01, -1.2716e-01,  1.3728e-01,  2.5130e-01,\n           1.1522e-01,  1.6465e-03, -3.2652e-01, -2.4203e-01,  2.7271e-01,\n           3.4703e-01,  1.4537e-01, -5.4230e-02,  2.0772e-01,  1.2222e-01,\n           2.2739e-01,  1.3996e-01, -4.1904e-01,  5.8535e-02,  3.3988e-01,\n           9.7815e-02,  1.4387e-01, -1.0417e-01, -3.0340e-01, -2.6406e-01,\n          -9.2786e-02,  3.6124e-02, -3.4181e-01, -1.1696e-01,  3.8384e-01,\n           6.0129e-02, -4.3528e-02, -1.7150e-01, -2.2895e-01, -2.2508e-02,\n          -1.3951e-01,  5.0391e-03,  9.7692e-02, -8.2600e-02, -4.4786e-01,\n          -1.0911e-01, -5.4455e-01, -1.0192e-01,  1.9583e-01, -3.3229e-01,\n           2.7528e-01, -2.9993e-01,  9.0975e-02,  4.1470e-01,  4.1915e-02,\n          -1.8994e-02, -2.1249e-01, -2.0769e-02,  1.0013e-01,  3.2178e-01,\n           2.6382e-01, -4.1503e-01,  1.1366e-01,  1.3142e-01,  2.8490e-01,\n           1.2126e-01, -7.1120e-02, -1.3761e-01,  1.4225e-01, -2.2768e-01,\n           1.6595e-01, -2.1633e-01,  2.0445e-01, -2.5149e-01, -2.3462e-01,\n           2.9504e-01, -4.3752e-01, -4.9766e-02,  9.5619e-02,  2.7203e-01,\n           8.3333e-03, -4.9078e-02, -8.8749e-02,  1.5907e-01,  1.5653e-01,\n           1.1979e-01, -4.1002e-01,  3.0015e-01, -1.6790e-02, -4.4993e-02,\n          -5.5496e-02,  1.8606e-01,  2.4183e-01,  1.0303e-01, -3.9973e-01,\n          -1.6404e-01,  1.3631e-01,  3.0170e-01, -2.3914e-01,  1.7749e-01,\n          -3.1045e-01, -4.2076e-01, -1.0955e-01,  2.1587e-01,  2.3329e-01,\n           1.7321e-01, -2.9311e-01,  1.7927e-01, -1.3326e-01, -4.4922e-01,\n          -3.6926e-01, -1.3418e-01,  2.4563e-01,  1.6821e-01,  1.7763e-01,\n           2.4371e-01,  4.0108e-02,  1.3898e-01,  1.4709e-01,  1.5897e-01,\n          -1.3558e-01,  1.8048e-01, -3.6979e-01, -4.8194e-02, -3.0224e-01,\n          -2.1270e-01, -2.5609e-01,  4.2424e-01, -2.4527e-01,  2.4194e-01,\n           4.1605e-01, -3.1440e-01, -1.0625e-01,  1.4996e-01,  1.3238e-01,\n           7.1264e-02, -1.3708e-01,  2.1019e-01,  2.0039e-01, -9.6688e-02,\n           2.6119e-01, -1.3896e-02,  2.9086e-01,  1.6212e-01,  6.1304e-02,\n           1.4799e-01,  1.3982e-01, -1.5339e-01,  7.9259e-02, -1.4050e-02,\n          -4.8840e-02, -2.5733e-01, -1.5727e-01,  2.3229e-01, -7.0259e-02,\n           2.4217e-02, -1.7690e-01, -6.4507e-02,  5.1651e-03,  3.9415e-01,\n          -3.7883e-01,  2.6267e-01,  7.3589e-02,  1.6995e-01, -2.3413e-01,\n          -2.3543e-01,  9.4571e-02,  2.0851e-01, -4.4865e-01,  2.3682e-02,\n           1.4342e-01,  9.5272e-02,  2.1424e-01,  2.8170e-01,  8.4311e-03,\n          -1.3219e-01,  5.3841e-01, -1.4480e-01, -1.4085e-01,  2.6909e-01,\n          -2.8319e-01, -2.8045e-01,  2.4999e-01, -2.1317e-02,  3.2974e-01,\n           1.5573e-01,  6.6449e-02,  7.5446e-02, -6.1085e-01,  6.0450e-02,\n          -4.6207e-01, -8.1808e-03,  2.8095e-02, -9.6467e-02, -2.0137e-01,\n           1.6036e-01,  2.8666e-01, -2.4361e-01, -1.6346e-02,  2.1179e-01,\n           5.0813e-02, -9.2329e-02,  4.7980e-01, -7.1775e-03,  2.2582e-01,\n          -6.1834e-02,  2.6959e-01, -2.2385e-01,  2.6324e-01, -2.7526e-01,\n          -1.1521e-01,  4.2042e-02,  9.7212e-02,  6.3397e-02, -4.0348e-02,\n          -3.4408e-01,  2.4880e-01, -1.5275e-02, -6.0177e-02, -6.2704e-02,\n           1.0854e-01, -3.1073e-02,  7.6387e-02,  8.8965e-02,  3.1935e-01,\n           2.2015e-01, -3.7582e-02, -3.8733e-01, -5.9448e-02, -1.1720e-01,\n           6.3451e-02,  4.9610e-02,  1.8308e-02,  4.6296e-01, -9.0140e-02,\n           1.0484e-04, -1.5601e-01,  2.8291e-01,  2.3006e-01,  1.6743e-01,\n           1.2103e-01,  7.1099e-02,  1.6337e-01, -3.8276e-02, -1.7334e-02,\n          -1.5610e-01, -2.4764e-01, -2.8213e-01,  2.3848e-01, -2.6679e-01,\n          -1.5903e-01,  1.5730e-01,  2.1328e-01, -1.3124e-01,  1.5489e-01,\n           3.3090e-01,  1.2297e-01, -1.5964e-01,  3.0954e-01, -9.6767e-02,\n           9.7459e-02,  3.1856e-01, -4.3148e-02,  1.8268e-01,  5.3642e-01,\n           2.2859e-01, -3.9894e-01, -5.3708e-02, -2.3796e-01, -5.4005e-03,\n           2.5037e-01, -1.1444e-01,  1.9435e-01,  4.0045e-01,  3.1514e-01,\n           4.9319e-01, -1.8669e-02, -1.3310e-01,  1.0838e-01,  2.5667e-01,\n           3.5776e-02, -1.9927e-01, -1.7757e-01,  2.7399e-01,  8.5346e-02,\n          -1.5680e-01, -1.2514e-02, -1.4217e-01,  6.1741e-02, -1.6586e-01,\n          -4.2413e-01,  4.6739e-02,  1.6991e-01, -4.7828e-01,  1.2750e-01,\n          -3.1037e-01,  4.4076e-02, -2.4381e-01,  2.4202e-01, -2.5880e-01,\n          -1.2796e-01,  4.1581e-01, -5.3722e-02,  1.7539e-02, -1.8507e-01,\n          -1.5450e-01,  2.4454e-02,  1.2036e-02, -4.8371e-02, -1.9533e-02,\n           3.5984e-01, -1.6613e-01,  4.6619e-02,  1.8877e-02,  2.2054e-01,\n          -7.7161e-02,  1.8295e-01,  1.7197e-02, -1.2908e-01, -3.9890e-01,\n           1.3663e-01, -2.1127e-01, -4.4377e-01, -3.9106e-01,  3.9458e-01,\n          -1.4675e-01, -2.6565e-01, -2.2609e-01, -2.7383e-01,  9.2205e-02,\n           2.2372e-01,  4.5662e-01, -3.9735e-01, -6.9926e-02,  4.9995e-01,\n          -6.7641e-02, -1.9660e-01,  3.0055e-01,  2.2292e-01, -3.3214e-01,\n           3.6988e-01,  2.7960e-01, -7.0076e-02,  2.8102e-02,  5.3083e-01,\n           1.4739e-01,  2.0253e-01, -1.9778e-01,  4.7751e-01, -2.5482e-01,\n           3.2002e-01, -1.5566e-01, -1.9210e-01, -2.3896e-01, -2.0563e-03,\n           3.5590e-01,  2.0330e-01, -4.4359e-01, -1.3284e-01,  5.6973e-02,\n           3.3119e-01, -4.0474e-01, -7.0760e-02,  3.6668e-02, -3.6576e-01,\n           1.2317e-01,  1.0435e-01,  2.5552e-01, -4.1451e-01, -3.0012e-02,\n           4.1847e-01, -3.2331e-01,  1.1822e-01,  3.0342e-01,  8.6966e-02,\n           3.6233e-01, -4.4335e-02, -1.0096e-02,  4.6188e-02, -2.5951e-01,\n          -2.9547e-02,  1.3714e-01,  5.7834e-01,  1.4212e-01, -4.0849e-01,\n           1.1055e-01,  2.4542e-01, -1.7543e-01,  3.1505e-01, -9.0253e-02,\n          -3.9626e-02,  2.6807e-01, -4.8430e-02,  1.4228e-01, -8.6239e-02,\n          -2.6031e-01, -3.4352e-01,  4.1868e-01, -2.1520e-01, -1.0780e-01,\n          -1.7410e-01, -9.8137e-02, -1.8509e-01,  3.0571e-02, -3.8891e-01,\n           3.6955e-01,  1.6003e-01, -1.9684e-01, -1.0920e-01, -1.0565e-01,\n          -1.8501e-01, -2.1207e-01, -2.7411e-01,  4.1682e-01, -1.7961e-01,\n          -4.8093e-01,  2.7099e-01,  3.0270e-02,  3.5706e-01,  4.5577e-02,\n           1.1035e-01, -7.1230e-02,  1.4230e-01,  1.1042e-01, -1.1875e-01,\n           2.6342e-01,  8.0511e-02, -5.9012e-01, -1.3688e-01, -2.5692e-01,\n           9.2115e-02,  2.3678e-01, -3.7912e-01,  2.9019e-02,  3.7096e-02,\n           1.7515e-01,  1.9512e-02, -1.2934e-01, -9.8336e-02,  4.1498e-01,\n           2.4189e-01,  2.8652e-01,  1.2910e-01,  2.1781e-01, -2.4346e-02,\n          -3.6331e-01,  2.8518e-02,  9.4826e-02, -1.8736e-01,  4.5255e-01,\n          -1.1637e-01, -3.9879e-01, -6.3205e-02,  4.2367e-01,  9.5923e-02,\n          -1.4679e-02, -4.5640e-02,  2.1090e-01,  1.7609e-01, -1.3064e-01,\n           1.9494e-01, -4.5788e-02, -1.4252e-01, -1.1313e-01,  7.2816e-02,\n          -2.2074e-01,  5.0540e-02, -1.6169e-01, -1.2910e-02, -2.1704e-01,\n           1.9197e-02, -1.9891e-01,  2.7839e-01, -3.5211e-01,  1.0466e-01,\n           4.7807e-02,  3.0521e-01, -3.6600e-01, -1.8887e-01, -4.7351e-02,\n           1.7521e-01,  2.7785e-01,  3.7207e-01,  1.3246e-02,  4.2034e-02,\n          -1.8040e-01, -2.6263e-01,  5.8420e-02, -2.0585e-01,  1.2869e-01,\n           7.8583e-02,  2.4585e-01, -3.1965e-01, -1.9843e-01,  2.2358e-01,\n          -5.8403e-02, -1.5437e-01,  4.6256e-01,  2.3912e-01,  1.9697e-01,\n           3.0291e-02,  2.3159e-01,  4.0198e-02, -1.7525e-01, -1.1614e-01,\n          -2.8817e-01,  9.5208e-02, -1.1072e-01, -6.8288e-02, -7.3571e-02,\n          -1.3836e-01, -2.4735e-01, -1.6295e-01,  1.5430e-01,  1.2541e-01,\n           2.5656e-02, -7.6363e-02, -4.1694e-02, -3.0698e-01,  3.1316e-01,\n           3.7009e-03,  7.9322e-02, -5.4263e-02,  5.9545e-02, -1.6118e-01,\n           2.4499e-01,  2.1877e-01,  6.6010e-02, -1.7677e-01, -5.8230e-02,\n          -3.2647e-01, -3.5972e-01,  5.6589e-02,  1.3940e-01,  1.3641e-01,\n          -9.7825e-02, -2.7857e-01, -1.9047e-03, -1.3070e-01,  1.8966e-01,\n           3.3023e-02, -1.7722e-01, -1.0800e-01, -6.4136e-02, -2.3364e-02,\n           6.9859e-02, -2.1221e-01, -1.9302e-01, -1.1282e-01, -8.9990e-02,\n          -6.8713e-02,  3.5237e-01, -7.0613e-02,  3.1450e-01, -1.7329e-01,\n           2.1153e-02, -1.9160e-01,  1.4463e-01, -5.5973e-02,  6.6437e-02,\n           2.8610e-01, -4.6652e-01, -1.6867e-01, -2.5767e-02, -1.9701e-01,\n          -1.6272e-01, -1.0195e-01, -2.6313e-02,  2.3033e-01, -3.5897e-01,\n           2.1991e-01, -1.2825e-01,  2.0155e-01, -5.1254e-02, -2.5490e-01,\n          -1.6225e-01,  7.7723e-03,  2.7402e-01, -3.4974e-01, -2.4202e-01,\n          -2.9979e-01, -1.2266e-01, -9.7564e-02, -2.7101e-01,  4.4363e-01,\n          -1.2672e-01, -6.8712e-02,  3.6890e-02,  4.4539e-01,  2.1421e-01,\n           1.7221e-01,  2.2777e-01, -3.2545e-03,  3.2152e-02,  1.2249e-01,\n          -4.9318e-01,  2.6493e-01, -2.6054e-01, -1.4131e-01,  4.9641e-03,\n           1.4107e-01, -1.5786e-02,  2.8980e-02, -1.6210e-01, -9.3445e-02,\n           2.3429e-01, -3.8397e-01, -3.3143e-02,  2.8311e-01,  1.5047e-01,\n          -2.7344e-01,  3.1570e-02,  1.2186e-01,  3.9147e-01,  8.0947e-02,\n          -2.3919e-01,  1.2583e-01, -3.7150e-01, -2.5058e-02, -1.9792e-01,\n          -3.0291e-01,  1.9151e-01, -9.3119e-02,  7.6530e-02, -8.1040e-02,\n          -2.8561e-01,  2.1643e-01, -6.1416e-02, -6.3492e-02,  4.4589e-01,\n           5.3158e-02, -9.5958e-02,  1.3302e-01,  2.6892e-02,  3.8728e-02,\n          -1.0086e-01,  2.8471e-01,  2.1443e-01, -3.1842e-01,  1.1572e-01,\n          -1.9164e-01, -4.5938e-02, -1.1187e-01]], grad_fn=<TanhBackward>))"
     },
     "metadata": {},
     "execution_count": 8
    }
   ],
   "source": [
    "out = model(input_ids=token_ids_tensor, attention_mask=mask_tensor)\n",
    "out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# outputのサイズ\n",
    "out[0]:最後のBertLayerの出力 \n",
    "out[1]:最後のBertLayerの0要素目（CLS）を入力したBertPoolerの出力"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "torch.Size([1, 192, 768])\ntorch.Size([1, 768])\n"
    }
   ],
   "source": [
    "print(out[0].shape)\n",
    "print(out[1].shape)"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7-final"
  },
  "orig_nbformat": 2,
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}