site/ja/hub/tutorials/senteval_for_universal_sentence_encoder_cmlm.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CGyzr0tfeUTQ"
      },
      "source": [
        "**Copyright 2021 The TensorFlow Hub Authors.**\n",
        "\n",
        "Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zV1OQAGReaGQ"
      },
      "outputs": [],
      "source": [
        "# Copyright 2021 The TensorFlow Hub Authors. All Rights Reserved.\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "# =============================================================================="
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L5bsDhkRfTpq"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>     <a target=\"_blank\" href=\"https://www.tensorflow.org/hub/tutorials/senteval_for_universal_sentence_encoder_cmlm\">     <img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\">     TensorFlow.org で表示</a>\n",
        "</td>\n",
        "  <td>     <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs-l10n/blob/master/site/ja/hub/tutorials/senteval_for_universal_sentence_encoder_cmlm.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\">Google Colabで実行</a>\n",
        "</td>\n",
        "  <td><a target=\"_blank\" href=\"https://github.com/tensorflow/docs-l10n/blob/master/site/ja/hub/tutorials/senteval_for_universal_sentence_encoder_cmlm.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\">GitHub でソースを表示</a></td>\n",
        "  <td><a href=\"https://storage.googleapis.com/tensorflow_docs/docs-l10n/site/ja/hub/tutorials/senteval_for_universal_sentence_encoder_cmlm.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\">ノートブックをダウンロード</a></td>\n",
        "  <td><a href=\"https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-base/1\"><img src=\"https://www.tensorflow.org/images/hub_logo_32px.png\"> TF Hub モデルを参照</a></td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "owWqOcw1e-RZ"
      },
      "source": [
        "# ユニバーサルセンテンスエンコーダ SentEval の実演\n",
        "\n",
        "この Colab は、文の埋め込みの品質を測定するためのライブラリである [SentEval](https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-base/1) ツールキットを使用した[ユニバーサルセンテンスエンコーダ CMLM モデル](https://github.com/facebookresearch/SentEval)を説明します。 SentEval ツールキットには、埋め込みモデルの一般化能力を評価し、エンコードされた言語プロパティを評価できる、さまざまな下流タスクのセットが含まれています。\n",
        "\n",
        "最初の 2 つのコードブロックを実行して環境をセットアップします。3 番目のコードブロックでは、SentEval タスクを選択してモデルを評価できます。この Colab を実行するには、GPU ランタイムをお勧めします。\n",
        "\n",
        "ユニバーサルセンテンスエンコーダ CMLM モデルの詳細については、https://openreview.net/forum?id=WDVD4lUCTzU を参照してください。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-CerULCLsjzV"
      },
      "outputs": [],
      "source": [
        "#@title Install dependencies\n",
        "!pip install --quiet \"tensorflow-text==2.11.*\"\n",
        "!pip install --quiet torch==1.8.1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LjqkqD6aiZGU"
      },
      "source": [
        "## SentEvalとタスクデータをダウンロードする\n",
        "\n",
        "このステップでは、github から SentEval をダウンロードし、データスクリプトを実行してタスクデータをダウンロードします。完了するまでに最大 5 分かかる場合があります。"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3UwhHQiKJmSc"
      },
      "outputs": [],
      "source": [
        "#@title Install SentEval and download task data\n",
        "!rm -rf ./SentEval\n",
        "!git clone https://github.com/facebookresearch/SentEval.git\n",
        "!cd $PWD/SentEval/data/downstream && bash get_transfer_data.bash > /dev/null 2>&1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7a2ohPn8vMe2"
      },
      "source": [
        "＃SentEval 評価タスクの実行。次のコードブロックは SentEval タスクを実行して結果を出力し、次のタスクのいずれかを選択して USE CMLM モデルを評価します。\n",
        "\n",
        "```\n",
        "MR\tCR\tSUBJ\tMPQA\tSST\tTREC\tMRPC\tSICK-E\n",
        "```\n",
        "\n",
        "実行するモデル、パラメーター、およびタスクを選択します。ラピッドプロトタイピングパラメータは、計算時間を短縮するために使用でき、より速く結果を得ることができます。\n",
        "\n",
        "通常、**'rapid prototyping'** パラメータを使用してタスクを完了するには 5〜15 分かかり、**'slower, best performance'** パラメータでは最大 1 時間かかります。\n",
        "\n",
        "```\n",
        "params = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 5}\n",
        "params['classifier'] = {'nhid': 0, 'optim': 'rmsprop', 'batch_size': 128,\n",
        "                                 'tenacity': 3, 'epoch_size': 2}\n",
        "```\n",
        "\n",
        "より良い結果を得るには、より遅い **'slower, best performance'** パラメータを使用してください。計算には最大 1 時間かかる場合があります。\n",
        "\n",
        "```\n",
        "params = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 10}\n",
        "params['classifier'] = {'nhid': 0, 'optim': 'adam', 'batch_size': 16,\n",
        "                                 'tenacity': 5, 'epoch_size': 6}\n",
        "```\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nenCcawjwowt"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\n",
        "\n",
        "import sys\n",
        "sys.path.append(f'{os.getcwd()}/SentEval')\n",
        "\n",
        "import tensorflow as tf\n",
        "\n",
        "# Prevent TF from claiming all GPU memory so there is some left for pytorch.\n",
        "gpus = tf.config.list_physical_devices('GPU')\n",
        "if gpus:\n",
        "  # Memory growth needs to be the same across GPUs.\n",
        "  for gpu in gpus:\n",
        "    tf.config.experimental.set_memory_growth(gpu, True)\n",
        "\n",
        "import tensorflow_hub as hub\n",
        "import tensorflow_text\n",
        "import senteval\n",
        "import time\n",
        "\n",
        "PATH_TO_DATA = f'{os.getcwd()}/SentEval/data'\n",
        "MODEL = 'https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-base/1' #@param ['https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-base/1', 'https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-large/1']\n",
        "PARAMS = 'rapid prototyping' #@param ['slower, best performance', 'rapid prototyping']\n",
        "TASK = 'CR' #@param ['CR','MR', 'MPQA', 'MRPC', 'SICKEntailment', 'SNLI', 'SST2', 'SUBJ', 'TREC']\n",
        "\n",
        "params_prototyping = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 5}\n",
        "params_prototyping['classifier'] = {'nhid': 0, 'optim': 'rmsprop', 'batch_size': 128,\n",
        "                                 'tenacity': 3, 'epoch_size': 2}\n",
        "\n",
        "params_best = {'task_path': PATH_TO_DATA, 'usepytorch': True, 'kfold': 10}\n",
        "params_best['classifier'] = {'nhid': 0, 'optim': 'adam', 'batch_size': 16,\n",
        "                                 'tenacity': 5, 'epoch_size': 6}\n",
        "\n",
        "params = params_best if PARAMS == 'slower, best performance' else params_prototyping\n",
        "\n",
        "preprocessor = hub.KerasLayer(\n",
        "    \"https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3\")\n",
        "encoder = hub.KerasLayer(\n",
        "    \"https://tfhub.dev/google/universal-sentence-encoder-cmlm/en-base/1\")\n",
        "\n",
        "inputs = tf.keras.Input(shape=tf.shape(''), dtype=tf.string)\n",
        "outputs = encoder(preprocessor(inputs))\n",
        "\n",
        "model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
        "\n",
        "def prepare(params, samples):\n",
        "    return\n",
        "\n",
        "def batcher(_, batch):\n",
        "    batch = [' '.join(sent) if sent else '.' for sent in batch]\n",
        "    return model.predict(tf.constant(batch))[\"default\"]\n",
        "\n",
        "\n",
        "se = senteval.engine.SE(params, batcher, prepare)\n",
        "print(\"Evaluating task %s with %s parameters\" % (TASK, PARAMS))\n",
        "start = time.time()\n",
        "results = se.eval(TASK)\n",
        "end = time.time()\n",
        "print('Time took on task %s : %.1f. seconds' % (TASK, end - start))\n",
        "print(results)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SNvsY6Hsvs0_"
      },
      "source": [
        "詳細情報\n",
        "\n",
        "- その他のテキスト埋め込みモデルは、[TensorFlow Hub](https://tfhub.dev) をご覧ください。\n",
        "- [多言語ユニバーサルセンテンスエンコーダ CMLM モデル](https://tfhub.dev/google/universal-sentence-encoder-cmlm/multilingual-base-br/1)もご覧ください。\n",
        "- 他の[ユニバーサルセンテンスエンコーダモデル](https://tfhub.dev/google/collections/universal-sentence-encoder/1)を確認してください\n",
        "\n",
        "## リファレンス\n",
        "\n",
        "- Ziyi Yang、Yinfei Yang、Daniel Cer、Jax Law、Eric Darve。[Universal Sentence Representations Learning with Conditional Masked Language Model. November 2020](https://openreview.net/forum?id=WDVD4lUCTzU)\n"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "senteval_for_universal_sentence_encoder_cmlm.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}