Use_Pre_Trained_Tokenizer.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Use Pre-Trained Tokenizer.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMDogv+1b98mT9yBxqcTFB2",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/JINU6497/JINUrepo/blob/master/Use_Pre_Trained_Tokenizer.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TC5c9G5GbjhM",
        "outputId": "70b3f227-5afd-4cdc-ff20-69d57a61cfa9"
      },
      "source": [
        "!pip install transformers\n",
        "\n",
        "from transformers import BertTokenizer\n",
        "\n",
        "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n",
        "\n",
        "print(len(tokenizer.vocab))"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: transformers in /usr/local/lib/python3.7/dist-packages (4.9.2)\n",
            "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers) (4.6.3)\n",
            "Requirement already satisfied: huggingface-hub==0.0.12 in /usr/local/lib/python3.7/dist-packages (from transformers) (0.0.12)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers) (4.62.0)\n",
            "Requirement already satisfied: sacremoses in /usr/local/lib/python3.7/dist-packages (from transformers) (0.0.45)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.7/dist-packages (from transformers) (5.4.1)\n",
            "Requirement already satisfied: tokenizers<0.11,>=0.10.1 in /usr/local/lib/python3.7/dist-packages (from transformers) (0.10.3)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (2019.12.20)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers) (2.23.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers) (3.0.12)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (1.19.5)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers) (21.0)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from huggingface-hub==0.0.12->transformers) (3.7.4.3)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers) (2.4.7)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers) (3.5.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2021.5.30)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2.10)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (1.24.3)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.15.0)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (1.0.1)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers) (7.1.2)\n",
            "30522\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WSpavljHfFtU"
      },
      "source": [
        "'bert-base-uncased'라는 이름의 이미 학습된 모델을 사용할 예정인데, 이때 이 모델의 학습에 사용된 Tokenizer도 일치시켜야 함. 이때 tokenizer.vocap을 통하여 Tokenizer의 크기를 확인해볼 수 있다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4NDZKFMTfYN0",
        "outputId": "9ca48994-8f24-4770-fa51-7df368971738"
      },
      "source": [
        "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n",
        "\n",
        "sentence = \"My dog is cute. He likes playing\"\n",
        "print(tokenizer.tokenize(sentence))"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['my', 'dog', 'is', 'cute', '.', 'he', 'likes', 'playing']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_0IP4vkfkOTq"
      },
      "source": [
        "여기서는  split과 다르지 않은 결과를 보여준다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pvFQRhwkjqVO",
        "outputId": "2308472e-fff7-4e3d-fbd2-131f32973f87"
      },
      "source": [
        "tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-uncased')\n",
        "print(len(tokenizer.vocab))\n",
        "\n",
        "print(tokenizer.tokenize(sentence))"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "105879\n",
            "['my', 'dog', 'is', 'cut', '##e', '.', 'he', 'likes', 'playing']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sxRYlfRHkSrq"
      },
      "source": [
        "다양한 언어를 담고 있는 다른 데이터에서 학습한 모델인 bert-base-multilingual-uncased의 Tokenizer를 가져와 나누면 cute가 cut과 ##e로 나뉘는 것을 볼 수 있다. 즉, 학습한 데이터에 따라 Tokenizer가 달라지는 것 확인 가능. 이때 ##e는 바로 이어지는 Token을 의미"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_jgeNRnGj9Ec",
        "outputId": "59a03910-b7a8-4055-f156-b4747713b0b7"
      },
      "source": [
        "sentence = '나는 책상 위에 사과를 먹었다. 알고 보니 그 사과는 Jason 것이었다. 그래서 Jason에게 사과를 했다'\n",
        "\n",
        "print(tokenizer.tokenize(sentence))"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['나는', 'ᄎ', '##ᅢᆨ', '##상', '위에', 'ᄉ', '##ᅡ', '##과', '##를', 'ᄆ', '##ᅥ', '##ᆨ', '##었다', '.', '알', '##고', 'ᄇ', '##ᅩ', '##니', '그', 'ᄉ', '##ᅡ', '##과', '##는', 'jason', '것이', '##었다', '.', '그', '##래', '##서', 'jason', '##에게', 'ᄉ', '##ᅡ', '##과', '##를', '했다']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y7B5gS0IkqFy"
      },
      "source": [
        "한국어에 대한 Tokenization도 제한적이지만 가능."
      ]
    }
  ]
}