자연어처리_2_정제&정규화_어간추출&표제어추출_불용어.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyPuNBsNoR//Pp6hvJ3604rn",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/moonstar0301/NLP_studynote/blob/main/%EC%9E%90%EC%97%B0%EC%96%B4%EC%B2%98%EB%A6%AC_2_%EC%A0%95%EC%A0%9C%26%EC%A0%95%EA%B7%9C%ED%99%94_%EC%96%B4%EA%B0%84%EC%B6%94%EC%B6%9C%26%ED%91%9C%EC%A0%9C%EC%96%B4%EC%B6%94%EC%B6%9C_%EB%B6%88%EC%9A%A9%EC%96%B4.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "토큰화 작업 전, 후에는 텍스트 데이터를 용도에 맞게 정제(cleaning) 및 정규화(normalization)하는 일이 항상 함께합니다.\n",
        "\n",
        "정제(cleaning) : 갖고 있는 코퍼스로부터 노이즈 데이터를 제거한다.\n",
        "\n",
        "정규화(normalization) : 표현 방법이 다른 단어들을 통합시켜서 같은 단어로 만들어준다."
      ],
      "metadata": {
        "id": "KXHsM0fNkgSY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "1. 규칙에 기반한 표기가 다른 단어들의 통합\n",
        "usa와 us는 같은 의미를 가지므로 하나의 단어로 정규화해볼 수 있습니다.\n",
        "\n",
        "\n",
        "2.대, 소문자 통합\n",
        "소문자 변환을 적용하면 ferrari만 쳐도 원하는 결과를 얻을수있다.\n",
        "물론 대문자와 소문자를 무작정 통합해서는 안된다. 대문자와 소문자가 구분되어야하는 경우도 있기 때문이다.\n",
        "예를들어 단어US 와 us는 구분되어야할 필요가있다.\n",
        "\n",
        "\n",
        "3. 불필요한 단어의 제거\n",
        "\n",
        "(1) 등장 빈도가 적은 단어\n",
        "\n",
        "(2) 길이가 짧은 단어\n",
        "영어권 언어에서 길이가 짧은 단어들은 대부분 불용어에 해당됩니다.\n",
        "영어 단어의 길이가 한국어 단어의 길이보다는 평균적으로 길다는 점입니다\n",
        "\n",
        "\n",
        "4 정규 표현식(Regular Expression)\n",
        "뉴스 기사를 크롤링 했다면, 기사마다 게재 시간이 적혀져 있을 수 있습니다. 정규 표현식은 이러한 코퍼스 내에 계속해서 등장하는 글자들을 규칙에 기반하여 한 번에 제거하는 방식으로서 매우 유용합니다."
      ],
      "metadata": {
        "id": "XWyz9NXXk-om"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "MdVwWi4CkV4r"
      },
      "outputs": [],
      "source": [
        "import re"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "text = \"I was wondering if anyone out there could enlighten me on this car.\"\n",
        "\n",
        "# 길이가 1~2인 단어들을 정규 표현식을 이용하여 삭제\n",
        "shortword = re.compile(r'\\W*\\b\\w{1,2}\\b')\n",
        "print(shortword.sub('', text))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "utF7HEmiAAfz",
        "outputId": "0857b6cf-aad9-4d2b-9ea4-468a9c97c779"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " was wondering anyone out there could enlighten this car.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n"
      ],
      "metadata": {
        "id": "vutXqNSmAtBj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "어간 추출(Stemming) and 표제어 추출(Lemmatization)"
      ],
      "metadata": {
        "id": "N1_C9H9tAxL0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "1. 표제어 추출(Lemmatization)\n",
        "표제어 : 기본 사전형 단어\n",
        "표제어 추출은 단어들로부터 표제어를 찾아가는 과정입니다.\n",
        "\n",
        "Ex) am, are, is는 서로 다른 스펠링이지만 그 뿌리 단어는 be라고 볼 수 있습니다.\n",
        "\n",
        "형태소 : 어간(stem), 접사(affix)\n",
        "형태학적 파싱은 이 두 가지 구성요소를 분리하는 작업을 말합니다.\n",
        "예를들어 cats면 cat(어간) 과 s(접사)를 분리합니다.\n",
        "\n",
        "NLTK에서는 표제어 추출을 위한 도구인 WordNetLemmatizer를 지원합니다."
      ],
      "metadata": {
        "id": "wX4Tya0BBt9e"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "nltk.download('wordnet')\n",
        "from nltk.stem import WordNetLemmatizer\n",
        "\n",
        "lemmatizer = WordNetLemmatizer()\n",
        "\n",
        "words = ['policy', 'doing', 'organization', 'have', 'going', 'love', 'lives', 'fly', 'dies', 'watched', 'has', 'starting']\n",
        "\n",
        "print(\"표제어 추출 전 :\", words)\n",
        "\n",
        "print(\"표제어 추출 후 :\", [lemmatizer.lemmatize(word) for word in words])\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SvtZxYT5ANA7",
        "outputId": "e0453526-7932-4395-a079-1204daf3e1e9"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "표제어 추출 전 : ['policy', 'doing', 'organization', 'have', 'going', 'love', 'lives', 'fly', 'dies', 'watched', 'has', 'starting']\n",
            "표제어 추출 후 : ['policy', 'doing', 'organization', 'have', 'going', 'love', 'life', 'fly', 'dy', 'watched', 'ha', 'starting']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "위에보면 dy나 ha와 같이 적절하지 못한 단어를 출력하고 있습니다.\n",
        "\n",
        "WordNetLemmatizer는 입력으로 단어가 동사 품사라는 사실을 알려줄 수 있습니다. 즉, dies와 watched, has가 문장에서 동사로 쓰였다는 것을 알려준다면 표제어 추출기는 품사의 정보를 보존하면서 정확한 Lemma를 출력하게 됩니다."
      ],
      "metadata": {
        "id": "k_L5TWw3E69R"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "lemmatizer.lemmatize('dies', 'v')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "7a3vs_fDEeMd",
        "outputId": "26f56873-6c4d-40bd-ee5e-c779a4c64fad"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'die'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "표제어 추출은 문맥을 고려하며 수행했을 때의 결과는 해당 단어의 품사 정보를 보존합니다. 하지만 어간 추출을 수행한 결과는 품사 정보가 보존되지 않습니다. 더 정확히는 어간 추출을 한 결과는 사전에 존재하지 않는 단어일 경우가 많습니다."
      ],
      "metadata": {
        "id": "a2c5czYNFLPk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "어간추출(Stemming)\n",
        "\n"
      ],
      "metadata": {
        "id": "wJ8nkwlsFp-y"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "어간 추출은 형태학적 분석을 단순화한 버전이라고 볼 수도 있고, 정해진 규칙만 보고 단어의 어미를 자르는 어림짐작의 작업이라고 볼 수도 있습니다. "
      ],
      "metadata": {
        "id": "RcuHl4pMICXO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "nltk.download('punkt')\n",
        "from nltk.stem import PorterStemmer\n",
        "from nltk.tokenize import word_tokenize\n",
        "\n",
        "stemmer = PorterStemmer()\n",
        "\n",
        "sentence = \"This was not the map we found in Billy Bones's chest, but an accurate copy, complete in all things--names and heights and soundings--with the single exception of the red crosses and the written notes.\"\n",
        "tokenized_sentence = word_tokenize(sentence)\n",
        "\n",
        "print('어간 추출 천 :', tokenized_sentence)\n",
        "print('어간 추출 후 :', [stemmer.stem(word) for word in tokenized_sentence])\n",
        "\n",
        "#tokenizer - tokenize\n",
        "#stemmer - stem"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DWVhDLIsFDzK",
        "outputId": "93df4a48-5502-4a94-cba7-62ef139ace56"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "어간 추출 천 : ['This', 'was', 'not', 'the', 'map', 'we', 'found', 'in', 'Billy', 'Bones', \"'s\", 'chest', ',', 'but', 'an', 'accurate', 'copy', ',', 'complete', 'in', 'all', 'things', '--', 'names', 'and', 'heights', 'and', 'soundings', '--', 'with', 'the', 'single', 'exception', 'of', 'the', 'red', 'crosses', 'and', 'the', 'written', 'notes', '.']\n",
            "어간 추출 후 : ['thi', 'wa', 'not', 'the', 'map', 'we', 'found', 'in', 'billi', 'bone', \"'s\", 'chest', ',', 'but', 'an', 'accur', 'copi', ',', 'complet', 'in', 'all', 'thing', '--', 'name', 'and', 'height', 'and', 'sound', '--', 'with', 'the', 'singl', 'except', 'of', 'the', 'red', 'cross', 'and', 'the', 'written', 'note', '.']\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "어간 추출 속도는 표제어 추출보다 일반적으로 빠른데, 포터 어간 추출기는 정밀하게 설계되어 정확도가 높으므로 영어 자연어 처리에서 어간 추출을 하고자 한다면 가장 준수한 선택입니다."
      ],
      "metadata": {
        "id": "hvd9K0nRJ2lI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#이번에는 포터 알고리즘과 랭커스터 스태머 알고리즘으로 각각 어간 추출을 진행했을 때, 이 둘의 결과를 비교해보겠습니다.\n",
        "\n",
        "from nltk.stem import PorterStemmer\n",
        "from nltk.stem import LancasterStemmer\n",
        "from nltk.tokenize import word_tokenize\n",
        "\n",
        "porter_stemmer = PorterStemmer()\n",
        "lancaster_stemmer = LancasterStemmer()\n",
        "\n",
        "sentence = \"This was not the map we found in Billy Bones's chest, but an accurate copy, complete in all things--names and heights and soundings--with the single exception of the red crosses and the written notes.\"\n",
        "words = word_tokenize(sentence)\n",
        "\n",
        "print(\"어간 추출 전 :\", words)\n",
        "print(\"포터 스태머의 어간 추출 후:\", [porter_stemmer.stem(word) for word in words])\n",
        "print(\"랭커스터 스태머의 어간 추출 후:\", [lancaster_stemmer.stem(word) for word in words])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9aw5J4WKJPVB",
        "outputId": "eb04c9f2-ef29-4280-d21b-7a117fc352ca"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "어간 추출 전 : ['This', 'was', 'not', 'the', 'map', 'we', 'found', 'in', 'Billy', 'Bones', \"'s\", 'chest', ',', 'but', 'an', 'accurate', 'copy', ',', 'complete', 'in', 'all', 'things', '--', 'names', 'and', 'heights', 'and', 'soundings', '--', 'with', 'the', 'single', 'exception', 'of', 'the', 'red', 'crosses', 'and', 'the', 'written', 'notes', '.']\n",
            "포터 스태머의 어간 추출 후: ['thi', 'wa', 'not', 'the', 'map', 'we', 'found', 'in', 'billi', 'bone', \"'s\", 'chest', ',', 'but', 'an', 'accur', 'copi', ',', 'complet', 'in', 'all', 'thing', '--', 'name', 'and', 'height', 'and', 'sound', '--', 'with', 'the', 'singl', 'except', 'of', 'the', 'red', 'cross', 'and', 'the', 'written', 'note', '.']\n",
            "랭커스터 스태머의 어간 추출 후: ['thi', 'was', 'not', 'the', 'map', 'we', 'found', 'in', 'bil', 'bon', \"'s\", 'chest', ',', 'but', 'an', 'acc', 'cop', ',', 'complet', 'in', 'al', 'thing', '--', 'nam', 'and', 'height', 'and', 'sound', '--', 'with', 'the', 'singl', 'exceiv', 'of', 'the', 'red', 'cross', 'and', 'the', 'writ', 'not', '.']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "한국어에서의 어간 추출\n",
        "한국어는 아래와 같이 5언 9품사의 구조를 가지고 있습니다.\n",
        "\n",
        "체언 : 명사, 대명사, 수사\n",
        "\n",
        "수식언 : 관형사, 부사\n",
        "\n",
        "관계언 : 조사\n",
        "\n",
        "독립언 : 감탄사\n",
        "\n",
        "용언 : 동사, 형용사\n",
        "\n",
        "활용(conjugation)\n",
        "활용이란 용언의 어간이 어미를 가지는 일을 말합니다\n",
        "활용은 어간이 어미를 취할 때, 어간의 모습이 일정하다면 규칙 활용, 어간이나 어미의 모습이 변하는 불규칙 활용으로 나뉩니다."
      ],
      "metadata": {
        "id": "sGI-VmtZNeOl"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n"
      ],
      "metadata": {
        "id": "aLfWPb9gOOWE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "불용어(Stopword)\n",
        "\n",
        "큰 의미가 없는 단어 토큰을 제거하는 작업이 필요합니다.\n",
        "\n",
        "예를 들면 I, my, me, over, 조사, 접미사 같은 단어들은 문장에서는 자주 등장하지만 실제 의미 분석을 하는데는 기의 기여하는 바가 없습니다. 이러한 단어들을 불용어라고합니다.\n",
        "\n",
        "NLTK에서는 100개 이상의 영어 단어들을 불용어 패키지 내에서 미리 정의하고 있습니다."
      ],
      "metadata": {
        "id": "HsWEzLP-OQhP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!curl -s https://raw.githubusercontent.com/teddylee777/machine-learning/master/99-Misc/01-Colab/mecab-colab.sh | bash\n",
        "from nltk.corpus import stopwords\n",
        "from nltk.tokenize import word_tokenize\n",
        "from konlpy.tag import Okt"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O9W5R6haLYUt",
        "outputId": "77d22814-45d2-4081-fff9-1546fb4fd4f4"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-04-29 11:00:00--  https://www.dropbox.com/s/9xls0tgtf3edgns/mecab-0.996-ko-0.9.2.tar.gz?dl=1\n",
            "Resolving www.dropbox.com (www.dropbox.com)... 162.125.3.18, 2620:100:6030:18::a27d:5012\n",
            "Connecting to www.dropbox.com (www.dropbox.com)|162.125.3.18|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: /s/dl/9xls0tgtf3edgns/mecab-0.996-ko-0.9.2.tar.gz [following]\n",
            "--2023-04-29 11:00:00--  https://www.dropbox.com/s/dl/9xls0tgtf3edgns/mecab-0.996-ko-0.9.2.tar.gz\n",
            "Reusing existing connection to www.dropbox.com:443.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://ucdf952888a086b33b0ead98a074.dl.dropboxusercontent.com/cd/0/get/B7E53PSZ8trUVc5ALJFKkSnDSUy9776IBHsly-SIOGu7rgD62snmsRmZjBRCOFuWyojOBrYloPsl07aOj3m-rYFysb1ccZfL7yrugpRGFa0G3Udf882jorJrBTNTeS6P99JIxqSywH3ABjGAmyVYYees-aEO9bT6IBUK3_2CFkXxjGnL9QrXdLey21jNVcYTcDg/file?dl=1# [following]\n",
            "--2023-04-29 11:00:01--  https://ucdf952888a086b33b0ead98a074.dl.dropboxusercontent.com/cd/0/get/B7E53PSZ8trUVc5ALJFKkSnDSUy9776IBHsly-SIOGu7rgD62snmsRmZjBRCOFuWyojOBrYloPsl07aOj3m-rYFysb1ccZfL7yrugpRGFa0G3Udf882jorJrBTNTeS6P99JIxqSywH3ABjGAmyVYYees-aEO9bT6IBUK3_2CFkXxjGnL9QrXdLey21jNVcYTcDg/file?dl=1\n",
            "Resolving ucdf952888a086b33b0ead98a074.dl.dropboxusercontent.com (ucdf952888a086b33b0ead98a074.dl.dropboxusercontent.com)... 162.125.3.15, 2620:100:6018:15::a27d:30f\n",
            "Connecting to ucdf952888a086b33b0ead98a074.dl.dropboxusercontent.com (ucdf952888a086b33b0ead98a074.dl.dropboxusercontent.com)|162.125.3.15|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 1414979 (1.3M) [application/binary]\n",
            "Saving to: ‘mecab-0.996-ko-0.9.2.tar.gz?dl=1.1’\n",
            "\n",
            "mecab-0.996-ko-0.9. 100%[===================>]   1.35M  --.-KB/s    in 0.07s   \n",
            "\n",
            "2023-04-29 11:00:01 (19.4 MB/s) - ‘mecab-0.996-ko-0.9.2.tar.gz?dl=1.1’ saved [1414979/1414979]\n",
            "\n",
            "mecab-0.996-ko-0.9.2/\n",
            "mecab-0.996-ko-0.9.2/example/\n",
            "mecab-0.996-ko-0.9.2/example/example.cpp\n",
            "mecab-0.996-ko-0.9.2/example/example_lattice.cpp\n",
            "mecab-0.996-ko-0.9.2/example/example_lattice.c\n",
            "mecab-0.996-ko-0.9.2/example/example.c\n",
            "mecab-0.996-ko-0.9.2/example/thread_test.cpp\n",
            "mecab-0.996-ko-0.9.2/mecab-config.in\n",
            "mecab-0.996-ko-0.9.2/man/\n",
            "mecab-0.996-ko-0.9.2/man/Makefile.am\n",
            "mecab-0.996-ko-0.9.2/man/mecab.1\n",
            "mecab-0.996-ko-0.9.2/man/Makefile.in\n",
            "mecab-0.996-ko-0.9.2/mecab.iss.in\n",
            "mecab-0.996-ko-0.9.2/config.guess\n",
            "mecab-0.996-ko-0.9.2/README\n",
            "mecab-0.996-ko-0.9.2/COPYING\n",
            "mecab-0.996-ko-0.9.2/CHANGES.md\n",
            "mecab-0.996-ko-0.9.2/README.md\n",
            "mecab-0.996-ko-0.9.2/INSTALL\n",
            "mecab-0.996-ko-0.9.2/config.sub\n",
            "mecab-0.996-ko-0.9.2/configure.in\n",
            "mecab-0.996-ko-0.9.2/swig/\n",
            "mecab-0.996-ko-0.9.2/swig/Makefile\n",
            "mecab-0.996-ko-0.9.2/swig/version.h.in\n",
            "mecab-0.996-ko-0.9.2/swig/version.h\n",
            "mecab-0.996-ko-0.9.2/swig/MeCab.i\n",
            "mecab-0.996-ko-0.9.2/aclocal.m4\n",
            "mecab-0.996-ko-0.9.2/LGPL\n",
            "mecab-0.996-ko-0.9.2/Makefile.am\n",
            "mecab-0.996-ko-0.9.2/configure\n",
            "mecab-0.996-ko-0.9.2/tests/\n",
            "mecab-0.996-ko-0.9.2/tests/autolink/\n",
            "mecab-0.996-ko-0.9.2/tests/autolink/unk.def\n",
            "mecab-0.996-ko-0.9.2/tests/autolink/dicrc\n",
            "mecab-0.996-ko-0.9.2/tests/autolink/dic.csv\n",
            "mecab-0.996-ko-0.9.2/tests/autolink/test\n",
            "mecab-0.996-ko-0.9.2/tests/autolink/char.def\n",
            "mecab-0.996-ko-0.9.2/tests/autolink/matrix.def\n",
            "mecab-0.996-ko-0.9.2/tests/autolink/test.gld\n",
            "mecab-0.996-ko-0.9.2/tests/t9/\n",
            "mecab-0.996-ko-0.9.2/tests/t9/unk.def\n",
            "mecab-0.996-ko-0.9.2/tests/t9/ipadic.pl\n",
            "mecab-0.996-ko-0.9.2/tests/t9/dicrc\n",
            "mecab-0.996-ko-0.9.2/tests/t9/dic.csv\n",
            "mecab-0.996-ko-0.9.2/tests/t9/test\n",
            "mecab-0.996-ko-0.9.2/tests/t9/char.def\n",
            "mecab-0.996-ko-0.9.2/tests/t9/matrix.def\n",
            "mecab-0.996-ko-0.9.2/tests/t9/mkdic.pl\n",
            "mecab-0.996-ko-0.9.2/tests/t9/test.gld\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/ipa.train\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/ipa.test\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/seed/\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/seed/rewrite.def\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/seed/feature.def\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/seed/unk.def\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/seed/dicrc\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/seed/dic.csv\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/seed/char.def\n",
            "mecab-0.996-ko-0.9.2/tests/cost-train/seed/matrix.def\n",
            "mecab-0.996-ko-0.9.2/tests/run-eval.sh\n",
            "mecab-0.996-ko-0.9.2/tests/run-cost-train.sh\n",
            "mecab-0.996-ko-0.9.2/tests/Makefile.am\n",
            "mecab-0.996-ko-0.9.2/tests/katakana/\n",
            "mecab-0.996-ko-0.9.2/tests/katakana/unk.def\n",
            "mecab-0.996-ko-0.9.2/tests/katakana/dicrc\n",
            "mecab-0.996-ko-0.9.2/tests/katakana/dic.csv\n",
            "mecab-0.996-ko-0.9.2/tests/katakana/test\n",
            "mecab-0.996-ko-0.9.2/tests/katakana/char.def\n",
            "mecab-0.996-ko-0.9.2/tests/katakana/matrix.def\n",
            "mecab-0.996-ko-0.9.2/tests/katakana/test.gld\n",
            "mecab-0.996-ko-0.9.2/tests/eval/\n",
            "mecab-0.996-ko-0.9.2/tests/eval/answer\n",
            "mecab-0.996-ko-0.9.2/tests/eval/system\n",
            "mecab-0.996-ko-0.9.2/tests/eval/test.gld\n",
            "mecab-0.996-ko-0.9.2/tests/shiin/\n",
            "mecab-0.996-ko-0.9.2/tests/shiin/unk.def\n",
            "mecab-0.996-ko-0.9.2/tests/shiin/dicrc\n",
            "mecab-0.996-ko-0.9.2/tests/shiin/dic.csv\n",
            "mecab-0.996-ko-0.9.2/tests/shiin/test\n",
            "mecab-0.996-ko-0.9.2/tests/shiin/char.def\n",
            "mecab-0.996-ko-0.9.2/tests/shiin/matrix.def\n",
            "mecab-0.996-ko-0.9.2/tests/shiin/mkdic.pl\n",
            "mecab-0.996-ko-0.9.2/tests/shiin/test.gld\n",
            "mecab-0.996-ko-0.9.2/tests/latin/\n",
            "mecab-0.996-ko-0.9.2/tests/latin/unk.def\n",
            "mecab-0.996-ko-0.9.2/tests/latin/dicrc\n",
            "mecab-0.996-ko-0.9.2/tests/latin/dic.csv\n",
            "mecab-0.996-ko-0.9.2/tests/latin/test\n",
            "mecab-0.996-ko-0.9.2/tests/latin/char.def\n",
            "mecab-0.996-ko-0.9.2/tests/latin/matrix.def\n",
            "mecab-0.996-ko-0.9.2/tests/latin/test.gld\n",
            "mecab-0.996-ko-0.9.2/tests/chartype/\n",
            "mecab-0.996-ko-0.9.2/tests/chartype/unk.def\n",
            "mecab-0.996-ko-0.9.2/tests/chartype/dicrc\n",
            "mecab-0.996-ko-0.9.2/tests/chartype/dic.csv\n",
            "mecab-0.996-ko-0.9.2/tests/chartype/test\n",
            "mecab-0.996-ko-0.9.2/tests/chartype/char.def\n",
            "mecab-0.996-ko-0.9.2/tests/chartype/matrix.def\n",
            "mecab-0.996-ko-0.9.2/tests/chartype/test.gld\n",
            "mecab-0.996-ko-0.9.2/tests/run-dics.sh\n",
            "mecab-0.996-ko-0.9.2/tests/ngram/\n",
            "mecab-0.996-ko-0.9.2/tests/ngram/unk.def\n",
            "mecab-0.996-ko-0.9.2/tests/ngram/dicrc\n",
            "mecab-0.996-ko-0.9.2/tests/ngram/dic.csv\n",
            "mecab-0.996-ko-0.9.2/tests/ngram/test\n",
            "mecab-0.996-ko-0.9.2/tests/ngram/char.def\n",
            "mecab-0.996-ko-0.9.2/tests/ngram/matrix.def\n",
            "mecab-0.996-ko-0.9.2/tests/ngram/test.gld\n",
            "mecab-0.996-ko-0.9.2/tests/Makefile.in\n",
            "mecab-0.996-ko-0.9.2/ltmain.sh\n",
            "mecab-0.996-ko-0.9.2/config.rpath\n",
            "mecab-0.996-ko-0.9.2/config.h.in\n",
            "mecab-0.996-ko-0.9.2/mecabrc.in\n",
            "mecab-0.996-ko-0.9.2/GPL\n",
            "mecab-0.996-ko-0.9.2/Makefile.train\n",
            "mecab-0.996-ko-0.9.2/ChangeLog\n",
            "mecab-0.996-ko-0.9.2/install-sh\n",
            "mecab-0.996-ko-0.9.2/AUTHORS\n",
            "mecab-0.996-ko-0.9.2/doc/\n",
            "mecab-0.996-ko-0.9.2/doc/bindings.html\n",
            "mecab-0.996-ko-0.9.2/doc/posid.html\n",
            "mecab-0.996-ko-0.9.2/doc/unk.html\n",
            "mecab-0.996-ko-0.9.2/doc/learn.html\n",
            "mecab-0.996-ko-0.9.2/doc/format.html\n",
            "mecab-0.996-ko-0.9.2/doc/libmecab.html\n",
            "mecab-0.996-ko-0.9.2/doc/mecab.css\n",
            "mecab-0.996-ko-0.9.2/doc/feature.html\n",
            "mecab-0.996-ko-0.9.2/doc/Makefile.am\n",
            "mecab-0.996-ko-0.9.2/doc/soft.html\n",
            "mecab-0.996-ko-0.9.2/doc/en/\n",
            "mecab-0.996-ko-0.9.2/doc/en/bindings.html\n",
            "mecab-0.996-ko-0.9.2/doc/dic-detail.html\n",
            "mecab-0.996-ko-0.9.2/doc/flow.png\n",
            "mecab-0.996-ko-0.9.2/doc/mecab.html\n",
            "mecab-0.996-ko-0.9.2/doc/index.html\n",
            "mecab-0.996-ko-0.9.2/doc/result.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/tab_a.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/globals_eval.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/classMeCab_1_1Tagger-members.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/functions_vars.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/doxygen.css\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/tab_r.gif\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/classMeCab_1_1Lattice.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/functions.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/classMeCab_1_1Tagger.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/mecab_8h_source.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/tabs.css\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/nav_f.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/tab_b.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/globals.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/nav_h.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/tab_h.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/classMeCab_1_1Model.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/globals_func.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/closed.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/tab_l.gif\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/structmecab__path__t-members.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/functions_func.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/globals_type.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/classMeCab_1_1Lattice-members.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/structmecab__node__t.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/namespacemembers_func.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/tab_s.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/structmecab__dictionary__info__t-members.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/namespacemembers_type.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/classMeCab_1_1Model-members.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/structmecab__dictionary__info__t.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/namespaces.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/namespacemembers.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/namespaceMeCab.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/structmecab__path__t.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/files.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/structmecab__node__t-members.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/index.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/annotated.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/globals_defs.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/classes.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/mecab_8h-source.html\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/doxygen.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/tab_b.gif\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/bc_s.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/open.png\n",
            "mecab-0.996-ko-0.9.2/doc/doxygen/mecab_8h.html\n",
            "mecab-0.996-ko-0.9.2/doc/dic.html\n",
            "mecab-0.996-ko-0.9.2/doc/partial.html\n",
            "mecab-0.996-ko-0.9.2/doc/feature.png\n",
            "mecab-0.996-ko-0.9.2/doc/Makefile.in\n",
            "mecab-0.996-ko-0.9.2/missing\n",
            "mecab-0.996-ko-0.9.2/BSD\n",
            "mecab-0.996-ko-0.9.2/NEWS\n",
            "mecab-0.996-ko-0.9.2/mkinstalldirs\n",
            "mecab-0.996-ko-0.9.2/src/\n",
            "mecab-0.996-ko-0.9.2/src/dictionary.h\n",
            "mecab-0.996-ko-0.9.2/src/writer.h\n",
            "mecab-0.996-ko-0.9.2/src/utils.h\n",
            "mecab-0.996-ko-0.9.2/src/string_buffer.cpp\n",
            "mecab-0.996-ko-0.9.2/src/tokenizer.cpp\n",
            "mecab-0.996-ko-0.9.2/src/make.bat\n",
            "mecab-0.996-ko-0.9.2/src/mecab.h\n",
            "mecab-0.996-ko-0.9.2/src/freelist.h\n",
            "mecab-0.996-ko-0.9.2/src/string_buffer.h\n",
            "mecab-0.996-ko-0.9.2/src/learner_tagger.h\n",
            "mecab-0.996-ko-0.9.2/src/dictionary_compiler.cpp\n",
            "mecab-0.996-ko-0.9.2/src/eval.cpp\n",
            "mecab-0.996-ko-0.9.2/src/mecab-system-eval.cpp\n",
            "mecab-0.996-ko-0.9.2/src/darts.h\n",
            "mecab-0.996-ko-0.9.2/src/param.h\n",
            "mecab-0.996-ko-0.9.2/src/char_property.h\n",
            "mecab-0.996-ko-0.9.2/src/learner_node.h\n",
            "mecab-0.996-ko-0.9.2/src/mecab-dict-gen.cpp\n",
            "mecab-0.996-ko-0.9.2/src/mecab-dict-index.cpp\n",
            "mecab-0.996-ko-0.9.2/src/winmain.h\n",
            "mecab-0.996-ko-0.9.2/src/thread.h\n",
            "mecab-0.996-ko-0.9.2/src/context_id.cpp\n",
            "mecab-0.996-ko-0.9.2/src/Makefile.am\n",
            "mecab-0.996-ko-0.9.2/src/connector.h\n",
            "mecab-0.996-ko-0.9.2/src/common.h\n",
            "mecab-0.996-ko-0.9.2/src/dictionary_rewriter.cpp\n",
            "mecab-0.996-ko-0.9.2/src/Makefile.msvc.in\n",
            "mecab-0.996-ko-0.9.2/src/dictionary_rewriter.h\n",
            "mecab-0.996-ko-0.9.2/src/feature_index.h\n",
            "mecab-0.996-ko-0.9.2/src/iconv_utils.cpp\n",
            "mecab-0.996-ko-0.9.2/src/char_property.cpp\n",
            "mecab-0.996-ko-0.9.2/src/mecab-test-gen.cpp\n",
            "mecab-0.996-ko-0.9.2/src/tagger.cpp\n",
            "mecab-0.996-ko-0.9.2/src/mecab-cost-train.cpp\n",
            "mecab-0.996-ko-0.9.2/src/learner.cpp\n",
            "mecab-0.996-ko-0.9.2/src/dictionary.cpp\n",
            "mecab-0.996-ko-0.9.2/src/lbfgs.cpp\n",
            "mecab-0.996-ko-0.9.2/src/ucs.h\n",
            "mecab-0.996-ko-0.9.2/src/writer.cpp\n",
            "mecab-0.996-ko-0.9.2/src/learner_tagger.cpp\n",
            "mecab-0.996-ko-0.9.2/src/lbfgs.h\n",
            "mecab-0.996-ko-0.9.2/src/libmecab.cpp\n",
            "mecab-0.996-ko-0.9.2/src/tokenizer.h\n",
            "mecab-0.996-ko-0.9.2/src/mecab.cpp\n",
            "mecab-0.996-ko-0.9.2/src/utils.cpp\n",
            "mecab-0.996-ko-0.9.2/src/dictionary_generator.cpp\n",
            "mecab-0.996-ko-0.9.2/src/param.cpp\n",
            "mecab-0.996-ko-0.9.2/src/context_id.h\n",
            "mecab-0.996-ko-0.9.2/src/mmap.h\n",
            "mecab-0.996-ko-0.9.2/src/viterbi.h\n",
            "mecab-0.996-ko-0.9.2/src/viterbi.cpp\n",
            "mecab-0.996-ko-0.9.2/src/stream_wrapper.h\n",
            "mecab-0.996-ko-0.9.2/src/feature_index.cpp\n",
            "mecab-0.996-ko-0.9.2/src/nbest_generator.h\n",
            "mecab-0.996-ko-0.9.2/src/ucstable.h\n",
            "mecab-0.996-ko-0.9.2/src/nbest_generator.cpp\n",
            "mecab-0.996-ko-0.9.2/src/iconv_utils.h\n",
            "mecab-0.996-ko-0.9.2/src/connector.cpp\n",
            "mecab-0.996-ko-0.9.2/src/Makefile.in\n",
            "mecab-0.996-ko-0.9.2/src/scoped_ptr.h\n",
            "mecab-0.996-ko-0.9.2/Makefile.in\n",
            "checking for a BSD-compatible install... /usr/bin/install -c\n",
            "checking whether build environment is sane... yes\n",
            "checking for a thread-safe mkdir -p... /usr/bin/mkdir -p\n",
            "checking for gawk... no\n",
            "checking for mawk... mawk\n",
            "checking whether make sets $(MAKE)... yes\n",
            "checking for gcc... gcc\n",
            "checking whether the C compiler works... yes\n",
            "checking for C compiler default output file name... a.out\n",
            "checking for suffix of executables... \n",
            "checking whether we are cross compiling... no\n",
            "checking for suffix of object files... o\n",
            "checking whether we are using the GNU C compiler... yes\n",
            "checking whether gcc accepts -g... yes\n",
            "checking for gcc option to accept ISO C89... none needed\n",
            "checking for style of include used by make... GNU\n",
            "checking dependency style of gcc... none\n",
            "checking for g++... g++\n",
            "checking whether we are using the GNU C++ compiler... yes\n",
            "checking whether g++ accepts -g... yes\n",
            "checking dependency style of g++... none\n",
            "checking how to run the C preprocessor... gcc -E\n",
            "checking for grep that handles long lines and -e... /usr/bin/grep\n",
            "checking for egrep... /usr/bin/grep -E\n",
            "checking whether gcc needs -traditional... no\n",
            "checking whether make sets $(MAKE)... (cached) yes\n",
            "checking build system type... x86_64-unknown-linux-gnu\n",
            "checking host system type... x86_64-unknown-linux-gnu\n",
            "checking how to print strings... printf\n",
            "checking for a sed that does not truncate output... /usr/bin/sed\n",
            "checking for fgrep... /usr/bin/grep -F\n",
            "checking for ld used by gcc... /usr/bin/ld\n",
            "checking if the linker (/usr/bin/ld) is GNU ld... yes\n",
            "checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B\n",
            "checking the name lister (/usr/bin/nm -B) interface... BSD nm\n",
            "checking whether ln -s works... yes\n",
            "checking the maximum length of command line arguments... 1572864\n",
            "checking whether the shell understands some XSI constructs... yes\n",
            "checking whether the shell understands \"+=\"... yes\n",
            "checking how to convert x86_64-unknown-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop\n",
            "checking how to convert x86_64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop\n",
            "checking for /usr/bin/ld option to reload object files... -r\n",
            "checking for objdump... objdump\n",
            "checking how to recognize dependent libraries... pass_all\n",
            "checking for dlltool... dlltool\n",
            "checking how to associate runtime and link libraries... printf %s\\n\n",
            "checking for ar... ar\n",
            "checking for archiver @FILE support... @\n",
            "checking for strip... strip\n",
            "checking for ranlib... ranlib\n",
            "checking command to parse /usr/bin/nm -B output from gcc object... ok\n",
            "checking for sysroot... no\n",
            "checking for mt... no\n",
            "checking if : is a manifest tool... no\n",
            "checking for ANSI C header files... yes\n",
            "checking for sys/types.h... yes\n",
            "checking for sys/stat.h... yes\n",
            "checking for stdlib.h... yes\n",
            "checking for string.h... yes\n",
            "checking for memory.h... yes\n",
            "checking for strings.h... yes\n",
            "checking for inttypes.h... yes\n",
            "checking for stdint.h... yes\n",
            "checking for unistd.h... yes\n",
            "checking for dlfcn.h... yes\n",
            "checking for objdir... .libs\n",
            "checking if gcc supports -fno-rtti -fno-exceptions... no\n",
            "checking for gcc option to produce PIC... -fPIC -DPIC\n",
            "checking if gcc PIC flag -fPIC -DPIC works... yes\n",
            "checking if gcc static flag -static works... yes\n",
            "checking if gcc supports -c -o file.o... yes\n",
            "checking if gcc supports -c -o file.o... (cached) yes\n",
            "checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes\n",
            "checking whether -lc should be explicitly linked in... no\n",
            "checking dynamic linker characteristics... GNU/Linux ld.so\n",
            "checking how to hardcode library paths into programs... immediate\n",
            "checking whether stripping libraries is possible... yes\n",
            "checking if libtool supports shared libraries... yes\n",
            "checking whether to build shared libraries... yes\n",
            "checking whether to build static libraries... yes\n",
            "checking how to run the C++ preprocessor... g++ -E\n",
            "checking for ld used by g++... /usr/bin/ld -m elf_x86_64\n",
            "checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes\n",
            "checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes\n",
            "checking for g++ option to produce PIC... -fPIC -DPIC\n",
            "checking if g++ PIC flag -fPIC -DPIC works... yes\n",
            "checking if g++ static flag -static works... yes\n",
            "checking if g++ supports -c -o file.o... yes\n",
            "checking if g++ supports -c -o file.o... (cached) yes\n",
            "checking whether the g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes\n",
            "checking dynamic linker characteristics... (cached) GNU/Linux ld.so\n",
            "checking how to hardcode library paths into programs... immediate\n",
            "checking for library containing strerror... none required\n",
            "checking whether byte ordering is bigendian... no\n",
            "checking for ld used by GCC... /usr/bin/ld -m elf_x86_64\n",
            "checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes\n",
            "checking for shared library run path origin... done\n",
            "checking for iconv... yes\n",
            "checking for working iconv... yes\n",
            "checking for iconv declaration... \n",
            "         extern size_t iconv (iconv_t cd, char * *inbuf, size_t *inbytesleft, char * *outbuf, size_t *outbytesleft);\n",
            "checking for ANSI C header files... (cached) yes\n",
            "checking for an ANSI C-conforming const... yes\n",
            "checking whether byte ordering is bigendian... (cached) no\n",
            "checking for string.h... (cached) yes\n",
            "checking for stdlib.h... (cached) yes\n",
            "checking for unistd.h... (cached) yes\n",
            "checking fcntl.h usability... yes\n",
            "checking fcntl.h presence... yes\n",
            "checking for fcntl.h... yes\n",
            "checking for stdint.h... (cached) yes\n",
            "checking for sys/stat.h... (cached) yes\n",
            "checking sys/mman.h usability... yes\n",
            "checking sys/mman.h presence... yes\n",
            "checking for sys/mman.h... yes\n",
            "checking sys/times.h usability... yes\n",
            "checking sys/times.h presence... yes\n",
            "checking for sys/times.h... yes\n",
            "checking for sys/types.h... (cached) yes\n",
            "checking dirent.h usability... yes\n",
            "checking dirent.h presence... yes\n",
            "checking for dirent.h... yes\n",
            "checking ctype.h usability... yes\n",
            "checking ctype.h presence... yes\n",
            "checking for ctype.h... yes\n",
            "checking for sys/types.h... (cached) yes\n",
            "checking io.h usability... no\n",
            "checking io.h presence... no\n",
            "checking for io.h... no\n",
            "checking windows.h usability... no\n",
            "checking windows.h presence... no\n",
            "checking for windows.h... no\n",
            "checking pthread.h usability... yes\n",
            "checking pthread.h presence... yes\n",
            "checking for pthread.h... yes\n",
            "checking for off_t... yes\n",
            "checking for size_t... yes\n",
            "checking size of char... 1\n",
            "checking size of short... 2\n",
            "checking size of int... 4\n",
            "checking size of long... 8\n",
            "checking size of long long... 8\n",
            "checking size of size_t... 8\n",
            "checking for size_t... (cached) yes\n",
            "checking for unsigned long long int... yes\n",
            "checking for stdlib.h... (cached) yes\n",
            "checking for unistd.h... (cached) yes\n",
            "checking for sys/param.h... yes\n",
            "checking for getpagesize... yes\n",
            "checking for working mmap... yes\n",
            "checking for main in -lstdc++... yes\n",
            "checking for pthread_create in -lpthread... yes\n",
            "checking for pthread_join in -lpthread... yes\n",
            "checking for getenv... yes\n",
            "checking for opendir... yes\n",
            "checking whether make is GNU Make... yes\n",
            "checking if g++ supports stl <vector> (required)... yes\n",
            "checking if g++ supports stl <list> (required)... yes\n",
            "checking if g++ supports stl <map> (required)... yes\n",
            "checking if g++ supports stl <set> (required)... yes\n",
            "checking if g++ supports stl <queue> (required)... yes\n",
            "checking if g++ supports stl <functional> (required)... yes\n",
            "checking if g++ supports stl <algorithm> (required)... yes\n",
            "checking if g++ supports stl <string> (required)... yes\n",
            "checking if g++ supports stl <iostream> (required)... yes\n",
            "checking if g++ supports stl <sstream> (required)... yes\n",
            "checking if g++ supports stl <fstream> (required)... yes\n",
            "checking if g++ supports template <class T> (required)... yes\n",
            "checking if g++ supports const_cast<> (required)... yes\n",
            "checking if g++ supports static_cast<> (required)... yes\n",
            "checking if g++ supports reinterpret_cast<> (required)... yes\n",
            "checking if g++ supports namespaces (required) ... yes\n",
            "checking if g++ supports __thread (optional)... yes\n",
            "checking if g++ supports template <class T> (required)... yes\n",
            "checking if g++ supports GCC native atomic operations (optional)... yes\n",
            "checking if g++ supports OSX native atomic operations (optional)... no\n",
            "checking if g++ environment provides all required features... yes\n",
            "configure: creating ./config.status\n",
            "config.status: creating Makefile\n",
            "config.status: creating src/Makefile\n",
            "config.status: creating src/Makefile.msvc\n",
            "config.status: creating man/Makefile\n",
            "config.status: creating doc/Makefile\n",
            "config.status: creating tests/Makefile\n",
            "config.status: creating swig/version.h\n",
            "config.status: creating mecab.iss\n",
            "config.status: creating mecab-config\n",
            "config.status: creating mecabrc\n",
            "config.status: creating config.h\n",
            "config.status: config.h is unchanged\n",
            "config.status: executing depfiles commands\n",
            "config.status: executing libtool commands\n",
            "config.status: executing default commands\n",
            "make  all-recursive\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "Making all in src\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/src'\n",
            "make[2]: Nothing to be done for 'all'.\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/src'\n",
            "Making all in man\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/man'\n",
            "make[2]: Nothing to be done for 'all'.\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/man'\n",
            "Making all in doc\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/doc'\n",
            "make[2]: Nothing to be done for 'all'.\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/doc'\n",
            "Making all in tests\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "make[2]: Nothing to be done for 'all'.\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "Making check in src\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/src'\n",
            "make[1]: Nothing to be done for 'check'.\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/src'\n",
            "Making check in man\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/man'\n",
            "make[1]: Nothing to be done for 'check'.\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/man'\n",
            "Making check in doc\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/doc'\n",
            "make[1]: Nothing to be done for 'check'.\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/doc'\n",
            "Making check in tests\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "make  check-TESTS\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./unk.def ... 2\n",
            "emitting double-array: 100% |###########################################| \n",
            "./model.def is not found. skipped.\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./dic.csv ... 177\n",
            "emitting double-array: 100% |###########################################| \n",
            "reading ./matrix.def ... 178x178\n",
            "emitting matrix      : 100% |###########################################| \n",
            "\n",
            "done!\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./unk.def ... 2\n",
            "emitting double-array: 100% |###########################################| \n",
            "./model.def is not found. skipped.\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./dic.csv ... 83\n",
            "emitting double-array: 100% |###########################################| \n",
            "reading ./matrix.def ... 84x84\n",
            "emitting matrix      : 100% |###########################################| \n",
            "\n",
            "done!\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./unk.def ... 2\n",
            "emitting double-array: 100% |###########################################| \n",
            "./model.def is not found. skipped.\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./dic.csv ... 450\n",
            "emitting double-array: 100% |###########################################| \n",
            "reading ./matrix.def ... 1x1\n",
            "\n",
            "done!\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./unk.def ... 2\n",
            "emitting double-array: 100% |###########################################| \n",
            "./model.def is not found. skipped.\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./dic.csv ... 162\n",
            "emitting double-array: 100% |###########################################| \n",
            "reading ./matrix.def ... 3x3\n",
            "emitting matrix      : 100% |###########################################| \n",
            "\n",
            "done!\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./unk.def ... 2\n",
            "emitting double-array: 100% |###########################################| \n",
            "./model.def is not found. skipped.\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./dic.csv ... 4\n",
            "emitting double-array: 100% |###########################################| \n",
            "reading ./matrix.def ... 1x1\n",
            "\n",
            "done!\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./unk.def ... 11\n",
            "emitting double-array: 100% |###########################################| \n",
            "./model.def is not found. skipped.\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./dic.csv ... 1\n",
            "reading ./matrix.def ... 1x1\n",
            "\n",
            "done!\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./unk.def ... 2\n",
            "emitting double-array: 100% |###########################################| \n",
            "./model.def is not found. skipped.\n",
            "./pos-id.def is not found. minimum setting is used\n",
            "reading ./dic.csv ... 1\n",
            "reading ./matrix.def ... 1x1\n",
            "\n",
            "done!\n",
            "PASS: run-dics.sh\n",
            "PASS: run-eval.sh\n",
            "seed/pos-id.def is not found. minimum setting is used\n",
            "reading seed/unk.def ... 40\n",
            "emitting double-array: 100% |###########################################| \n",
            "seed/model.def is not found. skipped.\n",
            "seed/pos-id.def is not found. minimum setting is used\n",
            "reading seed/dic.csv ... 4335\n",
            "emitting double-array: 100% |###########################################| \n",
            "reading seed/matrix.def ... 1x1\n",
            "\n",
            "done!\n",
            "reading corpus ...\n",
            "Number of sentences: 34\n",
            "Number of features:  64108\n",
            "eta:                 0.00005\n",
            "freq:                1\n",
            "eval-size:           6\n",
            "unk-eval-size:       4\n",
            "threads:             1\n",
            "charset:             EUC-JP\n",
            "C(sigma^2):          1.00000\n",
            "\n",
            "iter=0 err=1.00000 F=0.35771 target=2406.28355 diff=1.00000\n",
            "iter=1 err=0.97059 F=0.65652 target=1484.25231 diff=0.38318\n",
            "iter=2 err=0.91176 F=0.79331 target=863.32765 diff=0.41834\n",
            "iter=3 err=0.85294 F=0.89213 target=596.72480 diff=0.30881\n",
            "iter=4 err=0.61765 F=0.95467 target=336.30744 diff=0.43641\n",
            "iter=5 err=0.50000 F=0.96702 target=246.53039 diff=0.26695\n",
            "iter=6 err=0.35294 F=0.95472 target=188.93963 diff=0.23361\n",
            "iter=7 err=0.20588 F=0.99106 target=168.62665 diff=0.10751\n",
            "iter=8 err=0.05882 F=0.99777 target=158.64865 diff=0.05917\n",
            "iter=9 err=0.08824 F=0.99665 target=154.14530 diff=0.02839\n",
            "iter=10 err=0.08824 F=0.99665 target=151.94257 diff=0.01429\n",
            "iter=11 err=0.02941 F=0.99888 target=147.20825 diff=0.03116\n",
            "iter=12 err=0.00000 F=1.00000 target=147.34956 diff=0.00096\n",
            "iter=13 err=0.02941 F=0.99888 target=146.32592 diff=0.00695\n",
            "iter=14 err=0.00000 F=1.00000 target=145.77299 diff=0.00378\n",
            "iter=15 err=0.02941 F=0.99888 target=145.24641 diff=0.00361\n",
            "iter=16 err=0.00000 F=1.00000 target=144.96490 diff=0.00194\n",
            "iter=17 err=0.02941 F=0.99888 target=144.90246 diff=0.00043\n",
            "iter=18 err=0.00000 F=1.00000 target=144.75959 diff=0.00099\n",
            "iter=19 err=0.00000 F=1.00000 target=144.71727 diff=0.00029\n",
            "iter=20 err=0.00000 F=1.00000 target=144.66337 diff=0.00037\n",
            "iter=21 err=0.00000 F=1.00000 target=144.61349 diff=0.00034\n",
            "iter=22 err=0.00000 F=1.00000 target=144.62987 diff=0.00011\n",
            "iter=23 err=0.00000 F=1.00000 target=144.60060 diff=0.00020\n",
            "iter=24 err=0.00000 F=1.00000 target=144.59125 diff=0.00006\n",
            "iter=25 err=0.00000 F=1.00000 target=144.58619 diff=0.00004\n",
            "iter=26 err=0.00000 F=1.00000 target=144.58219 diff=0.00003\n",
            "iter=27 err=0.00000 F=1.00000 target=144.58059 diff=0.00001\n",
            "\n",
            "Done! writing model file ... \n",
            "model-ipadic.c1.0.f1.model is not a binary model. reopen it as text mode...\n",
            "reading seed/unk.def ... 40\n",
            "reading seed/dic.csv ... 4335\n",
            "emitting model-ipadic.c1.0.f1.dic/left-id.def/ model-ipadic.c1.0.f1.dic/right-id.def\n",
            "emitting model-ipadic.c1.0.f1.dic/unk.def ... 40\n",
            "emitting model-ipadic.c1.0.f1.dic/dic.csv ... 4335\n",
            "emitting matrix      : 100% |###########################################| \n",
            "copying seed/char.def to model-ipadic.c1.0.f1.dic/char.def\n",
            "copying seed/rewrite.def to model-ipadic.c1.0.f1.dic/rewrite.def\n",
            "copying seed/dicrc to model-ipadic.c1.0.f1.dic/dicrc\n",
            "copying seed/feature.def to model-ipadic.c1.0.f1.dic/feature.def\n",
            "copying model-ipadic.c1.0.f1.model to model-ipadic.c1.0.f1.dic/model.def\n",
            "\n",
            "done!\n",
            "model-ipadic.c1.0.f1.dic/pos-id.def is not found. minimum setting is used\n",
            "reading model-ipadic.c1.0.f1.dic/unk.def ... 40\n",
            "emitting double-array: 100% |###########################################| \n",
            "model-ipadic.c1.0.f1.dic/pos-id.def is not found. minimum setting is used\n",
            "reading model-ipadic.c1.0.f1.dic/dic.csv ... 4335\n",
            "emitting double-array: 100% |###########################################| \n",
            "reading model-ipadic.c1.0.f1.dic/matrix.def ... 346x346\n",
            "emitting matrix      : 100% |###########################################| \n",
            "\n",
            "done!\n",
            "              precision          recall         F\n",
            "LEVEL 0:    12.8959(57/442) 11.8998(57/479) 12.3779\n",
            "LEVEL 1:    12.2172(54/442) 11.2735(54/479) 11.7264\n",
            "LEVEL 2:    11.7647(52/442) 10.8559(52/479) 11.2921\n",
            "LEVEL 4:    11.7647(52/442) 10.8559(52/479) 11.2921\n",
            "PASS: run-cost-train.sh\n",
            "==================\n",
            "All 3 tests passed\n",
            "==================\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "Making install in src\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/src'\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/src'\n",
            "test -z \"/usr/local/lib\" || /usr/bin/mkdir -p \"/usr/local/lib\"\n",
            " /bin/bash ../libtool   --mode=install /usr/bin/install -c   libmecab.la '/usr/local/lib'\n",
            "libtool: install: /usr/bin/install -c .libs/libmecab.so.2.0.0 /usr/local/lib/libmecab.so.2.0.0\n",
            "libtool: install: (cd /usr/local/lib && { ln -s -f libmecab.so.2.0.0 libmecab.so.2 || { rm -f libmecab.so.2 && ln -s libmecab.so.2.0.0 libmecab.so.2; }; })\n",
            "libtool: install: (cd /usr/local/lib && { ln -s -f libmecab.so.2.0.0 libmecab.so || { rm -f libmecab.so && ln -s libmecab.so.2.0.0 libmecab.so; }; })\n",
            "libtool: install: /usr/bin/install -c .libs/libmecab.lai /usr/local/lib/libmecab.la\n",
            "libtool: install: /usr/bin/install -c .libs/libmecab.a /usr/local/lib/libmecab.a\n",
            "libtool: install: chmod 644 /usr/local/lib/libmecab.a\n",
            "libtool: install: ranlib /usr/local/lib/libmecab.a\n",
            "libtool: finish: PATH=\"/opt/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tools/node/bin:/tools/google-cloud-sdk/bin:/sbin\" ldconfig -n /usr/local/lib\n",
            "----------------------------------------------------------------------\n",
            "Libraries have been installed in:\n",
            "   /usr/local/lib\n",
            "\n",
            "If you ever happen to want to link against installed libraries\n",
            "in a given directory, LIBDIR, you must either use libtool, and\n",
            "specify the full pathname of the library, or use the `-LLIBDIR'\n",
            "flag during linking and do at least one of the following:\n",
            "   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable\n",
            "     during execution\n",
            "   - add LIBDIR to the `LD_RUN_PATH' environment variable\n",
            "     during linking\n",
            "   - use the `-Wl,-rpath -Wl,LIBDIR' linker flag\n",
            "   - have your system administrator add LIBDIR to `/etc/ld.so.conf'\n",
            "\n",
            "See any operating system documentation about shared libraries for\n",
            "more information, such as the ld(1) and ld.so(8) manual pages.\n",
            "----------------------------------------------------------------------\n",
            "test -z \"/usr/local/bin\" || /usr/bin/mkdir -p \"/usr/local/bin\"\n",
            "  /bin/bash ../libtool   --mode=install /usr/bin/install -c mecab '/usr/local/bin'\n",
            "libtool: install: /usr/bin/install -c .libs/mecab /usr/local/bin/mecab\n",
            "test -z \"/usr/local/libexec/mecab\" || /usr/bin/mkdir -p \"/usr/local/libexec/mecab\"\n",
            "  /bin/bash ../libtool   --mode=install /usr/bin/install -c mecab-dict-index mecab-dict-gen mecab-cost-train mecab-system-eval mecab-test-gen '/usr/local/libexec/mecab'\n",
            "libtool: install: /usr/bin/install -c .libs/mecab-dict-index /usr/local/libexec/mecab/mecab-dict-index\n",
            "libtool: install: /usr/bin/install -c .libs/mecab-dict-gen /usr/local/libexec/mecab/mecab-dict-gen\n",
            "libtool: install: /usr/bin/install -c .libs/mecab-cost-train /usr/local/libexec/mecab/mecab-cost-train\n",
            "libtool: install: /usr/bin/install -c .libs/mecab-system-eval /usr/local/libexec/mecab/mecab-system-eval\n",
            "libtool: install: /usr/bin/install -c .libs/mecab-test-gen /usr/local/libexec/mecab/mecab-test-gen\n",
            "test -z \"/usr/local/include\" || /usr/bin/mkdir -p \"/usr/local/include\"\n",
            " /usr/bin/install -c -m 644 mecab.h '/usr/local/include'\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/src'\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/src'\n",
            "Making install in man\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/man'\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/man'\n",
            "make[2]: Nothing to be done for 'install-exec-am'.\n",
            "test -z \"/usr/local/share/man/man1\" || /usr/bin/mkdir -p \"/usr/local/share/man/man1\"\n",
            " /usr/bin/install -c -m 644 mecab.1 '/usr/local/share/man/man1'\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/man'\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/man'\n",
            "Making install in doc\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/doc'\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/doc'\n",
            "make[2]: Nothing to be done for 'install-exec-am'.\n",
            "make[2]: Nothing to be done for 'install-data-am'.\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/doc'\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/doc'\n",
            "Making install in tests\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "make[2]: Nothing to be done for 'install-exec-am'.\n",
            "make[2]: Nothing to be done for 'install-data-am'.\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2/tests'\n",
            "make[1]: Entering directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "make[2]: Entering directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "test -z \"/usr/local/bin\" || /usr/bin/mkdir -p \"/usr/local/bin\"\n",
            " /usr/bin/install -c mecab-config '/usr/local/bin'\n",
            "test -z \"/usr/local/etc\" || /usr/bin/mkdir -p \"/usr/local/etc\"\n",
            " /usr/bin/install -c -m 644 mecabrc '/usr/local/etc'\n",
            "make[2]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "make[1]: Leaving directory '/tmp/mecab-0.996-ko-0.9.2'\n",
            "--2023-04-29 11:00:16--  https://www.dropbox.com/s/i8girnk5p80076c/mecab-ko-dic-2.1.1-20180720.tar.gz?dl=1\n",
            "Resolving www.dropbox.com (www.dropbox.com)... 162.125.3.18, 2620:100:6030:18::a27d:5012\n",
            "Connecting to www.dropbox.com (www.dropbox.com)|162.125.3.18|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: /s/dl/i8girnk5p80076c/mecab-ko-dic-2.1.1-20180720.tar.gz [following]\n",
            "--2023-04-29 11:00:16--  https://www.dropbox.com/s/dl/i8girnk5p80076c/mecab-ko-dic-2.1.1-20180720.tar.gz\n",
            "Reusing existing connection to www.dropbox.com:443.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://ucd57e69720f910b799d43b4f76a.dl.dropboxusercontent.com/cd/0/get/B7GT06YgeOKOrFjnLx_Uf7ZJNrF5igtWu0Nnz0vVHzyYsdgScAtNZ_t911_9FGJS4YgZA2X0qvRKQS6Cm7Zc9TqmcVzsYt27ai4e2UCFBowgXQ_l91AvUygPwHsE8jrhc3b_he96ZnTRFYGUAwkWleQJyCjnYNqUnBy1l3BaHHdTYjcYfpra4Txm_heX_-5iuNk/file?dl=1# [following]\n",
            "--2023-04-29 11:00:17--  https://ucd57e69720f910b799d43b4f76a.dl.dropboxusercontent.com/cd/0/get/B7GT06YgeOKOrFjnLx_Uf7ZJNrF5igtWu0Nnz0vVHzyYsdgScAtNZ_t911_9FGJS4YgZA2X0qvRKQS6Cm7Zc9TqmcVzsYt27ai4e2UCFBowgXQ_l91AvUygPwHsE8jrhc3b_he96ZnTRFYGUAwkWleQJyCjnYNqUnBy1l3BaHHdTYjcYfpra4Txm_heX_-5iuNk/file?dl=1\n",
            "Resolving ucd57e69720f910b799d43b4f76a.dl.dropboxusercontent.com (ucd57e69720f910b799d43b4f76a.dl.dropboxusercontent.com)... 162.125.3.15, 2620:100:601d:15::a27d:50f\n",
            "Connecting to ucd57e69720f910b799d43b4f76a.dl.dropboxusercontent.com (ucd57e69720f910b799d43b4f76a.dl.dropboxusercontent.com)|162.125.3.15|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 49775061 (47M) [application/binary]\n",
            "Saving to: ‘mecab-ko-dic-2.1.1-20180720.tar.gz?dl=1.1’\n",
            "\n",
            "mecab-ko-dic-2.1.1- 100%[===================>]  47.47M  90.3MB/s    in 0.5s    \n",
            "\n",
            "2023-04-29 11:00:18 (90.3 MB/s) - ‘mecab-ko-dic-2.1.1-20180720.tar.gz?dl=1.1’ saved [49775061/49775061]\n",
            "\n",
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "autoconf is already the newest version (2.69-11.1).\n",
            "0 upgraded, 0 newly installed, 0 to remove and 24 not upgraded.\n",
            "mecab-ko-dic-2.1.1-20180720/\n",
            "mecab-ko-dic-2.1.1-20180720/configure\n",
            "mecab-ko-dic-2.1.1-20180720/COPYING\n",
            "mecab-ko-dic-2.1.1-20180720/autogen.sh\n",
            "mecab-ko-dic-2.1.1-20180720/Place-station.csv\n",
            "mecab-ko-dic-2.1.1-20180720/NNG.csv\n",
            "mecab-ko-dic-2.1.1-20180720/README\n",
            "mecab-ko-dic-2.1.1-20180720/EF.csv\n",
            "mecab-ko-dic-2.1.1-20180720/MAG.csv\n",
            "mecab-ko-dic-2.1.1-20180720/Preanalysis.csv\n",
            "mecab-ko-dic-2.1.1-20180720/NNB.csv\n",
            "mecab-ko-dic-2.1.1-20180720/Person-actor.csv\n",
            "mecab-ko-dic-2.1.1-20180720/VV.csv\n",
            "mecab-ko-dic-2.1.1-20180720/Makefile.in\n",
            "mecab-ko-dic-2.1.1-20180720/matrix.def\n",
            "mecab-ko-dic-2.1.1-20180720/EC.csv\n",
            "mecab-ko-dic-2.1.1-20180720/NNBC.csv\n",
            "mecab-ko-dic-2.1.1-20180720/clean\n",
            "mecab-ko-dic-2.1.1-20180720/ChangeLog\n",
            "mecab-ko-dic-2.1.1-20180720/J.csv\n",
            "mecab-ko-dic-2.1.1-20180720/.keep\n",
            "mecab-ko-dic-2.1.1-20180720/feature.def\n",
            "mecab-ko-dic-2.1.1-20180720/Foreign.csv\n",
            "mecab-ko-dic-2.1.1-20180720/XPN.csv\n",
            "mecab-ko-dic-2.1.1-20180720/EP.csv\n",
            "mecab-ko-dic-2.1.1-20180720/NR.csv\n",
            "mecab-ko-dic-2.1.1-20180720/left-id.def\n",
            "mecab-ko-dic-2.1.1-20180720/Place.csv\n",
            "mecab-ko-dic-2.1.1-20180720/Symbol.csv\n",
            "mecab-ko-dic-2.1.1-20180720/dicrc\n",
            "mecab-ko-dic-2.1.1-20180720/NP.csv\n",
            "mecab-ko-dic-2.1.1-20180720/ETM.csv\n",
            "mecab-ko-dic-2.1.1-20180720/IC.csv\n",
            "mecab-ko-dic-2.1.1-20180720/Place-address.csv\n",
            "mecab-ko-dic-2.1.1-20180720/Group.csv\n",
            "mecab-ko-dic-2.1.1-20180720/model.def\n",
            "mecab-ko-dic-2.1.1-20180720/XSN.csv\n",
            "mecab-ko-dic-2.1.1-20180720/INSTALL\n",
            "mecab-ko-dic-2.1.1-20180720/rewrite.def\n",
            "mecab-ko-dic-2.1.1-20180720/Inflect.csv\n",
            "mecab-ko-dic-2.1.1-20180720/configure.ac\n",
            "mecab-ko-dic-2.1.1-20180720/NNP.csv\n",
            "mecab-ko-dic-2.1.1-20180720/CoinedWord.csv\n",
            "mecab-ko-dic-2.1.1-20180720/XSV.csv\n",
            "mecab-ko-dic-2.1.1-20180720/pos-id.def\n",
            "mecab-ko-dic-2.1.1-20180720/Makefile.am\n",
            "mecab-ko-dic-2.1.1-20180720/unk.def\n",
            "mecab-ko-dic-2.1.1-20180720/missing\n",
            "mecab-ko-dic-2.1.1-20180720/VCP.csv\n",
            "mecab-ko-dic-2.1.1-20180720/install-sh\n",
            "mecab-ko-dic-2.1.1-20180720/Hanja.csv\n",
            "mecab-ko-dic-2.1.1-20180720/MAJ.csv\n",
            "mecab-ko-dic-2.1.1-20180720/XSA.csv\n",
            "mecab-ko-dic-2.1.1-20180720/Wikipedia.csv\n",
            "mecab-ko-dic-2.1.1-20180720/tools/\n",
            "mecab-ko-dic-2.1.1-20180720/tools/add-userdic.sh\n",
            "mecab-ko-dic-2.1.1-20180720/tools/mecab-bestn.sh\n",
            "mecab-ko-dic-2.1.1-20180720/tools/convert_for_using_store.sh\n",
            "mecab-ko-dic-2.1.1-20180720/user-dic/\n",
            "mecab-ko-dic-2.1.1-20180720/user-dic/nnp.csv\n",
            "mecab-ko-dic-2.1.1-20180720/user-dic/place.csv\n",
            "mecab-ko-dic-2.1.1-20180720/user-dic/person.csv\n",
            "mecab-ko-dic-2.1.1-20180720/user-dic/README.md\n",
            "mecab-ko-dic-2.1.1-20180720/NorthKorea.csv\n",
            "mecab-ko-dic-2.1.1-20180720/VX.csv\n",
            "mecab-ko-dic-2.1.1-20180720/right-id.def\n",
            "mecab-ko-dic-2.1.1-20180720/VA.csv\n",
            "mecab-ko-dic-2.1.1-20180720/char.def\n",
            "mecab-ko-dic-2.1.1-20180720/NEWS\n",
            "mecab-ko-dic-2.1.1-20180720/MM.csv\n",
            "mecab-ko-dic-2.1.1-20180720/ETN.csv\n",
            "mecab-ko-dic-2.1.1-20180720/AUTHORS\n",
            "mecab-ko-dic-2.1.1-20180720/Person.csv\n",
            "mecab-ko-dic-2.1.1-20180720/XR.csv\n",
            "mecab-ko-dic-2.1.1-20180720/VCN.csv\n",
            "Looking in current directory for macros.\n",
            "configure.ac:2: warning: AM_INIT_AUTOMAKE: two- and three-arguments forms are deprecated.  For more info, see:\n",
            "configure.ac:2: https://www.gnu.org/software/automake/manual/automake.html#Modernize-AM_005fINIT_005fAUTOMAKE-invocation\n",
            "checking for a BSD-compatible install... /usr/bin/install -c\n",
            "checking whether build environment is sane... yes\n",
            "/tmp/mecab-ko-dic-2.1.1-20180720/missing: Unknown `--is-lightweight' option\n",
            "Try `/tmp/mecab-ko-dic-2.1.1-20180720/missing --help' for more information\n",
            "configure: WARNING: 'missing' script is too old or missing\n",
            "checking for a thread-safe mkdir -p... /usr/bin/mkdir -p\n",
            "checking for gawk... no\n",
            "checking for mawk... mawk\n",
            "checking whether make sets $(MAKE)... yes\n",
            "checking whether make supports nested variables... yes\n",
            "checking for mecab-config... /usr/local/bin/mecab-config\n",
            "checking that generated files are newer than configure... done\n",
            "configure: creating ./config.status\n",
            "config.status: creating Makefile\n",
            "make: Nothing to be done for 'all'.\n",
            "make[1]: Entering directory '/tmp/mecab-ko-dic-2.1.1-20180720'\n",
            "make[1]: Nothing to be done for 'install-exec-am'.\n",
            " /usr/bin/mkdir -p '/usr/local/lib/mecab/dic/mecab-ko-dic'\n",
            " /usr/bin/install -c -m 644 model.bin matrix.bin char.bin sys.dic unk.dic left-id.def right-id.def rewrite.def pos-id.def dicrc '/usr/local/lib/mecab/dic/mecab-ko-dic'\n",
            "make[1]: Leaving directory '/tmp/mecab-ko-dic-2.1.1-20180720'\n",
            "fatal: destination path 'mecab-python-0.996' already exists and is not an empty directory.\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: konlpy in /usr/local/lib/python3.10/dist-packages (0.6.0)\n",
            "Requirement already satisfied: numpy>=1.6 in /usr/local/lib/python3.10/dist-packages (from konlpy) (1.22.4)\n",
            "Requirement already satisfied: lxml>=4.1.0 in /usr/local/lib/python3.10/dist-packages (from konlpy) (4.9.2)\n",
            "Requirement already satisfied: JPype1>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from konlpy) (1.4.1)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from JPype1>=0.7.0->konlpy) (23.1)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#1. NLTK에서 불용어 확인하기\n",
        "import nltk\n",
        "nltk.download('stopwords')\n",
        "stop_words_list = stopwords.words('english')\n",
        "print(\"불용어 개수 :\", len(stop_words_list))\n",
        "print(\"불용어 10개 출력 :\", stop_words_list[:10])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HNRFQmq8p9tI",
        "outputId": "a02764c0-9e73-45e1-dcec-cbfae883b0f8"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "불용어 개수 : 179\n",
            "불용어 10개 출력 : ['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\"]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#2. NLTK를 통해서 불용어 제거하기\n",
        "import nltk\n",
        "nltk.download('punkt')\n",
        "example = \"Family is not an important thing. It's everything.\"\n",
        "stop_words = set(stopwords.words('english'))\n",
        "\n",
        "word_tokens = word_tokenize(example)\n",
        "result = []\n",
        "for word in word_tokens:\n",
        "  if word not in stop_words:\n",
        "    result.append(word)\n",
        "\n",
        "print(\"불용어 제거 전 :\", word_tokens)\n",
        "print(\"불용어 제거 후 :\", result)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3HTYUnlIvwsG",
        "outputId": "65cb4720-7a57-44bb-c8bd-d1d8bd3d669e"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "불용어 제거 전 : ['Family', 'is', 'not', 'an', 'important', 'thing', '.', 'It', \"'s\", 'everything', '.']\n",
            "불용어 제거 후 : ['Family', 'important', 'thing', '.', 'It', \"'s\", 'everything', '.']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#3. 한국어에서 불용어 제거하기\n",
        "\n",
        "okt = Okt()\n",
        "\n",
        "example = \"고기를 아무렇게나 구우려고 하면 안 돼. 고기라고 다 같은 게 아니거든. 예컨대 삼겹살을 구울 때는 중요한 게 있지.\"\n",
        "stop_words = \"를 아무렇게나 구 우려 고 안 돼 같은 게 구울 때 는\"\n",
        "\n",
        "stop_words = set(stop_words.split(' '))\n",
        "word_tokens = okt.morphs(example) #형태소 추출\n",
        "\n",
        "result = []\n",
        "for word in word_tokens:\n",
        "  if word not in stop_words:\n",
        "    result.append(word)\n",
        "\n",
        "print(\"불용어 제거 전 :\", word_tokens)\n",
        "print(\"불용어 제거 후 :\", result)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Q2-gVmZAxr_L",
        "outputId": "f8b6e4cb-2469-43cd-aca8-0398ee9fc382"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "불용어 제거 전 : ['고기', '를', '아무렇게나', '구', '우려', '고', '하면', '안', '돼', '.', '고기', '라고', '다', '같은', '게', '아니거든', '.', '예컨대', '삼겹살', '을', '구울', '때', '는', '중요한', '게', '있지', '.']\n",
            "불용어 제거 후 : ['고기', '하면', '.', '고기', '라고', '다', '아니거든', '.', '예컨대', '삼겹살', '을', '중요한', '있지', '.']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "_ntO4tuzy91k"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}