code/ch08/ch08.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8장. 감성 분석에 머신 러닝 적용하기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**아래 링크를 통해 이 노트북을 주피터 노트북 뷰어(nbviewer.jupyter.org)로 보거나 구글 코랩(colab.research.google.com)에서 실행할 수 있습니다.**\n",
    "\n",
    "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://nbviewer.jupyter.org/github/rickiepark/python-machine-learning-book-2nd-edition/blob/master/code/ch08/ch08.ipynb\"><img src=\"https://jupyter.org/assets/main-logo.svg\" width=\"28\" />주피터 노트북 뷰어로 보기</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/rickiepark/python-machine-learning-book-2nd-edition/blob/master/code/ch08/ch08.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />구글 코랩(Colab)에서 실행하기</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`watermark`는 주피터 노트북에 사용하는 파이썬 패키지를 출력하기 위한 유틸리티입니다. `watermark` 패키지를 설치하려면 다음 셀의 주석을 제거한 뒤 실행하세요."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install watermark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "last updated: 2020-05-22 \n",
      "\n",
      "CPython 3.7.3\n",
      "IPython 7.5.0\n",
      "\n",
      "numpy 1.18.4\n",
      "pandas 1.0.3\n",
      "sklearn 0.23.1\n",
      "nltk 3.4.1\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -u -d -v -p numpy,pandas,sklearn,nltk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 텍스트 처리용 IMDb 영화 리뷰 데이터 준비 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 영화 리뷰 데이터셋 구하기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IMDB 영화 리뷰 데이터셋은 [http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz](http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz)에서 내려받을 수 있습니다. 다운로드된 후 파일 압축을 해제합니다.\n",
    "\n",
    "A) 리눅스(Linux)나 macOS를 사용하면 새로운 터미널(Terminal) 윈도우를 열고 `cd` 명령으로 다운로드 디렉터리로 이동하여 다음 명령을 실행하세요. \n",
    "\n",
    "`tar -zxf aclImdb_v1.tar.gz`\n",
    "\n",
    "B) 윈도(Windows)를 사용하면 7Zip(http://www.7-zip.org) 같은 무료 압축 유틸리티를 설치하여 다운로드한 파일의 압축을 풀 수 있습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**코랩이나 리눅스에서 직접 다운로드하려면 다음 셀의 주석을 제거하고 실행하세요.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!wget http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**다음처럼 파이썬에서 직접 압축을 풀 수도 있습니다:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import tarfile\n",
    "\n",
    "if not os.path.isdir('aclImdb'):\n",
    "\n",
    "    with tarfile.open('aclImdb_v1.tar.gz', 'r:gz') as tar:\n",
    "        tar.extractall()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 영화 리뷰 데이터셋을 더 간편한 형태로 전처리하기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`pyprind`는 주피터 노트북에서 진행바를 출력하기 위한 유틸리티입니다. `pyprind` 패키지를 설치하려면 다음 셀의 주석을 제거한 뒤 실행하세요."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install pyprind"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "0% [##############################] 100% | ETA: 00:00:00\n",
      "Total time elapsed: 00:01:21\n"
     ]
    }
   ],
   "source": [
    "import pyprind\n",
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "# `basepath`를 압축 해제된 영화 리뷰 데이터셋이 있는\n",
    "# 디렉토리로 바꾸세요\n",
    "\n",
    "basepath = 'aclImdb'\n",
    "\n",
    "labels = {'pos': 1, 'neg': 0}\n",
    "pbar = pyprind.ProgBar(50000)\n",
    "df = pd.DataFrame()\n",
    "for s in ('test', 'train'):\n",
    "    for l in ('pos', 'neg'):\n",
    "        path = os.path.join(basepath, s, l)\n",
    "        for file in sorted(os.listdir(path)):\n",
    "            with open(os.path.join(path, file), \n",
    "                      'r', encoding='utf-8') as infile:\n",
    "                txt = infile.read()\n",
    "            df = df.append([[txt, labels[l]]], \n",
    "                           ignore_index=True)\n",
    "            pbar.update()\n",
    "df.columns = ['review', 'sentiment']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "데이터프레임을 섞습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "np.random.seed(0)\n",
    "df = df.reindex(np.random.permutation(df.index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "선택사항: 만들어진 데이터를 CSV 파일로 저장합니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv('movie_data.csv', index=False, encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>In 1974, the teenager Martha Moxley (Maggie Gr...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>OK... so... I really like Kris Kristofferson a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>***SPOILER*** Do not read this, if you think a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              review  sentiment\n",
       "0  In 1974, the teenager Martha Moxley (Maggie Gr...          1\n",
       "1  OK... so... I really like Kris Kristofferson a...          0\n",
       "2  ***SPOILER*** Do not read this, if you think a...          0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('movie_data.csv', encoding='utf-8')\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50000, 2)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BoW 모델 소개"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 단어를 특성 벡터로 변환하기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CountVectorizer의 fit_transform 메서드를 호출하여 BoW 모델의 어휘사전을 만들고 다음 세 문장을 희소한 특성 벡터로 변환합니다:\n",
    "1. The sun is shining\n",
    "2. The weather is sweet\n",
    "3. The sun is shining, the weather is sweet, and one and one is two"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "count = CountVectorizer()\n",
    "docs = np.array([\n",
    "        'The sun is shining',\n",
    "        'The weather is sweet',\n",
    "        'The sun is shining, the weather is sweet, and one and one is two'])\n",
    "bag = count.fit_transform(docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "어휘 사전의 내용을 출력해 보면 BoW 모델의 개념을 이해하는 데 도움이 됩니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'the': 6, 'sun': 4, 'is': 1, 'shining': 3, 'weather': 8, 'sweet': 5, 'and': 0, 'one': 2, 'two': 7}\n"
     ]
    }
   ],
   "source": [
    "print(count.vocabulary_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이전 결과에서 볼 수 있듯이 어휘 사전은 고유 단어와 정수 인덱스가 매핑된 파이썬 딕셔너리에 저장되어 있습니다. 그다음 만들어진 특성 벡터를 출력해 봅시다:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "특성 벡터의 각 인덱스는 CountVectorizer의 어휘 사전 딕셔너리에 저장된 정수 값에 해당됩니다. 예를 들어 인덱스 0에 있는 첫 번째 특성은 ‘and’ 단어의 카운트를 의미합니다. 이 단어는 마지막 문서에만 나타나네요. 인덱스 1에 있는 (특성 벡터의 두 번째 열) 단어 ‘is’는 세 문장에 모두 등장합니다. 특성 벡터의 이런 값들을 단어 빈도(term frequency) 라고도 부릅니다. 문서 d에 등장한 단어 t의 횟수를 *tf (t,d)*와 같이 씁니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0 1 0 1 1 0 1 0 0]\n",
      " [0 1 0 0 0 1 1 0 1]\n",
      " [2 3 2 1 1 1 2 1 1]]\n"
     ]
    }
   ],
   "source": [
    "print(bag.toarray())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## tf-idf를 사용해 단어 적합성 평가하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.set_printoptions(precision=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "텍스트 데이터를 분석할 때 클래스 레이블이 다른 문서에 같은 단어들이 나타나는 경우를 종종 보게 됩니다. 일반적으로 자주 등장하는 단어는 유용하거나 판별에 필요한 정보를 가지고 있지 않습니다. 이 절에서 특성 벡터에서 자주 등장하는 단어의 가중치를 낮추는 기법인 tf-idf(term frequency-inverse document frequency)에 대해 배우겠습니다. tf-idf는 단어 빈도와 역문서 빈도(inverse document frequency)의 곱으로 정의됩니다:\n",
    "\n",
    "$$\\text{tf-idf}(t,d)=\\text{tf (t,d)}\\times \\text{idf}(t,d)$$\n",
    "\n",
    "여기에서 tf(t, d)는 이전 절에서 보았던 단어 빈도입니다. *idf(t, d)*는 역문서 빈도로 다음과 같이 계산합니다:\n",
    "\n",
    "$$\\text{idf}(t,d) = \\text{log}\\frac{n_d}{1+\\text{df}(d, t)},$$\n",
    "\n",
    "여기에서 $n_d$는 전체 문서 개수이고 *df(d, t)*는 단어 t가 포함된 문서 d의 개수입니다. 분모에 상수 1을 추가하는 것은 선택 사항입니다. 훈련 샘플에 한 번도 등장하지 않는 단어가 있는 경우 분모가 0이 되지 않게 만듭니다. log는 문서 빈도 *df(d, t)*가 낮을 때 역문서 빈도 값이 너무 커지지 않도록 만듭니다.\n",
    "\n",
    "사이킷런 라이브러리에는 `CountVectorizer` 클래스에서 만든 단어 빈도를 입력받아 tf-idf로 변환하는 `TfidfTransformer` 클래스가 구현되어 있습니다:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.   0.43 0.   0.56 0.56 0.   0.43 0.   0.  ]\n",
      " [0.   0.43 0.   0.   0.   0.56 0.43 0.   0.56]\n",
      " [0.5  0.45 0.5  0.19 0.19 0.19 0.3  0.25 0.19]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "\n",
    "tfidf = TfidfTransformer(use_idf=True, \n",
    "                         norm='l2', \n",
    "                         smooth_idf=True)\n",
    "print(tfidf.fit_transform(count.fit_transform(docs))\n",
    "      .toarray())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이전 절에서 보았듯이 세 번째 문서에서 단어 ‘is’가 가장 많이 나타났기 때문에 단어 빈도가 가장 컸습니다. 동일한 특성 벡터를 tf-idf로 변환하면 단어 ‘is’는 비교적 작은 tf-idf를 가집니다(0.45). 이 단어는 첫 번째와 두 번째 문서에도 나타나므로 판별에 유용한 정보를 가지고 있지 않을 것입니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "수동으로 특성 벡터에 있는 각 단어의 tf-idf를 계산해 보면 `TfidfTransformer`가 앞서 정의한 표준 tf-idf 공식과 조금 다르게 계산한다는 것을 알 수 있습니다. 사이킷런에 구현된 역문서 빈도 공식은 다음과 같습니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\text{idf} (t,d) = log\\frac{1 + n_d}{1 + \\text{df}(d, t)}$$\n",
    "\n",
    "비슷하게 사이킷런에서 계산하는 tf-idf는 앞서 정의한 공식과 조금 다릅니다:\n",
    "\n",
    "$$\\text{tf-idf}(t,d) = \\text{tf}(t,d) \\times (\\text{idf}(t,d)+1)$$\n",
    "\n",
    "일반적으로 tf-idf를 계산하기 전에 단어 빈도(tf)를 정규화하지만 `TfidfTransformer` 클래스는 tf-idf를 직접 정규화합니다. 사이킷런의 `TfidfTransformer`는 기본적으로 L2 정규화를 적용합니다(norm=’l2’). 정규화되지 않은 특성 벡터 v를 L2-노름으로 나누면 길이가 1인 벡터가 반환됩니다:\n",
    "\n",
    "$$v_{\\text{norm}} = \\frac{v}{||v||_2} = \\frac{v}{\\sqrt{v_{1}^{2} + v_{2}^{2} + \\dots + v_{n}^{2}}} = \\frac{v}{\\big (\\sum_{i=1}^{n} v_{i}^{2}\\big)^\\frac{1}{2}}$$\n",
    "\n",
    "TfidfTransformer의 작동 원리를 이해하기 위해 세 번째 문서에 있는 단어 ‘is'의 tf-idf를 예로 들어 계산해 보죠.\n",
    "\n",
    "세 번째 문서에서 단어 ‘is’의 단어 빈도는 3입니다(tf=3). 이 단어는 세 개 문서에 모두 나타나기 때문에 문서 빈도가 3입니다(df=3). 따라서 역문서 빈도는 다음과 같이 계산됩니다:\n",
    "\n",
    "$$\\text{idf}(\"is\", d3) = log \\frac{1+3}{1+3} = 0$$\n",
    "\n",
    "이제 tf-idf를 계산하기 위해 역문서 빈도에 1을 더하고 단어 빈도를 곱합니다:\n",
    "\n",
    "$$\\text{tf-idf}(\"is\",d3)= 3 \\times (0+1) = 3$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf-idf of term \"is\" = 3.00\n"
     ]
    }
   ],
   "source": [
    "tf_is = 3\n",
    "n_docs = 3\n",
    "idf_is = np.log((n_docs+1) / (3+1))\n",
    "tfidf_is = tf_is * (idf_is + 1)\n",
    "print('tf-idf of term \"is\" = %.2f' % tfidf_is)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "세 번째 문서에 있는 모든 단어에 대해 이런 계산을 반복하면 tf-idf 벡터 [3.39, 3.0, 3.39, 1.29, 1.29, 1.29, 2.0, 1.69, 1.29]를 얻습니다. 이 특성 벡터의 값은 앞서 사용했던 TfidfTransformer에서 얻은 값과 다릅니다. tf-idf 계산에서 빠트린 마지막 단계는 다음과 같은 L2-정규화입니다::"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\text{tfi-df}_{norm} = \\frac{[3.39, 3.0, 3.39, 1.29, 1.29, 1.29, 2.0 , 1.69, 1.29]}{\\sqrt{[3.39^2, 3.0^2, 3.39^2, 1.29^2, 1.29^2, 1.29^2, 2.0^2 , 1.69^2, 1.29^2]}}$$\n",
    "\n",
    "$$=[0.5, 0.45, 0.5, 0.19, 0.19, 0.19, 0.3, 0.25, 0.19]$$\n",
    "\n",
    "$$\\Rightarrow \\text{tfi-df}_{norm}(\"is\", d3) = 0.45$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "결과에서 보듯이 사이킷런의 `TfidfTransformer`에서 반환된 결과와 같아졌습니다. tf-idf 계산 방법을 이해했으므로 다음 절로 넘어가 이 개념을 영화 리뷰 데이터셋에 적용해 보죠."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3.39, 3.  , 3.39, 1.29, 1.29, 1.29, 2.  , 1.69, 1.29])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf = TfidfTransformer(use_idf=True, norm=None, smooth_idf=True)\n",
    "raw_tfidf = tfidf.fit_transform(count.fit_transform(docs)).toarray()[-1]\n",
    "raw_tfidf "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.5 , 0.45, 0.5 , 0.19, 0.19, 0.19, 0.3 , 0.25, 0.19])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l2_tfidf = raw_tfidf / np.sqrt(np.sum(raw_tfidf**2))\n",
    "l2_tfidf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 텍스트 데이터 정제"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'is seven.<br /><br />Title (Brazil): Not Available'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[0, 'review'][-50:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "def preprocessor(text):\n",
    "    text = re.sub('<[^>]*>', '', text)\n",
    "    emoticons = re.findall('(?::|;|=)(?:-)?(?:\\)|\\(|D|P)',\n",
    "                           text)\n",
    "    text = (re.sub('[\\W]+', ' ', text.lower()) +\n",
    "            ' '.join(emoticons).replace('-', ''))\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'is seven title brazil not available'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor(df.loc[0, 'review'][-50:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'this is a test :) :( :)'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor(\"</a>This :) is :( a test :-)!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['review'] = df['review'].apply(preprocessor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        in 1974 the teenager martha moxley maggie grac...\n",
       "1        ok so i really like kris kristofferson and his...\n",
       "2         spoiler do not read this if you think about w...\n",
       "3        hi for all the people who have seen this wonde...\n",
       "4        i recently bought the dvd forgetting just how ...\n",
       "                               ...                        \n",
       "49995    ok lets start with the best the building altho...\n",
       "49996    the british heritage film industry is out of c...\n",
       "49997    i don t even know where to begin on this one i...\n",
       "49998    richard tyler is a little boy who is scared of...\n",
       "49999    i waited long to watch this movie also because...\n",
       "Name: review, Length: 50000, dtype: object"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['review'].map(preprocessor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 문서를 토큰으로 나누기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem.porter import PorterStemmer\n",
    "\n",
    "porter = PorterStemmer()\n",
    "\n",
    "def tokenizer(text):\n",
    "    return text.split()\n",
    "\n",
    "\n",
    "def tokenizer_porter(text):\n",
    "    return [porter.stem(word) for word in text.split()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['runners', 'like', 'running', 'and', 'thus', 'they', 'run']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer('runners like running and thus they run')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['runner', 'like', 'run', 'and', 'thu', 'they', 'run']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer_porter('runners like running and thus they run')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to /home/haesun/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "\n",
    "nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['runner', 'like', 'run', 'run', 'lot']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import stopwords\n",
    "\n",
    "stop = stopwords.words('english')\n",
    "[w for w in tokenizer_porter('a runner likes running and runs a lot')[-10:]\n",
    "if w not in stop]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 문서 분류를 위한 로지스틱 회귀 모델 훈련하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = df.loc[:25000, 'review'].values\n",
    "y_train = df.loc[:25000, 'sentiment'].values\n",
    "X_test = df.loc[25000:, 'review'].values\n",
    "y_test = df.loc[25000:, 'sentiment'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "tfidf = TfidfVectorizer(strip_accents=None,\n",
    "                        lowercase=False,\n",
    "                        preprocessor=None)\n",
    "\n",
    "param_grid = [{'vect__ngram_range': [(1, 1)],\n",
    "               'vect__stop_words': [stop, None],\n",
    "               'vect__tokenizer': [tokenizer, tokenizer_porter],\n",
    "               'clf__penalty': ['l1', 'l2'],\n",
    "               'clf__C': [1.0, 10.0, 100.0]},\n",
    "              {'vect__ngram_range': [(1, 1)],\n",
    "               'vect__stop_words': [stop, None],\n",
    "               'vect__tokenizer': [tokenizer, tokenizer_porter],\n",
    "               'vect__use_idf':[False],\n",
    "               'vect__norm':[None],\n",
    "               'clf__penalty': ['l1', 'l2'],\n",
    "               'clf__C': [1.0, 10.0, 100.0]},\n",
    "              ]\n",
    "\n",
    "lr_tfidf = Pipeline([('vect', tfidf),\n",
    "                     ('clf', LogisticRegression(solver='liblinear', random_state=0))])\n",
    "\n",
    "gs_lr_tfidf = GridSearchCV(lr_tfidf, param_grid,\n",
    "                           scoring='accuracy',\n",
    "                           cv=5,\n",
    "                           verbose=1,\n",
    "                           n_jobs=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**`n_jobs` 매개변수에 대하여**\n",
    "\n",
    "앞의 코드 예제에서 컴퓨터에 있는 모든 CPU 코어를 사용해 그리드 서치의 속도를 높이려면 (`n_jobs=1` 대신) `n_jobs=-1`로 지정하는 것이 좋습니다. 일부 시스템에서는 멀티프로세싱을 위해 `n_jobs=-1`로 지정할 때 `tokenizer` 와 `tokenizer_porter` 함수의 직렬화에 문제가 발생할 수 있습니다. 이런 경우 `[tokenizer, tokenizer_porter]`를 `[str.split]`로 바꾸어 문제를 해결할 수 있습니다. 다만 `str.split`로 바꾸면 어간 추출을 하지 못합니다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**코드 실행 시간에 대하여**\n",
    "\n",
    "다음 코드 셀을 실행하면 시스템에 따라 **30~60분 정도 걸릴 수 있습니다**. 매개변수 그리드에서 정의한 대로 2*2*2*3*5 + 2*2*2*3*5 = 240개의 모델을 훈련하기 때문입니다.\n",
    "\n",
    "**코랩을 사용할 경우에도 CPU 코어가 많지 않기 때문에 실행 시간이 오래 걸릴 수 있습니다.**\n",
    "\n",
    "너무 오래 기다리기 어렵다면 데이터셋의 훈련 샘플의 수를 다음처럼 줄일 수 있습니다:\n",
    "\n",
    "    X_train = df.loc[:2500, 'review'].values\n",
    "    y_train = df.loc[:2500, 'sentiment'].values\n",
    "    \n",
    "훈련 세트 크기를 줄이는 것은 모델 성능을 감소시킵니다. 그리드에 지정한 매개변수를 삭제하면 훈련한 모델 수를 줄일 수 있습니다. 예를 들면 다음과 같습니다:\n",
    "\n",
    "    param_grid = [{'vect__ngram_range': [(1, 1)],\n",
    "                   'vect__stop_words': [stop, None],\n",
    "                   'vect__tokenizer': [tokenizer],\n",
    "                   'clf__penalty': ['l1', 'l2'],\n",
    "                   'clf__C': [1.0, 10.0]},\n",
    "                  ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 48 candidates, totalling 240 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "/home/haesun/anaconda3/envs/python-ml/lib/python3.7/site-packages/sklearn/feature_extraction/text.py:386: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['abov', 'ani', 'becaus', 'befor', 'doe', 'dure', 'ha', 'hi', \"it'\", 'onc', 'onli', 'ourselv', \"she'\", \"should'v\", 'themselv', 'thi', 'veri', 'wa', 'whi', \"you'r\", \"you'v\", 'yourselv'] not in stop_words.\n",
      "  'stop_words.' % sorted(inconsistent))\n",
      "[Parallel(n_jobs=1)]: Done 240 out of 240 | elapsed: 145.7min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5,\n",
       "             estimator=Pipeline(steps=[('vect',\n",
       "                                        TfidfVectorizer(lowercase=False)),\n",
       "                                       ('clf',\n",
       "                                        LogisticRegression(random_state=0,\n",
       "                                                           solver='liblinear'))]),\n",
       "             n_jobs=1,\n",
       "             param_grid=[{'clf__C': [1.0, 10.0, 100.0],\n",
       "                          'clf__penalty': ['l1', 'l2'],\n",
       "                          'vect__ngram_range': [(1, 1)],\n",
       "                          'vect__stop_words': [['i', 'me', 'my', 'myself', 'we',\n",
       "                                                'our', 'ours', 'ourselves',\n",
       "                                                'you', \"you're\", \"you've\"...\n",
       "                                                'our', 'ours', 'ourselves',\n",
       "                                                'you', \"you're\", \"you've\",\n",
       "                                                \"you'll\", \"you'd\", 'your',\n",
       "                                                'yours', 'yourself',\n",
       "                                                'yourselves', 'he', 'him',\n",
       "                                                'his', 'himself', 'she',\n",
       "                                                \"she's\", 'her', 'hers',\n",
       "                                                'herself', 'it', \"it's\", 'its',\n",
       "                                                'itself', ...],\n",
       "                                               None],\n",
       "                          'vect__tokenizer': [<function tokenizer at 0x7fc16c4e9730>,\n",
       "                                              <function tokenizer_porter at 0x7fc16c4e97b8>],\n",
       "                          'vect__use_idf': [False]}],\n",
       "             scoring='accuracy', verbose=1)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs_lr_tfidf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "최적의 매개변수 조합: {'clf__C': 10.0, 'clf__penalty': 'l2', 'vect__ngram_range': (1, 1), 'vect__stop_words': None, 'vect__tokenizer': <function tokenizer at 0x7fc16c4e9730>} \n",
      "CV 정확도: 0.897\n"
     ]
    }
   ],
   "source": [
    "print('최적의 매개변수 조합: %s ' % gs_lr_tfidf.best_params_)\n",
    "print('CV 정확도: %.3f' % gs_lr_tfidf.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "테스트 정확도: 0.899\n"
     ]
    }
   ],
   "source": [
    "clf = gs_lr_tfidf.best_estimator_\n",
    "print('테스트 정확도: %.3f' % clf.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# 대용량 데이터 처리-온라인 알고리즘과 외부 메모리 학습"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 이 셀의 코드는 책에 포함되어 있지 않습니다. This cell is not contained in the book but\n",
    "# 이전 코드를 실행하지 않고 바로 시작할 수 있도록 편의를 위해 추가했습니다.\n",
    "\n",
    "import os\n",
    "import gzip\n",
    "\n",
    "\n",
    "if not os.path.isfile('movie_data.csv'):\n",
    "    if not os.path.isfile('movie_data.csv.gz'):\n",
    "        print('Please place a copy of the movie_data.csv.gz'\n",
    "              'in this directory. You can obtain it by'\n",
    "              'a) executing the code in the beginning of this'\n",
    "              'notebook or b) by downloading it from GitHub:'\n",
    "              'https://github.com/rasbt/python-machine-learning-'\n",
    "              'book-2nd-edition/blob/master/code/ch08/movie_data.csv.gz')\n",
    "    else:\n",
    "        in_f = gzip.open('movie_data.csv.gz', 'rb')\n",
    "        out_f = open('movie_data.csv', 'wb')\n",
    "        out_f.write(in_f.read())\n",
    "        in_f.close()\n",
    "        out_f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import re\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "\n",
    "# `stop` 객체를 앞에서 정의했지만 이전 코드를 실행하지 않고\n",
    "# 편의상 여기에서부터 코드를 실행하기 위해 다시 만듭니다.\n",
    "stop = stopwords.words('english')\n",
    "\n",
    "\n",
    "def tokenizer(text):\n",
    "    text = re.sub('<[^>]*>', '', text)\n",
    "    emoticons = re.findall('(?::|;|=)(?:-)?(?:\\)|\\(|D|P)', text.lower())\n",
    "    text = re.sub('[\\W]+', ' ', text.lower()) +\\\n",
    "        ' '.join(emoticons).replace('-', '')\n",
    "    tokenized = [w for w in text.split() if w not in stop]\n",
    "    return tokenized\n",
    "\n",
    "\n",
    "def stream_docs(path):\n",
    "    with open(path, 'r', encoding='utf-8') as csv:\n",
    "        next(csv)  # 헤더 넘기기\n",
    "        for line in csv:\n",
    "            text, label = line[:-3], int(line[-2])\n",
    "            yield text, label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('\"In 1974, the teenager Martha Moxley (Maggie Grace) moves to the high-class area of Belle Haven, Greenwich, Connecticut. On the Mischief Night, eve of Halloween, she was murdered in the backyard of her house and her murder remained unsolved. Twenty-two years later, the writer Mark Fuhrman (Christopher Meloni), who is a former LA detective that has fallen in disgrace for perjury in O.J. Simpson trial and moved to Idaho, decides to investigate the case with his partner Stephen Weeks (Andrew Mitchell) with the purpose of writing a book. The locals squirm and do not welcome them, but with the support of the retired detective Steve Carroll (Robert Forster) that was in charge of the investigation in the 70\\'s, they discover the criminal and a net of power and money to cover the murder.<br /><br />\"\"Murder in Greenwich\"\" is a good TV movie, with the true story of a murder of a fifteen years old girl that was committed by a wealthy teenager whose mother was a Kennedy. The powerful and rich family used their influence to cover the murder for more than twenty years. However, a snoopy detective and convicted perjurer in disgrace was able to disclose how the hideous crime was committed. The screenplay shows the investigation of Mark and the last days of Martha in parallel, but there is a lack of the emotion in the dramatization. My vote is seven.<br /><br />Title (Brazil): Not Available\"',\n",
       " 1)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(stream_docs(path='movie_data.csv'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_minibatch(doc_stream, size):\n",
    "    docs, y = [], []\n",
    "    try:\n",
    "        for _ in range(size):\n",
    "            text, label = next(doc_stream)\n",
    "            docs.append(text)\n",
    "            y.append(label)\n",
    "    except StopIteration:\n",
    "        pass\n",
    "    return docs, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import HashingVectorizer\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "\n",
    "\n",
    "vect = HashingVectorizer(decode_error='ignore', \n",
    "                         n_features=2**21,\n",
    "                         preprocessor=None, \n",
    "                         tokenizer=tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "from distutils.version import LooseVersion as Version\n",
    "from sklearn import __version__ as sklearn_version\n",
    "\n",
    "clf = SGDClassifier(loss='log', random_state=1, max_iter=1)\n",
    "\n",
    "doc_stream = stream_docs(path='movie_data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "0% [##############################] 100% | ETA: 00:00:00\n",
      "Total time elapsed: 00:00:21\n"
     ]
    }
   ],
   "source": [
    "import pyprind\n",
    "pbar = pyprind.ProgBar(45)\n",
    "\n",
    "classes = np.array([0, 1])\n",
    "for _ in range(45):\n",
    "    X_train, y_train = get_minibatch(doc_stream, size=1000)\n",
    "    if not X_train:\n",
    "        break\n",
    "    X_train = vect.transform(X_train)\n",
    "    clf.partial_fit(X_train, y_train, classes=classes)\n",
    "    pbar.update()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "정확도: 0.868\n"
     ]
    }
   ],
   "source": [
    "X_test, y_test = get_minibatch(doc_stream, size=5000)\n",
    "X_test = vect.transform(X_test)\n",
    "print('정확도: %.3f' % clf.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = clf.partial_fit(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 토픽 모델링"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 사이킷런의 LDA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>In 1974, the teenager Martha Moxley (Maggie Gr...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>OK... so... I really like Kris Kristofferson a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>***SPOILER*** Do not read this, if you think a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              review  sentiment\n",
       "0  In 1974, the teenager Martha Moxley (Maggie Gr...          1\n",
       "1  OK... so... I really like Kris Kristofferson a...          0\n",
       "2  ***SPOILER*** Do not read this, if you think a...          0"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('movie_data.csv', encoding='utf-8')\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "count = CountVectorizer(stop_words='english',\n",
    "                        max_df=.1,\n",
    "                        max_features=5000)\n",
    "X = count.fit_transform(df['review'].values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import LatentDirichletAllocation\n",
    "\n",
    "lda = LatentDirichletAllocation(n_components=10,\n",
    "                                random_state=123,\n",
    "                                learning_method='batch')\n",
    "X_topics = lda.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10, 5000)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda.components_.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "토픽 1:\n",
      "worst minutes awful comedy money\n",
      "토픽 2:\n",
      "sex women woman men black\n",
      "토픽 3:\n",
      "comedy musical music song dance\n",
      "토픽 4:\n",
      "family father mother wife son\n",
      "토픽 5:\n",
      "war book american documentary japanese\n",
      "토픽 6:\n",
      "episode guy series house girl\n",
      "토픽 7:\n",
      "role performance actor john book\n",
      "토픽 8:\n",
      "war music beautiful cinema history\n",
      "토픽 9:\n",
      "horror budget gore killer effects\n",
      "토픽 10:\n",
      "action original series animation disney\n"
     ]
    }
   ],
   "source": [
    "n_top_words = 5\n",
    "feature_names = count.get_feature_names()\n",
    "\n",
    "for topic_idx, topic in enumerate(lda.components_):\n",
    "    print(\"토픽 %d:\" % (topic_idx + 1))\n",
    "    print(\" \".join([feature_names[i]\n",
    "                    for i in topic.argsort()\\\n",
    "                        [:-n_top_words - 1:-1]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "각 토픽에서 가장 중요한 단어 다섯 개를 기반으로 LDA가 다음 토픽을 구별했다고 추측할 수 있습니다.\n",
    "\n",
    "1.\t대체적으로 형편없는 영화(실제 토픽 카테고리가 되지 못함)\n",
    "2.\t가족 영화\n",
    "3.\t전쟁 영화\n",
    "4.\t예술 영화\n",
    "5.\t범죄 영화\n",
    "6.\t공포 영화\n",
    "7.\t코미디 영화\n",
    "8.\tTV 쇼와 관련된 영화\n",
    "9.\t소설을 원작으로 한 영화\n",
    "10.\t액션 영화"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "카테고리가 잘 선택됐는지 확인하기 위해 공포 영화 카테고리에서 3개 영화의 리뷰를 출력해 보죠(공포 영화는 카테고리 6이므로 인덱스는 5입니다):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "공포 영화 #1:\n",
      "This is one of the funniest movies I've ever saw. A 14-year old boy Jason Shepherd wrote an English paper called \"Big Fat Liar\". When his skateboard was taken, he had to use his sister's bike to get to the college on time and he hit a limo. When he went into the limo, he met a famous producer from H ...\n",
      "\n",
      "공포 영화 #2:\n",
      "Where to start? Some guy has some Indian pot that he's cleaning, and suddenly Skeletor attacks. He hits a woman in the neck with an axe, she falls down, but then gets up and is apparently uninjured. She runs into the woods, and it turns out there's the basement of a shopping center out there in the  ...\n",
      "\n",
      "공포 영화 #3:\n",
      "***SPOILERS*** ***SPOILERS*** Some bunch of Afrikkaner-Hillbilly types are out in the desert looking for Diamonds when they find a hard mound in the middle of a sandy desert area. Spoilers: The dumbest one starts hitting the mound with a pick, and cracks it open. Then he looks into the hole and stic ...\n"
     ]
    }
   ],
   "source": [
    "horror = X_topics[:, 5].argsort()[::-1]\n",
    "\n",
    "for iter_idx, movie_idx in enumerate(horror[:3]):\n",
    "    print('\\n공포 영화 #%d:' % (iter_idx + 1))\n",
    "    print(df['review'][movie_idx][:300], '...')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "앞 코드에서 공포 영화 카테고리 중 최상위 3개의 리뷰에서 300자씩 출력했습니다. 정확히 어떤 영화에 속한 리뷰인지는 모르지만 공포 영화의 리뷰임을 알 수 있습니다."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}