02.01.01.word2vec_with_keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word2Vec implementation with keras  \n",
    "\n",
    "Word2Vec(Mikolov et al. 2013)은 한 문장 안에 함께 출현한 두 단어들 사이에는 의미적 유사도가 있다라는 아이디어를 통해 word의 semantic vector representation을 구할 수 있다는 것을 보임으로써 크게 주목을 받았습니다.  \n",
    "\n",
    "본 실습에서는 word2vec을 실제로 구현하는 과정을 통해, 자체적인 의미(meaning) 정보를 갖지 않은 심볼을 embedding하는 개념을 익히고자 합니다.\n",
    "\n",
    "(참고)  본 실습은 아래 튜토리얼을 참고하여 작성되었습니다.\n",
    "https://byeongkijeong.github.io/Word2vec-from-scratch-using-keras/\n",
    "https://adventuresinmachinelearning.com/word2vec-keras-tutorial/\n",
    "https://www.d2l.ai/chapter_natural-language-processing/word2vec-gluon.html\n",
    "https://towardsdatascience.com/art-of-vector-representation-of-words-5e85c59fee5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from collections import Counter\n",
    "from time import time\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from keras.layers import Dense, Dot, Embedding, Input, Reshape\n",
    "from keras.models import Model\n",
    "from keras.preprocessing.sequence import skipgrams\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "np.random.seed(777)\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"] = '-1'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## step 1. define dataset  \n",
    "\n",
    "본 실습은 abcnews-date-text.csv 라는 52MB짜리 corpus를 기반으로 진행합니다.  \n",
    "52MB는 의미있는 word vector를 추출하기에는 매우 작은 사이즈입니다. 이 실습의 결과로 얻어진 word vector를 통해 NLP 성능을 기대하기는 어렵습니다. 하지만 구현 과정을 빠르게 살펴보기에는 용이합니다.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    aba decides against community broadcasting lic...\n",
       "1       act fire witnesses must be aware of defamation\n",
       "2       a g calls for infrastructure protection summit\n",
       "3             air nz staff in aust strike for pay rise\n",
       "4        air nz strike to affect australian travellers\n",
       "5                    ambitious olsson wins triple jump\n",
       "6           antic delighted with record breaking barca\n",
       "7    aussie qualifier stosur wastes four memphis match\n",
       "8         aust addresses un security council over iraq\n",
       "9           australia is locked into war timetable opp\n",
       "Name: headline_text, dtype: object"
      ]
     },
     "execution_count": 180,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus = pd.read_csv(\"datasets/corpus/abcnews-date-text.csv\").iloc[:,1] \n",
    "corpus.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.strings.StringMethods"
      ]
     },
     "execution_count": 181,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(corpus)\n",
    "type(corpus.str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    aba decides against community broadcasting lic...\n",
       "1       act fire witnesses must be aware of defamation\n",
       "2       a g calls for infrastructure protection summit\n",
       "3             air nz staff in aust strike for pay rise\n",
       "4        air nz strike to affect australian travellers\n",
       "5                    ambitious olsson wins triple jump\n",
       "6           antic delighted with record breaking barca\n",
       "7    aussie qualifier stosur wastes four memphis match\n",
       "8         aust addresses un security council over iraq\n",
       "9           australia is locked into war timetable opp\n",
       "Name: headline_text, dtype: object"
      ]
     },
     "execution_count": 182,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 소문자로 변경\n",
    "corpus = corpus.str.lower()    \n",
    "# 숫자/알파벳/공백 을 제외하고 전부 제거 (with pandas.core.strings.StringMethods)\n",
    "corpus = corpus.str.replace('[^a-z0-9]+', ' ', regex=True)\n",
    "corpus.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1082168"
      ]
     },
     "execution_count": 183,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_list = corpus.values.tolist()\n",
    "len(corpus_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['aba decides against community broadcasting licence',\n",
       " 'act fire witnesses must be aware of defamation',\n",
       " 'a g calls for infrastructure protection summit',\n",
       " 'air nz staff in aust strike for pay rise',\n",
       " 'air nz strike to affect australian travellers']"
      ]
     },
     "execution_count": 184,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_list[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build Vocabulary Dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['aba',\n",
       " 'decides',\n",
       " 'against',\n",
       " 'community',\n",
       " 'broadcasting',\n",
       " 'licence',\n",
       " 'act',\n",
       " 'fire',\n",
       " 'witnesses',\n",
       " 'must',\n",
       " 'be',\n",
       " 'aware',\n",
       " 'of',\n",
       " 'defamation',\n",
       " 'a',\n",
       " 'g',\n",
       " 'calls',\n",
       " 'for',\n",
       " 'infrastructure',\n",
       " 'protection',\n",
       " 'summit',\n",
       " 'air',\n",
       " 'nz',\n",
       " 'staff',\n",
       " 'in',\n",
       " 'aust',\n",
       " 'strike',\n",
       " 'for',\n",
       " 'pay',\n",
       " 'rise',\n",
       " 'air',\n",
       " 'nz',\n",
       " 'strike',\n",
       " 'to',\n",
       " 'affect',\n",
       " 'australian',\n",
       " 'travellers',\n",
       " 'ambitious',\n",
       " 'olsson',\n",
       " 'wins',\n",
       " 'triple',\n",
       " 'jump',\n",
       " 'antic',\n",
       " 'delighted',\n",
       " 'with',\n",
       " 'record',\n",
       " 'breaking',\n",
       " 'barca',\n",
       " 'aussie',\n",
       " 'qualifier',\n",
       " 'stosur',\n",
       " 'wastes',\n",
       " 'four',\n",
       " 'memphis',\n",
       " 'match',\n",
       " 'aust',\n",
       " 'addresses',\n",
       " 'un',\n",
       " 'security',\n",
       " 'council',\n",
       " 'over',\n",
       " 'iraq',\n",
       " 'australia',\n",
       " 'is',\n",
       " 'locked',\n",
       " 'into',\n",
       " 'war',\n",
       " 'timetable',\n",
       " 'opp',\n",
       " 'australia',\n",
       " 'to',\n",
       " 'contribute',\n",
       " '10',\n",
       " 'million',\n",
       " 'in',\n",
       " 'aid',\n",
       " 'to',\n",
       " 'iraq',\n",
       " 'barca',\n",
       " 'take',\n",
       " 'record',\n",
       " 'as',\n",
       " 'robson',\n",
       " 'celebrates',\n",
       " 'birthday',\n",
       " 'in',\n",
       " 'bathhouse',\n",
       " 'plans',\n",
       " 'move',\n",
       " 'ahead',\n",
       " 'big',\n",
       " 'hopes',\n",
       " 'for',\n",
       " 'launceston',\n",
       " 'cycling',\n",
       " 'championship',\n",
       " 'big',\n",
       " 'plan',\n",
       " 'to',\n",
       " 'boost']"
      ]
     },
     "execution_count": 185,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# corpus에 있는 모든 line에서 word를 추출해서 하나의 list에 모두 집어넣습니다.\n",
    "words = np.concatenate(np.core.defchararray.split(corpus_list)).tolist()\n",
    "words[0:100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a',\n",
       " 'about',\n",
       " 'above',\n",
       " 'after',\n",
       " 'again',\n",
       " 'against',\n",
       " 'ain',\n",
       " 'all',\n",
       " 'am',\n",
       " 'an',\n",
       " 'and',\n",
       " 'any',\n",
       " 'are',\n",
       " 'aren',\n",
       " \"aren't\",\n",
       " 'as',\n",
       " 'at',\n",
       " 'be',\n",
       " 'because',\n",
       " 'been',\n",
       " 'before',\n",
       " 'being',\n",
       " 'below',\n",
       " 'between',\n",
       " 'both',\n",
       " 'but',\n",
       " 'by',\n",
       " 'can',\n",
       " 'couldn',\n",
       " \"couldn't\",\n",
       " 'd',\n",
       " 'did',\n",
       " 'didn',\n",
       " \"didn't\",\n",
       " 'do',\n",
       " 'does',\n",
       " 'doesn',\n",
       " \"doesn't\",\n",
       " 'doing',\n",
       " 'don',\n",
       " \"don't\",\n",
       " 'down',\n",
       " 'during',\n",
       " 'each',\n",
       " 'few',\n",
       " 'for',\n",
       " 'from',\n",
       " 'further',\n",
       " 'had',\n",
       " 'hadn',\n",
       " \"hadn't\",\n",
       " 'has',\n",
       " 'hasn',\n",
       " \"hasn't\",\n",
       " 'have',\n",
       " 'haven',\n",
       " \"haven't\",\n",
       " 'having',\n",
       " 'he',\n",
       " 'her',\n",
       " 'here',\n",
       " 'hers',\n",
       " 'herself',\n",
       " 'him',\n",
       " 'himself',\n",
       " 'his',\n",
       " 'how',\n",
       " 'i',\n",
       " 'if',\n",
       " 'in',\n",
       " 'into',\n",
       " 'is',\n",
       " 'isn',\n",
       " \"isn't\",\n",
       " 'it',\n",
       " \"it's\",\n",
       " 'its',\n",
       " 'itself',\n",
       " 'just',\n",
       " 'll',\n",
       " 'm',\n",
       " 'ma',\n",
       " 'me',\n",
       " 'mightn',\n",
       " \"mightn't\",\n",
       " 'more',\n",
       " 'most',\n",
       " 'mustn',\n",
       " \"mustn't\",\n",
       " 'my',\n",
       " 'myself',\n",
       " 'needn',\n",
       " \"needn't\",\n",
       " 'no',\n",
       " 'nor',\n",
       " 'not',\n",
       " 'now',\n",
       " 'o',\n",
       " 'of',\n",
       " 'off',\n",
       " 'on',\n",
       " 'once',\n",
       " 'only',\n",
       " 'or',\n",
       " 'other',\n",
       " 'our',\n",
       " 'ours',\n",
       " 'ourselves',\n",
       " 'out',\n",
       " 'over',\n",
       " 'own',\n",
       " 're',\n",
       " 's',\n",
       " 'same',\n",
       " 'shan',\n",
       " \"shan't\",\n",
       " 'she',\n",
       " \"she's\",\n",
       " 'should',\n",
       " \"should've\",\n",
       " 'shouldn',\n",
       " \"shouldn't\",\n",
       " 'so',\n",
       " 'some',\n",
       " 'such',\n",
       " 't',\n",
       " 'than',\n",
       " 'that',\n",
       " \"that'll\",\n",
       " 'the',\n",
       " 'their',\n",
       " 'theirs',\n",
       " 'them',\n",
       " 'themselves',\n",
       " 'then',\n",
       " 'there',\n",
       " 'these',\n",
       " 'they',\n",
       " 'this',\n",
       " 'those',\n",
       " 'through',\n",
       " 'to',\n",
       " 'too',\n",
       " 'under',\n",
       " 'until',\n",
       " 'up',\n",
       " 've',\n",
       " 'very',\n",
       " 'was',\n",
       " 'wasn',\n",
       " \"wasn't\",\n",
       " 'we',\n",
       " 'were',\n",
       " 'weren',\n",
       " \"weren't\",\n",
       " 'what',\n",
       " 'when',\n",
       " 'where',\n",
       " 'which',\n",
       " 'while',\n",
       " 'who',\n",
       " 'whom',\n",
       " 'why',\n",
       " 'will',\n",
       " 'with',\n",
       " 'won',\n",
       " \"won't\",\n",
       " 'wouldn',\n",
       " \"wouldn't\",\n",
       " 'y',\n",
       " 'you',\n",
       " \"you'd\",\n",
       " \"you'll\",\n",
       " \"you're\",\n",
       " \"you've\",\n",
       " 'your',\n",
       " 'yours',\n",
       " 'yourself',\n",
       " 'yourselves'}"
      ]
     },
     "execution_count": 186,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 너무 자주 출현하는 word들은 word vector 구성에 방해가 됩니다. 이런 불용어들을 제거해 줍니다. \n",
    "stopWords = set(stopwords.words('english'))\n",
    "stopWords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6907609"
      ]
     },
     "execution_count": 187,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "96035"
      ]
     },
     "execution_count": 188,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counter = Counter(words)\n",
    "len(counter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('to', 210213),\n",
       " ('in', 132680),\n",
       " ('for', 128091),\n",
       " ('of', 78259),\n",
       " ('on', 71547),\n",
       " ('over', 49557),\n",
       " ('the', 47359),\n",
       " ('police', 35420),\n",
       " ('at', 30776),\n",
       " ('with', 28817),\n",
       " ('after', 28804),\n",
       " ('new', 28470),\n",
       " ('man', 27627),\n",
       " ('a', 23804),\n",
       " ('and', 21669),\n",
       " ('up', 20688),\n",
       " ('as', 19890),\n",
       " ('says', 19340),\n",
       " ('from', 18493),\n",
       " ('us', 17137),\n",
       " ('by', 17081),\n",
       " ('govt', 16915),\n",
       " ('out', 16843),\n",
       " ('council', 16225),\n",
       " ('court', 16017),\n",
       " ('be', 15334),\n",
       " ('more', 15000),\n",
       " ('interview', 14868),\n",
       " ('fire', 13687),\n",
       " ('not', 13368),\n",
       " ('nsw', 12610),\n",
       " ('plan', 12197),\n",
       " ('australia', 12137),\n",
       " ('water', 11772),\n",
       " ('qld', 11632),\n",
       " ('wa', 11192),\n",
       " ('crash', 11019),\n",
       " ('death', 10925),\n",
       " ('into', 10643),\n",
       " ('off', 10511),\n",
       " ('sydney', 10409),\n",
       " ('against', 10236),\n",
       " ('health', 10199),\n",
       " ('charged', 9968),\n",
       " ('back', 9931),\n",
       " ('australian', 9926),\n",
       " ('no', 9853),\n",
       " ('report', 9441),\n",
       " ('down', 9363),\n",
       " ('call', 9165),\n",
       " ('murder', 9007),\n",
       " ('an', 8909),\n",
       " ('sa', 8896),\n",
       " ('hospital', 8669),\n",
       " ('day', 8595),\n",
       " ('car', 8520),\n",
       " ('may', 8395),\n",
       " ('calls', 8212),\n",
       " ('coast', 8211),\n",
       " ('win', 8161),\n",
       " ('woman', 8109),\n",
       " ('about', 8085),\n",
       " ('two', 8007),\n",
       " ('killed', 7999),\n",
       " ('s', 7965),\n",
       " ('accused', 7955),\n",
       " ('world', 7877),\n",
       " ('urged', 7808),\n",
       " ('found', 7674),\n",
       " ('home', 7671),\n",
       " ('government', 7651),\n",
       " ('south', 7541),\n",
       " ('missing', 7462),\n",
       " ('will', 7400),\n",
       " ('rural', 7329),\n",
       " ('first', 7307),\n",
       " ('set', 7203),\n",
       " ('claims', 7147),\n",
       " ('cup', 7034),\n",
       " ('is', 7019),\n",
       " ('attack', 6992),\n",
       " ('under', 6980),\n",
       " ('minister', 6970),\n",
       " ('election', 6965),\n",
       " ('boost', 6958),\n",
       " ('school', 6954),\n",
       " ('wins', 6901),\n",
       " ('face', 6810),\n",
       " ('dies', 6759),\n",
       " ('funding', 6733),\n",
       " ('market', 6715),\n",
       " ('help', 6655),\n",
       " ('talks', 6643),\n",
       " ('group', 6588),\n",
       " ('china', 6506),\n",
       " ('child', 6501),\n",
       " ('north', 6460),\n",
       " ('mp', 6430),\n",
       " ('drug', 6398),\n",
       " ('one', 6383)]"
      ]
     },
     "execution_count": 189,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counter.most_common(100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 190,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'into' in words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [],
   "source": [
    "words = [word for word in words if word not in stopWords]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 192,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'into' in words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "95889"
      ]
     },
     "execution_count": 193,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counter = Counter(words)\n",
    "len(counter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('police', 35420),\n",
       " ('new', 28470),\n",
       " ('man', 27627),\n",
       " ('says', 19340),\n",
       " ('us', 17137),\n",
       " ('govt', 16915),\n",
       " ('council', 16225),\n",
       " ('court', 16017),\n",
       " ('interview', 14868),\n",
       " ('fire', 13687),\n",
       " ('nsw', 12610),\n",
       " ('plan', 12197),\n",
       " ('australia', 12137),\n",
       " ('water', 11772),\n",
       " ('qld', 11632),\n",
       " ('wa', 11192),\n",
       " ('crash', 11019),\n",
       " ('death', 10925),\n",
       " ('sydney', 10409),\n",
       " ('health', 10199),\n",
       " ('charged', 9968),\n",
       " ('back', 9931),\n",
       " ('australian', 9926),\n",
       " ('report', 9441),\n",
       " ('call', 9165),\n",
       " ('murder', 9007),\n",
       " ('sa', 8896),\n",
       " ('hospital', 8669),\n",
       " ('day', 8595),\n",
       " ('car', 8520),\n",
       " ('may', 8395),\n",
       " ('calls', 8212),\n",
       " ('coast', 8211),\n",
       " ('win', 8161),\n",
       " ('woman', 8109),\n",
       " ('two', 8007),\n",
       " ('killed', 7999),\n",
       " ('accused', 7955),\n",
       " ('world', 7877),\n",
       " ('urged', 7808),\n",
       " ('found', 7674),\n",
       " ('home', 7671),\n",
       " ('government', 7651),\n",
       " ('south', 7541),\n",
       " ('missing', 7462),\n",
       " ('rural', 7329),\n",
       " ('first', 7307),\n",
       " ('set', 7203),\n",
       " ('claims', 7147),\n",
       " ('cup', 7034),\n",
       " ('attack', 6992),\n",
       " ('minister', 6970),\n",
       " ('election', 6965),\n",
       " ('boost', 6958),\n",
       " ('school', 6954),\n",
       " ('wins', 6901),\n",
       " ('face', 6810),\n",
       " ('dies', 6759),\n",
       " ('funding', 6733),\n",
       " ('market', 6715),\n",
       " ('help', 6655),\n",
       " ('talks', 6643),\n",
       " ('group', 6588),\n",
       " ('china', 6506),\n",
       " ('child', 6501),\n",
       " ('north', 6460),\n",
       " ('mp', 6430),\n",
       " ('drug', 6398),\n",
       " ('one', 6383),\n",
       " ('nt', 6378),\n",
       " ('labor', 6364),\n",
       " ('farmers', 6361),\n",
       " ('trial', 6286),\n",
       " ('deal', 6233),\n",
       " ('power', 6227),\n",
       " ('melbourne', 6217),\n",
       " ('road', 6204),\n",
       " ('hit', 6175),\n",
       " ('get', 6175),\n",
       " ('indigenous', 6162),\n",
       " ('sex', 6157),\n",
       " ('fears', 6135),\n",
       " ('open', 6131),\n",
       " ('pm', 6113),\n",
       " ('abc', 6113),\n",
       " ('plans', 6109),\n",
       " ('gold', 6084),\n",
       " ('year', 6027),\n",
       " ('high', 6018),\n",
       " ('centre', 5928),\n",
       " ('public', 5909),\n",
       " ('dead', 5903),\n",
       " ('national', 5827),\n",
       " ('charges', 5807),\n",
       " ('case', 5752),\n",
       " ('act', 5751),\n",
       " ('iraq', 5745),\n",
       " ('workers', 5732),\n",
       " ('rise', 5729),\n",
       " ('house', 5717)]"
      ]
     },
     "execution_count": 194,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counter.most_common(100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['bpa',\n",
       " 'referal',\n",
       " 'towms',\n",
       " 'collating',\n",
       " 'wallareenya',\n",
       " 'lobbed',\n",
       " 'samarasinghe',\n",
       " 'springstown',\n",
       " 'blinman',\n",
       " 'UNK']"
      ]
     },
     "execution_count": 195,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_n_ratio=0.8\n",
    "\n",
    "counter = Counter(dict(counter.most_common(int(top_n_ratio * len(counter)))))\n",
    "vocab = list(counter) + ['UNK']\n",
    "vocab[-10:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "76712"
      ]
     },
     "execution_count": 196,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab_size = len(vocab)\n",
    "vocab_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {},
   "outputs": [],
   "source": [
    "word2index = {word:index for index, word in enumerate(vocab)}\n",
    "index2word = {index:word for word, index in word2index.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "96"
      ]
     },
     "execution_count": 198,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word2index['iraq']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'iraq'"
      ]
     },
     "execution_count": 200,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index2word[96]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### word corpus -> indexed corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [],
   "source": [
    "indexed_corpus_list = []\n",
    "for doc in corpus_list:\n",
    "    indexed_corpus_list.append([word2index[word] if word in word2index else word2index['UNK'] for word in doc.split()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'aba decides against community broadcasting licence'"
      ]
     },
     "execution_count": 202,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_list[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[10889, 5930, 76711, 151, 10890, 1252]"
      ]
     },
     "execution_count": 203,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indexed_corpus_list[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10889"
      ]
     },
     "execution_count": 204,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word2index['aba']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1252"
      ]
     },
     "execution_count": 205,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word2index['licence']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make X-y Dataset with negative sampling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {},
   "outputs": [],
   "source": [
    "# (참고) https://keras.rstudio.com/reference/skipgrams.html\n",
    "\n",
    "def generating_wordpairs(indexed_corpus, vocab_size, window_size=4):\n",
    "    X = []\n",
    "    Y = []\n",
    "    for row in indexed_corpus:\n",
    "        x, y = skipgrams(sequence=row, vocabulary_size=vocab_size, window_size=window_size,\n",
    "                        negative_samples=1.0, shuffle=True, categorical=False, sampling_table=None, seed=None)\n",
    "        X = X + list(x)\n",
    "        Y = Y + list(y)\n",
    "    return X, Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, Y = generating_wordpairs(indexed_corpus_list[0:100], vocab_size, window_size=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[10890, 67427]"
      ]
     },
     "execution_count": 209,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'broadcasting'"
      ]
     },
     "execution_count": 210,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index2word[10890]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'maldinis'"
      ]
     },
     "execution_count": 211,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index2word[67427]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 212,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Construction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_5 (InputLayer)            (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_6 (InputLayer)            (None, 1)            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_3 (Embedding)         (None, 1, 30)        2301360     input_5[0][0]                    \n",
      "                                                                 input_6[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "reshape_7 (Reshape)             (None, 30, 1)        0           embedding_3[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "reshape_8 (Reshape)             (None, 30, 1)        0           embedding_3[1][0]                \n",
      "__________________________________________________________________________________________________\n",
      "dot_3 (Dot)                     (None, 1, 1)         0           reshape_7[0][0]                  \n",
      "                                                                 reshape_8[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "reshape_9 (Reshape)             (None, 1)            0           dot_3[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "dense_5 (Dense)                 (None, 16)           32          reshape_9[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_6 (Dense)                 (None, 1)            17          dense_5[0][0]                    \n",
      "==================================================================================================\n",
      "Total params: 2,301,409\n",
      "Trainable params: 2,301,409\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "#embedding_dim=300\n",
    "embedding_dim=30\n",
    "\n",
    "input_target = Input((1,))\n",
    "input_context = Input((1,))\n",
    "\n",
    "embedding_layer = Embedding(vocab_size, embedding_dim, input_length=1)\n",
    "\n",
    "target_embedding = embedding_layer(input_target)\n",
    "target_embedding = Reshape((embedding_dim, 1))(target_embedding)\n",
    "context_embedding = embedding_layer(input_context)\n",
    "context_embedding = Reshape((embedding_dim, 1))(context_embedding)\n",
    "\n",
    "hidden_layer = Dot(axes=1)([target_embedding, context_embedding])\n",
    "hidden_layer = Reshape((1,))(hidden_layer)\n",
    "\n",
    "output = Dense(16, activation='sigmoid')(hidden_layer)\n",
    "output = Dense(1, activation='sigmoid')(output)\n",
    "\n",
    "model = Model(inputs=[input_target, input_context], outputs=output)\n",
    "model.summary()\n",
    "model.compile(loss='binary_crossentropy', optimizer='sgd')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 0, loss=1.0534988641738892\n",
      "Iteration 10, loss=1.069413423538208\n",
      "Iteration 20, loss=0.9909926056861877\n",
      "Iteration 30, loss=0.8904520869255066\n",
      "Iteration 40, loss=0.5047919750213623\n",
      "Iteration 50, loss=1.015505313873291\n",
      "Iteration 60, loss=1.0681167840957642\n",
      "Iteration 70, loss=1.0520728826522827\n",
      "Iteration 80, loss=0.4729217290878296\n",
      "Iteration 90, loss=0.9086063504219055\n",
      "Iteration 100, loss=0.9128470420837402\n",
      "Iteration 110, loss=0.5534312725067139\n",
      "Iteration 120, loss=0.6119768619537354\n",
      "Iteration 130, loss=0.7974879741668701\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-217-4bb2104157fb>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mepochs\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m     \u001b[0midx_batch\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrandom\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mchoice\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mindexed_corpus_list\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbatch_size\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 6\u001b[1;33m     \u001b[0mX\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mY\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mgenerating_wordpairs\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0marray\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mindexed_corpus_list\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0midx_batch\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtolist\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mvocab_size\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      7\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      8\u001b[0m     \u001b[0mword_target\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mword_context\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mzip\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m*\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "epochs = 100000\n",
    "batch_size = 512\n",
    "\n",
    "for i in range(epochs):\n",
    "    idx_batch = np.random.choice(len(indexed_corpus_list), batch_size)\n",
    "    X, Y = generating_wordpairs(np.array(indexed_corpus_list)[idx_batch].tolist(), vocab_size)\n",
    "\n",
    "    word_target, word_context = zip(*X)\n",
    "    word_target = np.array(word_target, dtype=np.int32)\n",
    "    word_context = np.array(word_context, dtype=np.int32)\n",
    "\n",
    "    target = np.zeros((1,))\n",
    "    context = np.zeros((1,))\n",
    "    label = np.zeros((1,))\n",
    "    idx = np.random.randint(0, len(Y)-1)\n",
    "    target[0,] = word_target[idx]\n",
    "    context[0,] = word_context[idx]\n",
    "    label[0,] = Y[idx]\n",
    "    loss = model.train_on_batch([target, context], label)\n",
    "    if i % 10 == 0:\n",
    "        print(\"Iteration {}, loss={}\".format(i, loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### save trained word2vec into file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {},
   "outputs": [],
   "source": [
    "word2vec_file_path = 'word2vec.txt'\n",
    "f = open(word2vec_file_path, 'w')\n",
    "f.write('{} {}\\n'.format(vocab_size-1, embedding_dim))\n",
    "vectors = model.get_weights()[0]\n",
    "for word, i in word2index.items():\n",
    "    f.write('{} {}\\n'.format(word, ' '.join(map(str, list(vectors[i, :])))))\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load trained word2vec\n",
    "\n",
    "(참고) https://radimrehurek.com/gensim/models/keyedvectors.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\uhmpp\\Anaconda3\\envs\\py36\\lib\\site-packages\\smart_open\\smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
      "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([ 0.01219402,  0.03773722,  0.04487152,  0.01146974, -0.04884932,\n",
       "        0.02861242,  0.01513971, -0.04361826, -0.00498766,  0.03830883,\n",
       "       -0.00213171, -0.02341004, -0.02808711, -0.03443178,  0.03845153,\n",
       "        0.00785465, -0.03831436, -0.02630966, -0.02292718, -0.04224662,\n",
       "       -0.04796321,  0.00908273, -0.03531182,  0.01840568,  0.01029073,\n",
       "       -0.0419643 ,  0.0182987 ,  0.00931753,  0.00032729, -0.04848105],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 225,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gensim.models.keyedvectors import Word2VecKeyedVectors\n",
    "\n",
    "word_vectors = Word2VecKeyedVectors.load_word2vec_format(word2vec_file_path, binary=False)\n",
    "vector = word_vectors['computer']\n",
    "vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('perisher', 0.6857004165649414),\n",
       " ('unloved', 0.6776821613311768),\n",
       " ('keller', 0.6709513664245605),\n",
       " ('phenomenal', 0.6538717150688171),\n",
       " ('creation', 0.6509047746658325),\n",
       " ('supersystem', 0.6281493306159973),\n",
       " ('bagshaw', 0.6279741525650024),\n",
       " ('appreciation', 0.6266207695007324),\n",
       " ('heathcote', 0.6193178296089172),\n",
       " ('0501', 0.6160426139831543)]"
      ]
     },
     "execution_count": 226,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_vectors.similar_by_word(\"cat\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py36",
   "language": "python",
   "name": "py36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}