Ch 15. 문서 요약.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"Ch15. 문서 요약(Text Summarization).ipynb","provenance":[],"collapsed_sections":[],"authorship_tag":"ABX9TyPkXxCnJCWGR1Y/LLujPXtu"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"name":"python"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"aJph87X395xC"},"source":["# 실습 [15-1]<br>\n","**실습명: TextRank을 이용한 추출 요약**<br>\n",": 추출 요약기법 TextRank 알고리즘을 활용해서 스코어 랭킹값을 이용해 요약 문장 추출"]},{"cell_type":"code","metadata":{"id":"ykRHlmmBzI0c"},"source":["#Gensim 라이브러리 설치\n","from gensim.summarization.summarizer import summarize\n","\n","text = '''Rice Pudding - Poem by Alan Alexander Milne\n","... What is the matter with Mary Jane?\n","... She's crying with all her might and main,\n","... And she won't eat her dinner - rice pudding again -\n","... What is the matter with Mary Jane?\n","... What is the matter with Mary Jane?\n","... I've promised her dolls and a daisy-chain,\n","... And a book about animals - all in vain -\n","... What is the matter with Mary Jane?\n","... What is the matter with Mary Jane?\n","... She's perfectly well, and she hasn't a pain;\n","... But, look at her, now she's beginning again! -\n","... What is the matter with Mary Jane?\n","... What is the matter with Mary Jane?\n","... I've promised her sweets and a ride in the train,\n","... And I've begged her to stop for a bit and explain -\n","... What is the matter with Mary Jane?\n","... What is the matter with Mary Jane?\n","... She's perfectly well and she hasn't a pain,\n","... And it's lovely rice pudding for dinner again!\n","... What is the matter with Mary Jane?'''"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"xRgx0Xr1-wGH","executionInfo":{"status":"ok","timestamp":1624168412389,"user_tz":-540,"elapsed":313,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}},"outputId":"7a879f15-83b0-4eab-94d7-86075aca716c"},"source":["print(summarize(text)) #추출 요약 (통계적 특징을 고려해 추출 문장 정렬, 가장 스코어값이 두드러지는 문장 찾기, 새로운 문장 없음)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["And she won't eat her dinner - rice pudding again -\n","I've promised her dolls and a daisy-chain,\n","I've promised her sweets and a ride in the train,\n","And it's lovely rice pudding for dinner again!\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"E7XEE4Eh-_Uw"},"source":["# 실습 [15-2]<br>\n","**실습명: 비지도 학습을 이용한 음식점 리뷰 추출 요약**<br>\n",": Kaggle의 [아마존 음식 리뷰 데이터](https://www.kaggle.com/snap/amazon-fine-food-reviews)를 활용해 추출 요약 진행 (기계학습 접근법 사용)<br>\n","\n","1. K-means 클러스터링을 이용해 비지도 학습을 통해 클러스터 구성\n","2. 사전 학습된 워드 임베딩(glove)를 이용해 클러스터에 속해있는 문장들의 평균값 수치화\n","3. 요약문장과의 비교를 통해 어느 클러스터가 요약문장과 유사한지 계산"]},{"cell_type":"code","metadata":{"colab":{"resources":{"http://localhost:8080/nbextensions/google.colab/files.js":{"data":"Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgZG8gewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwoKICAgICAgbGV0IHBlcmNlbnREb25lID0gZmlsZURhdGEuYnl0ZUxlbmd0aCA9PT0gMCA/CiAgICAgICAgICAxMDAgOgogICAgICAgICAgTWF0aC5yb3VuZCgocG9zaXRpb24gLyBmaWxlRGF0YS5ieXRlTGVuZ3RoKSAqIDEwMCk7CiAgICAgIHBlcmNlbnQudGV4dENvbnRlbnQgPSBgJHtwZXJjZW50RG9uZX0lIGRvbmVgOwoKICAgIH0gd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCk7CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK","ok":true,"headers":[["content-type","application/javascript"]],"status":200,"status_text":""}},"base_uri":"https://localhost:8080/","height":111},"id":"N1bCoXAyArLj","executionInfo":{"status":"ok","timestamp":1624172338934,"user_tz":-540,"elapsed":2835216,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}},"outputId":"da2afdca-f26b-4ec5-c2f0-6ee4e66b25cb"},"source":["#관련 라이브러리 설치\n","import numpy as np\n","import pandas as pd\n","from matplotlib import pyplot as plt\n","import re #정규표현식\n","\n","from nltk.tokenize import sent_tokenize #문장 토큰화\n","from keras.preprocessing.text import Tokenizer\n","from keras.preprocessing.sequence import pad_sequences\n","\n","import gensim.models.keyedvectors as word2vec\n","import gc\n","import string\n","import nltk\n","nltk.download('punkt')\n","\n","from google.colab import files\n","uploaded = files.upload()"],"execution_count":null,"outputs":[{"output_type":"stream","text":["[nltk_data] Downloading package punkt to /root/nltk_data...\n","[nltk_data]   Unzipping tokenizers/punkt.zip.\n"],"name":"stdout"},{"output_type":"display_data","data":{"text/html":["\n","     <input type=\"file\" id=\"files-398a5e01-7e2e-44b8-ac40-56515d4558f4\" name=\"files[]\" multiple disabled\n","        style=\"border:none\" />\n","     <output id=\"result-398a5e01-7e2e-44b8-ac40-56515d4558f4\">\n","      Upload widget is only available when the cell has been executed in the\n","      current browser session. Please rerun this cell to enable.\n","      </output>\n","      <script src=\"/nbextensions/google.colab/files.js\"></script> "],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{"tags":[]}},{"output_type":"stream","text":["Saving Reviews.csv to Reviews.csv\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"rb7cNkphB8EV","colab":{"base_uri":"https://localhost:8080/","height":432},"executionInfo":{"status":"ok","timestamp":1624172365170,"user_tz":-540,"elapsed":3371,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}},"outputId":"fe0683d9-e3f4-4978-f9fa-78e65e5ceef7"},"source":["import io\n","df = pd.read_csv(io.BytesIO(uploaded['Reviews.csv'])) #Reviews csv 파일 읽기\n","df.head(3) "],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>Id</th>\n","      <th>ProductId</th>\n","      <th>UserId</th>\n","      <th>ProfileName</th>\n","      <th>HelpfulnessNumerator</th>\n","      <th>HelpfulnessDenominator</th>\n","      <th>Score</th>\n","      <th>Time</th>\n","      <th>Summary</th>\n","      <th>Text</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>1</td>\n","      <td>B001E4KFG0</td>\n","      <td>A3SGXH7AUHU8GW</td>\n","      <td>delmartian</td>\n","      <td>1</td>\n","      <td>1</td>\n","      <td>5</td>\n","      <td>1303862400</td>\n","      <td>Good Quality Dog Food</td>\n","      <td>I have bought several of the Vitality canned d...</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>2</td>\n","      <td>B00813GRG4</td>\n","      <td>A1D87F6ZCVE5NK</td>\n","      <td>dll pa</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>1</td>\n","      <td>1346976000</td>\n","      <td>Not as Advertised</td>\n","      <td>Product arrived labeled as Jumbo Salted Peanut...</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>3</td>\n","      <td>B000LQOCH0</td>\n","      <td>ABXLMWJIXXAIN</td>\n","      <td>Natalia Corres \"Natalia Corres\"</td>\n","      <td>1</td>\n","      <td>1</td>\n","      <td>4</td>\n","      <td>1219017600</td>\n","      <td>\"Delight\" says it all</td>\n","      <td>This is a confection that has been around a fe...</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   Id  ...                                               Text\n","0   1  ...  I have bought several of the Vitality canned d...\n","1   2  ...  Product arrived labeled as Jumbo Salted Peanut...\n","2   3  ...  This is a confection that has been around a fe...\n","\n","[3 rows x 10 columns]"]},"metadata":{"tags":[]},"execution_count":2}]},{"cell_type":"code","metadata":{"id":"gwakndB6CiAo","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1624172453266,"user_tz":-540,"elapsed":76900,"user":{"displayName":"DA YEON KI","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GiWfZD29yea7akQsMZ0L-JyyDBDzCak-4Q55tt-jw=s64","userId":"15097711384147757777"}},"outputId":"0a1a2b63-713f-4184-ba78-569ce6005c55"},"source":["#각 리뷰별로 문장 단위로 분할\n","def split_sentences(reviews):\n","  n_reviews = len(reviews)\n","  for i in range(n_reviews): #리뷰 수만큼\n","    review = reviews[i] \n","    sentences = sent_tokenize(review)\n","    for j in reversed(range(len(sentences))):\n","      sent = sentences[j]\n","      sentences[j] = sent.strip()\n","      if sent == '':\n","        setences.pop(j) #뒤에서부터 제거\n","    reviews[j] = sentences\n","\n","rev_list = list(df['Text'])\n","split_sentences(rev_list) #리뷰 데이터 split하기\n","df['sent_tokens'] = rev_list #split된 리뷰 데이터 sent_tokens에 넣기\n","\n","#각 리뷰별 문장 길이 계산 (5보다 긴 길이의 문장만 불러오기)\n","df['length_of_rv'] = df['sent_tokens'].map(lambda x: len(x))\n","choice_length = 5\n","df = df[df['length_of_rv'] > choice_length]\n","df.shape"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["(568453, 12)"]},"metadata":{"tags":[]},"execution_count":3}]},{"cell_type":"code","metadata":{"id":"HIYhkLbkEm-w"},"source":["#리뷰들의 최대 vocab 개수 5000개로 한정\n","list_sentences_train = df['Text']\n","max_features = 5000\n","tokenizer = Tokenizer(num_words=max_features)\n","tokenizer.fit_on_texts(list(list_sentences_train))\n","list_tokenized_train = tokenizer.texts_to_sequences(list_sentences_train)\n","maxlen = 200\n","X_t = pad_sequences(list_tokenized_train, maxlen=maxlen)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"tBZ0-5Tx2RES"},"source":["- glove: 벡터 공간 상에서 단어의 의미 표현 나타냄\n","- fasttext: 단어를 더 작은 단위로 쪼개서 표현"]},{"cell_type":"code","metadata":{"id":"zo-MoretFR_H"},"source":["def loadEmbeddingMatrix(typeToLoad):\n","        #사전 학습된 glove 임베딩 (glove=워드 임베딩 방법론)\n","        if(typeToLoad==\"glove\"):\n","            uploaded_glove = files.upload()\n","            EMBEDDING_FILE = io.BytesIO(uploaded['glove.twitter.27B.25d.txt'])\n","            EMBEDDING_FILE = 'glove.twitter.27B.25d.txt'\n","            embed_size = 25\n","        \n","        elif(typeToLoad==\"fasttext\"): #fasttext\n","            EMBEDDING_FILE='wiki.simple.vec/wiki.simple.vec'\n","            embed_size = 300\n","\n","        if(typeToLoad==\"glove\" or typeToLoad==\"fasttext\" ):\n","            embeddings_index = dict()\n","            #파일의 모든 line에 대해서 glove를 이용해서 수치화\n","            f = open(EMBEDDING_FILE, encoding='utf-8')\n","            for line in f:\n","                values = line.split()\n","                #1번째 인덱스는 단어\n","                word = values[0]\n","                #나머지 값들은 새로운 배열에 저장\n","                coefs = np.asarray(values[1:], dtype='float32')\n","                embeddings_index[word] = coefs\n","            f.close()\n","            print('Loaded %s word vectors.' % len(embeddings_index))\n","\n","        gc.collect()\n","        return embeddings_index"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"n3EdAVnfajFo"},"source":["#glove 로딩\n","emb_index = loadEmbeddingMatrix('glove')"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"dKUe5T0lGc3N"},"source":["#전체 문장들에 대해 단어들의 임베딩 평균값으로 문장 임베딩 계산\n","\n","def calculate_sentence_embedding(wordList):\n","    #wordList = re.sub(\"[^\\w]\", \" \",  sent).split()\n","    #print(wordList)\n","    emb_li =[]\n","    for k in wordList:\n","        embedding_vector = emb_index.get(k)\n","        if embedding_vector is not None:\n","            if(len(embedding_vector) == 25):\n","                emb_li.append(list(embedding_vector))\n","    #print(\"Lost words in translation: \", len(wordList)-len(emb_li))\n","    mean_arr = np.array(emb_li)\n","    #print(\"done calculating sentence emb for you\")\n","    return np.mean(mean_arr, axis=0)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"MAa4WtPnGm17"},"source":["#미리 정의된 함수(calculate_sentence_embedding)를 이용해서 문장에 대해 임베딩 할당 -> 전처리 과정\n","def get_sent_embedding(mylist):\n","    sent_emb = []\n","    n_sentences = len(mylist)\n","    for i in mylist:\n","        #print(\"my sentence : \", i)\n","        #print(\"\\nlower is\", i.lower())\n","        i = i.lower()\n","        wL = re.sub(\"[^\\w]\", \" \",  i).split()\n","        if(len(wL)>0):\n","            for k in wL:\n","                if(k in string.punctuation):\n","                    wL.remove(k)\n","            if(len(wL) <= 2):\n","                continue\n","        else:\n","            print(\"Sentence Removed: \",i)\n","            continue\n","        #print(wL)\n","        res = list(calculate_sentence_embedding(wL))\n","        sent_emb.append(res)\n","    return np.array(sent_emb)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"kpkiUUJQG1yR"},"source":["from sklearn.cluster import KMeans\n","from sklearn.metrics import pairwise_distances_argmin_min\n","#5000건 리뷰의 요약문장과 비교해 어떤 클러스터가 요약문장과 가장 유사한지 확인\n","\n","how_many_summaries = 5000\n","summary = [None]*how_many_summaries\n","for rv in range(how_many_summaries):\n","        review = df['sent_tokens'].iloc[rv]\n","        enc_email = get_sent_embedding(review)\n","        if(len(enc_email) > 0):\n","            n_clusters = int(np.ceil(len(enc_email)**0.5))\n","            kmeans = KMeans(n_clusters=n_clusters, random_state=0)\n","            kmeans = kmeans.fit(enc_email)\n","            avg = []\n","            closest = []\n","            for j in range(n_clusters):\n","                idx = np.where(kmeans.labels_ == j)[0]\n","                #print(\"IDX is: \", idx)\n","                avg.append(np.mean(idx))\n","            closest, _ = pairwise_distances_argmin_min(kmeans.cluster_centers_,\\\n","                                                       enc_email)\n","            ordering = sorted(range(n_clusters), key=lambda k: avg[k])\n","            summary[rv] = ' '.join([review[closest[idx]] for idx in ordering])\n","            print(\"Done for review # = \", rv)\n","        else:\n","            print(\"This is not a valid review\")"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"2Zt58RVsHDij"},"source":["df_5000 = df.iloc[:5000]\n","df_5000['PredictedSummary'] = summary\n","df_5000[['Text', 'PredictedSummary']].to_csv('top_5000_summary.csv')"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"sZ6-qrDeHJsD"},"source":["df_result = pd.read_csv('top_5000_summary.csv')\n","df_result.head(10)"],"execution_count":null,"outputs":[]}]}