5.朴素贝叶斯进行社会媒体挖掘.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Get Data from Twitter\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50% of ur friends in their 20s vs. the other 50% of ur friends in their 20s https://t.co/Fc57ZUOkQt\n",
      "open for a double surprise https://t.co/QQTjOWsVxD\n"
     ]
    }
   ],
   "source": [
    "import twitter\n",
    "import tweepy\n",
    "from tweepy import OAuthHandler\n",
    "consumer_key = \"hEDGEdp4p2Tp8U6h5vpOGOpm9\"\n",
    "consumer_secret = \"4C8P1YeOtsjebf6dYyTnRCWae3GWWoF9n6VLnQOKTEorlK3KbT\"\n",
    "access_token = \"846992744690208768-QQKUada7IjhguoQyjrARQ99nCkHVMop\"\n",
    "access_token_secret = \"5zfmqKV2eLPO2SU0XoW3DKtOUqwmvpPB1r7KoQ48OQnq8\"\n",
    "authorization = twitter.OAuth(access_token, access_token_secret, consumer_key, consumer_secret)\n",
    "t = twitter.Twitter(auth=authorization)\n",
    "\n",
    "auth = OAuthHandler(consumer_key,consumer_secret)\n",
    "auth.set_access_token(access_token,access_token_secret)\n",
    "\n",
    "api = tweepy.API(auth, proxy=\"127.0.0.1:8123\")\n",
    "\n",
    "for status in tweepy.Cursor(api.home_timeline).items(2):\n",
    "    print (status.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RT @machinelearnbot: Natural Language Processing with Deep Learning in Python https://t.co/X0yOxiezE8 #machinelearning\n",
      "@NationalTheatre for the faintly surreal St George where Monty Python meets Medieval Mystery play in riff on all things English.  Diverting.\n",
      "Disruptors: Entrepreneurs &amp; The Escape From Corporate America here  https://t.co/8SEKLMQtvg #python #programming @pythonbot_\n",
      "Python: Beginner's Guide To Programming Code With Python  https://t.co/3w2x9otZqb #software #development #python @pythonbot_\n",
      "RT @Amaka_Ekwo: Alert! \n",
      "Nigerian Army starts operation python poison across BiafraLand; forcefully injecting Biafran children with virus. #…\n",
      "俺の20mのフルアダルトの胴回りの太さはハンパねぇな。そう、大体直径1mくらいになっちゃうよ。\n",
      "django-carrot 0.1.5a0: A RabbitMQ asynchronous task queue for Django. https://t.co/mOPgCLJCqw\n",
      "puer 0.4.1:  https://t.co/oq6mpKqa9M\n",
      "@YouTube C++でコアを作ってFEにラップして+Pythonでスタンドアロン化してるのね。FE挟んでるからＤＣＣツールを選ばずどこでも使えると。\n",
      "ツール作る側からすれば最高の環境だな。\n",
      "RT @Demoore90210: Confirmation!\n",
      "Buhari intends to build the North; and send Military for Operation Python Dance, crorocdile smile to the So…\n",
      "RT @BiafraTelegraph: Biafra: Vaccination of Biafran children is continuation of Nigerian Army Operation Python Dance through biological… ht…\n",
      "RT @Okeke7Okeke: Nigeria army Operation Python Poison &amp; Monkey pox in Biafra Land must be stopped #BiafraLifeMatters @hrw @FLOTUS @UN https…\n",
      "RT @SciPyTip: From Python to Numpy https://t.co/nAJHJHAp13\n",
      "RT @Hypo_Thermick: Heavy Weapons Gal 👊\n",
      "#TeamFortress2 #tf2 #fanart based on one of the female merc concepts: https://t.co/U1rVIImzFf https:…\n",
      "RT @NkirukaNistoran: W'Bank to focus on northern Nigeria While operation python dance, crocodile smile and Monkey pox to focus on southern…\n",
      "RT @NLeconomyinTR: @pyistanbul &amp; @NLinIstanbul will co-organize a #Python programming language day on October 18th. Python was authored by…\n",
      "RT @KirkDBorne: Word Embeddings - A Natural Language Processing Crash Course: https://t.co/HrFVZS6ryb #BigData #DataScience #NLProc #ML by…\n",
      "RT @comradop: Like play, Like Play, IPOB is no longer daily topic online. \n",
      "\n",
      "Not even Kanu is talking 😂😂😂.  Python is a bastard 😂😂\n",
      "Pythonの勉強するよ\n",
      "RT @nardtree: Pythonの最強最悪のバグはメモリが破損していると、任意の箇所で任意のエラーが出ること（バグとは）\n",
      "RT @LibraryLadyTerr: Coding Club for kids for the next couple of months is going to be exploring Python. #hockessinlibrary #coding #Python…\n",
      "@CMDR_FOSDYKE Yeah! Some love for the python at last!\n",
      "mappyscript 0.0.3:  https://t.co/CDuTrF8pcD\n",
      "Weblate 2.17: A web-based translation tool with tight version control integration https://t.co/Medj2Xfwam\n",
      "amplpy 0.3.0rc1: Python API for AMPL https://t.co/NQYxvhTTjS\n",
      "appmode 0.0.2: A Jupyter extensions that turns notebooks into web applications. https://t.co/OrKVfEnwwz\n",
      "tbzuploader 2017.14.0:  https://t.co/WzZ1BPWAKD\n",
      "pyolite 1.5.9: Python wrapper for gitolite https://t.co/Pyp2AbdOIZ\n",
      "inspire-query-parser 1.0.6: A PEG-based query parser for INSPIRE. https://t.co/U0FYZ8juLX\n",
      "ccxt 1.9.131: A JavaScript / Python / PHP cryptocurrency trading library with support for 85+ exchanges https://t.co/cqBVWUm9Pf\n",
      "forgi 1.0: RNA Graph Library https://t.co/JefbWFfeoz\n",
      "autorandr 1.2.post1: Automatically select a display configuration based on connected devices https://t.co/3k1xPuUnHY\n",
      "tap-marketo 1.0.0: Singer.io tap for extracting data from the Marketo API https://t.co/uoHdWiOjs0\n",
      "Crocodile smilers March to d SOUTH-SOUTH, Python dancers March to d EAST while World bank March to d NORTH. Meaning of RESTRUCTURING to PMB!\n",
      "رابط الدورة \n",
      "https://t.co/pJUAVcupu5\n",
      "@alosefer , @OlaTechno . https://t.co/OEPdYtU9xb\n",
      "RT @Fope_: wrote a python program that draws a circle by first drawing a shitload of squares. https://t.co/svD9EI6H9e\n",
      "RT @NLeconomyinTR: @pyistanbul &amp; @NLinIstanbul will co-organize a #Python programming language day on October 18th. Python was authored by…\n",
      "RT @BiafrXit: School children running from free Python Dance military injection.. https://t.co/ocAEI30LWw\n",
      "Have the need for a #Python project, well, look no further, check out what we can do for you.\n",
      "https://t.co/exYDBx27Vh\n",
      "RT @IBMSPSS: #SPSSModeler new coding-free open source capabilities include exposed #Python algorithms. Learn more: https://t.co/BBFjUpNoQZ…\n",
      "Python 学習コース Ⅲを修了しました！ https://t.co/lb4FxXRDkR https://t.co/6FFR2RVXb7 #Progate\n",
      "RT @AstrobiologyNAI: The 2016 NASA Astrobiology Institute Annual Science Report is now available! \n",
      "https://t.co/yBT9L2JI27 https://t.co/FdE…\n",
      "BREAKING!! Over 30 Soldiers Killed, Time For The ''Python'' To Dance In The North And Leave The South-East(Graphic … https://t.co/jfRUmdQE8z\n",
      "@maisenza @IlTorto77 @Python_00 @nickdibauscia @JonnyVioni La prossima volta vado con #attimino\n",
      "Confirmation!\n",
      "Buhari intends to build the North; and send Military for Operation Python Dance, crorocdile smile to… https://t.co/0P5NllGfuK\n",
      "@FacuDiazT reflexiones de fascistas (autoreconocidos)  a lo Monty Python sobre accidente eurofighter. Es un no para… https://t.co/B9XcYRaHed\n",
      "RT @imocitynewsblog: VACCINATION OF BIAFRA CHILDREN, THE CONTINUATION OF NIGERIAN ARMY'S OPERATION PYTHON DANCE THROUGH BIOLOGICAL WEAPON h…\n",
      "dict.fromkeys( )メソッドを使うと、与えられたシーケンス(文字列、リスト、タプル)の要素をキーに持つ辞書が作られる。そのときに値を指定することも可能。何も指定しない場合はNoneとなる。例）dict.fromkeys([\"python\",\"ruby\"],\"lan\")\n",
      "@sanulovex6 please go through this https://t.co/A2QH6o91yJ\n",
      "プログラミング知識ゼロを前提に始まる文学部NLP講義、Emacを使わせたりPythonの’と”について説明なしにサンプルに混合して使ってたりよろしくない\n",
      "RT @pyblogsal: @facundobatista Concedido! Cuáles son sus otros dos deseos? https://t.co/jsjDmKN0WQ\n",
      "RT @YuriEngelhardt: Data Visualization Type Catalogs\n",
      "The Classic https://t.co/taCEovH6Q9\n",
      "The Thorough https://t.co/2pfLUuodo7\n",
      "The Python ht…\n",
      "RT @Zweshi: #WIP Gameplay code is coming along nicely, just a few more things to get done, hopefully this week! #Python #PyGame #indiedev #…\n",
      "fgoも自動運転あればなぁ\n",
      "RT @Zweshi: #WIP Gameplay code is coming along nicely, just a few more things to get done, hopefully this week! #Python #PyGame #indiedev #…\n",
      "RT @YuriEngelhardt: Data Visualization Type Catalogs\n",
      "The Classic https://t.co/taCEovH6Q9\n",
      "The Thorough https://t.co/2pfLUuodo7\n",
      "The Python ht…\n",
      "RT @pyblogsal: @facundobatista Concedido! Cuáles son sus otros dos deseos? https://t.co/jsjDmKN0WQ\n",
      "会社でマギレコ自動運転してる\n",
      "RT @taylorndean: lil tiny python https://t.co/3hXfXJR6qQ\n",
      "@voiceofbramley @BremainInSpain is this another repeat of the Monty Python sketch about the Romans? Shall I start..… https://t.co/LzpyURp5c4\n",
      "RT @infobeautiful: Get hands-on with this how-to #dataviz: https://t.co/lQ7o7yJiwR https://t.co/MUGfPVGRjv\n",
      "RT @rebecahdixon: Learn Python Programming by Making a Game ☞ https://t.co/aBcNysQcSu https://t.co/3T1E2fu9jY\n",
      "@hra1018 サブライムかPythonだけならpycharm\n",
      "See where Python, SAS, Matlab and more rank. https://t.co/zhWuVuqU9R\n",
      "RT @kankarara: PythonでMATLAB control system toolboxと同じようなことができるのか...初期教育はこれで充分かもしれないですね．\n",
      "https://t.co/BbhgqwZlvR\n",
      "Python leads to a broken build\n",
      "ReactとPythonで食っていきたい\n",
      "I've got much planned for my blog- python, social media plug ins, food photography. For now, blogspot and very basic. Change is coming!\n",
      "RT @randal_olson: #Python is the 2nd most popular language on #GitHub, behind only #JavaScript.\n",
      "\n",
      "(measured by Pull Requests) #programming #…\n",
      "RT @favourbia995203: Soldiers given force  immunisation is continuation of operation python dance,\n",
      "Killing #Biafrans  with python venom,\n",
      "@a…\n",
      "Harkとかいうもんに浸ってたからPython触っててもう感激.｡ﾟ+.(･∀･)ﾟ+.ﾟ\n",
      "RT @Voic_of_Justice: @HOPEFORBIAFRA @EkpereNwankwo @SaharaReporters @Amaka_Ekwo @realFFK @vanguardngrnews @channelstv @ReutersAfrica @IBTim…\n",
      "i ate like a hog, now I'm going to bask like a python.\n",
      "Rejoignez notre équipe de Professional Services à #Lyon ! Il y a des postes à pourvoir pour déployer @FluidTopics… https://t.co/H0Mp2PUash\n",
      "RT @python_devv: Docker Tutorial — Getting Started with Python, Redis, and Nginx. \n",
      "\n",
      "☞ https://t.co/LNFLKaCsKH\n",
      "\n",
      "#Python #Redis #Nginx https:…\n",
      "RT @odsc: Data Science Job Report 2017: R Passes SAS, But Python Leaves Them Both Behind https://t.co/YWvPIAdqOy #ODSC\n",
      "Nueva #OfertaDeEmpleo #Python #Django: Django freelance @ European Telemedicine Clinic https://t.co/ebuWhNGGYk\n",
      "Nigerian Journalist Drags President Buhari to ICC over Operation Python Dance II https://t.co/y8FUyc3ztV\n",
      "Python核のつらい気がしてきたんだけどPythonしか書けないんだよなぁ\n",
      "RT @Fope_: wrote a python program that draws a circle by first drawing a shitload of squares. https://t.co/svD9EI6H9e\n",
      "@IlTorto77 @Python_00 @NicolinoBerti @nickdibauscia @JonnyVioni #attimo\n",
      "RT @ipobworldnews: Biafra: [Biafra Killers Turned Healers]: The Nigerian Army Python Dance Exercise And Her Free Vaccination Exercise;… htt…\n",
      "python楽しくて30ページほど進めた\n",
      "Docker Tutorial — Getting Started with Python, Redis, and Nginx. \n",
      "\n",
      "☞ https://t.co/WjS01K1RKI\n",
      "\n",
      "#Python #Redis #Nginx https://t.co/aNszy1O8vo\n",
      "Pythonを書きたくない\n",
      "Uzahlaka kanjani when you want to get a python?  😂😂 https://t.co/UX5FNR6372\n",
      "ワイもpythonでゲーム作るか\n",
      "('ω')ﾏｲｽﾀｰﾊ?\n",
      "onegov.gazette 1.7.0: OneGov official notices website. https://t.co/s1Y8lnn0DD\n",
      "#Jobopening #BigData #developer in #London, #Uk Skills:  #Java, #J2EE- 6-8ys #Development  &amp; 3ys exp- #Hadoop stack… https://t.co/mqNQZtgORV\n",
      "#facepalm https://t.co/gPLOUZJPmF\n",
      "侵入者がこのあたりをうろついているというから、わざわざ出向いてきたというのに...残念だ\n",
      "80 kişilik dev ekipten absürt komedi: Monty Python and Holy Grail https://t.co/T62AhrXGll https://t.co/7dnyL68UjR\n",
      "@Python_00 @nickdibauscia @JonnyVioni @IlTorto77 @maisenza Maledetti\n",
      "RT @nineleaps: Resolve #methods in #Python\n",
      "#programming #pythonista #tech #algorithm\n",
      "https://t.co/u6efgr0vxz https://t.co/APtIfM8KQh\n",
      "RT @bamitav: \"#Habu: #Python Network Hacking Toolkit\" https://t.co/EQSKfXUifr #InfoSec #Security #CyberSecurity #CyberAttack #CyberWar #Hac…\n",
      "#Christian Louboutin ZIG 120 Python Strappy Platform Sandals Heels Shoe https://t.co/e9mQrkVA0O https://t.co/sC9eVKXfzS\n",
      "i want a python SO BAD but their food 😭\n",
      "The Python Power Coder BONUS Bundle for $45\n",
      "\n",
      "-70 hours of content and courses!\n",
      "\n",
      "https://t.co/6Ta9dYgxsS https://t.co/y8lw6VNNsv\n",
      "The Python Power Coder BONUS Bundle for $45\n",
      "\n",
      "-70 hours of content and courses!\n",
      "\n",
      "https://t.co/rj81f8VT13 https://t.co/EXOK1Abyaa\n",
      "The Python Power Coder BONUS Bundle for $45\n",
      "\n",
      "-70 hours of content and courses!\n",
      "\n",
      "https://t.co/n5Nwv3ArU2 https://t.co/UrIvYsNPrI\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import json\n",
    "\n",
    "output_filename = os.path.join(\".\", \"data\", \"python_tweets.json\")\n",
    "with open(output_filename, 'a') as output_file:\n",
    "    search_results = api.search(q=\"python\", count=100)\n",
    "    for tweet in search_results:\n",
    "        if 'text' in tweet._json:\n",
    "            print(tweet._json['text'])\n",
    "            output_file.write(json.dumps(tweet._json))\n",
    "            output_file.write(\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Label Twitter Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Labelling the class values for the twitter dataset.\n",
    "import os\n",
    "input_filename = os.path.join(\".\", \"data\", \"python_tweets.json\")\n",
    "classes_filename = os.path.join(\".\", \"data\", \"python_classes.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded 400 tweets\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "tweets = []\n",
    "with open(input_filename) as inf:\n",
    "    for line in inf:\n",
    "        if len(line.strip()) == 0:\n",
    "            continue\n",
    "        tweets.append(json.loads(line))\n",
    "print(\"Loaded {} tweets\".format(len(tweets)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "101\n"
     ]
    }
   ],
   "source": [
    "tweet_sample = tweets\n",
    "labels = []\n",
    "if os.path.exists(classes_filename):\n",
    "    with open(classes_filename) as inf:\n",
    "        labels = json.load(inf)\n",
    "print(len(labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_next_tweet():\n",
    "    return tweet_sample[len(labels)]['text']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "function set_label(label){\n",
       "    var kernel = IPython.notebook.kernel;\n",
       "    kernel.execute(\"labels.append(\" + label + \")\");\n",
       "    load_next_tweet();\n",
       "}\n",
       "\n",
       "function load_next_tweet(){\n",
       "   var code_input = \"get_tweet()\";\n",
       "   var kernel = IPython.notebook.kernel;\n",
       "   var callbacks = { 'iopub' : {'output' : handle_output}};\n",
       "   kernel.execute(code_input, callbacks, {silent:false});\n",
       "}\n",
       "\n",
       "function handle_output(out){\n",
       "   console.log(out);\n",
       "   var res = out.content.data[\"text/plain\"];\n",
       "   $(\"div#tweet_text\").html(res);\n",
       "}"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%javascript\n",
    "function set_label(label){\n",
    "    var kernel = IPython.notebook.kernel;\n",
    "    kernel.execute(\"labels.append(\" + label + \")\");\n",
    "    load_next_tweet();\n",
    "}\n",
    "\n",
    "function load_next_tweet(){\n",
    "   var code_input = \"get_tweet()\";\n",
    "   var kernel = IPython.notebook.kernel;\n",
    "   var callbacks = { 'iopub' : {'output' : handle_output}};\n",
    "   kernel.execute(code_input, callbacks, {silent:false});\n",
    "}\n",
    "\n",
    "function handle_output(out){\n",
    "   console.log(out);\n",
    "   var res = out.content.data[\"text/plain\"];\n",
    "   $(\"div#tweet_text\").html(res);\n",
    "}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div name=\"tweetbox\">\n",
       "    Instructions: Click in text box. Enter a 1 if the tweet is relevant, enter 0 otherwise.<br>\n",
       "    Tweet: <div id=\"tweet_text\" value=\"text\"></div><br>\n",
       "    <input type=text id=\"capture\"></input><br>\n",
       "</div>\n",
       "<script>\n",
       "function set_label(label){\n",
       "    var kernel = IPython.notebook.kernel;\n",
       "    kernel.execute(\"labels.append(\" + label + \")\");\n",
       "    load_next_tweet();\n",
       "}\n",
       "\n",
       "function load_next_tweet(){\n",
       "   var code_input = \"get_next_tweet()\";\n",
       "   var kernel = IPython.notebook.kernel;\n",
       "   var callbacks = { 'iopub' : {'output' : handle_output}};\n",
       "   kernel.execute(code_input, callbacks, {silent:false});\n",
       "}\n",
       "\n",
       "function handle_output(out){\n",
       "   console.log(out);\n",
       "   var res = out.content.data[\"text/plain\"];\n",
       "   $(\"div#tweet_text\").html(res);\n",
       "}\n",
       "$(\"input#capture\").keypress(function(e) {\n",
       "    console.log(e);\n",
       "  if(e.which == 48) {\n",
       "    // 0 pressed\n",
       "    set_label(0);\n",
       "    $(\"input#capture\").val(\"\");\n",
       "  }else if (e.which == 49){\n",
       "    // 1 pressed\n",
       "    set_label(1);  \n",
       "    $(\"input#capture\").val(\"\");\n",
       "  }\n",
       "});\n",
       "\n",
       "load_next_tweet();\n",
       "</script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<div name=\"tweetbox\">\n",
    "    Instructions: Click in text box. Enter a 1 if the tweet is relevant, enter 0 otherwise.<br>\n",
    "    Tweet: <div id=\"tweet_text\" value=\"text\"></div><br>\n",
    "    <input type=text id=\"capture\"></input><br>\n",
    "</div>\n",
    "<script>\n",
    "function set_label(label){\n",
    "    var kernel = IPython.notebook.kernel;\n",
    "    kernel.execute(\"labels.append(\" + label + \")\");\n",
    "    load_next_tweet();\n",
    "}\n",
    "\n",
    "function load_next_tweet(){\n",
    "   var code_input = \"get_next_tweet()\";\n",
    "   var kernel = IPython.notebook.kernel;\n",
    "   var callbacks = { 'iopub' : {'output' : handle_output}};\n",
    "   kernel.execute(code_input, callbacks, {silent:false});\n",
    "}\n",
    "\n",
    "function handle_output(out){\n",
    "   console.log(out);\n",
    "   var res = out.content.data[\"text/plain\"];\n",
    "   $(\"div#tweet_text\").html(res);\n",
    "}\n",
    "$(\"input#capture\").keypress(function(e) {\n",
    "    console.log(e);\n",
    "  if(e.which == 48) {\n",
    "    // 0 pressed\n",
    "    set_label(0);\n",
    "    $(\"input#capture\").val(\"\");\n",
    "  }else if (e.which == 49){\n",
    "    // 1 pressed\n",
    "    set_label(1);  \n",
    "    $(\"input#capture\").val(\"\");\n",
    "  }\n",
    "});\n",
    "\n",
    "load_next_tweet();\n",
    "</script>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "101\n"
     ]
    }
   ],
   "source": [
    "print(len(labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(classes_filename, 'w') as outf:\n",
    "    json.dump(labels, outf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classify Twitter Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded 400 tweets\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "tweets = []\n",
    "with open(input_filename) as inf:\n",
    "    for line in inf:\n",
    "        if len(line.strip()) == 0:\n",
    "            continue\n",
    "        tweets.append(json.loads(line)['text'])\n",
    "print(\"Loaded {} tweets\".format(len(tweets)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(classes_filename) as inf:\n",
    "    labels = json.load(inf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_samples = min(len(tweets), len(labels))\n",
    "sample_tweets = [t.lower() for t in tweets[:n_samples]]\n",
    "labels = labels[:n_samples]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "y_true = np.array(labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.0% have class 1\n"
     ]
    }
   ],
   "source": [
    "print(\"{:.1f}% have class 1\".format(np.mean(y_true == 1) * 100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.base import TransformerMixin\n",
    "from nltk import word_tokenize\n",
    "# import nltk\n",
    "class NLTKBOW(TransformerMixin):\n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "\n",
    "    def transform(self, X):\n",
    "        return [{word: True for word in word_tokenize(document)}\n",
    "                 for document in X]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/dlinking-lxy/more-space/pyworks/venv/lib/python3.5/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction import DictVectorizer\n",
    "from sklearn.naive_bayes import BernoulliNB\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/dlinking-lxy/more-space/pyworks/venv/lib/python3.5/site-packages/sklearn/cross_validation.py:552: Warning: The least populated class in y has only 3 members, which is too few. The minimum number of labels for any class cannot be less than n_folds=10.\n",
      "  % (min_labels, self.n_folds)), Warning)\n",
      "/home/dlinking-lxy/more-space/pyworks/venv/lib/python3.5/site-packages/sklearn/metrics/classification.py:1113: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "/home/dlinking-lxy/more-space/pyworks/venv/lib/python3.5/site-packages/sklearn/metrics/classification.py:1113: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n",
      "/home/dlinking-lxy/more-space/pyworks/venv/lib/python3.5/site-packages/sklearn/metrics/classification.py:1113: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.000\n"
     ]
    }
   ],
   "source": [
    "# nltk.download('punkt')\n",
    "pipeline = Pipeline([('bag-of-words', NLTKBOW()),\n",
    "                     ('vectorizer', DictVectorizer()),\n",
    "                     ('naive-bayes', BernoulliNB())\n",
    "                     ])\n",
    "scores = cross_val_score(pipeline, sample_tweets, y_true, cv=10, scoring='f1')\n",
    "print(\"Score: {:.3f}\".format(np.mean(scores)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "69px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}