example/DFML_demo_local.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 設定Spark\n",
    "import os\n",
    "import sys\n",
    "# SPARK_HOME=\"/opt/spark-1.4.1-bin-hadoop2.6\"\n",
    "# os.environ[\"SPARK_HOME\"] = SPARK_HOME\n",
    "# sys.path.append(os.path.join(SPARK_HOME, 'python'))\n",
    "# sys.path.append(os.path.join(SPARK_HOME, 'python/lib/py4j-0.8.2.1-src.zip'))\n",
    "#----\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "# import seaborn as sns\n",
    "# sns.set(rc={\"figure.figsize\": (14, 5)},palette=sns.color_palette(\"Set1\", 2))\n",
    "from pyspark.sql.types import *\n",
    "from pyspark.sql.functions import *\n",
    "from pyspark.ml.feature import *\n",
    "from pyspark.ml.classification import *\n",
    "from pyspark.ml.param import *\n",
    "from pyspark.ml import *\n",
    "# from pyspark import HiveContext\n",
    "# ctx = HiveContext(sc)\n",
    "ctx=sqlContext "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#DataFrame與Pipeline使用範例\n",
    "\n",
    "1. 使用 DataFrame 探索資料，觀察時間與spam的關係。\n",
    "2. 使用ML Pipeline進行spam預測。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#資料說明\n",
    "\n",
    "資料來源是 2015痞客邦 PIXNET HACKATHON 活動中開放的[資料集](https://pixnethackathon2015.events.pixnet.net/dataset/readfirst.html)\n",
    "<h2 align=\"center\">[痞客邦](https://www.pixnet.net/)</h2> \n",
    "痞客邦目前華文世界最大部落格服務網站，當中蘊含大量的優質文章，但也不乏惡意灌水的廣告文章，目前站方已經設計多種演算法來偵測劣質文章，也希望多方好手及駭客高手腦力激盪來找出更好的方法揪出網路流氓。身為台灣社群龍頭，以每天2800萬次瀏覽到訪網站的流量，痞客邦開放海量數據資料，讓參賽者可以看見台灣網友更豐富多元的面向與使用行為。 \n",
    "\n",
    "---\n",
    "文章資料集蘊含正常使用者所發佈的優質文章，及spam使用者所發佈的劣質廣告文章，資料收集時間為2015/4。\n",
    "因為data使用規定，本demo無法直接提供資料，請於資料連結內下載。\n",
    "資料欄位說明：\n",
    "```\n",
    "\"post_at\": 文章發佈時間 (Unix Timestamp),\n",
    "\"author\": 作者 ID ,\n",
    "\"tags\": [文章歸屬的標籤列表],\n",
    "\"title\": 文章的標題,\n",
    "\"hits\": 文章總人氣,\n",
    "\"content\": 部落格本文\n",
    "\"comment_count\": 多少人 comment 過,\n",
    "\"comment_ids\": [留言者ID] ,\n",
    "\"category\": 文章的分類,\n",
    "\"spam\": 是否為 Spam 文章,1為spam，0為正常。\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#讀取資料集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- category: string (nullable = true)\n",
      " |-- comment_count: long (nullable = true)\n",
      " |-- comment_ids: array (nullable = true)\n",
      " |    |-- element: string (containsNull = true)\n",
      " |-- hits: long (nullable = true)\n",
      " |-- tags: array (nullable = true)\n",
      " |    |-- element: string (containsNull = true)\n",
      " |-- title: string (nullable = true)\n",
      " |-- spam: double (nullable = true)\n",
      " |-- post_at: long (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "path = <<FILL-IN>> # the path of file: articles-half-a.json.\n",
    "panda_df = pd.read_json(path) #如果無法讀入，可以試試python的json套件\n",
    "df=ctx.createDataFrame(panda_df)\n",
    "df=df.withColumn('spam2',df['spam'].astype(DoubleType())).drop('spam').withColumnRenamed('spam2','spam')  # 1.4 bug\n",
    "df=df.withColumn('post_at2',(df['post_at']/1000000000).astype(LongType())).drop('post_at').withColumnRenamed('post_at2','post_at')  # 1.4 bug\n",
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------+-------------+----+----------+--------------------+----+\n",
      "|category|comment_count|hits|   post_at|               title|spam|\n",
      "+--------+-------------+----+----------+--------------------+----+\n",
      "|    時尚流行|            0|   0|1427897290|LACOSTE 2013 春夏新品預覽會| 1.0|\n",
      "|     汽機車|            0| 445|1428297431|     VIRAGE 引擎抖動、不順!| 0.0|\n",
      "+--------+-------------+----+----------+--------------------+----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 觀察table內的數值\n",
    "df.select('category','comment_count','hits','post_at','title','spam').show(2) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#使用 DataFrame 探索資料，觀察時間與spam的關係\n",
    "我們想要知道spam和時間的關係，是否spam大部份發生在晚上或白天，非spam是否有分布上的差異。   \n",
    "原始資料是巨大的表格形式，不利於觀察，我們利用Spark處理大資料的能力將大資料聚集成小資料，再利用Pandas將它視覺化。  \n",
    "\n",
    "步驟：\n",
    "1. 觀察原始時間資料。\n",
    "2. 定義解析時間用的UDF ,用udf將utc數字轉成時間類型。\n",
    "2. 計算每一小時的spam和非sapm數量，也就是聚集時間和spam成group並進行count。\n",
    "3. 視覺化，轉成pandas畫圖。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###觀察原始時間資料\n",
    "原始時間在欄位'post_at',格式是UTC second."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------+----+\n",
      "|   post_at|spam|\n",
      "+----------+----+\n",
      "|1427897290| 1.0|\n",
      "|1428297431| 0.0|\n",
      "|1428749080| 1.0|\n",
      "+----------+----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.select('post_at','spam').show(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###定義解析時間用的UDF \n",
    "1. 設定台灣時差\n",
    "2. 定義把utc時間轉成當天小時時間的function\n",
    "3. 定義SQL UDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from datetime import datetime,tzinfo,timedelta\n",
    "# 定義把utc時間轉成當天小時時間的function\n",
    "get_hour = lambda x: (datetime.utcfromtimestamp(float(x)) + timedelta(hours=8)).hour \n",
    "# 定義SQL UDF\n",
    "hourOfDay = udf(get_hour, IntegerType()) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###計算每一小時的spam和非sapm數量\n",
    "1. 使用udf，增加hour 欄位 \n",
    "2. group by (spam , hour) \n",
    "3. 計算每個group內的數量並且命名為 'count' \n",
    "4. 照時間順序排序結果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- spam: double (nullable = true)\n",
      " |-- hour: integer (nullable = true)\n",
      " |-- count: long (nullable = false)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df_hour=(df.withColumn('hour',hourOfDay(df['post_at'])) # 使用上述定義的UDF，增加hour欄位\n",
    "            .groupBy('spam','hour')                     # group by (spam , hour) \n",
    "            .agg(count('*').alias('count'))             # 計算每個group內的數量並且命名為 'count' \n",
    "            .orderBy(asc('hour')))                      # 照時間順序排序結果\n",
    "            \n",
    "df_hour.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###視覺化\n",
    "圖片藍色是spam文章，紅色是正常文章。  \n",
    "經由視覺化更有效地觀察資料，我們可以發現，發文數量呈現日多夜少的情形。\n",
    "非spam的發文數量於晚間12點達到最高峰，晚餐和睡眠時間最低。\n",
    "spam方面與非spam的發文數量相比，spam半夜特別多，且變化較大。  \n",
    "\n",
    "步驟：\n",
    "1. spark dataframe轉成pandas dataframe\n",
    "2. 使用pandas畫圖"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x7f4a8cea5110>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAz0AAAFVCAYAAADFUIFrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XucHXV5+PHPkrABTMQEIkEItxgeqVbkWtpiQUsVirdq\nQamXKN5+tkVrlQreoFoFBFTAilINCcjVqogXLBQVpSIFRbSKTxQhBEhCMBsgBBMS9vfHTMIS9gZ7\n5sxk9vN+vfLKOTOz8zzn7Dln9jnfW09/fz+SJEmS1Fab1Z2AJEmSJFXJokeSJElSq1n0SJIkSWo1\nix5JkiRJrWbRI0mSJKnVLHokSZIktZpFjySp8SLi4IhY1KFzzYuIj3biXJKkTYNFjySpdhFxe0S8\nsEvh+st/kqRxwqJHktQE/UBP3Ul0UkRMrDsHSVLBD2RJ0mNExEzgDOBAii/HLsrMYyJiM+D9wFuA\nLYHvAMdk5v0RcTBwfmbOHHCe24GjM/O7EXEi8EfAQ8DfAHcAczLzJxFxPrAT8I2IWAf8a2aeNkRu\nxwP/DKwEPpCZF0bEfsA3gO0zs7887pXAhzPzeUM8zGkR8U3gL4BfAX+Xmb8rf/bPysc/G1gAvCsz\nrxvwmN6cmVeX908EZmXm6yNiF+B35fNzAnAbcPCwT7YkqSts6ZEkbRARE4BvUvzBvjOwA3BRufuN\nwByKP+R3AyYDnxnmdBt3IXtpea6tgcvX/2xmvp6iCHpJZk4ZquABZgDbAM8o8zgnImZn5g3A74EX\nDzj29cD8Ic7TA7wGOBGYCvwW+BhAREwDvgV8GpgGfBL4VkRMHfCYBj6uwbrJ/QXwrI3ykSTVyKJH\nkjTQ/sD2wLGZ+VBmrs7MH5X7Xgucnpm3Z+aDwPHAa8oWoNH4YWZ+p2yN+RKw55PI70OZ+XBm/oCi\nOHl1uf084HWwoXB5EXDhEOfoB76amTdm5jrgAmB9i9DhQGbmBZn5SGZeDPyaomAbzGBd8k5c/9w9\n0QcnSaqGRY8kaaCZwMLMfGSQfdsDCwfcv4Oim/R2ozz30gG3VwFbPIGCCaAvMx8acH8hRasPFIXL\nSyNiK+BI4AeZuXTjEwyRy0MUrVaU57tjo2MXUrR4jVZHZpmTJHWORY8kaaBFwE5lN7eN3Q3sMuD+\nTsBaigLiQWCr9TvKn5/+BOKOZja1qWVRs97OwF0AmXkn8GPglRQtPuc/gdgD3VWed6ANcSge51MG\n7JsxyDmcGU6SGsaiR5I00PXAYuDkiNgqIrYoB/ZDMR7n3RGxS0RMBj4OXFy2Ci2gaLn564jYHPgg\nMOkJxF0KzBrFcf8aEZtHxPMpuqJ9ecC+84D3Ac8BvjrMOYabJe4KYPeIOCoiJkbEqynG53yz3P8z\nii59EyNiX+BVWORIUuNZ9EiSNigLmJcCz6To5rWIorsYwFyKFpQfUMxStgo4pvy5+4C/B74A3Ekx\nu9rAbl6DrY0z8P5JwAcjoi8i/nmQ1PopirE+ihan84G3Z+aCAcd8laL16WuZ+YdhHuaQuWTm74GX\nAO8B7gXeSzHBwvLyuA9RFGd9FBMhXDDMY5IkNURPf3+1n88R8TSKi+CzKS4GbwJ+A1xC0WXgduDI\nzFxRaSKSpNaLiN9QFEPfrTsXSVJzdKOl5wzg25m5B/BcillwjgOuyszdgavL+5IkPWnl2jz9FjyS\npI1VujhpRGwNPD8z5wBk5lrgvoh4GXBQedh84PtY+EiSnqSI+D7F2JvX15yKJKmBKi16gF2BZRFx\nLsV6DD8B/gnYbsBUoksZ/XSnkiQ9TmYeXHcOkqTmqrp720Rgb+Czmbk3xVSfj2nRKRepc+CnJEmS\npEpU3dJzJ3BnZt5Q3v9PihW8l0TEjMxcEhHbA/cMd5K1a9f1T5w42JIRkiRJkgQMsyRBpUVPWdQs\niojdy2lFDwF+Wf6bA5xS/n/ZcOfp61s1pjymT5/CsmUPjOkcY1V3DnXHb0IOdcdvQg51x29CDnXH\nN4dmxG9CDnXHb0IOdcdvQg51x29CDnXHN4dmxO9EDtOnTxlyX9UtPVCs4XBBRPQCt1JMWT0BuDQi\n3kw5ZXUX8pAkSZI0DlVe9GTmzcB+g+w6pOrYkiRJktSNdXokSZIkqTYWPZIkSZJazaJHkiRJUqtZ\n9EiSJElqNYseSZIkSa3WjSmrJUmSJG1kzZo1LFq0EIC+vsksX75yzOecOXNnent7x3yega644pvs\nt98BbLvtth09bzdZ9EiSJEk1WLRoIbce/mJ2mjiBpR043x1r18G3/otZs2Z34GyP+va3v8Guu86y\n6JEkSZL0xO00cQK7Tdy8a/EWL76b9773nTz3uXvxf/93M9OnP52TTjodmMJvfpOceupJrF69mh12\n2JHjj/8wN954Pb/+9S185CMfZIsttuDss+cyadKkDef78pcv5utf/yoTJkxg111348QTP8YXv/h5\n7r77Tu666y5WrFjBa1/7Bl760lewatUqjj/+vTzwwP2sW7eWt771HRx44EEsXnw373nPMey77z7c\ncMONPOtZf8Rhh72Ec889h76+FZxwwkfZY49nj+lxO6ZHkiRJGkfuvHMRr3rVkZx//qVMnjyFa675\nLgD/9m8n8A//8C7mz7+IWbOeybnnnsMLXnAIz3rWHpxwwseYO/eCxxQ8ABdcMJ958y5k/vyLOPbY\n4zds/93vbuXMMz/H5z8/l3PP/Q/uvfdeJk2axEknncrcuV/ijDM+x2c+8+kNx991150cffTRXHjh\nV7jjjoVcffWVnH32XP7xH9/FeeedO+bHbNEjSZIkjSPbb78Dz3xm0QUu4lksXnw3K1euZOXKley5\n514AHHro4fzsZzdt+Jn+/v5BzzVr1mxOPPEDXHnlFWy22QQAenp6OPDAg+jt7WXrrZ/G3nvvyy23\n/B/9/f187nOfYc6co3j3u/+ee+9dRl/f8g05zZ49m56eHnbddTf23Xd/AHbddRZLltw95sds0SNJ\nkiSNI729j3an22yzCTzyyCMj/kxPT8+g20899dO88pVHkPlr3vrWN7Bu3bohf/7KK6/gvvtWMHfu\nlzj33AuZOnUaq1evGSSnzdh888033B7qnE+ERY8kSZI0jvX39zN58mSmTHkqN9/8MwC+851vsdde\n+wCw1VZb8eCDj59Zrr+/n6VLl7D33vvyjnccw8qVK3nooYfo7+/n2muvYc2aNdx33wpuuukn7LHH\ns3nwwQeZOnUaEyZM4Kc/vZElSxZ37TE6kYEkSZJUkzvWjr0VY+C5Zo3iuI1bbdbf/8AHTuS0007i\nD3/4AzvssCPvf/8JAPz1X7+U00476XETGaxbt46PfvTDPPjgSvr7+zniiNcwefJkenp6mDVrNu98\n5/9jxYoVvOlNb2GbbbblRS86lPe975+ZM+c1ROzBzjvvOmJOg+17MnqG6p/XJMuWPTCmJKdPn8Ky\nZQ90Kp1NMoe64zchh7rjNyGHuuM3IYe645tDM+I3IYe64zchh7rjNyGHuuM3IYe649eZw8B1eqZN\nq3ednk4+B3PnnsOWW27FUUe9rqs5TJ8+ZcjqyJYeSZIkqQa9vb0b1tRpQvHXSR1onOkoix5JkiRJ\nHXP00W+rO4XHcSIDSZIkSa1m0SNJkiSp1Sx6JEmSJLWaRY8kSZKkVnMiA0mSJKkGA6es7uurd8rq\ntrPokSRJkmqwaNFC3njWFUyaOqMj51vdt4R5xxy2YRpsPcqiR5IkSarJpKkz2HKbHbsW76GHHuLD\nHz6OZcuW8cgj65gz5y2cffaZvOQlh/O9732f3t5JnHjix9hhhx259tofcN55c1m79mGe+tStOeGE\nf2Pq1Gl88YufZ/Hiu1m8+G6WLl3CMce8m1/84ufccMOP2Xbbp3PKKZ9k4sRmlRmO6ZEkSZLGieuv\n/xHbbvt05s27kPPOu4QDDvhTenp6eOpTn8r8+RfzqlcdyRlnnA7AnnvuxTnnzGPu3Av4y798ERdc\ncN6G8yxefDdnnvk5Tj75k3zkIx9iv/3+hPnzL2bSpElcd921dT28IVn0SJIkSePErFmzufHG6zn7\n7LO4+eaf8ZSnTAbg8MMPB+CQQ17ML3/5cwDuuWcp7373PzBnzmu46KLzuf323wHQ09PDAQf8GRMm\nTGC33WbR39/Pn/zJn5bnfyaLFy+u4ZENr1ntTpIkSZIqM3PmTsydewHXXXct//Efn2WfffZ73DE9\nPT0AfOpTn+Coo17Pn//587nppp8wd+45G46ZOHFzADbbbDMmTJj4mJ9dt25dxY/iibOlR5IkSRon\n7r33Xnp7e3nRiw7jqKNez4IFCcC3v/1tAK6++kqe85znArBq1YNsu+10AK644psbztHf39/lrMfO\nlh5JkiSpJqv7lnT1XL/73W/5938/g80262HixM15z3uO40Mfeh/3338/c+YcRW9vLyee+DEAjj76\nbXzoQ+9jypSnss8++7JkSdFtraenh7IxaMP9gTa62wgWPZIkSVINZs7cmXnHHAbAtGmdW6dnOPvv\nfwD773/A47a/5S1vYc6ctz9m24EHHsSBBx70uGOPPvptj7l/5ZXXDLmvKSx6JEmSpBr09vZuWFNn\n+vQpLFv2QE2ZNLBppsMc0yNJkiSNY1/+8td52tOeVncalbLokSRJktRqdm+TJKnL1qxZw4IFC4bt\nvz9z5s709vZ2MStJai+LHkmSumzRooW88awrmDR1xqD7V/ctYd4xh23o6y9JGhuLHkmSajBp6gy2\n3GbHutOQpHHBMT2SJEmSWs2iR5IkSVKrWfRIkiRJarXKx/RExO3A/cA64OHM3D8ipgGXADsDtwNH\nZuaKqnORJEmSNP50o6WnHzg4M/fKzP3LbccBV2Xm7sDV5X1JkiRJ6rhudW/r2ej+y4D55e35wCu6\nlIckSZKkcaZbLT3/HRE3RsRby23bZebS8vZSYLsu5CFJkiRpHOrp7++vNEBEbJ+ZiyNiOnAVcAxw\neWZOHXDM8sycNtQ51q5d1z9x4oRK85QkqVsWLFjAEWf+cMh1eh76/Z18+Z3PZ/fdd+9yZpK0Sdu4\nd9kGlU9kkJmLy/+XRcTXgP2BpRExIzOXRMT2wD3DnaOvb9WYcpg+fQrLlj0wpnOMVd051B2/CTnU\nHb8JOdQdvwk51B3fHJoRv+4cli9fOapjqs6v7t9D3fGbkEPd8ZuQQ93xzaEZ8TuRw/TpU4bcV2n3\ntojYKiKmlLefArwI+AVwOTCnPGwOcFmVeUiSJEkav6pu6dkO+FpErI91QWZeGRE3ApdGxJspp6yu\nOA9JkiRJ41SlRU9m3gY8b5Dty4FDqowtSZIkSdC9KaslSZIkqRYWPZIkSZJazaJHkiRJUqtZ9EiS\nJElqNYseSZIkSa1m0SNJkiSp1Sx6JEmSJLWaRY8kSZKkVrPokSRJktRqFj2SJEmSWs2iR5IkSVKr\nWfRIkiRJajWLHkmSJEmtZtEjSZIkqdUseiRJkiS1mkWPJEmSpFaz6JEkSZLUahY9kiRJklrNokeS\nJElSq1n0SJIkSWo1ix5JkiRJrWbRI0mSJKnVLHokSZIktZpFjyRJkqRWs+iRJEmS1GoWPZIkSZJa\nzaJHkiRJUqtZ9EiSJElqNYseSZIkSa1m0SNJkiSp1Sx6JEmSJLWaRY8kSZKkVrPokSRJktRqFj2S\nJEmSWs2iR5IkSVKrWfRIkiRJajWLHkmSJEmtNrHqABExAbgRuDMzXxoR04BLgJ2B24EjM3NF1XlI\nkiRJGp+60dLzLuBXQH95/zjgqszcHbi6vC9JkiRJlai06ImIHYG/Br4A9JSbXwbML2/PB15RZQ6S\nJEmSxreqW3o+BRwLPDJg23aZubS8vRTYruIcJEmSJI1jlRU9EfES4J7MvIlHW3keIzP7ebTbmyRJ\nkiR1XE9/fzU1R0R8HHg9sBbYAngq8FVgP+DgzFwSEdsD38vMZw13rrVr1/VPnDihkjwlSeq2BQsW\ncMSZP2TLbXYcdP9Dv7+TL7/z+ey+++5dzkySNmmDNrRAhbO3Zeb7gfcDRMRBwHsz8/UR8QlgDnBK\n+f9lI52rr2/VmHKZPn0Ky5Y9MKZzjFXdOdQdvwk51B2/CTnUHb8JOdQd3xyaEb/uHJYvXzmqY6rO\nr+7fQ93xm5BD3fGbkEPd8c2hGfE7kcP06VOG3NfNdXrWNymdDPxVRCwAXljelyRJkqRKVL5OD0Bm\nXgNcU95eDhzSjbiSJElS061Zs4YFCxYM2wo8c+bO9Pb2djGrdulK0SNJkiRpcIsWLeSNZ13BpKkz\nBt2/um8J8445jFmzZnc5s/aw6JEkSZJqNmnqjCEnN9HYdXNMjyRJkiR1nUWPJEmSpFaze5skSeo6\nB25LGqjqzwSLHkmS1HUO3JY0UNWfCRY9kiSpFg7cljRQlZ8JjumRJEmS1GoWPZIkSZJazaJHkiRJ\nUqtZ9EiSJElqNYseSZIkSa1m0SNJkiSp1Vo/ZXUTFj9rQg6SJA3ktUnSeNL6oqcJi581IQdJkgby\n2iRpPGl90QPNWPysCTlIkjSQ1yZJ44VjeiRJkiS1mkWPJEmSpFYbF93bJEmS9HhOaKHxwqJHkiRp\nnHJCC40XFj2SJEnjmBNaaDxwTI8kSZKkVrOlR5IkjUuOZ5HGD4seSZI0LjmeRRo/LHokSdK45XgW\naXxwTI8kSZKkVrPokSRJktRqdm+TJElSLUYzmQQ4oYTGzqJHkiRJtRhpMglwQgl1xojd2yLi0tFs\nkyRJkp6o9ZNJDPVvuIJIGq3RjOkZrKzeo9OJSJIkSVIVhuzeFhFvBd4G7B4RNwzYtTWQVScmSZIk\nSZ0w3JieK4HfAP8OvBfoKbffD9xccV6SJEmS1BFDFj2ZuRBYCDy7e+lIkiRJUmeNOHtbRDwL+CAw\na8Dx/Zm5f5WJSZIkSVInjGbK6ouBS4FzgXXltv7KMpIkSZKkDhpN0dOTmR+vPBNJGgdGsxCfi/BJ\nktRZoyl6rouIPTPTyQskaYxGWojPRfgkSeq80RQ9BwBHR0QCfyi3jTimJyK2AK4BJgG9wNcz8/iI\nmAZcAuwM3A4cmZkrnmT+krTJWb8QnyRJ6o7RFD3/9GROnJl/iIgXZOaqiJgIXBsRBwIvA67KzE9E\nxPuA48p/kiRJktRxIxY9mfn9J3vyzFxV3uwFJgB9FEXPQeX2+cD3seiRJEmSVJHRTFl9wyCbRzVl\ndURsBvyUYrrrszPzlxGxXWYuLQ9ZCmz3RBKWJEmSpCdiNN3bjh1wewvgKODu0Zw8Mx8BnhcRWwP/\nFREv2Gh/f0SMOP311KlbMXHihNGEfJy+vskjHjNt2mSmT5/ypM6/qeQAVH7+TSGHuuM3IYe64zch\nhzrjN+XzAMb376HuHJrwOqg7h7rjNyUHGN+fSaOJX3UO643n38N6bX4OnnD3toj4L+B/nkiQzLwv\nIr4F7AMsjYgZmbkkIrYH7hnp5/v6Vo10yJCGmxZ24DHLlj3wpGNsCjlMnz6l0vNvCjnUHb8JOdQd\nvwk51B2/CZ8HUP/zUHf8unNowuug7hzqjt+UHOp+L9T9HIwmftU5gL8HaMdzMFxBtNmTyGlrRtEl\nLSK2jYinlbe3BP4KuAm4HJhTHjYHuOxJ5CBJkiRJo/JEx/RsBuwGnD6Kc28PzC/H9WwGnJ+ZV0fE\nTcClEfFmyimrn3DWkiRJkjRKT3RMz1rgd5k54piezPwFsPcg25cDh4w6Q0mSJEkagxG7t5Vjev4H\nuJdiyukRx+BIkiRJUlOMWPRExL7Ab4GvUYy/+U1E7FN1YpIkSZLUCaOZyOBM4OjMnJ2Zs4E3l9sk\nSZIkqfFGM6bnKZl59fo7mfndiPhUhTlJHbdmzRoWLFgw7HSIM2fuTG9vb6tzkCQ1x2iuC+C1QeqE\n0RQ9qyLiBZn5PYCIOBh4sNKspA5btGghbzzrCiZNnTHo/tV9S5h3zGHMmjW71TlIkppjpOsCeG2Q\nOmU0Rc87ga9ExOryfi/wqupSkqoxaeoMttxmx3GfgySpObwuSN0xmqLnacB+wNPL+/cAz64sI0mS\nJEnqoNEUPacCe2XmUoCImECxOOleVSYmSZIkSZ0wmqKHzOwfcHtdRIxm1jdJUgM5qYYkabwZTdHz\nQEQckJk/BoiIA3AiA0naZDmphiRpvBlN0fMvwNci4lfl/T8CXlldSpKkqjl4WpI0noxY9GTmdRHx\nbOBPgX7gx5m5vPLMJEmSJKkDRjumZznwrYpzkSRJkqSOG1XRI0md4AB6NYWvRUl61Hj4TLTokdQ1\nDqBXU/halKRHjYfPRIseSV3lAHo1ha9FSXpU2z8TXW9HkiRJUqvZ0iONE+Ohv64kSdJgLHqkcWI8\n9NeVJEkajEWPNI60vb+uJEnSYBzTI0mSJKnVLHokSZIktZpFjyRJkqRWs+iRJEmS1GoWPZIkSZJa\nzaJHkiRJUqtZ9EiSJElqNYseSZIkSa1m0SNJkiSp1Sx6JEmSJLWaRY8kSZKkVptYdwKSpM5Zs2YN\nixYtHPaYO+4Yfr8kSZ020vWp6muTRY8ktciiRQu59fAXs9PECUMec8fqNfC6U7uYlSRpvBvp+lT1\ntcmiR5JaZqeJE9ht4uZD7l+0dm0Xs5EkqTDc9anqa5NjeiRJkiS1mkWPJEmSpFaze9s4sGbNGhYs\nWMDy5SuHPGbmzJ3p7e3tYlaSVJ26B8xKkpql0qInImYC5wFPB/qBczLzzIiYBlwC7AzcDhyZmSuq\nzGU8W7RoIW886womTZ0x6P7VfUuYd8xhzJo1u8uZSVI16h4wK0lqlqpbeh4G3p2ZP4uIycBPIuIq\n4E3AVZn5iYh4H3Bc+U8VmTR1Bltus2PdaUhS19Q5YFaS1CyVjunJzCWZ+bPy9krgFmAH4GXA/PKw\n+cArqsxDkiRJ0vjVtYkMImIXYC/gemC7zFxa7loKbNetPCRJkiSNL12ZyKDs2vYV4F2Z+UBEbNiX\nmf0R0d+NPCRJ1RtpEoGHH36YpUufwsqVa4Y8ZlOfXMWJFHwO1BwjvRYBtt76OV3KRnWpvOiJiM0p\nCp7zM/OycvPSiJiRmUsiYnvgnuHOMXXqVkwcZnXx4fT1TR7xmGnTJjN9+pQndf5NIYe64w/UjRiD\nacJzUHcOdcdvSg5Q3+sQqn8O+voms3TkwyrNYcGCBcNOInD96jV84fD3DDu5yuUnHMEOO+z+pOJD\nZ56HKp+D0UyksKm/H30OOhO/CTnUHX+sOYz4Wly7jmnX/Q+77/7kP3PGalP4PYw1ft2fy1XP3tYD\nfBH4VWZ+esCuy4E5wCnl/5cN8uMb9PWtetI5DDdN88Bjli174EnHaHoOdcdfb/r0KZXHGEoTnoO6\nc6g7flNyqPN1CNU/B6M5/2jPM5YcRppEYKTJVcb6OujE81D1c1Bl/NHoxmtxvD8HnYjfhBzqjj/W\nHEZ6La7X5mtDE+J343N5uIKo6paePwdeB/w8Im4qtx0PnAxcGhFvppyyuuI8JEmSJI1TlRY9mXkt\nQ0+WcEiVsdUsLpAqSZKkunRlIgPJBVIlSZJUF4sedY0LpEqSJKkOXVunR5IkSZLqYNEjSZIkqdU2\n+e5tLn4mSZIkaTibfNGzaNHCMS9+JkmSJKm9NvmiBxjz4meSJEmS2ssxPZIkSZJazaJHkiRJUqu1\nonubJEl6LCf6kaRHWfRIktRCTvQjSY+y6JEkqaWc6EeSCo7pkSRJktRqFj2SJEmSWs2iR5IkSVKr\nWfRIkiRJajWLHkmSJEmtZtEjSZIkqdWcslqSJElqMRcrtuiRJEmSWs3Fii16JEmSpNYb74sVO6ZH\nkiRJUqvZ0iNJkqRKOJZETWHRI0mSpEo4lkRNYdEjSZKkyoz3sSRqBsf0SJIkSWo1ix5JkiRJrWb3\nNkmtMdKAWYCtt35OrTk4aFfqHt+PGo2H+/u57bbbWL585ZDHzJy5M729vV3MSp1m0SOpNUYcMLt2\nHdOu+x+mTt2+vhwctCt1je9HjcaSdev4yHk3MmnqjEH3r+5bwrxjDmPWrNldzkydZNEjqVWGGzDb\nhBwctCt1l+9HjcakqTPYcpsd605DFXJMjyRJkqRWs+iRJEmS1GoWPZIkSZJazaJHkiRJUqtZ9EiS\nJElqNYseSZIkSa3mlNWSJElSRUazcLYL5VbPokeSJEmqyEiL5IIL5XZDpUVPRMwFDgfuycw/LrdN\nAy4BdgZuB47MzBVV5iFJkiTVZaSFs10ot3pVj+k5Fzh0o23HAVdl5u7A1eV9SZIkSapEpUVPZv4Q\n6Nto88uA+eXt+cArqsxBkiRJ0vhWx+xt22Xm0vL2UmC7GnKQJEmSNE7UOpFBZvZHRP9Ix02duhUT\nhxj81dc3maWD7hm9adMmM336lDGeZWh9fZNrzaHu+E3Ioe74Tcih7vjdyGG0nwdVP8Y6P5M6Eb8J\nOYz1tVh3DnXHb0IOdcdvSg7DGc1nYhNyqPt10IQc6o7fhBw29fdjHUXP0oiYkZlLImJ74J6RfqCv\nb9WQ+5YvXznmhJYvX8myZQ+M+TzDnb/OHOqO34Qc6o7fhBzqjt+NHEb7eVD3YxzNOap+Dpqew1hf\ni3XnUHf8JuRQd/ym5DDSuTeFHOp+HTQhh7rjNyGHTeH9OFxBVEf3tsuBOeXtOcBlNeQgSZIkaZyo\nesrqi4CDgG0jYhHwYeBk4NKIeDPllNVV5iBJkiRpfKu06MnMo4bYdUiVcSVJkuq2Zs0aFi1aOOT+\nO+4Yep+kzqp1IgNJkqS2WrRoIbce/mJ2GmIypjtWr4HXndrlrKTxyaJHkiSpIjtNnMBuEzcfdN+i\ntWu7nI00ftUxkYEkSZIkdY1FjyRJkqRWs+iRJEmS1GoWPZIkSZJazaJHkiRJUqtZ9EiSJElqNaes\nHqORFh4DFx+TmuLh/n5uu+02li9fOexxM2fuTG9vb5eykiRJVbPoGaORFh4DFx+TmmLJunV85Lwb\nmTR1xpDHrO5bwrxjDmPWrNldzEySJFXJoqcDhlt4DFx8TGqSSVNnsOU2O9adhiRJ6iLH9EiSJElq\nNYseSZIkSa1m9zZJHTPSxB5O6iFJkupg0SOpY0aa2MNJPSRJUh0seiR11HATeziphyRJqoNjeiRJ\nkiS1mkUY+POIAAAMtElEQVSPJEmSpFaz6JEkSZLUahY9kiRJklrNokeSJElSq1n0SJIkSWo1p6yW\nOsSFOSVJTeO1SSpY9Egd4sKckqSm8dokFSx6pA5yYU5JUtN4bZIc0yNJkiSp5Sx6JEmSJLWa3dta\nwEGKPgeSJEkamkVPCzhI0edAkiRJQ7PoaQkHKfocSJIkaXCO6ZEkSZLUarb0SC3huCZJkqTBWfRI\nLeG4JkmSpMFZ9Egt4rgmSZKkx3NMjyRJkqRWs+iRJEmS1Gp2b1NHOIhekiRJTVVb0RMRhwKfBiYA\nX8jMU+rKRWPnIHpJkiQ1VS1FT0RMAD4DHALcBdwQEZdn5i115KPOcBC9JEmSmqiuMT37A7/NzNsz\n82HgYuDlNeUiSZIkqcXqKnp2ABYNuH9nuU2SJEmSOqquMT39nTzZHWvXDbnv7nWPsLpvyZD7h9vX\nifhNyKEb8ZuQQ93xm5BD3fGbkMNY4jchBz+TfC12Kn4Tcqg7fhNy8DPJ52Ck+E3IYTy8H3v6+zta\nf4xKRBwAnJiZh5b3jwcecTIDSZIkSZ1WV0vPjcDsiNgFuBt4NXBUTblIkiRJarFaxvRk5lrgH4H/\nAn4FXOLMbZIkSZKqUEv3NkmSJEnqlrpmb5MkSZKkrrDokSRJktRqFj2SJEmSWq2u2dsqExF7AC/n\n0cVO7wQuH08TJZTPwTOA6zNz5YDth2bmd7qUw4HA8sz8VUQcDOwL3JSZV3cj/iD5nJeZb6gjdhn/\n+cD+wC8y88ouxDsAuCUz74uIrYDjgL2BXwIfz8z7upDDO4GvZeaiEQ+uJv4k4DXAXZn53xHxWuDP\nKCZPOSczH+5SHrOAVwI7Ao8ACVyYmfd3I74kSWrZRAYR8T6Kqa8vpih2AGZSTIl9SWaeVFduABHx\npsw8t+IY7wT+AbgF2At4V2ZeVu67KTP3qjJ+Geck4AXABOB7wF8A3wL+CvhGZp5acfxvUCyA2zNg\n8wuB7wL9mfmyKuOXOfxvZu5f3n4rxe/ka8CLgG9W/VqMiF8Bz83MtRHxH8CDwH8Ch5TbX1ll/DKH\n+4BVwK3AhcCXM3NZ1XEHxL+Q4jW4FbACmAx8leI5IDPndCGHdwEvAa4BDgduKnP5G+DvM/N7Vecg\nDSUinp6Z99SdR50iYpvM/H3deah7IuJpFF8EvgLYjuLvhXuAy4CTM3NFjekREVdk5mEVx9gaOJ7i\ny7hvZ+aFA/Z9NjP/vsr4ZZyZwAnAvcDJwKeA/Siuk++p4rOpbS09bwH+aONvcCPidIpvd2steoCP\nAJUWPcDbgH0yc2W5DtJ/RsQumfnpiuMO9HLguUAvsBTYsWxxOA24Hqi06KF4E/8K+ALFN+s9FC1N\np1Ucd6DNB9x+O/BXmblswHNQ9Wuxp5waHorXw97l7Wsj4uaKY6/3O2AfiiLjNcC/RsRPgIuAr2bm\nAxXH/+PM/OOImEixHtgzyiLwS8DPK4693luBPTNzXUR8ErgiMw+KiM8BlwPPqzqBJl/gu3FxL+PU\neoGv4+I+SA7TNtrUA/xvROwNkJnLK46/oadB+Zo8nbL1G3h3Zi6tMn4Z9xTgtPKzeF/gUuCRiOgF\n3pCZ3684/k3AV4CLMvPWKmMNk8N+wCeAuyjeE3Mpfg8LgLdl5k0Vx58CHAu8iuJL6TUUX4ydnZnz\nqow9wKXA1cDBwNLM7I+I7YE55b4XVZ3A+vfdIHoovrCu2rkUv/OvAEdHxKuA12bmH4A/7UJ8gHkU\n16HJwI/L+ydQ/A15NsVrpKPaVvSso+jWdvtG259R7qtcRPximN1P70IKPeu7tGXm7WXXsq9ExM48\ntuWjSmvKP7jXRsSt67tSZeZDEfFIF+LvC7wL+ABwbGbeFBF/yMxruhB7vQnlHxk9wIT1LRyZ+WBE\nrB3+RzvilxFxdGbOBW6OiP0y84aI2J3iItMVmfkIcCVwZfmHxWEUrbGnA9tWHH6zsovbVsCWwNbA\n74Et6N54xn6KAnhdGfcpAJl5R0RsPtwPdlCtF/gGXNyh/gv8PLp8cR/EvcDCjbbtAPyE4nW6W8Xx\nTwLWd68+HVgMvJSi1fPzFEV51Q7PzPeVt08DXj3gc/Eiii9pqvS08t/3ImIpRQv4JZl5d8VxB/os\n8OEyj+uAd1N8Bryw3Ff1++ECil4PhwJHULwnLgY+GBG7Z+b7K44PsEtmnjJwQ2YuBk6OiKO7EB/g\nBuAHQ+zbugvxZw3o8fG1iPgAcHVEvLwLsdfbJjPPAoiId2TmyeX2syLizVUEbFvR80/Af0fEb4H1\n4whmArMpFkPthqdTvJn7Btn3oy7EvycinpeZPwMoW3xeAnyRovWlG1ZHxFaZuYpiHAmw4du9youe\nzFwHfDIiLgU+FRH30P3X+lMp/pgA6I+I7TNzcfktVze8BTgjIj4ILAN+FBF3Urwv3tKlHB4jM9cA\nXwe+HhFP6ULIL1F083wYeA/ww4j4EXAAML8L8aFobbwhIq4Hng+cAkW3IooCrBvqvsDXfXGH+i/w\nXb+4D+JYii7G/5KZPy9zuS0zd+1S/IH2BZ6Xmf0Un9Fv7FLcCRGxedkbZIvMvAEgMxeUX8pUbUVm\nvjcijqX4PDgK+GlE3ELR+nNOF3KYmJlXQNHylZlfLrdfXfaKqdouA7r5fzIibszMj5SvgVuAbhQ9\nCyPiX4D561sYI2IGxRdBd3QhPsCvgbdn5oKNd0REN8bB9kbEZuUXk2TmxyLiLoqu2JO7EB8e+0X8\n+Rvtm1BFwFYVPZn5nYgIiqbaHSi+vboLuHFAV5+qfQuYPFgTcUR0o6XhDRR/5G2QmQ9HxBygGx+o\nAAeV36Cu/6Z/vYkUHypdkZl3AkeURV/lA/c3ir3LELvWUXyzWXX8FcCcslvPrhTP/Z2ZuaTq2AO8\nZqgdmflg1cEz86SIuAi4PzOXR8TVFH9snZWZXenil5lnlHGfRdGt5tfl9nsoxrp1Q90X+Lov7lD/\nBb7rF/eNZebp5RdBnyy/ADmhG3EHmB4R/0zxXGxc7HarF8JngW+X406/ExFnUIzzeyHwsy7lQFns\n/QD4QUQcQ9EF+NV05xr9cES8mOJ30BMRf5OZX4uIg4DVXYj/YEQ8PzN/WH7p8Hso/lYo/nzrildT\ndPm9JiK2K7ctpehyfGSXcjiRoXscHNOF+N8E/hK4av2GzJwXEUuAs7oQH+DyiJiSmQ9k5gfWb4yI\n2RQT/nRcqyYykCQ9VtnN8jjgZRRjeuDRC/zJXRjLcQTFrIW/HmTfK9ZPtFJxDqcCV2bmVRttP5Si\nCJ5dcfyPAp/YeBxbeXE/KTP/tsr4g+Tzcopv1HfJzO1GOr5DMU+k+CJyvbMz856yq+Up2aXZNSPi\nBcA7KHqATKSY9OgyYO7G44EriH1xZg75ZVA3RMT+FGN6FlN8LnyR4ovi31KM6bmx4vh7UrSAz6aY\nTfTozMyImA4clZlnVhl/QB57UHw5fv3A92V0d5bboWbaPWx9a1yb49eRg0WPJI1T0YUZJUeIv37c\nWW3qzqGu+FFMZT8rM3/h66AR74Va4zchh269DqIZs9zWmkPd8evKwcVJJWn8+kjN8f+15vhQfw61\nxM/MVZm5fuIdXwf1Pwd1x4f6c+jW62D9LLevAA4CPhQR/9Sl2EPl8MEu51B3/FpyaNWYHknSY8Xw\nM0pW3rWp7vhNyKHu+KPIofKZRX0O6o8/ihzGxXuBx89yexDdn+W27pl2645fSw4WPZLUbnXPKFl3\n/CbkUHf8JuRQd/wm5FB3/CbkUHd8aMYst3XnUHf8WnKw6JGkdqt7Rsm64zchh7rjNyGHuuM3IYe6\n4zchh7rjQzNmua07h7rj15KDExlIkiRJajUnMpAkSZLUahY9kiRJklrNokeSJElSq1n0SJJqFxGP\nlAtmSpLUcRY9kqSmqGx9iIjweidJ45hTVkuSmuKdEfE3wDbAsZn5VYCIOBT4ODABWAa8PTNvjYg3\nAodn5hHlcRvul7dfB9wPzAZeC/y8uw9HktQUfvMlSWqK+zJzf+D1wJkAEfF04Dzg7zJzT+BC4IJR\nnu9PgPdk5h9npgWPJI1jFj2SpKa4uPz/euAZEdFLUbjcnJm/LvfNA54XEU8ZxfmuzczbOp+mJGlT\nY9EjSWqKPwBk5rry/khdsNfy2OvYFhvtX9mhvCRJmziLHklSk/0Y2DMiorw/B/hpZj4I/BZ4bkT0\nlq1Cf1tXkpKkZrPokSQ1Qf9g9zNzGcUYnwsj4mbg7ygmKCAzfwz8N/BL4CrgVwPO0z/IOSVJ41RP\nf7/XBEmSJEntZUuPJEmSpFaz6JEkSZLUahY9kiRJklrNokeSJElSq1n0SJIkSWo1ix5JkiRJrWbR\nI0mSJKnVLHokSZIktdr/B0hLtjUQfVBxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f4a4fc59d90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# spark dataframe轉成pandas dataframe\n",
    "pandas_df = df_hour.toPandas() \n",
    "# 轉一下格式方便畫圖\n",
    "pandas_df = pandas_df.pivot_table(index='hour',columns='spam',values='count') \n",
    "pandas_df.set_axis(1,['not spam','spam'])\n",
    "# 使用pandas畫圖\n",
    "ax=pandas_df.plot(kind='bar') \n",
    "ax.set_title('count by hour')\n",
    "ax.set_ylabel('count')\n",
    "ax.set_xlabel('hour')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#使用ML Pipeline進行spam預測\n",
    "步驟：\n",
    "1. 前處理。  \n",
    "3. 建立pipeline。\n",
    "4. 分成訓練和測試資料。\n",
    "5. 使用訓練資料訓練pipeline。\n",
    "7. 使用測試資料評估性能。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###前處理 \n",
    "特徵抽取，加入時間特徵，利用上述dataframe udf加入hour特徵。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 加入時間特徵\n",
    "df_ml = df.withColumn('hour',hourOfDay(df['post_at']).astype(DoubleType())) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###建立 Pipeline\n",
    "本demo目的為Pipeline的使用，為了簡單起見，本demo只挑選幾個簡單的特徵：\n",
    "- 時間\"hour\", 時間特徵使用上述的發文小時時間, 在前處理步驟已經加入。\n",
    "- 點擊數\"hits\",\n",
    "- 留言數量'comment_count'。  \n",
    "\n",
    "我們使用線性分類器logistic regression，這種分類器只接受數值特徵，類別特徵必須先經過編碼轉換才能通過分類器。留言數和點擊數本身是數值特徵， 小時時間類別是類別特徵需要經過編碼轉成數值特徵。\n",
    "\n",
    "Pipeline流程圖：\n",
    "```\n",
    "發文時間 ---- 類別數值化 ------------\\  \n",
    "                                   \\  \n",
    "點擊數量 -------------------------- 結合 ---> 分類器  \n",
    "                                   /  \n",
    "留言數量 --------------------------/  \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 發文時間編碼\n",
    "hour_encoder = OneHotEncoder(inputCol=\"hour\", outputCol=\"hour_code\")  \n",
    "\n",
    "# 結合所有特徵\n",
    "assembler = VectorAssembler(inputCols=[\"hour_code\",\"hits\",'comment_count'], outputCol=\"features\")\n",
    "\n",
    "# 分類器\n",
    "log_regressor = LogisticRegression(featuresCol=\"features\",labelCol=\"spam\")\n",
    "\n",
    "#機器學習管線\n",
    "pipeline = Pipeline(stages=[hour_encoder,assembler,log_regressor])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###把資料分成訓練和測試資料\n",
    "在比例上，訓練資料佔七成，測試資料佔三成"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_train,df_test=df_ml.randomSplit([7.,3.],123)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###訓練pipeline\n",
    "pipeline是Estimator所以有fit方法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = pipeline.fit(df_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###評估pipeline\n",
    "1. 用訓練好的pipeline進行預測操是資料集。\n",
    "3. 使用DataFrame計算預測精準度。\n",
    "3. 利用dataframe觀察資料以利檢討模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 用訓練好的pipeline進行預測測試資料集。\n",
    "# 訓練好的pipeline是Transformer所以有transform方法。\n",
    "df_pred = model.transform(df_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- category: string (nullable = true)\n",
      " |-- comment_count: long (nullable = true)\n",
      " |-- comment_ids: array (nullable = true)\n",
      " |    |-- element: string (containsNull = true)\n",
      " |-- hits: long (nullable = true)\n",
      " |-- tags: array (nullable = true)\n",
      " |    |-- element: string (containsNull = true)\n",
      " |-- title: string (nullable = true)\n",
      " |-- spam: double (nullable = true)\n",
      " |-- post_at: long (nullable = true)\n",
      " |-- hour: double (nullable = true)\n",
      " |-- hour_code: vector (nullable = true)\n",
      " |-- features: vector (nullable = true)\n",
      " |-- rawPrediction: vector (nullable = true)\n",
      " |-- probability: vector (nullable = true)\n",
      " |-- prediction: double (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 觀察transofrm對dataframe做了什麼事，所有pipeline設定的特徵和預測都加入dataframe了。\n",
    "df_pred.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+\n",
      "|          accuracy|\n",
      "+------------------+\n",
      "|0.7128378378378378|\n",
      "+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "pred_label = df_pred.select(\"prediction\", \"spam\")\n",
    "# 建立accuracy function，使用內建dataframe functions達成UDAF\n",
    "accuracy = avg((pred_label.prediction==pred_label['spam']).astype(IntegerType())).alias(\"accuracy\") \n",
    "# 需要groupBy才能使用此function, 所以就group所有data吧\n",
    "pred_label.groupBy().agg(accuracy).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----+----+-------------+----+----------+\n",
      "|hour|hits|comment_count|spam|prediction|\n",
      "+----+----+-------------+----+----------+\n",
      "|13.0| 189|            0| 0.0|       1.0|\n",
      "|12.0|  34|            0| 0.0|       1.0|\n",
      "|20.0|   0|            0| 1.0|       0.0|\n",
      "|20.0|   0|            0| 1.0|       0.0|\n",
      "|20.0|   0|            0| 1.0|       0.0|\n",
      "|20.0|   0|            0| 1.0|       0.0|\n",
      "|15.0|   0|            0| 0.0|       1.0|\n",
      "|10.0|2012|            0| 0.0|       1.0|\n",
      "|15.0|3652|            4| 0.0|       1.0|\n",
      "|13.0|   0|            0| 0.0|       1.0|\n",
      "+----+----+-------------+----+----------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 檢討模型效能，觀察什麼樣的特徵導致錯誤預測。\n",
    "df_pred.filter(df_pred['spam']!=df_pred['prediction'])\\\n",
    "        .select('hour','hits','comment_count','spam','prediction')\\\n",
    "        .show(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "author: Wayne-Lin \n",
    "#=====使用完請隨手關燈sc.stop(),感謝您======"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sc.stop()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}