Ukara_Baseline_Code_Published.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ukara 1.0 Challenge Track 1 (baseline model )\n",
    "**Python notebook ini menjelaskan detail baseline model UKARA. Notebook ini hanya menampilkan eksperimen dengan menggunakan data training saja untuk DATA A. _Baseline performance_ dihasilkan dengan menggunakan data test yang akan di _reveal_ pada tanggal 16 September 2019.** <br>\n",
    "\n",
    "Ada beberapa step yang dilakukan dalam eksperimen ini yaitu:\n",
    "\n",
    "1. Preparation\n",
    "2. Ekstraksi Fitur\n",
    "3. Learning\n",
    "4. Menyimpan hasil prediksi ke json file\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>RES_ID</th>\n",
       "      <th>RESPONSE</th>\n",
       "      <th>LABEL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>TRA1</td>\n",
       "      <td>intetraksi/beradaptasi terhadap lingkungan yan...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>TRA2</td>\n",
       "      <td>seperti jatuhnya meteor tsunami gempa bumi</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>TRA3</td>\n",
       "      <td>hanya tuhan yang tahu tantangan nya itu apaan</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>TRA4</td>\n",
       "      <td>mereka akan sulit beradaptasi</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>TRA5</td>\n",
       "      <td>Tempat tinggal, ekonomi, dan pekerjaan</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  RES_ID                                           RESPONSE  LABEL\n",
       "0   TRA1  intetraksi/beradaptasi terhadap lingkungan yan...      1\n",
       "1   TRA2         seperti jatuhnya meteor tsunami gempa bumi      0\n",
       "2   TRA3      hanya tuhan yang tahu tantangan nya itu apaan      0\n",
       "3   TRA4                      mereka akan sulit beradaptasi      1\n",
       "4   TRA5             Tempat tinggal, ekonomi, dan pekerjaan      1"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sys\n",
    "import numpy as np\n",
    "import csv\n",
    "import pandas as pd\n",
    "from pandas import DataFrame\n",
    "from sklearn.model_selection import train_test_split\n",
    "np.random.seed(42)\n",
    "\n",
    "dataA = \"/home/yunita/Data/Project/essay_scoring/Tahap_2_2019/EvalAI/Data_A/data_train_A.csv\" # load data\n",
    "dataA_df = pd.read_csv(dataA, header=0)\n",
    "dataA_df.head() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "268"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataA_df.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Karena pada eksperimen ini hanya menggunakan data training, maka data dibagi menjadi 2 part yaitu training dan test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "trainingSet, testSet = train_test_split(dataA_df, test_size=0.2, stratify=dataA_df['LABEL'], random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Preparation\n",
    "Dalam tahap preparation, kita perlu mempersiapkan kamus *stopwords* terlebih dahulu.\n",
    "* Stopwords merupakan kata-kata yang biasanya tidak signifikan secara makna (non-content words) (i.e. yang, dari, ke, dia, kamu, etc). File **stopwords.txt** berisi 763 _predefined stopwords_ dalam bahasa Indonesia.\n",
    "\n",
    "**Anda bisa mendefiniskan sendiri kamus stopword yang akan digunakan, atau memilih tidak membuang stopword**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Jumlah stop words: 763\n"
     ]
    }
   ],
   "source": [
    "stopw = set([w for w in open('/home/yunita/Data/Code/essay_scoring/stopwords.txt','r')])\n",
    "print (\"Jumlah stop words:\", len(stopw))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Ekstraksi Fitur\n",
    "Tahap berikutnya adalah ekstraksi fitur. Pada _baseline model_, digunakan **Bag of words** sebagai fitur. Untuk ekstraksi fitur, digunakan salah satu python library yaitu Scikit Learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "count_vect = CountVectorizer(analyzer=\"word\", ngram_range=(1,1), stop_words=stopw, lowercase=True)\n",
    "\n",
    "train_vect = count_vect.fit_transform(trainingSet['RESPONSE'].astype(str)) #fit transform pada data train A\n",
    "test_vect = count_vect.transform(testSet['RESPONSE'].astype(str)) #transform data test \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Learning dan Evaluasi\n",
    "Tahap selanjutnya adalah learning dengan mengimplementasikan algoritma machine learning yaitu AdaBoost Classifier. Evaluasi dengan metric _accuracy_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import precision_score\n",
    "from sklearn.metrics import recall_score\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "\n",
    "AdaClassifier = AdaBoostClassifier() #menggunakan default parameter\n",
    "AdaClassifier.fit(train_vect , trainingSet['LABEL']) #fit pada data train\n",
    "Predict = AdaClassifier.predict(test_vect) #memprediksi label pada data test\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy pada test data: 0.759259259259\n"
     ]
    }
   ],
   "source": [
    "print (\"Accuracy pada test data:\", accuracy_score(testSet['LABEL'], Predict))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision baseline pada test data: 0.878787878788\n"
     ]
    }
   ],
   "source": [
    "print (\"Precision baseline pada test data:\", precision_score(testSet['LABEL'], Predict))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recall baseline pada test data: 0.763157894737\n"
     ]
    }
   ],
   "source": [
    "print (\"Recall baseline pada test data:\", recall_score(testSet['LABEL'], Predict))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1 baseline pada test data: 0.816901408451\n"
     ]
    }
   ],
   "source": [
    "print (\"F1 baseline pada test data:\", f1_score(testSet['LABEL'], Predict))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TPR baseline: 0.763157894737 dan FRP baseline: 0.25\n"
     ]
    }
   ],
   "source": [
    "tn_dev, fp_dev, fn_dev, tp_dev = confusion_matrix(testSet['LABEL'], Predict).ravel()\n",
    "TPR_dev = tp_dev/(tp_dev+fn_dev)\n",
    "FPR_dev = fp_dev/(fp_dev+tn_dev)\n",
    "print(\"TPR baseline:\", TPR_dev, \"dan FRP baseline:\", FPR_dev)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Menyimpan hasil prediksi ke json file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pandas import DataFrame\n",
    "\n",
    "prediksi_dataA = {'RES_ID': testSet['RES_ID'],\n",
    "                'LABEL': Predict\n",
    "                }\n",
    "df_dataA = DataFrame(prediksi_dataA, columns= ['RES_ID', 'LABEL'])\n",
    "df_dataA.to_json ('/home/yunita/Data/Code/essay_scoring/train/predictions_test.json', orient='records')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}