Machine_Learning_Algorithms_Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning Algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3 Types of ML Algorithms<br>\n",
    "1. Supervised Learning - consisits of target variable and predictors. Regression and Classification. Models:Regression, KNN, Decision Tree, Random Forest, Logistics Regression<br>\n",
    "2. Unsupervised Learning - Do not have any outcome variables to predict. Clustering. Segement customer, or picture. Models: Apriori, K-means<br>\n",
    "3. Reinforcement learning - The machine is trained to make specific decisions. Markov Decision Process"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Training data: data used to fit the model\n",
    "- Test data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split data into training and test data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "#train_X, test_X, train_y, test_y = train_test_split(X, y,random_state = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Minimize the sum of squared difference of distince between obersed value and estimated value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import linear_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Identify feature and response variable(s) and values must be numeric and numpy arrays<br>\n",
    " Load train and test data sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# instantiate a model--make an instance\n",
    "lr = linear_model.LinearRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train the model with trainning set and make prediction\n",
    "\n",
    "# lr.fit(X_train, y_train)\n",
    "# y_pred = lr.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Equation coefficient and Intercept\n",
    "\n",
    "# print('Coefficient: \\n', linear.coef_)\n",
    "# print('Intercept: \\n', linear.intercept_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Classification: used to estimate discrete values---Binary values like 0/1, yes/no, true/false"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit data to a logit function, and predicts probability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# instantiate a model--make an instance\n",
    "logreg = LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# logreg.fit(X_train, y_train)\n",
    "# y_pred = logreg.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Equation coefficient and Intercept\n",
    "# print('Coefficient: \\n', logreg.coef_)\n",
    "# print('Intercept: \\n', logreg.intercept_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Decision Tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split the population into two or more homogeneous sets. This is done based on most significant attributes/ independent variables to make as distinct groups as possible. <br>\n",
    "Tree's depth - how many splits it makes before coming to a prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For Classification\n",
    "# lgorithm default is gini, others - entropy\n",
    "dt = tree.DecisionTreeClassifier(criterion = 'gini')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For Regression\n",
    "dt = tree.DecisionTreeRegressor()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## KNN - K-nearest Neighbors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both Classification(more widely used) and Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- KNN is computationaly expensive<br>\n",
    "- Variables should be normalized, else higher range varibles can bias it<br>\n",
    "- Remove outlier, noise before doing KNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# default value of n_neighbors is 5\n",
    "knn = KNeighborsClassifier(n_neighbors=6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random Forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Random Forest, there are a collection of decision trees. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf = RandomForestClassifier()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor\n",
    "rf = RandomForestRegressor()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "if 0:\n",
    "    print ('Lin')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## K-Means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unsupervised learning - Clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How K-means forms cluster:<br>\n",
    "1. K-means picks k number of points for each cluster known as centroids.<br>\n",
    "2. Each data point forms a cluster with the closest centroids i.e. k clusters.<br>\n",
    "3. Finds the centroid of each cluster based on existing cluster members. Here we have new centroids.<br>\n",
    "4. As we have new centroids, repeat step 2 and 3. Find the closest distance for each data point from new centroids and get associated with new k-clusters. Repeat this process until convergence occurs i.e. centroids does not change."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How to determine value of K:<br>\n",
    "Sum of square of difference between centroid and the data points --- the number of cluster increases, the value keeps decreasing. If draw a plot, sum of square distince decreases sharply up to some value of k. Here, we can find the optimum number of cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans = KMeans(n_clusters = 3, random_state = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# kmeans.fit(X_train, y_train)\n",
    "# kmeans.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- MAE(Mean Absolute Error): absolute difference between predicted and actual value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_absolute_error\n",
    "# mean_absolute_error(y,predicted_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cross Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By improving the accuracy score, we might get into the situation of over-fitting. Cross Validation helps to achieve more generalized relationships."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Method: k-fold cross validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Steps:<br>\n",
    "1. Randomly split data into k folds.<br>\n",
    "2. For each k folds, build and train the model on k-1 folds of the data set, and test the model on the kth fold.<br>\n",
    "3. Record error/accuracy.<br>\n",
    "4. Repeat until each of the k fold of data has served as test set.<br>\n",
    "5. The average of your k recorded errors is called the cross-validation error and will serve as your performance metric for the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How to choose value of K --- often use k = 10<br>\n",
    "Lower value of k is more biased.<br>\n",
    "Large value of k is less biased, but can suffer from large variability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import KFold\n",
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "kf = KFold(n_splits = 10, random_state=0)\n",
    "modelCV = RandomForestClassifier()\n",
    "scoring = \"accuracy\"\n",
    "# results = cross_val_score(modelCV, X_train, y_train, cv=kf,scoring = scoring)\n",
    "# print ('10-fold cross validation average accuracy: {}'.format(results.mean()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Underfitting, Overfitting and Model Optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a way to measure model accuracy, we can experiment with altenative models and see which gives the best predictions. (different options built with the model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overfitting: model matches with training data almost perfectly, but does poorly in validation and other new data.<br>\n",
    "Underfitting: Model fails to capture important distinctions and patterns in the data<br>\n",
    "eg. for the decision tree model, more max_leaf_nodes, the more move from underfitting overfitting."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}