class08_Logistic-Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Data Mining:<br>Statistical Modeling and Learning from Data\n",
    "\n",
    "## Dr. Ciro Cattuto<br>Dr. Laetitia Gauvin<br>Dr. André Panisson\n",
    "\n",
    "### Exercises - Logistic Regression\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pylab inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating data\n",
    "\n",
    "In the following, we generated 50 samples from a bivariate Gaussian distribution $\\mathcal{ N } ((2, 0)^T , I)$ and labeled this class\n",
    "**RED**. Similarly, 50 more were drawn from $ \\mathcal{N} ((0, 2)^T , I)$ and labeled class **GREEN**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(2)\n",
    "nr_samples = 100\n",
    "\n",
    "# Generate 10 means from two bivariate Gaussian distributions\n",
    "samples_red   = np.random.multivariate_normal(mean=(0,2), cov=np.identity(2), size=nr_samples//2)\n",
    "samples_green = np.random.multivariate_normal(mean=(2,0), cov=np.identity(2), size=nr_samples//2)\n",
    "\n",
    "# Join the red and green datasets as X and the class definitions as y\n",
    "X = np.concatenate([samples_red, samples_green])\n",
    "y = np.zeros(nr_samples, dtype=int)\n",
    "y[nr_samples//2:] = 1\n",
    "\n",
    "# plot the red and green class points\n",
    "figure(num=None, figsize=(5, 5))\n",
    "plot(samples_red[:,0], samples_red[:,1], 'o', mec='r', mfc='none')\n",
    "plot(samples_green[:,0], samples_green[:,1], 'o', mec='g', mfc='none')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1: Logistic Regression\n",
    "\n",
    "Find the weight values $\\mathbf{w}$ that minimize the error \n",
    "\n",
    "$$E_{\\mathbf{in}}(\\mathbf{w}) = \\frac{1}{N} \\sum_{n=1}^N \n",
    "{ln(1 + e^{-\\mathbf{y}_n\\mathbf{w}^T\\mathbf{x}_n})}$$\n",
    "\n",
    "For this, implement the Gradient Descent algorithm with $\\mathbf{s}$ learning steps and learning rate $\\alpha$.  \n",
    "At each training step, update $\\mathbf{w}$ with this rule:\n",
    "\n",
    "$$\\mathbf{w}_i := \\mathbf{w}_i - \\alpha \\Delta E_{\\mathbf{in}} $$\n",
    "\n",
    "where\n",
    "\n",
    "$$\\Delta E_{\\mathbf{in}} = - \\frac{1}{N} \\sum_{n=1}^N \n",
    "{\\frac{\\mathbf{y}_n\\mathbf{x}_n}{1 + e^{\\mathbf{y}_n\\mathbf{w}^T\\mathbf{x}_n}}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare the dataset for the matrix manipulations\n",
    "Xext = np.insert(X, 0, ones(len(X)), axis=1)\n",
    "y[y==0] = -1\n",
    "\n",
    "N = Xext.shape[0] \n",
    "d = Xext.shape[1]\n",
    "s = 2000 # learning steps\n",
    "alpha = 0.2 # learning rate\n",
    "\n",
    "w = zeros(d)\n",
    "\n",
    "# YOUR CODE HERE\n",
    "for step in range(s):\n",
    "    grad = zeros(d)\n",
    "    \n",
    "\n",
    "    for n in range(N):\n",
    "        grad = grad + (y[n]*Xext[n])/ (1 + np.exp(y[n]*w.T.dot(Xext[n])))\n",
    "   \n",
    "    w = w - alpha * (-grad/N)\n",
    "    \n",
    "    #posso verificare la  norma ad ogni passo così da avere una soglia ben definita\n",
    "    if np.linalg.norm(grad) < 1e-4:\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grad #mostra il risultato del gradiente più il gradiente è piccolo più sono prossimo alla soluzione\n",
    "#alpha troppo grande da' salti troppo grandi da una parte all'altra e#prova su python 2 ! non riesce a minimizzare se metto una alpha troppo bassa non riesce a scendere abbastanza in fretta e quindi non riesce ad arrivare al minimo\n",
    "# se voglio la norma np.linalg.norm(grad) calcola così la norma l2 ovvero la frobenium norm\n",
    "#se metto step mi dice a quale passo ho ottenuto la convergenza"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we visualize the area which is classified as RED and GREEN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the logistic function\n",
    "def l(x):\n",
    "    return 1. / (1. + exp(-x))\n",
    "\n",
    "# define a colormap to colorize the class areas\n",
    "cmap = matplotlib.colors.LinearSegmentedColormap.from_list('colormap',['#FFC0C0', '#C0FFC0'])\n",
    "\n",
    "# create a mesh of points that cover the full graph area\n",
    "x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5\n",
    "y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5\n",
    "h = .02  # step size in the mesh\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n",
    "            np.arange(y_min, y_max, h))\n",
    "\n",
    "# use the classifier to predict the class of each mesh point\n",
    "Xtest = np.c_[xx.ravel(), yy.ravel()]\n",
    "Xext = np.insert(Xtest, 0, ones(len(Xtest)), axis=1)\n",
    "Z = l(Xext.dot(w))\n",
    "Z[Z>0.5] = 1\n",
    "Z[Z<=0.5] = 0\n",
    "\n",
    "# colorize the graph using the class of each mesh point\n",
    "Z = Z.reshape(xx.shape)\n",
    "figure(figsize=(5, 5))\n",
    "pcolormesh(xx, yy, Z, cmap=cmap)\n",
    "\n",
    "# plot the red and green class points\n",
    "plot(samples_red[:,0], samples_red[:,1], 'o', mec='r', mfc='none')\n",
    "plot(samples_green[:,0], samples_green[:,1], 'o', mec='g', mfc='none')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2: Logistic Regression with Scikit-Learn\n",
    "\n",
    "Find the values for the parameters $\\mathbf{w}$ using the model sklearn.linear_model.LogisticRegression  \n",
    "http://scikit-learn.org/dev/modules/generated/sklearn.linear_model.LogisticRegression.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "model = LogisticRegression()\n",
    "model.fit(X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we visualize the area which is classified as RED and GREEN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define a colormap to colorize the class areas\n",
    "cmap = matplotlib.colors.LinearSegmentedColormap.from_list('colormap',['#FFC0C0', '#C0FFC0'])\n",
    "\n",
    "# create a mesh of points that cover the full graph area\n",
    "x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5\n",
    "y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5\n",
    "h = .02  # step size in the mesh\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n",
    "            np.arange(y_min, y_max, h))\n",
    "\n",
    "# use the classifier to predict the class of each mesh point\n",
    "Xtest = np.c_[xx.ravel(), yy.ravel()]\n",
    "Z = model.predict(Xtest)\n",
    "\n",
    "# colorize the graph using the class of each mesh point\n",
    "Z = Z.reshape(xx.shape)\n",
    "figure(figsize=(5, 5))\n",
    "pcolormesh(xx, yy, Z, cmap=cmap)\n",
    "\n",
    "# plot the red and green class points\n",
    "plot(samples_red[:,0], samples_red[:,1], 'o', mec='r', mfc='none')\n",
    "plot(samples_green[:,0], samples_green[:,1], 'o', mec='g', mfc='none')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}