exercises/exercise8/08-exercise-pids2023.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "93119d59",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize Otter\n",
    "import otter\n",
    "grader = otter.Notebook(\"08-exercise-pids2023.ipynb\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "24138852",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-525c2fd5f3e93d2b",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# Exercise sheet 8\n",
    "**Hello everyone!**\n",
    "\n",
    "**Points: 15**\n",
    "\n",
    "Topics of this exercise sheet are:\n",
    "* Central Limit Theorem\n",
    "* Confidence Interval\n",
    "\n",
    "\n",
    "Please let us know if you have questions or problems! <br>\n",
    "Contact us during the exercise session, on [Piazza](https://piazza.com/class/leibdr4lk8n3w4), or [via email](https://sada.dmi.unibas.ch/en/teaching/pids23).\n",
    "\n",
    "Please submit this exercise sheet on **GRADESCOPE**.\n",
    "Naming conventions:\n",
    "* Name of notebook: \"08-exercise-pids2023.ipynb\"\n",
    "\n",
    "Your notebook will be automatically graded using gradescope. To get graded, you need to upload the file *08-exercise-pids2023.ipynb* to gradescope. If you are using JupyterHub, make sure you have saved the current notebook, then you can download this file by following: *File*->*Download* and save the file on you computer.\n",
    "\n",
    "The sheet is released before the exercise session on Tuesday. You have until the next exercise session on Tuesday to submit this sheet. Most of the questions of the exercise sheet can be handle during the session.\n",
    "\n",
    "**Handout date**: 25.04.2023 <br>\n",
    "**Submission date**: 02.05.2023 before 16:00 <br>\n",
    "\n",
    "\n",
    "### Please name the variables as suggested! Be careful with the type! Otherwise the grading system will fail you!\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "27cd51cd",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-41524afeb7ebe206",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from scipy import stats\n",
    "from nose.tools import assert_is_instance, assert_equal, assert_almost_equal, assert_true"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b3af4bc",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-53460d34058f3491",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# 1 (8 points)\n",
    "We define a random variable $X_\\text{mean}$ as follows:\n",
    "$$\n",
    "X_\\text{mean} = \\dfrac{1}{N} \\sum_{i=1}^N X_i \n",
    "$$\n",
    "Where $X_i \\sim$ Binomial($n,p$). As discussed in the last lecture, we can analytically compute the mean and variance of this random variable:\n",
    "$$\n",
    "\\mathbb{E}[X_\\text{mean}] = \\dfrac{1}{N} \\sum_{i=1}^N \\mathbb{E}[X_i] = np\n",
    "$$\n",
    "$$\n",
    "\\text{Var}[X_\\text{mean}] = \\dfrac{1}{N^2} \\sum_{i=1}^N \\text{Var}[X_i] = \\dfrac{np(1-p)}{N}\n",
    "$$\n",
    "In this exercise, we want to verify the analytical results experimentally.\n",
    "As the first step, let's define a Binomial random variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "180711ec",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-de32fce7c84d1792",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "n = 20 # The total number of trials\n",
    "p = 0.5 # The probability of success in each trial\n",
    "X = stats.binom(n, p)\n",
    "# Please check this page for more information: \n",
    "# https://docs.scipy.org/doc/scipy-0.13.0/reference/generated/scipy.stats.binom.html#scipy.stats.binom"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c4edc684",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-5d9cbb87ed2f5e87",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### 1a (2 points)\n",
    "Generate a matrix with size $m \\times N$ containing samples from the given Binomial distribution and store the matrix in variable 'x'. Then calculate the mean of each row of matrix 'x' and and store the row-wise means in a vector with size $m$ and call it 'mean'. Plot the histogram of the calculated 'mean' vector.\n",
    "\n",
    "**Important**: Please do not make a new cell for the plot. Plot the histogram in the same cell.\n",
    "\n",
    "**Important**: Please do not use a \"for-loop\" for sample generation; X.rvs can take the shape of the matrix and generate it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "083c6133",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-cda45808f0efe171",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "m = 100\n",
    "N = 10\n",
    "np.random.seed(0)\n",
    "..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df3562ac",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"Question 1a\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86e155b6",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-6fcab90fcffa4ae2",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### 1b (4 points)\n",
    "Now we want to repeat this process for $N=\\{1,2,...,k\\}$ instead of a single $N=10$. So write a code to generate the mean for each $N$ (as you did for $N=10$) and store all the calculated means in \"means\" which would be a matrix with size $k \\times m$.\n",
    "\n",
    "**Important**: Please do not use a \"for-loop\" for sample generation; X.rvs can take the shape of the matrix and generate it. You can use \"for-loop\" anywhere else."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1de80838",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-0f3266550e4db393",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "m = 100\n",
    "k = 50\n",
    "np.random.seed(0)\n",
    "means = np.zeros([k,m])\n",
    "..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bafc9a22",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"Question 1b\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "51c3efe0",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-0682fdc8882d815c",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### 1c (2 points)\n",
    "Now calculate the variance of the mean for a given N. This means that you need to calculate the variance of each row of the matrix \"means\" in 1b. Store the variances in a vector with size k and call this vector \"variences\". Plot the vector \"variences\" with respect to $N=\\{1,2,...,k\\}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "936f43b5",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-183fea12257ba84f",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b21aa8c3",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"Question 1c\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c183c18",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-3b593abf923267bc",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "Does the variance decrease as we increase the number of samples ($N$)? Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2021f50",
   "metadata": {
    "nbgrader": {
     "grade": true,
     "grade_id": "cell-dfab7379a295c647",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "29728b10",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-bf218f2a711395fc",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "# 2 (7 points)\n",
    "## Confidence Interval\n",
    "In this exercise, we work on confidence interval calculation using bottstrap technique.\n",
    "Let's generate $N$ samples from Gaussian distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4f28ee60",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-feab5992c895e9e3",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "mu = 0.0\n",
    "sigma = 1.0\n",
    "N = 400\n",
    "\n",
    "np.random.seed(0)\n",
    "X = stats.norm(loc = mu , scale = sigma)\n",
    "x = X.rvs(N)\n",
    "sns.histplot(x)\n",
    "\n",
    "# Please check this page for more information: \n",
    "# https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6ead6342",
   "metadata": {},
   "source": [
    "### 2a (3 points)\n",
    "\n",
    "As discussed in the previous lecture, we can use the bootstrap technique to find the confidence interval. In this exersice, we are going to estimate the confidence interval using bootstrap technique. For this we use the bootstrap function from the scipy.stats module.\n",
    "\n",
    "Given $N$ samples and a confidence level of 0.95, the scipy bootstrap function does the following steps to calculate the confidence interval:\n",
    "1. Randomly select $N$ samples with replacement from the given samples. Sampling with replacement allows us to generate different batches of samples with size $N$ from a single set.\n",
    "2. Calculate the mean of the $N$ selected samples.\n",
    "3. Repeat steps 1 and 2 for a total of $k$ times and store the means in a vector.\n",
    "4. Calculate the 2.5th and 97.5th percentiles of the means vector and store them in the 'CI' variable.\n",
    "\n",
    "This function calculates the confidence interval of a given statistic, such as mean or median. Let's use the scipy bootstrap function to estimate the confidence interval and store the function output in 'bootstrap_ci' variable. Note that we are estimating the 'mean' and using the 'percentile' method.\n",
    "Please visit [this](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html) website for further information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b0963460",
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import bootstrap\n",
    "import numpy as np\n",
    "\n",
    "k = 50\n",
    "confidence_level = 0.95\n",
    "np.random.seed(0)\n",
    "\n",
    "...\n",
    "\n",
    "CI_scipy = [bootstrap_ci.confidence_interval[0], bootstrap_ci.confidence_interval[1]]\n",
    "print(\"The {} confidence interval for N={} is: [{:.3f}, {:.3f}].\".format(confidence_level, N,CI_scipy[0], CI_scipy[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8efd898",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"Question 2a\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9bf4a0d0",
   "metadata": {},
   "source": [
    "### 2b (1 points)\n",
    "Compute the confidence interval for $N = 4000$ (instead of 400) and store the function output in 'bootstrap_ci_4000'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "5adefc2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.stats import bootstrap\n",
    "import numpy as np\n",
    "\n",
    "mu = 0.0\n",
    "sigma = 1.0\n",
    "N = 4000 # previously 400\n",
    "\n",
    "np.random.seed(0)\n",
    "X = stats.norm(loc = mu , scale = sigma)\n",
    "x = X.rvs(N)\n",
    "sns.histplot(x)\n",
    "\n",
    "k = 50\n",
    "confidence_level = 0.95\n",
    "\n",
    "...\n",
    "\n",
    "CI_scipy_4000 = [bootstrap_ci_4000.confidence_interval[0], bootstrap_ci_4000.confidence_interval[1]]\n",
    "print(\"The {} confidence interval for N={} is: [{:.3f}, {:.3f}].\".format(confidence_level,\n",
    "                                                                         N,CI_scipy_4000[0],\n",
    "                                                                         CI_scipy_4000[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a80c8c1d",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"Question 2b\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "55dbf3a0",
   "metadata": {},
   "source": [
    "Does the confidencence interval shrinke by increasing the number of samples ($N$)? Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99e72b6c",
   "metadata": {},
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c6979f11",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-a7768c9418cb5405",
     "locked": true,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "source": [
    "### 2c (3 points)\n",
    "\n",
    "In this exercise, we will implement the bootstrap technique from scratch to estimate the confidence interval of a given set of samples. Please implement the steps explained in question 2a. This will allow us to compare our implementation with the one provided by scipy.stats module. Please store the estimated confidence interval in the 'CI' variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "a27e8afe",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-5b337223591e2e4e",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "N = 4000 # previously 400\n",
    "\n",
    "np.random.seed(0)\n",
    "X = stats.norm(loc = mu , scale = sigma)\n",
    "x = X.rvs(N)\n",
    "\n",
    "k = 50\n",
    "confidence_level = 0.95\n",
    "\n",
    "...\n",
    "\n",
    "print(\"The {} confidence interval for N={} is: [{:.3f}, {:.3f}].\".format(confidence_level, N,CI[0], CI[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a921c868",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check(\"Question 2c\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6be93110",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "---\n",
    "\n",
    "To double-check your work, the cell below will rerun all of the autograder tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53012cdf",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "grader.check_all()"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Create Assignment",
  "kernelspec": {
   "display_name": "nf_torch",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "07cb0c0be5d5c09ece9af089e9de2ad5458cdd62d0537c1d9b85ed67c101dd7b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}