Statistics/Sample_With_Replacement/SampleWithReplacement.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "U7P3EBo0XxvD"
   },
   "source": [
    "<h1>Understanding Sampling with Replacement (Python)</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/TOCSampleWithReplacement.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sampling with replacement can be defined as random sampling that allows sampling units to occur more than once. Sampling with replacement consists of\n",
    "\n",
    "1. A sampling unit (like a glass bead or a row of data) being randomly drawn from a population (like a jar of beads or a dataset). \n",
    "2. Recording which sampling unit was drawn.\n",
    "3. Returning the sampling unit to the population.\n",
    "\n",
    "The reason why the sampling unit is returned to the population before the next sampling unit is drawn is to make sure the probability of selecting any particular sampling unit remains the same in future draws. There are many applications of sampling with replacement throughout data science. Many of these applications use bootstrapping which is a statistical procedure that uses sampling with replacement on a dataset to create many simulated samples. Datasets that are created with sampling with replacement so that they have the same number of samples as the original dataset are called bootstrapped datasets. Bootstrapped data is used in machine learning algorithms like [bagged trees](https://youtu.be/urb2wRxnGz4) and random forests as well as in statistical methods like [bootstrapped confidence intervals](https://machinelearningmastery.com/calculate-bootstrap-confidence-intervals-machine-learning-results-python/), and more.\n",
    "\n",
    "This tutorial will dive into sampling with and without replacement and will touch on some common applications of these concepts in data science. As always, the code used in this tutorial is available on my GitHub. With that, let's get started!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>What is Sampling with Replacement</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/TOCSampleWithReplacement.png)\n",
    "Caption: Sampling with replacement procedure"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sampling with replacement can be defined as random sampling that allows sampling units to occur more than once. Sampling with replacement consists of\n",
    "\n",
    "1. A sampling unit (like a glass bead or a row of data) being randomly drawn from a population (like a jar of beads or a dataset). \n",
    "2. Recording which sampling unit was drawn.\n",
    "3. Returning the sampling unit to the population.\n",
    "\n",
    "Imagine you have a jar of 12 unique glass beads like in the image above. If you are sampling with replacement from the jar, the chance of <b>randomly</b> selecting any 1 of the glass beads is 1/12. After selecting a bead, return it to the jar so that the probability of selecting any of the 12 beads in future sampling doesn't change (1/12). This means that if you repeat the process it is entirely possible you could randomly take out the same bead (1/12 chance in this case). \n",
    "\n",
    "This remaining parts of this section go over how sampling with replacement can be done using the Python libraries NumPy and Pandas and will go over related concepts like bootstrapped datasets and how many duplicate samples should you expect when sampling with replacement to create a bootstrapped dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Sampling with Replacement using NumPy</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to better understand sample with replacement, let's now simulate this process with Python. The code below loads NumPy and samples with replacement 12 times from a NumPy array containing unique numbers from 0 to 11"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([10,  8,  9,  3,  8,  8,  0,  5,  3, 10, 11,  9])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "np.random.seed(3)\n",
    "\n",
    "# a parameter: generate a list of unique random numbers (from 0 to 11)\n",
    "# size parameter: how many samples we want (12)\n",
    "# replace = True: sample with replacement\n",
    "np.random.choice(a=12, size=12, replace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how we have multiple repeating numbers. The reason why we sampled 12 times in the code above is because the original jar (dataset) we are sampling from has 12 beads (sampling units) in it. The 12 marbles we selected are now part of a bootstrapped dataset which is a dataset that is created with sampling with replacement that has the same number of values as the original dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> Sampling with Replacement using Pandas</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since most people aren't interested in the application of sampling beads out of a jar, it is important to mention a sampling unit can also be something like an entire row of data. The code below creates a bootstrapped dataset using Kaggle's King County dataset which contains the price at which houses were sold for in King County, which includes Seattle between May 2014 and May 2015. You can download the dataset from [Kaggle](https://www.kaggle.com/harlfoxem/housesalesprediction) or load it from my [GitHub](https://raw.githubusercontent.com/mGalarnyk/Tutorial_Data/master/King_County/kingCountyHouseData.csv)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>sqft_living</th>\n",
       "      <th>sqft_lot</th>\n",
       "      <th>floors</th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>3</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1780</td>\n",
       "      <td>7470</td>\n",
       "      <td>1.0</td>\n",
       "      <td>229500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>3</td>\n",
       "      <td>1.75</td>\n",
       "      <td>1370</td>\n",
       "      <td>9680</td>\n",
       "      <td>1.0</td>\n",
       "      <td>400000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>3</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1780</td>\n",
       "      <td>7470</td>\n",
       "      <td>1.0</td>\n",
       "      <td>229500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3</td>\n",
       "      <td>2.25</td>\n",
       "      <td>1715</td>\n",
       "      <td>6819</td>\n",
       "      <td>2.0</td>\n",
       "      <td>257500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1160</td>\n",
       "      <td>6000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>468000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>1.00</td>\n",
       "      <td>770</td>\n",
       "      <td>10000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>180000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1160</td>\n",
       "      <td>6000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>468000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>3</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1780</td>\n",
       "      <td>7470</td>\n",
       "      <td>1.0</td>\n",
       "      <td>229500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>3</td>\n",
       "      <td>1.50</td>\n",
       "      <td>1060</td>\n",
       "      <td>9711</td>\n",
       "      <td>1.0</td>\n",
       "      <td>291850.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>1.00</td>\n",
       "      <td>770</td>\n",
       "      <td>10000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>180000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>2.25</td>\n",
       "      <td>2570</td>\n",
       "      <td>7242</td>\n",
       "      <td>2.0</td>\n",
       "      <td>538000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1160</td>\n",
       "      <td>6000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>468000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>4</td>\n",
       "      <td>4.50</td>\n",
       "      <td>5420</td>\n",
       "      <td>101930</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1225000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>3</td>\n",
       "      <td>2.50</td>\n",
       "      <td>3560</td>\n",
       "      <td>9796</td>\n",
       "      <td>1.0</td>\n",
       "      <td>662500.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3</td>\n",
       "      <td>2.00</td>\n",
       "      <td>1680</td>\n",
       "      <td>8080</td>\n",
       "      <td>1.0</td>\n",
       "      <td>510000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    bedrooms  bathrooms  sqft_living  sqft_lot  floors      price\n",
       "8          3       1.00         1780      7470     1.0   229500.0\n",
       "13         3       1.75         1370      9680     1.0   400000.0\n",
       "8          3       1.00         1780      7470     1.0   229500.0\n",
       "6          3       2.25         1715      6819     2.0   257500.0\n",
       "11         2       1.00         1160      6000     1.0   468000.0\n",
       "2          2       1.00          770     10000     1.0   180000.0\n",
       "11         2       1.00         1160      6000     1.0   468000.0\n",
       "8          3       1.00         1780      7470     1.0   229500.0\n",
       "7          3       1.50         1060      9711     1.0   291850.0\n",
       "2          2       1.00          770     10000     1.0   180000.0\n",
       "1          3       2.25         2570      7242     2.0   538000.0\n",
       "11         2       1.00         1160      6000     1.0   468000.0\n",
       "5          4       4.50         5420    101930     1.0  1225000.0\n",
       "10         3       2.50         3560      9796     1.0   662500.0\n",
       "4          3       2.00         1680      8080     1.0   510000.0"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# Load dataset\n",
    "url = 'https://raw.githubusercontent.com/mGalarnyk/Tutorial_Data/master/King_County/kingCountyHouseData.csv'\n",
    "df = pd.read_csv(url)\n",
    "# Selecting columns I am interested in\n",
    "columns= ['bedrooms','bathrooms','sqft_living','sqft_lot','floors','price']\n",
    "df = df.loc[:, columns]\n",
    "\n",
    "# Only want to use 15 rows of the dataset for illustrative purposes. \n",
    "df = df.head(15)\n",
    "\n",
    "# Notice how we have 3 rows with the index label 8\n",
    "df.sample(n = 15, replace = True, random_state=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>How many duplicate samples/rows should you expect when sampling with replacement to create a bootstrapped dataset?</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is important to note that when you do sample with replacement to generate data you will likely get duplicate samples/rows. In practice, the average bootstrapped dataset contains about 63.2% of the original rows. This means that for any particular row of data in the original dataset, 36.8% of the bootstrapped datasets will not contain it. \n",
    "\n",
    "This subsection briefly shows how you can derive these numbers statistically and as well as get close to them by experiment using the Python library pandas. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Basic Statistics</b> \n",
    "\n",
    "Let's start by deriving how for any particular row of data in the original dataset, 36.8% of the bootstrapped datasets will not contain that row.\n",
    "\n",
    "Assume there are N rows of data in the original dataset. If you want to create a bootstrapped dataset, you need to sample with replacement N times.  \n",
    "\n",
    "For a SINGLE sample with replacement, the probability that a particular row of data is not randomly sampled with replacement from the dataset is\n",
    "\n",
    "$$1 - \\frac{1}{N}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since a bootstrapped dataset is obtained by sampling N times from a dataset of size N, we need to sample N times to find the probability that a particular row is not chosen in a given bootstrapped dataset. \n",
    "\n",
    "$$\\left(1 - \\frac{1}{N}\\right)^{N}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we take the limit as $N\\to\\infty$, we find that the probability is .368. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\lim_{N\\to\\infty}\\left(1 - \\frac{1}{N}\\right)^{N} = e^{-1} = .36787 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The probability that any particular row of data from the original dataset would be in the bootstrapped dataset is just 1 -  $e^{-1}$ = .63213. Note that in real life, the larger your dataset is (the larger N is), the more likely you will get close to these numbers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Using pandas</b> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below uses pandas to show that a bootstrapped dataset will contain about 63.2% of the original rows. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# Load dataset\n",
    "url = 'https://raw.githubusercontent.com/mGalarnyk/Tutorial_Data/master/King_County/kingCountyHouseData.csv'\n",
    "df = pd.read_csv(url)\n",
    "# Selecting columns I am interested in\n",
    "columns= ['bedrooms','bathrooms','sqft_living','sqft_lot','floors','price']\n",
    "df = df.loc[:, columns]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate Bootstrapped Dataset (dataset generated with sample with replacement which has the same number of values as original dataset)\n",
    "# % of original rows will vary depending on random_state\n",
    "bootstrappedDataset = df.sample(frac = 1, replace = True, random_state = 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the bootstrap sample below, note that it contains about 63.2% of the original samples/rows. This is because the sample size was large (len(df) is 21613). This also means that each bootstrapped dataset will not include about 36.8% of the rows from the original dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21613"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6317956785268126"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(bootstrappedDataset.index.unique()) / len(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>What is Sampling without Replacement</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/SampleWithoutReplacement.png)\n",
    "\n",
    "Sampling without replacement can be defined as random sampling that DOES NOT allow sampling units to occur more than once. Let's now go over a quick example of how sampling <b>without</b> replacement works.\n",
    "\n",
    "Imagine you have a jar of 12 unique glass beads like in the image above. If you are sampling <b>without</b> replacement from the jar, the chance of <b>randomly</b> selecting any 1 of the glass beads is 1/12. After selecting a bead, it is NOT returned to the jar so that the probability of selecting any of the remaining 11 beads in future sampling is now (1/11). This means that for each additional sample drawn, there are less and less beads in the jar until eventually there are no more beads to sample (after 12 samplings)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Sampling without Replacement using NumPy</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to ingrain this knowledge, let's now simulate this process with Python. The code below loads NumPy and samples <b>without</b> replacement 12 times from a NumPy array containing unique numbers from 0 to 11"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 5,  4,  1,  2, 11,  6,  7,  0,  3,  9,  8, 10])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "np.random.seed(3)\n",
    "\n",
    "# a parameter: generate a list of unique random numbers (from 0 to 11)\n",
    "# size parameter: how many samples we want (12)\n",
    "# replace = False: sample without replacement\n",
    "np.random.choice(a=12, size=12, replace=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how there aren't repeating numbers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that if you try to generate a sample using sampling WITHOUT replacement that is longer than the original sample (12 in this case), you will get an error. Going to back to the jar of beads example, you can't sample more beads than there are in the jar. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Cannot take a larger sample than population when 'replace=False'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m/var/folders/5m/6x56qhwd14d_f6qmsh2h09640000gn/T/ipykernel_49469/3827293918.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrandom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mseed\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrandom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchoice\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m12\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m20\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mreplace\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32mmtrand.pyx\u001b[0m in \u001b[0;36mnumpy.random.mtrand.RandomState.choice\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: Cannot take a larger sample than population when 'replace=False'"
     ]
    }
   ],
   "source": [
    "np.random.seed(3)\n",
    "np.random.choice(a=12, size=20, replace=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images/emptyMarbleJar.png)\n",
    "\n",
    "Caption: You can't sample more beads than there are in the jar. Image by [Michael Galarnyk](https://twitter.com/GalarnykMichael)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> Examples of Sampling without Replacement in Data Science</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sampling without replacement is used throughout data science. One very common use is in model validation procedures like [train test split](https://builtin.com/data-science/train-test-split) and [cross validation](https://scikit-learn.org/stable/modules/cross_validation.html). In short, each of these procedures allows you to simulate how a machine learning model would perform on new/unseen data. \n",
    "\n",
    "The image below shows the train test split procedure which consists of splitting a dataset into two pieces: a training set and a testing set. This consists of randomly sampling WITHOUT replacement about 75% (you can vary this) of the rows and putting them into your training set and putting the remaining 25% to your test set. Note that the colors in “Features” and “Target” indicate where their data will go (“X_train”, “X_test”, “y_train”, “y_test”) for a particular train test split.\n",
    "\n",
    "![](images/TrainTestSplit.png)\n",
    "\n",
    "If you would like to learn more about train test split, you can check out my blog post [Understanding Train Test Split](https://builtin.com/data-science/train-test-split)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Conclusion</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Understanding the concept of sampling with and without replacement is important in statistics and data science. Bootstrapped data is used in machine learning algorithms like [bagged trees](https://youtu.be/urb2wRxnGz4) and random forests as well as in statistical methods like [bootstrapped confidence intervals](https://machinelearningmastery.com/calculate-bootstrap-confidence-intervals-machine-learning-results-python/), and more.\n",
    "\n",
    "A future tutorials will take some of this knowledge and go over how it is applied to understanding bagged trees and random forests. If you have any questions or thoughts on the tutorial, feel free to reach out in the comments below or through [Twitter](https://twitter.com/GalarnykMichael)."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "colab": {
   "collapsed_sections": [],
   "name": "SampleWithReplacement.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}