07_Intro_sklearn_student.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "964303fb-e336-4c3d-a204-6878d58eb72a",
   "metadata": {},
   "source": [
    "# Introducing Scikit-Learn\n",
    "\n",
    "This exercise generally follows section [5.02 of the PDSH](https://jakevdp.github.io/PythonDataScienceHandbook/05.02-introducing-scikit-learn.html).\n",
    "\n",
    "## Why not the `iris` dataset?\n",
    "\n",
    "For a number of reasons, I have switched out the dataset for a different one. The main reason is that the `iris` dataset that is used in the text, and indeed in **many** tutorials and texts, is ethically challenged. As outlined in the Armchair Ecology blog post *[It's time to retire the iris dataset](https://armchairecology.blog/iris-dataset/)*, [here](https://towardsdatascience.com/the-iris-dataset-a-little-bit-of-history-and-biology-fb4812f5a7b5), and many other sources, the dataset, while collected by Edgar Anderson, was first published by R. A. Fisher in 1936 in the *Annals of Eugenics*; you can read the article [here](https://onlinelibrary.wiley.com/doi/epdf/10.1111/j.1469-1809.1936.tb02137.x) if you wish. To quote from the [Armchair Ecology blog](https://armchairecology.blog/iris-dataset/):\n",
    "\n",
    " >...for you see, `iris` was collected and first published with the express intent to advance the science of eugenics, and parading it around in 2020 is an unacceptable endorsement of a repulsive (but still very much alive) way to subsume science under ideology. For this reason, `iris` should not be appearing in teaching material with such ubiquity, unless it is to remind students that white supremacy has always tried to use quantitative methods to push its agenda, and that quantitative sciences have a foundation in providing arguments to scientific racism and classism.\n",
    "> \n",
    "> By using this dataset in 2020, we are sending a very strong message. Maybe we do not care about the role of science in creating and re-inforcing inequalities and structures of oppression. Maybe we are at peace with the fact that a lot of early quantitative techniques in the biological sciences have been designed to support eugenics and grant it legitimacy, as phrenology did before. Maybe we are eager to pardon the racism of pioneers of the field if their contributions are important enough. Maybe we just don’t care about the social consequences of our science. In any case, just as much as one does not publish in Annals of Eugenics by accident, the decision to keep using this dataset is an endorsement, albeit an implicit one, of being able to draw a straight line from mainstream academic science to white supremacy. Every time we use `iris` as foundational in data science education, we are re-drawing this straight line, over and over again. Whether we intend to do so matters little, if at all.\n",
    "\n",
    "## The `breast_cancer` dataset\n",
    "\n",
    "The dataset for this example is another one built into `sklearn` and comes with this metadata:\n",
    "\n",
    ">This is a copy of UCI ML Breast Cancer Wisconsin (Diagnostic) datasets. https://goo.gl/U2Uwz2\n",
    ">\n",
    ">Features are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass. They describe characteristics of the cell nuclei present in the image.\n",
    ">\n",
    ">Separating plane described above was obtained using Multisurface Method-Tree (MSM-T) [K. P. Bennett, “Decision Tree Construction Via Linear Programming.” Proceedings of the 4th Midwest Artificial Intelligence and Cognitive Science Society, pp. 97-101, 1992], a classification method which uses linear programming to construct a decision tree. Relevant features were selected using an exhaustive search in the space of 1-4 features and 1-3 separating planes.\n",
    ">\n",
    ">The actual linear program used to obtain the separating plane in the 3-dimensional space is that described in: [K. P. Bennett and O. L. Mangasarian: “Robust Linear Programming Discrimination of Two Linearly Inseparable Sets”, Optimization Methods and Software 1, 1992, 23-34].\n",
    "\n",
    "\n",
    "## Data as tables\n",
    "\n",
    "Using our `breast_cancer` dataset, let's follow the example from [5.02 of the PDSH](https://jakevdp.github.io/PythonDataScienceHandbook/05.02-introducing-scikit-learn.html). \n",
    "\n",
    "Because the data are stored and accessed a little differently, there are a few differences. The `sklearn` documentation for [load_breast_cancer](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html) shows that we can load the data into a dataframe (to follow along with the text) using `as_frame=True`, but that the return (what is put into the `breast_cancer` variable below) is a dictionary object. So, to access the dataframe portion of that we use `breast_cancer.frame.head()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46c55bc9-b5bb-4f0b-9613-865d03bdc29f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "393dd6c2-0b39-4e19-a796-b97637d93492",
   "metadata": {},
   "source": [
    "Our data consist of 569 rows, one for each patient. We have 30 columns of baseline data: age, sex, bmi, bp, and blood serum values s1-6. Lastly we have the measure of disease progression one-year later, labeled: target; 0=malignant, 1=benign.\n",
    "\n",
    "The 30 columns are our **features**, our goal is to make a **model** to predict the **target**.\n",
    "\n",
    "As in the text, we can plot the data to explore it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "422cf16a-7e60-4750-824b-e7d73791a61c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import seaborn as sns; sns.set()\n",
    "\n",
    "# Plot only some of the columns since there are so many...takes a long time to do all\n",
    "sns.pairplot(data=breast_cancer.frame,\n",
    "             x_vars=['mean radius','mean texture','mean perimeter','mean area','mean smoothness','mean compactness','mean concavity','mean concave points','mean symmetry'],\n",
    "             y_vars=['mean radius','mean texture','mean perimeter','mean area','mean smoothness','mean compactness','mean concavity','mean concave points','mean symmetry'],\n",
    "             hue='target', size=1.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab57e159-afe5-4e9e-9a0c-91bffdda977d",
   "metadata": {},
   "source": [
    "The next bit of PDSH goes into getting the format expected by `sklearn`, and most other toolkits: a dataframe of features, one row per sample and a vector of labels/outputs one cell per sample. This image from the text does a good job illustrating this:\n",
    "\n",
    "![Diagram showing the feature matrix and target vector for sklearn](https://jakevdp.github.io/PythonDataScienceHandbook/figures/05.02-samples-features.png)\n",
    "\n",
    "While we could follow the text and convert our `breast_cancer.frame` as needed, it is easier to use another handy method of `sklearn` and import the dataset formatted as needed by re-importing the data, this time, setting `return_X_y=True`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b466d264-5697-4ae8-8fb0-ad863c4d2f5e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "34aa481e-daff-4d67-8b40-7ad5d23e96f3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b31f0c5-1b0d-4ddd-a88d-dda4f13ebeb1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0c73a8d1-8829-4fc5-9b58-5dd13c5eab96",
   "metadata": {},
   "source": [
    "## Linear regression with `sklearn`: 1. quick and dirty\n",
    "\n",
    "Let's follow the text and use some made-up data to do a quick linear regression with `sklearn`. After that, we'll come back and add a little more of the theory we started to look at in lecture, and then come back to the breast cancer data.\n",
    "\n",
    "Remember our generic AI cycle diagram from lecture:\n",
    "\n",
    "![Diagram of a generic AI cycle](images/AI-cycle.png)\n",
    "\n",
    "Make some x,y values for the examples...note the in this case, we have *one* feature, $x$, and our target, $y$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b61b29d8-ee23-418b-8b57-993e33730efe",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f05ab05-6b70-49f7-a015-b4f39221f9fb",
   "metadata": {},
   "source": [
    "Following the steps in the text, we need to:\n",
    "\n",
    "### 1. Choose a class of model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bfaf5e81-e34c-4cbc-8ccd-77f2b746e92d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d3155a2c-228c-4c11-955d-83e98d1984dc",
   "metadata": {},
   "source": [
    "### 2. Choose model hyperparameters\n",
    "\n",
    "We haven't really talked about parameters vs hyperparameters.Our model will have two parameters: the slope and y-intercept. Those are the things the model is estimating. Other aspects of how we build and improve the model are hyperparameters...things like, the loss function, the optimizer, and some more nitty-gritty details as noted in the text (normalization, regularization, etc.). Let's instantiate a model and show its (admittedly simple details):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e10a2a50-ba5b-4648-823d-13fe74c96a51",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "9e543997-93ef-4472-8ba1-0858ab1f650a",
   "metadata": {},
   "source": [
    "### 3. Arrange data into features matrix and target vector\n",
    "\n",
    "Right now, $x$ is 'just' an array (look at `x.shape` and `type(x)`). We need it to be a matrix--in this case it will only have one column, but everything is setup to take a matrix. We'll also take this opportunity to name it with the conventional capital-$X$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "191a16db-cc74-4d4f-9cdf-d3509206b66f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "1321d8db-2ae3-47a9-964c-bf9355ecb8e7",
   "metadata": {},
   "source": [
    "### 4. Fit the model to our data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5d2fe77-ea22-4683-b218-9d8e138e6c9d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "76bbf717-8484-4e4b-91f3-681232c04ba5",
   "metadata": {},
   "source": [
    "That was fast! Now we can look at the parameters of the model, which in this case are stored in `model.coef_` (the slope) and `model.intercept_` (the y-intercept)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82b341de-e732-4f79-abc4-2bb95cfe56c5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b41c3c9-ee03-4e78-bec3-1fdc2eeaf823",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "4d9f714d-62c8-4497-9edd-3f4065adef25",
   "metadata": {},
   "source": [
    "Given that we created the $y$ values with the formula `y = 2 * x - 1 + rng.randn(50)`, we expected the `model.coef_` to be 2 and the `model.intercept_` to be -1...not too bad!\n",
    "\n",
    "### 5. Predict labels for unknown data\n",
    "\n",
    "So far, we've got a model based on *training data*. Usually, what we want to do is use this model to predict the response (the $y$'s) for new samples.\n",
    "\n",
    "We can create some new data and reshape it into the required matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1dffd120-5aff-4100-b275-a846760c3b95",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "# Visilize the results\n",
    "  # Scatter plot of our training data\n",
    "  # Our linear model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d40548a3-6e05-4009-baf7-f39040bc2bd8",
   "metadata": {},
   "source": [
    "## Linear regression with `sklearn`: 2. Going back to our theory...\n",
    "\n",
    "A lot happened here with what is a handful of lines of code. `sklearn` is great in that it makes all of this easy. But while we have a relatively simple model, I want to go back and look a the details a bit more in-depth.\n",
    "\n",
    "We will stick with some made up data, almost exactly the same as above, but let's make 100 points, and split the data into 80 for training and 20 for testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df0f2a15-5958-4f67-82d0-6e00f6b645f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "rng = np.random.RandomState(42)\n",
    "x = 10 * rng.rand(100)\n",
    "y = 2 * x - 1 + rng.randn(100)\n",
    "plt.scatter(x, y)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d061963f-20de-4df4-8c4e-954131e29839",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Shuffles the indices to split train and validation datasets\n",
    "idx = np.arange(100)\n",
    "np.random.shuffle(idx)\n",
    "\n",
    "# Uses first 80 random indices for train\n",
    "train_idx = idx[:80]\n",
    "\n",
    "# Uses the remaining indices for validation\n",
    "val_idx = idx[80:]\n",
    "\n",
    "# Generates train and validation sets\n",
    "x_train, y_train = x[train_idx], y[train_idx]\n",
    "x_val, y_val = x[val_idx], y[val_idx]\n",
    "\n",
    "# Plot the test and train datasets\n",
    "fig, ax = plt.subplots(1,2)\n",
    "\n",
    "ax[0].scatter(x_train,y_train, color='blue')\n",
    "ax[0].set_title('Generated Data - Train')\n",
    "ax[0].set_xlabel('x')\n",
    "ax[0].set_ylabel('y')\n",
    "ax[0].set_ylim(1,20) # Set y axis range so both plots are the same.\n",
    "\n",
    "ax[1].scatter(x_val,y_val, color='orange')\n",
    "ax[1].set_title('Generated Data - Validatoin')\n",
    "ax[1].set_xlabel('x')\n",
    "ax[1].set_ylim(1,20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f99c112b-47cd-4c8d-9d96-6a5bfda5b15d",
   "metadata": {},
   "source": [
    "Ok, so now we have our data, let's step through the process as outlined above, but with a bit more detail...\n",
    "\n",
    "### 1. Choose a class of model\n",
    "\n",
    "We are going to do a linear model again, but now, we are going to define our parameters manually. Here's out linear model:\n",
    "\n",
    "$$ y = a + bx + \\epsilon $$\n",
    "\n",
    "Where $a$ is our y-intercept and $b$ is the slope. Again, with how we generated our data, we expect $a=1$ and $b=2$, but let's see...\n",
    "\n",
    "So we need two parameters, $a$ and $b$. What should we set these to? How about we randomly guess as a start?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06767768-dbdd-4b73-9797-8a34fbc31552",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initializes parameters \"a\" and \"b\" randomly\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8c880d4-d4d3-44a2-9706-27ea5144ad66",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's write a function to plot our model predictions\n",
    "def check_model(a, b, x, y):\n",
    "    xfit = np.linspace(-1,11)\n",
    "    yfit =  b * xfit - a\n",
    "\n",
    "    # Visilize the results\n",
    "    plt.scatter(x,y) # Scatter plot of our training data\n",
    "    plt.plot(xfit,yfit) # Our linear model\n",
    "    \n",
    "check_model(a,b, x_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcda9650-133e-453c-ad2e-2b28f83cd82a",
   "metadata": {},
   "source": [
    "### 2. Choose model hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "420f5de2-e86c-432d-a651-b498bfcce6f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sets hyper-parameters \n",
    "# learning rate\n",
    "\n",
    "\n",
    "# Defines number of epochs\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11f697b0-8def-4e54-82ff-1dd191071d9a",
   "metadata": {},
   "source": [
    "### 3. Arrange data into features matrix and target vector\n",
    "\n",
    "In this case, because of the way we are going to make the model, we are ok leaving the x_train as an array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58f7a6cc-dbc8-4647-bf93-5258cb4c77f9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8a72369f-24f7-43b8-9eb8-2098d50a14cc",
   "metadata": {},
   "source": [
    "### 4. Fit the model to our data\n",
    "\n",
    "Fitting, or training, the model has several steps that we'll break out here.\n",
    "\n",
    "#### Model training step 1: Compute the model's predicted output\n",
    "\n",
    "Using the current model parameters, what output do we get? We call these estimates $\\hat{y}$, or y-hat. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5a84082-d93d-4d80-b0ed-e5f2c798c883",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "28fd0cbb-addb-4573-803e-4c65c16aeb3b",
   "metadata": {},
   "source": [
    "#### Model training step 2: Compute the loss\n",
    "\n",
    "Compare the predicted values to the known values and calculate a loss. We'll use mean squared error as our loss function that has the formula:\n",
    "\n",
    "$$ MSE = \\frac{1}{N}\\sum_{i=1}^N ( y_i - \\hat{y_i} )^2 $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae4de7bf-c069-462c-87e1-3fc45496d15d",
   "metadata": {},
   "outputs": [],
   "source": [
    " # The part in parentheses\n",
    " #Square the error, and get the mean"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5494ecfe-b0f3-4859-b519-10641009e4af",
   "metadata": {},
   "source": [
    "#### Model training step 3: Compute the gradient\n",
    "\n",
    "Now we need to use our optimization tools to figure out how to update the model parameters. I'm not going to derive this here, but there are many place, one of which is Daniel Goody's post [here](https://towardsdatascience.com/understanding-pytorch-with-an-example-a-step-by-step-tutorial-81fc5f8c4e8e) if you want to get into the calculus that get's us here. \n",
    "\n",
    "The main thing to remember is that for each of our parameters, we are looking at the slope of the loss function based on where we are on the loss surface. Based on that slope, we can determine which way to update the parameter (increase it or decrease it) and by how much to change it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5282f0f7-d9e0-48ee-ba5c-110aab0be501",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate the gradients for our two parameters, a and b.\n",
    "\n",
    "a_grad = -2 * error.mean()\n",
    "b_grad = -2 * (x_train * error).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1321a43-fa90-44c5-bd7e-0bd66d30f381",
   "metadata": {},
   "source": [
    "#### Model training step 4: Update the parameters\n",
    "\n",
    "Now that we know how to update the model parameters, we can update them. We'll use the learning rate hyperparameter to limit how much we update the parameters so that we don't overshoot when we make updates. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3344e613-77bb-4356-ac7d-960c29f22f47",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = a - lr * a_grad\n",
    "b = b - lr * b_grad\n",
    "\n",
    "print(a,b)\n",
    "check_model(a,b,x_train,y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91559e64-c0d7-4e77-b58b-ecd170e808dd",
   "metadata": {},
   "source": [
    "That's about it! We've gone through **one epoch** of learning! Our parameters, $a$ and $b$, have been updated, and hopefully our model is better at predicting our response. But machine learning is iterative...our model is still fairly bad..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "706a3dab-9f1e-4612-aca6-e8ec72d8de2d",
   "metadata": {},
   "source": [
    "#### Repeat the model fitting steps\n",
    "\n",
    "No problem! Let's run through the process a bunch (`n_epochs = 1000`) of times..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8f2eb2b-fd14-44cf-bdf6-171e6e414526",
   "metadata": {},
   "outputs": [],
   "source": [
    "for epoch in range(n_epochs):\n",
    "    # Step 1: Computes our model's predicted output\n",
    "    yhat = a + b * x_train\n",
    "    \n",
    "    # Step 2: Compue loss\n",
    "    error = (y_train - yhat)\n",
    "    loss = (error ** 2).mean()\n",
    "    \n",
    "    # Step 3: Compute gradients\n",
    "    a_grad = -2 * error.mean()\n",
    "    b_grad = -2 * (x_train * error).mean()\n",
    "    \n",
    "    # Step 4: Update parameters using gradients and the learning rate\n",
    "    a = a - lr * a_grad\n",
    "    b = b - lr * b_grad\n",
    "    \n",
    "    if epoch % 50 == 0: # Print a,b and graph every 50 epochs\n",
    "        print(f'Epoch: {epoch}; a={a},b={b}')\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77d47a39-c641-4002-acc5-4b5134a8507b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now we can check out model performance with the validation data\n",
    "\n",
    "check_model(a,b, x_val, y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dedfd344-e9cf-49b2-9228-03f173b380c0",
   "metadata": {},
   "source": [
    "## Finally...back to the breast cancer data!\n",
    "\n",
    "Now, rather than a continuous variable for regression, we have a binary variable for classification. We can pickup following the text... \n",
    "\n",
    "### Supervised learning example: Breast cancer classification\n",
    "\n",
    "Two quick notes here:\n",
    "1. The text uses `from sklearn.cross_validation import train_test_split`...that won't work because recent versions of `sklearn` moved `train_test_split` to `model_selection`.\n",
    "1. This is common in AI coding! Programming packages change fast and tutorials and example code break quickly. Learning how to deal with that is part of the fun and equally important 😅! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8dde9442-2dba-473b-85e4-35091fa82f01",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "Xtrain, Xtest, ytrain, ytest = train_test_split(X_breast_cancer, y_breast_cancer,\n",
    "                                                random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5132ec6e-85b7-49f1-9205-7197f061115f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import GaussianNB # 1. choose model class\n",
    "                        # 2. instantiate model\n",
    "                        # 3. fit model to data\n",
    "                        # 4. predict on new data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30191d90-96ba-4d7c-b57e-aa3e542d6284",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f3977e7-f6f5-4778-8428-7b095e9985f4",
   "metadata": {},
   "source": [
    "With a few lines of code, we have a model than can predict breast cancer occurrence with an accuracy of 94%! That's quite amazing! (It's also fairly easy data an we kind of saw that with the pairplots above...)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d1de0b5-0345-43b7-8f82-2406a27ab6a2",
   "metadata": {},
   "source": [
    "## Unsupervised learning example: Breast cancer dimensionality\n",
    "\n",
    "Continuing with the text, let's try dimensionality reduction, in this case, using principle component analysis (PCA).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "642023c5-5cf8-4f63-81d8-ed81cad9886c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA  # 1. Choose the model class\n",
    "                       # 2. Instantiate the model with hyperparameters\n",
    "                      # 3. Fit to data. Notice y is not specified!\n",
    "                       # 4. Transform the data to two dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df5ca61f-4478-45c7-8acf-3b1f571ac646",
   "metadata": {},
   "outputs": [],
   "source": [
    "breast_cancer.frame['PCA1'] = X_2D[:, 0]\n",
    "breast_cancer.frame['PCA2'] = X_2D[:, 1]\n",
    "sns.lmplot(\"PCA1\", \"PCA2\", hue='target', data=breast_cancer.frame, fit_reg=False);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "713b4e5f-c393-4741-9101-b22dd253dbef",
   "metadata": {},
   "source": [
    "We could go on following the text, though I'm not sure clustering is that relevant in this case.\n",
    "\n",
    "Let's explore the famous digits dataset...\n",
    "\n",
    "## Application: Exploring the Hand-written Digits\n",
    "\n",
    "This dataset is a subset of the famous AI/ML dataset MNIST which we may revisit later in the semester. Here's the metadata:\n",
    "> The data set contains images of hand-written digits: 10 classes where each class refers to a digit.\n",
    ">\n",
    "> Preprocessing programs made available by NIST were used to extract normalized bitmaps of handwritten digits from a preprinted form. From a total of 43 people, 30 contributed to the training set and different 13 to the test set. 32x32 bitmaps are divided into nonoverlapping blocks of 4x4 and the number of on pixels are counted in each block. This generates an input matrix of 8x8 where each element is an integer in the range 0..16. This reduces dimensionality and gives invariance to small distortions.\n",
    ">\n",
    ">For info on NIST preprocessing routines, see M. D. Garris, J. L. Blue, G. T. Candela, D. L. Dimmick, J. Geist, P. J. Grother, S. A. Janet, and C. L. Wilson, NIST Form-Based Handprint Recognition System, NISTIR 5469, 1994.\n",
    "\n",
    "\n",
    "### Loading and visualizing the data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ac6fb94-e870-49f6-8550-42ebcb73ca96",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2dd723f5-e9bc-4fc6-acd7-bee220a52b92",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig, axes = plt.subplots(10, 10, figsize=(8, 8),\n",
    "                         subplot_kw={'xticks':[], 'yticks':[]},\n",
    "                         gridspec_kw=dict(hspace=0.1, wspace=0.1))\n",
    "\n",
    "for i, ax in enumerate(axes.flat):\n",
    "    ax.imshow(digits.images[i], cmap='binary', interpolation='nearest')\n",
    "    ax.text(0.05, 0.05, str(digits.target[i]),\n",
    "            transform=ax.transAxes, color='green')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd2094b8-1dc0-4212-84d1-7dae43bb325a",
   "metadata": {},
   "source": [
    "#### Prepare the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5eb0322-9711-4744-9272-592f1570b03a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7832978c-020d-4bfb-9ce7-3d9759e0d9e5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "b0d8b563-7f37-4da9-aa52-9e8c530f62ef",
   "metadata": {},
   "source": [
    "### Unsupervised learning: Dimensionality reduction\n",
    "\n",
    "You can read more in the text, but I'm going to skip over the details of making the graph below...mostly a way of visualizing our 64-dimensional data in 2-dimensions.\n",
    "\n",
    "Then we use a classification algorithm, Gaussian naive Bayes, on the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c285c34a-9251-46d2-992f-6e295b0213c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.manifold import Isomap\n",
    "iso = Isomap(n_components=2)\n",
    "iso.fit(digits.data)\n",
    "data_projected = iso.transform(digits.data)\n",
    "data_projected.shape\n",
    "\n",
    "plt.scatter(data_projected[:, 0], data_projected[:, 1], c=digits.target,\n",
    "            edgecolor='none', alpha=0.5,\n",
    "            cmap=plt.cm.get_cmap('rainbow', 10))\n",
    "plt.colorbar(label='digit label', ticks=range(10))\n",
    "plt.clim(-0.5, 9.5);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b4d8361-7738-4503-bf33-a49d7913487f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split the data into training and testing\n",
    "Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, random_state=0)\n",
    "\n",
    "# Fit the Gaissian Naive Bayes model\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "model = GaussianNB()\n",
    "model.fit(Xtrain, ytrain)\n",
    "y_model = model.predict(Xtest)\n",
    "\n",
    "# Get the accuracy\n",
    "from sklearn.metrics import accuracy_score\n",
    "accuracy_score(ytest, y_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a7bed89-b496-494b-a96b-6e6a5a908cf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# See where our model makes mistakes\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "mat = confusion_matrix(ytest, y_model)\n",
    "\n",
    "sns.heatmap(mat, square=True, annot=True, cbar=False)\n",
    "plt.xlabel('predicted value')\n",
    "plt.ylabel('true value');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0db7c1ab-9cf2-4979-af96-befeba298d7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# See some examples of correct and incorrect classifications\n",
    "fig, axes = plt.subplots(10, 10, figsize=(8, 8),\n",
    "                         subplot_kw={'xticks':[], 'yticks':[]},\n",
    "                         gridspec_kw=dict(hspace=0.1, wspace=0.1))\n",
    "\n",
    "test_images = Xtest.reshape(-1, 8, 8)\n",
    "\n",
    "for i, ax in enumerate(axes.flat):\n",
    "    ax.imshow(test_images[i], cmap='binary', interpolation='nearest')\n",
    "    ax.text(0.05, 0.05, str(y_model[i]),\n",
    "            transform=ax.transAxes,\n",
    "            color='green' if (ytest[i] == y_model[i]) else 'red')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Tensorflow-2.6.0",
   "language": "python",
   "name": "tensorflow-2.6.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}