project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classifying Song Genres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a code cell without any tag. You can put convenience code here,\n",
    "# but it won't be included in any way in the final project.\n",
    "# For example, to be able to run tests locally in the notebook\n",
    "# you need to install the following:\n",
    "# pip install nose\n",
    "# pip install git+https://github.com/datacamp/ipython_nose\n",
    "# and then load in the ipython_nose extension like this:\n",
    "%load_ext ipython_nose"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "type:NotebookTask"
    ]
   },
   "source": [
    "## 1. Preparing our dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@context"
    ]
   },
   "source": [
    "_These recommendations are so on point! How does this playlist know me so well?_\n",
    "\n",
    "<img src=\"img/iphone_music.jpg\" alt=\"Project Image Record\" width=\"600px\"/>\n",
    "\n",
    "Over the past few years, streaming services with huge catalogs have become the primary means through which most people listen to their favorite music. But at the same time, the sheer amount of music on offer can mean users might be a bit overwhelmed when trying to look for newer music that suits their tastes.\n",
    "\n",
    "For this reason, streaming services have looked into means of categorizing music to allow for personalized recommendations. One method involves direct analysis of the raw audio information in a given song, scoring the raw data on a variety of metrics. Today, we'll be examining data compiled by a research group known as The Echo Nest. Our goal is to look through this dataset and classify songs as being either 'Hip-Hop' or 'Rock' - all without listening to a single one ourselves. In doing so, we will learn how to clean our data, do some exploratory data visualization, and use feature reduction towards the goal of feeding our data through some simple machine learning algorithms, such as decision trees and logistic regression.\n",
    "\n",
    "To begin with, let's load the metadata about our tracks alongside the track metrics compiled by The Echo Nest. A song is about more than its title, artist, and number of listens. We have another dataset that has musical features of each track such as `danceability` and `acousticness` on a scale from -1 to 1. These exist in two different files, which are in different formats - CSV and JSON. While CSV is a popular file format for denoting tabular data, JSON is another common file format in which databases often return the results of a given query.\n",
    "\n",
    "Let's start by creating two pandas `DataFrames` out of these files that we can merge so we have features and labels (often also referred to as `X` and `y`) for the classification later on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@instructions"
    ]
   },
   "source": [
    "Read in the data using `pandas` and merge the DataFrames into one usable dataset.\n",
    "\n",
    "- Using the pandas `read_csv()` function, read in the file with the track metadata (`datasets/fma-rock-vs-hiphop.csv`) and name the DataFrame `tracks`.\n",
    "- Using the pandas `read_json()` function, read in the JSON file with the track acoustic metrics (`datasets/echonest-metrics.json`) and name the DataFrame `echonest_metrics`. Set the `precise_float` argument to `True` when reading in your data.\n",
    "- Merge the DataFrames on matching `track_id` values. Only retain the `track_id` and `genre_top` columns of `tracks`. `echonest_metrics` should be the first (left) data frame in the merge.\n",
    "- Inspect the DataFrame using the `.info()` method.\n",
    "\n",
    "<hr>\n",
    "\n",
    "## Good to know\n",
    "\n",
    "This project lets you apply what you learned in [Supervised Learning with scikit-learn](https://www.datacamp.com/courses/supervised-learning-with-scikit-learn), plus data preprocessing, dimensionality reduction, and machine learning using the `scikit-learn` package. We recommend you are familiar with these topics before starting this project.\n",
    "\n",
    "Helpful links:\n",
    "- Documentation for pandas [`read_csv()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html), [`read_json()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_json.html) and [`pd.merge()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.merge.html) functions\n",
    "- Variance of the PCA features [exercise](https://campus.datacamp.com/courses/unsupervised-learning-in-python/decorrelating-your-data-and-dimension-reduction?ex=7)\n",
    "- Train/test/split + Fit/Predict/Accuracy [exercise](https://campus.datacamp.com/courses/supervised-learning-with-scikit-learn/classification?ex=11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@hint"
    ]
   },
   "source": [
    "You can select columns of a DataFrame by providing a list to the indexer i.e. using the `[]`. A correct solution for the merge looks like this:\n",
    "\n",
    "```python\n",
    "echo_tracks = echonest_metrics.merge(tracks[['column1_name', 'column2_name']], on='column2_name')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Read in track metadata with genre labels\n",
    "tracks = ...\n",
    "\n",
    "# Read in track metrics with the features\n",
    "echonest_metrics = ...\n",
    "\n",
    "# Merge the relevant columns of tracks and echonest_metrics\n",
    "echo_tracks = ...\n",
    "\n",
    "# Inspect the resultant dataframe\n",
    "..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@solution"
    ]
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Read in track metadata with genre labels\n",
    "tracks = pd.read_csv('datasets/fma-rock-vs-hiphop.csv')\n",
    "\n",
    "# Read in track metrics with the features\n",
    "echonest_metrics = pd.read_json('datasets/echonest-metrics.json', precise_float=True)\n",
    "\n",
    "# Merge the relevant columns of tracks and echonest_metrics\n",
    "echo_tracks = echonest_metrics.merge(tracks[['genre_top', 'track_id']], on='track_id')\n",
    "\n",
    "# Inspect the resultant dataframe\n",
    "echo_tracks.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@tests"
    ]
   },
   "outputs": [],
   "source": [
    "%%nose\n",
    "\n",
    "def test_tracks_read():\n",
    "    try:\n",
    "        pd.testing.assert_frame_equal(tracks, pd.read_csv('datasets/fma-rock-vs-hiphop.csv'))\n",
    "    except AssertionError:\n",
    "        assert False, \"The tracks data frame was not read in correctly.\"\n",
    "\n",
    "def test_metrics_read():\n",
    "    ech_met_test = pd.read_json('datasets/echonest-metrics.json', precise_float=True)\n",
    "    try:\n",
    "        pd.testing.assert_frame_equal(echonest_metrics, ech_met_test)\n",
    "    except AssertionError:\n",
    "        assert False, \"The echonest_metrics data frame was not read in correctly.\"\n",
    "        \n",
    "def test_merged_shape(): \n",
    "    merged_test = echonest_metrics.merge(tracks[['genre_top', 'track_id']], on='track_id')\n",
    "    try:\n",
    "        pd.testing.assert_frame_equal(echo_tracks, merged_test)\n",
    "    except AssertionError:\n",
    "        assert False, ('The two datasets should be merged on matching track_id values '\n",
    "                       'keeping only the track_id and genre_top columns of tracks.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "type:NotebookTask"
    ]
   },
   "source": [
    "## 2. Pairwise relationships between continuous variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@context"
    ]
   },
   "source": [
    "We typically want to avoid using variables that have strong correlations with each other -- hence avoiding feature redundancy -- for a few reasons:\n",
    "- To keep the model simple and improve interpretability (with many features, we run the risk of overfitting).\n",
    "- When our datasets are very large, using fewer features can drastically speed up our computation time.\n",
    "\n",
    "To get a sense of whether there are any strongly correlated features in our data, we will use built-in functions in the `pandas` package."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@instructions"
    ]
   },
   "source": [
    "Explore correlations in our dataset using pandas `corr` function.\n",
    "- Visually inspect the correlation table generated from `DataFrame.corr()` for any strong correlations.\n",
    "\n",
    "<hr>\n",
    "\n",
    "Helpful links:\n",
    "- pandas DataFrame `corr` method [documentation](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.corr.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@hint"
    ]
   },
   "source": [
    "Does your code look something like this?\n",
    "\n",
    "```python\n",
    "corr_metrics = df.corr()\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# Create a correlation matrix\n",
    "corr_metrics = ...\n",
    "corr_metrics.style.background_gradient()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@solution"
    ]
   },
   "outputs": [],
   "source": [
    "# Create a correlation matrix\n",
    "corr_metrics = echonest_metrics.corr()\n",
    "corr_metrics.style.background_gradient()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@tests"
    ]
   },
   "outputs": [],
   "source": [
    "%%nose\n",
    "\n",
    "def test_corr_matrix():\n",
    "    assert all(corr_metrics == echonest_metrics.corr()) and isinstance(corr_metrics, pd.core.frame.DataFrame), \\\n",
    "        'The correlation matrix can be computed using the .corr() method.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "type:NotebookTask"
    ]
   },
   "source": [
    "## 3. Normalizing the feature data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@context"
    ]
   },
   "source": [
    "As mentioned earlier, it can be particularly useful to simplify our models and use as few features as necessary to achieve the best result. Since we didn't find any particular strong correlations between our features, we can instead use a common approach to reduce the number of features called **principal component analysis (PCA)**. \n",
    "\n",
    "It is possible that the variance between genres can be explained by just a few features in the dataset. PCA rotates the data along the axis of highest variance, thus allowing us to determine the relative contribution of each feature of our data towards the variance between classes. \n",
    "\n",
    "However, since PCA uses the absolute variance of a feature to rotate the data, a feature with a broader range of values will overpower and bias the algorithm relative to the other features. To avoid this, we must first normalize our data. There are a few methods to do this, but a common way is through *standardization*, such that all features have a mean = 0 and standard deviation = 1 (the resultant is a z-score)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@instructions"
    ]
   },
   "source": [
    "Prepare our dataset for training a model, and standardize the data.\n",
    "- Define our features from `echo_tracks` by removing `genre_top` and `track_id` from the DataFrame using `DataFrame.drop()` along axis 1.\n",
    "- Define our labels -- in this case, the `genre_top` column from `echo_tracks`.\n",
    "- Import the `StandardScaler` from the `sklearn.preprocessing` module\n",
    "- Define an instance of the `StandardScaler` called `scaler` without passing any arguments and use the `fit_transform` method to scale `features` and save to a new variable called `scaled_train_features`\n",
    "\n",
    "\n",
    "<hr>\n",
    "\n",
    "Helpful links:\n",
    "- pandas `DataFrame.drop()` method [documentation](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop.html)\n",
    "- Square brackets in Pandas [exercise](https://campus.datacamp.com/courses/intermediate-python-for-data-science/dictionaries-pandas?ex=15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@hint"
    ]
   },
   "source": [
    "The `axis` argument of the `drop()` method by convention considers `0` for rows and `1` for columns to look through\n",
    "to drop. You can drop columns named `col1` and `col2` from the dataframe `df` and save to a new dataframe called\n",
    "`df_drop` like so:  \n",
    "\n",
    "```python\n",
    "df_drop = df.drop(columns=['col1', 'col2'])\n",
    "```\n",
    "\n",
    "You can use the `StandardScaler` to standardize the data in `my_data` like so after you define an instance, here called `scaler` using the `fit_transform` method of the `StandardScaler`.\n",
    "\n",
    "```python\n",
    "scaler = StandardScaler()\n",
    "scaler.fit_transform(my_data)\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# Define our features \n",
    "features = ...\n",
    "\n",
    "# Define our labels\n",
    "labels = ...\n",
    "\n",
    "# Import the StandardScaler\n",
    "...\n",
    "\n",
    "# Scale the features and set the values to a new variable\n",
    "scaler = ...\n",
    "scaled_train_features = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@solution"
    ]
   },
   "outputs": [],
   "source": [
    "# Define our features\n",
    "features = echo_tracks.drop(columns=['genre_top', 'track_id']) \n",
    "\n",
    "# Define our labels\n",
    "labels = echo_tracks['genre_top']\n",
    "\n",
    "# Import the StandardScaler\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# Scale the features and set the values to a new variable\n",
    "scaler = StandardScaler()\n",
    "scaled_train_features = scaler.fit_transform(features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@tests"
    ]
   },
   "outputs": [],
   "source": [
    "%%nose\n",
    "\n",
    "import sys\n",
    "\n",
    "def test_dropped_columns():\n",
    "    try:\n",
    "        pd.testing.assert_frame_equal(features, echo_tracks.drop(columns=['genre_top', 'track_id']))\n",
    "    except AssertionError:\n",
    "        assert False, 'Use the .drop method to remove the genre_top and track_id columns.'\n",
    "        \n",
    "def test_labels_df():\n",
    "    try:\n",
    "        pd.testing.assert_series_equal(labels, echo_tracks['genre_top'])\n",
    "    except AssertionError:\n",
    "        assert False, 'Does your labels DataFrame only contain the genre_top column?'\n",
    "        \n",
    "def test_standardscaler_import():\n",
    "    assert 'sklearn.preprocessing' in list(sys.modules.keys()), \\\n",
    "        'The StandardScaler can be imported from sklearn.preprocessing.'\n",
    "        \n",
    "def test_scaled_train_features():\n",
    "#     assert scaled_train_features.shape == (4802, 8) and round(scaled_train_features[0][0], 2) == -0.19, \\\n",
    "    assert (scaled_train_features == scaler.fit_transform(features)).all(), \\\n",
    "        \"Use the StandardScaler's fit_transform method to scale your features.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "type:NotebookTask"
    ]
   },
   "source": [
    "## 4. Principal Component Analysis on our scaled data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@context"
    ]
   },
   "source": [
    "Now that we have preprocessed our data, we are ready to use PCA to determine by how much we can reduce the dimensionality of our data. We can use **scree-plots** and **cumulative explained ratio plots** to find the number of components to use in further analyses.\n",
    "\n",
    "Scree-plots display the number of components against the variance explained by each component, sorted in descending order of variance. Scree-plots help us get a better sense of which components explain a sufficient amount of variance in our data. When using scree plots, an 'elbow' (a steep drop from one data point to the next) in the plot is typically used to decide on an appropriate cutoff."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@instructions"
    ]
   },
   "source": [
    "Use PCA to determine the explained variance of our features.\n",
    "- Import the `matplotlib.pyplot` module as `plt`, and our `PCA()` class from `sklearn.decomposition`\n",
    "- Create our PCA class using `PCA()`, fit the model on our `scaled_train_features` using `PCA.fit()`, and retrieve the explained variance ratio\n",
    "- Make a scree plot of the variance explained by each component\n",
    "\n",
    "<hr>\n",
    "\n",
    "We run PCA on all our features at first, which is done by default if `n_components` is not specified.\n",
    "\n",
    "Helpful links:\n",
    "- sklearn PCA [documentation](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html)\n",
    "- matplotlib `bar` plot [documentation](https://matplotlib.org/api/_as_gen/matplotlib.axes.Axes.bar.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@hint"
    ]
   },
   "source": [
    "You can get the number of components and the explained variance ratio of an array of features named `my_features` from a `PCA` object called `pca`  like this after fitting `pca`.\n",
    "\n",
    "```python\n",
    "pca.fit(my_features)\n",
    "print(pca.explained_variance_ratio_)\n",
    "print(pca.n_components_)\n",
    "```\n",
    "\n",
    "You can create a barplot by passing first the x-values which hold x-coordinates of each bar and then the corresponding y-values which hold the value of each bar, like so:\n",
    "\n",
    "```python\n",
    "fig, ax = plt.subplots()\n",
    "ax.bar(range(6), [5,1,0,2,3,0])\n",
    "```\n",
    "\n",
    "When creating the plot, keep in mind that the number of components goes on the x-axis and the explained variance on the y-axis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# This is just to make plots appear in the notebook\n",
    "%matplotlib inline\n",
    "\n",
    "# Import our plotting module, and PCA class\n",
    "#... YOUR CODE ...\n",
    "\n",
    "# Get our explained variance ratios from PCA using all features\n",
    "pca = ...\n",
    "...\n",
    "exp_variance = ...\n",
    "\n",
    "# plot the explained variance using a barplot\n",
    "fig, ax = plt.subplots()\n",
    "ax.bar(..., ...)\n",
    "ax.set_xlabel('Principal Component #')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@solution"
    ]
   },
   "outputs": [],
   "source": [
    "# This is just to make plots appear in the notebook\n",
    "%matplotlib inline\n",
    "\n",
    "# Import our plotting module, and PCA class\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "# Get our explained variance ratios from PCA using all features\n",
    "pca = PCA()\n",
    "pca.fit(scaled_train_features)\n",
    "exp_variance = pca.explained_variance_ratio_\n",
    "\n",
    "# plot the explained variance using a barplot\n",
    "fig, ax = plt.subplots()\n",
    "ax.bar(range(pca.n_components_), exp_variance)\n",
    "ax.set_xlabel('Principal Component #')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@tests"
    ]
   },
   "outputs": [],
   "source": [
    "%%nose\n",
    "\n",
    "import sklearn\n",
    "import numpy as np\n",
    "import sys\n",
    "\n",
    "def test_pca_import():\n",
    "    assert ('sklearn.decomposition' in list(sys.modules.keys())), \\\n",
    "        'Have you imported the PCA object from sklearn.decomposition?'\n",
    "\n",
    "def test_pca_obj():\n",
    "    assert isinstance(pca, sklearn.decomposition.PCA), \\\n",
    "        \"Use scikit-learn's PCA() object to create your own PCA object here.\"\n",
    "        \n",
    "def test_exp_variance():\n",
    "    rounded_array = np.array([0.24, 0.18, 0.14, 0.13, 0.11, 0.08, 0.07, 0.05])\n",
    "    rounder = lambda t: round(t, ndigits = 2)\n",
    "    vectorized_round = np.vectorize(rounder)\n",
    "    assert all(vectorized_round(exp_variance) == rounded_array), \\\n",
    "        'Following the PCA fit, the explained variance ratios can be obtained via the explained_variance_ratio_ method.'\n",
    "        \n",
    "def test_scree_plot():\n",
    "    expected_xticks = [float(n) for n in list(range(-1, 9))]\n",
    "    assert list(ax.get_xticks()) == expected_xticks, \\\n",
    "        'Plot the number of pca components (on the x-axis) against the explained variance (on the y-axis).'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "type:NotebookTask"
    ]
   },
   "source": [
    "## 5. Further visualization of PCA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@context"
    ]
   },
   "source": [
    "Unfortunately, there does not appear to be a clear elbow in this scree plot, which means it is not straightforward to find the number of intrinsic dimensions using this method. \n",
    "\n",
    "But all is not lost! Instead, we can also look at the **cumulative explained variance plot** to determine how many features are required to explain, say, about 85% of the variance (cutoffs are somewhat arbitrary here, and usually decided upon by 'rules of thumb'). Once we determine the appropriate number of components, we can perform PCA with that many components, ideally reducing the dimensionality of our data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@instructions"
    ]
   },
   "source": [
    "Plot the cumulative explained variance of our PCA.\n",
    "- Import the `numpy` package as `np`.\n",
    "- Calculate the cumulative sums of our explained variance using `np.cumsum()`.\n",
    "- Plot the cumulative explained variances using `ax.plot` and look for the number of components at which we can account for >85% of our variance; assign this to `n_components`.\n",
    "- Perform PCA using `n_components` and project our data onto these components.\n",
    "\n",
    "<hr>\n",
    "\n",
    "Helpful links:\n",
    "- numpy `cumsum()` function [documentation](https://docs.scipy.org/doc/numpy/reference/generated/numpy.cumsum.html)\n",
    "- sklearn PCA [documentation](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@hint"
    ]
   },
   "source": [
    "Don't forget that Python indexing starts at 0 when looking at your plot.\n",
    "\n",
    "You can use a `PCA` object that has been trained on an array named `my_features` to then project the data onto the components like this:\n",
    "\n",
    "```python\n",
    "pca.transform(my_features)\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# Import numpy\n",
    "...\n",
    "\n",
    "# Calculate the cumulative explained variance\n",
    "cum_exp_variance = ...\n",
    "\n",
    "# Plot the cumulative explained variance and draw a dashed line at 0.85.\n",
    "fig, ax = plt.subplots()\n",
    "...\n",
    "ax.axhline(y=0.85, linestyle='--')\n",
    "\n",
    "# choose the n_components where about 85% of our variance can be explained\n",
    "n_components = ...\n",
    "\n",
    "# Perform PCA with the chosen number of components and project data onto components\n",
    "pca = PCA(n_components, random_state=10)\n",
    "pca.fit(scaled_train_features)\n",
    "pca_projection = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@solution"
    ]
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Calculate the cumulative explained variance\n",
    "cum_exp_variance = np.cumsum(exp_variance)\n",
    "\n",
    "# Plot the cumulative explained variance and draw a dashed line at 0.85.\n",
    "fig, ax = plt.subplots()\n",
    "ax.plot(cum_exp_variance)\n",
    "ax.axhline(y=0.85, linestyle='--')\n",
    "\n",
    "# choose the n_components where about 85% of our variance can be explained\n",
    "n_components = 6\n",
    "\n",
    "# Perform PCA with the chosen number of components and project data onto components\n",
    "pca = PCA(n_components, random_state=10)\n",
    "pca.fit(scaled_train_features)\n",
    "pca_projection = pca.transform(scaled_train_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@tests"
    ]
   },
   "outputs": [],
   "source": [
    "%%nose\n",
    "\n",
    "import sys\n",
    "\n",
    "def test_np_import():\n",
    "    assert 'numpy' in list(sys.modules.keys()), \\\n",
    "        'Have you imported numpy?'\n",
    "    \n",
    "\n",
    "def test_cumsum():\n",
    "    cum_exp_variance_correct = np.cumsum(exp_variance)\n",
    "    assert all(cum_exp_variance == cum_exp_variance_correct), \\\n",
    "    'Use np.cumsum to calculate the cumulative sum of the exp_variance array.'\n",
    "    \n",
    "    \n",
    "def test_n_comp():\n",
    "    assert n_components == 6, \\\n",
    "    ('Check the values in cum_exp_variance if it is difficult '\n",
    "    'to determine the number of components from the plot.')\n",
    "    \n",
    "    \n",
    "def test_trans_pca():\n",
    "    pca_test = PCA(n_components, random_state=10)\n",
    "    pca_test.fit(scaled_train_features)\n",
    "    assert (pca_projection == pca_test.transform(scaled_train_features)).all(), \\\n",
    "    'Transform the scaled features and assign them to the pca_projection variable.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "type:NotebookTask"
    ]
   },
   "source": [
    "## 6. Train a decision tree to classify genre"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@context"
    ]
   },
   "source": [
    "Now we can use the lower dimensional PCA projection of the data to classify songs into genres. To do that, we first need to split our dataset into 'train' and 'test' subsets, where the 'train' subset will be used to train our model while the 'test' dataset allows for model performance validation.\n",
    "\n",
    "Here, we will be using a simple algorithm known as a decision tree. Decision trees are rule-based classifiers that take in features and follow a 'tree structure' of binary decisions to ultimately classify a data point into one of two or more categories. In addition to being easy to both use and interpret, decision trees allow us to visualize the 'logic flowchart' that the model generates from the training data.\n",
    "\n",
    "Here is an example of a decision tree that demonstrates the process by which an input image (in this case, of a shape) might be classified based on the number of sides it has and whether it is rotated.\n",
    "\n",
    "<img src=\"img/simple_decision_tree.png\" alt=\"Decision Tree Flow Chart Example\" width=\"350px\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@instructions"
    ]
   },
   "source": [
    "Prepare our training and test sets and train our first classifier.\n",
    "- Import the `train_test_split()` function from `sklearn.model_selection` module\n",
    "- Import the `DecisionTreeClassifier` from `sklearn.tree` module\n",
    "- Split our projected data into train and test, features and labels, respectively using `train_test_split()` with `random_state=10`.\n",
    "- Create our decision tree classifier using `DecisionTreeClassifier()` and `random_state=10` and train the model using the `model.fit()` notation\n",
    "- Find the predicted labels of the `test_features` from our trained model using the `model.predict()` notation.\n",
    "\n",
    "<hr>\n",
    "\n",
    "Helpful links:\n",
    "- scikit-learn `train_test_split()` function [documentation](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html)\n",
    "- scikit-learn `DecisionTreeClassifier()` class [documentation](http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@hint"
    ]
   },
   "source": [
    "The `train_test_split()` function can be used to split `features` and `labels` as training and test sets like this after importing it from `sklearn.model_selection`:\n",
    "\n",
    "```python\n",
    "from sklearn.model_selection import train_test_split\n",
    "train_features, test_features, train_labels, test_labels = train_test_split(features, labels)\n",
    "```\n",
    "\n",
    "You can fit your decision tree classifier named `tree` and train on your `train_features` and `train_labels` like this:\n",
    "\n",
    "```python\n",
    "tree = DecisionTreeClassifier()\n",
    "tree.fit(train_features, train_labels)\n",
    "```\n",
    "\n",
    "Don't forget to set your `random_state` for any function that might use some randomness in order to make your result reproducible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# Import train_test_split function and Decision tree classifier\n",
    "# ... YOUR CODE ...\n",
    "\n",
    "# Split our data\n",
    "train_features, test_features, train_labels, test_labels = ...\n",
    "\n",
    "# Train our decision tree\n",
    "tree = ...\n",
    "...\n",
    "\n",
    "# Predict the labels for the test data\n",
    "pred_labels_tree = ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@solution"
    ]
   },
   "outputs": [],
   "source": [
    "# Import train_test_split function and Decision tree classifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "# Split our data\n",
    "train_features, test_features, train_labels, test_labels = train_test_split(\n",
    "    pca_projection, labels, random_state=10)\n",
    "\n",
    "# Train our decision tree\n",
    "tree = DecisionTreeClassifier(random_state=10)\n",
    "tree.fit(train_features, train_labels)\n",
    "\n",
    "# Predict the labels for the test data\n",
    "pred_labels_tree = tree.predict(test_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@tests"
    ]
   },
   "outputs": [],
   "source": [
    "%%nose\n",
    "\n",
    "import sys\n",
    "\n",
    "def test_train_test_split_import():\n",
    "    assert 'sklearn.model_selection' in list(sys.modules.keys()), \\\n",
    "        'Have you imported train_test_split from sklearn.model_selection?'\n",
    "\n",
    "    \n",
    "def test_decision_tree_import():\n",
    "    assert 'sklearn.tree' in list(sys.modules.keys()), \\\n",
    "        'Have you imported DecisionTreeClassifier from sklearn.tree?'\n",
    "    \n",
    "    \n",
    "def test_train_test_split():\n",
    "    train_test_res = train_test_split(pca_projection, labels, random_state=10)\n",
    "    assert (train_features == train_test_res[0]).all(), \\\n",
    "        'Did you correctly call the train_test_split function?'\n",
    "    \n",
    "    \n",
    "def test_tree():\n",
    "    assert tree.get_params() == DecisionTreeClassifier(random_state=10).get_params(), \\\n",
    "        'Did you create the decision tree correctly?'\n",
    "    \n",
    "    \n",
    "def test_tree_fit():\n",
    "    assert hasattr(tree, 'classes_'), \\\n",
    "        'Did you fit the tree to the training data?'\n",
    "    \n",
    "    \n",
    "def test_tree_pred():\n",
    "    assert (pred_labels_tree == 'Rock').sum() == 971, \\\n",
    "        'Did you correctly use the fitted tree object to make a prediction from the test features?'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "type:NotebookTask"
    ]
   },
   "source": [
    "## 7. Compare our decision tree to a logistic regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@context"
    ]
   },
   "source": [
    "Although our tree's performance is decent, it's a bad idea to immediately assume that it's therefore the perfect tool for this job -- there's always the possibility of other models that will perform even better! It's always a worthwhile idea to at least test a few other algorithms and find the one that's best for our data.\n",
    "\n",
    "Sometimes simplest is best, and so we will start by applying **logistic regression**. Logistic regression makes use of what's called the logistic function to calculate the odds that a given data point belongs to a given class. Once we have both models, we can compare them on a few performance metrics, such as false positive and false negative rate (or how many points are inaccurately classified). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@instructions"
    ]
   },
   "source": [
    "Train our logistic regression and compare the performance with our decision tree.\n",
    "- Create our logistic regression model using LogisticRegression() and set `random_state` to `10`.\n",
    "- Train the model using the `model.fit()` notation and assign the predicted labels for the `test_features` to `pred_labels_logit`.\n",
    "- Import the `classification_report` from the `sklearn.metrics` package\n",
    "- Print the classification reports for our trained Decision Tree and Logistic Regression models. \n",
    "\n",
    "<hr>\n",
    "\n",
    "Helpful links:\n",
    "- scikit-learn `LogisticRegression()` class [documentation](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html)\n",
    "- scikit-learn `classification_report()` function [documentation](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@hint"
    ]
   },
   "source": [
    "You can fit your `LogisticRegression` named `my_logreg` on `features` and `labels` like this after instantiating an instance of `LogisticRegression`:\n",
    "\n",
    "```python\n",
    "my_logreg = LogisticRegresion()\n",
    "my_logreg.fit(features, labels)\n",
    "```\n",
    "\n",
    "You can get the classification report from the labels of each data point called `labels` and the predicted labels called `predicted` like this:\n",
    "\n",
    "```python\n",
    "report = classification_report(labels, predicted)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# Import LogisticRegression\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# Train our logistic regression and predict labels for the test set\n",
    "logreg = ...\n",
    "...\n",
    "pred_labels_logit = ...\n",
    "\n",
    "# Create the classification report for both models\n",
    "from sklearn.metrics import classification_report\n",
    "class_rep_tree = ...\n",
    "class_rep_log = ...\n",
    "\n",
    "print(\"Decision Tree: \\n\", class_rep_tree)\n",
    "print(\"Logistic Regression: \\n\", class_rep_log)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@solution"
    ]
   },
   "outputs": [],
   "source": [
    "# Import LogisticRegression\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# Train our logisitic regression\n",
    "logreg = LogisticRegression(random_state=10)\n",
    "logreg.fit(train_features, train_labels)\n",
    "pred_labels_logit = logreg.predict(test_features)\n",
    "\n",
    "# Create the classification report for both models\n",
    "from sklearn.metrics import classification_report\n",
    "class_rep_tree = classification_report(test_labels, pred_labels_tree)\n",
    "class_rep_log = classification_report(test_labels, pred_labels_logit)\n",
    "\n",
    "print(\"Decision Tree: \\n\", class_rep_tree)\n",
    "print(\"Logistic Regression: \\n\", class_rep_log)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@tests"
    ]
   },
   "outputs": [],
   "source": [
    "%%nose\n",
    "\n",
    "def test_logreg():\n",
    "    assert logreg.get_params() == LogisticRegression(random_state=10).get_params(), \\\n",
    "        'The logreg variable should be created using LogisticRegression().'\n",
    "\n",
    "    \n",
    "def test_logreg_pred():\n",
    "    assert abs((pred_labels_logit == 'Rock').sum() - 1028) < 10, \\\n",
    "        'The labels should be predicted from the test_features.'\n",
    "    \n",
    "    \n",
    "def test_class_rep_tree():\n",
    "    assert isinstance(class_rep_tree, str), \\\n",
    "        'Did you create the classification report correctly for the decision tree?'\n",
    "    \n",
    "    \n",
    "def test_class_rep_log():\n",
    "    assert isinstance(class_rep_tree, str), \\\n",
    "        'Did you create the classification report correctly for the logistic regression?'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "type:NotebookTask"
    ]
   },
   "source": [
    "## 8. Balance our data for greater performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@context"
    ]
   },
   "source": [
    "Both our models do similarly well, boasting an average precision of 87% each. However, looking at our classification report, we can see that rock songs are fairly well classified, but hip-hop songs are disproportionately misclassified as rock songs. \n",
    "\n",
    "Why might this be the case? Well, just by looking at the number of data points we have for each class, we see that we have far more data points for the rock classification than for hip-hop, potentially skewing our model's ability to distinguish between classes. This also tells us that most of our model's accuracy is driven by its ability to classify just rock songs, which is less than ideal.\n",
    "\n",
    "To account for this, we can weight the value of a correct classification in each class inversely to the occurrence of data points for each class. Since a correct classification for \"Rock\" is not more important than a correct classification for \"Hip-Hop\" (and vice versa), we only need to account for differences in _sample size_ of our data points when weighting our classes here, and not relative importance of each class. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@instructions"
    ]
   },
   "source": [
    "Balance our dataset such that the number of tracks for each genre is the same.\n",
    "- Subset only the hip-hop tracks from `echo_tracks` using `df.loc[]`, and the same for the rock tracks\n",
    "- Sample `rock_only` such that there is the same number of data points as there are hip-hop data points. Set the `random_state` to 10.\n",
    "- Concatenate the `rock_only` and `hop_only` (in that order) DataFrames using the `pd.concat()` function by passing a list of these DataFrames.\n",
    "- Redefine our train and test sets using `train_test_split` with the PCA projection of the balanced dataframe.\n",
    "\n",
    "<hr>\n",
    "\n",
    "Helpful links:\n",
    "- pandas `DataFrame.loc[]` indexing [documentation](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.loc.html)\n",
    "- pandas `concat()` function [documentation](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.concat.html)\n",
    "- pandas `DataFrame.sample()` method [documentation](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.sample.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@hint"
    ]
   },
   "source": [
    "You can filter rows of a dataframe named `df` which satisfy a boolean statement where the value in a column named `col1` is greater than 10, and then sample only 20 of these rows like this:\n",
    "\n",
    "```python\n",
    "df_filter_sample = df.loc[df['col1'] > 10].sample(20, random_state=0)\n",
    "```\n",
    "\n",
    "You can concatenate two DataFrames named `df1` and `df2` like this:\n",
    "\n",
    "```python\n",
    "df_concat = pd.concat([df1, df2])\n",
    "```\n",
    "\n",
    "Whenever you're using a function that takes a random sample, or shuffles data, you should set the `random_state` if you want your result to be reproducible. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# Subset only the hip-hop tracks, and then only the rock tracks\n",
    "hop_only = ...\n",
    "rock_only = ...\n",
    "\n",
    "# sample the rocks songs to be the same number as there are hip-hop songs\n",
    "rock_only = ...\n",
    "\n",
    "# concatenate the dataframes rock_only and hop_only\n",
    "rock_hop_bal = ...\n",
    "\n",
    "# The features, labels, and pca projection are created for the balanced dataframe\n",
    "features = rock_hop_bal.drop(['genre_top', 'track_id'], axis=1) \n",
    "labels = rock_hop_bal['genre_top']\n",
    "pca_projection = pca.fit_transform(scaler.fit_transform(features))\n",
    "\n",
    "# Redefine the train and test set with the pca_projection from the balanced data\n",
    "train_features, test_features, train_labels, test_labels = train_test_split(..., random_state=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@solution"
    ]
   },
   "outputs": [],
   "source": [
    "# Subset a balanced proportion of data points\n",
    "hop_only = echo_tracks.loc[echo_tracks['genre_top'] == 'Hip-Hop']\n",
    "rock_only = echo_tracks.loc[echo_tracks['genre_top'] == 'Rock']\n",
    "\n",
    "# subset only the rock songs, and take a sample the same size as there are hip-hop songs\n",
    "rock_only = rock_only.sample(hop_only.shape[0], random_state=10)\n",
    "\n",
    "# concatenate the dataframes hop_only and rock_only\n",
    "rock_hop_bal = pd.concat([rock_only, hop_only])\n",
    "\n",
    "# The features, labels, and pca projection are created for the balanced dataframe\n",
    "features = rock_hop_bal.drop(['genre_top', 'track_id'], axis=1) \n",
    "labels = rock_hop_bal['genre_top']\n",
    "pca_projection = pca.fit_transform(scaler.fit_transform(features))\n",
    "\n",
    "# Redefine the train and test set with the pca_projection from the balanced data\n",
    "train_features, test_features, train_labels, test_labels = train_test_split(\n",
    "    pca_projection, labels, random_state=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@tests"
    ]
   },
   "outputs": [],
   "source": [
    "%%nose\n",
    "\n",
    "def test_hop_only():\n",
    "    try:\n",
    "        pd.testing.assert_frame_equal(hop_only, echo_tracks.loc[echo_tracks['genre_top'] == 'Hip-Hop'])\n",
    "    except AssertionError:\n",
    "        assert False, \"The hop_only data frame was not assigned correctly.\"\n",
    "        \n",
    "\n",
    "def test_rock_only():\n",
    "    try:\n",
    "        pd.testing.assert_frame_equal(\n",
    "            rock_only, echo_tracks.loc[echo_tracks['genre_top'] == 'Rock'].sample(hop_only.shape[0], random_state=10))\n",
    "    except AssertionError:\n",
    "        assert False, \"The rock_only data frame was not assigned correctly.\"\n",
    "        \n",
    "        \n",
    "def test_rock_hop_bal():\n",
    "    hop_only = echo_tracks.loc[echo_tracks['genre_top'] == 'Hip-Hop']\n",
    "    rock_only = echo_tracks.loc[echo_tracks['genre_top'] == 'Rock'].sample(hop_only.shape[0], random_state=10)\n",
    "    try:\n",
    "        pd.testing.assert_frame_equal(\n",
    "            rock_hop_bal, pd.concat([rock_only, hop_only]))\n",
    "    except AssertionError:\n",
    "        assert False, \"The rock_hop_bal data frame was not assigned correctly.\"\n",
    "        \n",
    "        \n",
    "def test_train_features():\n",
    "    assert round(train_features[0][0], 4) == -0.7311 and round(train_features[-1][-1], 4) == 0.5624, \\\n",
    "    'The train_test_split was not performed correctly.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "type:NotebookTask"
    ]
   },
   "source": [
    "## 9. Does balancing our dataset improve model bias?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@context"
    ]
   },
   "source": [
    "We've now balanced our dataset, but in doing so, we've removed a lot of data points that might have been crucial to training our models. Let's test to see if balancing our data improves model bias towards the \"Rock\" classification while retaining overall classification performance. \n",
    "\n",
    "Note that we have already reduced the size of our dataset and will go forward without applying any dimensionality reduction. In practice, we would consider dimensionality reduction more rigorously when dealing with vastly large datasets and when computation times become prohibitively large."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@instructions"
    ]
   },
   "source": [
    "Compare the two model performances on the balanced data.\n",
    "- Create and train your decision tree using `DecisionTreeClassifier()` and a random state of `10`, then predict on the `test_features`.\n",
    "- Create and train your logistic regression using `LogisticRegression()` and a random state of `10`, then predict on the `test_features`.\n",
    "- Compare the performance of the two models using `classification_report()`. \n",
    "\n",
    "<hr>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@hint"
    ]
   },
   "source": [
    "You can predict labels on features called `my_features` using a scikit-learn object called `my_model` after fitting `my_model` on the training data, `train_features` and `train_labels` like this:\n",
    "\n",
    "```python\n",
    "my_model.fit(train_features, train_labels)\n",
    "predicted = my_model.predict(my_features)\n",
    "```\n",
    "\n",
    "Most of the classifiers initialize or run with some degree of randomness, you should set your `random_state` to make the result reproducible. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "# Train our decision tree on the balanced data\n",
    "tree = ...\n",
    "...\n",
    "pred_labels_tree = ...\n",
    "\n",
    "# Train our logistic regression on the balanced data\n",
    "logreg = ...\n",
    "...\n",
    "pred_labels_logit = ...\n",
    "\n",
    "# Compare the models\n",
    "print(\"Decision Tree: \\n\", classification_report(...))\n",
    "print(\"Logistic Regression: \\n\", classification_report(...))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@solution"
    ]
   },
   "outputs": [],
   "source": [
    "# Train our decision tree on the balanced data\n",
    "tree = DecisionTreeClassifier(random_state=10)\n",
    "tree.fit(train_features, train_labels)\n",
    "pred_labels_tree = tree.predict(test_features)\n",
    "\n",
    "# Train our logistic regression on the balanced data\n",
    "logreg = LogisticRegression(random_state=10)\n",
    "logreg.fit(train_features, train_labels)\n",
    "pred_labels_logit = logreg.predict(test_features)\n",
    "\n",
    "# compare the models\n",
    "print(\"Decision Tree: \\n\", classification_report(test_labels, pred_labels_tree))\n",
    "print(\"Logistic Regression: \\n\", classification_report(test_labels, pred_labels_logit))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@tests"
    ]
   },
   "outputs": [],
   "source": [
    "%%nose\n",
    "\n",
    "def test_tree_bal():\n",
    "    assert (pred_labels_tree == 'Rock').sum() == 226, \\\n",
    "    'The pred_labels_tree variable should contain the predicted labels from the test_features.'\n",
    "    \n",
    "    \n",
    "def test_logit_bal():\n",
    "    assert (pred_labels_logit == 'Rock').sum() == 221, \\\n",
    "    'The pred_labels_logit variable should contain the predicted labels from the test_features.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "type:NotebookTask"
    ]
   },
   "source": [
    "## 10. Using cross-validation to evaluate our models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@context"
    ]
   },
   "source": [
    "Success! Balancing our data has removed bias towards the more prevalent class. To get a good sense of how well our models are actually performing, we can apply what's called **cross-validation** (CV). This step allows us to compare models in a more rigorous fashion.\n",
    "\n",
    "Since the way our data is split into train and test sets can impact model performance, CV attempts to split the data multiple ways and test the model on each of the splits. Although there are many different CV methods, all with their own advantages and disadvantages, we will use what's known as **K-fold** CV here. K-fold first splits the data into K different, equally sized subsets. Then, it iteratively uses each subset as a test set while using the remainder of the data as train sets. Finally, we can then aggregate the results from each fold for a final model performance score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@instructions"
    ]
   },
   "source": [
    "Use cross-validation to get a better sense of your model performance.\n",
    "- Create a variable called `kf` to store your cv using `KFold()` with `10` folds.\n",
    "- Train each of your models using `cross_val_score()` on the original `pca_projection` and `labels` variables.\n",
    "- Print the mean of the cross-validation scores for each model using `np.mean()`.\n",
    "\n",
    "<hr>\n",
    "\n",
    "Helpful links:\n",
    "- `KFold()` [documentation](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html)\n",
    "- `cross_val_score()` [documentation](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html#sklearn.model_selection.cross_val_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "@hint"
    ]
   },
   "source": [
    "You can train and cross-validate a scikit-learn model named `my_model` after defining our KFold cross-validation named `kf` like this:\n",
    "\n",
    "```python\n",
    "kf = KFold(5)\n",
    "cv_score = cross_val_score(my_model, features, labels, cv=kf)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@sample_code"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import KFold, cross_val_score\n",
    "\n",
    "# Set up our K-fold cross-validation\n",
    "kf = ...\n",
    "\n",
    "tree = DecisionTreeClassifier(random_state=10)\n",
    "logreg = LogisticRegression(random_state=10)\n",
    "\n",
    "# Train our models using KFold cv\n",
    "tree_score = ...\n",
    "logit_score = ...\n",
    "\n",
    "# Print the mean of each array of scores\n",
    "print(\"Decision Tree:\", ..., \"Logistic Regression:\", ...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@solution"
    ]
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import KFold, cross_val_score\n",
    "\n",
    "# Set up our K-fold cross-validation\n",
    "kf = KFold(10)\n",
    "\n",
    "tree = DecisionTreeClassifier(random_state=10)\n",
    "logreg = LogisticRegression(random_state=10)\n",
    "\n",
    "# Train our models using KFold cv\n",
    "tree_score = cross_val_score(tree, pca_projection, labels, cv=kf)\n",
    "logit_score = cross_val_score(logreg, pca_projection, labels, cv=kf)\n",
    "\n",
    "# Print the mean of each array o scores\n",
    "print(\"Decision Tree:\", np.mean(tree_score), \"Logistic Regression:\", np.mean(logit_score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "@tests"
    ]
   },
   "outputs": [],
   "source": [
    "%%nose\n",
    "\n",
    "def test_kf():\n",
    "    assert kf.__repr__() == 'KFold(n_splits=10, random_state=None, shuffle=False)', \\\n",
    "    'The k-fold cross-validation was not setup correctly.'\n",
    "    \n",
    "    \n",
    "def test_tree_score():\n",
    "    assert np.isclose(round((tree_score.sum() / tree_score.shape[0]), 4), 0.7242, atol=1e-3), \\\n",
    "    'The tree_score was not calculated correctly.'\n",
    "    \n",
    "    \n",
    "def test_log_score():\n",
    "    assert np.isclose(round((logit_score.sum() / logit_score.shape[0]), 4), 0.7753, atol=1e-3), \\\n",
    "    'The logit_score was not calculated correctly.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*The recommended number of tasks in a DataCamp Project is between 8 and 10, so feel free to add more if necessary. You can't have more than 12 tasks.*"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}