2024-01-19_school_budgeting_with_machine_learning_in_python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9490a0f9-06b1-4a60-9dfb-3b94fdf23494",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "- Notebook Author: [Trenton McKinney][1]\n",
    "- Course: [DataCamp: Case Study: School Budgeting with Machine Learning in Python][2] - This notebook was created as a reproducible reference\n",
    "- If you find the content beneficial, consider a [DataCamp Subscription][3]\n",
    "\n",
    "[1]: https://trenton3983.github.io/\n",
    "[2]: https://app.datacamp.com/learn/courses/case-study-school-budgeting-with-machine-learning-in-python\n",
    "[3]: https://www.datacamp.com/pricing?period=yearly"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bb2056a3b466fc1",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "# Course Description\n",
    "\n",
    "Data science isn't just for predicting ad-clicks-it's also useful for social impact! This course is a case study from a machine learning competition on DrivenData. You'll explore a problem related to school district budgeting. By building a model to automatically classify items in a school's budget, it makes it easier and faster for schools to compare their spending with other schools. In this course, you'll begin by building a baseline model that is a simple, first-pass approach. In particular, you'll do some natural language processing to prepare the budgets for modeling. Next, you'll have the opportunity to try your own techniques and see how they compare to participants from the competition. Finally, you'll see how the winner was able to combine a number of expert techniques to build the most accurate model.\n",
    "\n",
    "The case study discusses the use of several types of models for a school budgeting problem with machine learning in Python. The models used include:\r\n",
    "\r\n",
    "1. Multi-class logistic regression: This is used as a starting point to quickly go from raw data to predictions. The model treats each label column independently and trains a logistic regression classifier for each label column separately.\r\n",
    "\r\n",
    "2. Random Forest Classifier: This model is mentioned as a possible choice to use in a scikit-learn Pipeline for classification. It's known for its performance and ease of use in classification tasks.\r\n",
    "\r\n",
    "3. Naive Bayes Classifier: This is another type of model that could be used in a scikit-learn Pipeline. It is based on applying Bayes' theorem with strong (naïve) independence assumptions between the features.\r\n",
    "\r\n",
    "4. K-Nearest Neighbors (k-NN) Classifier: This model is listed as an alternative that could be experimented with in the pipeline. It classifies data based on the closest training examples in the feature space.\r\n",
    "\r\n",
    "5. Ensemble methods: The document also alludes to the use of ensemble methods, which combine the predictions of multiple machine learning models to improve accuracy. Specifically, it mentions the winner of a competition using an ensemble of many models for classification.\r\n",
    "\r\n",
    "6. Deep Convolutional Neural Network: It is indicated that the winner of a machine learning competition mentioned in the case study used a 500-layer deep convolutional neural network to master the budget data.\r\n",
    "\r\n",
    "These models are typical in machine learning tasks that involve classification and pattern recognition. They were chosen for their effectiveness in dealing with the specific characteristics of the school budget data, such as its multi-class nature and the presence of both numerical and textual data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c7c6063-3393-4a95-b9f6-e41bf3ceee0a",
   "metadata": {},
   "source": [
    "# Imports\n",
    "\n",
    "- DataCamp package versions vs. local versions:\n",
    "  - `sklearn v0.20.4` vs. `1.3.2`\n",
    "  - `numpy v1.17.4` vs. `1.26.3`\n",
    "  - `pandas v0.24.2` vs. `2.2.1`\n",
    "  - `matplotlib v3.1.2` vs. `3.8.1`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efec3b43-ff5b-4cba-b0bc-75701ec7b24b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from typing import List, Union\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c133d10-081b-43a9-bf0f-5063bb1b0e52",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "from sklearn.feature_extraction.text import CountVectorizer, HashingVectorizer\n",
    "\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "from sklearn.preprocessing import FunctionTransformer, MaxAbsScaler, PolynomialFeatures\n",
    "from sklearn.pipeline import FeatureUnion\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.feature_selection import chi2, SelectKBest\n",
    "from sklearn.metrics import make_scorer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85079ff5-7209-4ec2-a92a-ec9247162bad",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(r'D:\\users\\trenton\\Dropbox\\PythonProjects\\DataCamp\\functions')\n",
    "\n",
    "from multilabel import multilabel_train_test_split\n",
    "from score_sub import score_submission, _multi_multi_log_loss\n",
    "from sparse_interactions import SparseInteractions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ace076f-4df9-4283-a268-ab8ca45405b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option('display.max_columns', 700)\n",
    "pd.set_option('display.max_rows', 400)\n",
    "pd.set_option('display.min_rows', 10)\n",
    "pd.set_option('display.expand_frame_repr', True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5698fbde-4512-4417-97aa-bb18e8d8d17b",
   "metadata": {},
   "source": [
    "# Saving data from DataCamp to a local csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b2bf183-f310-4383-9a58-4537e197982f",
   "metadata": {},
   "source": [
    "```python\n",
    "# output the data to a format that can be copied and pasted into notepad++\n",
    "# find all non and replace with np.nan\n",
    "# copy all of lists and replace ... in data = np.array([...])\n",
    "# print the columns names df.columns.tolist() and then create a variable in a notebook\n",
    "# create a dataframe with df = pd.DataFrame(data=data, columns=columns)\n",
    "\n",
    "# create the dataframe in the DataCamp console\n",
    "df = pd.read_csv('TrainingData.csv')  # do not specify index_col=0\n",
    "\n",
    "# print each row as a list with a comma at the end and then copy all of the lists into notepad++\n",
    "for r in df.to_numpy():\n",
    "    print(list(r), ',')\n",
    "\n",
    "# print the column names and copy them\n",
    "df.columns.tolist()\n",
    "\n",
    "# in jupyterlab\n",
    "data = np.array([...])  # replace ... with the cleaned lists from notepad++\n",
    "columns = ...  # pasted from DataCamp console\n",
    "\n",
    "df = pd.DataFrame(data=data, columns=columns)\n",
    "\n",
    "df.to_csv('TrainingData.csv', index=False)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3731136-2d6e-4986-b2ae-34fa1744790c",
   "metadata": {},
   "source": [
    "# Exploring the Raw Data\n",
    "\n",
    "In this chapter, you'll be introduced to the problem you'll be solving in this course. How do you accurately classify line-items in a school budget based on what that money is being used for? You will explore the raw text and numeric values in the dataset, both quantitatively and visually. And you'll learn how to measure success when trying to predict class labels for each row of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40a7635053b66a3",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# set the current working directory if needed\n",
    "cwd = os.getcwd()\n",
    "if cwd != 'D:/Users/trenton/Dropbox/PythonProjects/DataCamp':\n",
    "    os.chdir(r'D:\\users\\trenton\\Dropbox\\PythonProjects\\DataCamp')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f73a14f-5566-47e6-83f2-44da71584c93",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/2024-01-19_school_budgeting_with_machine_learning_in_python/TrainingData.csv', index_col=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3aff791-1d21-427b-a658-1028258ad240",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "563556b0-7a29-4d6b-b556-e9ac8376a9b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49000e82632f3734",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## Introducing the challenge\n",
    "\n",
    "1. Introducing the challenge: Hello DataCampers! How's it going? Really glad you've decided to join us for this course. We have an exciting journey ahead of us through some real data and some incredibly useful tips and tricks from expert data scientists. I'm Peter Bull, a data scientist and a co-founder of DrivenData. Our mission is to bring the power of data science to social impact organizations. One of the ways we do that is by running online data science challenges for non-profits, NGOs, and social enterprises. In our challenges, a global community of data scientists--like you!--competes to solve a particular problem. We'll work\n",
    "- Learn from the expert who won DrivenData's challenge\n",
    "  - Natural language processing\n",
    "  - Feature engineering\n",
    "  - Efficency boosting hashing tricks\n",
    "- Use data to have a social impact\n",
    "2. Introducing the challenge: through one of these competitions as a case-study, and we'll show you how the winner achieved the best score. In the course, we'll do some natural language processing, some feature engineering, and boost our computational efficiency. In addition to these pro-tips, we'll look at one of the ways in which we can use data to have a social impact.\n",
    "- Budgets for schools are huge, complex, and not standardized\n",
    "  - Hundreds of hours each year are spent manually labelling\n",
    "- Goal: Build a machine learning algorithm that can automate the process\n",
    "- Budget data\n",
    "  - Line-item: \"Algebra books for 8th grade students\"\n",
    "  - Labels: \"Textbooks\", \"Math\", \"Middle School\"\n",
    "- This is a supervised learning problem\n",
    "3. Introducing the challenge: School budgets in the United States are incredibly complex, and there are no standards for reporting how money is spent. Schools want to be able to measure their performance--for example, are we spending more on textbooks than our neighboring schools, and is that investment worthwhile? However to do this comparison takes hundreds of hours each year in which analysts hand-categorize each line-item. Our goal is to build a machine learning algorithm that can automate that process. For each line item, we have some text fields that tell us about the expense--for example, a line might say something like \"Algebra books for 8th grade students\". We also have the amount of the expense in dollars. This line item then has a set of labels attached to it. For example, this one might have labels like \"Textbooks,\" \"Math,\" and \"Middle School.\" These labels are our target variable. This is a supervised learning problem where we want to use correctly labeled data to build an algorithm that can suggest labels for unlabeled lines. This is in contrast to an unsupervised learning problem where we don't have labels, and we are using an algorithm to automatically which line-items might go together. For this problem,\n",
    "\n",
    "4. Over 100 target variables!: we have over 100 unique target variables that could be attached to a single line item. Because we want to predict a category for each line item, this is a classification problem. This is as opposed to a regression problem where we want to predict a numeric value for a line item--for example, predicting house prices. Here are some of the actual categories that we need to determine: Is this expense for\n",
    "\n",
    "5. Over 100 target variables!: pre-kindergarten education (which is important because it has different funding sources)? Or, is there a particular\n",
    "\n",
    "6. Over 100 target variables!: Student_Type that this expense supports? Overall, there are 9 columns with many different possible categories in each column.\n",
    "- This is a classification problem\n",
    "  - Pre_K:\n",
    "    - NO_LABEL\n",
    "    - Non PreK\n",
    "    - PreK\n",
    "  - Sharing:\n",
    "    - Leadership & Management\n",
    "    - NO_LABEL\n",
    "    - School Reported\n",
    "  - Reporting:\n",
    "    - NO_LABEL\n",
    "    - Non-School\n",
    "    - School\n",
    "  - Student_Type:\n",
    "    - Alternative\n",
    "    - At Risk\n",
    "7. How we can help: If you talk to the people who actually do this work, it is impossible for a human to label these line items with 100% accuracy. To take this into account, we don't want our algorithm to just say \"This line is for textbooks.\" We want it to say: \"It's most likely this line is for textbooks, and I'm 60% sure that it is. If it's not textbooks, I'm 30% sure it's 'office supplies.'\" By making these suggestions, analysts can prioritize their time. This is called a human-in-the-loop machine learning system.\n",
    "\n",
    "8. How we can help: We will predict a probability between 0 (the algorithm thinks this label is very unlikely for this line item) and 1 (the algorithm thinks this label is very likely). We'll take a quick break to review and then come back to talk about how to load the data.\n",
    "- Predictions will be **probabilites** for each label\n",
    "\n",
    "9. Let's practice!: We'll take a quick break to review and then come back to talk about how to load the data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c86eb33c6ab0387",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "### What category of problem is this?\n",
    "\n",
    "You're no novice to data science, but let's make sure we agree on the basics.\n",
    "\n",
    "As Peter from [DrivenData][1] explained in the video, you're going to be working with school district budget data. This data can be classified in many ways according to certain labels, e.g. `Function: Career & Academic Counseling`, or `Position_Type: Librarian`.\n",
    "\n",
    "Your goal is to develop a model that predicts the probability for each possible label by relying on some correctly labeled examples.\n",
    "\n",
    "What type of machine learning problem is this?\n",
    "\n",
    "**Answer the question**\n",
    "\n",
    "- ~~Reinforcement Learning, because the model is learning from the data through a system of rewards and punishments.~~\n",
    "- ~~Unsupervised Learning, because the model doesn't output labels with certainty.~~\n",
    "- ~~Unsupervised Learning, because not all data is correctly classified to begin with.~~\n",
    "- **Supervised Learning, because the model will be trained using labeled examples.** - **Using correctly labeled budget line items to train means this is a supervised learning problem.**\n",
    "\n",
    "[1]: https://www.drivendata.org/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6795696c3d9d179",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "### What is the goal of the algorithm?\n",
    "\n",
    "As you know from previous courses, there are different types of supervised machine learning problems. In this exercise you will tell us what type of supervised machine learning problem this is, and why you think so.\n",
    "\n",
    "Remember, your goal is to correctly label budget line items by training a supervised model to predict the probability of each possible label, taking most probable label as the correct label.\n",
    "\n",
    "**Answer the question**\n",
    "\n",
    "- ~~Regression, because the model will output probabilities.~~\n",
    "- **Classification, because predicted probabilities will be used to select a label class.** - **Specifically, we have ourselves a multi-class-multi-label classification problem (quite a mouthful!), because there are 9 broad categories that each take on many possible sub-label instances.**\n",
    "- ~~Regression, because probabilities take a continuous value between 0 and 1.~~\n",
    "- ~~Classification, because the model will output probabilities.~~"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "458b87d7192810f3",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## Exploring the data\n",
    "\n",
    "1. Exploring the data: We left off the last video by showing that we'd be predicting probabilities for each budget line item. Let's be clear what this looks like in practice.\n",
    "\n",
    "2. A column for each possible value: or example, we'll say we are predicting the hair type and eye color of people. We have the categories \"curly,\" \"straight,\" and \"wavy\" for hair and \"brown\" and \"blue\" for eyes. If we are predicting probabilities, we need a value for each possible value in each column. In this case, the target would have the columns for each hair type and for each eye color. Later in the course, we'll talk more about how to perform this transformation. We'll begin by loading our data.\n",
    "\n",
    "3. Load and preview the data: As an example, in this video we'll work with a small sample dataset. For the exercises, you'll be loading the actual school budget dataset. You've seen pandas functions for loading data before, and in this case we'll be working with a file of comma-separated values: a CSV. First, we'll import pandas giving it the alias pd so we don't have to type pandas every time we want to use a function. We'll use the function read_csv and pass the filename to our dataset. Then, the function head shows us the first 5 rows of the DataFrame and the column names. We can see this sample dataset has both numeric data and text data.\n",
    "\n",
    "4. Summarize the data: The function info tells us a bit more information about our dataset. Importantly, it tells us the datatype of each column. Columns that can be recognized as a numeric type--integers and floats--will be recognized as such. Columns that can't will get the generic type of object. Additionally, info tells us if any of the columns have missing values. As we can see, the column with_missing has 95 non-null entries, which means that there are 5 rows that are missing a value in the with_missing column. The next summary function we want to use is describe. This function gives us summary statistics of numeric columns in our data frame. In addition to also giving us a hint about missing values with count, we can get a sense for the mean, standard deviation, minimum and maximum of the column.\n",
    "\n",
    "5. Let's practice!: Now it's your turn to load the actual dataset we'll be working with using read_csv and then look at what it contains with the functions head, info, and describe."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c401786732f7ced",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "### Loading the data\n",
    "\n",
    "Now it's time to check out the dataset! You'll use pandas (which has been pre-imported as `pd`) to load your data into a DataFrame and then do some Exploratory Data Analysis (EDA) of it.\n",
    "\n",
    "The training data is available as `TrainingData.csv`. Your first task is to load it into a DataFrame in the IPython Shell using `pd.read_csv()` along with the keyword argument `index_col=0`.\n",
    "\n",
    "Use methods such as `.info()`, `.head()`, and `.tail()` to explore the budget data and the properties of the features and labels.\n",
    "\n",
    "Some of the column names correspond to **features** - descriptions of the budget items - such as the `Job_Title_Description` column. The values in this column tell us if a budget item is for a teacher, custodian, or other employee.\n",
    "\n",
    "Some columns correspond to the budget item **labels** you will be trying to predict with your model. For example, the `Object_Type` column describes whether the budget item is related classroom supplies, salary, travel expenses, etc.\n",
    "\n",
    "Use d`f.info()` in the IPython Shell to answer the following questions:\n",
    "\n",
    "- How many rows are there in the training data?\n",
    "- How many columns are there in the training data?\n",
    "- How many non-null entries are in the `Job_Title_Description` column?\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- ~~25 rows, 1560 columns, 1560 non-null entries in `Job_Title_Description`.~~\n",
    "- ~~25 rows, 1560 columns, 1131 non-null entries in `Job_Title_Description`.~~\n",
    "- **1560 rows, 25 columns, 1131 non-null entries in `Job_Title_Description`.**\n",
    "- ~~1560 rows, 25 columns, 1560 non-null entries in `Job_Title_Description`.~~"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "978f84ff-4bc2-49b6-8d96-8b41011bd643",
   "metadata": {},
   "source": [
    "### Summarize the Data\n",
    "\n",
    "You'll continue your EDA in this exercise by computing summary statistics for the numeric data in the dataset. The data has been pre-loaded into a DataFrame called `df`.\n",
    "\n",
    "You can use `df.info()` in the IPython Shell to determine which columns of the data are numeric, specifically type `float64`. You'll notice that there are two numeric columns, called `FTE` and `Total`.\n",
    "\n",
    "- `FTE`: Stands for \"full-time equivalent\". If the budget item is associated to an employee, this number tells us the percentage of full-time that the employee works. A value of 1 means the associated employee works for the school full-time. A value close to 0 means the item is associated to a part-time or contracted employee.\n",
    "- `Total`: Stands for the total cost of the expenditure. This number tells us how much the budget item cost.\n",
    "\n",
    "After printing summary statistics for the numeric data, your job is to plot a histogram of the non-null `FTE` column to see the distribution of part-time and full-time employees in the dataset.\n",
    "\n",
    "This course touches on a lot of concepts you may have forgotten, so if you ever need a quick refresher, download the [Scikit-Learn Cheat Sheet][1] and keep it handy!\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Print summary statistics of the numeric columns in the DataFrame df using the `.describe()` method.\n",
    "- Import `matplotlib.pyplot` as `plt`.\n",
    "- Create a histogram of the non-null `'FTE'` column. You can do this by passing `df['FTE'].dropna()` to `plt.hist()`.\n",
    "- The title has been specified and axes have been labeled, so hit submit to see how often school employees work full-time!\n",
    "\n",
    "[1]: https://datacamp-community-prod.s3.amazonaws.com/5433fa18-9f43-44cc-b228-74672efcd116"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ffeeb68-9cba-42b1-9ea3-e5e514aa40a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print the summary statistics\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e48c25f-deb4-411b-b8e3-cff93f10de68",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the histogram\n",
    "plt.hist(df['FTE'].dropna(), ec='k')  # .dropna() isn't needed\n",
    "\n",
    "# Add title and labels\n",
    "plt.title('Distribution of %full-time \\n employee works')\n",
    "plt.xlabel('% of full-time')\n",
    "plt.ylabel('num employees')\n",
    "\n",
    "# Display the histogram\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bae073d7-a2cf-4215-a253-efaad52c4cac",
   "metadata": {},
   "source": [
    "**The high variance in expenditures makes sense (some purchases are cheap some are expensive). Also, it looks like the FTE column is bimodal. That is, there are some part-time and some full-time employees.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a4152c6-4d74-44dc-8764-34b7703bffd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dtypes.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64e812736b0f97ba",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## Looking at the datatypes\n",
    "\n",
    "1. Looking at the datatypes: We've seen we have some numeric values and some text values in our dataset. It's common to have data where each value is from a known set of categories. For example, a column season_of_year may have the values winter, spring, summer, and fall. These kinds of data are not simply strings. Let's look at\n",
    "\n",
    "2. Objects instead of categories: our sample dataset again. If we look at the label column, we can see that it takes either the value a or the value b. We can treat these kinds of variables in a way that solves two problems for us.\n",
    "\n",
    "3. Encode labels as categories: The first problem is that our machine learning algorithms work on numbers. We need a numeric representation of these strings before we can do any sort of model-fitting. The second problem is that strings can be slow. We never know ahead of time how long a string is, so our computers have to take more time processing strings than numbers, which have a precise number of bits. In pandas, there is a special datatype called category that encodes our categorical data numerically, and--because of this numerical encoding--it can speed up our code. In pandas, we can call the astype function with the string category to change a column's type from object to category. Here are two rows of the 'label' column from the sample dataframe. When the data is loaded, pandas assumes that these variables are strings, so the dtype is object. By calling astype('category'), we are returned a categorical variable. As we can see, pandas is already smarter about the values that appear in the column--in this case, the two values a and b.### Slide 5: Dummy variable encoding. We have actually converted these strings into a numeric representation of the categories.\n",
    "- ML algorithms work on numbers, not strings\n",
    "  - Need a numeric representation of these strings\n",
    "- Strings can be slow compared to numbers\n",
    "- In pandas, `category` dtype encodes categorical data numerically\n",
    "  - Can speed up code\n",
    "\n",
    "4. Dummy variable encoding: To see this numeric representation, we can use the get_dummies function in pandas. This is called get_dummies because this process is called creating \"dummy variables\". Our dummy variables dataframe has two columns: the first, if the value is \"label_a,\" the second if the value is \"label_b.\" Each row contains a 1 if that row is of that category, and a 0 if not. This is also called a \"binary indicator\" representation. Note that the prefix_sep parameter is useful to tell the get_dummies function what character should separate the original column name and the column value for our dummy variable. Before we create categorical representations of our budget data, we want to mention\n",
    "- Also called a `binary indicator` representation\n",
    "\n",
    "5. Lambda functions: lambda functions, a feature of the Python language. Instead of using the def syntax you may have seen before, lambda functions let you make simple, one-line functions. For example, we may want a function that squares a variable. We can define a lambda function that takes a parameter, the variable x. The function itself just multiplies x by x and returns the result. We can call this function just like any other Python function, and it returns whatever the one line of code evaluates to.\n",
    "- Alternative to `def` syntax\n",
    "- Easy way to make simple, one-line functions\n",
    "\n",
    "6. Encode labels as categories: In the sample dataframe, we only have one relevant column called label. But, you'll remember from looking at the budget data that there are multiple columns we want to make categorical. To make multiple columns into categories, we need to apply the function to each column separately. We will use a small lambda function to convert each column to a category. We then use the apply method on a pandas dataframe to apply this function to each of the relevant columns separately by passing the axis equals 0 parameter. Now it's your turn to use astype('category')\n",
    "- In the sample dataframe, we only have one relevant column\n",
    "- In the budget data, there are multiple columns that need to be made categorical\n",
    "\n",
    "7. Let's practice!: to encode categorical variables in our actual dataset to help speed up the models that you're going to use on your data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18c40a7b3e553c52",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "### Exploring datatypes in pandas\n",
    "\n",
    "It's always good to know what datatypes you're working with, especially when the inefficient pandas type `object` may be involved. Towards that end, let's explore what we have.\n",
    "\n",
    "The data has been loaded into the workspace as `df`. Your job is to look at the DataFrame attribute `.dtypes` in the IPython Shell, and call its `.value_counts()` method in order to answer the question below.\n",
    "\n",
    "Make sure to call `df.dtypes.value_counts()`, and not `df.value_counts()`! Check out the difference in the Shell. `df.value_counts()` will return an error, because it is a Series method, not a DataFrame method.\n",
    "\n",
    "How many columns with dtype `object` are in the data?\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- ~~2.~~\n",
    "- **23.**\n",
    "- ~~64.~~\n",
    "- ~~25.~~"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d21a4285-a60d-439e-a0bb-4d4e7cbdd00c",
   "metadata": {},
   "source": [
    "### Encode the Labels as Categorical Variables\n",
    "\n",
    "Remember, your ultimate goal is to predict the probability that a certain label is attached to a budget line item. You just saw that many columns in your data are the inefficient `object` type. Does this include the labels you're trying to predict? Let's find out!\n",
    "\n",
    "There are 9 columns of labels in the dataset. Each of these columns is a category that has [many possible values it can take][1]. The 9 labels have been loaded into a list called `LABELS`. In the Shell, check out the type for these labels using `df[LABELS].dtypes`.\n",
    "\n",
    "You will notice that every label is encoded as an object datatype. Because `category` datatypes are [much more efficient][2] your task is to convert the labels to category types using the `.astype()` method.\n",
    "\n",
    "Note: `.astype()` only works on a pandas Series. Since you are working with a pandas DataFrame, you'll need to use the `.apply()` method and provide a lambda function called categorize_label that applies `.astype()` to each column, x.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Define the lambda function `categorize_label` to convert column `x` into `x.astype('category')`.\n",
    "- Use the `LABELS` list provided to convert the subset of data `df[LABELS]` to categorical types using the `.apply()` method and `categorize_label`. Don't forget `axis=0`.\n",
    "- Print the converted `.dtypes` attribute of `df[LABELS]`.\n",
    "\n",
    "[1]: https://www.drivendata.org/competitions/4/box-plots-for-education/page/15/#labels_list\n",
    "[2]: http://matthewrocklin.com/blog/work/2015/06/18/Categoricals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bed5b8b-46e5-4c58-8178-d0df14ba9d40",
   "metadata": {},
   "outputs": [],
   "source": [
    "LABELS_td = ['Function', 'Use', 'Sharing', 'Reporting', 'Student_Type', 'Position_Type', 'Object_Type', 'Pre_K', 'Operating_Status']\n",
    "categorize_label = lambda x: x.astype('category')\n",
    "df[LABELS_td] = df[LABELS_td].apply(categorize_label)\n",
    "print(df[LABELS_td].dtypes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06f438ff-2132-487b-a4fc-47d2fc318245",
   "metadata": {},
   "source": [
    "### Counting Unique Labels\n",
    "\n",
    "As Peter mentioned in the video, there are over 100 unique labels. In this exercise, you will explore this fact by counting and plotting the number of unique values for each category of label.\n",
    "\n",
    "The dataframe `df` and the `LABELS` list have been loaded into the workspace; the `LABELS` columns of `df` have been converted to category types.\n",
    "\n",
    "pandas, which has been pre-imported as `pd`, provides a `pd.Series.nunique` method for counting the number of unique values in a Series.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Create the DataFrame `num_unique_labels` by using the `.apply()` method on `df[LABELS]` with `pd.Series.nunique` as the argument.\n",
    "- Create a bar plot of `num_unique_labels` using pandas' `.plot(kind='bar')` method.\n",
    "- The axes have been labeled for you, so hit submit to see the number of unique values for each label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa046f05-270d-4ba9-8d7e-bbcf6c4edfae",
   "metadata": {
    "is_executing": true
   },
   "outputs": [],
   "source": [
    "num_unique_labels = df[LABELS_td].apply(pd.Series.nunique)\n",
    "ax = num_unique_labels.plot(kind='bar', ylabel='Number of Unique Categories', xlabel='Column Name')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c8094575c32431c",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## How do we measure success?\n",
    "\n",
    "1. How do we measure success?: The next step is to decide how we decide if our algorithm works. Choosing how to evaluate your machine learning model is one of the most important decisions an analyst makes. The decision balances the real-world use of the algorithm, the mathematical properties of the evaluation function, and the interpretability of the measure. How do we measure success?: Often we hear the question \"how accurate is your model?\" Accuracy is a simple measure that tells us what percentage of rows we got right. However, sometimes accuracy doesn't tell the whole story. Consider the case of identifying spam emails. Let's say that only 1% of the emails I receive are spam. The other 99% are legitimate emails. I can build a classifier that is 99% accurate just by assuming every message is legitimate, and never marking any message as spam. But this model isn't useful at all because every message, even the spam, ends up in my inbox. The metric we use for this problem is called log loss. Log loss is what is generally called a \"loss function,\" and it is a measure of error. We want our error to be as small as possible, which is the opposite of a metric like accuracy, where we want to maximize the value.\n",
    "- Accuracy can be misleading when classes are imbalanced\n",
    "  - Legitimate email: 99%, Spam: 1%\n",
    "  - Model that never predicts spam will be 99% accurate!\n",
    "- Metric used in this problem: log loss\n",
    "It is a loss function\n",
    "  - Measure of error\n",
    "  - Want to minimize the error (unlike accuracy)\n",
    "\n",
    "2. Log loss binary classification: Let's look at how logloss is calculated. It takes the actual value, 1 or 0, and it takes our prediction, which is a probability between 0 and 1. The greek letter sigma (which looks like an uppercase E below) indicates that we're taking the sum of the logloss measures for each row of the dataset. We then multiply this sum by -1 over N, the number of rows, to get a single value for loss. We will unpack this math a little more by looking at an example. Consider the case where the true label is 0, but we predict confidently that the label is 1. In this case, because y is 0, the first term becomes 0. This means the logloss is calculated by (1 - y) times log(1 - p). This simplifies to log(1 - 0-point-9) or log(0-point-1), which is 2-point-3. Now, Log loss binary classification: example: consider the case that the correct label is 1, but our model is not sure and our prediction is right in the middle (0-point-5). Our logloss is 0-point-69. Since We are trying to minimize log loss, we can see that it is better to be less confident than it is to be confident and wrong.\n",
    "- Log loss for binary classi,cation\n",
    "  - ![][sb5]\n",
    "- Actual value: y = {1=yes, 0=no}\n",
    "- Prediction (probability that the value is 1): p\n",
    "- Try label is 0, but we predict confidently that the label is 1\n",
    "  - logloss = (1 - y) * log(1 - p)\n",
    "  - logloss = (1 - 0) * log(1 - 0.9)\n",
    "  - logloss = 2.3\n",
    "- Try label is 1, but our model is not sure and our prediction is right in the middle (0.5)\n",
    "  - logloss = -1/N * sum(y * log(p) + (1 - y) * log(1 - p))\n",
    "  - lobloss = 0.69\n",
    "  - Better to be less confident than to be confident and wrong\n",
    "3. Computing log loss with NumPy: Here is an implementation of logloss. The most important detail is the clip function which sets a maximum and minimum value for the elements in an array. Since log(0) is negative infinity, we want to offset our predictions ever so slightly from being exactly 1 or exactly 0 so that our score remains a real number. In this example we use the eps variable to be 0-point-00 (thirteen zeros) 1, which is close enough to zero to not effect our overall scores. After adjusting the predictions slightly with clip, we calculate logloss using the formula. If we call this function on the examples we looked at earlier, we can see that the confident and wrong item returns the expected value of 2-point-3 and the prediction that is right in the middle returns 0-point-69. We have implemented it here to demonstrate how to take a mathematical equation and turn it into a function to use for evaluation, just like you may need to if you were participating in a machine learning competition.\n",
    "\n",
    "4. Let's practice!: Now let's develop some intuition for how the logloss metric performs with a few examples.\n",
    "\n",
    "[sb5]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2024-01-19_school_budgeting_with_machine_learning_in_python/sb05.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaa6ec4b-6f90-4a2c-820e-0adaeca72b2f",
   "metadata": {},
   "source": [
    "### `def compute_log_loss`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8fe98be5-ce50-4caf-a513-1d2898cd8a64",
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_log_loss(predicted: Union[float, List[float]], actual: [int, List[int]], eps: float=1e-14) -> float:\n",
    "    \"\"\"\n",
    "    Computes the logarithmic loss between predicted and \n",
    "    actual when these are 1D arrays\n",
    "\n",
    "    :param predicted: The predicted probabilities as floats between 0-1\n",
    "    :param actual: The actual binary labels. Either 0 or 1\n",
    "    :param eps (optional): log(0) is inf, so we need to offset our\n",
    "                           precidted values slightly by eps from 0 to 1.\n",
    "    \"\"\"\n",
    "\n",
    "    predicted = np.clip(predicted, eps, 1 - eps)\n",
    "    loss = -1 * np.mean(actual * np.log(predicted)\n",
    "                        + (1 - actual)\n",
    "                        * np.log(1 - predicted))\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23552926-321e-4bf9-9669-7068bc1e60b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = [(0.85, 1), (0.99, 0), (0.51, 0)]\n",
    "\n",
    "for p, y in data:\n",
    "    print(compute_log_loss(p, y))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd275576-9af3-40e1-9d3a-f97860f3a316",
   "metadata": {},
   "source": [
    "### Penalizing highly confident wrong answers\n",
    "\n",
    "As Peter explained in the video, log loss provides a steep penalty for predictions that are both wrong and confident, i.e., a high probability is assigned to the incorrect class.\n",
    "\n",
    "Suppose you have the following 3 examples:\n",
    "\n",
    "```lang-none\n",
    "A: y = 1, p = 0.85\n",
    "B: y = 0, p = 0.99\n",
    "C: y = 0, p = 0.51\n",
    "```\n",
    "\n",
    "Select the ordering of the examples which corresponds to the lowest to highest log loss scores. y is an indicator of whether the example was classified correctly. You shouldn't need to crunch any numbers!\n",
    "\n",
    "- **Lowest: A, Middle: C, Highest: B** - **Of the two incorrect predictions, B will have a higher log loss because it is confident and wrong.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5deea260-0aea-4c79-ba4e-b4815d7559fa",
   "metadata": {},
   "source": [
    "### Computing log loss with Numpy\n",
    "\n",
    "To see how the log loss metric handles the trade-off between accuracy and confidence, we will use some sample data generated with NumPy and compute the log loss using the provided function `compute_log_loss`(), which Peter showed you in the video.\n",
    "\n",
    "5 one-dimensional numeric arrays simulating different types of predictions have been pre-loaded: `actual_labels`, `correct_confident`, `correct_not_confident`, `wrong_not_confident`, and `wrong_confident`.\n",
    "\n",
    "Your job is to compute the log loss for each sample set provided using the `compute_log_loss(predicted_values, actual_values)`. It takes the predicted values as the first argument and the actual values as the second argument.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Using the `compute_log_loss()` function, compute the log loss for the following predicted values (in each case, the actual values are contained in `actual_labels`):\n",
    "  - `correct_confident`\n",
    "  - `correct_not_confident`\n",
    "  - `wrong_not_confident`\n",
    "  - `wrong_confident`\n",
    "  - `actual_labels`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3e9931b-c32a-43e9-b7c2-53fd2010b57b",
   "metadata": {},
   "outputs": [],
   "source": [
    "correct_confident = np.array([0.95, 0.95, 0.95, 0.95, 0.95, 0.05, 0.05, 0.05, 0.05, 0.05])\n",
    "correct_not_confident = np.array([0.65, 0.65, 0.65, 0.65, 0.65, 0.35, 0.35, 0.35, 0.35, 0.35])\n",
    "wrong_not_confident = np.array([0.35, 0.35, 0.35, 0.35, 0.35, 0.65, 0.65, 0.65, 0.65, 0.65])\n",
    "wrong_confident = np.array([0.05, 0.05, 0.05, 0.05, 0.05, 0.95, 0.95, 0.95, 0.95, 0.95])\n",
    "actual_labels = np.array([1., 1., 1., 1., 1., 0., 0., 0., 0., 0.])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2cbe99d-ee7b-4b22-823a-5fb204bba355",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute and print log loss for 1st case\n",
    "correct_confident_loss = compute_log_loss(correct_confident, actual_labels)\n",
    "print(\"Log loss, correct and confident: {}\".format(correct_confident_loss)) \n",
    "\n",
    "# Compute log loss for 2nd case\n",
    "correct_not_confident_loss = compute_log_loss(correct_not_confident, actual_labels)\n",
    "print(\"Log loss, correct and not confident: {}\".format(correct_not_confident_loss)) \n",
    "\n",
    "# Compute and print log loss for 3rd case\n",
    "wrong_not_confident_loss = compute_log_loss(wrong_not_confident, actual_labels)\n",
    "print(\"Log loss, wrong and not confident: {}\".format(wrong_not_confident_loss)) \n",
    "\n",
    "# Compute and print log loss for 4th case\n",
    "wrong_confident_loss = compute_log_loss(wrong_confident, actual_labels)\n",
    "print(\"Log loss, wrong and confident: {}\".format(wrong_confident_loss)) \n",
    "\n",
    "# Compute and print log loss for actual labels\n",
    "actual_labels_loss = compute_log_loss(actual_labels, actual_labels)\n",
    "print(\"Log loss, actual labels: {}\".format(actual_labels_loss)) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d71d5d17-738b-4d7a-837c-5349702dc891",
   "metadata": {},
   "source": [
    "**Log loss penalizes highly confident wrong answers much more than any other type. This will be a good metric to use on your models.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d3842af-d395-4ab3-8d59-1ba934a936ef",
   "metadata": {},
   "source": [
    "# Creating a simple first model\n",
    "\n",
    "In this chapter, you'll build a first-pass model. You'll use numeric data only to train the model. Spoiler alert - throwing out all the text data is bad for performance! But you'll learn how to format your predictions. Then, you'll be introduced to natural language processing (NLP) in order to start working with the large amounts of text in the data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "396e356e-6dd2-49e2-9a4a-decbd15701d5",
   "metadata": {},
   "source": [
    "## It's time to build a model\n",
    "\n",
    "1. It's time to build a model: When approaching a machine learning problem, and in particular looking at a dataset from a machine learning competition, it's always a good approach to start with a very simple model. Creating a simple model first helps to give us a sense of how challenging a question actually is. Before we dig deep into complex models where many more things can go wrong, we want to understand how much signal we can pull out using basic methods. We'll start with a model that just uses the numeric data columns. In building our first model, we want to go from raw data to predictions as quickly as possible. In this case, we'll use multi-class logistic regression, which treats each label column as independent. The model will train a logistic regression classifier for each of these columns separately and then use those models to predict whether the label appears or not for any given rows. After writing out our predictions to a CSV, we'll simulate submitting them to the competition and seeing what our score would be.\n",
    "- Always a good approach to start with a very simple model\n",
    "- Gives a sense of how challenging the problem is\n",
    "- Many more things can go wrong in complex models\n",
    "- How much signal can we pull out using basic methods?\n",
    "- Train basic model on numeric data only\n",
    "  - Want to go from raw data to predictions quickly\n",
    "- Multi-class logistic regression\n",
    "  - Train classifier of each label separately and use those to predict\n",
    "- Format predictions and save to CSV\n",
    "- Compute log loss score of your predictions\n",
    "2. Splitting the multi-class dataset: In the supervised learning course, we talked about splitting our data into a training set and a test set. However, because of the nature of our data, the simple approach to a train-test split won't work. Some labels that only appear in a small fraction of the dataset. If we split our dataset randomly, we may end up with labels in our test set that never appeared in our training set. Our model won't be able to predict a class that it has never seen before! One approach to this problem is called StratifiedShuffleSplit, which is mentioned in the supervised learning course. However, this scikit-learn function only works if you have a single target variable. In our case, we have many target variables. To work around this issue, we've provided a utility function, multilabel_train_test_split, that will ensure that all of the classes are represented in both the test and training sets. We'll have a link to that code in the exercises if you're curious.\n",
    "- Recall: Train-test split\n",
    "  - Will not work here\n",
    "  - May end up with labels in test set that never appear in training set\n",
    "- Solution: `StratifiedShuffleSplit`\n",
    "  - Only works with a single target variable\n",
    "  - We have many target variables\n",
    "  - `multilabel_train_test_split()`\n",
    "3. Splitting the data: First, we'll subset our data to just the numeric columns. NUMERIC_COLUMNS is a variable we provide that contains a list of the column names for the columns that are numbers rather than text. Then we'll do a minimal amount of preprocessing where we fill the NaNs that are in the dataset with -1000. In this case, we choose -1000, because we want our algorithm to respond to NaN's differently than 0. We'll create our array of target variables using the get_dummies function in pandas. Again, the get_dummies function takes our categories, and produces a binary indicator for our targets, which is the format that scikit-learn needs to build a model. Finally, we use the multilabel_train_test_split function that is provided to split the dataset in to a training set and a test set.\n",
    "```python\n",
    "data_to_train = df[NUMERIC_COLUMNS].fillna(-1000)\n",
    "labels_to_use = pd.get_dummies(df[LABELS])\n",
    "X_train, X_test, y_train, y_test = multilabel_train_test_split(\n",
    "    data_to_train,\n",
    "    labels_to_use,\n",
    "    size=0.2, seed=123)\n",
    "```\n",
    "4. Training the model: Now we can import our standard LogisticRegression classifier from sklearn dot linear_model. We'll also import the OneVsRestClassifier from the sklearn dot multiclass module. OneVsRest let's us treat each column of y independently. Essentially, it fits a separate classifier for each of the columns. This is just one strategy you can use if you have multiple classes. Take a look at the scikit-learn documentation for other strategies you could consider. Now we can train that classifier by calling fit and passing our features in X_train and the corresponding labels that are in y_train.\n",
    "```python\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "clf = OneVsRestClassifier(LogisticRegression())\n",
    "clf.fit(X_train, y_train)\n",
    "```\n",
    "- `OneVsRestClassifier`:\n",
    "  - Treats each column of y independently\n",
    "  - Fits a separate classifier for each of the columns\n",
    "5. Let's practice!: Now it's your turn to build a model and make some predictions!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7cdde3a-ed94-48f6-935e-718d56ecd376",
   "metadata": {},
   "source": [
    "### Setting up a train-test split in scikit-learn\n",
    "\n",
    "Alright, you've been patient and awesome. It's finally time to start training models!\n",
    "\n",
    "The first step is to split the data into a training set and a test set. Some labels don't occur very often, but we want to make sure that they appear in both the training and the test sets. We provide a function that will make sure at least `min_count` examples of each label appear in each split: `multilabel_train_test_split`.\n",
    "\n",
    "Feel free to check out the full code for `multilabel_train_test_split` [here][1].\n",
    "\n",
    "You'll start with a simple model that uses just the numeric columns of your DataFrame when calling `multilabel_train_test_split`. The data has been read into a DataFrame `df` and a list consisting of just the numeric columns is available as `NUMERIC_COLUMNS`.\n",
    "\n",
    "[1]: https://github.com/drivendataorg/box-plots-sklearn/blob/master/src/data/multilabel.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe20bd07-e061-42df-9d1b-b9828dcd499f",
   "metadata": {},
   "outputs": [],
   "source": [
    "NUMERIC_COLUMNS_td = df.select_dtypes(include='number').columns.tolist()\n",
    "\n",
    "# Create the new DataFrame: numeric_data_only\n",
    "numeric_data_only = df[NUMERIC_COLUMNS_td].fillna(-1000)\n",
    "\n",
    "# Get labels and convert to dummy variables: label_dummies\n",
    "label_dummies = pd.get_dummies(df[LABELS_td])\n",
    "\n",
    "# Create training and test sets\n",
    "X_train, X_test, y_train, y_test = multilabel_train_test_split(numeric_data_only,\n",
    "                                                               label_dummies,\n",
    "                                                               size=0.2, \n",
    "                                                               seed=123)\n",
    "\n",
    "# Print the info\n",
    "print(\"X_train info:\")\n",
    "print(X_train.info())\n",
    "print(\"\\nX_test info:\")  \n",
    "print(X_test.info())\n",
    "print(\"\\ny_train info:\")  \n",
    "print(y_train.info())\n",
    "print(\"\\ny_test info:\")  \n",
    "print(y_test.info()) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57c77394-64ca-4471-90a8-73cd2bd38f14",
   "metadata": {},
   "source": [
    "### Training a model\n",
    "\n",
    "With split data in hand, you're only a few lines away from training a model.\n",
    "\n",
    "In this exercise, you will import the logistic regression and one versus rest classifiers in order to fit a multi-class logistic regression model to the `NUMERIC_COLUMNS` of your feature data.\n",
    "\n",
    "Then you'll test and print the accuracy with the `.score()` method to see the results of training.\n",
    "\n",
    "**Before you train!** Remember, we're ultimately going to be using logloss to score our model, so don't worry too much about the accuracy here. Keep in mind that you're throwing away all of the text data in the dataset - that's by far most of the data! So don't get your hopes up for a killer performance just yet. We're just interested in getting things up and running at the moment.\n",
    "\n",
    "All data necessary to call `multilabel_train_test_split()` has been loaded into the workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41707c4f-006f-48fc-9847-1f6cd6789a89",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate the classifier: clf\n",
    "clf = OneVsRestClassifier(LogisticRegression())\n",
    "\n",
    "# Fit the classifier to the training data\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "# Print the accuracy\n",
    "print(\"Accuracy: {}\".format(clf.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da410827-3364-45df-94ed-125da1a6750c",
   "metadata": {},
   "source": [
    "**The good news is that your workflow didn't cause any errors. The bad news is that your model scored the lowest possible accuracy: 0.0! But hey, you just threw away ALL of the text data in the budget. Later, you won't. Before you add the text data, let's see how the model does when scored by log loss.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae519164-080b-4993-a82e-80a218b78500",
   "metadata": {},
   "source": [
    "## Making predictions\n",
    "\n",
    "1. Making predictions: Once our classifier is trained, we can use it to make predictions on new data. We could use our test set that we've withheld, but we want to simulate actually competing in a data science competition,\n",
    "\n",
    "2. Predicting on holdout data: so we will make predictions on the holdout set that the competition provides. As we did with our training data, we load the holdout data using the read_csv function from pandas. We then perform the same simple preprocessing we used earlier. First, we select just the numeric columns. Then we use fillna to replace NaN values with -1000.Finally, we call the predict_proba method on our trained classifier. Remember, we want to predict probabilities for each label, not just whether or not the label appears. If we simply used the predict method instead, we would end up with a 0 or 1 in every case. Because log loss penalizes you for being confident and wrong, the score for this submission would be significantly worse than if we use predict_proba.\n",
    "```python\n",
    "houldout = pd.read_csv('HoldoutData.csv', index_col=0)\n",
    "holdout = holdout[NUMERIC_COLUMNS].fillna(-1000)\n",
    "predictions = clf.predict_proba(holdout)\n",
    "```\n",
    "\n",
    "- Using `.predict()`\n",
    "  - would result in an output of 0 or 1\n",
    "  - Log loss penalized being confident and wrong\n",
    "  - Worse performance compared to `.predict_proba()`\n",
    "3. Submitting your predictions as a csv: In data science competitions, it's a standard practice to write your predictions as a CSV and then upload that CSV to the competition platform. From the competition documentation, we can see that the submission format for this competition expects a dataframe that has each of the individual labels as the column headers and probabilities for each of the columns. The to_csv function on a DataFrame, will take our predictions and write them out to a file. One quick note: you'll notice that our columns have the original column name separated from the value by two underscores. This is because some of the column names already contained single a underscore.\n",
    "- All formatting and submission instructions are provided by the competition\n",
    "  - Standard practice to write predictions to a CSV and upload to competition platform\n",
    "  - `to_csv()` method on a DataFrame\n",
    "  - Submission format\n",
    "    - Each column is an individual label\n",
    "    - Each row is a prediction for that label\n",
    "    - Write to a file\n",
    "4. Format and submit predictions: The predictions that were generated by predict_proba is just an array of values. It doesn't have column names or an index like our submission format does. We'll fix this by turning those values into a DataFrame. To get the column names, we use get_dummies on our target variables and then borrow those column names for our new dataframe. Our index will be the same index that we read into pandas with read_csv, and the data is the predictions themselves. As we noted in the previous slide, we want to separate the original column names from the column values with a double underscore when we call get_dummies. To do this, we will use the keyword argument prefix_sep equals double underscore with the get_dummies function. Finally, we can call to_csv and pass the filename of the file we want to write out to disk. We can call the score_submission function that is provided to see how our submission would have scored in the competition!\n",
    "```python\n",
    "prediction_df = pd.DataFrame(columns=pd.get_dummies(df[LABELS], prefix_sep='__').columns, index=houldout.index, data=predictions)\n",
    "prediction_df.to_csv('predictions.csv')\n",
    "score = score_submission(pred_path='predictions.csv')\n",
    "```\n",
    "5. DrivenData leaderboard: On the DrivenData website, you would submit this CSV file of predictions. We would generate a score for you on the holdout set and then post that score to the leaderboard. Here is an example of the leaderboard from this competition. As the competition proceeds, people make more and more submissions and their place on the leaderboard changes as they build better and better models.\n",
    "\n",
    "6. Let's practice!: Now you'll get the chance to make some predictions and see what your score is."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de4a60c0-2f7f-479e-81de-a7dffaf8515b",
   "metadata": {},
   "source": [
    "### Use your model to predict values on holdout data\n",
    "\n",
    "You're ready to make some predictions! Remember, the train-test-split you've carried out so far is for model development. The original competition provides an additional test set, for which you'll never actually see the correct labels. This is called the \"holdout data.\"\n",
    "\n",
    "The point of the holdout data is to provide a fair test for machine learning competitions. If the labels aren't known by anyone but DataCamp, DrivenData, or whoever is hosting the competition, you can be sure that no one submits a mere copy of labels to artificially pump up the performance on their model.\n",
    "\n",
    "Remember that the original goal is to predict the **probability of each label**. In this exercise you'll do just that by using the `.predict_proba()` method on your trained model.\n",
    "\n",
    "First, however, you'll need to load the holdout data, which is available in the workspace as the file `HoldoutData.csv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c1f53fd-8846-4a21-9605-90ca91ca08f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate the classifier: clf\n",
    "clf = OneVsRestClassifier(LogisticRegression())\n",
    "\n",
    "# Fit it to the training data\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "# Load the holdout data: holdout\n",
    "holdout = pd.read_csv('data/2024-01-19_school_budgeting_with_machine_learning_in_python/HoldoutData.csv', index_col=0)\n",
    "\n",
    "# Generate predictions: predictions\n",
    "NUMERIC_COLUMNS_hd = holdout.select_dtypes(include='number').columns\n",
    "predictions = clf.predict_proba(holdout[NUMERIC_COLUMNS_hd].fillna(-1000))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "986f6251-2ebf-403c-9e64-6192cf7ea705",
   "metadata": {},
   "source": [
    "### Writing out your results to a csv for submission\n",
    "\n",
    "Writing out your results to a csv for submission\n",
    "At last, you're ready to submit some predictions for scoring. In this exercise, you'll write your predictions to a .csv using the `.to_csv()` method on a pandas DataFrame. Then you'll evaluate your performance according to the LogLoss metric discussed earlier!\n",
    "\n",
    "You'll need to make sure your submission obeys the [correct format][1].\n",
    "\n",
    "To do this, you'll use your `predictions` values to create a new DataFrame, `prediction_df`.\n",
    "\n",
    "**Interpreting LogLoss & Beating the Benchmark**:\n",
    "\n",
    "When interpreting your log loss score, keep in mind that the score will change based on the number of samples tested. To get a sense of how this very basic model performs, compare your score to the **DrivenData benchmark model performance: 2.0455**, which merely submitted uniform probabilities for each class.\n",
    "\n",
    "Remember, the lower the log loss the better. Is your model's log loss lower than 2.0455?\n",
    "\n",
    "\n",
    "[1]: https://www.drivendata.org/competitions/4/page/15/#sub_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a69f822a-ed43-4139-ada1-390c9fab0bba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Format predictions in DataFrame: prediction_df\n",
    "# pd.get_dummies(df[LABELS], ...) is correctly using df, not holdout\n",
    "prediction_df = pd.DataFrame(columns=pd.get_dummies(df[LABELS_td], prefix_sep='__').columns,\n",
    "                             index=holdout.index,\n",
    "                             data=predictions)\n",
    "\n",
    "# Save prediction_df to csv\n",
    "prediction_df.to_csv('data/2024-01-19_school_budgeting_with_machine_learning_in_python/predictions.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea2ce94e-6704-40d2-a91f-5676f30a6d8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# requires functions and variables from https://goodboychan.github.io/python/datacamp/machine_learning/2020/06/05/01-School-Budgeting-with-Machine-Learning-in-Python.html\n",
    "# BOX_PLOTS_COLUMN_INDICES, def _multi_multi_log_loss, and def score_submission\n",
    "\n",
    "# Submit the predictions for scoring: score\n",
    "score = score_submission(pred_path='data/2024-01-19_school_budgeting_with_machine_learning_in_python/predictions.csv', holdout_path='data/2024-01-19_school_budgeting_with_machine_learning_in_python/TestSetLabelsSample.csv')\n",
    "\n",
    "# Print score\n",
    "print('Your model, trained with numeric data only, yields logloss score: {}'.format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b68f7382-5f4b-4f01-a69a-dda406e6b18d",
   "metadata": {},
   "source": [
    "**Even though your basic model scored 0.0 accuracy, it nevertheless performs better than the benchmark score of 2.0455. You've now got the basics down and have made a first pass at this complicated supervised learning problem. It's time to step up your game and incorporate the text data.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4af67139-1f04-41ce-a068-79785e51e4a7",
   "metadata": {},
   "source": [
    "## A very brief introduction to NLP\n",
    "\n",
    "1. A very brief introduction to NLP: Some of our data comes in the form or freefrom text. When we have data that is text, we often want to process this text to create features for our algorithms. This is called Natural Language Processing, or NLP. We'll cover a couple of basic techniques for processing text data. A very brief introduction to NLP: Data for natural language processing can be text, as it is in our case, entire documents (for example, magazine articles or emails), or transcriptions of human speech (like the script I'm reading for this video!). The first step in processing this kind of data is called \"tokenization\". Tokenization is the process of splitting a long string into segments. Usually, this means taking a string and splitting it into a list of strings where we have one string for each word. For example, we might split the string \"Natural Language Processing\" into a list of three separate tokens: \"Natural,\" \"Language,\" and \"Processing\".\n",
    "- Data for NLP:\n",
    "  - Text, documents, speech, ...\n",
    "- Tokenization\n",
    "  - Splitting a string into segments\n",
    "  - Store segments as list\n",
    "- Example: \"Natural Language Processing\"\n",
    "  - `['Natural', 'Language', 'Processing']`\n",
    "2. Tokens and token patterns: Let's take a look at an example of actual data from our school budget dataset. We have the full string \"PETRO-VEND FUEL AND FLUIDS.\" If we want to tokenize on whitespace, that is split into words every time there is a space, tab, or return in the text, we end up with 4 tokens. The token \"PETRO-VEND,\" the token \"FUEL,\" the token \"AND,\" and the token \"FLUIDS.\" For some datasets, we may want to split words based on other characters than whitespace. For example, in this dataset we may observe that often we have words that are combined with a hyphen, like \"PETRO-VEND.\" We can opt in this case to tokenize on whitespace and punctuation. Here we break into tokens every time we see a space or any mark of punctuation. In this case, we end up with 5 tokens: The token \"PETRO,\" the token \"VEND,\" the token \"FUEL,\" the token \"AND,\" and the token \"FLUIDS.\" Now that we have our 5 tokens, we want to use them as part of our machine learning algorithm.\n",
    "- Tokenize on whitespace\n",
    "  - Petro-vend fuel and fluids\n",
    "    - Petro-vend | fuel | and | fluids\n",
    "- Tokenize on whitespace and punctuation\n",
    "  - Petro | vend | fuel | and | fluids\n",
    "3. Bag of word presentation: Often, the first way to do this is to simply count the number of times that a particular token appears in a row. This is called a \"bag of words\" representation, because you can imagine our vocabulary as a bag of all of our words, and we just count the number of times a particular word was pulled out of that bag. If you're following along closely, you may have noticed that this approach discards information about word order. That is, the phrase \"red, not blue\" would be treated the same as \"blue, not red.\" A slightly more sophisticated approach is to create what are called\n",
    "- Count the number of times a particular token appears\n",
    "- \"Bag of words\"\n",
    "  - Count the number of times a word was pulled out of the bag\n",
    "- This approach discards information about word order\n",
    "  - \"Red, not blue\" is the same as \"blue, not red\"\n",
    "4. 1-gram, 2-gram, …, n-gram: \"n-grams.\" In addition to a column for every token we see, which is called a \"1-gram,\" we may have a column for every ordered pair of two words. In that case, we'd have a column for \"PETRO-VEND,\" a column for \"VEND FUEL,\" a column for \"FUEL AND,\" and a column for \"AND FLUIDS.\" These are called 2-grams (or bi-grams). N can be any number, for example, we may also include 3-grams (or tri-grams) in this example. There are three columns for trigrams: one column for \"PETRO-VEND FUEL,\" one for \"VEND FUEL AND,\" and one for \"FUEL AND FLUIDS.\" Now it's time for a little practice with tokenization and n-grams.\n",
    "- 1-gram, 2-gram, ..., n-gram\n",
    "  - ![][1]\n",
    " \n",
    "[1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2024-01-19_school_budgeting_with_machine_learning_in_python/sb01.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26493578-8b50-4d8b-b5d9-70cf54c38f8e",
   "metadata": {},
   "source": [
    "### Tokenizating text\n",
    "\n",
    "As we talked about in the video, [tokenization][1] is the process of chopping up a character sequence into pieces called tokens.\n",
    "\n",
    "How do we determine what constitutes a token? Often, tokens are separated by whitespace. But we can specify other delimiters as well. For example, if we decided to tokenize on punctuation, then any punctuation mark would be treated like a whitespace. How we tokenize text in our DataFrame can affect the statistics we use in our model.\n",
    "\n",
    "A particular cell in our budget DataFrame may have the string content `Title I - Disadvantaged Children/Targeted Assistance`. The number of n-grams generated by this text data is sensitive to whether or not we tokenize on punctuation, as you'll show in the following exercise.\n",
    "\n",
    "How many tokens (1-grams) are in the string\n",
    "\n",
    "`Title I - Disadvantaged Children/Targeted Assistance`\n",
    "\n",
    "if we tokenize on whitespace and punctuation?\n",
    "\n",
    "**Answer**\n",
    "\n",
    "- 6\n",
    "\n",
    "[1]: https://nlp.stanford.edu/IR-book/html/htmledition/tokenization-1.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7658c31c-bb85-4322-b400-cb3d19c6fedf",
   "metadata": {},
   "source": [
    "### Testing your NLP credentials with n-grams\n",
    "\n",
    "You're well on your way to NLP superiority. Let's test your mastery of n-grams!\n",
    "\n",
    "In the workspace, we have the loaded a python list, `one_grams`, which contains all 1-grams of the string `petro-vend fuel and fluids`, tokenized on punctuation. Specifically,\n",
    "\n",
    "`one_grams = ['petro', 'vend', 'fuel', 'and', 'fluids']`\n",
    "\n",
    "In this exercise, your job is to determine the **sum** of the sizes of 1-grams, 2-grams and 3-grams generated by the string `petro-vend fuel and fluids`, tokenized on punctuation.\n",
    "\n",
    "Recall that the n-gram of a sequence consists of all ordered subsequences of length n.\n",
    "\n",
    "**Answer**\n",
    "\n",
    "- 12 - The number of `1-grams + 2-grams + 3-grams` is `5 + 4 + 3 = 12`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bda3cb6-c027-42da-815b-de5c1b5bfe8a",
   "metadata": {},
   "source": [
    "## Representing text numerically\n",
    "\n",
    "1. Representing text numerically: Let's talk about how we take the tokenizations we have created and turn them into an array that we can feed into a machine learning algorithm. We mentioned the bag-of-words representation in our last video. This is one of the simplest ways to represent text in a machine learning algorithm. It discards information about grammar and word order, just assuming that the number of times a word occurs is enough information.\n",
    "- Bag-of-words\n",
    "  - Simple way to represent text in machine learning\n",
    "  - Discards information about grammar and word order\n",
    "  - Computes frequency of occurrence\n",
    "2. Scikit-learn tools for bag-of-words: Scikit-learn provides a very useful tool for creating bag-of-words representations. It is called the CountVectorizer. The CountVectorizer works by taking an array of strings and doing three things. First, it tokenizes all of the strings. Then, it makes note of all of the words that appear, which we call the \"vocabulary\". Finally, it counts the number of times that each token in the vocabulary appears in every given row.\n",
    "- `CountVectorizer()`\n",
    "  - Tokenizes all the strings\n",
    "  - Builds a 'vocabulary'\n",
    "  - Counts the occurrences of each token in the vocabulary\n",
    "3. Using `CountVectorizer()` on column of main dataset: The `CountVectorizer` is part of the text submodule in the feature_extraction module in scikit-learn. After importing the CountVectorizer, we will define a regular expression that does a split on whitespace. We won't go into the details here of how this regular expression works, but there are great resources for learning regular expressions online. We'll also make sure that our text column does not have any NaN values, simply replacing those with empty strings instead. Finally, we'll create a CountVectorizer object where we pass in the token_pattern that we have created. This creates an object that we can use to create bag-of-words representations of text. The `CountVectorizer` object that we've created can be used with the fit and transform pattern, just like any other preprocessor in scikit-learn. fit will parse all of the strings for tokens and then create the vocabulary. Here, we use the word vocabulary specifically to mean all of the tokens that appear in this dataset. transform will tokenize the text and then produce the array of counts. As part of the exercises, we'll look at how changing our token_pattern will change the number of tokens that the CountVectorizer recognizes.\n",
    "```python\n",
    "from sklearn.feature_extraction.text import CountVecotrizer\n",
    "\n",
    "TOKENS_BASIC = '\\\\\\\\S+(?=\\\\\\\\s+)'\n",
    "df.Program_Description.fillna('', inplace=True)\n",
    "vec_basic = CountVectorizer(token_pattern=TOKENS_BASIC)\n",
    "\n",
    "vec_basic.fit(df.Program_Description)\n",
    "msg = 'There are {} token in Program_Desction if tokean are any non-whitespace'\n",
    "print(msg.format(len(vec_basic.get_feature_names())))\n",
    "```\n",
    "4. Let's practice!: Now it's time to get some practice using the CountVectorizer from scikit-learn."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77c900a0-2c4e-4aab-b89c-f6ab32b49a50",
   "metadata": {},
   "source": [
    "### Creating a bag-of-words in scikit-learn\n",
    "\n",
    "In this exercise, you'll study the effects of tokenizing in different ways by comparing the bag-of-words representations resulting from different token patterns.\n",
    "\n",
    "You will focus on one feature only, the `Position_Extra` column, which describes any additional information not captured by the `Position_Type` label.\n",
    "\n",
    "For example, in the Shell you can check out the budget item in row 8960 of the data using `df.loc[8960]`. Looking at the output reveals that this `Object_Description` is overtime pay. For who? The Position Type is merely \"other\", but the Position Extra elaborates: \"BUS DRIVER\". Explore the column further to see more instances. It has a lot of NaN values.\n",
    "\n",
    "Your task is to turn the raw text in this column into a bag-of-words representation by creating tokens that contain only alphanumeric characters.\n",
    "\n",
    "For comparison purposes, the first 15 tokens of `vec_basic`, which splits `df.Position_Extra` into tokens when it encounters only whitespace characters, have been printed along with the length of the representation.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `CountVectorizer` from `sklearn.feature_extraction.text`.\n",
    "- Fill missing values in `df.Position_Extra` using `.fillna('')` to replace NaNs with empty strings. Specify the additional keyword argument `inplace=True` so that you don't have to assign the result back to `df`.\n",
    "- Instantiate the `CountVectorizer` as `vec_alphanumeric` by specifying the `token_pattern` to be `TOKENS_ALPHANUMERIC`.\n",
    "- Fit `vec_alphanumeric` to `df.Position_Extra`.\n",
    "- Hit submit to see the `len` of the fitted representation as well as the first 15 elements, and compare to `vec_basic`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3529766f-ddf4-4c1a-856a-3e9244c7f769",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the token pattern: TOKENS_ALPHANUMERIC\n",
    "TOKENS_ALPHANUMERIC = '[A-Za-z0-9]+(?=\\\\s+)'\n",
    "\n",
    "# Fill missing values in df.Position_Extra\n",
    "df.fillna({'Position_Extra': ''}, inplace=True)\n",
    "\n",
    "# Instantiate the CountVectorizer: vec_alphanumeric\n",
    "vec_alphanumeric = CountVectorizer(token_pattern=TOKENS_ALPHANUMERIC)\n",
    "\n",
    "# Fit to the data\n",
    "vec_alphanumeric.fit(df.Position_Extra)\n",
    "\n",
    "# Print the number of tokens and first 15 tokens\n",
    "msg = \"There are {} tokens in Position_Extra if we split on non-alpha numeric\"\n",
    "print(msg.format(len(vec_alphanumeric.get_feature_names_out())))\n",
    "print(vec_alphanumeric.get_feature_names_out()[:15])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3140f45f-e334-4675-9474-6b649752bc8b",
   "metadata": {},
   "source": [
    "**Treating only alpha-numeric characters as tokens gives you a smaller number of more meaningful tokens.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80aba1af-baa2-4e3b-a86b-879b10e9bc68",
   "metadata": {},
   "source": [
    "### Combining text columns for tokenization\n",
    "\n",
    "In order to get a bag-of-words representation for all the text data in our DataFrame, you must first convert the text data in each row of the DataFrame into a single string.\n",
    "\n",
    "In the previous exercise, this wasn't necessary because you only looked at one column of data, so each row was already just a single string. `CountVectorizer` expects each row to just be a single string, so in order to use all of the text columns, you'll need a method to turn a list of strings into a single string.\n",
    "\n",
    "In this exercise, you'll complete the function definition `combine_text_columns()`. When completed, this function will convert all training text data in your DataFrame to a single string per row that can be passed to the vectorizer object and made into a bag-of-words using the `.fit_transform()` method.\n",
    "\n",
    "Note that the function uses `NUMERIC_COLUMNS` and `LABELS` to determine which columns to drop. These lists have been loaded into the workspace.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Use the `.drop()` method on `data_frame` with `to_drop` and `axis=` as arguments to drop the non-text data. Save the result as `text_data`.\n",
    "- Fill in missing values (inplace) in `text_data` with blanks (\"\"), using the `.fillna()` method.\n",
    "- Complete the `.apply()` method by writing a lambda function that uses the `.join()` method to join all the items in a row with a space in between."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43ce88b5-901a-4d35-a0d6-28d1fe57f80f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define combine_text_columns()\n",
    "def combine_text_columns(data_frame: pd.DataFrame, to_drop: list=NUMERIC_COLUMNS_td + LABELS_td) -> pd.Series:\n",
    "    \"\"\" converts all text in each row of data_frame to single vector \"\"\"\n",
    "    \n",
    "    # Drop non-text columns that are in the df\n",
    "    to_drop = set(to_drop) & set(data_frame.columns.tolist())\n",
    "    text_data = data_frame.drop(to_drop, axis=1)\n",
    "    \n",
    "    # Replace nans with blanks\n",
    "    text_data.fillna('', inplace=True)\n",
    "    \n",
    "    # Join all text items in a row that have a space in between\n",
    "    return text_data.apply(lambda x: \" \".join(x), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bba63d58-a5d9-4334-b33b-571ca73f9851",
   "metadata": {},
   "source": [
    "### What's in a token?\n",
    "\n",
    "Now you will use `combine_text_columns` to convert all training text data in your DataFrame to a single vector that can be passed to the vectorizer object and made into a bag-of-words using the `.fit_transform()` method.\n",
    "\n",
    "You'll compare the effect of tokenizing using any non-whitespace characters as a token and using only alphanumeric characters as a token.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `CountVectorizer` from `sklearn.feature_extraction.text`.\n",
    "- Instantiate `vec_basic` and `vec_alphanumeric` using, respectively, the `TOKENS_BASIC` and `TOKENS_ALPHANUMERIC` patterns.\n",
    "- Create the text vector by using the `combine_text_columns()` function on `df`.\n",
    "- Using the `.fit_transform()` method with `text_vector`, fit and transform first `vec_basic` and then `vec_alphanumeric`. Print the number of tokens they contain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab3b92ae-b23b-4b44-89ec-8ce65c364652",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the basic token pattern\n",
    "TOKENS_BASIC = '\\\\S+(?=\\\\s+)'\n",
    "\n",
    "# Create the alphanumeric token pattern\n",
    "TOKENS_ALPHANUMERIC = '[A-Za-z0-9]+(?=\\\\s+)'\n",
    "\n",
    "# Instantiate basic CountVectorizer: vec_basic\n",
    "vec_basic = CountVectorizer(token_pattern=TOKENS_BASIC)\n",
    "\n",
    "# Instantiate alphanumeric CountVectorizer: vec_alphanumeric\n",
    "vec_alphanumeric = CountVectorizer(token_pattern=TOKENS_ALPHANUMERIC)\n",
    "\n",
    "# Create the text vector\n",
    "text_vector = combine_text_columns(df)\n",
    "\n",
    "# Fit and transform vec_basic\n",
    "vec_basic.fit_transform(text_vector)\n",
    "\n",
    "# Print number of tokens of vec_basic\n",
    "print(\"There are {} tokens in the dataset\".format(len(vec_basic.get_feature_names_out())))\n",
    "\n",
    "# Fit and transform vec_alphanumeric\n",
    "vec_alphanumeric.fit_transform(text_vector)\n",
    "\n",
    "# Print number of tokens of vec_alphanumeric\n",
    "print(\"There are {} alpha-numeric tokens in the dataset\".format(len(vec_alphanumeric.get_feature_names_out())))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "615f1006-ef42-4a30-8ef7-d8c4946b2fda",
   "metadata": {},
   "source": [
    "**Notice that tokenizing on alpha-numeric tokens reduced the number of tokens, just as in the last exercise. We'll keep this in mind when building a better model with the Pipeline object next.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c93fbc6-1f47-4655-9e76-b88d2aa82fa3",
   "metadata": {},
   "source": [
    "# Improving your model\n",
    "\n",
    "Here, you'll improve on your benchmark model using pipelines. Because the budget consists of both text and numeric data, you'll learn how to build pipelines that process multiple types of data. You'll also explore how the flexibility of the pipeline workflow makes testing different approaches efficient, even in complicated problems like this one!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb87d184-9b25-4124-95c3-2ae39c930d79",
   "metadata": {},
   "source": [
    "## Pipelines, feature & text preprocessing\n",
    "\n",
    "1. **Pipelines, feature & text preprocessing**: You've submitted your first simple model. Now it's time to combine what we've learned about NLP with our model pipeline and incorporate the text data into our algorithm.\n",
    "- Repeatable way to go from raw data to trained model\n",
    "- Pipeline object takes sequential list of steps\n",
    "  - Output of one step is input to next step\n",
    "- Each step is a tuple with two elements\n",
    "  - Name: string\n",
    "  - Transform: obj implementing .fit() and .transform()\n",
    "- Flexible: a step can itself be another pipeline!\n",
    "\n",
    "2. **The pipeline workflow**: The supervised learning course introduced pipelines, which are a repeatable way to go from raw data to a trained machine learning model. The scikit-learn Pipeline object takes a sequential list of steps where the output of one step is the input to the next. Each step is represented with a name for the step, that is simply a string, and an object that implements the fit and the transform methods. A good example is the `CountVectorizer` that we used earlier. As we'll see Pipelines are a very flexible way to represent your workflow. In fact, you can even have a sub-pipeline as one of the steps! The beauty of the Pipeline is that it encapsulates every transformation from raw data to a trained model.\n",
    "\n",
    "```python\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "pl = Pipeline([('clf', OneVsRestClassifier(LogisticRegression()))])\n",
    "```\n",
    "\n",
    "3. **Instantiate simple pipeline with one step**: After importing the relevant modules, we'll start with a one-step pipeline. Although we don't need a pipeline for a single step, this exercise will get us familiar with the syntax. Remember, we create a Pipeline by passing it a series of named steps. In this case the name is the string 'clf', and the step is the one-vs-rest logistic regression classifier we created earlier.\n",
    "\n",
    "4. **Train and test with sample numeric data**: The sample dataset we've been working with has numeric data in the numeric columns and the with missing column. We'll start by building a model to predict the label column with the numeric data.\n",
    "\n",
    "5. **Train and test with sample numeric data**: First, we'll use the train_test split function from sklearn dot model_selection. Our X will just be the numeric column, and our Y will be the dummy encoding of the label column. We'll call the fit method on our Pipeline, just like a normal classifier in sklearn. We can see that it returns a Pipeline that has one step to do our logistic regression.\n",
    "\n",
    "```python\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(sample_df[['numeric']], pd.get_dummies(sample_df['label']), random_state=2)\n",
    "pl.fit(X_train, y_train)\n",
    "```\n",
    "\n",
    "6. **Train and test with sample numeric data**: By using the score function on our pipeline, we can see how our Pipeline performs on the test set. The default scoring method for this classifier is accuracy. That's good enough for our sample dataset.\n",
    "\n",
    "```python\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print('accuracy on numeric data, no nans: ', accuracy)\n",
    "```\n",
    "\n",
    "7. **Adding more steps to the pipeline**: Let's add the with_missing column to our training set. Oops! Now when we call fit, we see a value error that explains that our input has NaN values.\n",
    "\n",
    "```python\n",
    "X_train, X_test, y_train, y_test = train_test_split(sample_df[['numeric', 'with_missing']], pd.get_dummies(sample_df['label']), random_state=2)\n",
    "pl.fit(X_train, y_train)\n",
    "```\n",
    "\n",
    "8. **Preprocessing numeric features with missing data**: To address this, we'll add an Imputer to our pipeline, which will fill in the NaN values. To do so, we add a step to the Pipeline with the name 'imp' and an Imputer object. The default imputation in scikit-learn is to fill missing values with the mean of the column in question. You can look at the documentation of the Imputer object to see other possible imputation strategies.\n",
    "\n",
    "```python\n",
    "from sklearn.preprocessing import Imputer\n",
    "X_train, X_test, y_train, y_test = train_test_split(sample_df[['numeric', 'with_missing']], pd.get_dummies(sample_df['label']), random_state=2)\n",
    "pl = Pipeline([('imp', Imputer()), ('clf', OneVsRestClassifier(LogisticRegression()))])\n",
    "```\n",
    "\n",
    "9. **Preprocessing numeric features with missing data**: We can call fit and score on the new pipeline, just like we did before. But, in this case our model also includes the column with_missing which had NaN values that were imputed.\n",
    "\n",
    "```python\n",
    "pipeline.fit(X_train, y_train)\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print('accuracy on all numeric, incl nans: ', accuracy)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c61b9581-3f9a-4341-bda2-3d7eba8a3a9d",
   "metadata": {},
   "source": [
    "### Instantiate pipeline\n",
    "\n",
    "To make your life easier as you start to work with all the data in your original DataFrame, `df`, it's time to turn to one of scikit-learn's most useful objects: the `Pipeline`.\n",
    "\n",
    "For the next few exercises, you'll reacquaint yourself with pipelines and train a classifier on some synthetic (sample) data of multiple datatypes before using the same techniques on the main dataset.\n",
    "\n",
    "The sample data is stored in the DataFrame, `sample_df`, which has three kinds of feature data: numeric, text, and numeric with missing values. It also has a label column with two classes, `a` and `b`.\n",
    "\n",
    "In this exercise, your job is to instantiate a pipeline that trains using the `numeric` column of the sample data.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `Pipeline` from `sklearn.pipeline`.\n",
    "- Create training and test sets using the numeric data only. Do this by specifying `sample_df[['numeric']]` in `train_test_split()`.\n",
    "- Instantiate a pipeline as pl by adding the classifier step. Use a name of `'clf'` and the same classifier from Chapter 2: `OneVsRestClassifier(LogisticRegression())`.\n",
    "- Fit your pipeline to the training data and compute its accuracy to see it in action! Since this is toy data, you'll use the default scoring method for now. In the next chapter, you'll return to log loss scoring."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf2527ea-e93b-4ce5-84ab-90e69008de3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_df = pd.read_csv('data/2024-01-19_school_budgeting_with_machine_learning_in_python/sample_df.csv')\n",
    "\n",
    "sample_df.fillna({'text': ''}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e50ae90-12f6-4532-ae56-464828c8e31e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split and select numeric data only, no nans \n",
    "X_train, X_test, y_train, y_test = train_test_split(sample_df[['numeric']], pd.get_dummies(sample_df['label']),  random_state=22)\n",
    "\n",
    "# Instantiate Pipeline object: pl\n",
    "pl = Pipeline([('clf', OneVsRestClassifier(LogisticRegression()))])\n",
    "\n",
    "# Fit the pipeline to the training data\n",
    "pl.fit(X_train, y_train)\n",
    "\n",
    "# Compute and print accuracy\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print(\"Accuracy on sample data - numeric, no nans: \", accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b2e3fc7-5908-4b3e-8685-bbd7afcbbd6e",
   "metadata": {},
   "source": [
    "**Now it's time to incorporate numeric data with missing values by adding a preprocessing step!**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5cfccfe-8a5c-4a77-a1fb-d18f053f8ab6",
   "metadata": {},
   "source": [
    "### Preprocessing numeric features\n",
    "\n",
    "What would have happened if you had included the with `'with_missing'` column in the last exercise? Without imputing missing values, the pipeline would **not** be happy (try it and see). So, in this exercise you'll improve your pipeline a bit by using the `Imputer()` imputation transformer from scikit-learn to fill in missing values in your sample data.\n",
    "\n",
    "By default, the [imputer transformer][1] replaces NaNs with the mean value of the column. That's a good enough imputation strategy for the sample data, so you won't need to pass anything extra to the imputer.\n",
    "\n",
    "After importing the transformer, you will edit the steps list used in the previous exercise by inserting a `(name, transform)` tuple. Recall that steps are processed sequentially, so make sure the new tuple encoding your preprocessing step is put in the right place.\n",
    "\n",
    "The `sample_df` is in the workspace, in case you'd like to take another look. Make sure to select **both** numeric columns- in the previous exercise we couldn't use `with_missing` because we had no preprocessing step!\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `Imputer` from `sklearn.preprocessing`.\n",
    "- Create training and test sets by selecting the correct subset of `sample_df`: `numeric` and `with_missing`.\n",
    "- Add the tuple `('imp', Imputer())` to the correct position in the pipeline. `Pipeline` processes steps sequentially, so the imputation step should come __before__ the classifier step.\n",
    "- Complete the `.fit()` and `.score()` methods to fit the pipeline to the data and compute the accuracy.\n",
    "\n",
    "[1]: http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.Imputer.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e778c2e-db1d-4cab-99ce-d79784497263",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create training and test sets using only numeric data\n",
    "X_train, X_test, y_train, y_test = train_test_split(sample_df[['numeric', 'with_missing']],\n",
    "                                                    pd.get_dummies(sample_df['label']), \n",
    "                                                    random_state=456)\n",
    "\n",
    "# Insantiate Pipeline object: pl\n",
    "pl = Pipeline([('imp', SimpleImputer()), ('clf', OneVsRestClassifier(LogisticRegression()))])\n",
    "\n",
    "# Fit the pipeline to the training data\n",
    "pl.fit(X_train, y_train)\n",
    "\n",
    "# Compute and print accuracy\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print(\"Accuracy on sample data - all numeric, incl nans: \", accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d329b86a-7f73-4f5f-b1ca-d078cab6c63b",
   "metadata": {},
   "source": [
    "**Now you know how to use preprocessing in pipelines with numeric data, and it looks like the accuracy has improved because of it! Text data preprocessing is next!**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b418a9f-9474-488e-8977-6d57ad98d868",
   "metadata": {},
   "source": [
    "## Test features and feature union\n",
    "\n",
    "1. **Text features and feature unions**: In this video, we'll look at how we process text data in a pipeline and then how we put it all together.\n",
    "\n",
    "2. **Preprocessing text features**: Our sample dataframe contains one column we haven't used yet, the text column. First, we'll change our train and test data just to be working with the text data for now. Then, we'll add a step to our Pipeline with the name vec and the CountVectorizer object from scikit-learn. Now our pipeline will use the CountVectorizer on our dataset, and then it will pass the result into our classifier.\n",
    "\n",
    "```python\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "X_train, X_test, y_train, y_test = train_test_split(sample_df['text'], pd.get_dummies(sample_df['label']), random_state=2)\n",
    "pl = Pipeline([('vec', CountVectorizer()), ('clf', OneVsRestClassifier(LogisticRegression()))])\n",
    "```\n",
    "\n",
    "3. **Preprocessing text features**: Again, we can call the fit and score methods on the Pipeline. The score function now reports our accuracy if we just use the text data.\n",
    "\n",
    "```python\n",
    "pl.fit(X_train, y_train)\n",
    "\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print('accuracy on sample data: ', accuracy)\n",
    "```\n",
    "\n",
    "4. **Preprocessing multiple dtypes**: Let's say we want to use all of our data in a single pipeline. We can't just have a Pipeline that has a `CountVectorizer` step, Imputation step, and then a classifier. The `CountVectorizer` won't know what to do with the numeric columns, and we don't want to perform imputation on the text columns. In order to build our pipeline, we need to separately operate on the text columns and on the numeric columns. There are two tools: FunctionTransformer and FeatureUnion that will help us build a Pipeline to work with both our text and numeric data. The first utility that we cover is the FunctionTransformer.\n",
    "- Want to use all available features in one pipeline\n",
    "- Problem\n",
    "  - Pipeline steps for numeric and text preprocessing can't follow each other\n",
    "  - e.g., output of `CountVectorizer` can't be input to `Imputer`\n",
    "- Solution\n",
    "  - `FunctionTransformer()` & `FeatureUnion()`\n",
    "\n",
    "5. **FunctionTransformer**: `FunctionTransformer` has a simple job: take a Python function, and turn it into an object that the scikit-learn Pipeline can understand. We'll write two simple functions: one that takes the whole dataframe, and returns just the numeric columns. The other will take the whole dataframe and return just the text columns. Using these function transformers, we can build a separate Pipeline for our numeric data and for our text data.\n",
    "- Turns a Python function into an object that a scikit-learn pipeline can understand\n",
    "- Need to write two functions for pipeline preprocessing\n",
    "  - Take entire DataFrame, return numeric columns\n",
    "  - Take entire DataFrame, return text columns\n",
    "- Can then preprocess numeric and text data in separate pipelines\n",
    "\n",
    "6. **Putting it all together**: First, we'll do our train_test split on the entire dataset and import the FunctionTransformer and FeatureUnion utilities.\n",
    "\n",
    "```python\n",
    "X_train, X_test, y_train, y_test = train_test_split(sample_df[['numeric','with_missing', 'text']], pd.get_dummies(sample_df['label']), random_state=2)\n",
    "\n",
    "from sklearn.preprocessing import FunctionTransformer\n",
    "from sklearn.pipeline import FeatureUnion\n",
    "```\n",
    "\n",
    "7. **Putting it all together**: Next, we'll create two `FunctionTransformer` objects. The first takes a dataframe and just returns the column named 'text'. The second takes a dataframe and returns the columns named 'numeric' and 'with_missing'. These two function transformer objects will let us set up separate Pipelines that operate on the selected columns only. Note that we've passed the parameter validate equals False. This simply tells scikit-learn it doesn't need to check for NaNs or validate the dtypes of the input. We'll do that work ourselves.\n",
    "\n",
    "```python\n",
    "get_text_data = FunctionTransformer(lambda x: x['text'], validate=False)\n",
    "get_numeric_data = FunctionTransformer(lambda x: x[['numeric', 'with_missing']], validate=False)\n",
    "```\n",
    "\n",
    "8. **FeatureUnion Text and Numeric Features**: `FeatureUnion` from the sklearn dot pipeline module is the other utility we need. We know that our text pipeline generates the array on the left, our text features.\n",
    "\n",
    "9. **FeatureUnion Text and Numeric Features**: And our numeric pipeline generates the array on the right, our numeric features.\n",
    "\n",
    "10. **FeatureUnion Text and Numeric Features**: The FeatureUnion object puts these two sets of features together as a single array that will be the input to our classifier.\n",
    "\n",
    "```python\n",
    "from sklearn.pipeline import FeatureUnion\n",
    "union = FeatureUnion([('numeric', numeric_pipeline), ('text', text_pipeline)])\n",
    "```\n",
    "\n",
    "11. **Putting it all together**: Here's our entire pipeline, which processes our text and numeric data. First, we create our separate text and numeric pipelines. Remember, `get_numeric_data` and `get_text_data` are the `FunctionTransformers` that we just created. These pipelines output our numeric features and our text features respectively. Then we create our overall pipeline, which has two steps: First, the `FeatureUnion` takes a list of objects, calls each one, and then concatenates the output into a wide array out of all the results. In this case, it will call our numeric pipeline and then our text pipeline. Once we have this array of all of our features, we can pass it to our classifier. We can call fit and score on this pipeline, just like our simpler ones before it.\n",
    "\n",
    "```python\n",
    "numeric_pipeline = Pipeline([('selector', get_numeric_data), ('imputer', Imputer())])\n",
    "text_pipeline = Pipeline([('selector', get_text_data), ('vectorizer', CountVectorizer())])\n",
    "pl = Pipeline([('union', FeatureUnion([('numeric', numeric_pipeline), ('text', text_pipeline)])),\n",
    "               ('clf', OneVsRestClassifier(LogisticRegression()))])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3791aa28-417a-45b4-845e-66208018c0a4",
   "metadata": {},
   "source": [
    "### Preprocessing text features\n",
    "\n",
    "Here, you'll perform a similar preprocessing pipeline step, only this time you'll use the `text` column from the sample data.\n",
    "\n",
    "To preprocess the text, you'll turn to `CountVectorizer()` to generate a bag-of-words representation of the data, as in Chapter 2. Using the [default][1] arguments, add a `(step, transform)` tuple to the steps list in your pipeline.\n",
    "\n",
    "Make sure you select only the text column for splitting your training and test sets.\n",
    "\n",
    "As usual, your `sample_df` is ready and waiting in the workspace.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `CountVectorizer` from `sklearn.feature_extraction.tex`t.\n",
    "- Create training and test sets by selecting the correct subset of `sample_df`: `'text'`.\n",
    "- Add the `CountVectorizer` step (with the name `'vec'`) to the correct position in the pipeline.\n",
    "- Fit the pipeline to the training data and compute its accuracy.\n",
    "\n",
    "[1]: http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a5bb903-c8e6-40d0-a85e-9940277aaf4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split out only the text data\n",
    "X_train, X_test, y_train, y_test = train_test_split(sample_df['text'],\n",
    "                                                    pd.get_dummies(sample_df['label']), \n",
    "                                                    random_state=456)\n",
    "\n",
    "# Instantiate Pipeline object: pl\n",
    "pl = Pipeline([\n",
    "        ('vec', CountVectorizer()),\n",
    "        ('clf', OneVsRestClassifier(LogisticRegression()))\n",
    "    ])\n",
    "\n",
    "# Fit to the training data\n",
    "pl.fit(X_train, y_train)\n",
    "\n",
    "# Compute and print accuracy\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print(\"\\nAccuracy on sample data - just text data: \", accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ce6e626-2b74-4b38-a66e-0db2856c88a1",
   "metadata": {},
   "source": [
    "### Multiple types of processing: FunctionTransformer\n",
    "\n",
    "The next two exercises will introduce new topics you'll need to make your pipeline truly excel.\n",
    "\n",
    "Any step in the pipeline must be an object that implements the `fit` and `transform` methods. The [`FunctionTransformer`][1] creates an object with these methods out of any Python function that you pass to it. We'll use it to help select subsets of data in a way that plays nicely with pipelines.\n",
    "\n",
    "You are working with numeric data that needs imputation, and text data that needs to be converted into a bag-of-words. You'll create functions that separate the text from the numeric variables and see how the `.fit()` and `.transform()` methods work.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Compute the selector `get_text_data` by using a lambda function and `FunctionTransformer()` to obtain all `'text'` columns.\n",
    "- Compute the selector `get_numeric_data` by using a lambda function and `FunctionTransformer()` to obtain all the numeric columns (including missing data). These are `'numeric'` and `'with_missing'`.\n",
    "- Fit and transform `get_text_data` using the `.fit_transform()` method with `sample_df` as the argument.\n",
    "- Fit and transform `get_numeric_data` using the same approach as above.\n",
    "\n",
    "\n",
    "[1]: http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.FunctionTransformer.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e8b01c8-6105-4bff-98cc-69c1273d5b54",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Obtain the text data: get_text_data\n",
    "get_text_data = FunctionTransformer(lambda x: x['text'], validate=False)\n",
    "\n",
    "# Obtain the numeric data: get_numeric_data\n",
    "get_numeric_data = FunctionTransformer(lambda x: x[['numeric', 'with_missing']], validate=False)\n",
    "\n",
    "# Fit and transform the text data: just_text_data\n",
    "just_text_data = get_text_data.fit_transform(sample_df)\n",
    "\n",
    "# Fit and transform the numeric data: just_numeric_data\n",
    "just_numeric_data = get_numeric_data.fit_transform(sample_df)\n",
    "\n",
    "# Print head to check results\n",
    "print('Text Data')\n",
    "display(just_text_data.head())\n",
    "print('\\nNumeric Data')\n",
    "display(just_numeric_data.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86c59a0d-8836-4885-8785-66303ab6b9b6",
   "metadata": {},
   "source": [
    "### Multiple types of processing: FeatureUnion\n",
    "\n",
    "Now that you can separate text and numeric data in your pipeline, you're ready to perform separate steps on each by nesting pipelines and using `FeatureUnion()`.\n",
    "\n",
    "These tools will allow you to streamline all preprocessing steps for your model, even when multiple datatypes are involved. Here, for example, you don't want to impute our text data, and you don't want to create a bag-of-words with our numeric data. Instead, you want to deal with these separately and then join the results together using `FeatureUnion()`.\n",
    "\n",
    "In the end, you'll still have only two high-level steps in your pipeline: preprocessing and model instantiation. The difference is that the first preprocessing step actually consists of a pipeline for numeric data and a pipeline for text data. The results of those pipelines are joined using `FeatureUnion()`.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- In the `process_and_join_features`:\n",
    "  - Add the steps `('selector', get_numeric_data)` and `('imputer', Imputer())` to the `'numeric_features'` preprocessing step.\n",
    "  - Add the equivalent steps for the `text_features` preprocessing step. That is, use `get_text_data` and a `CountVectorizer` step with the name `'vectorizer'`.\n",
    "- Add the transform step `process_and_join_features` to `'union'` in the main pipeline, `pl`.\n",
    "- Hit submit to see the pipeline in action!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee33d9fa-0f80-43a0-9f7a-acf42a9633d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split using ALL data in sample_df\n",
    "X_train, X_test, y_train, y_test = train_test_split(sample_df[['numeric', 'with_missing', 'text']],\n",
    "                                                    pd.get_dummies(sample_df['label']), \n",
    "                                                    random_state=22)\n",
    "\n",
    "# Create a FeatureUnion with nested pipeline: process_and_join_features\n",
    "process_and_join_features = FeatureUnion(\n",
    "            transformer_list = [\n",
    "                ('numeric_features', Pipeline([\n",
    "                    ('selector', get_numeric_data),\n",
    "                    ('imputer', SimpleImputer())\n",
    "                ])),\n",
    "                ('text_features', Pipeline([\n",
    "                    ('selector', get_text_data),\n",
    "                    ('vectorizer', CountVectorizer())\n",
    "                ]))\n",
    "             ]\n",
    "        )\n",
    "\n",
    "# Instantiate nested pipeline: pl\n",
    "pl = Pipeline([\n",
    "        ('union', process_and_join_features),\n",
    "        ('clf', OneVsRestClassifier(LogisticRegression()))\n",
    "    ])\n",
    "\n",
    "\n",
    "# Fit pl to the training data\n",
    "pl.fit(X_train, y_train)\n",
    "\n",
    "# Compute and print accuracy\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print(\"\\nAccuracy on sample data - all data: \", accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2094f7db-b88f-49bc-bede-fa634fa0c411",
   "metadata": {},
   "source": [
    "## Choosing a classification model\n",
    "\n",
    "1. **Choosing a classification model**: Now that we understand how to pull together text and numeric data into a single machine learning pipeline, we'll return to talking about our school budget dataset. In the sample dataset, there was only one text column. This is the format that `CountVectorizer` expects.\n",
    "\n",
    "2. **Main dataset: lots of text**: However, our main dataset has 14 text columns. In an interactive exercise during our NLP chapter we wrote a function combine_text_columns that put together all the text columns into a single column. We'll re-use this function as part of our Pipeline to process the text in the budget dataset.\n",
    "\n",
    "```python\n",
    "LABELS = ['Function', 'Use', 'Sharing', 'Reporting', 'Student_Type', 'Position_Type', 'Object_Type', 'Pre_K', 'Operating_Status']\n",
    "NON_LABELS = [c for c in df.columns if c not in LABELS]\n",
    "len(NON_LABELS) - len(NUMERIC_COLUMNS)\n",
    "```\n",
    "\n",
    "3. **Using pipeline with the main dataset**: Again, as we did in the last chapter, we'll use the `get_dummies` function from pandas to create our label array, and we'll also create our train-test split using the `multilabel_train_test_split` function. Hopefully, this is starting to feel familiar.\n",
    "\n",
    "4. **Using pipeline with the main dataset**: Now we'll create a pipeline for working with our main dataset. The beauty of this code, is that it has one change from the code we used for the sample dataset: we create a `FunctionTransformer` object using our combine_text_columns function instead of the simple selection function we used in the sample dataset. Other than this one change, the pipeline that processes the text, imputes the numeric columns, joins those results together and then fits a multilabel logistic regression remains the same.\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('TrainingSetSample.csv', index_col=0)\n",
    "dummy_labels = pd.get_dummies(df[LABELS])\n",
    "X_train, X_test, y_train, y_test = multilabel_train_test_split(\n",
    "    df[NON_LABELS], dummy_labels,\n",
    "    0.2)\n",
    "\n",
    "get_text_data = FunctionTransformer(combine_text_columns, validate=False)\n",
    "get_numeric_data = FunctionTransformer(lambda x: x[NUMERIC_COLUMNS], validate=False)\n",
    "pl = Pipeline([\n",
    "    ('union', FeatureUnion([\n",
    "        ('numeric_features', Pipeline([\n",
    "            ('selector', get_numeric_data),\n",
    "            ('imputer', Imputer())\n",
    "        ])),\n",
    "        ('text_features', Pipeline([\n",
    "            ('selector', get_text_data),\n",
    "            ('vectorizer', CountVectorizer())\n",
    "        ]))\n",
    "    ])\n",
    "     ),\n",
    "    ('clf', OneVsRestClassifier(LogisticRegression()))\n",
    "])\n",
    "```\n",
    "\n",
    "5. **Performance using main dataset**: We can now fit this Pipeline on the school budget dataset. As part of our exercises, we'll look at the results of this pipeline. Now we have infrastructure for processing our features and fitting a model. This infrastructure allows us to easily experiment with adapting different parts of the pipeline. Part of your challenge in the exercises will be to improve the performance of the pipeline.\n",
    "\n",
    "```python\n",
    "pl.fit(X_train, y_train)\n",
    "```\n",
    "\n",
    "6. **Flexibility of model step**: For example, we can easily experiment with different classes of models. All of our Pipeline code can stay the same, but we can update the last step to be a different class of model instead of `LogisticRegression`. For example, we could try a `RandomForestClassifier`, `NaiveBayesClassifier`, or a `KNeighborsClassifier` instead. In fact, you can look at the scikit-learn documentation and choose any model class you want!\n",
    "- Is current model the best?\n",
    "- Can quickly try different models with pipelines\n",
    "  - Pipeline preprocessing steps unchanged\n",
    "  - Edit the model step in your pipeline\n",
    "  - Random Forest, Naive Bayes, k-NN\n",
    "\n",
    "7. **Easily try new models using pipeline**: Here's an example of how simple it is to change the classification method in our scikit-learn Pipeline. We simply import a different class of model--in this case, we'll use a `RandomForestClassifier`. Then, the only other line that needs to change is the one that defines the classifier inside the Pipeline. This makes it very simple for us to try a large number of different model classes and determine which one is the best for the problem that we're working on.\n",
    "\n",
    "```python\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "pl = Pipeline([\n",
    "    ('union', FeatureUnion(\n",
    "        transformer_list=[\n",
    "            ('numeric_features', Pipeline([\n",
    "                ('selector', get_numeric_data),\n",
    "                ('imputer', Imputer())\n",
    "            ])),\n",
    "            ('text_features', Pipeline([\n",
    "                ('selector', get_text_data),\n",
    "                ('vectorizer', CountVectorizer())\n",
    "            ]))\n",
    "        ]\n",
    "    )),\n",
    "    ('clf', OneVsRest(RandomForestClassifier()))\n",
    "])\n",
    "```\n",
    "\n",
    "8. **Let's practice**: Now your task is to implement the classification pipeline on the school budget dataset. You will also get the chance to experiment with the Pipeline steps and model class. See how much you can improve the performance of the model!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f7630fa-b210-4e3e-af80-195ad51a036e",
   "metadata": {},
   "source": [
    "### Using FunctionTransformer on the main dataset\n",
    "\n",
    "In this exercise you're going to use `FunctionTransformer` on the primary budget data, before instantiating a multiple-datatype pipeline in the next exercise.\n",
    "\n",
    "Recall from Chapter 2 that you used a custom function `combine_text_columns` to select and properly format **text data** for tokenization; it is loaded into the workspace and ready to be put to work in a function transformer!\n",
    "\n",
    "Concerning the **numeric data**, you can use `NUMERIC_COLUMNS`, preloaded as usual, to help design a subset-selecting lambda function.\n",
    "\n",
    "You're all finished with sample data. The original `df` is back in the workspace, ready to use.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Complete the call to `multilabel_train_test_split()` by selecting `df[NON_LABELS]`.\n",
    "- Compute `get_text_data` by using `FunctionTransformer()` and passing in `combine_text_columns`. Be sure to also specify `validate=False`.\n",
    "- Use `FunctionTransformer()` to compute `get_numeric_data`. In the lambda function, select out the `NUMERIC_COLUMNS` of `x`. Like you did when computing `get_text_data`, also specify `validate=False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1499b850-5879-48fb-a124-53a80461f3ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/2024-01-19_school_budgeting_with_machine_learning_in_python/TrainingData.csv', index_col=0)\n",
    "LABELS_td = ['Function', 'Use', 'Sharing', 'Reporting', 'Student_Type', 'Position_Type', 'Object_Type', 'Pre_K', 'Operating_Status']\n",
    "categorize_label = lambda x: x.astype('category')\n",
    "df[LABELS_td] = df[LABELS_td].apply(categorize_label)\n",
    "NUMERIC_COLUMNS_td = df.select_dtypes(include='number').columns.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e405cbc-73e6-477c-89df-4af790bf0cfb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the dummy encoding of the labels\n",
    "dummy_labels = pd.get_dummies(df[LABELS_td])\n",
    "\n",
    "# Get the columns that are features in the original df\n",
    "NON_LABELS = [c for c in df.columns if c not in LABELS_td]\n",
    "\n",
    "# Split into training and test sets\n",
    "X_train, X_test, y_train, y_test = multilabel_train_test_split(df[NON_LABELS],\n",
    "                                                               dummy_labels,\n",
    "                                                               0.2, \n",
    "                                                               seed=123)\n",
    "\n",
    "# Preprocess the text data: get_text_data\n",
    "get_text_data = FunctionTransformer(combine_text_columns, validate=False)\n",
    "\n",
    "# Preprocess the numeric data: get_numeric_data\n",
    "get_numeric_data = FunctionTransformer(lambda x: x[NUMERIC_COLUMNS_td], validate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58fa6948-164a-44a6-b7ca-b1413af9f703",
   "metadata": {},
   "source": [
    "### Add a model to the pipeline\n",
    "\n",
    "You're about to take everything you've learned so far and implement it in a `Pipeline` that works with the real, [DrivenData][1] budget line item data you've been exploring.\n",
    "\n",
    "**Surprise!** The structure of the pipeline is exactly the same as earlier in this chapter:\n",
    "\n",
    "- the **preprocessing step** uses `FeatureUnion` to join the results of nested pipelines that each rely on `FunctionTransformer` to select multiple datatypes\n",
    "- the **model step** stores the model object\n",
    "\n",
    "- You can then call familiar methods like `.fit()` and `.score()` on the `Pipeline` object `pl`.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Complete the `'numeric_features'` transform with the following steps:\n",
    "  - `get_numeric_data`, with the name `'selector'`.\n",
    "  - `Imputer()`, with the name `'imputer'`.\n",
    "- Complete the `'text_features'` transform with the following steps:\n",
    "  - `get_text_data`, with the name `'selector'`.\n",
    "  - `CountVectorizer()`, with the name `'vectorizer'`.\n",
    "- Fit the pipeline to the training data.\n",
    "- Hit submit to compute the accuracy!\n",
    "\n",
    "\n",
    "[1]: https://www.drivendata.org/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a898a53-93fc-4bf8-bb3b-bd824949bed3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Complete the pipeline: pl\n",
    "pl = Pipeline([\n",
    "        ('union', FeatureUnion(\n",
    "            transformer_list = [\n",
    "                ('numeric_features', Pipeline([\n",
    "                    ('selector', get_numeric_data),\n",
    "                    ('imputer', SimpleImputer())\n",
    "                ])),\n",
    "                ('text_features', Pipeline([\n",
    "                    ('selector', get_text_data),\n",
    "                    ('vectorizer', CountVectorizer())\n",
    "                ]))\n",
    "             ]\n",
    "        )),\n",
    "        ('clf', OneVsRestClassifier(LogisticRegression(max_iter=1000)))\n",
    "    ])\n",
    "\n",
    "# Fit to the training data\n",
    "pl.fit(X_train, y_train)\n",
    "\n",
    "# Compute and print accuracy\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print(\"\\nAccuracy on budget dataset: \", accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "483f1868-9786-4dba-bcad-fb76a052b4ba",
   "metadata": {},
   "source": [
    "**Now that you've built the entire pipeline, you can easily start trying out different models by just modifying the `'clf'` step.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c6cfec0-a466-4e82-891a-58060b5d3443",
   "metadata": {},
   "source": [
    "### Try a different class of model\n",
    "\n",
    "Now you're cruising. One of the great strengths of pipelines is how easy they make the process of testing different models.\n",
    "\n",
    "Until now, you've been using the model step `('clf', OneVsRestClassifier(LogisticRegression()))` in your pipeline.\n",
    "\n",
    "But what if you want to try a different model? Do you need to build an entirely new pipeline? New nests? New FeatureUnions? Nope! You just have a simple one-line change, as you'll see in this exercise.\n",
    "\n",
    "In particular, you'll swap out the logistic-regression model and replace it with a [random forest][1] classifier, which uses the statistics of an ensemble of decision trees to generate predictions.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import the `RandomForestClassifier` from `sklearn.ensemble`.\n",
    "- Add a `RandomForestClassifier()` step named `'clf'` to the pipeline.\n",
    "- Hit submit to fit the pipeline to the training data and compute its accuracy.\n",
    "\n",
    "\n",
    "[1]: https://en.wikipedia.org/wiki/Random_forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4717873b-25bc-4800-8bca-96f90cc07179",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Edit model step in pipeline\n",
    "pl = Pipeline([\n",
    "        ('union', FeatureUnion(\n",
    "            transformer_list = [\n",
    "                ('numeric_features', Pipeline([\n",
    "                    ('selector', get_numeric_data),\n",
    "                    ('imputer', SimpleImputer())\n",
    "                ])),\n",
    "                ('text_features', Pipeline([\n",
    "                    ('selector', get_text_data),\n",
    "                    ('vectorizer', CountVectorizer())\n",
    "                ]))\n",
    "             ]\n",
    "        )),\n",
    "        ('clf', RandomForestClassifier(n_jobs=-1))\n",
    "    ])\n",
    "\n",
    "# Fit to the training data\n",
    "pl.fit(X_train, y_train)\n",
    "\n",
    "# Compute and print accuracy\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print(\"\\nAccuracy on budget dataset: \", accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72b2afe7-9dfe-4b47-aa04-1c6fd2c99604",
   "metadata": {},
   "source": [
    "**An accuracy improvement- amazing! All your work building the pipeline is paying off. It's now very simple to test different models!**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "428a37b1-98eb-4e9f-ae94-3147a6209bd0",
   "metadata": {},
   "source": [
    "### Can you adjust the model or parameters to improve accuracy?\n",
    "\r\n",
    "You just saw a substantial improvement in accuracy by swapping out the model. Pipelines are amazing!\r\n",
    "\r\n",
    "Can you make it better? Try changing the paramet`er n_estimat`ors `of RandomForestClassifie`r(), whose default value `is` 10, `to` 15."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09bcf191-ec06-4e2c-9513-c32c1fbe4f4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Edit model step in pipeline\n",
    "pl = Pipeline([\n",
    "        ('union', FeatureUnion(\n",
    "            transformer_list = [\n",
    "                ('numeric_features', Pipeline([\n",
    "                    ('selector', get_numeric_data),\n",
    "                    ('imputer', SimpleImputer())\n",
    "                ])),\n",
    "                ('text_features', Pipeline([\n",
    "                    ('selector', get_text_data),\n",
    "                    ('vectorizer', CountVectorizer())\n",
    "                ]))\n",
    "             ]\n",
    "        )),\n",
    "        ('clf', RandomForestClassifier(n_estimators=15, n_jobs=-1))\n",
    "    ])\n",
    "\n",
    "# Fit to the training data\n",
    "pl.fit(X_train, y_train)\n",
    "\n",
    "# Compute and print accuracy\n",
    "accuracy = pl.score(X_test, y_test)\n",
    "print(\"\\nAccuracy on budget dataset: \", accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee2f5449-be8f-4f0b-b392-ce01fe66d4a1",
   "metadata": {},
   "source": [
    "**It's time to get serious and work with the log loss metric. You'll learn expert techniques in the next chapter to take the model to the next level.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eda5985446aa34ac",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "# Learning from the experts\n",
    "\n",
    "In this chapter, you will learn the tricks used by the competition winner, and implement them yourself using scikit-learn."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab754cdc16525336",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## Learning from the expert: processing\n",
    "\n",
    "1. Learning from the expert: processing: In this chapter, we're going to see the tips and tricks that got the winning competitor to the top of the leaderboard.\n",
    "\n",
    "2. Learning from the expert: Some tricks are about text processing, some are about statistical methods, and some are about computational efficiency. While some of this may be new, some of it may be familiar from this course or other work you've done on DataCamp. By knowing which tools to combine, we can build extremely effective models.\n",
    "- Text processing\n",
    "- Statistical methods\n",
    "- Computational efficiency\n",
    "3. Learning from the expert: text preprocessing: In our introduction to natural language processing, we introduced the same tools that the winner used. These are common tools for manipulating text, and are often effective for improving models based on text data. The first trick the winner used was to tokenize on punctuation. By noticing that there are lots of characters like hyphens, periods, and underscores in the text we are working with, the winner picked out a better way of separating words than just spaces. The other trick was to use a range of n-grams in the model. By including unigrams and bigrams, the winner was more likely to capture important information that appears as multiple tokens in the text--for example, \"middle school.\"\n",
    "- NLP tricks for text data\n",
    "  - Tokenize on punctuation to avoid hyphens, underscores, etc.\n",
    "  - Include unigrams and bi-grams in the model to capture important information involving multiple tokens (e.g., \"middle school\")\n",
    "4. N-grams and tokenization: One of the massive benefits of building our models on top of scikit-learn is that many of these common tools are implemented for us and are well-tested by a large community. To change our tokenization and add bi-grams, we can change our call to `CountVectorizer`. We'll pass the regular expression to only accept alphanumeric characters in our tokens. We defined this expression back in chapter 2. We'll also add the parameter `ngram_range` equals (1, 2). This tells the `CountVectorizer` to include 1-grams and 2-grams in the vectorization. These changes to the call to `CountVectorizer` are the only changes we need to make to add these preprocessing steps. We can easily update our pipeline to include this change, which will be part of the exercises.\n",
    "```python\n",
    "vec = CountVectorizer(token_pattern=TOKENS_ALPHANUMERIC, ngram_range=(1, 2))\n",
    "```\n",
    "- Simple changes to `CountVectorizer` can improve your model\n",
    "  - `token_pattern` parameter to avoid breaking on hyphens\n",
    "  - `ngram_range` parameter to include unigrams and bigrams\n",
    "    - `ngram_range=(min_n, max_n)`\n",
    "  - alphanumeric tokenization\n",
    "    - `token_pattern=TOKENS_ALPHANUMERIC`\n",
    "5. Range of n-grams in scikit-learn: Then we fit our updated pipeline in the same way that we've been doing throughout the course: call the fit method on our Pipeline object. Getting predictions from our model is also the same as it has been throughout.\n",
    "\n",
    "6. Range of n-grams in scikit-learn: We use the predict_proba function to get our class probabilities. Because we have built our preprocessing into our Pipeline, we don't need to do any additional processing on our holdout data when we load it. The pipeline represents the entire prediction process from raw data to class probabilities.\n",
    "```python\n",
    "pl.fit(X_train, y_train)\n",
    "\n",
    "holdout = pd.read_csv('HoldoutData.csv', index_col=0)\n",
    "predictions = pl.predict_proba(holdout)\n",
    "prediction_df = pd.DataFrame(columns=pd.get_dummies(\n",
    "    df[LABELS]).columns, index=holdout.index, data=predictions)\n",
    "prediction_df.to_csv('predictions.csv')\n",
    "score = score_submission(pred_path='predictions.csv')\n",
    "```\n",
    "7. Let's practice!: Now it's your turn to implement these text processing tricks and see how they improve your score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ec602c90eef1da2",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/2024-01-19_school_budgeting_with_machine_learning_in_python/TrainingData.csv', index_col=0)\n",
    "LABELS_td = ['Function', 'Use', 'Sharing', 'Reporting', 'Student_Type', 'Position_Type', 'Object_Type', 'Pre_K', 'Operating_Status']\n",
    "categorize_label = lambda x: x.astype('category')\n",
    "df[LABELS_td] = df[LABELS_td].apply(categorize_label)\n",
    "NUMERIC_COLUMNS_td = df.select_dtypes(include='number').columns.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a24eff8b180a662",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "### How many tokens?\n",
    "\n",
    "Recall from previous chapters that how you tokenize text affects the n-gram statistics used in your model.\n",
    "\n",
    "Going forward, you'll use alphanumeric sequences, and only alphanumeric sequences, as tokens. Alphanumeric tokens contain only letters a-z and numbers 0-9 (no other characters). In other words, you'll tokenize on punctuation to generate n-gram statistics.\n",
    "\n",
    "In this exercise, you'll make sure you remember how to tokenize on punctuation.\n",
    "\n",
    "Assuming we tokenize on punctuation, accepting only alphanumeric sequences as tokens, how many tokens are in the following string from the main dataset?\n",
    "\n",
    "`'PLANNING,RES,DEV,& EVAL      '`\n",
    "If you want, we've loaded this string into the workspace as SAMPLE_STRING, but you may not need it to answer the question.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- ~~4, because `RES` and `DEV` are not tokens~~\n",
    "- **4, because `,` and `&` are not tokens** - **Commas, \"&\", and whitespace are not alphanumeric tokens.**\n",
    "- ~~7, because there are 4 different words, some commas, an `&` symbol, and whitespace~~\n",
    "- ~~7, because there are 7 whitespaces~~"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77efb023315aabc2",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "### Deciding what's a word\n",
    "\n",
    "Before you build up to the winning pipeline, it will be useful to look a little deeper into how the text features will be processed.\n",
    "\n",
    "In this exercise, you will use `CountVectorizer` on the training data `X_train` (preloaded into the workspace) to see the effect of tokenization on punctuation.\n",
    "\n",
    "Remember, since `CountVectorizer` expects a vector, you'll need to use the preloaded function, `combine_text_columns` before fitting to the training data.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Create `text_vector` by preprocessing `X_train` using `combine_text_columns`. This is important, or else you won't get any tokens!\n",
    "- Instantiate `CountVectorizer` as `text_features`. Specify the keyword argument `token_pattern=TOKENS_ALPHANUMERIC`.\n",
    "- Fit `text_features` to the `text_vector`.\n",
    "- Hit submit to print the first 10 tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80df8f194a816c0e",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Create the text vector\n",
    "text_vector = combine_text_columns(X_train)\n",
    "\n",
    "# Create the token pattern: TOKENS_ALPHANUMERIC\n",
    "TOKENS_ALPHANUMERIC = '[A-Za-z0-9]+(?=\\\\s+)'\n",
    "\n",
    "# Instantiate the CountVectorizer: text_features\n",
    "text_features = CountVectorizer(token_pattern=TOKENS_ALPHANUMERIC)\n",
    "\n",
    "# Fit text_features to the text vector\n",
    "text_features.fit(text_vector)\n",
    "\n",
    "# Print the first 10 tokens\n",
    "print(text_features.get_feature_names_out()[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7902824bf0429a3",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "### N-gram range in scikit-learn\n",
    "\n",
    "In this exercise, you'll insert a `CountVectorizer` instance into your pipeline for the main dataset, and compute multiple n-gram features to be used in the model.\n",
    "\n",
    "To look for ngram relationships at multiple scales, you will use the `ngram_range` parameter as Peter discussed in the video.\n",
    "\n",
    "**Special functions**: You'll notice a couple of new steps provided in the pipeline in this and many of the remaining exercises. Specifically, the `dim_red` step following the `vectorizer` step, and the `scale` step preceding the `clf` (classification) step.\n",
    "\n",
    "These have been added to account for the fact that you're using a reduced-size sample of the full dataset in this course. To make sure the models perform as the expert competition winner intended, we have to apply a [dimensionality reduction][1] technique, which is what the `dim_red` step does. We have to [scale the features][2] to lie between -1 and 1, which is what the `scale` step does.\n",
    "\n",
    "The `dim_red` step uses a scikit-learn function called SelectKBest(), applying something called the [chi-squared test][3] to select the K \"best\" features. The `scale` step uses a scikit-learn function called `MaxAbsScaler()` to squash the relevant features into the interval -1 to 1.\n",
    "\n",
    "You won't need to do anything extra with these functions here, complete the vectorizing pipeline steps below. However, notice how easy it was to add more processing steps to our pipeline!\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `CountVectorizer` from `sklearn.feature_extraction.text`.\n",
    "- Add a `CountVectorizer` step to the pipeline with the name `'vectorizer'`.\n",
    "- Set the token pattern to be `TOKENS_ALPHANUMERIC`.\n",
    "- Set the `ngram_range` to be `(1, 2)`.\n",
    "\n",
    "\n",
    "[1]: https://en.wikipedia.org/wiki/Dimensionality_reduction\n",
    "[2]: https://en.wikipedia.org/wiki/Feature_scaling\n",
    "[3]: https://en.wikipedia.org/wiki/Chi-squared_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c53832552c0b5a6",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Select 300 best features\n",
    "chi_k = 300\n",
    "\n",
    "# Perform preprocessing\n",
    "get_text_data = FunctionTransformer(combine_text_columns, validate=False)\n",
    "get_numeric_data = FunctionTransformer(lambda x: x[NUMERIC_COLUMNS_td], validate=False)\n",
    "\n",
    "# Create the token pattern: TOKENS_ALPHANUMERIC\n",
    "TOKENS_ALPHANUMERIC = '[A-Za-z0-9]+(?=\\\\s+)'\n",
    "\n",
    "# Instantiate pipeline: pl\n",
    "pl = Pipeline([\n",
    "        ('union', FeatureUnion(\n",
    "            transformer_list = [\n",
    "                ('numeric_features', Pipeline([\n",
    "                    ('selector', get_numeric_data),\n",
    "                    ('imputer', SimpleImputer())\n",
    "                ])),\n",
    "                ('text_features', Pipeline([\n",
    "                    ('selector', get_text_data),\n",
    "                    ('vectorizer', CountVectorizer(token_pattern=TOKENS_ALPHANUMERIC, ngram_range=(1, 2))),\n",
    "                    ('dim_red', SelectKBest(chi2, k=chi_k))\n",
    "                ]))\n",
    "             ]\n",
    "        )),\n",
    "        ('scale', MaxAbsScaler()),\n",
    "        ('clf', OneVsRestClassifier(LogisticRegression()))\n",
    "    ])\n",
    "\n",
    "# fit the pipeline to our training data\n",
    "pl.fit(X_train, y_train.values)\n",
    "\n",
    "# print the score of our trained pipeline on our test set\n",
    "log_loss_scorer = make_scorer(_multi_multi_log_loss)\n",
    "print(\"Logloss score of trained pipeline: \", log_loss_scorer(pl, X_test, y_test.values))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c74e054d7bc917d",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## Learning from the expert: a stats trick\n",
    "\n",
    "1. **Learning from the expert: a stats trick**: Now that you have experience with the text processing tricks that won the competition, we'll introduce some additional tools that the winner used.\n",
    "\n",
    "2. **Learning from the expert: interaction terms**: The statistical tool that the winner used is called interaction terms. We use bigrams to account for when words appear in a certain order. However, what if terms are not next to each other? For example, consider \"English Teacher for 2nd Grade\" and \"2nd Grade - budget for English Teacher\". If I want to identify this line item as a staff position for an elementary school, it helps to know that both \"2nd grade\" and \"English teacher\" appear. Interaction terms let us mathematically describe when tokens appear together.\n",
    "- Statistical tool that the winner used: interaction terms\n",
    "- Example\n",
    "  - English teacher for 2nd grade\n",
    "  - 2nd grade - budget for English teacher\n",
    "- Interaction terms mathematically describe when tokens appear together\n",
    "\n",
    "3. **Interaction terms: the math**: We're going to walk through a brief bit of math here to help explain interaction terms. If we look at a simple linear model, we have\n",
    "\n",
    "4. **Interaction terms: the math**: Beta-one times X1, beta 2 times X2.\n",
    "\n",
    "5. **Interaction terms: the math**: Here, X1 and X2 represent whether a particular token appears in the row. The betas are the coefficients, which represent how important that particular token is. To add an interaction term,\n",
    "\n",
    "6. **Interaction terms: the math**: we add another column to our array, X3 that equals X1 times X2. Because X1 and X2 are either 0 or 1, when we multiply them, we only get a 1 if both occur. This new column, X3, has its own coefficient, Beta 3, which can separately measure how important it is that X1 and X2 appear together. Again,\n",
    "- ![][sb2]\n",
    "\n",
    "7. **Adding interaction features with scikit-learn**: scikit-learn provides us with a straightforward way of adding interaction terms. In scikit-learn this functionality is called `PolynomialFeatures`, and we import it from the sklearn dot preprocessing module. As an example, we'll show the same x matrix as the last slide. When we create a `PolynomialFeatures` we tell it the degree of features to include. In our example, we looked at multiplying 2 columns together to see if they co-occurred, so degree equals 2, however, you could 3, 4, or more. While this can sometimes improve the model results, adding larger degrees very quickly scales the number of features outside what is computationally feasible. The `interaction_only` equals True parameter tells `PolynomialFeatures` that we don't need to multiply a column by itself, and we'll talk about include_bias on the next slide. When we fit and transform our array, x, we can see that we get the expected output from the last slide with the new column added. We opted not to include a bias term in our model, but you could have if you wanted to. A bias term is an offset for a model.\n",
    "```python\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "\n",
    "x = pd.DataFrame({'x1': [0, 1], 'x2': [1, 1]})\n",
    "\n",
    "interaction = PolynomialFeatures(degree=2,\n",
    "                                 interaction_only=True,\n",
    "                                 include_bias=False)\n",
    "interaction.fit_transform(x)\n",
    "\n",
    "[out]:\n",
    "array([[0., 1., 0.],\n",
    "       [1., 1., 1.]])\n",
    "```\n",
    "8. **A note about bias terms**: For example, if we plot age on the x-axis, and weights on the y-axis, we'd have a line that had a value 7-point-5 lbs when the baby is born, because a baby has a weight even when it is 0 years old! A bias term allows your model to account for situations where an x value of 0 should have a y value that is not 0, like in this example.\n",
    "- Bias term allows your model to have non-zero y value when x is 0\n",
    "- ![][sb3]\n",
    "\n",
    "9. **Sparse interaction features**: As we mentioned, adding interaction terms makes your X array grow exponentially. Because of computational concerns, our `CountVectorizer` returns an object called a sparse matrix. We won't dig into the details here, but the standard `PolynomialFeatures` object is not compatible with a sparse matrix. We provide you with a replacement, `SparseInteractions` which works with a sparse matrix. You only need to pass it the `degree` parameter for it to work in the same way.\n",
    "- `SparseInteractions(degree=2).fit_transform(x).toarray()`\n",
    "- The number of interaction terms grows exponentially\n",
    "- Our vectorizer saves memory by using a sparse matrix\n",
    "- `PolynomialFeatures` does not support sparse matrices\n",
    "- We have provided `SparseInteractions` to work for this problem\n",
    "10. **Let's practice!**: Now it's your turn to practice adding interaction terms with `SparseInteractions` to your code, and you can see what effect they have on your score!\n",
    "\n",
    "\n",
    "[sb2]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2024-01-19_school_budgeting_with_machine_learning_in_python/sb02.jpg\n",
    "[sb3]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2024-01-19_school_budgeting_with_machine_learning_in_python/sb03.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56a2ae7c66d6aed8",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "### Which models of the data include interaction terms?\n",
    "\n",
    "Recall from the video that interaction terms involve products of features.\n",
    "\n",
    "Suppose we have two features `x` and `y`, and we use models that process the features as follows:\n",
    "\n",
    "1. βx + βy + ββ\n",
    "2. βxy + βx + βy\n",
    "3. βx + βy + βx^2 + βy^2\n",
    "\n",
    "where β is a coefficient in your model (not a feature).\n",
    "\n",
    "Which expression(s) include interaction terms?\n",
    "\n",
    "**Answer the question**\n",
    "\n",
    "- ~~The first expression.~~\n",
    "- **The second expression.** - **The second expression includes the interaction term `βxy`, where the `xy` term is present, which represents interactions between features**\n",
    "- ~~The third expression.~~\n",
    "- ~~The first and third expressions.~~"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bb31eefd17fb468",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "### Implement interaction modeling in scikit-learn\n",
    "\n",
    "It's time to add interaction features to your model. The `PolynomialFeatures` object in scikit-learn does just that, but here you're going to use a custom interaction object, `SparseInteractions`. Interaction terms are a statistical tool that lets your model express what happens if two features appear together in the same row.\n",
    "\n",
    "`SparseInteractions` does the same thing as `PolynomialFeatures`, but it uses sparse matrices to do so. You can get the code for `SparseInteractions` at [this GitHub Gist][1].\n",
    "\n",
    "`PolynomialFeatures` and `SparseInteractions` both take the argument `degree`, which tells them what polynomial degree of interactions to compute.\n",
    "\n",
    "You're going to consider interaction terms of `degree=2` in your pipeline. You will insert these steps after the preprocessing steps you've built out so far, but before the classifier steps.\n",
    "\n",
    "Pipelines with interaction terms take a while to train (since you're making n features into n-squared features!), so as long as you set it up right, we'll do the heavy lifting and tell you what your score is!\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Add the interaction terms step using `SparseInteractions()` with `degree=2`. Give it a name of `'int'`, and make sure it is after the preprocessing step but before scaling.\n",
    "\n",
    "\n",
    "[1]: https://github.com/drivendataorg/box-plots-sklearn/blob/master/src/features/SparseInteractions.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ea3733bf27f574b",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Instantiate pipeline: pl\n",
    "pl = Pipeline([\n",
    "    ('union', FeatureUnion(\n",
    "        transformer_list=[\n",
    "            ('numeric_features', Pipeline([\n",
    "                ('selector', get_numeric_data),\n",
    "                ('imputer', SimpleImputer())\n",
    "            ])),\n",
    "            ('text_features', Pipeline([\n",
    "                ('selector', get_text_data),\n",
    "                ('vectorizer', CountVectorizer(token_pattern=TOKENS_ALPHANUMERIC,\n",
    "                                               ngram_range=(1, 2))),\n",
    "                ('dim_red', SelectKBest(chi2, k=chi_k))\n",
    "            ]))\n",
    "        ]\n",
    "    )),\n",
    "    ('int', SparseInteractions(degree=2)),\n",
    "    ('scale', MaxAbsScaler()),\n",
    "    ('clf', OneVsRestClassifier(LogisticRegression()))\n",
    "])\n",
    "\n",
    "# fit the pipeline to our training data\n",
    "pl.fit(X_train, y_train.values)\n",
    "\n",
    "# print the score of our trained pipeline on our test set\n",
    "log_loss_scorer = make_scorer(_multi_multi_log_loss)\n",
    "print(\"Logloss score of trained pipeline: \", log_loss_scorer(pl, X_test, y_test.values))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1d78a3253a84130",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "## Learning from the expert: the winning model\n",
    "\n",
    "1. Learning from the expert: the winning model: Congratulations! You've implemented nearly all of the winning pipeline. Just a couple more tweaks.\n",
    "\n",
    "2. Learning from the expert: hashing trick: We've mentioned that we need to balance adding new features with the computational cost of additional columns. For example, adding 3-grams or 4-grams is going to have an enormous increase in the size of the array. As the array grows in size, we need more computational power to fit our model. The \"hashing trick\" is a way of limiting the size of the matrix that we create without sacrificing too much model accuracy. A hash function takes an input, in this case a token, and outputs a hash value. For example, the hash value may be an integer like in this example below. We explicitly state how many possible outputs the hashing function may have. For example, we may say that we will only have 250 outputs of the hash function. The `HashingVectorizer` then maps every token to one of those 250 columns. Some columns will have multiple tokens that map to them. Interestingly, the original paper about the hashing function demonstrates that even if two tokens hash to the same value, there is very little effect on model accuracy in real world problems.\n",
    "- Adding new features may cause enormous increase in array size\n",
    "- Hashing is a way of increasing memory efficiency\n",
    "  - ![][sb4]\n",
    "- Hash function limits possible outputs, fixing array size\n",
    "3. When to use the hashing trick: We've been working with a small subset of the data for this course, however, the size of the full dataset is part of the challenge. In this situation, we want to do whatever we can to make our array of features smaller. Doing so is called \"**dimensionality reduction**\". The hashing trick is particularly useful in cases like this where we have a very large amount of text data.\n",
    "- Want to make array of features as small as possible\n",
    "  - Dimensionality reduction\n",
    "  - Particularly useful on large datasets\n",
    "  - e.g., lots of text data!\n",
    "4. Implementing the hashing trick in scikit-learn: Implementing the hashing trick is very simple in scikit-learn. Instead of using the `CountVectorizer`, which creates the bag of words representation, we change to the `HashingVectorizer`. We can pass this the same `token_pattern` and `ngram_range` as before. The parameters norm equals None and non_negative equals `True` let you drop in the `HashingVectorizer` as a replacement for the `CountVectorizer`. For more details on those two parameters, you can see the scikit-learn documentation. We can now use the `HashingVectorizer` in a `Pipeline` instead of the `CountVectorizer`.\n",
    "```python\n",
    "from sklearn.feature_extraction.text import HashingVectorizer\n",
    "vec = HashingVectorizer(norm=None, non_negative=True, token_pattern=TOKENS_ALPHANUMERIC, ngram_range=(1, 2))\n",
    "```\n",
    "5. The model that won it all: We've worked through the three categories of tools that the winner put together to win the competition. The NLP trick was to use bigrams and tokenize on punctuation. The statistical trick was to add interaction terms to the model. The computational trick was to use a `HashingVectorizer`. You can now code all of these in a scikit-learn Pipeline. However, there's one thing we haven't talked about yet. What is the class of model that the winner used? Was it a deep convolutional neural network? Extreme gradient boosted trees? An ensemble of local-expert elastic net regressions?\n",
    "- You now know all the expert moves to make on this dataset\n",
    "  - NLP: Range of n-grams, punctuation tokenization\n",
    "  - Stats: Interaction terms\n",
    "  - Computation: Hashing trick\n",
    "- What class of model was used?\n",
    "6. The model that won it all: No! The winning model was, in fact, the simple model we started with: **logistic regression**! The competition wasn't won by a cutting edge algorithm. Instead, it was won by thinking carefully about how to create features for the model and then adding a couple of easily implemented tricks. So, instead of reaching for those advanced algorithms that are hard to interpret and expensive to train, it's always worth it to see how far you can get with simpler methods.\n",
    "- And the winning model was…\n",
    "- Logistic regression!\n",
    "  - Carefully create features\n",
    "  - Easily implemented tricks\n",
    "  - Favor simplicity over complexity and see how far it takes you!\n",
    "7. Let's practice!: Now it's your chance to use all these tools together to create the winning Pipeline.\n",
    "\n",
    "[sb4]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2024-01-19_school_budgeting_with_machine_learning_in_python/sb04.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52bfe7ce36bc53c3",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "### Why is hashing a useful trick?\n",
    "\n",
    "In the video, Peter explained that a [hash][1] function takes an input, in your case a token, and outputs a hash value. For example, the input may be a string and the hash value may be an integer.\n",
    "\n",
    "We've loaded a familiar python datatype, a dictionary called `hash_dict`, that makes this mapping concept a bit more explicit. In fact, [python dictionaries ARE hash tables][2]!\n",
    "\n",
    "Print `hash_dict` in the IPython Shell to get a sense of how strings can be mapped to integers.\n",
    "\n",
    "By explicitly stating how many possible outputs the hashing function may have, we limit the size of the objects that need to be processed. With these limits known, computation can be made more efficient and we can get results faster, even on large datasets.\n",
    "\n",
    "Using the above information, answer the following:\n",
    "\n",
    "Why is hashing a useful trick?\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- ~~Hashing isn't useful unless you're working with numbers.~~\n",
    "- ~~Some problems are memory-bound and not easily parallelizable, but hashing parallelizes them.~~\n",
    "- **Some problems are memory-bound and not easily parallelizable, and hashing enforces a fixed length computation instead of using a mutable datatype (like a dictionary)**. - **Enforcing a fixed length can speed up calculations drastically, especially on large datasets!**\n",
    "- ~~Hashing enforces a mutable length computation instead of using a fixed length datatype, like a dictionary.~~\n",
    "\n",
    "\n",
    "[1]: https://en.wikipedia.org/wiki/Feature_hashing#Feature_vectorization_using_the_hashing_trick\n",
    "[2]: http://stackoverflow.com/questions/114830/is-a-python-dictionary-an-example-of-a-hash-table\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7ebe268149f404",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "hash_dict = {'and': 780, 'fluids': 354, 'fuel': 895, 'petro': 354, 'vend': 785}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33cad809b347f58c",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "### Implementing the hashing trick in scikit-learn\n",
    "\n",
    "In this exercise you will check out the scikit-learn implementation of `HashingVectorizer` before adding it to your pipeline later.\n",
    "\n",
    "As you saw in the video, `HashingVectorizer` acts just like `CountVectorizer` in that it can accept `token_pattern` and `ngram_range` parameters. The important difference is that it creates hash values from the text, so that we get all the computational advantages of hashing!\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `HashingVectorizer` from `sklearn.feature_extraction.text`.\n",
    "- Instantiate the `HashingVectorizer` as hashing_vec using the `TOKENS_ALPHANUMERIC` pattern.\n",
    "- Fit and transform `hashing_vec` using `text_data`. Save the result as `hashed_text`.\n",
    "- Hit submit to see some of the resulting hash values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77076c2c027b7e6",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Get text data: text_data\n",
    "text_data = combine_text_columns(X_train)\n",
    "\n",
    "# Create the token pattern: TOKENS_ALPHANUMERIC\n",
    "TOKENS_ALPHANUMERIC = '[A-Za-z0-9]+(?=\\\\s+)' \n",
    "\n",
    "# Instantiate the HashingVectorizer: hashing_vec\n",
    "hashing_vec = HashingVectorizer(token_pattern=TOKENS_ALPHANUMERIC)\n",
    "\n",
    "# Fit and transform the Hashing Vectorizer\n",
    "hashed_text = hashing_vec.fit_transform(text_data)\n",
    "\n",
    "# Create DataFrame and print the head\n",
    "hashed_df = pd.DataFrame(hashed_text.data)\n",
    "print(hashed_df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdb2743b64c9a1be",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "**As you can see, some text is hashed to the same value, but as Peter mentioned in the video, this doesn't neccessarily hurt performance.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d69d28740e93d850",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "### Build the winning model\n",
    "\n",
    "You have arrived! This is where all of your hard work pays off. It's time to build the model that won DrivenData's competition.\n",
    "\n",
    "You've constructed a robust, powerful pipeline capable of processing training and testing data. Now that you understand the data and know all of the tools you need, you can essentially solve the whole problem in a relatively small number of lines of code. Wow!\n",
    "\n",
    "All you need to do is add the `HashingVectorizer` step to the pipeline to replace the `CountVectorizer` step.\n",
    "\n",
    "The parameters `non_negative=True`, `norm=None`, and `binary=False` make the `HashingVectorizer` perform similarly to the default settings on the `CountVectorizer` so you can just replace one with the other.\n",
    "\n",
    "**Instructions**\n",
    "\n",
    "- Import `HashingVectorizer` from `sklearn.feature_extraction.text`.\n",
    "- Add a `HashingVectorizer` step to the pipeline.\n",
    "  - Name the step `'vectorizer'`.\n",
    "  - Use the `TOKENS_ALPHANUMERIC` token pattern.\n",
    "  - Specify the `ngram_range` to be `(1, 2)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4eb2b008d3998ad8",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Instantiate the winning model pipeline: pl\n",
    "pl = Pipeline([\n",
    "    ('union', FeatureUnion(\n",
    "        transformer_list=[\n",
    "            ('numeric_features', Pipeline([\n",
    "                ('selector', get_numeric_data),\n",
    "                ('imputer', SimpleImputer())\n",
    "            ])),\n",
    "            ('text_features', Pipeline([\n",
    "                ('selector', get_text_data),\n",
    "                ('vectorizer', HashingVectorizer(token_pattern=TOKENS_ALPHANUMERIC,\n",
    "                                                 alternate_sign=False, norm=None, binary=False,\n",
    "                                                 ngram_range=(1, 2))),\n",
    "                ('dim_red', SelectKBest(chi2, k=chi_k))\n",
    "            ]))\n",
    "        ]\n",
    "    )),\n",
    "    ('int', SparseInteractions(degree=2)),\n",
    "    ('scale', MaxAbsScaler()),\n",
    "    ('clf', OneVsRestClassifier(LogisticRegression()))\n",
    "])\n",
    "\n",
    "# fit the pipeline to our training data\n",
    "pl.fit(X_train, y_train.values)\n",
    "\n",
    "# print the score of our trained pipeline on our test set\n",
    "log_loss_scorer = make_scorer(_multi_multi_log_loss)\n",
    "print(\"Logloss score of trained pipeline: \", log_loss_scorer(pl, X_test, y_test.values))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b10ec584bb223614",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "**Looks like the performance is about the same, but this is expected since the HashingVectorizer should work the same as the CountVectorizer. Try this pipeline out on the whole dataset on your local machine to see its full power!**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e336c29b4d7d3fb",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "### What tactics got the winner the best score?\n",
    "\n",
    "Now you've implemented the winning model from start to finish. If you want to use this model locally, [this Jupyter notebook][1] contains all the code you've worked so hard on. You can now take that code and build on it!\n",
    "\n",
    "Let's take a moment to reflect on why this model did so well. What tactics got the winner the best score?\n",
    "\n",
    "**Answer the question**\n",
    "\n",
    "- The winner used a 500 layer deep convolutional neural network to master the budget data.\n",
    "- The winner used an ensemble of many models for classification, taking the best results as predictions.\n",
    "- **The winner used skillful NLP, efficient computation, and simple but powerful stats tricks to master the budget data.** - **Often times simpler is better, and understanding the problem in depth leads to simpler solutions!**\n",
    "\n",
    "[1]: https://github.com/datacamp/course-resources-ml-with-experts-budgets/blob/master/notebooks/1.0-full-model.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "191e4453897802f8",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "# Next steps and the social impact of your work\n",
    "\n",
    "1. Next steps and the social impact of your work: Congratulations on finishing this course! I hope you enjoyed it and learned some useful tools to add to your data science toolbox. Your new skills have the ability to really make a difference. The volume of data is growing everywhere, not only in the tech industry. And with this growth in data comes opportunity: the opportunity to use your data science skills to make people's lives better.\n",
    "\n",
    "2. Can you do better?: One of our goals has been to demonstrate how flexible and powerful the Pipeline API is in scikit-learn. We've successively added small code changes that have had a big effect on our model. Now we want you to think about how you might improve things. Some areas to look at might be: Including stemming or stop-word removal in your natural language processing. Trying a different class of model like RandomForest, KNN or Naive Bayes. Doing further processing on the numeric columns and handling NaNs differently than the default Imputer. One of the benefits of scikit-learn is GridSearch (as you learned about in the supervised learning course). Check out the scikit-learn documentation to see how to perform grid search over every object in a Pipeline. You could even look around the scikit-learn docs and experiment with a new technique you haven't used before. Feel free to sign up on DrivenData and experiment with some of these methods on the full dataset. Learning how these approaches effect real-world data can be a huge boost to your skills.\n",
    "- You've seen the .exibility of the pipeline steps\n",
    "- Quickly test ways of improving your submission\n",
    "  - NLP: Stemming, stop-word removal\n",
    "  - Model: RandomForest, k-NN, Naïve Bayes\n",
    "  - Numeric Preprocessing: Imputation strategies\n",
    "  - Optimization: Grid search over pipeline objects\n",
    "  - Experiment with new scikit-learn techniques\n",
    "- Work with the full dataset at DrivenData!\n",
    "3. Hundreds of hours saved: The goal of building this model was to make an education non-profit more effective in helping school districts with their budgeting decisions. Having a model like the one we have built saves hundreds of hours a year in human time that was spent simply labeling line items. Now budget analysts can spend their time on the decisions that really matter.\n",
    "- Make schools more e)cient by improving their budgeting decisions\n",
    "- Saves hundreds of hours each year that humans spent labeling line items\n",
    "- Can spend more time on the decisions that really ma\n",
    "4. DrivenData: Data Science to save the world: If you're interested in other ways to use data science to have a social impact, we invite you to check out www-dot-drivendata-dot-org. We've always got competitions just like this running where you can look at new datasets, try interesting methods, learn from the community, and have an impact.\n",
    "- Other ways to use data science to have a social impact at [DrivenData][1]\n",
    "- Improve your data science skills while helping meaningful organizations thrive\n",
    "- Win some cash prizes while you're at it!\n",
    "\n",
    "[1]: https://www.drivendata.org/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9be75127-db98-4283-afb4-84cfc07eef71",
   "metadata": {},
   "source": [
    "# Certificate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5974cd79b36dd8be",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "![certificate][1]\n",
    "\n",
    "[1]: https://raw.githubusercontent.com/trenton3983/DataCamp/master/Images/2024-01-19_school_budgeting_with_machine_learning_in_python/2024-01-27_DataCamp_school_budgeting_with_machine_learning_in_python.jpg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbb89005b7ba5544",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py312",
   "language": "python",
   "name": "py312"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}