Student_MLE_MiniProject_ML.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yzr5Eo_qxHQi"
      },
      "source": [
        "# Mini Project: Build a Machine Learning Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_NvUsqw8xg26"
      },
      "source": [
        "## Predict Total Fare on the NYC Taxi Dataset\n",
        "\n",
        "Welcome to the NYC Taxi Fare Prediction project! In this Colab, we will continue using the NYC Taxi Dataset to predict the fare amount for taxi rides using a subset of available features. We will go through three main stages: building a baseline model, creating a full model, and performing hyperparameter tuning to enhance our predictions.\n",
        "\n",
        "Now that you've completed exploratory data analysis on this dataset you should have a good understanding of the feature space.\n",
        "\n",
        "## Project Objectives\n",
        "\n",
        "The primary objectives of this project are as follows:\n",
        "\n",
        "Baseline Model: We will start by building a simple baseline model to establish a benchmark for our predictions. This model will serve as a starting point to compare the performance of our subsequent models.\n",
        "\n",
        "Full Model: Next, we will develop a more comprehensive model that leverages machine learning techniques to improve prediction accuracy. We will use Scikit-Learn's model pipeline to build a framework that enables rapid experimentation.\n",
        "\n",
        "Hyperparameter Tuning: Lastly, we will optimize our full model by fine-tuning its hyperparameters. By systematically adjusting the parameters that control model behavior, we aim to achieve the best possible performance for our prediction task."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lbJFWLELlI6N"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.model_selection import train_test_split, GridSearchCV\n",
        "from sklearn.metrics import mean_absolute_error\n",
        "\n",
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.preprocessing import StandardScaler, OneHotEncoder, FunctionTransformer\n",
        "from sklearn.compose import ColumnTransformer\n",
        "from sklearn.linear_model import LinearRegression\n",
        "from sklearn.ensemble import RandomForestRegressor"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gJtGZwwG2WTW"
      },
      "source": [
        "Load the NYC taxi dataset into a Pandas DataFrame and do a few basic checks to ensure the data is loaded properly. Note, there are several months of data that can be used. For simplicity, use the Yellow Taxi 2022-01 parquet file [here](https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2022-01.parquet). Here are your tasks:\n",
        "\n",
        "  1. Load the `yellow_tripdata_2022-01.parquet` file into Pandas.\n",
        "  2. Print the first 5 rows of data.\n",
        "  3. Drop any rows of data that contain NULL values.\n",
        "  4. Create a new feature, 'trip_duration' that captures the duration of the trip in minutes.\n",
        "  5. Create a varible named 'target_variable' to store the name of the thing we're trying to predict, 'total_amount'.\n",
        "  6. Create a list called 'feature_cols' containing the feature names that we'll be using to predict our target variable. The list should contain 'VendorID', 'trip_distance', 'payment_type', 'PULocationID', 'DOLocationID', and 'trip_duration'."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "db--eb8zlNNg"
      },
      "outputs": [],
      "source": [
        "# Load the dataset into a pandas DataFrame (from https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TslBuHoXl_o1"
      },
      "outputs": [],
      "source": [
        "# Display the first few rows of the dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4ybS7HV6HHQL"
      },
      "outputs": [],
      "source": [
        "# Drop rows with missing values."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZC4p9PEjnSm8"
      },
      "outputs": [],
      "source": [
        "# Create new feature, 'trip_duration'."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "R5d84tINrG9d"
      },
      "outputs": [],
      "source": [
        "# Create a list called feature_col to store column names"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eq2kgevn51LY"
      },
      "source": [
        "Splitting a dataset into training and test sets is a crucial step in machine learning model development. It allows us to evaluate the performance and generalization ability of our models accurately. The training set is used to train the model, while the test set serves as an independent sample for evaluating its performance.\n",
        "\n",
        "1. **Model Training**: The training set is used to fit the model, allowing it to learn the underlying patterns and relationships between the features and the target variable. By exposing the model to a diverse range of examples in the training set, it can capture the underlying structure of the data.\n",
        "\n",
        "2. **Model Evaluation**: The test set, which is independent of the training set, is crucial for evaluating how well the trained model generalizes to unseen data. It provides an unbiased assessment of the model's performance on new instances. By measuring the model's accuracy, precision, recall, or other evaluation metrics on the test set, we can estimate how well the model will perform on unseen data.\n",
        "\n",
        "3. **Preventing Overfitting**: Overfitting occurs when a model learns the training data's noise and idiosyncrasies instead of the underlying patterns. By evaluating the model on the test set, we can identify if the model is overfitting. If the model performs significantly worse on the test set compared to the training set, it indicates overfitting. In such cases, we might need to adjust the model, feature selection, or regularization techniques to improve generalization.\n",
        "\n",
        "4. **Hyperparameter Tuning**: Splitting the dataset allows us to perform hyperparameter tuning on the model. Hyperparameters are configuration settings that control the learning process, such as learning rate, regularization strength, or the number of hidden layers in a neural network. By using a validation set (often created from a portion of the training set), we can iteratively adjust the hyperparameters and select the best combination that maximizes the model's performance on the validation set. The final evaluation on the test set provides an unbiased estimate of the model's performance.\n",
        "\n",
        "By splitting the dataset into training and test sets, we can ensure that our models are both well-trained and accurately evaluated. This separation helps us understand how the model will perform on new, unseen data, which is critical for assessing its effectiveness and making informed decisions about its deployment.\n",
        "\n",
        "Here is your task:\n",
        "\n",
        "  1. Use Scikit-Learn's [train_test_split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) to split the data into training and test sets. Don't forget to set the random state."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "grdbA-I6rMGC"
      },
      "outputs": [],
      "source": [
        "# Split dataset into training and test sets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VIAhf_oA7PGx"
      },
      "source": [
        "The importance of a baseline model, even if it uses a simple strategy like always predicting the mean, cannot be understated. Here's why a baseline model is valuable:\n",
        "\n",
        "1. **Performance Comparison**: A baseline model serves as a reference point for evaluating the performance of more sophisticated models. By establishing a simple yet reasonable baseline, we can determine whether our advanced models offer any significant improvement over this basic approach. It helps us set realistic expectations and gauge the effectiveness of our efforts.\n",
        "\n",
        "2. **Model Complexity**: A baseline model provides insight into the complexity required to solve the prediction task. If a simple strategy like predicting the median performs reasonably well, it suggests that the problem might not necessitate complex modeling techniques. Conversely, if the baseline model performs poorly, it indicates the presence of more intricate patterns that need to be captured by more sophisticated models.\n",
        "\n",
        "3. **Minimum Performance Requirement**: A baseline model can establish a minimum performance requirement for a predictive task. If we cannot outperform the baseline, it suggests that our models have failed to capture even the most fundamental relationships within the data. In such cases, we may need to revisit our data preprocessing steps, feature engineering techniques, or consider other external factors affecting the task.\n",
        "\n",
        "4. **Identifying Data Issues**: A baseline model can help identify potential issues within the dataset. If the baseline model performs poorly, it may indicate problems like missing values, outliers, or data inconsistencies. These issues can be further investigated and resolved to improve the overall model performance.\n",
        "\n",
        "While a baseline model like always predicting the median may not offer the highest prediction accuracy, its importance lies in its role as a starting point for model development and evaluation. It provides a solid foundation for comparing and assessing the performance of more complex models, ensuring that any improvements made are meaningful and significant.\n",
        "\n",
        "Here is your task:\n",
        "\n",
        "  1. Create a model that always predicts the mean total fare of the training dataset. Use Scikit-Learn's [mean_absolute_error](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html) to evaluate this model. Is it any good?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6vV1UsNTFX22"
      },
      "outputs": [],
      "source": [
        "# Create a baseline for mean absolute error of total amount"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sCKDGKpg8MVb"
      },
      "source": [
        "With a baseline metric in place, we can try to build a machine learning model. Obviously, if the model can't beat the baseline then there are some major issues to be resolved.\n",
        "\n",
        "It's always a good idea to start with a simple machine learning model, like linear regression, and build upon it if necessary.\n",
        "\n",
        "Here are your tasks:\n",
        "\n",
        "  1. Use Scikit-Learn's [ColumnTransformer](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html) to preprocess the categorical and continuous features independently. Apply the [StandardScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html) to the continuous columns and [OneHotEncoder](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html) to the categorical columns.\n",
        "\n",
        "  One-hot encoding is a popular technique used to represent categorical variables numerically in machine learning models. It transforms categorical features into a binary vector representation, where each category is represented by a binary column. Here's an explanation of one-hot encoding:\n",
        "\n",
        "  When working with categorical variables, such as colors (e.g., red, blue, green) or vehicle types (e.g., car, truck, motorcycle), machine learning algorithms often require numerical inputs. However, directly assigning numerical values to categories can introduce unintended relationships or orderings between them. For example, assigning the values 0, 1, and 2 to the categories red, blue, and green may imply a sequential relationship, which is not desired.\n",
        "\n",
        "  One-hot encoding solves this problem by creating new binary columns, equal to the number of unique categories in the original feature. Each binary column represents a specific category and takes a value of 1 if the data point belongs to that category, and 0 otherwise. This encoding ensures that no implicit ordering or relationship exists between the categories.\n",
        "\n",
        "  2. Integrate the preprocessor in the previous step with Scikit-Learn's [LinearRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html) model using a [Pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html).\n",
        "\n",
        "  3. Train the pipeline on the training data.\n",
        "\n",
        "  4. Evaluate the model using mean absolute error as a metric on the test data. Does the model beat the baseline?\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Use Scikit-Learn's ColumnTransformer to preprocess the categorical and\n",
        "# continuous features independently."
      ],
      "metadata": {
        "id": "KckTWwsiah5p"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yVzxrDu6w506"
      },
      "outputs": [],
      "source": [
        "# Create a pipeline object containing the column transformations and regression\n",
        "# model."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Fit the pipeline on the training data."
      ],
      "metadata": {
        "id": "7Dhli3fkalIS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Make predictions on the test data."
      ],
      "metadata": {
        "id": "mUkcHyJpamxC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d1669p0UBHrx"
      },
      "source": [
        "Random Forest Regression and Linear Regression are two commonly used regression algorithms, each with its own advantages and suitability for different scenarios. Random Forest Regression offers several advantages over Linear Regression, including:\n",
        "\n",
        "1. **Non-linearity**: Random Forest Regressor is capable of capturing non-linear relationships between features and the target variable. In contrast, Linear Regression assumes a linear relationship between the features and the target. When faced with non-linear relationships or complex feature interactions, Random Forest Regressor can provide more accurate predictions.\n",
        "\n",
        "2. **Robustness to Outliers**: Random Forest Regressor is generally more robust to outliers compared to Linear Regression. Outliers can disproportionately impact the coefficients and predictions of Linear Regression models. However, as an ensemble of decision trees, Random Forest Regressor can mitigate the effect of outliers by averaging predictions from multiple trees.\n",
        "\n",
        "3. **Feature Importance**: Random Forest Regressor provides a measure of feature importance, which helps identify the most influential features for making predictions. This information is useful for feature selection, understanding the underlying relationships in the data, and gaining insights into the problem domain. Unlike Linear Regression, which provides coefficient values indicating the direction and magnitude of relationships, Random Forest Regressor explicitly highlights feature importance.\n",
        "\n",
        "4. **Handling of Categorical Variables**: Random Forest Regressor can effectively handle categorical variables without requiring pre-processing steps like one-hot encoding. It can directly incorporate categorical variables into the model, making it more convenient when working with mixed data types. In contrast, Linear Regression often requires categorical variables to be encoded or transformed before use.\n",
        "\n",
        "5. **Handling of High-Dimensional Data**: Random Forest Regressor can handle datasets with a large number of features (high dimensionality) by automatically selecting subsets of features during the construction of individual decision trees. This reduces the risk of overfitting, which is a concern with Linear Regression when dealing with high-dimensional data.\n",
        "\n",
        "6. **Resistance to Multicollinearity**: Random Forest Regressor is less affected by multicollinearity, which occurs when predictor variables are highly correlated. In Linear Regression, highly correlated features can lead to unstable coefficient estimates, making it challenging to interpret the individual effects of each feature. Random Forest Regressor, as an ensemble approach, is less impacted by multicollinearity because each tree is built independently.\n",
        "\n",
        "Here are your tasks:\n",
        "\n",
        "  1. Build a Random Forest Regressor model using Scikit-Learn's [RandomForestRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html) and train it on the train data.\n",
        "\n",
        "  2. Evaluate the performance of the model on the test data using mean absolute error as a metric. Mess around with various input parameter configurations to see how they affect the model. Can you beat the performance of the linear regression model?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QU4tTQKl4q_n"
      },
      "outputs": [],
      "source": [
        "# Build random forest regressor model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "P6SIX9EK5RYi"
      },
      "outputs": [],
      "source": [
        "# Make predictions on the test data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5CuYk9WBFF2X"
      },
      "source": [
        "Hyperparameter tuning plays a critical role in machine learning model development. It involves selecting the optimal values for the hyperparameters, which are configuration settings that control the behavior of the learning algorithm. Here's why hyperparameter tuning is so important in ML:\n",
        "\n",
        "1. **Optimizing Model Performance**: The choice of hyperparameters can significantly impact the model's performance. By fine-tuning the hyperparameters, we can improve the model's accuracy, precision, recall, or other performance metrics. It helps to extract the maximum predictive power from the chosen algorithm and ensures that the model is well-suited to the specific problem at hand.\n",
        "\n",
        "2. **Avoiding Overfitting and Underfitting**: Hyperparameter tuning helps strike a balance between overfitting and underfitting.\n",
        "\n",
        "3. **Exploring Model Complexity**: Hyperparameter tuning enables us to explore the complexity of the model. For instance, in algorithms like decision trees or neural networks, we can adjust the number of layers, the number of neurons, or the maximum depth of the tree. By systematically modifying these hyperparameters, we can understand how different levels of complexity impact the model's performance and find the right balance between simplicity and complexity.\n",
        "\n",
        "Note, there are multiple approaches to hyperparemeter tuning.  \n",
        "\n",
        "While grid search is the easiest to understand and implement there are many advantages of Bayesian search over grid search for hyperparameter tuning:\n",
        "\n",
        "1. **Efficiency**: Bayesian search is generally more efficient than grid search. Grid search explores all possible combinations of hyperparameter values, which can be computationally expensive and time-consuming, especially when dealing with a large number of hyperparameters or a wide range of values. Bayesian search, on the other hand, intelligently selects the next hyperparameter configuration to evaluate based on the results of previous evaluations. It focuses on areas of the hyperparameter space that are more likely to yield better performance, reducing the number of evaluations needed.\n",
        "\n",
        "2. **Flexibility**: Bayesian search is flexible in handling continuous and discrete hyperparameters. It can handle both types of hyperparameters naturally and effectively. In contrast, grid search is more suitable for discrete hyperparameters but may struggle with continuous ones, as it requires discretization or defining a finite set of values to search over.\n",
        "\n",
        "3. **Adaptive Search**: Bayesian search adapts its search strategy based on the results of previous evaluations. It maintains a probability distribution over the hyperparameter space, updating it with each evaluation. This allows it to dynamically allocate more evaluations to promising regions and explore unexplored areas. In contrast, grid search follows a fixed and predefined search grid, regardless of the results of previous evaluations.\n",
        "\n",
        "4. **Better Convergence**: Bayesian search has the potential to converge to the optimal hyperparameter configuration more quickly.\n",
        "\n",
        "Here are your tasks:\n",
        "\n",
        "  1. Perform a grid-search on a Random Forest Regressor model. Only search the space for the parameters 'n_estimators', 'max_depth', and 'min_samples_split'. Note, this can take some time to run. Make sure you set reasonable boundaries for the search space. Use Scikit-Learn's [GridSearchCV](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) method.\n",
        "\n",
        "  2. After you've identified the best parameters, train a random forest regression model using these parameters on the full training data.\n",
        "\n",
        "  3. Evaluate the model from the previous step using the test data. How does your model perform?"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Define the hyperparameters to tune."
      ],
      "metadata": {
        "id": "HlmZeNDOa1zs"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "NWaMWOH6Kdho"
      },
      "outputs": [],
      "source": [
        "# Perform grid search to find the best hyperparameters. This could take a while."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Get the best model and its parameters."
      ],
      "metadata": {
        "id": "_JO-wsm7a58f"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "W7ier_bt23RS"
      },
      "outputs": [],
      "source": [
        "# Fit the best classifier on the training data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "ZHPrwD41tq13"
      },
      "outputs": [],
      "source": [
        "# Make predictions on the test data"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}