Preprocessing/Data_representation_in_scikit_learn.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Data representation in scikit-learn.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BiE4HQuXA1jb"
      },
      "source": [
        "# **Import libaray**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CYcSX5ofA8Yz"
      },
      "source": [
        "import numpy as np \n",
        "import pandas as pd \n",
        "import seaborn as sns \n",
        "import matplotlib.pyplot as plt\n",
        "from prettytable import PrettyTable\n",
        "from sklearn.metrics import roc_curve, auc\n",
        "from mlxtend.plotting import plot_confusion_matrix \n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import classification_report, confusion_matrix\n",
        "from sklearn.tree import DecisionTreeClassifier\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.svm import LinearSVC\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-dEH-aOwYdNx"
      },
      "source": [
        "# **Introduction**\n",
        "\n",
        "Let’s start with the basics and consider the data representation used in scikit-learn, which is essentially a tabular dataset.At a high-level, for a supervised learning problem the tabular dataset will be comprised of both X and y variables while an unsupervised learning problem will constitute of only X variables.\n",
        "\n",
        "At a high-level, X variables are also known as independent variables and they can be either quantitative or qualitative descriptions of samples of interests while the y variable is also known as the dependent variable and they are essentially the target or response variable that predictive models are built to predict."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Rl_C9QLeAc8"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1OH5lF4gI10mL3T4dG2YYhhjp2Ich7p_U)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dSRa4CGgeNHG"
      },
      "source": [
        "For example, if we’re building a predictive model to predict whether individuals have a disease or not the disease/non-disease status is the y variable whereas health indicators obtained by clinical test results are used as X variables.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "69-GFSEUYpQA"
      },
      "source": [
        "# **Loading data from CSV files via Pandas**\n",
        "\n",
        "Practically, the contents of a dataset can be stored in a CSV file and it can be read in using the Pandas library via the pd.read_csv() function. Thus, the data structure of the loaded data is known as the Pandas DataFrame."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fYNNsGgxmxyQ",
        "outputId": "db152256-b938-4017-cba9-ab9d37289e47"
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bkoAdyxInLR4"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "data = pd.read_csv(\"//content/drive/MyDrive/Datasets/Student field Recommendation /Placement_Data_Full_Class.csv\")"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OE_f4-bToHWA",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        },
        "outputId": "47538ec6-98c7-43f0-d835-f443e3e86808"
      },
      "source": [
        "data.head()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sl_no</th>\n",
              "      <th>gender</th>\n",
              "      <th>ssc_p</th>\n",
              "      <th>ssc_b</th>\n",
              "      <th>hsc_p</th>\n",
              "      <th>hsc_b</th>\n",
              "      <th>hsc_s</th>\n",
              "      <th>degree_p</th>\n",
              "      <th>degree_t</th>\n",
              "      <th>workex</th>\n",
              "      <th>etest_p</th>\n",
              "      <th>specialisation</th>\n",
              "      <th>mba_p</th>\n",
              "      <th>status</th>\n",
              "      <th>salary</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>M</td>\n",
              "      <td>67.00</td>\n",
              "      <td>Others</td>\n",
              "      <td>91.00</td>\n",
              "      <td>Others</td>\n",
              "      <td>Commerce</td>\n",
              "      <td>58.00</td>\n",
              "      <td>Sci&amp;Tech</td>\n",
              "      <td>No</td>\n",
              "      <td>55.0</td>\n",
              "      <td>Mkt&amp;HR</td>\n",
              "      <td>58.80</td>\n",
              "      <td>Placed</td>\n",
              "      <td>270000.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>M</td>\n",
              "      <td>79.33</td>\n",
              "      <td>Central</td>\n",
              "      <td>78.33</td>\n",
              "      <td>Others</td>\n",
              "      <td>Science</td>\n",
              "      <td>77.48</td>\n",
              "      <td>Sci&amp;Tech</td>\n",
              "      <td>Yes</td>\n",
              "      <td>86.5</td>\n",
              "      <td>Mkt&amp;Fin</td>\n",
              "      <td>66.28</td>\n",
              "      <td>Placed</td>\n",
              "      <td>200000.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>M</td>\n",
              "      <td>65.00</td>\n",
              "      <td>Central</td>\n",
              "      <td>68.00</td>\n",
              "      <td>Central</td>\n",
              "      <td>Arts</td>\n",
              "      <td>64.00</td>\n",
              "      <td>Comm&amp;Mgmt</td>\n",
              "      <td>No</td>\n",
              "      <td>75.0</td>\n",
              "      <td>Mkt&amp;Fin</td>\n",
              "      <td>57.80</td>\n",
              "      <td>Placed</td>\n",
              "      <td>250000.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>M</td>\n",
              "      <td>56.00</td>\n",
              "      <td>Central</td>\n",
              "      <td>52.00</td>\n",
              "      <td>Central</td>\n",
              "      <td>Science</td>\n",
              "      <td>52.00</td>\n",
              "      <td>Sci&amp;Tech</td>\n",
              "      <td>No</td>\n",
              "      <td>66.0</td>\n",
              "      <td>Mkt&amp;HR</td>\n",
              "      <td>59.43</td>\n",
              "      <td>Not Placed</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>M</td>\n",
              "      <td>85.80</td>\n",
              "      <td>Central</td>\n",
              "      <td>73.60</td>\n",
              "      <td>Central</td>\n",
              "      <td>Commerce</td>\n",
              "      <td>73.30</td>\n",
              "      <td>Comm&amp;Mgmt</td>\n",
              "      <td>No</td>\n",
              "      <td>96.8</td>\n",
              "      <td>Mkt&amp;Fin</td>\n",
              "      <td>55.50</td>\n",
              "      <td>Placed</td>\n",
              "      <td>425000.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   sl_no gender  ssc_p    ssc_b  ...  specialisation  mba_p      status    salary\n",
              "0      1      M  67.00   Others  ...          Mkt&HR  58.80      Placed  270000.0\n",
              "1      2      M  79.33  Central  ...         Mkt&Fin  66.28      Placed  200000.0\n",
              "2      3      M  65.00  Central  ...         Mkt&Fin  57.80      Placed  250000.0\n",
              "3      4      M  56.00  Central  ...          Mkt&HR  59.43  Not Placed       NaN\n",
              "4      5      M  85.80  Central  ...         Mkt&Fin  55.50      Placed  425000.0\n",
              "\n",
              "[5 rows x 15 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ShNfkqfBeoWq"
      },
      "source": [
        "Afterwards, data processing can be performed on the DataFrame using the wide range of Pandas functions for handling missing data (i.e. dropping missing data or filling them in with imputed values), selecting specific column or range of columns, performing feature transformations, conditional filtering of data, etc.\n",
        "In the following example, we will separate the DataFrame as X and y variables, which will be used shortly for model building."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y7r4GCZ_r578"
      },
      "source": [
        "# **Prepraring X and Y**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j9t6Ik9SAUPX"
      },
      "source": [
        "X=data.drop('specialisation',axis=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f52X64tDAYXY"
      },
      "source": [
        "y=data[['specialisation']]\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5CKqdX86iIBc"
      },
      "source": [
        "# **References**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OZF79FvqiNhO"
      },
      "source": [
        "[How to Master Scikit-learn for Data Science](https://towardsdatascience.com/how-to-master-scikit-learn-for-data-science-c29214ec25b0)"
      ]
    }
  ]
}