IT496_ViktoriiaO_naive_bayes_name_gender_classifier.ipynb

{
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.0"
    },
    "colab": {
      "name": "IT496_ViktoriiaO_naive-bayes-name-gender-classifier.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat_minor": 0,
  "nbformat": 4,
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/vovsiienko/IT496_Course_acitvities/blob/master/IT496_ViktoriiaO_naive_bayes_name_gender_classifier.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Source: https://www.kaggle.com/code/springboardroger/naive-bayes-name-gender-classifier\n",
        "\n",
        "This notebook takes the gender names dataset from Kaggle (sourced from Social Security records) and uses it as training data for a bigram Naive Bayes classifier that probabilistically classifies lists of names into a likely split of the genders within by determining patterns within gendered names. "
      ],
      "metadata": {
        "_cell_guid": "a7d5b056-89be-ce41-4897-d173a49ffe0d",
        "id": "jmWULdQVkNzM"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here, we import all of the needed libraries. "
      ],
      "metadata": {
        "_cell_guid": "3f11424d-6839-9cf9-3795-dd29ca9aa624",
        "id": "dxhijNYUkNzW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "import requests\n",
        "from bs4 import BeautifulSoup\n",
        "\n",
        "import matplotlib as mlp\n",
        "\n",
        "mlp.use('TKAgg')\n",
        "#%matplotlib tk\n",
        "#%matplotlib inline\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "from sklearn.feature_extraction.text import CountVectorizer\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import f1_score\n",
        "mlp.rcParams.update({'font.family': \"Open Sans\", 'font.size' : 16})"
      ],
      "metadata": {
        "_cell_guid": "b1905623-3283-5cc8-3757-2239d45d0863",
        "id": "t85DThRVkNzX"
      },
      "execution_count": 52,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we import the data file (the Social Security database) and do some basic data cleaning -- so we can display the first five entries and get a sense of the data."
      ],
      "metadata": {
        "_cell_guid": "37090842-1784-d201-8bf1-752b66451b1e",
        "id": "wPVJPEuAkNzc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#import social security names database from Kaggle in int form\n",
        "names = pd.read_csv(\"/content/drive/MyDrive/IT496/NationalNames.csv\", dtype = {'Count': np.int32})\n",
        "names = names.fillna(0)\n",
        "names.head()"
      ],
      "metadata": {
        "_cell_guid": "db8f040f-e46e-597d-3d94-62db18410132",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "AEYjXI_IkNzd",
        "outputId": "53a4bbf7-45ee-44ab-d70c-00f9e34b6584"
      },
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   Id       Name  Year Gender  Count\n",
              "0   1       Mary  1880      F   7065\n",
              "1   2       Anna  1880      F   2604\n",
              "2   3       Emma  1880      F   2003\n",
              "3   4  Elizabeth  1880      F   1939\n",
              "4   5     Minnie  1880      F   1746"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-b8edce7c-84ce-4d75-8220-97913b0e25b4\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Id</th>\n",
              "      <th>Name</th>\n",
              "      <th>Year</th>\n",
              "      <th>Gender</th>\n",
              "      <th>Count</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>Mary</td>\n",
              "      <td>1880</td>\n",
              "      <td>F</td>\n",
              "      <td>7065</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>Anna</td>\n",
              "      <td>1880</td>\n",
              "      <td>F</td>\n",
              "      <td>2604</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>Emma</td>\n",
              "      <td>1880</td>\n",
              "      <td>F</td>\n",
              "      <td>2003</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>Elizabeth</td>\n",
              "      <td>1880</td>\n",
              "      <td>F</td>\n",
              "      <td>1939</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>Minnie</td>\n",
              "      <td>1880</td>\n",
              "      <td>F</td>\n",
              "      <td>1746</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b8edce7c-84ce-4d75-8220-97913b0e25b4')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-b8edce7c-84ce-4d75-8220-97913b0e25b4 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-b8edce7c-84ce-4d75-8220-97913b0e25b4');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 53
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's groupby so we can get away from counting sums across years, and get the aggregated count of female and male occurrences for each name."
      ],
      "metadata": {
        "_cell_guid": "2cd9e928-cbdf-504d-93a2-a5a115d7fc0f",
        "id": "XRbONNrVkNze"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "namechart = names.groupby(['Name', 'Gender'], as_index = False)['Count'].sum()\n",
        "namechart.head(5)"
      ],
      "metadata": {
        "_cell_guid": "de808c7e-c122-fd5d-b61f-451e6d3322e0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "8HqoDfankNzf",
        "outputId": "35b3abe4-ff8d-418d-e4d4-6161f7876f38"
      },
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "        Name Gender  Count\n",
              "0      Aaban      M     72\n",
              "1      Aabha      F     21\n",
              "2      Aabid      M      5\n",
              "3  Aabriella      F     10\n",
              "4      Aadam      M    196"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-a01a2410-ba96-49aa-81ae-eed0798cc450\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Name</th>\n",
              "      <th>Gender</th>\n",
              "      <th>Count</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Aaban</td>\n",
              "      <td>M</td>\n",
              "      <td>72</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Aabha</td>\n",
              "      <td>F</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Aabid</td>\n",
              "      <td>M</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Aabriella</td>\n",
              "      <td>F</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Aadam</td>\n",
              "      <td>M</td>\n",
              "      <td>196</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a01a2410-ba96-49aa-81ae-eed0798cc450')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-a01a2410-ba96-49aa-81ae-eed0798cc450 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-a01a2410-ba96-49aa-81ae-eed0798cc450');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's add a column to categorize different names into a male or female bucket based on whether or not the frequency of males for a name outnumbers the frequency of females. "
      ],
      "metadata": {
        "_cell_guid": "74565e45-b9bd-ac6c-bdf1-20076849bc01",
        "id": "O4xs4vRTkNzh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "namechartdiff = namechart.reset_index().pivot('Name', 'Gender', 'Count')\n",
        "namechartdiff = namechartdiff.fillna(0)\n",
        "namechartdiff[\"Mpercent\"] = ((namechartdiff[\"M\"] - namechartdiff[\"F\"])/(namechartdiff[\"M\"] + namechartdiff[\"F\"]))\n",
        "namechartdiff['gender'] = np.where(namechartdiff['Mpercent'] > 0.001, 'male', 'female')\n",
        "namechartdiff.head()"
      ],
      "metadata": {
        "_cell_guid": "fc6a4039-0f20-2367-bb08-031c27bcad22",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 238
        },
        "id": "mXFn0TsDkNzi",
        "outputId": "26062f22-50fd-4f10-9a59-2789455d91ab"
      },
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Gender        F      M  Mpercent  gender\n",
              "Name                                    \n",
              "Aaban       0.0   72.0       1.0    male\n",
              "Aabha      21.0    0.0      -1.0  female\n",
              "Aabid       0.0    5.0       1.0    male\n",
              "Aabriella  10.0    0.0      -1.0  female\n",
              "Aadam       0.0  196.0       1.0    male"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-7f1f8cd2-c990-4138-841f-77f49399f2c9\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>Gender</th>\n",
              "      <th>F</th>\n",
              "      <th>M</th>\n",
              "      <th>Mpercent</th>\n",
              "      <th>gender</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Name</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Aaban</th>\n",
              "      <td>0.0</td>\n",
              "      <td>72.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>male</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Aabha</th>\n",
              "      <td>21.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>-1.0</td>\n",
              "      <td>female</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Aabid</th>\n",
              "      <td>0.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>male</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Aabriella</th>\n",
              "      <td>10.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>-1.0</td>\n",
              "      <td>female</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Aadam</th>\n",
              "      <td>0.0</td>\n",
              "      <td>196.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>male</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7f1f8cd2-c990-4138-841f-77f49399f2c9')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-7f1f8cd2-c990-4138-841f-77f49399f2c9 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-7f1f8cd2-c990-4138-841f-77f49399f2c9');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 55
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's now break down 'the strings of names into bigram blocks of characters with CountVectorizer."
      ],
      "metadata": {
        "_cell_guid": "ed8f3898-bed6-060e-dab2-334ff6c65a6e",
        "id": "nnsxlxoBkNzk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "char_vectorizer = CountVectorizer(analyzer='char', ngram_range=(2, 2))\n",
        "X = char_vectorizer.fit_transform(namechartdiff.index)\n",
        "X = X.tocsc()\n",
        "y = (namechartdiff.gender == 'male').values.astype(np.int)\n",
        "print(X)"
      ],
      "metadata": {
        "_cell_guid": "62c8b415-1f28-b177-9ada-6b9a1bba7431",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "p-AxzO-qkNzm",
        "outputId": "3b397e2b-89e5-4d13-9a91-0ef191898ef1"
      },
      "execution_count": 56,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  (0, 0)\t1\n",
            "  (1, 0)\t1\n",
            "  (2, 0)\t1\n",
            "  (3, 0)\t1\n",
            "  (4, 0)\t1\n",
            "  (5, 0)\t1\n",
            "  (6, 0)\t1\n",
            "  (7, 0)\t1\n",
            "  (8, 0)\t1\n",
            "  (9, 0)\t1\n",
            "  (10, 0)\t1\n",
            "  (11, 0)\t1\n",
            "  (12, 0)\t1\n",
            "  (13, 0)\t1\n",
            "  (14, 0)\t1\n",
            "  (15, 0)\t1\n",
            "  (16, 0)\t1\n",
            "  (17, 0)\t1\n",
            "  (18, 0)\t1\n",
            "  (19, 0)\t1\n",
            "  (20, 0)\t1\n",
            "  (21, 0)\t1\n",
            "  (22, 0)\t1\n",
            "  (23, 0)\t1\n",
            "  (24, 0)\t1\n",
            "  :\t:\n",
            "  (64876, 616)\t1\n",
            "  (67519, 616)\t1\n",
            "  (67520, 616)\t1\n",
            "  (67521, 616)\t1\n",
            "  (72287, 616)\t1\n",
            "  (73357, 616)\t1\n",
            "  (73358, 616)\t1\n",
            "  (76118, 616)\t1\n",
            "  (81252, 616)\t1\n",
            "  (81253, 616)\t1\n",
            "  (81254, 616)\t1\n",
            "  (81255, 616)\t1\n",
            "  (81256, 616)\t1\n",
            "  (83577, 616)\t1\n",
            "  (88001, 616)\t1\n",
            "  (88002, 616)\t1\n",
            "  (88270, 616)\t1\n",
            "  (91145, 616)\t1\n",
            "  (91333, 616)\t1\n",
            "  (91334, 616)\t1\n",
            "  (91335, 616)\t1\n",
            "  (91336, 616)\t1\n",
            "  (91604, 616)\t1\n",
            "  (92000, 616)\t1\n",
            "  (93888, 616)\t1\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:4: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.\n",
            "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
            "  after removing the cwd from sys.path.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's split our training and test data now. "
      ],
      "metadata": {
        "_cell_guid": "164e4c22-6a07-95ed-b0da-4177bf72fe51",
        "id": "NxEaWqT5kNzn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "itrain, itest = train_test_split(range(namechartdiff.shape[0]), train_size=0.7)\n",
        "mask=np.ones(namechartdiff.shape[0], dtype='int')\n",
        "mask[itrain]=1\n",
        "mask[itest]=0\n",
        "mask = (mask==1)"
      ],
      "metadata": {
        "_cell_guid": "0cb381da-a5a3-bf1d-17d2-fa2d69529f00",
        "id": "NiGonzZYkNzo"
      },
      "execution_count": 57,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we train the model."
      ],
      "metadata": {
        "_cell_guid": "716d9611-3afa-0cae-a752-d3b2fd942c24",
        "id": "ck83qRNJkNzp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "Xtrainthis=X[mask]\n",
        "Ytrainthis=y[mask]\n",
        "Xtestthis=X[~mask]\n",
        "Ytestthis=y[~mask]\n",
        "clf = MultinomialNB(alpha = 1)\n",
        "clf.fit(Xtrainthis, Ytrainthis)\n",
        "training_accuracy = clf.score(Xtrainthis,Ytrainthis)\n",
        "test_accuracy = clf.score(Xtestthis,Ytestthis)\n",
        "        \n",
        "print(training_accuracy)\n",
        "print(test_accuracy)"
      ],
      "metadata": {
        "_cell_guid": "5c16bf2e-30b1-e45b-9e36-bbbacd5e6b6b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1sLDD4LPkNzq",
        "outputId": "fc2b8f6e-3306-4593-a1fd-a21c158c2aa3"
      },
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.741152125620036\n",
            "0.7380977739908403\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now let's define a function that will allow us to easily look up and predict individual names. "
      ],
      "metadata": {
        "_cell_guid": "0de0a946-fc03-c77c-a446-5b4a9ba139c2",
        "id": "PQg33gYEkNzr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def lookup(x):\n",
        "    str(x)\n",
        "    new = char_vectorizer.transform([x])\n",
        "    y_pred = clf.predict(new)\n",
        "    if (y_pred == 1):\n",
        "        print(\"This is most likely a male name!\")\n",
        "    else:\n",
        "        print(\"This is most likely a female name!\")\n",
        "    "
      ],
      "metadata": {
        "_cell_guid": "a346a458-411f-9da9-c472-25bf37ff4970",
        "id": "pa6VttW4kNzr"
      },
      "execution_count": 59,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's lookup a name and determine whether it's male or female"
      ],
      "metadata": {
        "_cell_guid": "42a189f7-302f-7302-f442-e20c37324bf7",
        "id": "mGX906RSkNzs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "lookup(\"Aadam\")"
      ],
      "metadata": {
        "_cell_guid": "1f16a771-4c51-0d2d-c65a-703b0a220edb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2itIj7_xkNzs",
        "outputId": "e0decb2c-00ce-4813-cf1b-a40cd48e5bcb"
      },
      "execution_count": 60,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "This is most likely a male name!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "lookup(\"Mary\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AXLYr1FTsU9N",
        "outputId": "7a063623-1f7a-4b62-9865-050ff8e9b7a2"
      },
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "This is most likely a female name!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "lookup(\"John\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_CXIUlTI2ZLo",
        "outputId": "a7f985c6-e5b0-40bc-8735-67a16f37fc4e"
      },
      "execution_count": 64,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "This is most likely a male name!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "lookup(\"James\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RgfEcSJI2gc5",
        "outputId": "58c1d983-0de8-4fdb-99c5-cb10d85789a5"
      },
      "execution_count": 65,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "This is most likely a male name!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "lookup(\"Emma\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k7hcVgMH2l4b",
        "outputId": "9f53cf69-822a-43db-9b6d-888ac00ea93a"
      },
      "execution_count": 66,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "This is most likely a male name!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "lookup(\"Anna\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kOBkQ-Sp22vj",
        "outputId": "f48ac48b-c42f-492f-d7f7-5f43a8434d3f"
      },
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "This is most likely a female name!\n"
          ]
        }
      ]
    }
  ]
}