Notebooks/3_w2v.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "w2v.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "fRqjeaLlmZHe",
        "_7J7_WOz2VK8"
      ],
      "toc_visible": true,
      "machine_shape": "hm"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "source": [
        "## Word2Vec Modeling"
      ],
      "cell_type": "markdown",
      "metadata": {}
    },
    {
      "source": [
        "This notebook follows on the [non-neural network notebook](https://github.com/kcalizadeh/phil_nlp/blob/master/Notebooks/2_non-neural_models.ipynb). Here, however, we leave aside classificatory modeling and instead aim to understand how the different schools use their words. We achieve this by using word2vec. Initial results trained on the school in isolation were not very promising. But after importing the GloVe pre-trained vectors, we were able to get a compelling picture of how each school uses key terms. "
      ],
      "cell_type": "markdown",
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RrxG_6U1lGEa"
      },
      "source": [
        "### Imports and Mounting Drive"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WRT6BjwbiM8n",
        "outputId": "35987b02-de21-4822-8e28-dff14aaa70a7"
      },
      "source": [
        "# this cell mounts drive, sets the correct directory, then imports all functions\n",
        "# and relevant libraries via the functions.py file\n",
        "from google.colab import drive\n",
        "import sys\n",
        "\n",
        "# install relevent libraries not included with colab\n",
        "!pip install lime\n",
        "\n",
        "drive.mount('/gdrive',force_remount=True)\n",
        "\n",
        "drive_path = '/gdrive/MyDrive/Colab_Projects/Phil_NLP'\n",
        "\n",
        "sys.path.append(drive_path)"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting lime\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/f5/86/91a13127d83d793ecb50eb75e716f76e6eda809b6803c5a4ff462339789e/lime-0.2.0.1.tar.gz (275kB)\n",
            "\r\u001b[K     |█▏                              | 10kB 18.1MB/s eta 0:00:01\r\u001b[K     |██▍                             | 20kB 24.3MB/s eta 0:00:01\r\u001b[K     |███▋                            | 30kB 29.0MB/s eta 0:00:01\r\u001b[K     |████▊                           | 40kB 21.8MB/s eta 0:00:01\r\u001b[K     |██████                          | 51kB 19.0MB/s eta 0:00:01\r\u001b[K     |███████▏                        | 61kB 16.8MB/s eta 0:00:01\r\u001b[K     |████████▎                       | 71kB 13.3MB/s eta 0:00:01\r\u001b[K     |█████████▌                      | 81kB 14.1MB/s eta 0:00:01\r\u001b[K     |██████████▊                     | 92kB 13.9MB/s eta 0:00:01\r\u001b[K     |███████████▉                    | 102kB 13.7MB/s eta 0:00:01\r\u001b[K     |█████████████                   | 112kB 13.7MB/s eta 0:00:01\r\u001b[K     |██████████████▎                 | 122kB 13.7MB/s eta 0:00:01\r\u001b[K     |███████████████▌                | 133kB 13.7MB/s eta 0:00:01\r\u001b[K     |████████████████▋               | 143kB 13.7MB/s eta 0:00:01\r\u001b[K     |█████████████████▉              | 153kB 13.7MB/s eta 0:00:01\r\u001b[K     |███████████████████             | 163kB 13.7MB/s eta 0:00:01\r\u001b[K     |████████████████████▏           | 174kB 13.7MB/s eta 0:00:01\r\u001b[K     |█████████████████████▍          | 184kB 13.7MB/s eta 0:00:01\r\u001b[K     |██████████████████████▋         | 194kB 13.7MB/s eta 0:00:01\r\u001b[K     |███████████████████████▊        | 204kB 13.7MB/s eta 0:00:01\r\u001b[K     |█████████████████████████       | 215kB 13.7MB/s eta 0:00:01\r\u001b[K     |██████████████████████████▏     | 225kB 13.7MB/s eta 0:00:01\r\u001b[K     |███████████████████████████▍    | 235kB 13.7MB/s eta 0:00:01\r\u001b[K     |████████████████████████████▌   | 245kB 13.7MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▊  | 256kB 13.7MB/s eta 0:00:01\r\u001b[K     |███████████████████████████████ | 266kB 13.7MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 276kB 13.7MB/s \n",
            "\u001b[?25hRequirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from lime) (3.2.2)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from lime) (1.19.5)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from lime) (1.4.1)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from lime) (4.41.1)\n",
            "Requirement already satisfied: scikit-learn>=0.18 in /usr/local/lib/python3.6/dist-packages (from lime) (0.22.2.post1)\n",
            "Requirement already satisfied: scikit-image>=0.12 in /usr/local/lib/python3.6/dist-packages (from lime) (0.16.2)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->lime) (2.8.1)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->lime) (1.3.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->lime) (0.10.0)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->lime) (2.4.7)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.18->lime) (1.0.0)\n",
            "Requirement already satisfied: PyWavelets>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image>=0.12->lime) (1.1.1)\n",
            "Requirement already satisfied: pillow>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image>=0.12->lime) (7.0.0)\n",
            "Requirement already satisfied: imageio>=2.3.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image>=0.12->lime) (2.4.1)\n",
            "Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image>=0.12->lime) (2.5)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.1->matplotlib->lime) (1.15.0)\n",
            "Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx>=2.0->scikit-image>=0.12->lime) (4.4.2)\n",
            "Building wheels for collected packages: lime\n",
            "  Building wheel for lime (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for lime: filename=lime-0.2.0.1-cp36-none-any.whl size=283846 sha256=290015c2bd2d8aa5c9a9bdfde358b2ddc2ba83efd753ad878000ab8dd381560d\n",
            "  Stored in directory: /root/.cache/pip/wheels/4c/4f/a5/0bc765457bd41378bf3ce8d17d7495369d6e7ca3b712c60c89\n",
            "Successfully built lime\n",
            "Installing collected packages: lime\n",
            "Successfully installed lime-0.2.0.1\n",
            "Mounted at /gdrive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "E_FSrsN8l0J4",
        "outputId": "068df069-ebc7-454e-f0bb-e0ee5933790f"
      },
      "source": [
        "from functions import *\n",
        "%load_ext autoreload\n",
        "%autoreload 2\n",
        "\n",
        "np.random_seed=17"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/externals/six.py:31: FutureWarning:\n",
            "\n",
            "The module is deprecated in version 0.21 and will be removed in version 0.23 since we've dropped support for Python 2.7. Please rely on the official version of six (https://pypi.org/project/six/).\n",
            "\n",
            "/usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:144: FutureWarning:\n",
            "\n",
            "The sklearn.neighbors.base module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.neighbors. Anything that cannot be imported from sklearn.neighbors is now part of the private API.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fRqjeaLlmZHe"
      },
      "source": [
        "### Load the Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 364
        },
        "id": "PHxy7346l3hZ",
        "outputId": "4d21f3de-a4b2-4524-a2de-a3fe39531d4c"
      },
      "source": [
        "df = pd.read_csv('/gdrive/MyDrive/Colab_Projects/Phil_NLP/phil_nlp.csv')\n",
        "\n",
        "df.sample(5)"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>title</th>\n",
              "      <th>author</th>\n",
              "      <th>school</th>\n",
              "      <th>sentence_spacy</th>\n",
              "      <th>sentence_str</th>\n",
              "      <th>sentence_length</th>\n",
              "      <th>sentence_lowered</th>\n",
              "      <th>tokenized_txt</th>\n",
              "      <th>lemmatized_str</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>34917</th>\n",
              "      <td>Plato - Complete Works</td>\n",
              "      <td>Plato</td>\n",
              "      <td>plato</td>\n",
              "      <td>However, there remains the view that they can ...</td>\n",
              "      <td>However, there remains the view that they can ...</td>\n",
              "      <td>84</td>\n",
              "      <td>however, there remains the view that they can ...</td>\n",
              "      <td>['however', 'there', 'remains', 'the', 'view',...</td>\n",
              "      <td>however , there remain the view that -PRON- c...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>245359</th>\n",
              "      <td>Off The Beaten Track</td>\n",
              "      <td>Heidegger</td>\n",
              "      <td>phenomenology</td>\n",
              "      <td>lated iiber gesetzt us?</td>\n",
              "      <td>lated iiber gesetzt us?</td>\n",
              "      <td>23</td>\n",
              "      <td>lated iiber gesetzt us?</td>\n",
              "      <td>['lated', 'iiber', 'gesetzt', 'us']</td>\n",
              "      <td>lated iiber gesetzt -PRON- ?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>278996</th>\n",
              "      <td>The Phenomenology Of Spirit</td>\n",
              "      <td>Hegel</td>\n",
              "      <td>german_idealism</td>\n",
              "      <td>I perceive in them the free. .)</td>\n",
              "      <td>I perceive in them the free. .)</td>\n",
              "      <td>31</td>\n",
              "      <td>i perceive in them the free. .)</td>\n",
              "      <td>['perceive', 'in', 'them', 'the', 'free']</td>\n",
              "      <td>-PRON- perceive in -PRON- the free . . )</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>213589</th>\n",
              "      <td>Anti-Oedipus</td>\n",
              "      <td>Deleuze</td>\n",
              "      <td>continental</td>\n",
              "      <td>It is the thing, unnamable, the generalized de...</td>\n",
              "      <td>It is the thing, unnamable, the generalized de...</td>\n",
              "      <td>207</td>\n",
              "      <td>it is the thing, unnamable, the generalized de...</td>\n",
              "      <td>['it', 'is', 'the', 'thing', 'unnamable', 'the...</td>\n",
              "      <td>-PRON- be the thing , unnamable , the general...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>196800</th>\n",
              "      <td>The Order Of Things</td>\n",
              "      <td>Foucault</td>\n",
              "      <td>continental</td>\n",
              "      <td>But this equality does not mean that one excha...</td>\n",
              "      <td>But this equality does not mean that one excha...</td>\n",
              "      <td>277</td>\n",
              "      <td>but this equality does not mean that one excha...</td>\n",
              "      <td>['but', 'this', 'equality', 'does', 'not', 'me...</td>\n",
              "      <td>but this equality do not mean that one exchan...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                              title  ...                                     lemmatized_str\n",
              "34917        Plato - Complete Works  ...   however , there remain the view that -PRON- c...\n",
              "245359         Off The Beaten Track  ...                       lated iiber gesetzt -PRON- ?\n",
              "278996  The Phenomenology Of Spirit  ...           -PRON- perceive in -PRON- the free . . )\n",
              "213589                 Anti-Oedipus  ...   -PRON- be the thing , unnamable , the general...\n",
              "196800          The Order Of Things  ...   but this equality do not mean that one exchan...\n",
              "\n",
              "[5 rows x 9 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UR-d0hRI7hmQ"
      },
      "source": [
        "# using gensim's built-in tokenizer \r\n",
        "df['gensim_tokenized'] = df['sentence_str'].map(lambda x: simple_preprocess(x.lower(),deacc=True,\r\n",
        "                                                        max_len=100))"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "z-mTBCSDojS1",
        "outputId": "37b1790d-7fb1-449b-d4e1-bfb4d2259f90"
      },
      "source": [
        "# check how it worked\r\n",
        "print(df.iloc[290646]['sentence_str'])\r\n",
        "df['gensim_tokenized'][290646]"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "A spider conducts operations that resemble those of a weaver, and a bee puts to shame many an architect in the construction of her cells.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['spider',\n",
              " 'conducts',\n",
              " 'operations',\n",
              " 'that',\n",
              " 'resemble',\n",
              " 'those',\n",
              " 'of',\n",
              " 'weaver',\n",
              " 'and',\n",
              " 'bee',\n",
              " 'puts',\n",
              " 'to',\n",
              " 'shame',\n",
              " 'many',\n",
              " 'an',\n",
              " 'architect',\n",
              " 'in',\n",
              " 'the',\n",
              " 'construction',\n",
              " 'of',\n",
              " 'her',\n",
              " 'cells']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_KxV2JTIoyxE"
      },
      "source": [
        "Hmm, an interesting observation. \r\n",
        "\r\n",
        "For our w2v models, first we'll focus on a single school, since a single school is more likely to have consistency in their use of a word.\r\n",
        "\r\n",
        "Unfortunately, we didn't have much luck with just training on the texts alone. The code for it is left here for posterity, but it was when we worked with GloVe as the base that we had results that were actually useful."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Th5FA84Ip-ha"
      },
      "source": [
        "### Word 2 Vec Training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7cv5UgQEqEGK"
      },
      "source": [
        "#### German Idealism as a Test Case"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hFuFkbN7qN2w"
      },
      "source": [
        "We start by examining the texts of German Idealism to get a feel for what kind of parameters would work best."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "78GJHWsMF2he"
      },
      "source": [
        "def make_w2v(series, stopwords=[], size=200, window=5, min_count=5, workers=-1, \r\n",
        "             epochs=20, lowercase=True, sg=0, seed=17, cbow_mean=1, alpha=0.025,\r\n",
        "             sample=0.001, use_bigrams=True, threshold=10, bigram_min=5):\r\n",
        "  # turn the series into a list, lower it, clean it\r\n",
        "    sentences = [sentence for sentence in series]\r\n",
        "    if lowercase:\r\n",
        "      cleaned = []\r\n",
        "      for sentence in sentences:\r\n",
        "        cleaned_sentence = [word.lower() for word in sentence]\r\n",
        "        cleaned_sentence = [word for word in sentence if word not in stopwords]\r\n",
        "        cleaned.append(cleaned_sentence)\r\n",
        "    else:\r\n",
        "      cleaned = []\r\n",
        "      for sentence in sentences:\r\n",
        "        cleaned_sentence = [word for word in sentence]\r\n",
        "        cleaned_sentence = [word for word in sentence if word not in stopwords]\r\n",
        "        cleaned.append(cleaned_sentence)\r\n",
        "\r\n",
        "  # incorporate bigrams\r\n",
        "    if use_bigrams:\r\n",
        "      bigram = Phrases(cleaned, min_count=bigram_min, threshold=threshold, delimiter=b' ')\r\n",
        "      bigram_phraser = Phraser(bigram)\r\n",
        "      tokens_list = []\r\n",
        "      for sent in cleaned:\r\n",
        "        tokens_ = bigram_phraser[sent]\r\n",
        "        tokens_list.append(tokens_)\r\n",
        "      cleaned = tokens_list\r\n",
        "    else:\r\n",
        "      cleaned = cleaned\r\n",
        "\r\n",
        "  # build the model\r\n",
        "    model = Word2Vec(cleaned, size=size, window=window, \r\n",
        "                     min_count=min_count, workers=workers, seed=seed, sg=sg,\r\n",
        "                     cbow_mean=cbow_mean, alpha=alpha, sample=sample)\r\n",
        "    model.train(series, total_examples=model.corpus_count, epochs=epochs)\r\n",
        "    model_wv = model.wv\r\n",
        "    \r\n",
        "  # clear it to avoid unwanted transference\r\n",
        "    del model\r\n",
        "\r\n",
        "    return model_wv"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ONQLm1mhOO6w"
      },
      "source": [
        "gi_wv = make_w2v(df[df['school'] == 'german_idealism']['gensim_tokenized'], threshold=12)"
      ],
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lUf7kAzvrh7K"
      },
      "source": [
        "We can check this model by trying out a few words. For that purpose we have a testing function that tries some common word combinations."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LMJat1b1rhwK"
      },
      "source": [
        "pairs_to_try = [(['law', 'moral'], []),\r\n",
        "                (['self', 'consciousness'], []),\r\n",
        "                (['dialectic'], []),\r\n",
        "                (['logic'], []),\r\n",
        "]"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tXOnjOLpOcxi",
        "outputId": "23f3ccf4-810c-4cf4-f9a1-7d5b912fbc30"
      },
      "source": [
        "test_w2v_pos_neg(gi_wv, pairs_to_try)"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Positive - ['law', 'moral']\tNegative - []\n",
            "- joins (0.31005)\n",
            "- tis (0.24875)\n",
            "- immediacy (0.23487)\n",
            "- feeling (0.22275)\n",
            "- disappeared (0.22237)\n",
            "\n",
            "Positive - ['self', 'consciousness']\tNegative - []\n",
            "- manifests itself (0.26899)\n",
            "- deepest (0.23347)\n",
            "- submitted (0.23027)\n",
            "- positivity (0.22937)\n",
            "- defend (0.22893)\n",
            "\n",
            "Positive - ['dialectic']\tNegative - []\n",
            "- calls (0.26034)\n",
            "- widespread (0.2585)\n",
            "- steel spring (0.25831)\n",
            "- ostensive (0.24845)\n",
            "- deliberations (0.22695)\n",
            "\n",
            "Positive - ['logic']\tNegative - []\n",
            "- frdm (0.25433)\n",
            "- physician (0.25169)\n",
            "- sunders (0.25144)\n",
            "- therein (0.24754)\n",
            "- ease (0.23451)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jgu0ee-frhZQ"
      },
      "source": [
        "Although some of these make a modicum of sense a lot of them seem like just gibberish. Let's try messing with some parameters.\r\n",
        "\r\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eS8zfzZHrhNe"
      },
      "source": [
        "##### Trying Skip-gram instead of C-bow"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qiV-E9xzrhCL"
      },
      "source": [
        "# make a base model with the preset parameters\r\n",
        "skip_gi_wv = make_w2v(series = df[df['school'] == 'german_idealism']['gensim_tokenized'], \r\n",
        "                         stopwords=[], sg=1, seed=0)"
      ],
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eS_Jzbgirg1y",
        "outputId": "cd3ac11c-b95a-4738-cd9b-d70133b6a31a"
      },
      "source": [
        "test_w2v(skip_gi_wv, pairs_to_try)"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Positive - ['law', 'moral']\tNegative - []\n",
            "- ive (0.30441)\n",
            "- obscure (0.27291)\n",
            "- manifestation (0.26622)\n",
            "- import (0.244)\n",
            "- ostensive (0.23673)\n",
            "\n",
            "Positive - ['self', 'consciousness']\tNegative - []\n",
            "- drop (0.29426)\n",
            "- definition (0.27958)\n",
            "- ratiocination (0.2355)\n",
            "- choice (0.23224)\n",
            "- pleasure (0.22293)\n",
            "\n",
            "Positive - ['dialectic']\tNegative - []\n",
            "- observe (0.2774)\n",
            "- abroad (0.26245)\n",
            "- paradoxical (0.251)\n",
            "- lordship (0.24905)\n",
            "- moral feeling (0.24531)\n",
            "\n",
            "Positive - ['logic']\tNegative - []\n",
            "- factor (0.28102)\n",
            "- renounced (0.26257)\n",
            "- breazeale (0.23544)\n",
            "- rerum (0.23437)\n",
            "- nervous system (0.23188)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XnNnqhueyaK-"
      },
      "source": [
        "These seem mildy more sensible. Let's tweak the other parameters."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lAv1PjLD7ZSh"
      },
      "source": [
        "##### Parameter Testing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y6Y3AggQyaVD",
        "outputId": "8bf20ab5-ce0d-43af-9dce-13b204ec0ed1"
      },
      "source": [
        "model_v1 = make_w2v(df[df['school'] == 'german_idealism']['gensim_tokenized'],\r\n",
        "                       stopwords=[],\r\n",
        "                       size=500,\r\n",
        "                       window=5,\r\n",
        "                       min_count=25,\r\n",
        "                       epochs=10,\r\n",
        "                       sg=1, \r\n",
        "                       seed=45)\r\n",
        "\r\n",
        "len(model_v1.vocab)"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2950"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SVVqIAf8yadZ",
        "outputId": "e5c1e456-f207-4a74-a54b-4a8570461301"
      },
      "source": [
        "test_w2v(model_v1, pairs_to_try)"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Positive - ['law', 'moral']\tNegative - []\n",
            "- rationality (0.15593)\n",
            "- avoid (0.15046)\n",
            "- association (0.14209)\n",
            "- accord (0.13843)\n",
            "- untrue (0.13657)\n",
            "\n",
            "Positive - ['self', 'consciousness']\tNegative - []\n",
            "- expand (0.14267)\n",
            "- merely (0.13895)\n",
            "- syllogisms (0.13827)\n",
            "- entity (0.13225)\n",
            "- positing (0.13193)\n",
            "\n",
            "Positive - ['dialectic']\tNegative - []\n",
            "- excludes (0.15812)\n",
            "- completed (0.14345)\n",
            "- important (0.1395)\n",
            "- too (0.13569)\n",
            "- primarily (0.13489)\n",
            "\n",
            "Positive - ['logic']\tNegative - []\n",
            "- sure (0.16182)\n",
            "- feeling (0.13781)\n",
            "- preceding (0.13777)\n",
            "- intuitions (0.13683)\n",
            "- inherently (0.13238)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kn8zvu2z2U_o"
      },
      "source": [
        "Despite tweaking parameters far and wide, it's difficult to get any results that are compellingly sensible. In most cases there are one or two terms in the similarity list that make some sense but others that are just strange or unconnected"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_7J7_WOz2VK8"
      },
      "source": [
        "#### Trying Another School"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rZdUtK642VcE",
        "outputId": "f0c7c9a7-3a01-4e83-e57d-bdd72e9193a0"
      },
      "source": [
        "cm_w2v = make_w2v(df[df['school'] == 'communism']['gensim_tokenized'],\r\n",
        "                       stopwords=[],\r\n",
        "                       size=700,\r\n",
        "                       window=10,\r\n",
        "                       min_count=10,\r\n",
        "                       epochs=25,\r\n",
        "                       sg=1, \r\n",
        "                       seed=10)\r\n",
        "\r\n",
        "type(cm_w2v)"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "gensim.models.keyedvectors.Word2VecKeyedVectors"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FYoO0ozP2VmK"
      },
      "source": [
        "pairs_to_try=[(['material', 'conditions'], []),\r\n",
        "              (['worker'], ['owner']),\r\n",
        "              (['alienation', 'labor'], []),\r\n",
        "              (['capital'], [])]"
      ],
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1809IvVx2Vuy",
        "outputId": "66aaaac8-5348-4628-fb22-c551af11a86c"
      },
      "source": [
        "test_w2v(cm_w2v, pairs_to_try)"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Positive - ['material', 'conditions']\tNegative - []\n",
            "- constant (0.14083)\n",
            "- november (0.1261)\n",
            "- revolutions (0.12528)\n",
            "- italics (0.12458)\n",
            "- just (0.12193)\n",
            "\n",
            "Positive - ['worker']\tNegative - ['owner']\n",
            "- themselves (0.14149)\n",
            "- slow (0.1371)\n",
            "- rises (0.12916)\n",
            "- every day (0.12628)\n",
            "- streets (0.12155)\n",
            "\n",
            "Positive - ['alienation', 'labor']\tNegative - []\n",
            "- motive power (0.14868)\n",
            "- higher (0.13019)\n",
            "- keeps (0.12603)\n",
            "- certain extent (0.11582)\n",
            "- wood (0.11516)\n",
            "\n",
            "Positive - ['capital']\tNegative - []\n",
            "- set (0.13807)\n",
            "- des (0.13375)\n",
            "- be replaced (0.11977)\n",
            "- specifically (0.11826)\n",
            "- examined (0.1142)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6hUYWlYr2V3L"
      },
      "source": [
        "Here the results were similar - a few words that made some sense and plenty that were just odd.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ef5ZnODM2V_P"
      },
      "source": [
        "### Transfer Learning with GloVe"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3VAOaelq2WIr"
      },
      "source": [
        "We'll import GloVe vectors as w2v, then use those as a base from which to train new vectors that are tuned to our corpus."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UcFZMRT82WRC"
      },
      "source": [
        "# load the vectors. other vector sizes were used but yielded generally less sensible models\r\n",
        "glove_file = datapath('/gdrive/MyDrive/Colab_Projects/Phil_NLP/glove.6B.50d.txt')\r\n",
        "tmp_file = get_tmpfile(\"test_word2vec.txt\")\r\n",
        "\r\n",
        "_ = glove2word2vec(glove_file, tmp_file)\r\n",
        "\r\n",
        "glove_vectors = KeyedVectors.load_word2vec_format(tmp_file)"
      ],
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "12hPyZ2aHugg"
      },
      "source": [
        "pairs_to_try = [(['law', 'moral'], []),\r\n",
        "                (['self', 'consciousness'], []),\r\n",
        "                (['dialectic'], []),\r\n",
        "                (['logic'], []),\r\n",
        "]"
      ],
      "execution_count": 46,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NjdDWtd72Tli",
        "outputId": "98ba86ac-7f60-4ddd-c226-dac9f34907ef"
      },
      "source": [
        "# check out how GloVe works on our test pairs\r\n",
        "test_w2v_pos_neg(glove_vectors, pairs_to_try)"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Positive - ['law', 'moral']\tNegative - []\n",
            "- morality (0.82654)\n",
            "- legal (0.82652)\n",
            "- laws (0.81529)\n",
            "- constitutional (0.80616)\n",
            "- fundamental (0.80217)\n",
            "\n",
            "Positive - ['self', 'consciousness']\tNegative - []\n",
            "- sense (0.83446)\n",
            "- mind (0.79755)\n",
            "- vision (0.78202)\n",
            "- belief (0.78031)\n",
            "- life (0.77984)\n",
            "\n",
            "Positive - ['dialectic']\tNegative - []\n",
            "- hegelian (0.88376)\n",
            "- dialectical (0.83417)\n",
            "- dialectics (0.80672)\n",
            "- materialist (0.77674)\n",
            "- metaphysics (0.77488)\n",
            "\n",
            "Positive - ['logic']\tNegative - []\n",
            "- reasoning (0.81405)\n",
            "- intuitionistic (0.76531)\n",
            "- concepts (0.75831)\n",
            "- logical (0.75604)\n",
            "- theory (0.75026)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BhFyauZoo7KW"
      },
      "source": [
        "Ok these make a lot more sense right from the start. But we want them to be trained on our actual philosophical texts - that way we can see how different thinkers use different words and potentially use the vectors for classification.\r\n",
        "\r\n",
        "So in the cells below we train the existing GloVe model on on the German Idealist texts as a test."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pSOA8Zz2o7vX"
      },
      "source": [
        "# isolate the relevant school\r\n",
        "documents = df[df['school'] == 'german_idealism']['gensim_tokenized']\r\n",
        "\r\n",
        "# format the series to be used\r\n",
        "stopwords = []\r\n",
        "\r\n",
        "sentences = [sentence for sentence in documents]\r\n",
        "cleaned = []\r\n",
        "for sentence in sentences:\r\n",
        "  cleaned_sentence = [word.lower() for word in sentence]\r\n",
        "  cleaned_sentence = [word for word in sentence if word not in stopwords]\r\n",
        "  cleaned.append(cleaned_sentence)\r\n",
        "\r\n",
        "# get bigrams\r\n",
        "bigram = Phrases(cleaned, min_count=20, threshold=10, delimiter=b' ')\r\n",
        "bigram_phraser = Phraser(bigram)\r\n",
        "\r\n",
        "bigramed_tokens = []\r\n",
        "for sent in cleaned:\r\n",
        "    tokens = bigram_phraser[sent]\r\n",
        "    bigramed_tokens.append(tokens)\r\n",
        "\r\n",
        "# run again to get trigrams\r\n",
        "trigram = Phrases(bigramed_tokens, min_count=20, threshold=10, delimiter=b' ')\r\n",
        "trigram_phraser = Phraser(trigram)\r\n",
        "\r\n",
        "trigramed_tokens = []\r\n",
        "for sent in bigramed_tokens:\r\n",
        "    tokens = trigram_phraser[sent]\r\n",
        "    trigramed_tokens.append(tokens)\r\n",
        "\r\n",
        "# build a toy model to update with\r\n",
        "base_model = Word2Vec(size=300, min_count=5)\r\n",
        "base_model.build_vocab(trigramed_tokens)\r\n",
        "total_examples = base_model.corpus_count\r\n",
        "\r\n",
        "# add GloVe's vocabulary & weights\r\n",
        "base_model.build_vocab([list(glove_vectors.vocab.keys())], update=True)\r\n",
        "\r\n",
        "# train on our data\r\n",
        "base_model.train(trigramed_tokens, total_examples=total_examples, epochs=base_model.epochs)\r\n",
        "base_model_wv = base_model.wv\r\n",
        "del base_model"
      ],
      "execution_count": 38,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "g0FjXahncQya",
        "outputId": "69f321dd-f2c2-4cd6-d1ad-e0928e4ede1e"
      },
      "source": [
        "test_w2v(base_model_wv, pairs_to_try)"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Positive - ['law', 'moral']\tNegative - []\n",
            "- freedom (0.82611)\n",
            "- happiness (0.81553)\n",
            "- rule (0.80918)\n",
            "- moral law (0.80705)\n",
            "- morality (0.80598)\n",
            "\n",
            "Positive - ['self', 'consciousness']\tNegative - []\n",
            "- self consciousness (0.90358)\n",
            "- objectivity (0.86174)\n",
            "- immediacy (0.85362)\n",
            "- essence (0.85233)\n",
            "- negativity (0.85204)\n",
            "\n",
            "Positive - ['dialectic']\tNegative - []\n",
            "- antinomy (0.94207)\n",
            "- method (0.90006)\n",
            "- remainder (0.89822)\n",
            "- morals (0.8973)\n",
            "- critique (0.89044)\n",
            "\n",
            "Positive - ['logic']\tNegative - []\n",
            "- pure reason (0.87167)\n",
            "- metaphysics (0.86658)\n",
            "- method (0.86523)\n",
            "- philosophy (0.83807)\n",
            "- science (0.81908)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GR3MnHFbcl7d"
      },
      "source": [
        "We can immediately see that these make a lot more sense (and the similarity scores are a lot higher). 'Self' + 'consciousness' is rightly associated with 'self consciousness' and 'moral' + 'law' with 'moral law'. It even identifies the German Idealist tendency to unify logic and metaphysics. \r\n",
        "\r\n",
        "This is a massive improvement - these vectors can be fairly said to reflect how german idealists use these terms. Moreover, they are significantly different than the original GloVe model, which indicates that there was real learning going on here.\r\n",
        "\r\n",
        "For comparison, let's check these same terms, but as used by Phenomenologists."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0s1Jkkc7dqk9"
      },
      "source": [
        "def train_glove(school, glove_vectors, threshold=10, stopwords=[],\r\n",
        "                min_count=20):\r\n",
        "  # isolate the relevant school\r\n",
        "  documents = df[df['school'] ==school]['gensim_tokenized']\r\n",
        "\r\n",
        "  # format the series to be used\r\n",
        "  stopwords = []\r\n",
        "\r\n",
        "  sentences = [sentence for sentence in documents]\r\n",
        "  cleaned = []\r\n",
        "  for sentence in sentences:\r\n",
        "    cleaned_sentence = [word.lower() for word in sentence]\r\n",
        "    cleaned_sentence = [word for word in sentence if word not in stopwords]\r\n",
        "    cleaned.append(cleaned_sentence)\r\n",
        "\r\n",
        "  # get bigrams\r\n",
        "  bigram = Phrases(cleaned, min_count=min_count, threshold=threshold, \r\n",
        "                   delimiter=b' ')\r\n",
        "  bigram_phraser = Phraser(bigram)\r\n",
        "\r\n",
        "  bigramed_tokens = []\r\n",
        "  for sent in cleaned:\r\n",
        "      tokens = bigram_phraser[sent]\r\n",
        "      bigramed_tokens.append(tokens)\r\n",
        "\r\n",
        "  # run again to get trigrams\r\n",
        "  trigram = Phrases(bigramed_tokens, min_count=min_count, threshold=threshold, \r\n",
        "                    delimiter=b' ')\r\n",
        "  trigram_phraser = Phraser(trigram)\r\n",
        "\r\n",
        "  trigramed_tokens = []\r\n",
        "  for sent in bigramed_tokens:\r\n",
        "      tokens = trigram_phraser[sent]\r\n",
        "      trigramed_tokens.append(tokens)\r\n",
        "\r\n",
        "  # build a toy model to update with\r\n",
        "  model = Word2Vec(size=300, min_count=5)\r\n",
        "  model.build_vocab(trigramed_tokens)\r\n",
        "  total_examples = model.corpus_count\r\n",
        "\r\n",
        "  # add GloVe's vocabulary & weights\r\n",
        "  model.build_vocab([list(glove_vectors.vocab.keys())], update=True)\r\n",
        "\r\n",
        "  # train on our data\r\n",
        "  model.train(trigramed_tokens, total_examples=total_examples, epochs=model.epochs)\r\n",
        "  model_wv = model.wv\r\n",
        "  del model\r\n",
        "  return model_wv"
      ],
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nIMLCNVHdxID"
      },
      "source": [
        "ph_model = train_glove(school='phenomenology', glove_vectors=glove_vectors)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BI3_J1HGnxd-",
        "outputId": "412836cb-d853-4142-997f-037b6eec41ef"
      },
      "source": [
        "test_w2v(ph_model, pairs_to_try)"
      ],
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Positive - ['law', 'moral']\tNegative - []\n",
            "- distribution (0.99305)\n",
            "- alleged (0.99293)\n",
            "- algebraic (0.99275)\n",
            "- range (0.99253)\n",
            "- interconnection (0.99223)\n",
            "\n",
            "Positive - ['self', 'consciousness']\tNegative - []\n",
            "- nature (0.95029)\n",
            "- potentiality (0.94732)\n",
            "- certainty (0.93127)\n",
            "- authentic (0.92803)\n",
            "- existence (0.92441)\n",
            "\n",
            "Positive - ['dialectic']\tNegative - []\n",
            "- inversion (0.98864)\n",
            "- emergence (0.98773)\n",
            "- mouth (0.98767)\n",
            "- aid (0.98666)\n",
            "- succession (0.98643)\n",
            "\n",
            "Positive - ['logic']\tNegative - []\n",
            "- mathematics (0.98822)\n",
            "- gathering (0.98441)\n",
            "- primary (0.98374)\n",
            "- apprehension (0.98373)\n",
            "- disclosure (0.98349)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iRp6eNmHeP01"
      },
      "source": [
        "Using the phenomenology vectors on some central terms of German idealism once again yields some pretty compelling results, except for where the words are rarely used by the phenomenologists. This is to be expected. Let's try the word vectors on some central terms of phenomenology."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QWzc_fFGnXnF",
        "outputId": "d10ffc90-988d-4e88-846c-94d9445a931c"
      },
      "source": [
        "pairs_to_try = [(['perception'], []),\r\n",
        "                (['dasein'], []),\r\n",
        "                (['consciousness'], []),\r\n",
        "                (['method'], []),]\r\n",
        "\r\n",
        "test_w2v(ph_model, pairs_to_try)"
      ],
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Positive - ['perception']\tNegative - []\n",
            "- representation (0.9493)\n",
            "- movement (0.94042)\n",
            "- condition (0.93799)\n",
            "- act (0.93541)\n",
            "- attitude (0.922)\n",
            "\n",
            "Positive - ['dasein']\tNegative - []\n",
            "- being (0.89689)\n",
            "- itself (0.8852)\n",
            "- truth (0.87685)\n",
            "- consciousness (0.84428)\n",
            "- understanding (0.8317)\n",
            "\n",
            "Positive - ['consciousness']\tNegative - []\n",
            "- care (0.91553)\n",
            "- truth (0.91257)\n",
            "- future (0.90781)\n",
            "- nature (0.90509)\n",
            "- movement (0.90323)\n",
            "\n",
            "Positive - ['method']\tNegative - []\n",
            "- necessity (0.96889)\n",
            "- ontology (0.9638)\n",
            "- spirit (0.96361)\n",
            "- historicity (0.95475)\n",
            "- metaphysics (0.95457)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bZBBODD-neYi"
      },
      "source": [
        "These look pretty strong. Overall, the GloVe-traiend vectors seem to be an effective tool for revealing how a word is used by a school. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N5UPKzWjrWMQ"
      },
      "source": [
        "#### Training on every school & author"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BIREGokArZQK"
      },
      "source": [
        "To further explore this, we'll train w2v models in this way for each school and examine how each of them looks at the same word - 'philosophy.' We can use these in our future dashboard work."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "R8_XI24ogh17",
        "outputId": "1acac61a-fe05-4a69-dac0-eb82dc9dfde1"
      },
      "source": [
        "w2v_dict = {}\r\n",
        "\r\n",
        "for school in df['school'].unique():\r\n",
        "  w2v_dict[school] = train_glove(school, glove_vectors=glove_vectors)\r\n",
        "  print(f'{school} completed')"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "plato completed\n",
            "aristotle completed\n",
            "empiricism completed\n",
            "rationalism completed\n",
            "analytic completed\n",
            "continental completed\n",
            "phenomenology completed\n",
            "german_idealism completed\n",
            "communism completed\n",
            "capitalism completed\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "euZktiqighpT",
        "outputId": "a5b4ceee-734c-4fe8-b2ab-d3ec6b7a5f6b"
      },
      "source": [
        "for school in df['school'].unique():\r\n",
        "  print(f'\\t{school.upper()}')\r\n",
        "  print('----------------------')\r\n",
        "  test_w2v(w2v_dict[school], [(['philosophy'], [])])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\tPLATO\n",
            "----------------------\n",
            "Positive - ['philosophy']\tNegative - []\n",
            "- relief (0.94646)\n",
            "- springs (0.94496)\n",
            "- friendship (0.94237)\n",
            "- greece (0.93422)\n",
            "- regime (0.93311)\n",
            "\n",
            "\tARISTOTLE\n",
            "----------------------\n",
            "Positive - ['philosophy']\tNegative - []\n",
            "- respiration (0.91699)\n",
            "- oratory (0.87146)\n",
            "- shrillness (0.87085)\n",
            "- holders (0.86975)\n",
            "- memory (0.85944)\n",
            "\n",
            "\tEMPIRICISM\n",
            "----------------------\n",
            "Positive - ['philosophy']\tNegative - []\n",
            "- religion (0.93495)\n",
            "- mankind (0.92261)\n",
            "- doctrine (0.92259)\n",
            "- inquiry (0.9155)\n",
            "- faith (0.90897)\n",
            "\n",
            "\tRATIONALISM\n",
            "----------------------\n",
            "Positive - ['philosophy']\tNegative - []\n",
            "- person (0.95782)\n",
            "- return (0.92869)\n",
            "- death (0.92862)\n",
            "- fall (0.92596)\n",
            "- public (0.9258)\n",
            "\n",
            "\tANALYTIC\n",
            "----------------------\n",
            "Positive - ['philosophy']\tNegative - []\n",
            "- philosophical (0.90815)\n",
            "- hahn (0.85215)\n",
            "- carnap (0.84692)\n",
            "- semantics (0.84195)\n",
            "- davidson (0.83312)\n",
            "\n",
            "\tCONTINENTAL\n",
            "----------------------\n",
            "Positive - ['philosophy']\tNegative - []\n",
            "- metaphysics (0.9689)\n",
            "- unreason (0.95789)\n",
            "- history (0.95635)\n",
            "- consciousness (0.94364)\n",
            "- proposition (0.93498)\n",
            "\n",
            "\tPHENOMENOLOGY\n",
            "----------------------\n",
            "Positive - ['philosophy']\tNegative - []\n",
            "- method (0.9159)\n",
            "- metaphysics (0.91034)\n",
            "- spirit (0.90868)\n",
            "- phenomenology (0.89536)\n",
            "- science (0.89134)\n",
            "\n",
            "\tGERMAN_IDEALISM\n",
            "----------------------\n",
            "Positive - ['philosophy']\tNegative - []\n",
            "- science (0.87517)\n",
            "- metaphysics (0.86891)\n",
            "- method (0.81549)\n",
            "- pure reason (0.78349)\n",
            "- definitions (0.78128)\n",
            "\n",
            "\tCOMMUNISM\n",
            "----------------------\n",
            "Positive - ['philosophy']\tNegative - []\n",
            "- ministers (0.99902)\n",
            "- inches (0.999)\n",
            "- syndicates (0.99897)\n",
            "- divisions (0.99894)\n",
            "- along (0.99891)\n",
            "\n",
            "\tCAPITALISM\n",
            "----------------------\n",
            "Positive - ['philosophy']\tNegative - []\n",
            "- queen (0.99179)\n",
            "- tartar (0.98992)\n",
            "- greek (0.98812)\n",
            "- treaties (0.98806)\n",
            "- uniform (0.98774)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xmP9U0WrghgT"
      },
      "source": [
        "Interestingly, many of these top words align quite strongly with the school's general attitude towards philosophy. Continental thinkers mentioning unreason, analytic philosophers focusing on semantics, and phenomenologists associating philosophy with a method all track well. The ones that don't make sense are those that don't problematize the nature of philosophy to any great degree - capitalist thinkers aren't out there trying to discuss the nature of philosophy.\r\n",
        "\r\n",
        "We'd also like vectors trained for each individual author. We can use these in our dashboard to enable intra-school comparisons of authors and generally allow for more fine-grained data exploration."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "V6KZckIvrDv8"
      },
      "source": [
        "#@title Glove Training Function Modified for Authors\n",
        "def train_glove_author(school, glove_vectors, threshold=10, stopwords=[],\n",
        "                min_count=20):\n",
        "  # isolate the relevant school\n",
        "  documents = df[df['author'] ==school]['gensim_tokenized']\n",
        "\n",
        "  # format the series to be used\n",
        "  stopwords = []\n",
        "\n",
        "  sentences = [sentence for sentence in documents]\n",
        "  cleaned = []\n",
        "  for sentence in sentences:\n",
        "    cleaned_sentence = [word.lower() for word in sentence]\n",
        "    cleaned_sentence = [word for word in sentence if word not in stopwords]\n",
        "    cleaned.append(cleaned_sentence)\n",
        "\n",
        "  # get bigrams\n",
        "  bigram = Phrases(cleaned, min_count=min_count, threshold=threshold, \n",
        "                   delimiter=b' ')\n",
        "  bigram_phraser = Phraser(bigram)\n",
        "\n",
        "  bigramed_tokens = []\n",
        "  for sent in cleaned:\n",
        "      tokens = bigram_phraser[sent]\n",
        "      bigramed_tokens.append(tokens)\n",
        "\n",
        "  # run again to get trigrams\n",
        "  trigram = Phrases(bigramed_tokens, min_count=min_count, threshold=threshold, \n",
        "                    delimiter=b' ')\n",
        "  trigram_phraser = Phraser(trigram)\n",
        "\n",
        "  trigramed_tokens = []\n",
        "  for sent in bigramed_tokens:\n",
        "      tokens = trigram_phraser[sent]\n",
        "      trigramed_tokens.append(tokens)\n",
        "\n",
        "  # build a toy model to update with\n",
        "  model = Word2Vec(size=300, min_count=5)\n",
        "  model.build_vocab(trigramed_tokens)\n",
        "  total_examples = model.corpus_count\n",
        "\n",
        "  # add GloVe's vocabulary & weights\n",
        "  model.build_vocab([list(glove_vectors.vocab.keys())], update=True)\n",
        "\n",
        "  # train on our data\n",
        "  model.train(trigramed_tokens, total_examples=total_examples, epochs=model.epochs)\n",
        "  model_wv = model.wv\n",
        "  del model\n",
        "  return model_wv"
      ],
      "execution_count": 53,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9_IheEwhqWRi",
        "outputId": "a9ff4dae-0d01-419a-9a4f-40cf5a6a6c4c"
      },
      "source": [
        "for author in df['author'].unique():\r\n",
        "  w2v_dict[author] = train_glove_author(author, glove_vectors=glove_vectors)\r\n",
        "  print(f'{author} completed')"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Plato completed\n",
            "Aristotle completed\n",
            "Locke completed\n",
            "Hume completed\n",
            "Berkeley completed\n",
            "Spinoza completed\n",
            "Leibniz completed\n",
            "Descartes completed\n",
            "Malebranche completed\n",
            "Russell completed\n",
            "Moore completed\n",
            "Wittgenstein completed\n",
            "Lewis completed\n",
            "Quine completed\n",
            "Popper completed\n",
            "Kripke completed\n",
            "Foucault completed\n",
            "Derrida completed\n",
            "Deleuze completed\n",
            "Merleau-Ponty completed\n",
            "Husserl completed\n",
            "Heidegger completed\n",
            "Kant completed\n",
            "Fichte completed\n",
            "Hegel completed\n",
            "Marx completed\n",
            "Lenin completed\n",
            "Smith completed\n",
            "Ricardo completed\n",
            "Keynes completed\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "av0A9Ohwsizl"
      },
      "source": [
        "With this finished - our next step is to train one on the entire corpus for use in classification."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YaV6IV5esoW7"
      },
      "source": [
        "#### Building a Model for the full Corpus"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qQu70aYwChpm"
      },
      "source": [
        "documents = df['gensim_tokenized']\r\n",
        "\r\n",
        "# format the series to be used\r\n",
        "stopwords = []\r\n",
        "\r\n",
        "sentences = [sentence for sentence in documents]\r\n",
        "cleaned = []\r\n",
        "for sentence in sentences:\r\n",
        "  cleaned_sentence = [word.lower() for word in sentence]\r\n",
        "  cleaned_sentence = [word for word in sentence if word not in stopwords]\r\n",
        "  cleaned.append(cleaned_sentence)\r\n",
        "\r\n",
        "# get bigrams\r\n",
        "bigram = Phrases(cleaned, min_count=30, threshold=10, \r\n",
        "                  delimiter=b' ')\r\n",
        "bigram_phraser = Phraser(bigram)\r\n",
        "\r\n",
        "bigramed_tokens = []\r\n",
        "for sent in cleaned:\r\n",
        "    tokens = bigram_phraser[sent]\r\n",
        "    bigramed_tokens.append(tokens)\r\n",
        "\r\n",
        "# run again to get trigrams\r\n",
        "trigram = Phrases(bigramed_tokens, min_count=30, threshold=10, \r\n",
        "                  delimiter=b' ')\r\n",
        "trigram_phraser = Phraser(trigram)\r\n",
        "\r\n",
        "trigramed_tokens = []\r\n",
        "for sent in bigramed_tokens:\r\n",
        "    tokens = trigram_phraser[sent]\r\n",
        "    trigramed_tokens.append(tokens)\r\n",
        "\r\n",
        "# build a toy model to update with\r\n",
        "all_text_model = Word2Vec(size=300, min_count=5)\r\n",
        "all_text_model.build_vocab(trigramed_tokens)\r\n",
        "total_examples = all_text_model.corpus_count\r\n",
        "\r\n",
        "# add GloVe's vocabulary & weights\r\n",
        "all_text_model.build_vocab([list(glove_vectors.vocab.keys())], update=True)\r\n",
        "\r\n",
        "# train on our data\r\n",
        "all_text_model.train(trigramed_tokens, total_examples=total_examples, \r\n",
        "                     epochs=all_text_model.epochs)\r\n",
        "all_text_wv = all_text_model.wv"
      ],
      "execution_count": 50,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Wk3zbVgDbxl"
      },
      "source": [
        "As a test case, let's see how the philosophy thinks of itself as compared to how glove thinks of philosophy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3GNhMEqvDnpN",
        "outputId": "b6271d35-e5b0-444d-d8ba-63bee3140904"
      },
      "source": [
        "for model in [1, 2]:\r\n",
        "  if model == 1:\r\n",
        "    print(f'\\tPHILOSOPHY CORPUS')\r\n",
        "    print('------------------------------------')\r\n",
        "    test_w2v(all_text_wv, [(['philosophy'], [])])\r\n",
        "  if model == 2:\r\n",
        "    print(f'\\tBASE GLOVE')\r\n",
        "    print('------------------------------------')\r\n",
        "    test_w2v(glove_vectors, [(['philosophy'], [])])\r\n"
      ],
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\tPHILOSOPHY CORPUS\n",
            "------------------------------------\n",
            "Positive - ['philosophy']\tNegative - []\n",
            "- theology (0.8004)\n",
            "- metaphysics (0.77228)\n",
            "- religion (0.73747)\n",
            "- science (0.72425)\n",
            "- philosophical (0.71879)\n",
            "\n",
            "\tBASE GLOVE\n",
            "------------------------------------\n",
            "Positive - ['philosophy']\tNegative - []\n",
            "- theology (0.88151)\n",
            "- philosophical (0.84362)\n",
            "- mathematics (0.83389)\n",
            "- psychology (0.82387)\n",
            "- sociology (0.81085)\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DpeVE7mUFzlY"
      },
      "source": [
        "This sort of stands to reason - 'metaphysics' often has a different meaning outside of philosophical discussion, so it's not surprising to see it as the most changed term here. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A24iStA_rtOf"
      },
      "source": [
        "#### Finalized exporting"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wUxG_7YlrvIP"
      },
      "source": [
        "All in all, things look good, so let's export the vectors so that they can be used in our neural networks and in our dash app. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mg6rnhbD235n"
      },
      "source": [
        "all_text_wv.save_word2vec_format('/gdrive/MyDrive/Colab_Projects/Phil_NLP/w2v_models/w2v_for_nn.bin')\r\n",
        "all_text_wv.save('/gdrive/MyDrive/Colab_Projects/Phil_NLP/w2v_models/w2v_for_nn.wordvectors')"
      ],
      "execution_count": 59,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-nw5Hf0JStaD"
      },
      "source": [
        "for source in w2v_dict.keys():\r\n",
        "  w2v_dict[source].save(f'/gdrive/MyDrive/Colab_Projects/Phil_NLP/w2v_models/{source}_w2v.wordvectors')"
      ],
      "execution_count": 60,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V0wn9YYoGXel"
      },
      "source": [
        "And that's it! With this initial foray into deep NLP under our belt, we turned now to the task of using neural networks in classification. For that work, see our [neural networks notebook](https://github.com/kcalizadeh/phil_nlp/blob/master/Notebooks/4_neural_networks.ipynb)."
      ]
    }
  ]
}