testing_all_the_news.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "testing all-the-news.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Interpause/pseudo-text/blob/master/testing_all_the_news.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4vLD25r6zym1",
        "colab_type": "text"
      },
      "source": [
        "#overview\n",
        "I'm assuming everything in all-the-news is mostly true. It doesn't matter much for this model anyways since I am aiming to have it actively scrap relevant articles. I'll later figure out how to get adversarial examples (must indicate true news as fake when all given articles are fake; does this stop the model from picking on finer details like word usage?)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XSs795bcLHfS",
        "colab_type": "text"
      },
      "source": [
        "# Conclusion\n",
        "This dataset is pretty bad. It will take a lot of cleaning to use.... I would still have used this datset though but I have found something better. Not exactly a dataset, but a modern newspaper scraper that can very accurately parse most mainstream news sources, including author, category, etc. I will be using that..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "skJrMHrg0Zfs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import sqlite3\n",
        "import pandas as pd"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K-7_UOKazuTu",
        "colab_type": "code",
        "outputId": "6d8e887a-08e2-401c-b3a8-7c7f036f9051",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "#Online instance only.\n",
        "from google.colab import drive #drive.flush_and_unmount()\n",
        "import os\n",
        "drive.mount('/content/drive')\n",
        "os.chdir('/content/drive/My Drive/CSIT Internship/code')"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UEBPadUz0WY7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "with sqlite3.connect('./raw/all-the-news.db') as db:\n",
        "    raw = pd.read_sql_query('SELECT * from longform',db)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Uv417hhKODBc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df2 = pd.read_pickle('./raw/all-the-news.pkl')\n",
        "import bz2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1eb0nChqSbpx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df2.to_pickle('./raw/all-the-news.pkl.xz',compression='xz')\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "l4xDKB44-IkD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df = pd.read_pickle('./raw/all-the-news.pkl.xz')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZJb1Vu_Z2ghY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df = df.applymap(lambda x: x.strip() if isinstance(x, str) else x)\n",
        "df = df.dropna(subset=['content'])\n",
        "df = df[df.content.str.len() != 0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xY3NAzuBIszU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df = df[df.content.str.len() >= 20]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nQJYnyyII93x",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "remove = ('No.',',','[ ]',\"['']\",'[, ]','[] ,','“','[]')\n",
        "def filterWeird(x):\n",
        "    return all(not chk == x for chk in remove)\n",
        "df = df[df.content.apply(filterWeird)]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3Dc6GtUaAamA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df = df.rename(columns={'content':'text'})"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4H7_41ckWKg6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import unicodedata\n",
        "#normally also helps remove foreign languages\n",
        "removables = ['AD']\n",
        "def clean(x):\n",
        "    x.title = ' '.join(w for w in unicodedata.normalize('NFKD',x.title.strip()).encode('ascii','ignore').decode('utf8','ignore').split() if w not in removables)\n",
        "    x.text = ' '.join(w for w in unicodedata.normalize('NFKD',x.text.strip()).encode('ascii','ignore').decode('utf8','ignore').split() if w not in removables)\n",
        "    if len(x.title) < 1 or len(x.text.split()) < 150:\n",
        "        x.title = 'None'\n",
        "        x.text = 'None'\n",
        "    return x"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9uJyIVtitNPe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df = df.dropna(subset=('text',))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zhf5fVkcby7i",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df = df.apply(clean,axis=1)[df.text != 'None']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R9a813peIEkT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df = df.drop_duplicates(subset=('text',))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "liPbuMayJSxH",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "434c2a86-5a44-4aa8-8696-dffa9b682364"
      },
      "source": [
        "if not 'uuid' in df:\n",
        "    import uuid\n",
        "    df = df.assign(uuid=tuple(uuid.uuid4() for x in range(len(df))))\n",
        "    print('done')"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "done\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rVFRTooJ1L3U",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df.text = df.text.str.lower()\n",
        "df.title = df.title.str.lower()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8GcJrUwojMX3",
        "colab_type": "code",
        "outputId": "7815df72-59bc-4bc8-88ba-9faf3ec06ba1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "subset_impt = ('text',)\n",
        "df = df.drop_duplicates(subset=subset_impt).dropna(subset=subset_impt).drop_duplicates(subset=subset_impt).dropna(subset=subset_impt).apply(clean,axis=1)[df.text != 'None'].drop_duplicates(subset=subset_impt).dropna(subset=subset_impt).reset_index(drop=True)"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:2: UserWarning: Boolean Series key will be reindexed to match DataFrame index.\n",
            "  \n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rjDqyUXqb7IF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df = df[df.text != 'None']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U6DwUq5gJuzu",
        "colab_type": "code",
        "outputId": "cac21d2f-d1fa-4909-a116-f5f0fd3a749f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "set(df[df.text.str.len() < 100].text)"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "set()"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h6a0ozFsKjJ_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 638
        },
        "outputId": "9e17191b-5e7a-4f01-9c39-d0fac877cc24"
      },
      "source": [
        "df.groupby('publication').count()"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>title</th>\n",
              "      <th>author</th>\n",
              "      <th>date</th>\n",
              "      <th>text</th>\n",
              "      <th>year</th>\n",
              "      <th>month</th>\n",
              "      <th>category</th>\n",
              "      <th>digital</th>\n",
              "      <th>section</th>\n",
              "      <th>url</th>\n",
              "      <th>uuid</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>publication</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Atlantic</th>\n",
              "      <td>7204</td>\n",
              "      <td>7204</td>\n",
              "      <td>6253</td>\n",
              "      <td>7204</td>\n",
              "      <td>7204</td>\n",
              "      <td>7204</td>\n",
              "      <td>7204</td>\n",
              "      <td>7204</td>\n",
              "      <td>7204</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>7204</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Breitbart</th>\n",
              "      <td>21337</td>\n",
              "      <td>21337</td>\n",
              "      <td>21337</td>\n",
              "      <td>21337</td>\n",
              "      <td>21337</td>\n",
              "      <td>21337</td>\n",
              "      <td>21337</td>\n",
              "      <td>21337</td>\n",
              "      <td>21337</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>21337</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Business Insider</th>\n",
              "      <td>5396</td>\n",
              "      <td>5396</td>\n",
              "      <td>4017</td>\n",
              "      <td>5396</td>\n",
              "      <td>5396</td>\n",
              "      <td>5396</td>\n",
              "      <td>5396</td>\n",
              "      <td>5396</td>\n",
              "      <td>5396</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>5396</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Buzzfeed News</th>\n",
              "      <td>4516</td>\n",
              "      <td>4516</td>\n",
              "      <td>4516</td>\n",
              "      <td>4516</td>\n",
              "      <td>4516</td>\n",
              "      <td>4516</td>\n",
              "      <td>4516</td>\n",
              "      <td>4516</td>\n",
              "      <td>4516</td>\n",
              "      <td>4516</td>\n",
              "      <td>4516</td>\n",
              "      <td>4516</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>CNN</th>\n",
              "      <td>11041</td>\n",
              "      <td>11041</td>\n",
              "      <td>6856</td>\n",
              "      <td>11041</td>\n",
              "      <td>11041</td>\n",
              "      <td>11041</td>\n",
              "      <td>11041</td>\n",
              "      <td>11041</td>\n",
              "      <td>11041</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>11041</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Fox News</th>\n",
              "      <td>4126</td>\n",
              "      <td>4126</td>\n",
              "      <td>1110</td>\n",
              "      <td>4126</td>\n",
              "      <td>4126</td>\n",
              "      <td>4126</td>\n",
              "      <td>4126</td>\n",
              "      <td>4126</td>\n",
              "      <td>4126</td>\n",
              "      <td>4124</td>\n",
              "      <td>4124</td>\n",
              "      <td>4126</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Guardian</th>\n",
              "      <td>8590</td>\n",
              "      <td>8590</td>\n",
              "      <td>7223</td>\n",
              "      <td>8571</td>\n",
              "      <td>8590</td>\n",
              "      <td>8571</td>\n",
              "      <td>8571</td>\n",
              "      <td>8590</td>\n",
              "      <td>8590</td>\n",
              "      <td>8590</td>\n",
              "      <td>8590</td>\n",
              "      <td>8590</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Los Angeles Times</th>\n",
              "      <td>8585</td>\n",
              "      <td>8585</td>\n",
              "      <td>8576</td>\n",
              "      <td>8574</td>\n",
              "      <td>8585</td>\n",
              "      <td>8574</td>\n",
              "      <td>8574</td>\n",
              "      <td>8585</td>\n",
              "      <td>8585</td>\n",
              "      <td>30</td>\n",
              "      <td>30</td>\n",
              "      <td>8585</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>NPR</th>\n",
              "      <td>11750</td>\n",
              "      <td>11750</td>\n",
              "      <td>11466</td>\n",
              "      <td>11750</td>\n",
              "      <td>11750</td>\n",
              "      <td>11750</td>\n",
              "      <td>11750</td>\n",
              "      <td>11750</td>\n",
              "      <td>11750</td>\n",
              "      <td>7665</td>\n",
              "      <td>11750</td>\n",
              "      <td>11750</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>National Review</th>\n",
              "      <td>6136</td>\n",
              "      <td>6136</td>\n",
              "      <td>6136</td>\n",
              "      <td>6136</td>\n",
              "      <td>6136</td>\n",
              "      <td>6136</td>\n",
              "      <td>6136</td>\n",
              "      <td>0</td>\n",
              "      <td>6136</td>\n",
              "      <td>6136</td>\n",
              "      <td>6136</td>\n",
              "      <td>6136</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>New Inquiry</th>\n",
              "      <td>2618</td>\n",
              "      <td>2618</td>\n",
              "      <td>2618</td>\n",
              "      <td>2618</td>\n",
              "      <td>2618</td>\n",
              "      <td>2618</td>\n",
              "      <td>2618</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>2618</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>New York Post</th>\n",
              "      <td>15334</td>\n",
              "      <td>15334</td>\n",
              "      <td>15328</td>\n",
              "      <td>15334</td>\n",
              "      <td>15334</td>\n",
              "      <td>15334</td>\n",
              "      <td>15334</td>\n",
              "      <td>0</td>\n",
              "      <td>15334</td>\n",
              "      <td>15334</td>\n",
              "      <td>15334</td>\n",
              "      <td>15334</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>New York Times</th>\n",
              "      <td>29914</td>\n",
              "      <td>29914</td>\n",
              "      <td>29914</td>\n",
              "      <td>29914</td>\n",
              "      <td>29914</td>\n",
              "      <td>29914</td>\n",
              "      <td>29914</td>\n",
              "      <td>29914</td>\n",
              "      <td>29914</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>29914</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Reuters</th>\n",
              "      <td>10706</td>\n",
              "      <td>10706</td>\n",
              "      <td>10706</td>\n",
              "      <td>10706</td>\n",
              "      <td>10706</td>\n",
              "      <td>10706</td>\n",
              "      <td>10706</td>\n",
              "      <td>10706</td>\n",
              "      <td>10706</td>\n",
              "      <td>0</td>\n",
              "      <td>10706</td>\n",
              "      <td>10706</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Talking Points Memo</th>\n",
              "      <td>3968</td>\n",
              "      <td>3968</td>\n",
              "      <td>1401</td>\n",
              "      <td>1855</td>\n",
              "      <td>3968</td>\n",
              "      <td>1855</td>\n",
              "      <td>1855</td>\n",
              "      <td>3968</td>\n",
              "      <td>3968</td>\n",
              "      <td>3968</td>\n",
              "      <td>3968</td>\n",
              "      <td>3968</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Verge</th>\n",
              "      <td>1098</td>\n",
              "      <td>1098</td>\n",
              "      <td>1098</td>\n",
              "      <td>1098</td>\n",
              "      <td>1098</td>\n",
              "      <td>1098</td>\n",
              "      <td>1098</td>\n",
              "      <td>1098</td>\n",
              "      <td>1098</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1098</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Vox</th>\n",
              "      <td>4919</td>\n",
              "      <td>4919</td>\n",
              "      <td>4919</td>\n",
              "      <td>4919</td>\n",
              "      <td>4919</td>\n",
              "      <td>4919</td>\n",
              "      <td>4919</td>\n",
              "      <td>4919</td>\n",
              "      <td>4919</td>\n",
              "      <td>0</td>\n",
              "      <td>4919</td>\n",
              "      <td>4919</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Washington Post</th>\n",
              "      <td>10976</td>\n",
              "      <td>10976</td>\n",
              "      <td>10939</td>\n",
              "      <td>10976</td>\n",
              "      <td>10976</td>\n",
              "      <td>10976</td>\n",
              "      <td>10976</td>\n",
              "      <td>10976</td>\n",
              "      <td>10976</td>\n",
              "      <td>10976</td>\n",
              "      <td>10976</td>\n",
              "      <td>10976</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                        id  title  author  ...  section    url   uuid\n",
              "publication                                ...                       \n",
              "Atlantic              7204   7204    6253  ...        0      0   7204\n",
              "Breitbart            21337  21337   21337  ...        0      0  21337\n",
              "Business Insider      5396   5396    4017  ...        0      0   5396\n",
              "Buzzfeed News         4516   4516    4516  ...     4516   4516   4516\n",
              "CNN                  11041  11041    6856  ...        0      0  11041\n",
              "Fox News              4126   4126    1110  ...     4124   4124   4126\n",
              "Guardian              8590   8590    7223  ...     8590   8590   8590\n",
              "Los Angeles Times     8585   8585    8576  ...       30     30   8585\n",
              "NPR                  11750  11750   11466  ...     7665  11750  11750\n",
              "National Review       6136   6136    6136  ...     6136   6136   6136\n",
              "New Inquiry           2618   2618    2618  ...        0      0   2618\n",
              "New York Post        15334  15334   15328  ...    15334  15334  15334\n",
              "New York Times       29914  29914   29914  ...        0      0  29914\n",
              "Reuters              10706  10706   10706  ...        0  10706  10706\n",
              "Talking Points Memo   3968   3968    1401  ...     3968   3968   3968\n",
              "Verge                 1098   1098    1098  ...        0      0   1098\n",
              "Vox                   4919   4919    4919  ...        0   4919   4919\n",
              "Washington Post      10976  10976   10939  ...    10976  10976  10976\n",
              "\n",
              "[18 rows x 12 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ARzqPyvE_ni0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df = df.drop(columns =['id','author','year','month','digital'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8uK9oNRNC7Vd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df.to_pickle('./raw/all-the-news.pkl.xz',compression='xz')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mX0LySOeDiLE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}