Reddit-Submissions-Comments-Extraction-via-Pushshift.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Using Pushshift Module to extract Submissions.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j-QQANye3ecE"
      },
      "source": [
        "# Using Pushshift Module to extract Submissions Data from Reddit via Python\n",
        "\n",
        "PRAW is pretty good at gettin reddit data but there are some limitations with it.\n",
        "Including the removal of the [subreddit.submissions endpoint](https://www.reddit.com/r/changelog/comments/7tus5f/update_to_search_api/.). \n",
        "\n",
        "So for extracting Reddit submissions and the primarily data such as upvotes and comments count, I put together this notebook using Pushshift.\n",
        "\n",
        "If you still prefer PRAW for extract submissions, I have written a code [template here](https://github.com/SeyiAgboola/Seyi_Projects/blob/master/submission_list.py).\n",
        "\n",
        "I will also host the code on GitHub.\n",
        "\n",
        "More info on the removal of the [subreddit.submissions endpoint](https://www.reddit.com/r/redditdev/comments/8bia9n/praw_psa_the_subredditsubmissions_method_no/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GyarnetivJPG"
      },
      "source": [
        "# Import modules"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VfmIce345UaB"
      },
      "source": [
        "import pandas as pd\n",
        "import requests #Pushshift accesses Reddit via an url so this is needed\n",
        "import json #JSON manipulation\n",
        "import csv #To Convert final table into a csv file to save to your machine\n",
        "import time\n",
        "import datetime"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pj8JGor4vMwC"
      },
      "source": [
        "# Pushshift URL Examples"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yH8nB39CzZfU"
      },
      "source": [
        "#We can access the Pushshift API through building an URL with the relevant parameters without even needing Reddit credentials.\n",
        "#These are some examples. You can follow the links and they will generate a page with JSON data\n",
        "search_ps4_after_date = \"https://api.pushshift.io/reddit/search/submission/?q=screenshot&after=1514764800&before=1517443200&subreddit=PS4\"\n",
        "search_science = \"https://api.pushshift.io/reddit/search/submission/?q=science\""
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AFPXbeD-0kGc"
      },
      "source": [
        "# Parameters for your Pushshift URL\n",
        "These are probably the most important parameters to consider when building your Pushshift URL:\n",
        "\n",
        "* size — increase limit of returned entries to 1000\n",
        "* after — where to start the search\n",
        "* before — where to end the search\n",
        "* title — to search only within the submission’s title\n",
        "* subreddit — to narrow it down to a particular subreddit"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mDma-H_k0frf"
      },
      "source": [
        "#Adapted from this https://gist.github.com/dylankilkenny/3dbf6123527260165f8c5c3bc3ee331b\n",
        "#This function builds an Pushshift URL, accesses the webpage and stores JSON data in a nested list\n",
        "def getPushshiftData(query, after, before, sub):\n",
        "    #Build URL\n",
        "    url = 'https://api.pushshift.io/reddit/search/submission/?title='+str(query)+'&size=1000&after='+str(after)+'&before='+str(before)+'&subreddit='+str(sub)\n",
        "    #Print URL to show user\n",
        "    print(url)\n",
        "    #Request URL\n",
        "    r = requests.get(url)\n",
        "    #Load JSON data from webpage into data variable\n",
        "    data = json.loads(r.text)\n",
        "    #return the data element which contains all the submissions data\n",
        "    return data['data']"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H524E6kkvh1O"
      },
      "source": [
        "# Extract key information from Submissions\n",
        "\n",
        "We want key data for further analysis including: \n",
        "* Submission Title\n",
        "* URL \n",
        "* Flair\n",
        "* Author\n",
        "* Submission post ID\n",
        "* Score\n",
        "* Upload Time\n",
        "* No. of Comments \n",
        "* Permalink.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N6qX7glQ1-nl"
      },
      "source": [
        "#This function will be used to extract the key data points from each JSON result\n",
        "def collectSubData(subm):\n",
        "    #subData was created at the start to hold all the data which is then added to our global subStats dictionary.\n",
        "    subData = list() #list to store data points\n",
        "    title = subm['title']\n",
        "    url = subm['url']\n",
        "    #flairs are not always present so we wrap in try/except\n",
        "    try:\n",
        "        flair = subm['link_flair_text']\n",
        "    except KeyError:\n",
        "        flair = \"NaN\"    \n",
        "    author = subm['author']\n",
        "    sub_id = subm['id']\n",
        "    score = subm['score']\n",
        "    created = datetime.datetime.fromtimestamp(subm['created_utc']) #1520561700.0\n",
        "    numComms = subm['num_comments']\n",
        "    permalink = subm['permalink']\n",
        "    #Put all data points into a tuple and append to subData\n",
        "    subData.append((sub_id,title,url,author,score,created,numComms,permalink,flair))\n",
        "    #Create a dictionary entry of current submission data and store all data related to it\n",
        "    subStats[sub_id] = subData"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vCDRrn9nwRsj"
      },
      "source": [
        "# Update your Search Settings here"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V0fNU7eS2mwT"
      },
      "source": [
        "#Create your timestamps and queries for your search URL\n",
        "#https://www.unixtimestamp.com/index.php > Use this to create your timestamps\n",
        "after = \"1577836800\" #Submissions after this timestamp (1577836800 = 01 Jan 20)\n",
        "before = \"1607040000\" #Submissions before this timestamp (1607040000 = 04 Dec 20)\n",
        "query = \"Cyberpunk 2077\" #Keyword(s) to look for in submissions\n",
        "sub = \"Games\" #Which Subreddit to search in\n",
        "\n",
        "#subCount tracks the no. of total submissions we collect\n",
        "subCount = 0\n",
        "#subStats is the dictionary where we will store our data.\n",
        "subStats = {}"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6wP_j8pp2-M8"
      },
      "source": [
        "# We need to run this function outside the loop first to get the updated after variable\n",
        "data = getPushshiftData(query, after, before, sub)\n",
        "# Will run until all posts have been gathered i.e. When the length of data variable = 0\n",
        "# from the 'after' date up until before date\n",
        "while len(data) > 0: #The length of data is the number submissions (data[0], data[1] etc), once it hits zero (after and before vars are the same) end\n",
        "    for submission in data:\n",
        "        collectSubData(submission)\n",
        "        subCount+=1\n",
        "    # Calls getPushshiftData() with the created date of the last submission\n",
        "    print(len(data))\n",
        "    print(str(datetime.datetime.fromtimestamp(data[-1]['created_utc'])))\n",
        "    #update after variable to last created date of submission\n",
        "    after = data[-1]['created_utc']\n",
        "    #data has changed due to the new after variable provided by above code\n",
        "    data = getPushshiftData(query, after, before, sub)\n",
        "    \n",
        "print(len(data))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OaeTGu7mwyoU"
      },
      "source": [
        "# Check your Submission Extraction was successful"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uVLuSJ8e8p7i"
      },
      "source": [
        "print(str(len(subStats)) + \" submissions have added to list\")\n",
        "print(\"1st entry is:\")\n",
        "print(list(subStats.values())[0][0][1] + \" created: \" + str(list(subStats.values())[0][0][5]))\n",
        "print(\"Last entry is:\")\n",
        "print(list(subStats.values())[-1][0][1] + \" created: \" + str(list(subStats.values())[-1][0][5]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BAm_zZZfw521"
      },
      "source": [
        "# Save data to CSV file"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MBikywNJ8ufl"
      },
      "source": [
        "def updateSubs_file():\n",
        "    upload_count = 0\n",
        "    #location = \"\\\\Reddit Data\\\\\" >> If you're running this outside of a notebook you'll need this to direct to a specific location\n",
        "    print(\"input filename of submission file, please add .csv\")\n",
        "    filename = input() #This asks the user what to name the file\n",
        "    file = filename\n",
        "    with open(file, 'w', newline='', encoding='utf-8') as file: \n",
        "        a = csv.writer(file, delimiter=',')\n",
        "        headers = [\"Post ID\",\"Title\",\"Url\",\"Author\",\"Score\",\"Publish Date\",\"Total No. of Comments\",\"Permalink\",\"Flair\"]\n",
        "        a.writerow(headers)\n",
        "        for sub in subStats:\n",
        "            a.writerow(subStats[sub][0])\n",
        "            upload_count+=1\n",
        "            \n",
        "        print(str(upload_count) + \" submissions have been uploaded\")\n",
        "updateSubs_file()"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}