hacker-news-posts.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring Posts on Hacker News "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "In this project, I am going to explore submissions to the site [Hacker News](https://news.ycombinator.com/). The dataset can be found [here](https://www.kaggle.com/hacker-news/hacker-news-posts).\n",
    "\n",
    "Submissions to the site can be grouped into two categories: questions (`Ask HN`) and posts that showcase something (`Show HN`). I'll examine which type of post receives more comments. I'll also explore whether there are certain times of the day when posts receive more comments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "import datetime as dt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First I load the dataset as a list of lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open('hacker_news.csv')\n",
    "hn = list(csv.reader(f))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# separate header row\n",
    "headers = hn[0]\n",
    "hn = hn[1:]\n",
    "\n",
    "# display headers\n",
    "print(headers)\n",
    "\n",
    "# display first 5 rows\n",
    "print(hn[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filter data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next I filter the data to include only posts in the `Ask HN` and `Show HN` categories. Specifically, I add entries from the original dataset, for each type of post, to separate lists: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ask_posts = []\n",
    "show_posts = []\n",
    "other_posts = []\n",
    "\n",
    "for row in hn:\n",
    "    title = row[1]\n",
    "    if title.lower().startswith('ask hn'):\n",
    "        ask_posts.append(row)\n",
    "    elif title.lower().startswith('show hn'):\n",
    "        show_posts.append(row)\n",
    "    else:\n",
    "        other_posts.append(row)\n",
    "\n",
    "# display data for one post in each list\n",
    "print(ask_posts[:1])\n",
    "print(show_posts[:1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# how many posts in each list?\n",
    "print(\"We have {0} 'Ask HN' posts and {1} 'Show HN' posts!\".format(len(ask_posts), len(show_posts)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Which type of posts receives more comments on average?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will determine whether `Ask HN` or `Show HN` receive more comments on average:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "total_ask_comments = 0\n",
    "for row in ask_posts:\n",
    "    # get number of comments for this entry\n",
    "    num_comments = int(row[4])\n",
    "    # add to tally of total comments for Ask HN posts\n",
    "    total_ask_comments += num_comments\n",
    "\n",
    "# compute average number of comments on Ask HN posts\n",
    "avg_ask_comments = total_ask_comments/len(ask_posts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "total_show_comments = 0\n",
    "for row in show_posts:\n",
    "    # get number of comments for this entry\n",
    "    num_comments = int(row[4])\n",
    "    # add to tally of total comments for Show HN posts\n",
    "    total_show_comments += num_comments\n",
    "\n",
    "# compute average number of comments on Ask HN posts\n",
    "avg_show_comments = total_show_comments/len(show_posts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# compare the results\n",
    "print('Average comments on Ask HN posts: {:.2f}'.format(avg_ask_comments))\n",
    "print('Average comments on Show HN posts: {:.2f}'.format(avg_show_comments))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On average, `Ask HN` posts receive more comments than `Show HN` posts -- not surprising since `Ask HN` posts are usually queries to the Hacker News community."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## At what times of day do 'Ask HN' posts receive the most comments?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, I'll examine the times of day at which `Ask HN` posts receive the most comments. To start, I iterate over the `ask_posts` data, creating a list where each element is a given post, the entries then refer to the post's date/time created and the corresponding number of comments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# iterate over ask_posts data\n",
    "# and create a list where each element has post date/time and number of comments\n",
    "result_list = []\n",
    "for row in ask_posts:\n",
    "    created_at = row[6]\n",
    "    result_list.append([row[6], int(row[4])])\n",
    "    \n",
    "# show first entry in list\n",
    "result_list[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, I create dictionaries to store the number of posts and number of total post comments per hour of the day:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# empty dicts to store # posts and comments by hour\n",
    "counts_by_hour = {}\n",
    "comments_by_hour = {}\n",
    "\n",
    "# iterate over result_list and extract hour of each post\n",
    "# then build dictionaries based on counts and comments per hour\n",
    "for element in result_list:\n",
    "    \n",
    "    # create date/time object\n",
    "    post_dt = dt.datetime.strptime(element[0], '%m/%d/%Y %H:%M')\n",
    "    # extract hour of post\n",
    "    post_hour = post_dt.strftime('%H')\n",
    "    \n",
    "    if post_hour not in counts_by_hour:\n",
    "        counts_by_hour[post_hour] = 1\n",
    "        comments_by_hour[post_hour] = element[1]\n",
    "    elif post_hour in counts_by_hour:\n",
    "        counts_by_hour[post_hour] += 1\n",
    "        comments_by_hour[post_hour] += element[1]\n",
    "\n",
    "# show each dictionary\n",
    "print(counts_by_hour)    \n",
    "print(comments_by_hour)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I can use these dictionaries to calculate, for each hour of the day, the average number of comments per post in a list of lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate average number of comments per post for each hour\n",
    "avg_by_hour = []\n",
    "for hour in counts_by_hour:\n",
    "    avg_by_hour.append([hour, comments_by_hour[hour]/counts_by_hour[hour]])\n",
    "    \n",
    "print(avg_by_hour)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This isn't very nice to read or easy to gain insights from, so I'll sort the list of lists in descending order of average comments per hour:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a list equivalent to avg_by_hour, but swap the columns (for sorting)\n",
    "swap_avg_by_hour = []\n",
    "for row in avg_by_hour:\n",
    "    swap_avg_by_hour.append([row[1], row[0]])\n",
    "    \n",
    "print(swap_avg_by_hour)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use the sorted() function to sort swap_avg_by_hour in descending order\n",
    "# (this sorts by average number of comments)\n",
    "sorted_swap = sorted(swap_avg_by_hour, reverse = True)\n",
    "print(sorted_swap)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Top 5 Hours for Ask Post Comments:')\n",
    "for row in sorted_swap[0:5]:\n",
    "    hr = dt.datetime.strptime(row[1], '%H').strftime('%H:%M')\n",
    "    print('{hr}: {avg:.2f} average comments per post'.format(hr=hr, avg=row[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that `Ask HN` posts receive the most comments later in the day: at 3pm, 2pm, 8pm, 4pm, and 9pm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## At what times of day do 'Show HN' posts receive the most comments?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, I'll do the same thing for `Show HN` posts, examining the hours during the day where the highest average number comments per post are made:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# iterate over show_posts data\n",
    "# and create a list where each element has post date/time and number of comments\n",
    "result_list = []\n",
    "for row in show_posts:\n",
    "    created_at = row[6]\n",
    "    result_list.append([row[6], int(row[4])])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# empty dicts to store # posts and comments by hour\n",
    "counts_by_hour = {}\n",
    "comments_by_hour = {}\n",
    "\n",
    "# iterate over result_list and extract hour of each post\n",
    "# then build dictionaries based on counts and comments per hour\n",
    "for element in result_list:\n",
    "    \n",
    "    # create date/time object\n",
    "    post_dt = dt.datetime.strptime(element[0], '%m/%d/%Y %H:%M')\n",
    "    # extract hour of post\n",
    "    post_hour = post_dt.strftime('%H')\n",
    "    \n",
    "    if post_hour not in counts_by_hour:\n",
    "        counts_by_hour[post_hour] = 1\n",
    "        comments_by_hour[post_hour] = element[1]\n",
    "    elif post_hour in counts_by_hour:\n",
    "        counts_by_hour[post_hour] += 1\n",
    "        comments_by_hour[post_hour] += element[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate average number of comments per post for each hour\n",
    "avg_by_hour = []\n",
    "for hour in counts_by_hour:\n",
    "    avg_by_hour.append([hour, comments_by_hour[hour]/counts_by_hour[hour]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a list equivalent to avg_by_hour, but swap the columns (for sorting)\n",
    "swap_avg_by_hour = []\n",
    "for row in avg_by_hour:\n",
    "    swap_avg_by_hour.append([row[1], row[0]])\n",
    "\n",
    "# use the sorted() function to sort swap_avg_by_hour in descending order\n",
    "# (this sorts by average number of comments)\n",
    "sorted_swap = sorted(swap_avg_by_hour, reverse = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Top 5 Hours for Show Post Comments:')\n",
    "for row in sorted_swap[0:5]:\n",
    "    hr = dt.datetime.strptime(row[1], '%H').strftime('%H:%M')\n",
    "    print('{hr}: {avg:.2f} average comments per post'.format(hr=hr, avg=row[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to `Ask HN` posts, `Show HN` posts receive the highest number of average comments later in the day: at 6pm, midnight, 2pm, 11pm, and 10pm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this report, I explored data on posts to the site `Hacker News` and discovered several things:\n",
    "\n",
    "- There are more posts made in the `Ask HN` category (questions) than in the `Show HN` category (which showcase something).\n",
    "- `Ask HN` and `Show HN` posts receive the most comments on average later in the day, particularly into the evening hours."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}