Exploring Hacker News Posts.ipynb

{
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2-final"
  },
  "orig_nbformat": 2,
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.9.2 64-bit",
   "metadata": {
    "interpreter": {
     "hash": "42ffdba7f0de544957ef69efdb450b60517fb3f2f7c6065358c1f9d2622fefe6"
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2,
 "cells": [
  {
   "source": [
    "# Exploring Hacker News Posts\n",
    "***\n",
    "This is a guided-project under Dataquest's curriculum and is meant for personal development.\n",
    "***\n",
    "Hacker News is a reddit-like website that is popular among technology and startup circles. There are many types of posts on Hacker News, for example, sharing information or news, asking questions, or showing interesting projects. \n",
    "\n",
    "This project is to analyze posts which titles begin with *Ask HN* and *Show HN*, which one has more comments on average and whether there is a point in time where post receives more comments. Title begins with *Ask HN* is used when a user would like to ask a question, and title begins with *Show HN* is used when a user would like to show a project or any interesting things to the community.\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "## Step 1 - Data preparation\n",
    "\n",
    "First, the data is downloaded from <a href=\"https://www.kaggle.com/hacker-news/hacker-news-posts\">Kaggle</a> in CSV format. The data will be imported, cleaned, and separated into 3 groups (ask posts, show posts, and other posts) per our goal.\n",
    "\n",
    "### Step 1a - Import data\n",
    "The data set contains Hacker News posts during 12-month period (up to 26 September 2016). There are 7 columns in this data set.\n",
    "\n",
    "|  Column name  | Description                              |\n",
    "|---------------|------------------------------------------|\n",
    "| id            | ID of the post                           |\n",
    "| title         | Title of the post                        |\n",
    "| url           | URL of the item being linked to          |\n",
    "| num_points    | Number of votes the posts received       |\n",
    "| num_comments  | Number of comments the post received     |\n",
    "| author        | User that submitted the post             |\n",
    "| created_at    | Date and time the post was made (in EST) |"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function is to extract data in the file to a more readable format.\n",
    "def explore_data(dataset, start, end, rows_and_columns=False):\n",
    "    dataset_slice = dataset[start:end]\n",
    "    for row in dataset_slice:\n",
    "        print(row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at']\n\n['12579008', 'You have two days to comment if you want stem cells to be classified as your own', 'http://www.regulations.gov/document?D=FDA-2015-D-3719-0018', '1', '0', 'altstar', '9/26/2016 3:26']\n['12579005', 'SQLAR  the SQLite Archiver', 'https://www.sqlite.org/sqlar/doc/trunk/README.md', '1', '0', 'blacksqr', '9/26/2016 3:24']\n['12578997', 'What if we just printed a flatscreen television on the side of our boxes?', 'https://medium.com/vanmoof/our-secrets-out-f21c1f03fdc8#.ietxmez43', '1', '0', 'pavel_lishin', '9/26/2016 3:19']\n['12578989', 'algorithmic music', 'http://cacm.acm.org/magazines/2011/7/109891-algorithmic-composition/fulltext', '1', '0', 'poindontcare', '9/26/2016 3:16']\n['12578979', 'How the Data Vault Enables the Next-Gen Data Warehouse and Data Lake', 'https://www.talend.com/blog/2016/05/12/talend-and-Â\\x93the-data-vaultÂ\\x94', '1', '0', 'markgainor1', '9/26/2016 3:14']\n\nNumber of rows: 293,119\nNumber of columns: 7\n"
     ]
    }
   ],
   "source": [
    "from csv import reader\n",
    "#1. Open the file\n",
    "data = open(\"HN_posts_year_to_Sep_26_2016.csv\", encoding=\"utf-8\")\n",
    "#2. Convert the file into a list\n",
    "hn = list(reader(data))\n",
    "#3. Separate headers and data\n",
    "headers = hn[0]\n",
    "hn = hn[1:]\n",
    "\n",
    "print(headers)\n",
    "print()\n",
    "explore_data(hn, 0 , 5)\n",
    "print()\n",
    "print(f\"Number of rows: {len(hn):,.0f}\")\n",
    "print(f\"Number of columns: {len(hn[0]):,.0f}\")"
   ]
  },
  {
   "source": [
    "### Step 1b - Clean data\n",
    "First, we will check that each rows contain all columns. Then we will remove posts with zero comment as we focus only posts with comment(s)."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Check ended.\nEach rows contain all columns: True\n"
     ]
    }
   ],
   "source": [
    "# Check each row contain all columns\n",
    "col_row = True\n",
    "\n",
    "while col_row:\n",
    "    for row in hn:\n",
    "        if len(headers) != len(row):\n",
    "            print(row, \"\\n\")\n",
    "            print(\"Index of the row is \" + str(hn.index(row)))\n",
    "            col_row = False\n",
    "    print(\"Check ended.\")\n",
    "    print(f\"Each rows contain all columns: {col_row}\")\n",
    "    break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Hacker News posts with comment(s): 80,401 posts\n\n['12578975', 'Saving the Hassle of Shopping', 'https://blog.menswr.com/2016/09/07/whats-new-with-your-style-feed/', '1', '1', 'bdoux', '9/26/2016 3:13']\n['12578908', 'Ask HN: What TLD do you use for local development?', '', '4', '7', 'Sevrene', '9/26/2016 2:53']\n['12578822', 'Amazons Algorithms Dont Find You the Best Deals', 'https://www.technologyreview.com/s/602442/amazons-algorithms-dont-find-you-the-best-deals/', '1', '1', 'yarapavan', '9/26/2016 2:26']\n"
     ]
    }
   ],
   "source": [
    "# Remove posts with zero comments\n",
    "hn_final = []\n",
    "\n",
    "for row in hn:\n",
    "    if int(row[4]) != 0:\n",
    "        hn_final.append(row)\n",
    "\n",
    "print(f\"Hacker News posts with comment(s): {len(hn_final):,.0f} posts\")\n",
    "print()\n",
    "explore_data(hn_final, 0, 3)"
   ]
  },
  {
   "source": [
    "### Step 1c - Separate data\n",
    "The data set will now be separated into 3 groups: ask posts, show posts, and other posts. for our analysis, we will focus only at ask posts and show posts."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Ask HN : 6,911 posts\n['12578908', 'Ask HN: What TLD do you use for local development?', '', '4', '7', 'Sevrene', '9/26/2016 2:53']\n['12578522', 'Ask HN: How do you pass on your work when you die?', '', '6', '3', 'PascLeRasc', '9/26/2016 1:17']\n['12577870', 'Ask HN: Why join a fund when you can be an angel?', '', '1', '3', 'anthony_james', '9/25/2016 22:48']\n\nShow HN : 5,059 posts\n['12577142', 'Show HN: Jumble  Essays on the go #PaulInYourPocket', 'https://itunes.apple.com/us/app/jumble-find-startup-essay/id1150939197?ls=1&mt=8', '1', '1', 'ryderj', '9/25/2016 20:06']\n['12576813', 'Show HN: Learn Japanese Vocab via multiple choice questions', 'http://japanese.vul.io/', '1', '1', 'soulchild37', '9/25/2016 19:06']\n['12576090', 'Show HN: Markov chain Twitter bot. Trained on comments left on Pornhub', 'https://twitter.com/botsonasty', '3', '1', 'keepingscore', '9/25/2016 16:50']\n\nOther posts : 68,431 posts\n['12578975', 'Saving the Hassle of Shopping', 'https://blog.menswr.com/2016/09/07/whats-new-with-your-style-feed/', '1', '1', 'bdoux', '9/26/2016 3:13']\n['12578822', 'Amazons Algorithms Dont Find You the Best Deals', 'https://www.technologyreview.com/s/602442/amazons-algorithms-dont-find-you-the-best-deals/', '1', '1', 'yarapavan', '9/26/2016 2:26']\n['12578694', 'Emergency dose of epinephrine that does not cost an arm and a leg', 'http://m.imgur.com/gallery/th6Ua', '2', '1', 'dredmorbius', '9/26/2016 1:54']\n\n"
     ]
    }
   ],
   "source": [
    "# Separate Ask HN, Show HN and other posts into different lists\n",
    "ask_posts = []\n",
    "show_posts = []\n",
    "other_posts = []\n",
    "\n",
    "for row in hn_final:\n",
    "    title = row[1]\n",
    "    if title.lower().startswith(\"ask hn\"):\n",
    "        ask_posts.append(row)\n",
    "    elif title.lower().startswith(\"show hn\"):\n",
    "        show_posts.append(row)\n",
    "    else:\n",
    "        other_posts.append(row)\n",
    "\n",
    "assert len(hn_final) == len(ask_posts) + len(show_posts) + len(other_posts)\n",
    "\n",
    "print(f\"Ask HN : {len(ask_posts):,.0f} posts\")\n",
    "explore_data(ask_posts, 0, 3)\n",
    "print()\n",
    "print(f\"Show HN : {len(show_posts):,.0f} posts\")\n",
    "explore_data(show_posts, 0, 3)\n",
    "print()\n",
    "print(f\"Other posts : {len(other_posts):,.0f} posts\")\n",
    "explore_data(other_posts, 0, 3)\n",
    "print()"
   ]
  },
  {
   "source": [
    "## Step 2 - Data analysis\n",
    "Our analysis is aim to answer 2 questions:\n",
    "\n",
    "1. which group has more comments on average \n",
    "2. whether there is a point in time where post receives more comments. \n",
    "\n",
    "### Which group has more comments on average?\n",
    "For each groups, an average number of comments is calculated, then compared against each other."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Total comments in Ask HN: 94,986 comments\nAverage comments per an Ask HN post: 14 comments\n"
     ]
    }
   ],
   "source": [
    "# Find total number of comments for Ask HN posts\n",
    "total_ask_comments = 0\n",
    "\n",
    "for row in ask_posts:\n",
    "    comments = int(row[4])\n",
    "    total_ask_comments += comments\n",
    "\n",
    "print(f\"Total comments in Ask HN: {total_ask_comments:,.0f} comments\")\n",
    "\n",
    "# Find average number of comments for Ask HN posts\n",
    "avg_ask_comments = total_ask_comments / len(ask_posts)\n",
    "\n",
    "print(f\"Average comments per an Ask HN post: {avg_ask_comments:,.0f} comments\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Total comments in Show HN: 49,633 comments\nAverage comments per a Show HN post: 10 comments\n"
     ]
    }
   ],
   "source": [
    "# Find total number of comments for Show HN posts\n",
    "total_show_comments = 0\n",
    "\n",
    "for row in show_posts:\n",
    "    comments = int(row[4])\n",
    "    total_show_comments += comments\n",
    "\n",
    "print(f\"Total comments in Show HN: {total_show_comments:,.0f} comments\")\n",
    "\n",
    "# Find average number of comments for Show HN posts\n",
    "avg_show_comments = total_show_comments / len(show_posts)\n",
    "\n",
    "print(f\"Average comments per a Show HN post: {avg_show_comments:,.0f} comments\")"
   ]
  },
  {
   "source": [
    "On average, *Ask HN* posts receive more comments than *Show HN* posts, 14 to 10 posts respectively.\n",
    "\n",
    "### Is there a point in time where post receives more comments?\n",
    "Here we use our ask posts as our sample data set. We will analyze at which hour of day the ask posts receive more comments."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "['9/26/2016 2:53', 7]\n['9/26/2016 1:17', 3]\n['9/25/2016 22:48', 3]\n"
     ]
    }
   ],
   "source": [
    "# Create a new list containing selected columns (created_at and num_comments) for the analysis\n",
    "import datetime as dt\n",
    "\n",
    "result_list = []\n",
    "\n",
    "for row in ask_posts:\n",
    "    created_at = row[-1]\n",
    "    comments = int(row[4])\n",
    "    result = [created_at, comments]\n",
    "    result_list.append(result)\n",
    "\n",
    "explore_data(result_list, 0, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Number of posts created during each hour:\n{'02': 227, '01': 223, '22': 287, '21': 407, '19': 420, '17': 404, '15': 467, '14': 378, '13': 326, '11': 251, '10': 219, '09': 176, '07': 157, '03': 212, '16': 415, '08': 190, '00': 231, '23': 276, '20': 392, '18': 452, '12': 274, '04': 186, '06': 176, '05': 165}\n\nNumber of comments for posts created during each hour:\n{'02': 2996, '01': 2089, '22': 3372, '21': 4500, '19': 3954, '17': 5547, '15': 18525, '14': 4972, '13': 7245, '11': 2797, '10': 3013, '09': 1477, '07': 1585, '03': 2154, '16': 4466, '08': 2362, '00': 2277, '23': 2297, '20': 4462, '18': 4877, '12': 4234, '04': 2360, '06': 1587, '05': 1838}\n"
     ]
    }
   ],
   "source": [
    "# Create 2 dictionaries to keep track of (1) number of posts created during each hour, and (2) number of comments for posts created during each hour.\n",
    "counts_by_hour = {}\n",
    "comments_by_hour = {}\n",
    "\n",
    "for row in result_list:\n",
    "    # number of posts created during each hour\n",
    "    created_dt = row[0]\n",
    "    dt_format = \"%m/%d/%Y %H:%M\"\n",
    "    created_dt = dt.datetime.strptime(created_dt, dt_format)\n",
    "    created_hour = dt.datetime.strftime(created_dt, \"%H\")\n",
    "    counts_by_hour.setdefault(created_hour, 0)\n",
    "    counts_by_hour[created_hour] += 1\n",
    "    # number of comments for posts created during each hour\n",
    "    comments = row[1]\n",
    "    comments_by_hour.setdefault(created_hour, 0)\n",
    "    comments_by_hour[created_hour] += comments\n",
    "\n",
    "check_tot_posts = 0\n",
    "for k, v in counts_by_hour.items():\n",
    "    check_tot_posts += v\n",
    "\n",
    "check_tot_comments = 0\n",
    "for k, v in comments_by_hour.items():\n",
    "    check_tot_comments += v\n",
    "\n",
    "assert check_tot_posts == len(ask_posts)\n",
    "assert check_tot_comments == total_ask_comments\n",
    "\n",
    "print(\"Number of posts created during each hour:\")\n",
    "print(counts_by_hour)\n",
    "print()\n",
    "print(\"Number of comments for posts created during each hour:\")\n",
    "print(comments_by_hour)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Top 5 hours for Ask HN comments (on average):\n15:00 >> 39.67 comments per post\n13:00 >> 22.22 comments per post\n12:00 >> 15.45 comments per post\n10:00 >> 13.76 comments per post\n17:00 >> 13.73 comments per post\n"
     ]
    }
   ],
   "source": [
    "# Calculate average number of comments for posts created during each hour\n",
    "avg_by_hour = []\n",
    "\n",
    "for hour in comments_by_hour:\n",
    "    avg = comments_by_hour[hour] / counts_by_hour[hour]\n",
    "    avg_by_hour.append([hour, avg])\n",
    "\n",
    "# Swap the row to sort by average number of comments\n",
    "swap_avg_by_hour = []\n",
    "\n",
    "for row in range(len(avg_by_hour)):\n",
    "    swap = avg_by_hour[row][1], avg_by_hour[row][0]\n",
    "    swap_avg_by_hour.append(swap)\n",
    "\n",
    "sorted_swap = sorted(swap_avg_by_hour, reverse=True)\n",
    "\n",
    "# Show Top 5 result\n",
    "print(\"Top 5 hours for Ask HN comments (on average):\")\n",
    "for row in sorted_swap[0:5]:\n",
    "    dt_hour = row[1]\n",
    "    hour_format = \"%H\"\n",
    "    dt_hour1 = dt.datetime.strptime(dt_hour, hour_format)\n",
    "    dt_hour2 = dt.datetime.strftime(dt_hour1, \"%H:00\")\n",
    "    print(f\"{dt_hour2} >> {row[0]:,.2f} comments per post\")"
   ]
  },
  {
   "source": [
    "Based on data above, a user should submit an *Ask HN* post during 15.00hrs (EST) to get more comments or answers.\n",
    "\n",
    "## Conclusion\n",
    "This project is to analyze posts which titles begin with Ask HN and Show HN, which one has more comments on average and whether there is a point in time where post receives more comments.\n",
    "\n",
    "Per our analysis, a user should submit an *Ask HN* post during 15.00hrs (EST) to get more comments."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "## Next steps\n",
    "### a. Calculate percentage of posts in each group that receive comments\n",
    "There are 293,119 posts in our data set. However, there are only 80,401 posts with comments. We would like to know further the percentage of posts with comments for both *Ask HN* and *Show HN*."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Total Ask HN : 9,139 posts\n['12578908', 'Ask HN: What TLD do you use for local development?', '', '4', '7', 'Sevrene', '9/26/2016 2:53']\n['12578522', 'Ask HN: How do you pass on your work when you die?', '', '6', '3', 'PascLeRasc', '9/26/2016 1:17']\n['12577870', 'Ask HN: Why join a fund when you can be an angel?', '', '1', '3', 'anthony_james', '9/25/2016 22:48']\n\nTotal Show HN : 10,158 posts\n['12577142', 'Show HN: Jumble  Essays on the go #PaulInYourPocket', 'https://itunes.apple.com/us/app/jumble-find-startup-essay/id1150939197?ls=1&mt=8', '1', '1', 'ryderj', '9/25/2016 20:06']\n['12576813', 'Show HN: Learn Japanese Vocab via multiple choice questions', 'http://japanese.vul.io/', '1', '1', 'soulchild37', '9/25/2016 19:06']\n['12576090', 'Show HN: Markov chain Twitter bot. Trained on comments left on Pornhub', 'https://twitter.com/botsonasty', '3', '1', 'keepingscore', '9/25/2016 16:50']\n\nTotal Other posts : 273,822 posts\n['12578975', 'Saving the Hassle of Shopping', 'https://blog.menswr.com/2016/09/07/whats-new-with-your-style-feed/', '1', '1', 'bdoux', '9/26/2016 3:13']\n['12578822', 'Amazons Algorithms Dont Find You the Best Deals', 'https://www.technologyreview.com/s/602442/amazons-algorithms-dont-find-you-the-best-deals/', '1', '1', 'yarapavan', '9/26/2016 2:26']\n['12578694', 'Emergency dose of epinephrine that does not cost an arm and a leg', 'http://m.imgur.com/gallery/th6Ua', '2', '1', 'dredmorbius', '9/26/2016 1:54']\n\n"
     ]
    }
   ],
   "source": [
    "# Find total number of posts for each group\n",
    "ask_posts_all = []\n",
    "show_posts_all = []\n",
    "other_posts_all = []\n",
    "\n",
    "for row in hn:\n",
    "    title = row[1]\n",
    "    if title.lower().startswith(\"ask hn\"):\n",
    "        ask_posts_all.append(row)\n",
    "    elif title.lower().startswith(\"show hn\"):\n",
    "        show_posts_all.append(row)\n",
    "    else:\n",
    "        other_posts_all.append(row)\n",
    "\n",
    "assert len(hn) == len(ask_posts_all) + len(show_posts_all) + len(other_posts_all)\n",
    "\n",
    "print(f\"Total Ask HN : {len(ask_posts_all):,.0f} posts\")\n",
    "explore_data(ask_posts, 0, 3)\n",
    "print()\n",
    "print(f\"Total Show HN : {len(show_posts_all):,.0f} posts\")\n",
    "explore_data(show_posts, 0, 3)\n",
    "print()\n",
    "print(f\"Total Other posts : {len(other_posts_all):,.0f} posts\")\n",
    "explore_data(other_posts, 0, 3)\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Rate which Ask HN posts received comment(s): 75.62%\nRate which Show HN posts received comment(s): 49.80%\nRate which other posts received comment(s): 24.99%\n"
     ]
    }
   ],
   "source": [
    "# Calculate percentage of posts with comment(s)\n",
    "\n",
    "perc_ask_post = (len(ask_posts) / len(ask_posts_all)) * 100\n",
    "perc_show_post = (len(show_posts) / len(show_posts_all)) * 100\n",
    "perc_other_post = (len(other_posts) / len(other_posts_all)) * 100\n",
    "\n",
    "print(f\"Rate which Ask HN posts received comment(s): {perc_ask_post:,.2f}%\")\n",
    "print(f\"Rate which Show HN posts received comment(s): {perc_show_post:,.2f}%\")\n",
    "print(f\"Rate which other posts received comment(s): {perc_other_post:,.2f}%\")"
   ]
  },
  {
   "source": [
    "The percentages above strengthen our conclusion that an *Ask HN* post should be submitted. 75% of all Ask HN posts received comments, while only 50% of Show HN posts received comments. Surprisingly, only 25% of other posts got comments.\n",
    "\n",
    "### b. Determine whether Ask HN or Show HN receive more points on average"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Hacker News posts with point(s): 293,119 posts\n"
     ]
    }
   ],
   "source": [
    "# Remove posts with zero points\n",
    "hn_points = []\n",
    "\n",
    "for row in hn:\n",
    "    if int(row[3]) != 0:\n",
    "        hn_points.append(row)\n",
    "\n",
    "print(f\"Hacker News posts with point(s): {len(hn_points):,.0f} posts\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Total points in Ask HN: 103,378 points\nAverage points per an Ask HN post: 11 points\n"
     ]
    }
   ],
   "source": [
    "# Find total number of points for Ask HN posts\n",
    "total_ask_points = 0\n",
    "\n",
    "for row in ask_posts_all:\n",
    "    points = int(row[3])\n",
    "    total_ask_points += points\n",
    "\n",
    "print(f\"Total points in Ask HN: {total_ask_points:,.0f} points\")\n",
    "\n",
    "# Find average number of points for Ask HN posts\n",
    "avg_ask_points = total_ask_points / len(ask_posts_all)\n",
    "\n",
    "print(f\"Average points per an Ask HN post: {avg_ask_points:,.0f} points\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Total points in show HN: 150,781 points\nAverage points per an show HN post: 15 points\n"
     ]
    }
   ],
   "source": [
    "# Find total number of points for show HN posts\n",
    "total_show_points = 0\n",
    "\n",
    "for row in show_posts_all:\n",
    "    points = int(row[3])\n",
    "    total_show_points += points\n",
    "\n",
    "print(f\"Total points in show HN: {total_show_points:,.0f} points\")\n",
    "\n",
    "# Find average number of points for show HN posts\n",
    "avg_show_points = total_show_points / len(show_posts_all)\n",
    "\n",
    "print(f\"Average points per an show HN post: {avg_show_points:,.0f} points\")"
   ]
  },
  {
   "source": [
    "On average, *Show HN* posts receive more points than *Ask HN* posts.\n",
    "\n",
    "In conclusion, if a user would like to have more comments, an *Ask HN* post should be created. If a user would like to have more votes, a *Show HN* post should be created."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "### c. Determine at which time where posts receive more points"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Ask posts:\n['9/26/2016 2:53', 4]\n['9/26/2016 1:17', 6]\n['9/25/2016 22:57', 1]\n\nShow posts:\n['9/26/2016 0:36', 2]\n['9/26/2016 0:01', 1]\n['9/25/2016 23:44', 1]\n"
     ]
    }
   ],
   "source": [
    "# Create a new list containing selected columns (created_at and num_points) for the analysis\n",
    "import datetime as dt\n",
    "\n",
    "# Ask posts\n",
    "point_ask_list = []\n",
    "\n",
    "for row in ask_posts_all:\n",
    "    created_at = row[-1]\n",
    "    points = int(row[3])\n",
    "    result = [created_at, points]\n",
    "    point_ask_list.append(result)\n",
    "\n",
    "print(\"Ask posts:\")\n",
    "explore_data(point_ask_list, 0, 3)\n",
    "print()\n",
    "\n",
    "# Show posts\n",
    "point_show_list = []\n",
    "\n",
    "for row in show_posts_all:\n",
    "    created_at = row[-1]\n",
    "    points = int(row[3])\n",
    "    result = [created_at, points]\n",
    "    point_show_list.append(result)\n",
    "\n",
    "print(\"Show posts:\")\n",
    "explore_data(point_show_list, 0, 3)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Number of ask posts created during each hour:\n{'02': 269, '01': 282, '22': 383, '21': 518, '19': 552, '17': 587, '15': 646, '14': 513, '13': 444, '11': 312, '10': 282, '09': 222, '07': 226, '03': 271, '23': 343, '20': 510, '16': 579, '08': 257, '00': 301, '18': 614, '12': 342, '04': 243, '06': 234, '05': 209}\n\nNumber of points for ask posts created during each hour:\n{'02': 2944, '01': 2662, '22': 3601, '21': 5042, '19': 4782, '17': 7155, '15': 13978, '14': 5390, '13': 7962, '11': 2856, '10': 3789, '09': 1763, '07': 2040, '03': 2539, '23': 2616, '20': 4491, '16': 5970, '08': 2744, '00': 2835, '18': 6850, '12': 4643, '04': 2650, '06': 2030, '05': 2046}\n"
     ]
    }
   ],
   "source": [
    "# Ask posts\n",
    "\n",
    "# Create 2 dictionaries to keep track of (1) number of posts created during each hour, and (2) number of points for posts created during each hour.\n",
    "counts_ask_by_hour = {}\n",
    "points_ask_by_hour = {}\n",
    "\n",
    "for row in point_ask_list:\n",
    "    # number of posts created during each hour\n",
    "    created_dt = row[0]\n",
    "    dt_format = \"%m/%d/%Y %H:%M\"\n",
    "    created_dt = dt.datetime.strptime(created_dt, dt_format)\n",
    "    created_hour = dt.datetime.strftime(created_dt, \"%H\")\n",
    "    counts_ask_by_hour.setdefault(created_hour, 0)\n",
    "    counts_ask_by_hour[created_hour] += 1\n",
    "    # number of points for posts created during each hour\n",
    "    points = row[1]\n",
    "    points_ask_by_hour.setdefault(created_hour, 0)\n",
    "    points_ask_by_hour[created_hour] += points\n",
    "\n",
    "check_tot_posts = 0\n",
    "for k, v in counts_ask_by_hour.items():\n",
    "    check_tot_posts += v\n",
    "\n",
    "check_tot_points = 0\n",
    "for k, v in points_ask_by_hour.items():\n",
    "    check_tot_points += v\n",
    "\n",
    "assert check_tot_posts == len(ask_posts_all)\n",
    "assert check_tot_points == total_ask_points\n",
    "\n",
    "print(\"Number of ask posts created during each hour:\")\n",
    "print(counts_ask_by_hour)\n",
    "print()\n",
    "print(\"Number of points for ask posts created during each hour:\")\n",
    "print(points_ask_by_hour)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Number of show posts created during each hour:\n{'00': 276, '23': 319, '20': 525, '19': 556, '18': 656, '16': 801, '14': 696, '10': 323, '09': 302, '08': 316, '06': 192, '03': 206, '21': 430, '17': 761, '15': 836, '11': 402, '07': 236, '04': 194, '13': 610, '12': 516, '01': 247, '22': 377, '02': 209, '05': 172}\n\nNumber of points for show posts created during each hour:\n{'00': 4291, '23': 5060, '20': 6948, '19': 8928, '18': 9935, '16': 11487, '14': 10503, '10': 4303, '09': 3762, '08': 4640, '06': 3071, '03': 2168, '21': 5990, '17': 10563, '15': 11657, '11': 7742, '07': 3303, '04': 2707, '13': 10381, '12': 10787, '01': 2931, '22': 5026, '02': 2764, '05': 1834}\n"
     ]
    }
   ],
   "source": [
    "# Show posts\n",
    "\n",
    "# Create 2 dictionaries to keep track of (1) number of posts created during each hour, and (2) number of points for posts created during each hour.\n",
    "counts_show_by_hour = {}\n",
    "points_show_by_hour = {}\n",
    "\n",
    "for row in point_show_list:\n",
    "    # number of posts created during each hour\n",
    "    created_dt = row[0]\n",
    "    dt_format = \"%m/%d/%Y %H:%M\"\n",
    "    created_dt = dt.datetime.strptime(created_dt, dt_format)\n",
    "    created_hour = dt.datetime.strftime(created_dt, \"%H\")\n",
    "    counts_show_by_hour.setdefault(created_hour, 0)\n",
    "    counts_show_by_hour[created_hour] += 1\n",
    "    # number of points for posts created during each hour\n",
    "    points = row[1]\n",
    "    points_show_by_hour.setdefault(created_hour, 0)\n",
    "    points_show_by_hour[created_hour] += points\n",
    "\n",
    "check_tot_posts = 0\n",
    "for k, v in counts_show_by_hour.items():\n",
    "    check_tot_posts += v\n",
    "\n",
    "check_tot_points = 0\n",
    "for k, v in points_show_by_hour.items():\n",
    "    check_tot_points += v\n",
    "\n",
    "assert check_tot_posts == len(show_posts_all)\n",
    "assert check_tot_points == total_show_points\n",
    "\n",
    "print(\"Number of show posts created during each hour:\")\n",
    "print(counts_show_by_hour)\n",
    "print()\n",
    "print(\"Number of points for show posts created during each hour:\")\n",
    "print(points_show_by_hour)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Top 5 hours for Ask HN points (on average):\n15:00 >> 21.64 points per post\n13:00 >> 17.93 points per post\n12:00 >> 13.58 points per post\n10:00 >> 13.44 points per post\n17:00 >> 12.19 points per post\n"
     ]
    }
   ],
   "source": [
    "# Ask posts\n",
    "\n",
    "# Calculate average number of points for posts created during each hour\n",
    "avg_ask_by_hour = []\n",
    "\n",
    "for hour in points_ask_by_hour:\n",
    "    avg = points_ask_by_hour[hour] / counts_ask_by_hour[hour]\n",
    "    avg_ask_by_hour.append([hour, avg])\n",
    "\n",
    "# Swap the row to sort by average number of points\n",
    "swap_avg_ask_by_hour = []\n",
    "\n",
    "for row in range(len(avg_ask_by_hour)):\n",
    "    swap = avg_ask_by_hour[row][1], avg_ask_by_hour[row][0]\n",
    "    swap_avg_ask_by_hour.append(swap)\n",
    "\n",
    "sorted_swap_ask = sorted(swap_avg_ask_by_hour, reverse=True)\n",
    "\n",
    "# Show Top 5 result\n",
    "print(\"Top 5 hours for Ask HN comments (on average):\")\n",
    "for row in sorted_swap_ask[0:5]:\n",
    "    dt_hour = row[1]\n",
    "    hour_format = \"%H\"\n",
    "    dt_hour1 = dt.datetime.strptime(dt_hour, hour_format)\n",
    "    dt_hour2 = dt.datetime.strftime(dt_hour1, \"%H:00\")\n",
    "    print(f\"{dt_hour2} >> {row[0]:,.2f} points per post\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Top 5 hours for Show HN points (on average):\n12:00 >> 20.91 points per post\n11:00 >> 19.26 points per post\n13:00 >> 17.02 points per post\n19:00 >> 16.06 points per post\n06:00 >> 15.99 points per post\n"
     ]
    }
   ],
   "source": [
    "# Show posts\n",
    "\n",
    "# Calculate average number of points for posts created during each hour\n",
    "avg_show_by_hour = []\n",
    "\n",
    "for hour in points_show_by_hour:\n",
    "    avg = points_show_by_hour[hour] / counts_show_by_hour[hour]\n",
    "    avg_show_by_hour.append([hour, avg])\n",
    "\n",
    "# Swap the row to sort by average number of points\n",
    "swap_avg_show_by_hour = []\n",
    "\n",
    "for row in range(len(avg_show_by_hour)):\n",
    "    swap = avg_show_by_hour[row][1], avg_show_by_hour[row][0]\n",
    "    swap_avg_show_by_hour.append(swap)\n",
    "\n",
    "sorted_swap_show = sorted(swap_avg_show_by_hour, reverse=True)\n",
    "\n",
    "# Show Top 5 result\n",
    "print(\"Top 5 hours for Show HN points (on average):\")\n",
    "for row in sorted_swap_show[0:5]:\n",
    "    dt_hour = row[1]\n",
    "    hour_format = \"%H\"\n",
    "    dt_hour1 = dt.datetime.strptime(dt_hour, hour_format)\n",
    "    dt_hour2 = dt.datetime.strftime(dt_hour1, \"%H:00\")\n",
    "    print(f\"{dt_hour2} >> {row[0]:,.2f} points per post\")"
   ]
  },
  {
   "source": [
    "Per our results above, *Ask HN* posts created during 15:00hrs (EST) received more points on average. This is aligned with our result for comments. On the other hand, *Show HN* posts created during 12.00hrs received slightly more points than posts created during 11.00hrs."
   ],
   "cell_type": "markdown",
   "metadata": {}
  }
 ]
}