Text_Pre_Processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text_Pre_Processing\n",
    "#### Author: Armin Berger\n",
    "\n",
    "Date: 29/08/2020\n",
    "\n",
    "Version: 1.0\n",
    "\n",
    "Environment: Python 2.7.11 and Jupyter notebook\n",
    "\n",
    "Libraries used:\n",
    "\n",
    "* landid      (to check for the langauge of a tweet, included in Anaconda Python 2.7) \n",
    "* re          (for regular expression, included in Anaconda Python 2.7) \n",
    "* numpy       (for numpy array, included in Anaconda Python 2.7)\n",
    "* pandas      (for dataframe, included in Anaconda Python 2.7) \n",
    "* itertools   (for combining lists, included in Anaconda Python 2.7)\n",
    "* nltk        (for tokenizing and stemming, included in Anaconda Python 2.7)\n",
    "* sklearn     (for sparse matrix, included in Anaconda Python 2.7)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:blue\"> 2.0 General Steps </span>\n",
    "\n",
    "In order to achieve the required outcomes of task 2 we first need to take several general steps that will help us in all sub-task of task2.\n",
    "\n",
    "Firstly, we will import all required libraries.\n",
    "\n",
    "Secondly, we will parse all of the the xlsx tweet file data and save its tokens in a dict.\n",
    "\n",
    "Thirdly, we will get all the context dependent stop words( appear more than 60 days) and all the rare words (appear less than 5 days)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Firstly, we will import a multitude of libraries to help us with this task\n",
    "\n",
    "import re # used for regular expression\n",
    "\n",
    "import langid # used to check whether a tweet is in english \n",
    "\n",
    "# character encoding standard\n",
    "import unicodedata\n",
    "\n",
    "# nltk used for parsing and cleaning text\n",
    "import nltk\n",
    "from nltk.stem import PorterStemmer # porter stemmer = applies 5 rules\n",
    "from nltk.tokenize import word_tokenize, sent_tokenize\n",
    "from nltk.probability import FreqDist\n",
    "from nltk.tokenize import RegexpTokenizer\n",
    "from nltk.util import ngrams\n",
    "\n",
    "# for vectorisation\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# obtaining ounctuations\n",
    "import string\n",
    "\n",
    "# for data frame manipulation \n",
    "import pandas as pd\n",
    "\n",
    "# for combining lists into one singualr list\n",
    "from itertools import chain\n",
    "\n",
    "import collections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Parse the xlsx File and get all tokens per sheet \n",
    "\n",
    "In this step we parse all the information of the provided Excel file which contains the tweets. \n",
    "\n",
    "One of the main obstacles while reading in the tweet data from the xlsx file was the varying postion of the sheets headers as well as the lack of headers for some sheets. Thus, I dedcided to drop all empty rows or columns as well as assign a header row to sheets missing headers, so that each dataframe had a header row in the same postition.\n",
    "\n",
    "After a number of wrangling steps and tokenization of each english tweet we save the retreived tokens per sheet and date information in a dict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "## GENERAL TOOLS: defining global variables/methods that get used throughout the code\n",
    "\n",
    "# read in provided conetext independent stop words from text file\n",
    "stop_words = open('stopwords_en.txt','r').readlines()\n",
    "\n",
    "# remove \\n from each stopword and save as a set\n",
    "stop_words = set([x.replace(\"\\n\",\"\") for x in stop_words])\n",
    "\n",
    "# set the tokenizer to the desired regular expression\n",
    "tokenizer = RegexpTokenizer(r\"[a-zA-Z]+(?:[-'][a-zA-Z]+)?\")\n",
    "\n",
    "# calling the PorterStemmer function\n",
    "porter = PorterStemmer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "## NOTE: this code block runs for a while but creates the desired output (please don't restart it, just let it run)\n",
    "\n",
    "## READ IN TWEET DATA AND TOKENIZE \n",
    "\n",
    "# create a dict that saves all tokens per sheet as a list \n",
    "all_tweets_dict = {}\n",
    "\n",
    "# read in excel with mutiple sheets \n",
    "tweet_data_frames = pd.ExcelFile('26255367.xlsx')\n",
    "\n",
    "# save the title of each sheet in a list\n",
    "list_sheet_titles = tweet_data_frames.sheet_names\n",
    "\n",
    "# iterate through all the sheets in the dataframe \n",
    "for item in list_sheet_titles:\n",
    "    \n",
    "    # save each read in sheet as a data frame\n",
    "    # parse is similar to the read_excel methode and transforms a specified sheet \n",
    "    # into a data frame\n",
    "    sheet_data_frame = tweet_data_frames.parse(sheet_name=item)\n",
    "    \n",
    "    # drop all empty columns\n",
    "    sheet_data_frame = sheet_data_frame.dropna(axis='columns', how='all')\n",
    "    \n",
    "    # drop all empty rows\n",
    "    sheet_data_frame = sheet_data_frame.dropna(axis=0, how='all')\n",
    "    \n",
    "    # turn the frist row into the header of each column\n",
    "    # adapted from user ostrokach on the 02/09/2020 \n",
    "    # from: https://stackoverflow.com/questions/31328861/python-pandas-replacing-header-with-top-row\n",
    "    \n",
    "    # we search for the column text, first row\n",
    "    if sheet_data_frame.iloc[0][0] == 'text':\n",
    "        sheet_data_frame = sheet_data_frame.rename(columns=sheet_data_frame.iloc[0]).drop(sheet_data_frame.index[0]) \n",
    "    \n",
    "    # reset the index of each data frame\n",
    "    sheet_data_frame = sheet_data_frame.reset_index()\n",
    "    \n",
    "    # after creating a uniform dataframe format for each sheet read in we save the tweet text\n",
    "    # as a list\n",
    "    text_per_sheet = sheet_data_frame['text'].values.tolist()\n",
    "    \n",
    "    # create list that saves all lists of tokens per sheet\n",
    "    list_all_token_per_sheet = []\n",
    "    \n",
    "    # now tokenize the tweet text\n",
    "    for tweet in text_per_sheet:\n",
    "            \n",
    "        # check if a tweet consists of alpahbestic characters \n",
    "        if type(tweet) == str:\n",
    "            tweet_language = langid.classify(tweet)\n",
    "            # check if a tweet is in english using the langid package\n",
    "            if tweet_language[0] == 'en':\n",
    "\n",
    "                # set tweet to lower case\n",
    "                tweet = tweet.lower()\n",
    "\n",
    "                # now we tokenize each sentence with the regular expression provided to us\n",
    "\n",
    "                # we now apply the tokenizer that was used during the week 5 tutorial\n",
    "                # result is a list of lists containg all the tweet word tokens of the read in sheet\n",
    "                unigram_tokens = tokenizer.tokenize(tweet)\n",
    "\n",
    "                # append list of tokens to all tokens list\n",
    "                list_all_token_per_sheet.append(unigram_tokens)\n",
    "\n",
    "    # combine all list of tokens into one list per sheet\n",
    "    list_all_token_per_sheet = list(chain.from_iterable(list_all_token_per_sheet))\n",
    "    \n",
    "    # assign the date as a key and the list of tweet text as a value\n",
    "    all_tweets_dict[item] = list_all_token_per_sheet\n",
    "    \n",
    "    \n",
    "# the final result is a list of dicts that conatain the tweets date and their text\n",
    "# infromation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Generate all rare words and context dependent stop words\n",
    "\n",
    "In this step we seek to find all rare words and context-dependent stop words with a threshold of less than 5 days and more than 60 days."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "## WORD FREQUENCY\n",
    "\n",
    "# firstly, in order try to find both context dependent stop words and rare words we need to get the sheet/document \n",
    "# frequency of each word accross all the sheets \n",
    "\n",
    "# get all the tokens into one list using list comprehsions\n",
    "# adapted logic from tutorial week 5\n",
    "token_document_frequencey = list(chain.from_iterable([set(item) for item in all_tweets_dict.values()]))\n",
    "\n",
    "# get the document/sheet frequency of each token(in how many sheets does a word appear)\n",
    "# using the function FreqDist introduced in tutorial week 5\n",
    "token_document_frequencey = FreqDist(token_document_frequencey)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "## CONTEXT DEPEDENT STOPWORDS\n",
    "\n",
    "# after getting the document frequency of each token we can sub select the too frequent and rare ones\n",
    "\n",
    "# get all context dependent stop words by only selecting the ones with a frequency count larger or equal to 61\n",
    "# list for all context_dependent_stop_words\n",
    "context_dependent_stop_words = []\n",
    "\n",
    "# iterate through the items of frequency dict\n",
    "for keys, values in token_document_frequencey.items():\n",
    "    \n",
    "    # only select words with a frequency larger or equal to 61\n",
    "    if values >= 61:\n",
    "        \n",
    "        # append key as stop word\n",
    "        context_dependent_stop_words.append(key)\n",
    "        \n",
    "# save all context_dependent_stop_words as a set for fast retrival \n",
    "context_dependent_stop_words = set(context_dependent_stop_words)\n",
    "\n",
    "\n",
    "## RARE WORDS\n",
    "\n",
    "# list for rare words\n",
    "rare_words = []\n",
    "\n",
    "# iterate through the items of frequency dict\n",
    "for keys, values in token_document_frequencey.items():\n",
    "    \n",
    "    # check for frequency\n",
    "    if values <= 4:\n",
    "        \n",
    "        # append key as rare word\n",
    "        rare_words.append(keys)\n",
    "        \n",
    "# save all rare_words as a set for fast retrival \n",
    "rare_words = set(rare_words)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:blue\">Task 2.1 - Generate Corpus Vocabulary </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Get all unigrams for the corpus vocabulary\n",
    "\n",
    "Since all of the data within the xlsx file is already read in saved we can directly start using it. To get all unigrams we need to filter out all the undesired words. \n",
    "\n",
    "1. Get a list of all unigrams by chaining together all dictionary values \n",
    "\n",
    "2. Remove all regular stop words\n",
    "\n",
    "3. Remove all rare words\n",
    "\n",
    "4. Remove any token that is in context_dependent_stop_words\n",
    "\n",
    "5. Remove any token with a length smaller than three\n",
    "\n",
    "6. Stemm the remaining tokens using the PorterStemmer\n",
    "\n",
    "7. Append all stemmed tokens to the corpus_vocabulary list\n",
    "\n",
    "8. Only keep unique words by turning it into a set\n",
    "\n",
    "9. Sort the set alphabetically\n",
    "\n",
    "Done, now you have all the unigrams for the corpus vocabulary!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# chain together all unigrams that were stored as dict values\n",
    "all_unigram_tokens = list(chain.from_iterable(all_tweets_dict.values()))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "## FILTER OUT WORDS\n",
    "\n",
    "# list for all tokens in the corpus vocabulary\n",
    "unigram_vocabulary = []\n",
    "\n",
    "# iterate through the list of all tokens\n",
    "for word in all_unigram_tokens:\n",
    "    \n",
    "    # check if word is in regular stopwords \n",
    "    if word not in stop_words:\n",
    "        \n",
    "        # check if word is in rare words\n",
    "        if word not in rare_words:\n",
    "            \n",
    "            # check if word is in context dependent stopwords\n",
    "            if word not in context_dependent_stop_words:\n",
    "                \n",
    "                # check if word is equal or longer than len() = 3 \n",
    "                if len(word) >= 3:\n",
    "                    \n",
    "                    # stem each word using the porter stemmer\n",
    "                    word = porter.stem(word)\n",
    "                    \n",
    "                    # append each word to the corpus_vocabulary list\n",
    "                    unigram_vocabulary.append(word)\n",
    "\n",
    "                        \n",
    "# sort vocabs and turn list into set\n",
    "unigram_vocabulary = sorted(set(unigram_vocabulary))\n",
    "\n",
    "# now we have all the unigrams "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Get all bigrams for the corpus vocabulary\n",
    "\n",
    "Since all of the data within the xlsx file is already read in saved we can directly start using it. To get all unigrams we need to filter out all the undesired words. \n",
    "\n",
    "1. Get a list of all tokens by chaining together all dictionary values \n",
    "\n",
    "2. Remove all regular stop words\n",
    "\n",
    "3. Remove all rare words\n",
    "\n",
    "4. Remove any token that is in context_dependent_stop_words\n",
    "\n",
    "5. Remove any token with a length smaller than three\n",
    "\n",
    "6. Stemm the remaining tokens using the PorterStemmer\n",
    "\n",
    "7. Append all stemmed tokens to the corpus_vocabulary list\n",
    "\n",
    "8. Only keep unique words by turning it into a set\n",
    "\n",
    "9. Sort the set alphabetically\n",
    "\n",
    "Done, now you have all the bigrams for the corpus vocabulary!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get list of all tokens by chaining together all dictinary values\n",
    "all_tokens = list(chain.from_iterable([item for item in all_tweets_dict.values()]))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "## CREATE BIGRAMS\n",
    "\n",
    "# most of this code was adapted from \"tutorial_05_answer\" example that was posted on moodle\n",
    "\n",
    "# set the nltk.collocations methode for later use\n",
    "bigram_measures = nltk.collocations.BigramAssocMeasures()\n",
    "\n",
    "# use the methode nltk.collocations.BigramCollocationFinder.from_words to get all bigrams\n",
    "bigram_finder = nltk.collocations.BigramCollocationFinder.from_words(all_tokens)\n",
    "\n",
    "# use of lambda function to remove all words samller than 3\n",
    "bigram_finder.apply_word_filter(lambda w: len(w) < 3)\n",
    "\n",
    "# get top 200 bigrams using nbest(pmi,2 00)\n",
    "top_200_bigrams = bigram_finder.nbest(bigram_measures.pmi, 200)\n",
    "\n",
    "# new list for top 200 concatinated bigrams\n",
    "top_200_bigrams_connected = []\n",
    "\n",
    "# iterate through all bigrams and connect them with '_'\n",
    "for item in top_200_bigrams:\n",
    "    \n",
    "    # append all connected bigrams \n",
    "    top_200_bigrams_connected.append(item[0] + '_' + item[1])\n",
    "    \n",
    "# Now we have the top 200 bigrams "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.  Combine unigrams and bigrams into one list and sort "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# new list for both unigrams and bigrams of corpus vocabulary \n",
    "corpus_vocabulary = list(unigram_vocabulary) + list(top_200_bigrams_connected)\n",
    "\n",
    "# sort the new list\n",
    "corpus_vocabulary = sorted(corpus_vocabulary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Write unigrams and bigrams to txt file\n",
    "\n",
    "In the final step we write all unigrams and bigrams to a txt file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we create a txt file and set it to write mode (w)\n",
    "token_file = open('<26255367>_vocab.txt','w')\n",
    "\n",
    "# iterate through all the vocabs\n",
    "for i in range(len(corpus_vocabulary)):\n",
    "        \n",
    "    # combine both the vocab and its id\n",
    "    info = f\"{corpus_vocabulary[i]}:{i}\"\n",
    "    \n",
    "    # write to file\n",
    "    token_file.write(info)\n",
    "    \n",
    "    # go to the next file line\n",
    "    token_file.write(\"\\n\")\n",
    "\n",
    "# we close the xml file after writing to it\n",
    "token_file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:blue\">Task 2.2 - Generate the Sparse Representation of the Corpus Vocabulary</span>\n",
    "\n",
    "After getting the corpus vocabulary we can create a count vector which is used for further text analysis. To create the sparse representation we need to follow the following set of steps:\n",
    "\n",
    "1. Create a dict called corpus_vocab_dict_id that contains all tokens of the corpus vocabulary as keys and id as values\n",
    "\n",
    "2. Create a dict called sparse_vocab_dict in which each key-value pair represents each day/sheet. Only stem each token and only append as a value if they are in the corpus vocabulary \n",
    "\n",
    "3. Create a text file to which we write the sparse representation\n",
    "\n",
    "4. Assign each token in sparse_vocab_dict with their matching id number from corpus_vocab_dict_id\n",
    "\n",
    "5. Write the date of the iterated sheet to the text file\n",
    "\n",
    "6. Get frequency of each token in tokens_per_sheet_id\n",
    "\n",
    "7. Go the next file at the end of the sheet and close the file\n",
    "\n",
    "Now you have the sparse representation of the corpus vocabulary written to a text file!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dict to save all the vocabs and their id in \n",
    "corpus_vocab_dict_id = {}\n",
    "\n",
    "# iterate through the list of vocabs\n",
    "for i in range(len(corpus_vocabulary)):\n",
    "    \n",
    "    # add each vocab and its id to the dict\n",
    "    corpus_vocab_dict_id[corpus_vocabulary[i]] = i\n",
    "\n",
    "    \n",
    "# create a dict for each day/sheet which only contains all tweet tokens that are part\n",
    "# of the corpus vocabulary\n",
    "sparse_vocab_dict = {}\n",
    "\n",
    "# loop through the dict containing all tokens \n",
    "for key,value in all_tweets_dict.items():\n",
    "    \n",
    "    # create a list to save all the tokens in\n",
    "    vocab_token_list = []\n",
    "    \n",
    "    # loop through each token\n",
    "    for item in value:\n",
    "        \n",
    "        # stem token\n",
    "        word = porter.stem(item)\n",
    "        \n",
    "        # check if token is part of the vocabulary\n",
    "        if word in unigram_vocabulary:\n",
    "            \n",
    "            # append all tokens that are part of the vocabulary\n",
    "            vocab_token_list.append(word)\n",
    "    \n",
    "    # add the date as key with the list of tokens as a value to sparse_vocab_dict\n",
    "    sparse_vocab_dict[key] = vocab_token_list\n",
    "            \n",
    "\n",
    "## now we can create the actual sparse representation and write it to a text file \n",
    "\n",
    "# create a text file to write to\n",
    "sparse_rep = open('<26255367>_countVec.txt', 'w')\n",
    "\n",
    "for key, value in sparse_vocab_dict.items():\n",
    "    \n",
    "    # use list comprehension to loop through all tokens in value and assign them their\n",
    "    # matching number \n",
    "    tokens_per_sheet_id = [corpus_vocab_dict_id[x] for x in value]\n",
    "    \n",
    "    # write date to the file \n",
    "    sparse_rep.write(key)\n",
    "    \n",
    "    # CHECK AGAIN\n",
    "    for keys, vals in FreqDist(tokens_per_sheet_id).items():\n",
    "        sparse_rep.write(f',{keys}:{vals}')\n",
    "    \n",
    "    # after each sheet/day is done go to the next line like shown in the sample output\n",
    "    sparse_rep.write('\\n')\n",
    "    \n",
    "    \n",
    "# close the text file\n",
    "sparse_rep.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:blue\">Task 2.3 - Find Top 100 Unigrams each day</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Parse the xlsx File\n",
    "\n",
    "Like in the previous task we need to read in the xlsx file and each of the sheets.\n",
    "Since we have done that before we will be reusing the read in data from earlier saved in the dict: all_tweets_dict.\n",
    "\n",
    "In this case, it is important to save the data from each sheet separately since we searching the top 100 unigrams and bigrams of each day (each sheet)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Create unigrams for each sheet\n",
    "\n",
    "In this step, we will extract all text data from each sheet (representing one day of tweets) and then process the text to extract the top 100 unigrams. To do that we need to follow the following steps:\n",
    "\n",
    "1. Iterate through the dict containing all tokens by date/sheet.\n",
    "   For each sheet then:\n",
    "\n",
    "2. Iterate through all tokens per sheet\n",
    "\n",
    "3. Remove all regular stop words\n",
    "\n",
    "4. Remove all rare words\n",
    "\n",
    "5. Remove any token that is in context_dependent_stop_words\n",
    "\n",
    "6. Remove any token with a length smaller than three\n",
    "\n",
    "7. Stemm the remaining tokens using the PorterStemmer\n",
    "\n",
    "8. Append all stemmed tokens to the all_tokens_each_day list\n",
    "\n",
    "9. Only keep unique words by turning it into a set\n",
    "\n",
    "10. Get frequency of words using FreqDist \n",
    "\n",
    "11. Save word and frequency in the list of tuples\n",
    "\n",
    "12. Sort the list based on word frequency and save in a dict call all_unigrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dict that saves the top 100 unigrams of all excel sheets \n",
    "all_unigrams = {}\n",
    "\n",
    "# iterate through the \n",
    "for key,values in all_tweets_dict.items():\n",
    "    \n",
    "    all_tokens_each_day = []\n",
    "    list_of_tup = []\n",
    "    \n",
    "    # loop through all tokens per sheet\n",
    "    for word in values:\n",
    "        \n",
    "        # check if word is in regular stopwords \n",
    "        if word not in stop_words:\n",
    "\n",
    "            # check if word is in rare words\n",
    "            if word not in rare_words:\n",
    "\n",
    "                # check if word is in context dependent stopwords\n",
    "                if word not in context_dependent_stop_words:\n",
    "\n",
    "                    # check if word is equal or longer than len() = 3 \n",
    "                    if len(word) >= 3:\n",
    "\n",
    "                        # stem each word using the porter stemmer\n",
    "                        word = porter.stem(word)\n",
    "                        #print(word)\n",
    "\n",
    "                        # append each word to the top_100_tokens_each_day list\n",
    "                        all_tokens_each_day.append(word)\n",
    "    \n",
    "    # count frequency of each word per sheet              \n",
    "    top_100_tokens_each_day = FreqDist(all_tokens_each_day)\n",
    "\n",
    "    # iterate through the items of frequency dict\n",
    "    for keys, values in top_100_tokens_each_day.items():\n",
    "        list_of_tup.append((keys,values))\n",
    "    \n",
    "    # use lambda function to sort the list of tuples by the frequencey of each token\n",
    "    # used the logic provided by user: Sven Marnach\n",
    "    # from: https://stackoverflow.com/questions/8459231/sort-tuples-based-on-second-parameter\n",
    "    # accessed on: 11/09/2020\n",
    "    list_of_tup = sorted(list_of_tup,key=lambda x: x[1], reverse = True) \n",
    "    \n",
    "    # append key and value to dict \n",
    "    all_unigrams[key] = list_of_tup[:100]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Write top 100 unigrams  each day + frequency to txt file\n",
    "\n",
    "In the final step, we write the top 100 unigrams of each day and their respective frequency to a txt file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we create a txt file and set it to write mode (w)\n",
    "unigram_file = open('<26255367>_100uni.txt','w')\n",
    "\n",
    "# iterate through dict containg all unigrams and their counts\n",
    "for key,value in all_unigrams.items():\n",
    "        \n",
    "        # write date to the file\n",
    "        unigram_file.write(f\"{key}:\")\n",
    "        \n",
    "        # write list of top 100 unigrams of each sheet to the file\n",
    "        unigram_file.write(str(value))\n",
    "        \n",
    "        # do to next line\n",
    "        unigram_file.write(\"\\n\")\n",
    "\n",
    "# we close the text file after writing to it\n",
    "unigram_file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:blue\">Task 2.4 - Find Top 100 Bigrams each day</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A bigram is a combination of two words that commonly appear together(like \"monash\",\"university\"). To collect the top 100 bigrams of each day/sheet we need to follow the following set of steps:\n",
    "\n",
    "1. Iterate through the dict containing all tokens by date/sheet.\n",
    "   For each sheet then:\n",
    "\n",
    "2. Iterate through all tokens per sheet\n",
    "\n",
    "3. Remove all regular stop words\n",
    "\n",
    "4. Remove all rare words\n",
    "\n",
    "5. Remove any token that is in context_dependent_stop_words\n",
    "\n",
    "6. Remove any token with a length smaller than three\n",
    "\n",
    "7. Stemm the remaining tokens using the PorterStemmer\n",
    "\n",
    "8. Append all stemmed tokens to the all_tokens_each_day list\n",
    "\n",
    "9. Only keep unique words by turning the list into a set\n",
    "\n",
    "10. Get frequency of words using FreqDist \n",
    "\n",
    "11. Save word and frequency in a list of tuples\n",
    "\n",
    "12. Sort the list based on word frequency and save in a dict call all_bigrams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Generate top 100 bigrams each day"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dict that saves the top 100 bigrams of all excel sheets \n",
    "all_bigrams = {}\n",
    "\n",
    "# iterate through the values in each dict\n",
    "for key,values in all_tweets_dict.items():\n",
    "    \n",
    "    # create list to hold bigrams and their frequency\n",
    "    list_of_tup_bi = []\n",
    "    \n",
    "    # in this step we get all possible bigrams for each day/sheet using ngrams() and then\n",
    "    # count their frequency using FreqDist function\n",
    "    bigrams_frequency = FreqDist(ngrams(values, n=2))\n",
    "\n",
    "\n",
    "    # iterate through the items of frequency dict\n",
    "    for keys, values in bigrams_frequency.items():\n",
    "        list_of_tup_bi.append((keys,values))\n",
    "    \n",
    "    # use lambda function to sort the list of tuples by the frequencey of each token\n",
    "    # used the logic provided by user: Sven Marnach\n",
    "    # from: https://stackoverflow.com/questions/8459231/sort-tuples-based-on-second-parameter\n",
    "    # accessed on: 11/09/2020\n",
    "    list_of_tup_bi = sorted(list_of_tup_bi ,key=lambda x: x[1], reverse = True) \n",
    "    \n",
    "    # append key and value to dict\n",
    "    if len(list_of_tup_bi) > 100:\n",
    "        all_bigrams[key] = list_of_tup_bi[:100]\n",
    "    else:\n",
    "        all_bigrams[key] = list_of_tup_bi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Write top 100 bigrams  each day and their frequency to txt file\n",
    "\n",
    "In the final step we write the top 100 bigrams and their respective frequencey to a txt file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we create a txt file and set it to write mode (w)\n",
    "bigram_file = open('<26255367>_100bi.txt','w')\n",
    "\n",
    "# iterate through dict containg all bigrams and their counts\n",
    "for key,value in all_bigrams.items():\n",
    "        \n",
    "        # write date to the file\n",
    "        bigram_file.write(f\"{key}:\")\n",
    "        \n",
    "        # write list of top 100 bigrams each sheet to file\n",
    "        bigram_file.write(str(value))\n",
    "        \n",
    "        # go to next line\n",
    "        bigram_file.write(\"\\n\")\n",
    "        \n",
    "\n",
    "# we close the text file after writing to it\n",
    "bigram_file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Summary\n",
    "Give a short summary of your work done above, such as your findings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main difficulty of task 2 was to \"re-engineer\" the text processing steps taken to arrive at the sample output. Since the order of steps was not given and in some cases steps were skipped or executed in a different order it was hard to match the sample output.\n",
    "\n",
    "Thus, to get closer to the desired output I had to read in the sample output files and analyze them to see patterns within the output. Such patterns included whether words seemed to be stemmed, whether they were shorter than len(3), or whether they contained stopword or rare words. After finding such patterns in the different output files, one could figure out which steps were taken and similarly process the input files.\n",
    "\n",
    "Since the tutorials worked on similar tasks, I reused the logic of the code in many areas. Whenever I reused code logic from the tutorials I noted it right above the code.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}