Week6_lecture.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture notes from week 6\n",
    "Programming for the Behavioral Sciences\n",
    "\n",
    "Text and string processing. The goal of this lecture is count the words in parts on Hamlet. What is the most frequently used work in this book, for instance?\n",
    "\n",
    "More information about basic string and text operations can be found here:\n",
    "\n",
    "http://www.pythonforbeginners.com/basics/string-manipulation-in-python\n",
    "\n",
    "https://www.tutorialspoint.com/python/python_strings.htm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic string operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['This', 'is', 'my', 'string']\n",
      "This is my string\n",
      "11\n",
      "This is my thing\n",
      "This is my string contains 17.00 characters\n"
     ]
    }
   ],
   "source": [
    "# Create a string\n",
    "my_str = 'This is my string'\n",
    "\n",
    "# Split it...\n",
    "my_str_split = my_str.split()\n",
    "print(my_str_split)\n",
    "\n",
    "# ... and restore (join) it again\n",
    "my_str_joined = ' '.join(my_str_split) # ' ' means join with space\n",
    "print(my_str_joined)\n",
    "\n",
    "# Find first occurence of word containing 'str'\n",
    "print(my_str.find('str')) \n",
    "\n",
    "# Replace all occurences of 'str' with 'th'\n",
    "print(my_str.replace('str','th'))\n",
    "\n",
    "# Print the length of the string. %.2f means that the values should have two decimals (a float).\n",
    "print(my_str + ' contains %.2f characters' % len(my_str))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Count words in Hamlet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1083, 'the'], [939, 'and'], [727, 'to'], [670, 'of'], [540, 'a'], [523, 'i'], [519, 'my'], [433, 'you'], [420, 'in'], [358, 'ham.']]\n"
     ]
    }
   ],
   "source": [
    "filename = 'img\\hamlet.txt'\n",
    "\n",
    "# Open file for reading\n",
    "f = open(filename,'r')\n",
    "\n",
    "# Create an empty dictionary\n",
    "worddict = {}\n",
    "\n",
    "# Loop over each line in file\n",
    "for line in f:\n",
    "    # Loop over each word in line\n",
    "    words = line.split()\n",
    "    for i, word in enumerate(words):   \n",
    "        # Make all words lowercase (so that The and the are counted as the same, for instance)\n",
    "        w = word.lower()\n",
    "        if w in worddict:\n",
    "            worddict[w] +=1\n",
    "        else:\n",
    "            worddict[w] = 1\n",
    "\n",
    "# Close the file for reading and writing\n",
    "f.close()  \n",
    "\n",
    "# Create a list from the dictionary\n",
    "wordlist = []\n",
    "for key, value in worddict.iteritems():\n",
    "    wordlist.append([value, key])\n",
    "    \n",
    "# Sort worddict such that the most common words are on the top   \n",
    "wordlist.sort(reverse=True)  \n",
    "\n",
    "# Print the 10 most common words\n",
    "print(wordlist[:10])\n",
    "\n",
    "# Write output to file       \n",
    "filename_out = 'my_hamlet_frequencies.txt'        \n",
    "f_out = open(filename_out,'w')\n",
    "for w in wordlist:\n",
    "    f_out.write('\\t'.join((w[1],str(w[0]),'\\n')))\n",
    "f_out.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use collections to do the same thing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('the', 1091),\n",
       " ('and', 969),\n",
       " ('to', 767),\n",
       " ('of', 675),\n",
       " ('i', 633),\n",
       " ('a', 571),\n",
       " ('you', 558),\n",
       " ('my', 520),\n",
       " ('in', 451),\n",
       " ('it', 421)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "from collections import Counter\n",
    "\n",
    "# Read all the words into a list\n",
    "words = re.findall(r'\\w+', open('img/hamlet.txt').read().lower())\n",
    "\n",
    "# Present the 10 \n",
    "Counter(words).most_common(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare the results using our own implementation and Counter. Why do they give slightly different results?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}