Slides

Assignments/Assignment_10.2

@@ -0,0 +1,18 @@
+Hello Fellows,
+
+Write a program to read through the mbox-short.txt and figure out the distribution by hour of the day for each of the messages. 
+You can pull the hour out from the 'From ' line by finding the time and then splitting the string a second time using a colon.
+
+From stephen.marquard@uct.ac.za Sat Jan  5 09:14:16 2008
+
+Once you have accumulated the counts for each hour, print out the counts, sorted by hour as shown below.
+
+
+Begin to write the program with the following code below:
+name = raw_input("Enter file:")
+if len(name) < 1 : name = "mbox-short.txt"
+handle = open(name)
+
+
+Happy Coding,
+Tunisia

Assignments/Assignment_11.py

@@ -0,0 +1,34 @@
+Finding Numbers in a Haystack
+In this assignment you will read through and parse a file with text and numbers. You will extract all the numbers in the file and compute the sum of the numbers.
+Data Files
+We provide two files for this assignment. One is a sample file where we give you the sum for your testing and the other is the actual data you need to process for the assignment.
+Sample data: http://python-data.dr-chuck.net/regex_sum_42.txt (There are 87 values with a sum=445822)
+Actual data: http://python-data.dr-chuck.net/regex_sum_242232.txt (There are 101 values and the sum ends with 786)
+
+These links open in a new window. Make sure to save the file into the same folder as you will be writing your Python program. Note: Each student will have a distinct data file for the assignment - so only use your own data file for analysis.
+Data Format
+The file contains much of the text from the introduction of the textbook except that random numbers are inserted throughout the text. Here is a sample of the output you might see:
+Why should you learn to write programs? 7746
+12 1929 8827
+Writing programs (or programming) is a very creative 
+7 and rewarding activity.  You can write programs for 
+many reasons, ranging from making your living to solving
+8837 a difficult data analysis problem to having fun to helping 128
+someone else solve a problem.  This book assumes that 
+everyone needs to know how to program ...
+
+
+The sum for the sample text above is 27486. The numbers can appear anywhere in the line. There can be any number of numbers in each line (including none).
+Handling The Data
+The basic outline of this problem is to read the file, look for integers using the re.findall(), looking for a regular expression of '[0-9]+' and then converting the extracted strings to integers and summing up the integers.
+Turn in Assignent
+Enter the sum from the actual data and your Python code below:
+Sum:  (ends with 786) 
+Python code:
+
+Optional: Just for Fun
+There are a number of different ways to approach this problem. While we don't recommend trying to write the most compact code possible, it can sometimes be a fun exercise. Here is a a redacted version of two-line version of this program using list comprehension:
+import re
+print sum( [ ****** *** * in **********('[0-9]+',**************************.read()) ] )
+
+

Assignments/Assignment_12.py

@@ -0,0 +1,18 @@
+'''Exploring the HyperText Transport Protocol
+You are to retrieve the following document using the HTTP protocol in a way that you can examine the HTTP Response headers.
+http://www.pythonlearn.com/code/intro-short.txt
+There are three ways that you might retrieve this web page and look at the response headers:
+Preferred: Modify the socket1.py program to retrieve the above URL and print out the headers and data.
+Open the URL in a web browser with a developer console or FireBug and manually examine the headers that are returned.
+Use the telnet program as shown in lecture to retrieve the headers and content.
+Enter the header values in each of the fields below and press "Submit".
+Last-Modified:
+
+ETag:
+
+Content-Length:
+
+Cache-Control:
+
+Content-Type:
+'''

Assignments/Assignment_13.1

@@ -0,0 +1,37 @@
+Scraping Numbers from HTML using BeautifulSoup In this assignment you will write a Python program similar to http://www.pythonlearn.com/code/urllink2.py. The program will use urllib to read the HTML from the data files below, and parse the data, extracting numbers and compute the sum of the numbers in the file.
+We provide two files for this assignment. One is a sample file where we give you the sum for your testing and the other is the actual data you need to process for the assignment.
+Sample data: http://python-data.dr-chuck.net/comments_42.html (Sum=2553)
+Actual data: http://python-data.dr-chuck.net/comments_242237.html (Sum ends with 67)
+
+You do not need to save these files to your folder since your program will read the data directly from the URL. Note: Each student will have a distinct data url for the assignment - so only use your own data url for analysis.
+Data Format
+The file is a table of names and comment counts. You can ignore most of the data in the file except for lines like the following:
+<tr><td>Modu</td><td><span class="comments">90</span></td></tr>
+<tr><td>Kenzie</td><td><span class="comments">88</span></td></tr>
+<tr><td>Hubert</td><td><span class="comments">87</span></td></tr>
+
+
+You are to find all the <span> tags in the file and pull out the numbers from the tag and sum the numbers.
+Look at the sample code provided. It shows how to find all of a certain kind of tag, loop through the tags and extract the various aspects of the tags.
+...
+# Retrieve all of the anchor tags
+tags = soup('a')
+for tag in tags:
+   # Look at the parts of a tag
+   print 'TAG:',tag
+   print 'URL:',tag.get('href', None)
+   print 'Contents:',tag.contents[0]
+   print 'Attrs:',tag.attrs
+
+
+You need to adjust this code to look for span tags and pull out the text content of the span tag, convert them to integers and add them up to complete the assignment.
+Sample Execution
+$ python solution.py 
+Enter - http://python-data.dr-chuck.net/comments_42.html
+Count 50
+Sum 2482
+
+
+Turning in the Assignment
+Enter the sum from the actual data and your Python code below:
+Sum:  (ends with 67) 

Assignments/Assignment_13.2

@@ -0,0 +1,31 @@
+Following Links in Python
+In this assignment you will write a Python program that expands on http://www.pythonlearn.com/code/urllinks.py. The program will use urllib to read the HTML from the data files below, extract the href= vaues from the anchor tags, scan for a tag that is in a particular position relative to the first name in the list, follow that link and repeat the process a number of times and report the last name you find.
+We provide two files for this assignment. One is a sample file where we give you the name for your testing and the other is the actual data you need to process for the assignment
+Sample problem: Start at http://python-data.dr-chuck.net/known_by_Fikret.html 
+Find the link at position 3 (the first name is 1). Follow that link. Repeat this process 4 times. The answer is the last name that you retrieve.
+Sequence of names: Fikret Montgomery Mhairade Butchi Anayah 
+Last name in sequence: Anayah
+
+Actual problem: Start at: http://python-data.dr-chuck.net/known_by_Ross.html 
+Find the link at position 18 (the first name is 1). Follow that link. Repeat this process 7 times. The answer is the last name that you retrieve.
+Hint: The first character of the name of the last page that you will load is: S
+
+Strategy
+The web pages tweak the height between the links and hide the page after a few seconds to make it difficult for you to do the assignment without writing a Python program. But frankly with a little effort and patience you can overcome these attempts to make it a little harder to complete the assignment without writing a Python program. But that is not the point. The point is to write a clever Python program to solve the program.
+Sample execution
+Here is a sample execution of a solution:
+$ python solution.py 
+Enter URL: http://python-data.dr-chuck.net/known_by_Fikret.html
+Enter count: 4
+Enter position: 3
+Retrieving: http://python-data.dr-chuck.net/known_by_Fikret.html
+Retrieving: http://python-data.dr-chuck.net/known_by_Montgomery.html
+Retrieving: http://python-data.dr-chuck.net/known_by_Mhairade.html
+Retrieving: http://python-data.dr-chuck.net/known_by_Butchi.html
+Retrieving: http://python-data.dr-chuck.net/known_by_Anayah.html
+
+
+The answer to the assignment for this execution is "Anayah".
+Turning in the Assignment
+Enter the last name retrieved and your Python code below:
+Name:  (name starts with S)

Assignments/Assignment_14

@@ -0,0 +1,33 @@
+Extracting Data from XML
+In this assignment you will write a Python program somewhat similar to http://www.pythonlearn.com/code/geoxml.py. The program will prompt for a URL, read the XML data from that URL using urlliband then parse and extract the comment counts from the XML data, compute the sum of the numbers in the file.
+We provide two files for this assignment. One is a sample file where we give you the sum for your testing and the other is the actual data you need to process for the assignment.
+Sample data: http://python-data.dr-chuck.net/comments_42.xml (Sum=2553)
+Actual data: http://python-data.dr-chuck.net/comments_242234.xml (Sum ends with 93)
+
+You do not need to save these files to your folder since your program will read the data directly from the URL. Note: Each student will have a distinct data url for the assignment - so only use your own data url for analysis.
+Data Format and Approach
+The data consists of a number of names and comment counts in XML as follows:
+<comment>
+  <name>Matthias</name>
+  <count>97</count>
+</comment>
+
+
+You are to look through all the <comment> tags and find the <count> values sum the numbers. The closest sample code that shows how to parse XML is geoxml.py. But since the nesting of the elements in our data is different than the data we are parsing in that sample code you will have to make real changes to the code.
+To make the code a little simpler, you can use an XPath selector string to look through the entire tree of XML for any tag named 'count' with the following line of code:
+counts = tree.findall('.//count')
+
+
+Take a look at the Python ElementTree documentation and look for the supported XPath syntax for details. You could also work from the top of the XML down to the comments node and then loop through the child nodes of the comments node.
+Sample Execution
+$ python solution.py 
+Enter location: http://python-data.dr-chuck.net/comments_42.xml
+Retrieving http://python-data.dr-chuck.net/comments_42.xml
+Retrieved 4204 characters
+Count: 50
+Sum: 2482
+
+
+Turning in the Assignment
+Enter the sum from the actual data and your Python code below:
+Sum:  (ends with 93) ANSWER:

Assignments/Assignment_15_1

@@ -0,0 +1,25 @@
+Calling a JSON API
+In this assignment you will write a Python program somewhat similar to http://www.pythonlearn.com/code/geojson.py. The program will prompt for a location, contact a web service and retrieve JSON for the web service and parse that data, and retrieve the first place_id from the JSON. A place ID is a textual identifier that uniquely identifies a place as within Google Maps.
+API End Points
+To complete this assignment, you should use this API endpoint that has a static subset of the Google Data:
+http://python-data.dr-chuck.net/geojson
+
+
+This API uses the same parameters (sensor and address) as the Google API. This API also has no rate limit so you can test as often as you like. If you visit the URL with no parameters, you get a list of all of the address values which can be used with this API.
+To call the API, you need to provide a sensor=false parameter and the address that you are requesting as the address= parameter that is properly URL encoded using the urllib.urlencode() fuction as shown in http://www.pythonlearn.com/code/geojson.py
+Just for fun, you can also test your program with the real Google API:
+http://maps.googleapis.com/maps/api/geocode/json?sensor=false&address=University+of+Michigan
+
+
+Singe Google's data is always changing, the data returned from the Google API could easily be different than from my local copy API. And the Google API has rate limits. But your code should work with the Google API with no modifications other than the base URL.
+Test Data / Sample Execution
+You can test to see if your program is working with a location of "South Federal University" which will have a place_id of "ChIJJ8oO7_B_bIcR2AlhC8nKlok".
+$ python solution.py
+Enter location: South Federal University 
+Retrieving http://...
+Retrieved 2101 characters
+Place id ChIJJ8oO7_B_bIcR2AlhC8nKlok 
+
+
+Turn In
+Please run your program to find the place_id for "University of Connecticut" and enter the place_id and your Python code below. Hint: The first seven characters of the place_id are "ChIJGbL ..."

Assignments/Assignment_15_2

@@ -0,0 +1,37 @@
+Extracting Data from JSON
+In this assignment you will write a Python program somewhat similar to http://www.pythonlearn.com/code/json2.py. The program will prompt for a URL, read the JSON data from that URL using urlliband then parse and extract the comment counts from the JSON data, compute the sum of the numbers in the file and enter the sum below:
+We provide two files for this assignment. One is a sample file where we give you the sum for your testing and the other is the actual data you need to process for the assignment.
+Sample data: http://python-data.dr-chuck.net/comments_42.json (Sum=2553)
+Actual data: http://python-data.dr-chuck.net/comments_242238.json (Sum ends with 77)
+
+You do not need to save these files to your folder since your program will read the data directly from the URL. Note: Each student will have a distinct data url for the assignment - so only use your own data url for analysis.
+Data Format
+The data consists of a number of names and comment counts in JSON as follows:
+{
+  comments: [
+    {
+      name: "Matthias"
+      count: 97
+    },
+    {
+      name: "Geomer"
+      count: 97
+    }
+    ...
+  ]
+}
+
+
+The closest sample code that shows how to parse JSON and extract a list is json2.py. You might also want to look at geoxml.py to see how to prompt for a URL and retrieve data from a URL.
+Sample Execution
+$ python solution.py 
+Enter location: http://python-data.dr-chuck.net/comments_42.json
+Retrieving http://python-data.dr-chuck.net/comments_42.json
+Retrieved 2733 characters
+Count: 50
+Sum: 2482
+
+
+Turning in the Assignment
+Enter the sum from the actual data and your Python code below:
+Sum:  (ends with 77)

Assignments/Assignment_16_1

@@ -0,0 +1,20 @@
+Instructions
+If you don't already have it, install the SQLite Browser from http://sqlitebrowser.org/.
+
+Then, create a SQLITE database or use an existing database and create a table in the database called "Ages":
+
+CREATE TABLE Ages ( 
+  name VARCHAR(128), 
+  age INTEGER
+)
+Then make sure the table is empty by deleting any rows that you previously inserted, and insert these rows and only these rows with the following commands:
+
+DELETE FROM Ages;
+INSERT INTO Ages (name, age) VALUES ('Malakhy', 37);
+INSERT INTO Ages (name, age) VALUES ('Marlee', 23);
+INSERT INTO Ages (name, age) VALUES ('Timucin', 18);
+INSERT INTO Ages (name, age) VALUES ('Anneroy', 17);
+Once the inserts are done, run the following SQL command:
+SELECT hex(name || age) AS X FROM Ages ORDER BY X
+Find the first row in the resulting record set and enter the long string that looks like 53656C696E613333.
+Note: This assignment must be done using SQLite - in particular, the SELECT query above will not work in any other database. So you cannot use MySQL or Oracle for this assignment.

Assignments/Assignment_16_2

@@ -0,0 +1,14 @@
+Counting Organizations
+This application will read the mailbox data (mbox.txt) count up the number email messages per organization (i.e. domain name of the email address) using a database with the following schema to maintain the counts.
+
+CREATE TABLE Counts (org TEXT, count INTEGER)
+When you have run the program on mbox.txt upload the resulting database file above for grading.
+If you run the program multiple times in testing or with dfferent files, make sure to empty out the data before each run.
+
+You can use this code as a starting point for your application: http://www.pythonlearn.com/code/emaildb.py.
+
+The data file for this application is the same as in previous assignments: http://www.pythonlearn.com/code/mbox.txt.
+
+Because the sample code is using an UPDATE statement and committing the results to the database as each record is read in the loop, it might take as long as a few minutes to process all the data. The commit insists on completely writing all the data to disk every time it is called.
+
+The program can be speeded up greatly by moving the commit operation outside of the loop. In any database program, there is a balance between the number of operations you execute between commits and the importance of not losing the results of operations that have not yet been committed.

Assignments/Assignment_17

@@ -0,0 +1,43 @@
+Musical Track Database
+This application will read an iTunes export file in XML and produce a properly normalized database with this structure:
+
+CREATE TABLE Artist (
+    id  INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
+    name    TEXT UNIQUE
+);
+
+CREATE TABLE Genre (
+    id  INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
+    name    TEXT UNIQUE
+);
+
+CREATE TABLE Album (
+    id  INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
+    artist_id  INTEGER,
+    title   TEXT UNIQUE
+);
+
+CREATE TABLE Track (
+    id  INTEGER NOT NULL PRIMARY KEY 
+        AUTOINCREMENT UNIQUE,
+    title TEXT  UNIQUE,
+    album_id  INTEGER,
+    genre_id  INTEGER,
+    len INTEGER, rating INTEGER, count INTEGER
+);
+If you run the program multiple times in testing or with different files, make sure to empty out the data before each run.
+
+You can use this code as a starting point for your application: http://www.pythonlearn.com/code/tracks.zip. The ZIP file contains the Library.xml file to be used for this assignment. You can export your own tracks from iTunes and create a database, but for the database that you turn in for this assignment, only use the Library.xml data that is provided.
+
+To grade this assignment, the program will run a query like this on your uploaded database and look for the data it expects to see:
+
+SELECT Track.title, Artist.name, Album.title, Genre.name 
+    FROM Track JOIN Genre JOIN Album JOIN Artist 
+    ON Track.genre_id = Genre.ID and Track.album_id = Album.id 
+        AND Album.artist_id = Artist.id
+    ORDER BY Artist.name LIMIT 3
+The expected result of this query on your database is:
+Track	Artist	Album	Genre
+Chase the Ace	AC/DC	Who Made Who	Rock
+D.T.	AC/DC	Who Made Who	Rock
+For Those About To Rock (We Salute You)	AC/DC	Who Made Who	Rock

Assignments/Assignment_18

@@ -0,0 +1,16 @@
+This application will read roster data in JSON format, parse the file, and then produce an SQLite database that contains a User, Course, and Member table and populate the tables from the data file.
+
+You can base your solution on this code: http://www.pythonlearn.com/code/roster.py - this code is incomplete as you need to modify the program to store the role column in the Member table to complete the assignment.
+
+Each student gets their own file for the assignment. Download this file (https://pr4e.dr-chuck.com/tsugi/mod/sql-intro/roster_data.php?PHPSESSID=3d72f3abd510051b658e7c284146cfb5) and save it as roster_data.json. Move the downloaded file into the same folder as your roster.py program.
+
+Once you have made the necessary changes to the program and it has been run successfully reading the above JSON data, run the following SQL command:
+
+
+SELECT hex(User.name || Course.title || Member.role ) AS X FROM 
+    User JOIN Member JOIN Course 
+    ON User.id = Member.user_id AND Member.course_id = Course.id
+    ORDER BY X
+
+
+Find the first row in the resulting record set and enter the long string that looks like 53656C696E613333.