HW5.ipynb

{
  "metadata": {
    "name": ""
  }, 
  "nbformat": 3, 
  "nbformat_minor": 0, 
  "worksheets": [
    {
      "cells": [
        {
          "source": [
            "# Homework 5: Networks and Congress\n", 
            "\n", 
            "*Due Friday, November 15, 11:59pm*\n", 
            "\n", 
            "<img src=\"http://img.washingtonpost.com/rf/image_1024w/2010-2019/WashingtonPost/2011/08/05/National-Politics/Images/uscap.JPG\">\n", 
            "\n", 
            "<br>"
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "cell_type": "code", 
          "language": "python", 
          "outputs": [], 
          "collapsed": false, 
          "prompt_number": 10, 
          "input": [
            "%matplotlib inline\n", 
            "\n", 
            "import json\n", 
            "\n", 
            "import numpy as np\n", 
            "import networkx as nx\n", 
            "import requests\n", 
            "from pattern import web\n", 
            "import matplotlib.pyplot as plt\n", 
            "\n", 
            "# set some nicer defaults for matplotlib\n", 
            "from matplotlib import rcParams\n", 
            "\n", 
            "#these colors come from colorbrewer2.org. Each is an RGB triplet\n", 
            "dark2_colors = [(0.10588235294117647, 0.6196078431372549, 0.4666666666666667),\n", 
            "                (0.8509803921568627, 0.37254901960784315, 0.00784313725490196),\n", 
            "                (0.4588235294117647, 0.4392156862745098, 0.7019607843137254),\n", 
            "                (0.9058823529411765, 0.1607843137254902, 0.5411764705882353),\n", 
            "                (0.4, 0.6509803921568628, 0.11764705882352941),\n", 
            "                (0.9019607843137255, 0.6705882352941176, 0.00784313725490196),\n", 
            "                (0.6509803921568628, 0.4627450980392157, 0.11372549019607843),\n", 
            "                (0.4, 0.4, 0.4)]\n", 
            "\n", 
            "rcParams['figure.figsize'] = (10, 6)\n", 
            "rcParams['figure.dpi'] = 150\n", 
            "rcParams['axes.color_cycle'] = dark2_colors\n", 
            "rcParams['lines.linewidth'] = 2\n", 
            "rcParams['axes.grid'] = False\n", 
            "rcParams['axes.facecolor'] = 'white'\n", 
            "rcParams['font.size'] = 14\n", 
            "rcParams['patch.edgecolor'] = 'none'\n", 
            "\n", 
            "def remove_border(axes=None, top=False, right=False, left=True, bottom=True):\n", 
            "    \"\"\"\n", 
            "    Minimize chartjunk by stripping out unnecessary plot borders and axis ticks\n", 
            "    \n", 
            "    The top/right/left/bottom keywords toggle whether the corresponding plot border is drawn\n", 
            "    \"\"\"\n", 
            "    ax = axes or plt.gca()\n", 
            "    ax.spines['top'].set_visible(top)\n", 
            "    ax.spines['right'].set_visible(right)\n", 
            "    ax.spines['left'].set_visible(left)\n", 
            "    ax.spines['bottom'].set_visible(bottom)\n", 
            "    \n", 
            "    #turn off all ticks\n", 
            "    ax.yaxis.set_ticks_position('none')\n", 
            "    ax.xaxis.set_ticks_position('none')\n", 
            "    \n", 
            "    #now re-enable visibles\n", 
            "    if top:\n", 
            "        ax.xaxis.tick_top()\n", 
            "    if bottom:\n", 
            "        ax.xaxis.tick_bottom()\n", 
            "    if left:\n", 
            "        ax.yaxis.tick_left()\n", 
            "    if right:\n", 
            "        ax.yaxis.tick_right()"
          ], 
          "metadata": {}
        }, 
        {
          "source": [
            "The website [govtrack.us](http://www.govetrack.us) collects data on activities in the Senate and House of Representatives. It's a great source of information for making data-driven assessments about Congress."
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "source": [
            "### Problem 1.\n", 
            "\n", 
            "The directories at http://www.govtrack.us/data/congress/113/votes/2013 contain JSON information about every vote cast for the current (113th)  Congress. Subdirectories beginning with \"S\" correspond to Senate votes, while subdirectories beginning with \"H\" correspond to House votes.\n", 
            "\n", 
            "Write two functions: one that downloads and parses a single Senate vote page given the vote number, and another that repeatedly calls this function to build a full collection of Senate votes from the 113th Congress."
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "cell_type": "code", 
          "language": "python", 
          "outputs": [], 
          "collapsed": false, 
          "prompt_number": 11, 
          "input": [
            "\"\"\"\n", 
            "Function\n", 
            "--------\n", 
            "get_senate_vote\n", 
            "\n", 
            "Scrapes a single JSON page for a particular Senate vote, given by the vote number\n", 
            "\n", 
            "Parameters\n", 
            "----------\n", 
            "vote : int\n", 
            "   The vote number to fetch\n", 
            "   \n", 
            "Returns\n", 
            "-------\n", 
            "vote : dict\n", 
            "   The JSON-decoded dictionary for that vote\n", 
            "   \n", 
            "Examples\n", 
            "--------\n", 
            ">>> get_senate_vote(11)['bill']\n", 
            "{u'congress': 113,\n", 
            " u'number': 325,\n", 
            " u'title': u'A bill to ensure the complete and timely payment of the obligations of the United States Government until May 19, 2013, and for other purposes.',\n", 
            " u'type': u'hr'}\n", 
            "\"\"\"\n", 
            "#your code here\n"
          ], 
          "metadata": {}
        }, 
        {
          "cell_type": "code", 
          "language": "python", 
          "outputs": [], 
          "collapsed": false, 
          "prompt_number": 12, 
          "input": [
            "\"\"\"\n", 
            "Function\n", 
            "--------\n", 
            "get_all_votes\n", 
            "\n", 
            "Scrapes all the Senate votes from http://www.govtrack.us/data/congress/113/votes/2013,\n", 
            "and returns a list of dicts\n", 
            "\n", 
            "Parameters\n", 
            "-----------\n", 
            "None\n", 
            "\n", 
            "Returns\n", 
            "--------\n", 
            "votes : list of dicts\n", 
            "    List of JSON-parsed dicts for each senate vote\n", 
            "\"\"\"\n", 
            "#Your code here\n"
          ], 
          "metadata": {}
        }, 
        {
          "cell_type": "code", 
          "language": "python", 
          "outputs": [], 
          "collapsed": false, 
          "prompt_number": 13, 
          "input": [
            "vote_data = get_all_votes()"
          ], 
          "metadata": {}
        }, 
        {
          "source": [
            "### Problem 2\n", 
            "\n", 
            "Now, turn these data into a NetworkX graph, according to the spec below. For details on using NetworkX, consult the lab materials for November 1, as well as the [NetworkX documentation](http://networkx.github.io/)."
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "cell_type": "code", 
          "language": "python", 
          "outputs": [], 
          "collapsed": false, 
          "prompt_number": 14, 
          "input": [
            "\"\"\"\n", 
            "Function\n", 
            "--------\n", 
            "vote_graph\n", 
            "\n", 
            "Parameters\n", 
            "----------\n", 
            "data : list of dicts\n", 
            "    The vote database returned from get_vote_data\n", 
            "\n", 
            "Returns\n", 
            "-------\n", 
            "graph : NetworkX Graph object, with the following properties\n", 
            "    1. Each node in the graph is labeled using the `display_name` of a Senator (e.g., 'Lee (R-UT)')\n", 
            "    2. Each node has a `color` attribute set to 'r' for Republicans, \n", 
            "       'b' for Democrats, and 'k' for Independent/other parties.\n", 
            "    3. The edges between two nodes are weighted by the number of \n", 
            "       times two senators have cast the same Yea or Nay vote\n", 
            "    4. Each edge also has a `difference` attribute, which is set to `1 / weight`.\n", 
            "\n", 
            "Examples\n", 
            "--------\n", 
            ">>> graph = vote_graph(vote_data)\n", 
            ">>> graph.node['Lee (R-UT)']\n", 
            "{'color': 'r'}  # attributes for this senator\n", 
            ">>> len(graph['Lee (R-UT)']) # connections to other senators\n", 
            "101\n", 
            ">>> graph['Lee (R-UT)']['Baldwin (D-WI)']  # edge relationship between Lee and Baldwin\n", 
            "{'difference': 0.02, 'weight': 50}\n", 
            "\"\"\"\n", 
            "#Your code here\n"
          ], 
          "metadata": {}
        }, 
        {
          "cell_type": "code", 
          "language": "python", 
          "outputs": [], 
          "collapsed": false, 
          "prompt_number": 15, 
          "input": [
            "votes = vote_graph(vote_data)    "
          ], 
          "metadata": {}
        }, 
        {
          "source": [
            "### How (and how not) to visualize networks\n", 
            "\n", 
            "Network plots often *look* impressive, but creating sensible network plots is tricky. From Ben Fry, the author of the Processing program:\n", 
            "<blockquote>\n", 
            "Usually a graph layout isn\u2019t the best option for data sets larger than a few dozen nodes. You\u2019re most likely to wind up with enormous spider webs or balls of string, and the mess seen so far is more often the case than not. Graphs can be a powerful way to represent relationships between data, but they are also a very abstract concept, which means that they run the danger of meaning something only to the creator of the graph. Often, simply showing the structure of the data says very little about what it actually means, even though it\u2019s a perfectly accurate means of representing the data. Everything looks like a graph, but almost nothing should ever be drawn as one.\n", 
            "</blockquote>\n", 
            "\n", 
            "Let's look at bad and better ways of visualizing the senate vote network.\n", 
            "\n", 
            "First, consider the \"default\" plot from networkx."
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "cell_type": "code", 
          "language": "python", 
          "outputs": [], 
          "collapsed": false, 
          "prompt_number": 32, 
          "input": [
            "#this makes sure draw_spring results are the same at each call\n", 
            "np.random.seed(1)  \n", 
            "\n", 
            "color = [votes.node[senator]['color'] for senator in votes.nodes()]\n", 
            "\n", 
            "#determine position of each node using a spring layout\n", 
            "pos = nx.spring_layout(votes, iterations=200)\n", 
            "\n", 
            "#plot the edges\n", 
            "nx.draw_networkx_edges(votes, pos, alpha = .05)\n", 
            "\n", 
            "#plot the nodes\n", 
            "nx.draw_networkx_nodes(votes, pos, node_color=color)\n", 
            "\n", 
            "#draw the labels\n", 
            "lbls = nx.draw_networkx_labels(votes, pos, alpha=5, font_size=8)\n", 
            "\n", 
            "#coordinate information is meaningless here, so let's remove it\n", 
            "plt.xticks([])\n", 
            "plt.yticks([])\n", 
            "remove_border(left=False, bottom=False)"
          ], 
          "metadata": {}
        }, 
        {
          "source": [
            "The spring layout tries to group nodes with large edge-weights near to each other. In this context, that means it tries to organize the Senate into similarly-voting cliques. However, there's simply too much going on in this plot -- we should simplify the representation."
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "source": [
            "### Problem 3\n", 
            "\n", 
            "Compute the `Minimum Spanning Tree` of this graph, using the `difference` edge attribute as the weight to minimize. A [Minimum Spanning Tree](http://en.wikipedia.org/wiki/Minimum_spanning_tree) is the subset of edges which trace at least one path through all nodes (\"spanning\"), with minimum total edge weight. You can think of it as a simplification of a network.\n", 
            "\n", 
            "Plot this new network, making modifications as necessary to prevent the graph from becoming too busy."
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "cell_type": "code", 
          "language": "python", 
          "outputs": [], 
          "collapsed": false, 
          "prompt_number": 33, 
          "input": [
            "#Your code here\n"
          ], 
          "metadata": {}
        }, 
        {
          "source": [
            "### Problem 4\n", 
            "\n", 
            "While this graph has less information, the remaining information is easier to digest. What does the Minimum Spanning Tree mean in this context? How does this graph relate to partisanship in the Senate? Which nodes in this graph are the most and least bi-partisan?"
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "source": [
            "*Your answer here*"
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "source": [
            "### Problem 5\n", 
            "\n", 
            "(For this problem, use the full graph for centrality computation, and not the Minimum Spanning Tree)\n", 
            "\n", 
            "Networkx can easily compute [centrality](https://en.wikipedia.org/wiki/Centrality) measurements. \n", 
            "\n", 
            "Briefly discuss what ``closeness_centrality`` means, both mathematically and in the context of the present graph -- how does the centrality relate to partisanship? Choose a way to visualize the `closeness_centrality` score for each member of the Senate, using edge `difference` as the distance measurement. Determine the 5 Senators with the highest and lowest centralities. \n", 
            "\n", 
            "Comment on your results. In particular, note the outliers John Kerry (who recently resigned his Senate seat when he became Secretary of State), Mo Cowan (Kerry's interim replacement) and Ed Markey (Kerry's permanent replacement) have low centrality scores -- why?"
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "source": [
            "*Your discussion here*"
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "cell_type": "code", 
          "language": "python", 
          "outputs": [], 
          "collapsed": false, 
          "prompt_number": 12, 
          "input": [
            "#Your code here\n"
          ], 
          "metadata": {}
        }, 
        {
          "source": [
            "### Problem 6\n", 
            "\n", 
            "Centrality isn't a perfect proxy for bipartisanship, since it gauges how centralized a node is to the network as a whole, and not how similar a Democrat node is to the Republican sub-network (and vice versa).\n", 
            "\n", 
            "Can you come up with another measure that better captures bipartisanship than closeness centrality? Develop your own metric -- how does it differ from the closeness centrality? Use visualizations to support your points."
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "cell_type": "code", 
          "language": "python", 
          "outputs": [], 
          "collapsed": false, 
          "prompt_number": 13, 
          "input": [
            "#your code here\n"
          ], 
          "metadata": {}
        }, 
        {
          "source": [
            "*Your discussion here*"
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "source": [
            "## Leadership in the Senate\n", 
            "\n", 
            "There are many metrics to quantify the leadership in the Senate.\n", 
            "\n", 
            "  1. Senate leaders sponsor and co-sponsor lots of bills\n", 
            "  2. Leaders sit on many committees, as well as more important committees\n", 
            "  3. Leaders usually have been in office for a long time\n", 
            "  \n", 
            "Another approach uses the philosophy behind how Google ranks search results. The core idea behind Google's PageRank algorithm is:\n", 
            "\n", 
            "1. A \"good\" website (i.e. one to rank highly in search results) is linked to by many other websites\n", 
            "2. A link found on a \"good\" website is more important than a link found on a \"bad\" website\n", 
            "\n", 
            "The PageRank algorithm thus assigns scores to nodes in a graph based on how many neighbors a node has, as well as the score of those neighbors.\n", 
            "\n", 
            "This technique can be adapted to rank Senate leadership. Here, nodes correspond to Senators, and edges correspond to a senator co-sponsoring a bill sponsored by another Senator. The weight of each edge from node A to B is the number of times Senator A has co-sponsored a bill whose primary sponsor is Senator B. If you interpret the PageRank scores of such a network to indicate Senate leadership, you are then assuming:\n", 
            "\n", 
            "1. Leaders sponsor more bills\n", 
            "1. Leaders attract co-sponsorship from other leaders\n", 
            "\n", 
            "### Problem 7\n", 
            "\n", 
            "Govtrack stores information about each Senate bill in the current congress at http://www.govtrack.us/data/congress/113/bills/s/. As in problem 1, write two functions to scrape these data -- the first function downloads a single bill, and the second function calls the first to loop over all bills."
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "cell_type": "code", 
          "language": "python", 
          "outputs": [], 
          "collapsed": false, 
          "prompt_number": 42, 
          "input": [
            "\"\"\"\n", 
            "Function\n", 
            "--------\n", 
            "get_senate_bill\n", 
            "\n", 
            "Scrape the bill data from a single JSON page, given the bill number\n", 
            "\n", 
            "Parameters\n", 
            "-----------\n", 
            "bill : int\n", 
            "   Bill number to fetch\n", 
            "   \n", 
            "Returns\n", 
            "-------\n", 
            "A dict, parsed from the JSON\n", 
            "\n", 
            "Examples\n", 
            "--------\n", 
            ">>> bill = get_senate_bill(10)\n", 
            ">>> bill['sponsor']\n", 
            "{u'district': None,\n", 
            " u'name': u'Reid, Harry',\n", 
            " u'state': u'NV',\n", 
            " u'thomas_id': u'00952',\n", 
            " u'title': u'Sen',\n", 
            " u'type': u'person'}\n", 
            ">>> bill['short_title']\n", 
            "u'Agriculture Reform, Food, and Jobs Act of 2013'\n", 
            "\"\"\"\n", 
            "#your code here\n"
          ], 
          "metadata": {}
        }, 
        {
          "cell_type": "code", 
          "language": "python", 
          "outputs": [], 
          "collapsed": false, 
          "prompt_number": 43, 
          "input": [
            "\"\"\"\n", 
            "Function\n", 
            "--------\n", 
            "get_all_bills\n", 
            "\n", 
            "Scrape all Senate bills at http://www.govtrack.us/data/congress/113/bills/s\n", 
            "\n", 
            "Parameters\n", 
            "----------\n", 
            "None\n", 
            "\n", 
            "Returns\n", 
            "-------\n", 
            "A list of dicts, one for each bill\n", 
            "\"\"\"\n", 
            "#your code here\n"
          ], 
          "metadata": {}
        }, 
        {
          "cell_type": "code", 
          "language": "python", 
          "outputs": [], 
          "collapsed": false, 
          "prompt_number": 44, 
          "input": [
            "bill_list = get_all_bills()"
          ], 
          "metadata": {}
        }, 
        {
          "source": [
            "### Problem 8\n", 
            "\n", 
            "Write a function to builded a Directed Graph (DiGraph) from these data, according to the following spec:"
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "cell_type": "code", 
          "language": "python", 
          "outputs": [], 
          "collapsed": false, 
          "prompt_number": 45, 
          "input": [
            "\"\"\"\n", 
            "Function\n", 
            "--------\n", 
            "bill_graph\n", 
            "\n", 
            "Turn the bill graph data into a NetworkX Digraph\n", 
            "\n", 
            "Parameters\n", 
            "----------\n", 
            "data : list of dicts\n", 
            "    The data returned from get_all_bills\n", 
            "\n", 
            "Returns\n", 
            "-------\n", 
            "graph : A NetworkX DiGraph, with the following properties\n", 
            "    * Each node is a senator. For a label, use the 'name' field \n", 
            "      from the 'sponsor' and 'cosponsors' dict items\n", 
            "    * Each edge from A to B is assigned a weight equal to how many \n", 
            "      bills are sponsored by B and co-sponsored by A\n", 
            "\"\"\"\n", 
            "#Your code here\n"
          ], 
          "metadata": {}
        }, 
        {
          "cell_type": "code", 
          "language": "python", 
          "outputs": [], 
          "collapsed": false, 
          "prompt_number": 46, 
          "input": [
            "bills = bill_graph(bill_list)"
          ], 
          "metadata": {}
        }, 
        {
          "source": [
            "### Problem 9\n", 
            "\n", 
            "Using `nx.pagerank_numpy`, compute the PageRank score for each senator in this graph. Visualize the results. Determine the 5 Senators with the highest\n", 
            "PageRank scores. How effective is this approach at identifying leaders? How does the PageRank rating compare to the degree of each node?\n", 
            "\n", 
            "Note: you can read about individual Senators by searching for them on the [govtrack website](https://www.govtrack.us/)."
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "cell_type": "code", 
          "language": "python", 
          "outputs": [], 
          "collapsed": false, 
          "prompt_number": 47, 
          "input": [
            "#Your code here\n"
          ], 
          "metadata": {}
        }, 
        {
          "cell_type": "code", 
          "language": "python", 
          "outputs": [], 
          "collapsed": false, 
          "prompt_number": 56, 
          "input": [
            "#Your code here\n"
          ], 
          "metadata": {}
        }, 
        {
          "source": [
            "*Your discussion here*"
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "source": [
            "### Interactive Visualization\n", 
            "\n", 
            "Producing a good node link layout is not quite so simple. Nevertheless, we will give it a try. \n", 
            "\n", 
            "We will use [Gephi](https://gephi.org/) for interactive graph visualization. Gephi supports a wide variety of graph file formats, and NetworkX exports to several of them. We'll use the Graph Exchange XML Format (GEXF)."
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "cell_type": "code", 
          "language": "python", 
          "outputs": [], 
          "collapsed": false, 
          "prompt_number": 118, 
          "input": [
            "nx.write_gexf(votes, 'votes.gexf')"
          ], 
          "metadata": {}
        }, 
        {
          "source": [
            "### Problem 10: Analysis with Gephi\n", 
            "\n", 
            "Download and install [Gephi](https://gephi.org/). See the [lab](http://goo.gl/SzHioP) for a brief introduction. Load the exported votes file. Try to produce a layout that clearly separates Democrats from Republicans (hint: filter on edge weight and re-layout once you filtered). Run PageRank and some other statistics and try encoding them with node color and node size. Run the \"Modularity\" statistic and encode the results in color.\n", 
            "\n", 
            "Include a screenshot of your \"best\" visualization and embed the image here with `IPython.display.Image`. Make sure to include this image in your submission.\n", 
            "\n", 
            "Explain your observations. Is the network visualization very helpful? Try to visualize your LinkedIn network (see the lab) or the one provided in the lab. Which dataset is more suitable for visualization and why is there a difference?\n"
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "cell_type": "code", 
          "language": "python", 
          "outputs": [], 
          "collapsed": false, 
          "input": [
            "from IPython.display import Image\n", 
            "\n", 
            "path = 'name_of_your_screenshot'\n", 
            "Image(path)"
          ], 
          "metadata": {}
        }, 
        {
          "source": [
            "*Your answer here*"
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "source": [
            "### How to Submit\n", 
            "\n", 
            "To submit your homework, create a folder named lastname_firstinitial_hw5 and place this notebook file in the folder. Double check that this file is still called HW5.ipynb, and that it contains your solutions. Also include any Gephi screenshots. Compress the folder (please use .zip compression) and submit to the CS109 dropbox in the appropriate folder. If we cannot access your work because these directions are not followed correctly, we will not grade your work."
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }, 
        {
          "source": [
            "---\n", 
            "*css tweaks in this cell*\n", 
            "<style>\n", 
            "div.text_cell_render {\n", 
            "    line-height: 150%;\n", 
            "    font-size: 110%;\n", 
            "    width: 800px;\n", 
            "    margin-left:50px;\n", 
            "    margin-right:auto;\n", 
            "    }\n", 
            "</style>"
          ], 
          "cell_type": "markdown", 
          "metadata": {}
        }
      ], 
      "metadata": {}
    }
  ]
}