notebooks/Day_20_CommonCrawl_Starter.ipynb

{
 "metadata": {
  "name": "Day_20_CommonCrawl_Starter"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Goals"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "For us to learn:\n",
      "\n",
      "* the basics of how to process CommonCrawl data by counting files and tallying file sizes in the CC crawl\n",
      "* how to do this processing in parallel fashion using PiCloud, Amazon AWS (specifically S3), the [`boto` library](http://boto.readthedocs.org/en/latest/) \n",
      "\n",
      "We start by writing a function to calculate stats on one given \"valid segment\" in the Common Crawl.  Then we'll learn how to calculate the stats for all valid segments and aggregate the data."
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Prerequisites"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Although strictly speaking, you can do all the work directly on PiCloud (where I'm handling the dependencies), you'll likely want to get PiCloud, boto, s3cmd set up locally.  See [Day 19 notes](http://nbviewer.ipython.org/urls/raw.github.com/rdhyee/working-open-data/master/notebooks/Day_19_CC_etc.ipynb) and [Day 16 PiCloud intro](http://nbviewer.ipython.org/urls/raw.github.com/rdhyee/working-open-data/master/notebooks/Day_16_PiCloud_intro.ipynb) for a refresher.  **One big reason for working locally is that you'll get charged for the time you are running a PiCloud notebook server** -- and when you are thinking, it's nice to not have to worry about the time (even if it is $0.05/hour for a running a c1 PiCloud instance.)\n",
      "\n",
      "Also ask for help if you are having problems."
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Moving notebooks between local storage and PiCloud"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "One way to copy this notebook from local machine to PiCloud -- use cloud.bucket.put"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "To see what's in your PiCloud bucket"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import cloud\n",
      "cloud.bucket.list()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 1,
       "text": [
        "[u'notebook/Day_02_class_starter.ipynb',\n",
        " u'notebook/Day_02_completed.ipynb',\n",
        " u'notebook/Day_04_completed.ipynb',\n",
        " u'notebook/Day_04_starter.ipynb',\n",
        " u'notebook/Day_05_plotting.ipynb',\n",
        " u'notebook/Day_07_array_len_and_multiply.ipynb',\n",
        " u'notebook/Day_08_basemap_globe_example.ipynb',\n",
        " u'notebook/Day_08_completed.ipynb',\n",
        " u'notebook/Day_08_freebase_intro.ipynb',\n",
        " u'notebook/Day_08_starter.ipynb',\n",
        " u'notebook/Day_10_A_fixed_width_parsing_completed.ipynb',\n",
        " u'notebook/Day_10_freebase_cursor_completed.ipynb',\n",
        " u'notebook/Day_10_requests_lxml.ipynb',\n",
        " u'notebook/Day_14_PfDA_revisited.ipynb',\n",
        " u'notebook/Day_14_PfDA_starter.ipynb',\n",
        " u'notebook/Day_14_basemap_redux.ipynb',\n",
        " u'notebook/Day_14_date_time.ipynb',\n",
        " u'notebook/Day_15_Sample_Python_Questions.ipynb',\n",
        " u'notebook/Day_16_PiCloud_intro.ipynb',\n",
        " u'notebook/Day_17_Midterm.ipynb',\n",
        " u'notebook/Day_17_Midterm_with_Key.ipynb',\n",
        " u'notebook/Day_18_Common_Crawl.ipynb',\n",
        " u'notebook/Day_19_CC_etc.ipynb',\n",
        " u'notebook/Day_20_CommonCrawl.ipynb',\n",
        " u'notebook/Primer.ipynb',\n",
        " u'notebook/basemap_example.ipynb',\n",
        " u'notebook/notebook_javascript_examples.ipynb',\n",
        " u'notebook/vtk_example.ipynb']"
       ]
      }
     ],
     "prompt_number": 1
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "copying from local computer to PiCloud"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# http://docs.picloud.com/moduledoc.html#module-cloud.bucket\n",
      "\n",
      "import os\n",
      "# only if we not running on picloud....\n",
      "if not os.path.exists('/home/picloud/notebook'):\n",
      "    pass\n",
      "    # normally I keep this line commented to prevent accidental copying if I run the notebook through.\n",
      "    cloud.bucket.put('Day_20_CommonCrawl_Starter.ipynb', 'notebook/Day_20_CommonCrawl_Starter.ipynb')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "copying from PiCloud to local machine"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import os\n",
      "\n",
      "if not os.path.exists('/home/picloud/notebook'):\n",
      "    pass\n",
      "    # normally I keep this line commented to prevent accidental copying if I run the notebook through.\n",
      "    # note the new local name -- to make it less likely to overwrite something I'm doing locally.\n",
      "    #cloud.bucket.get('notebook/Day_20_CommonCrawl_Starter.ipynb', 'Day_20_CommonCrawl_Starter_from_picloud.ipynb')\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**Warning:** I don't think you'll immediately see the notebook changes reflected in an already running PiCloud notebook server -- at least, that was my experience."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "There are other ways to interact with PiCloud -- using picloud ssh-info and scp --See [SSH into a job](http://docs.picloud.com/job_mgmt_adv.html#client-adv-ssh-into-job) and some [rough notes](https://www.evernote.com/shard/s1/sh/a9fab233-1857-4f01-8437-805be0e6fe22/593a1224dd150d6cf1cea6bb9886c22d).  The following code shows how to use `picloud ssh-info JID` to get the right ssh scp commands.\n",
      "\n",
      "You can read off the job id for your PiCloud notebook server from the upper right corner of https://www.picloud.com/accounts/notebook/:\n",
      "\n",
      "<img src=\"https://www.evernote.com/shard/s1/sh/646f953b-2c86-4b52-9deb-6dbe4f6ebc9e/26894da1003bf19d26cb0033f89097b4/res/a6f9765c-fa23-4521-9b02-3660134c9b80/PiCloud_%7C_Notebook-20130404-110444.jpg.jpg?resizeSmall&width=832\">\n",
      "\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import re\n",
      "\n",
      "# put the job id of your notebook server after ssh-info\n",
      "\n",
      "NOTEBOOK_SERVER_RUNNING = False\n",
      "NOTEBOOK_SERVER_JID = 501\n",
      "\n",
      "def to_picloud(nb_name):\n",
      "    scp_to_command =  \"scp -q -i {identity} -P {port} {nb_name} {username}@{address}:/home/picloud/notebook/\".format(nb_name=nb_name, **ssh_info_output)\n",
      "    return scp_to_command\n",
      "\n",
      "if NOTEBOOK_SERVER_RUNNING:\n",
      "    ssh_info_output = !picloud ssh-info $NOTEBOOK_SERVER_JID\n",
      "    ssh_info_output = dict(zip( *[filter(None, re.split(\"\\s+\", l)) for l in ssh_info_output]))\n",
      "#print ssh_info_output\n",
      "\n",
      "    ssh_command = \"ssh -q -i {identity} {username}@{address} -p {port}\".format(**ssh_info_output)\n",
      "\n",
      "\n",
      "    print ssh_command\n",
      "    print to_picloud(\"Day_20_CommonCrawl_Starter.ipynb\")\n",
      "\n",
      "# you can even run the scp command from within iPython notebook -- uncomment following lines\n",
      "#    to_picloud = to_picloud(\"Day_20_CommonCrawl_Starter.ipynb\")\n",
      "#    ! $to_picloud"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**Running scp to the live notebook server machine will actually update the notebooks.**"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Learning about Common Crawl structure"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Good to review Dave Lester's talk: http://www.slideshare.net/davelester/introduction-to-common-crawl  \n",
      "\n",
      "If you need general intro to Common Crawl, watch the [Common Crawl Video](https://www.youtube.com/watch?v=ozX4GvUWDm4)."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Common Crawl data stored in Amazon S3"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The Common Crawl data structure is documented at https://commoncrawl.atlassian.net/wiki/display/CRWL/About+the+Data+Set. To quote the docs:\n",
      "\n",
      "The entire Common Crawl data set is stored on Amazon S3 as a Public Data Set:\n",
      "\n",
      "    http://aws.amazon.com/datasets/41740\n",
      "\n",
      "The data set is divided into three major subsets:\n",
      "\n",
      "* Archived Crawl #1 - s3://aws-publicdatasets/common-crawl/crawl-001/ - crawl data from 2008/2010\n",
      "* Archived Crawl #2 - s3://aws-publicdatasets/common-crawl/crawl-002/ - crawl data from 2009/2010\n",
      "* Current Crawl - s3://aws-publicdatasets/common-crawl/parse-output/ - crawl data from 2012\n",
      "\n",
      "The two archived crawl data sets are stored in folders organized by the year, month, date, and hour the content was crawled.  For example:\n",
      "\n",
      "    s3://aws-publicdatasets/common-crawl/crawl-002/2010/01/06/10/1262847572760_10.arc.gz\n",
      "\n",
      "The current crawl data set is stored in the \"parse-output\" folder in a similar manner to how Nutch stores archives.  Crawl data is stored in a \"segments\" subfolder, then in a folder that starts with the UNIX timestamp of crawl start time.  For example:\n",
      "\n",
      "    s3://aws-publicdatasets/common-crawl/parse-output/segment/1341690169105/1341826131693_45.arc.gz\n"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Using s3cmd and boto to confirm the examples from the documentation"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# this key, secret access to aws-publicdatasets only -- created for WwOD 13 student usage\n",
      "\n",
      "# turns out there is an anonymous mode in boto for public data sets:\n",
      "# https://github.com/keiw/common_crawl_index/commit/ad341d0a41a828f260c9c08419dadff0dac6cf5b#L0R33\n",
      "#  conn=S3Connection(anon=True) will work instead of conn= S3Connection(KEY, SECRET) -- but there seems to be \n",
      "# a bug in how S3Connection gets pickled for anon=True -- so for now, just use the KEY, SECRET\n",
      "\n",
      "KEY = 'AKIAJH2FD7572FCTVSSQ'\n",
      "SECRET = '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'"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You can use this key/secret pair to configure both `boto` and `s3cmd`"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# s3cmd installed in custom PiCloud environment -- and maybe in your local environment too\n",
      "\n",
      "# confirm s3://aws-publicdatasets/common-crawl/crawl-002/2010/01/06/10/1262847572760_10.arc.gz\n",
      "# doc for s3cmd: http://s3tools.org/s3cmd\n",
      "\n",
      "!s3cmd ls s3://aws-publicdatasets/common-crawl/crawl-002/2010/01/06/10/1262847572760_10.arc.gz"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "2012-01-05 19:19 100001092   s3://aws-publicdatasets/common-crawl/crawl-002/2010/01/06/10/1262847572760_10.arc.gz\r\n"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "EXERCISE:  use s3cmd to confirm existence of `s3://aws-publicdatasets/common-crawl/parse-output/segment/1341690169105/1341826131693_45.arc.gz`"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "\n",
      "\n",
      "\n",
      "\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 6
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "using s3cmd to look at parse-output and valid_segments.txt in current crawl"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# looking at parse-output itself\n",
      "\n",
      "!s3cmd ls s3://aws-publicdatasets/common-crawl/parse-output"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "                       DIR   s3://aws-publicdatasets/common-crawl/parse-output-test/\r\n",
        "                       DIR   s3://aws-publicdatasets/common-crawl/parse-output/\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "2012-09-04 05:03         0   s3://aws-publicdatasets/common-crawl/parse-output-test_$folder$\r\n",
        "2012-11-09 11:28         0   s3://aws-publicdatasets/common-crawl/parse-output_$folder$\r\n"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# looking at what is contained by parse-output \"folder\"\n",
      "\n",
      "!s3cmd ls s3://aws-publicdatasets/common-crawl/parse-output/"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "                       DIR   s3://aws-publicdatasets/common-crawl/parse-output/checkpoint_staging/\r\n",
        "                       DIR   s3://aws-publicdatasets/common-crawl/parse-output/checkpoints/\r\n",
        "                       DIR   s3://aws-publicdatasets/common-crawl/parse-output/segment/\r\n",
        "                       DIR   s3://aws-publicdatasets/common-crawl/parse-output/valid_segments2/\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "2012-10-17 00:11         0   s3://aws-publicdatasets/common-crawl/parse-output/checkpoint_staging_$folder$\r\n",
        "2012-11-09 00:10         0   s3://aws-publicdatasets/common-crawl/parse-output/checkpoints_$folder$\r\n",
        "2012-09-05 05:13         0   s3://aws-publicdatasets/common-crawl/parse-output/segment_$folder$\r\n",
        "2012-11-09 11:28      2478   s3://aws-publicdatasets/common-crawl/parse-output/valid_segments.txt\r\n",
        "2012-09-05 05:13         0   s3://aws-publicdatasets/common-crawl/parse-output/valid_segments2_$folder$\r\n",
        "2012-07-09 15:07         0   s3://aws-publicdatasets/common-crawl/parse-output/valid_segments_$folder$\r\n"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "There is a list of \"valid segments\" in \n",
      "\n",
      "    s3://aws-publicdatasets/common-crawl/parse-output/valid_segments.txt\n",
      "\n",
      "-- a list of segments that are part of the current crawl.  Let's download it and study it.\n",
      "\n",
      "See [discussion about valid segments](https://groups.google.com/forum/#!msg/common-crawl/QYTmnttZZyo/NPiXvK8ZeiMJ)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!s3cmd ls s3://aws-publicdatasets/common-crawl/parse-output/valid_segments.txt"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "2012-11-09 11:28      2478   s3://aws-publicdatasets/common-crawl/parse-output/valid_segments.txt\r\n"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# we can download it:\n",
      "\n",
      "!s3cmd get s3://aws-publicdatasets/common-crawl/parse-output/valid_segments.txt"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "ERROR: Parameter problem: File ./valid_segments.txt already exists. Use either of --force / --continue / --skip-existing or give it a new name.\r\n"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!head valid_segments.txt"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1346823845675\r\n",
        "1346823846036\r\n",
        "1346823846039\r\n",
        "1346823846110\r\n",
        "1346823846125\r\n",
        "1346823846150\r\n",
        "1346823846176\r\n",
        "1346876860445\r\n",
        "1346876860454\r\n",
        "1346876860467\r\n"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "using boto to study parse-output and valid_segments.txt"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# http://boto.s3.amazonaws.com/s3_tut.html\n",
      "\n",
      "import boto\n",
      "from boto.s3.connection import S3Connection\n",
      "\n",
      "from itertools import islice\n",
      "\n",
      "conn = S3Connection(KEY,SECRET)\n",
      "\n",
      "# turns out there is an anonymous mode in boto for public data sets:\n",
      "# https://github.com/keiw/common_crawl_index/commit/ad341d0a41a828f260c9c08419dadff0dac6cf5b#L0R33\n",
      "#conn=S3Connection(anon=True)\n",
      "\n",
      "bucket = conn.get_bucket('aws-publicdatasets')\n",
      "for key in islice(bucket.list(prefix=\"common-crawl/parse-output/\", delimiter=\"/\"),None):\n",
      "    print key.name.encode('utf-8')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "common-crawl/parse-output/checkpoint_staging_$folder$"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "common-crawl/parse-output/checkpoints_$folder$\n",
        "common-crawl/parse-output/segment_$folder$\n",
        "common-crawl/parse-output/valid_segments.txt\n",
        "common-crawl/parse-output/valid_segments2_$folder$\n",
        "common-crawl/parse-output/valid_segments_$folder$\n",
        "common-crawl/parse-output/checkpoint_staging/\n",
        "common-crawl/parse-output/checkpoints/\n",
        "common-crawl/parse-output/segment/\n",
        "common-crawl/parse-output/valid_segments2/\n"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# get valid_segments\n",
      "# https://commoncrawl.atlassian.net/wiki/display/CRWL/About+the+Data+Set\n",
      "\n",
      "import boto\n",
      "from boto.s3.connection import S3Connection\n",
      "\n",
      "conn = S3Connection(KEY, SECRET)\n",
      "bucket = conn.get_bucket('aws-publicdatasets')\n",
      "\n",
      "k = bucket.get_key(\"common-crawl/parse-output/valid_segments.txt\")\n",
      "s = k.get_contents_as_string()\n",
      "\n",
      "valid_segments = filter(None, s.split(\"\\n\"))\n",
      "\n",
      "print len(valid_segments), valid_segments[0]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "177 1346823845675\n"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# valid_segments are Unix timestamps (in ms) -- confirm current crawl is from 2012\n",
      "\n",
      "import datetime\n",
      "datetime.datetime.fromtimestamp(float(valid_segments[0])/1000.)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 14,
       "text": [
        "datetime.datetime(2012, 9, 4, 22, 44, 5, 675000)"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Using boto to compile stats on each valid segment"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "As of the time of this writing (April 4, 2013), there are 177 valid segments in the current crawl.  Now, it's time to figure out how to write a Python function called `segment_stats` that takes a segment id and an optional `stop` parameter (for the max number of keys to iterate through) of the form\n",
      "\n",
      "    def segment_stats(seg_id, stop=None):\n",
      "        pass\n",
      "        # YOUR EXERCISE TO FILL IN\n",
      "\n",
      "and returns a `dict` with 2 keys:  \n",
      "\n",
      "* `count` holding the number of keys inside the given valid segment\n",
      "* `size` holding the total number of bytes held in the keys\n",
      "\n",
      "broken down by file type (there are 3 major types):\n",
      "\n",
      "* `arg.gz` for the \n",
      "* 'metadata' for the metadata files\n",
      "* 'textData' for the textdata files\n",
      "* 'success' for success files\n",
      "\n",
      "For example:\n",
      "\n",
      "    segment_stats('1346823845675', None)\n",
      "\n",
      "should return:\n",
      "\n",
      "    {\n",
      "     'count': {'arc.gz': 11904, 'metadata': 4377, 'success': 1, 'textData': 4377},\n",
      "     'size': {'arc.gz': 967409519222,\n",
      "          'metadata': 187079951008,\n",
      "          'success': 0,\n",
      "          'textData': 129994977292}\n",
      "    }"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Start by looking at a small subset of keys from valid_segments[0]"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Since it can take 10-50 seconds or so to retrieve all the keys in a valid segment, it's worth limiting to say first 10 to get a feel for what you can do with a key.  Run the following:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from itertools import islice\n",
      "\n",
      "import boto\n",
      "from boto.s3.connection import S3Connection\n",
      "\n",
      "conn = S3Connection(KEY, SECRET)\n",
      "bucket = conn.get_bucket('aws-publicdatasets')\n",
      "for key in islice(bucket.list(prefix=\"common-crawl/parse-output/segment/1346823845675/\", delimiter=\"/\"),10):\n",
      "    print key.name.encode('utf-8')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "common-crawl/parse-output/segment/1346823845675/1346864466526_10.arc.gz\n",
        "common-crawl/parse-output/segment/1346823845675/1346864469604_0.arc.gz\n",
        "common-crawl/parse-output/segment/1346823845675/1346864469638_1.arc.gz\n",
        "common-crawl/parse-output/segment/1346823845675/1346864471290_4.arc.gz\n",
        "common-crawl/parse-output/segment/1346823845675/1346864477152_29.arc.gz\n",
        "common-crawl/parse-output/segment/1346823845675/1346864479613_6.arc.gz\n",
        "common-crawl/parse-output/segment/1346823845675/1346864480261_2.arc.gz\n",
        "common-crawl/parse-output/segment/1346823845675/1346864480936_5.arc.gz\n",
        "common-crawl/parse-output/segment/1346823845675/1346864484063_39.arc.gz\n",
        "common-crawl/parse-output/segment/1346823845675/1346864484163_3.arc.gz\n"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# WARNING -- this might take a bit of time to run -- run it to see how long it takes you to get all the keys in this\n",
      "# segment.  time depends on where you are running this code\n",
      "\n",
      "%time all_files = list(islice(bucket.list(prefix=\"common-crawl/parse-output/segment/1346823845675/\", delimiter=\"/\"),None))\n",
      "print len(all_files), all_files[0]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "CPU times: user 3.64 s, sys: 0.16 s, total: 3.79 s\n",
        "Wall time: 46.83 s\n",
        "20659 <Key: aws-publicdatasets,common-crawl/parse-output/segment/1346823845675/1346864466526_10.arc.gz>\n"
       ]
      }
     ],
     "prompt_number": 16
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "But it's useful now to have `all_files` to hold all the keys under the segment `1346823845675`  Note, for example, you can get the size of the file and the name -- and the type of file (boto.s3.key.Key)"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# http://boto.readthedocs.org/en/latest/ref/s3.html#module-boto.s3.key\n",
      "\n",
      "file0 = all_files[0]\n",
      "type(file0), file0.name, file0.size"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 17,
       "text": [
        "(boto.s3.key.Key,\n",
        " u'common-crawl/parse-output/segment/1346823845675/1346864466526_10.arc.gz',\n",
        " 100011998)"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import boto\n",
      "from boto.s3.connection import S3Connection\n",
      "\n",
      "# this key, secret access to aws-publicdatasets only -- createdd for WwOD 13 student usage\n",
      "KEY = 'AKIAJH2FD7572FCTVSSQ'\n",
      "SECRET = '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'\n",
      "\n",
      "from itertools import islice\n",
      "from pandas import DataFrame\n",
      "\n",
      "conn= S3Connection(KEY, SECRET)\n",
      "bucket = conn.get_bucket('aws-publicdatasets')\n",
      "\n",
      "# you might find this conversion function between DataFrame and a list of a regular dict useful\n",
      "#https://gist.github.com/mikedewar/1486027#comment-804797\n",
      "def df_to_dictlist(df):\n",
      "    return [{k:df.values[i][v] for v,k in enumerate(df.columns)} for i in range(len(df))]\n",
      "\n",
      "def cc_file_type(path):\n",
      "\n",
      "    fname = path.split(\"/\")[-1]\n",
      "    \n",
      "    if fname[-7:] == '.arc.gz':\n",
      "        return 'arc.gz'\n",
      "    elif fname[:9] == 'textData-':\n",
      "        return 'textData'\n",
      "    elif fname[:9] == 'metadata-':\n",
      "        return 'metadata'\n",
      "    elif fname == '_SUCCESS':\n",
      "        return 'success'\n",
      "    else:\n",
      "        return 'other'\n",
      "    \n",
      "def segment_stats(seg_id, stop=None):\n",
      "    \n",
      "    # FILL IN WITH YOUR CODE\n",
      "    \n",
      "    return {\n",
      "      'count': {'arc.gz': 11904, 'metadata': 4377, 'success': 1, 'textData': 4377},\n",
      "      'size': {'arc.gz': 967409519222,\n",
      "      'metadata': 187079951008,\n",
      "      'success': 0,\n",
      "      'textData': 129994977292}\n",
      "     }\n",
      "    "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "/Users/raymondyee/.virtualenvs/epd1/lib/python2.7/site-packages/pytz/__init__.py:35: UserWarning: Module argparse was already imported from /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/argparse.pyc, but /Users/raymondyee/.virtualenvs/epd1/lib/python2.7/site-packages is being added to sys.path\n",
        "  from pkg_resources import resource_stream\n"
       ]
      }
     ],
     "prompt_number": 18
    }
   ],
   "metadata": {}
  }
 ]
}