notebooks/Day_21_CommonCrawl_Content.ipynb

{
 "metadata": {
  "name": "Day_21_CommonCrawl_Content"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Although we're putting a lot of emphasis in WwOD on doing the basic task of counting files and bytes in Common Crawl, this notebook will show you how to look at the *content* of the files in Common Crawl.  "
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Setup"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# this key, secret access to aws-publicdatasets only -- created for WwOD 13 student usage\n",
      "\n",
      "# turns out there is an anonymous mode in boto for public data sets:\n",
      "# https://github.com/keiw/common_crawl_index/commit/ad341d0a41a828f260c9c08419dadff0dac6cf5b#L0R33\n",
      "#  conn=S3Connection(anon=True) will work instead of conn= S3Connection(KEY, SECRET) -- but there seems to be \n",
      "# a bug in how S3Connection gets pickled for anon=True -- so for now, just use the KEY, SECRET\n",
      "\n",
      "KEY = 'AKIAJH2FD7572FCTVSSQ'\n",
      "SECRET = '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'"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# http://boto.s3.amazonaws.com/s3_tut.html\n",
      "\n",
      "import boto\n",
      "from boto.s3.connection import S3Connection\n",
      "\n",
      "from itertools import islice\n",
      "\n",
      "conn = S3Connection(KEY,SECRET)\n",
      "\n",
      "# turns out there is an anonymous mode in boto for public data sets:\n",
      "# https://github.com/keiw/common_crawl_index/commit/ad341d0a41a828f260c9c08419dadff0dac6cf5b#L0R33\n",
      "#conn=S3Connection(anon=True)\n",
      "\n",
      "bucket = conn.get_bucket('aws-publicdatasets')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Integration with url index"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Look at http://urlsearch.commoncrawl.org/\n",
      "\n",
      "* [blog post about URL search](http://commoncrawl.org/url-search-tool/)\n",
      "* [blog post about the URL search index](http://commoncrawl.org/common-crawl-url-index/)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "For example, let's look up ischool.berkeley.edu in the URL index:\n",
      "\n",
      "http://urlsearch.commoncrawl.org/?q=ischool.berkeley.edu\n",
      "\n",
      "You can also download the results as a json file \n",
      "\n",
      "http://urlsearch.commoncrawl.org/download?q=edu.berkeley.ischool\n",
      "\n",
      "which can be parsed:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import requests\n",
      "import json\n",
      "s = requests.get(\"http://urlsearch.commoncrawl.org/download?q=edu.berkeley.ischool\")\n",
      "data = [json.loads(row) for row in s.content.split(\"\\n\") if row]\n",
      "print len(data)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "547\n"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# http://urlsearch.commoncrawl.org/page/1346876860493/1346901517112/422/320051/596\n",
      "u = data[0]\n",
      "u"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 4,
       "text": [
        "{u'arcFileDate': 1346901517112L,\n",
        " u'arcFileOffset': 320051,\n",
        " u'arcFileParition': 422,\n",
        " u'arcSourceSegmentId': 1346876860493L,\n",
        " u'compressedSize': 596,\n",
        " u'url': u'http://people.ischool.berkeley.edu/~rosario/papers.html'}"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# form the urlsearch url from the information returned\n",
      "urlsearch_url = \"http://urlsearch.commoncrawl.org/page/{arcSourceSegmentId}/{arcFileDate}/{arcFileParition}/{arcFileOffset}/{compressedSize}\".format(**u)\n",
      "urlsearch_url"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 5,
       "text": [
        "'http://urlsearch.commoncrawl.org/page/1346876860493/1346901517112/422/320051/596'"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# can also look up the corresponding arc.gz file in S3\n",
      "\n",
      "!s3cmd ls s3://aws-publicdatasets/common-crawl/parse-output/segment/1346876860493/1346901517112_422.arc.gz "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "2012-09-06 04:03 100067216   s3://aws-publicdatasets/common-crawl/parse-output/segment/1346876860493/1346901517112_422.arc.gz\r\n"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Grabbing pieces of the .arc.gz files"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "I think it's possible to use the Python module [warc](https://github.com/internetarchive/warc/blob/master/docs/index.rst) to parse out the .arc.gz files but if we have the offset and size (provided by the URL index), we don't have to grab the entire file -- but just the piece we want."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**Range** specification in S3\n",
      "\n",
      "http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectGET.html  ->  Downloads the specified range bytes of an object. For more information about the HTTP Range header, go to http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35\n",
      "\n",
      "try: \n",
      "\n",
      "bytes={offset}-499\n",
      "\n",
      "\n",
      "OK -- the offset and compressed size can still be used even with gzip compression -- see https://github.com/trivio/common_crawl_index#retrieving-a-page\n",
      "\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#https://github.com/trivio/common_crawl_index#retrieving-a-page\n",
      "\n",
      "from StringIO import StringIO\n",
      "from gzip import GzipFile\n",
      "\n",
      "\n",
      "def arc_file(s3, bucket, info):\n",
      "\n",
      "    bucket = s3.lookup(bucket)\n",
      "    keyname = \"/common-crawl/parse-output/segment/{arcSourceSegmentId}/{arcFileDate}_{arcFileParition}.arc.gz\".format(**info)\n",
      "    key = bucket.lookup(keyname)\n",
      "    \n",
      "    start = info['arcFileOffset']\n",
      "    end = start + info['compressedSize'] - 1\n",
      "    \n",
      "    headers={'Range' : 'bytes={}-{}'.format(start, end)}\n",
      "    \n",
      "    chunk = StringIO(\n",
      "         key.get_contents_as_string(headers=headers)\n",
      "    )\n",
      "    \n",
      "    return GzipFile(fileobj=chunk).read()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "u"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 8,
       "text": [
        "{u'arcFileDate': 1346901517112L,\n",
        " u'arcFileOffset': 320051,\n",
        " u'arcFileParition': 422,\n",
        " u'arcSourceSegmentId': 1346876860493L,\n",
        " u'compressedSize': 596,\n",
        " u'url': u'http://people.ischool.berkeley.edu/~rosario/papers.html'}"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s = arc_file(conn, 'aws-publicdatasets', u)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# voila\n",
      "\n",
      "print s"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "http://people.ischool.berkeley.edu/~rosario/papers.html 128.32.78.16 20120522225235 text/html 821\n",
        "HTTP/1.1 200 OK\r\n",
        "Date:Tue, 22 May 2012 22:53:48 GMT\r\n",
        "Server:Apache/2.2.22 (Fedora)\r\n",
        "Last-Modified:Mon, 08 Apr 2002 18:25:30 GMT\r\n",
        "ETag:\"5a1d0a0-208-39e213165da80\"\r\n",
        "Accept-Ranges:bytes\r\n",
        "Content-Length:520\r\n",
        "Connection:close\r\n",
        "Content-Type:text/html; charset=UTF-8\r\n",
        "x-commoncrawl-DetectedCharset:UTF-8\r\n",
        "\r\n",
        "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\r\n",
        "<html> <head>\r\n",
        "<title>Barbara Rosario. Publications</title>\r\n",
        "</head>\r\n",
        "\r\n",
        "\r\n",
        "<frameset cols =\"20%,*\" frameborder=\"NO\" border=\"0\" framespacing=\"0\"> \r\n",
        "  <frame src=\"navigation_research_papers.html\" frameborder=\"NO\" name=\"navigation\">\r\n",
        "  <frame src=\"papers_frame.html\" frameborder=\"NO\" name=\"view_window\">\r\n",
        "</frameset>\r\n",
        "<noframes>\r\n",
        "Sorry, this document can be viewed only with a frame-capable browser.\r\n",
        "Back to the <a href=\"index.html\">Home Page</a> \r\n",
        "</noframes>\r\n",
        "</html>\r\n",
        "\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Parsing the metadata and textdata files"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Where to go next:  tempting to go implement the index crawling described in:\n",
      "\n",
      "http://commoncrawl.org/common-crawl-url-index/\n",
      "\n",
      "index itself:  The index itself is located public datasets bucket at s3://aws-publicdatasets/common-crawl/projects/url-index/url-index.1356128792.\n",
      "\n",
      "Lot more to explore at https://github.com/trivio/common_crawl_index"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "metadata and text files"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# example -- let's look at a the structure of a metadata file\n",
      "# grab 'common-crawl/parse-output/segment/1346823845675/metadata-00000'\n",
      "\n",
      "k = bucket.get_key('common-crawl/parse-output/segment/1346823845675/metadata-00000')\n",
      "k.size"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "pyout",
       "prompt_number": 11,
       "text": [
        "41857708"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "public URLs -- don't need to generate signature: https://aws-publicdatasets.s3.amazonaws.com/common-crawl/parse-output/segment/1346823845675/metadata-00000"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# easiest way to get file into the local directory -- warning file size is 41857708\n",
      "\n",
      "!wget https://aws-publicdatasets.s3.amazonaws.com/common-crawl/parse-output/segment/1346823845675/metadata-00000"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# alternative -- use boto to download to local file -- this method will be useful if you want grab content from S3\n",
      "fp = open('metadata-00000', 'wb')\n",
      "k.get_file(fp)\n",
      "fp.close()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The metadata and textdata files in Common Crawl are [Hadoop sequences files](http://wiki.apache.org/hadoop/SequenceFile), specifically, https://github.com/matteobertozzi/Hadoop/blob/master/python-hadoop/examples/SequenceFileReader.py.  To parse them, we will use library from https://github.com/matteobertozzi/Hadoop -- here are some installation instructions.  (I've installed these libraries on the PiCloud `rdhyee/Working_with_Open_Data` environment\n",
      "\n",
      "    git clone git://github.com/matteobertozzi/Hadoop.git\n",
      "    cd Hadoop/python-hadoop\n",
      "    python setup.py install\n",
      "\n",
      "\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import sys\n",
      "import json\n",
      "from hadoop.io import SequenceFile\n",
      "from itertools import islice\n",
      "\n",
      "\n",
      "def SequenceFileIterator(path):\n",
      "    reader = SequenceFile.Reader(path)\n",
      "\n",
      "    key_class = reader.getKeyClass()\n",
      "    value_class = reader.getValueClass()\n",
      "\n",
      "    key = key_class()\n",
      "    value = value_class()\n",
      "\n",
      "    position = reader.getPosition()\n",
      "\n",
      "    while reader.next(key, value):\n",
      "        yield (position, key.toString(), value.toString())\n",
      "        position = reader.getPosition()\n",
      "\n",
      "    reader.close()    \n",
      "    \n",
      "\n",
      "path = \"metadata-00000\"\n",
      "\n",
      "# read parts of the metdata-0000 file \n",
      "for (i, (pos, k, v)) in enumerate(islice(SequenceFileIterator(path), 1)):\n",
      "    v = json.loads(v)\n",
      "    archiveInfo = v.get('archiveInfo', None)\n",
      "    print i, k, archiveInfo\n",
      "    print \"metadata available:\", v.keys()\n",
      "    \n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0 http://www.museo-cb.com/museo-cb/audio-y-video/frequency-vhs/ {u'arcFileParition': 0, u'compressedSize': 9801, u'arcSourceSegmentId': 1346823845675L, u'arcFileDate': 1346864469604L, u'arcFileOffset': 157}\n",
        "metadata available: [u'download_size', u'disposition', u'http_headers', u'charset_detector', u'content_len', u'attempt_time', u'http_result', u'content', u'archiveInfo', u'server_ip', u'text_simhash', u'charset_detected', u'parsed_as', u'mime_type', u'md5']\n"
       ]
      }
     ],
     "prompt_number": 12
    }
   ],
   "metadata": {}
  }
 ]
}