TopPages.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Cleaning Your Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's take a web access log, and figure out the most-viewed pages on a website from it! Sounds easy, right?\n",
    "\n",
    "Let's set up a regex that lets us parse an Apache access log line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "format_pat= re.compile(\n",
    "    r\"(?P<host>[\\d\\.]+)\\s\"\n",
    "    r\"(?P<identity>\\S*)\\s\"\n",
    "    r\"(?P<user>\\S*)\\s\"\n",
    "    r\"\\[(?P<time>.*?)\\]\\s\"\n",
    "    r'\"(?P<request>.*?)\"\\s'\n",
    "    r\"(?P<status>\\d+)\\s\"\n",
    "    r\"(?P<bytes>\\S*)\\s\"\n",
    "    r'\"(?P<referer>.*?)\"\\s'\n",
    "    r'\"(?P<user_agent>.*?)\"\\s*'\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Here's the full path to the log file I'm analyzing; change this if you want to run this stuff yourself:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "logPath = \"E:\\\\sundog-consult\\\\Udemy\\\\DataScience\\\\access_log.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we'll whip up a little script to extract the URL in each access, and use a dictionary to count up the number of times each one appears. Then we'll sort it and print out the top 20 pages. What could go wrong?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "'dict' object has no attribute 'has_key'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-3-8efc8eba3391>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      8\u001b[0m             \u001b[0mrequest\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0maccess\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'request'\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      9\u001b[0m             \u001b[1;33m(\u001b[0m\u001b[0maction\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mURL\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mprotocol\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mrequest\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 10\u001b[1;33m             \u001b[1;32mif\u001b[0m \u001b[0mURLCounts\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mhas_key\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mURL\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     11\u001b[0m                 \u001b[0mURLCounts\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mURL\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mURLCounts\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mURL\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     12\u001b[0m             \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mAttributeError\u001b[0m: 'dict' object has no attribute 'has_key'"
     ]
    }
   ],
   "source": [
    "URLCounts = {}\n",
    "\n",
    "with open(logPath, \"r\") as f:\n",
    "    for line in (l.rstrip() for l in f):\n",
    "        match= format_pat.match(line)\n",
    "        if match:\n",
    "            access = match.groupdict()\n",
    "            request = access['request']\n",
    "            (action, URL, protocol) = request.split()\n",
    "            if URLCounts.has_key(URL):\n",
    "                URLCounts[URL] = URLCounts[URL] + 1\n",
    "            else:\n",
    "                URLCounts[URL] = 1\n",
    "\n",
    "results = sorted(URLCounts, key=lambda i: int(URLCounts[i]), reverse=True)\n",
    "\n",
    "for result in results[:20]:\n",
    "    print(result + \": \" + str(URLCounts[result]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Hm. The 'request' part of the line is supposed to look something like this:\n",
    "\n",
    "GET /blog/ HTTP/1.1\n",
    "\n",
    "There should be an HTTP action, the URL, and the protocol. But it seems that's not always happening. Let's print out requests that don't contain three items:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['_\\\\xb0ZP\\\\x07tR\\\\xe5']\n",
      "[]\n",
      "[]\n",
      "[]\n",
      "[]\n",
      "[]\n",
      "[]\n",
      "[]\n",
      "[]\n",
      "[]\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "URLCounts = {}\n",
    "\n",
    "with open(logPath, \"r\") as f:\n",
    "    for line in (l.rstrip() for l in f):\n",
    "        match= format_pat.match(line)\n",
    "        if match:\n",
    "            access = match.groupdict()\n",
    "            request = access['request']\n",
    "            fields = request.split()\n",
    "            if (len(fields) != 3):\n",
    "                print(fields)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Huh. In addition to empty fields, there's one that just contains garbage. Well, let's modify our script to check for that case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/xmlrpc.php: 68494\n",
      "/wp-login.php: 1923\n",
      "/: 440\n",
      "/blog/: 138\n",
      "/robots.txt: 123\n",
      "/sitemap_index.xml: 118\n",
      "/post-sitemap.xml: 118\n",
      "/page-sitemap.xml: 117\n",
      "/category-sitemap.xml: 117\n",
      "/orlando-headlines/: 95\n",
      "/san-jose-headlines/: 85\n",
      "http://51.254.206.142/httptest.php: 81\n",
      "/comics-2/: 76\n",
      "/travel/: 74\n",
      "/entertainment/: 72\n",
      "/business/: 70\n",
      "/national-headlines/: 70\n",
      "/world/: 70\n",
      "/weather/: 70\n",
      "/national/: 70\n"
     ]
    }
   ],
   "source": [
    "URLCounts = {}\n",
    "\n",
    "with open(logPath, \"r\") as f:\n",
    "    for line in (l.rstrip() for l in f):\n",
    "        match= format_pat.match(line)\n",
    "        if match:\n",
    "            access = match.groupdict()\n",
    "            request = access['request']\n",
    "            fields = request.split()\n",
    "            if (len(fields) == 3):\n",
    "                URL = fields[1]\n",
    "                if URL in URLCounts:\n",
    "                    URLCounts[URL] = URLCounts[URL] + 1\n",
    "                else:\n",
    "                    URLCounts[URL] = 1\n",
    "\n",
    "results = sorted(URLCounts, key=lambda i: int(URLCounts[i]), reverse=True)\n",
    "\n",
    "for result in results[:20]:\n",
    "    print(result + \": \" + str(URLCounts[result]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "It worked! But, the results don't really make sense. What we really want is pages accessed by real humans looking for news from our little news site. What the heck is xmlrpc.php? A look at the log itself turns up a lot of entries like this:\n",
    "\n",
    "46.166.139.20 - - [05/Dec/2015:05:19:35 +0000] \"POST /xmlrpc.php HTTP/1.0\" 200 370 \"-\" \"Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)\"\n",
    "\n",
    "I'm not entirely sure what the script does, but it points out that we're not just processing GET actions. We don't want POSTS, so let's filter those out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/: 434\n",
      "/blog/: 138\n",
      "/robots.txt: 123\n",
      "/sitemap_index.xml: 118\n",
      "/post-sitemap.xml: 118\n",
      "/page-sitemap.xml: 117\n",
      "/category-sitemap.xml: 117\n",
      "/orlando-headlines/: 95\n",
      "/san-jose-headlines/: 85\n",
      "http://51.254.206.142/httptest.php: 81\n",
      "/comics-2/: 76\n",
      "/travel/: 74\n",
      "/entertainment/: 72\n",
      "/business/: 70\n",
      "/national-headlines/: 70\n",
      "/world/: 70\n",
      "/weather/: 70\n",
      "/national/: 70\n",
      "/defense-sticking-head-sand/: 69\n",
      "/about/: 69\n"
     ]
    }
   ],
   "source": [
    "URLCounts = {}\n",
    "\n",
    "with open(logPath, \"r\") as f:\n",
    "    for line in (l.rstrip() for l in f):\n",
    "        match= format_pat.match(line)\n",
    "        if match:\n",
    "            access = match.groupdict()\n",
    "            request = access['request']\n",
    "            fields = request.split()\n",
    "            if (len(fields) == 3):\n",
    "                (action, URL, protocol) = fields\n",
    "                if (action == 'GET'):\n",
    "                    if URL in URLCounts:\n",
    "                        URLCounts[URL] = URLCounts[URL] + 1\n",
    "                    else:\n",
    "                        URLCounts[URL] = 1\n",
    "\n",
    "results = sorted(URLCounts, key=lambda i: int(URLCounts[i]), reverse=True)\n",
    "\n",
    "for result in results[:20]:\n",
    "    print(result + \": \" + str(URLCounts[result]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "That's starting to look better. But, this is a news site - are people really reading the little blog on it instead of news pages? That doesn't make sense. Let's look at a typical /blog/ entry in the log:\n",
    "\n",
    "54.165.199.171 - - [05/Dec/2015:09:32:05 +0000] \"GET /blog/ HTTP/1.0\" 200 31670 \"-\" \"-\"\n",
    "\n",
    "Hm. Why is the user agent blank? Seems like some sort of malicious scraper or something. Let's figure out what user agents we are dealing with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0): 68484\n",
      "-: 4035\n",
      "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0): 1724\n",
      "W3 Total Cache/0.9.4.1: 468\n",
      "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html): 278\n",
      "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html): 248\n",
      "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36: 158\n",
      "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0: 144\n",
      "Mozilla/5.0 (iPad; CPU OS 8_4 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12H143 Safari/600.1.4: 120\n",
      "Mozilla/5.0 (Linux; Android 5.1.1; SM-G900T Build/LMY47X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36: 47\n",
      "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm): 43\n",
      "Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+): 41\n",
      "Opera/9.80 (Windows NT 6.0) Presto/2.12.388 Version/12.14: 40\n",
      "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots): 27\n",
      "Mozilla/5.0 (Linux; Android 5.1.1; SM-G900T Build/LMY47X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.76 Mobile Safari/537.36: 15\n",
      "Ruby: 15\n",
      "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36: 13\n",
      "Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/): 11\n",
      "Mozilla/5.0 (Windows NT 5.1; rv:6.0.2) Gecko/20100101 Firefox/6.0.2: 11\n",
      "MobileSafari/600.1.4 CFNetwork/711.4.6 Darwin/14.0.0: 10\n",
      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36: 9\n",
      "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots): 9\n",
      "Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/): 7\n",
      "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp): 6\n",
      "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html): 6\n",
      "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1): 4\n",
      "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.28) Gecko/20120306 Firefox/3.6.28 (.NET CLR 3.5.30729): 4\n",
      "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://fulltext.sblog.cz/): 4\n",
      "Mozilla/5.0 zgrab/0.x: 4\n",
      "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.66 Safari/537.36: 4\n",
      "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0: 3\n",
      "Mozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot ): 3\n",
      "Mozilla/4.0 (compatible: FDSE robot): 3\n",
      "Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; 2Pac; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022): 3\n",
      "Mozilla/5.0: 3\n",
      "Mozilla/5.0 (Windows NT 6.1; rv:34.0) Gecko/20100101 Firefox/34.0: 3\n",
      "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.9.168 Version/11.50: 3\n",
      "Mozilla/5.0 (Windows NT 6.2; rv:31.0) Gecko/20100101 Firefox/31.0: 2\n",
      "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36: 2\n",
      "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0: 2\n",
      "Mozilla/5.0 (Windows NT 6.0; rv:31.0) Gecko/20100101 Firefox/31.0: 2\n",
      "Mozilla/5.0 (Windows NT 6.2; rv:24.0) Gecko/20100101 Firefox/24.0: 2\n",
      "Googlebot-Image/1.0: 2\n",
      "Mozilla/5.0 (Windows NT 6.1; rv:28.0) Gecko/20100101 Firefox/28.0: 2\n",
      "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 5.1; Trident/5.0; .NET CLR 2.0.50727; .NET CLR 3.5.30729): 2\n",
      "Mozilla/5.0 (Windows NT 5.1; rv:28.0) Gecko/20100101 Firefox/28.0: 2\n",
      "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0: 2\n",
      "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0: 2\n",
      "Mozilla/5.0 (Windows NT 6.0; rv:29.0) Gecko/20120101 Firefox/29.0: 2\n",
      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36: 2\n",
      "Mozilla/4.0 (compatible; Netcraft Web Server Survey): 2\n",
      "Mozilla/5.0 (Windows NT 5.1; rv:36.0) Gecko/20100101 Firefox/36.0: 2\n",
      "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36: 2\n",
      "Mozilla/5.0 (Windows NT 5.1; rv:2.0.1) Gecko/20100101 Firefox/5.0: 2\n",
      "Mozilla/5.0 (X11; U; FreeBSD x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16: 2\n",
      "netEstate NE Crawler (+http://www.website-datenbank.de/): 2\n",
      "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/5.0; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E): 2\n",
      "Opera/9.80 (Windows NT 6.1); U) Presto/2.10.289 Version/12.02: 2\n",
      "Mozilla/5.0 (Windows NT 6.1; rv:33.0) Gecko/20100101 Firefox/33.0: 1\n",
      "Mozilla/3.0 (compatible; Indy Library): 1\n",
      "Mozilla/5.0 (Windows; U; Windows NT 5.0; fr-FR; rv:0.9.4) Gecko/20011019 Netscape6/6.2: 1\n",
      "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36: 1\n",
      "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko: 1\n",
      "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:42.0) Gecko/20100101 Firefox/42.0: 1\n",
      "NokiaE5-00/SymbianOS/9.1 Series60/3.0 3gpp-gba: 1\n",
      "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0: 1\n",
      "Mozilla/5.0 (Windows NT 5.1; rv:32.0) Gecko/20100101 Firefox/31.0: 1\n",
      "Mozilla/5.0 (Windows NT 6.1; rv:22.0) Gecko/20130405 Firefox/22.0: 1\n",
      "Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows 95) Opera 7.03 [de]: 1\n",
      "Opera/9.80 (Windows NT 6.2; WOW64); U) Presto/2.12.388 Version/12.14: 1\n",
      "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0: 1\n",
      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36 Scanning for research (researchscan.comsys.rwth-aachen.de): 1\n",
      "Telesphoreo: 1\n",
      "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36: 1\n",
      "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php): 1\n",
      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2251.0 Safari/537.36: 1\n",
      "Mozilla/5.0 (X11; U; Linux i686; pl-PL; rv:1.9.0.2) Gecko/20121223 Ubuntu/9.25 (jaunty) Firefox/3.8: 1\n",
      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.59.10 (KHTML, like Gecko) Version/5.1.9 Safari/534.59.10: 1\n",
      "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.3.1.1) Gecko/20101203 Firefox/3.6.12 (.NET CLR 3.5.30309): 1\n",
      "Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Firefox/31.0: 1\n",
      "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36: 1\n",
      "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.0.13) Gecko/2009073022 Firefox/3.0.13 (.NET CLR 3.5.30729): 1\n",
      "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; MAGWJS; rv:11.0) like Gecko: 1\n",
      "Scrapy/1.0.3 (+http://scrapy.org): 1\n",
      "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36: 1\n"
     ]
    }
   ],
   "source": [
    "UserAgents = {}\n",
    "\n",
    "with open(logPath, \"r\") as f:\n",
    "    for line in (l.rstrip() for l in f):\n",
    "        match= format_pat.match(line)\n",
    "        if match:\n",
    "            access = match.groupdict()\n",
    "            agent = access['user_agent']\n",
    "            if agent in UserAgents:\n",
    "                UserAgents[agent] = UserAgents[agent] + 1\n",
    "            else:\n",
    "                UserAgents[agent] = 1\n",
    "\n",
    "results = sorted(UserAgents, key=lambda i: int(UserAgents[i]), reverse=True)\n",
    "\n",
    "for result in results:\n",
    "    print(result + \": \" + str(UserAgents[result]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Yikes! In addition to '-', there are also a million different web robots accessing the site and polluting my data. Filtering out all of them is really hard, but getting rid of the ones significantly polluting my data in this case should be a matter of getting rid of '-', anything containing \"bot\" or \"spider\", and W3 Total Cache."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/: 77\n",
      "/orlando-headlines/: 36\n",
      "/?page_id=34248: 28\n",
      "/wp-content/cache/minify/000000/M9AvyUjVzUstLy7PLErVz8lMKkosqtTPKtYvTi7KLCgpBgA.js: 27\n",
      "/wp-content/cache/minify/000000/lY7dDoIwDIVfiG0KxkfxfnbdKO4HuxICTy-it8Zw15PzfSftzPCckJem-x4qUWArqBPl5mygZLEgyhdOaoxToGyGaiALiOfUnIz0qDLOdSZGE-nOlpc3kopDzrSyavVVt_veb5qSDVhjsQ6dHh_B_eE_z2pYIGJ7iBWKeEio_eT9UQe4xHhDll27mGRryVu_pRc.js: 27\n",
      "/wp-content/cache/minify/000000/M9bPKixNLarUy00szs8D0Zl5AA.js: 27\n",
      "/wp-content/cache/minify/000000/fY45DoAwDAQ_FMvkRQgFA5ZyWLajiN9zNHR0O83MRkyt-pIctqYFJPedKyYzfHg2PzOFiENAzaD07AxcpKmTolORvDjZt8KEfhBUGjZYCf8Fb0fvA1TXCw.css: 25\n",
      "/?author=1: 21\n",
      "/wp-content/cache/minify/000000/hcrRCYAwDAXAhXyEjiQ1YKAh4SVSx3cE7_uG7ASr4M9qg3kGWyk1adklK84LHtRj_My6Y0Pfqcz-AA.js: 20\n",
      "/wp-content/uploads/2014/11/nhn1.png: 19\n",
      "/wp-includes/js/wp-emoji-release.min.js?ver=4.3.1: 17\n",
      "/wp-content/cache/minify/000000/BcGBCQAgCATAiUSaKYSERPk3avzuht4SkBJnt4tHJdqgnPBqKldesTcN1R8.js: 17\n",
      "/wp-login.php: 16\n",
      "/comics-2/: 12\n",
      "/world/: 12\n",
      "/favicon.ico: 10\n",
      "/wp-content/uploads/2014/11/babyblues.jpg: 6\n",
      "/wp-content/uploads/2014/11/jumble.jpg: 6\n",
      "/wp-content/uploads/2014/11/garfield.jpg: 6\n",
      "/wp-content/uploads/2014/11/violentcrime.jpg: 6\n"
     ]
    }
   ],
   "source": [
    "URLCounts = {}\n",
    "\n",
    "with open(logPath, \"r\") as f:\n",
    "    for line in (l.rstrip() for l in f):\n",
    "        match= format_pat.match(line)\n",
    "        if match:\n",
    "            access = match.groupdict()\n",
    "            agent = access['user_agent']\n",
    "            if (not('bot' in agent or 'spider' in agent or \n",
    "                    'Bot' in agent or 'Spider' in agent or\n",
    "                    'W3 Total Cache' in agent or agent =='-')):\n",
    "                request = access['request']\n",
    "                fields = request.split()\n",
    "                if (len(fields) == 3):\n",
    "                    (action, URL, protocol) = fields\n",
    "                    if (action == 'GET'):\n",
    "                        if URL in URLCounts:\n",
    "                            URLCounts[URL] = URLCounts[URL] + 1\n",
    "                        else:\n",
    "                            URLCounts[URL] = 1\n",
    "\n",
    "results = sorted(URLCounts, key=lambda i: int(URLCounts[i]), reverse=True)\n",
    "\n",
    "for result in results[:20]:\n",
    "    print(result + \": \" + str(URLCounts[result]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now, our new problem is that we're getting a bunch of hits on things that aren't web pages. We're not interested in those, so let's filter out any URL that doesn't end in / (all of the pages on my site are accessed in that manner - again this is applying knowledge about my data to the analysis!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/: 77\n",
      "/orlando-headlines/: 36\n",
      "/comics-2/: 12\n",
      "/world/: 12\n",
      "/about/: 4\n",
      "/australia/: 4\n",
      "/weather/: 4\n",
      "/national-headlines/: 3\n",
      "/feed/: 2\n",
      "/sample-page/feed/: 2\n",
      "/science/: 2\n",
      "/technology/: 2\n",
      "/business/: 1\n",
      "/entertainment/: 1\n",
      "/travel/feed/: 1\n",
      "/san-jose-headlines/: 1\n"
     ]
    }
   ],
   "source": [
    "URLCounts = {}\n",
    "\n",
    "with open(logPath, \"r\") as f:\n",
    "    for line in (l.rstrip() for l in f):\n",
    "        match= format_pat.match(line)\n",
    "        if match:\n",
    "            access = match.groupdict()\n",
    "            agent = access['user_agent']\n",
    "            if (not('bot' in agent or 'spider' in agent or \n",
    "                    'Bot' in agent or 'Spider' in agent or\n",
    "                    'W3 Total Cache' in agent or agent =='-')):\n",
    "                request = access['request']\n",
    "                fields = request.split()\n",
    "                if (len(fields) == 3):\n",
    "                    (action, URL, protocol) = fields\n",
    "                    if (URL.endswith(\"/\")):\n",
    "                        if (action == 'GET'):\n",
    "                            if URL in URLCounts:\n",
    "                                URLCounts[URL] = URLCounts[URL] + 1\n",
    "                            else:\n",
    "                                URLCounts[URL] = 1\n",
    "\n",
    "results = sorted(URLCounts, key=lambda i: int(URLCounts[i]), reverse=True)\n",
    "\n",
    "for result in results[:20]:\n",
    "    print(result + \": \" + str(URLCounts[result]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This is starting to look more believable! But if you were to dig even deeper, you'd find that the /feed/ pages are suspect, and some robots are still slipping through. However, it is accurate to say that Orlando news, world news, and comics are the most popular pages accessed by a real human on this day.\n",
    "\n",
    "The moral of the story is - know your data! And always question and scrutinize your results before making decisions based on them. If your business makes a bad decision because you provided an analysis of bad source data, you could get into real trouble.\n",
    "\n",
    "Be sure the decisions you make while cleaning your data are justifiable too - don't strip out data just because it doesn't support the results you want!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Activity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "These results still aren't perfect; URL's that include \"feed\" aren't actually pages viewed by humans. Modify this code further to strip out URL's that include \"/feed\". Even better, extract some log entries for these pages and understand where these views are coming from."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}