ezproxy_daily_users.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# EzProxy Daily : Users : Successful Logins"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***This runs best in Jupyter, either on a local machine or on a server you have file access to.*** \n",
    "\n",
    "To review users accessing EzProxy and whether they are chaining multiple sessions, make sure you place the audit logs into the /data folder and that they are named in the syntax of \"YYYYMMDD.txt\" (for example, \"20190314.txt\"). These audit files are usually in the /audit sub-folder of your EzProxy application folder on the server. Your audit logs will need to be in the following format:\n",
    "\n",
    "> **%h %{ezproxy-session}i %u %t \"%r\" %s %b**\n",
    "\n",
    "Please also maked sure that you place the EzProxy log files in the /data_e folder and that they are named in the syntax of \"ezproxyYYYYMMDD.log\" (for example, \"ezproxy20190101.log\"). These ezproxy files are usually in the /log sub-folder of your EzProxy application folder on the server. Your EzProxy logs will need to be in the following format, which is slightly different from the audit log format:\n",
    "\n",
    "> **%h %{ezproxy-session}i %U %u %t \"%r\" %s %b**\n",
    "\n",
    "Once you have some files in the approprate data folders, *run cells 1 through to 8*. If there are no warnings or errors, then you will be presented with a calendar dropdown menu, from which you can select the date for audting. Once you select a date, it will read the audit log and then refresh with a 'username' dropdown, which presents users with the number of sessions they have held on the day in question. From this, you can select a user to review their activity based on the number of sessions they have generated. Usual bahaviour rarely goes into double digits; suspicious behaviour is always into double digits and often stands out from the rest of the list. In the image below, there are two users which warranted a review: one was legitimate (left side), the other was not legitimate and the user had to be contacted after their username was blocked (right side)\n",
    "\n",
    "![User Behaviour: Good on left, Bad on right](./docs/user_behaviour_reduced.jpg)\n",
    "\n",
    "Once you select a username, the program will take a little moment to visualise where the user logged into EzProxy and where the user operated from after their logging in session was authenticated by EzProxy. In normal circumstances, both locations will match but in suspicious circumstances you may find that there are differences in locations or one location for logging in and multiple locations for general use. The latter scenario is usually indicative of the session being proxied out to other users (which sites like 'pubmed007' often do). You can verify this by clicking on the 'View Activity' button. This will generate a high-level list of resources accessed and other sites appearing in the logs. It is in the 'Other Sites' list where you will see any sites like 'pubmed007' or '2447.net'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Activate all cells"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[NbConvertApp] Converting notebook ezproxy_daily_users.ipynb to script\n",
      "[NbConvertApp] Writing 21599 bytes to ezproxy_daily_users.py\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "!jupyter nbconvert --to script ezproxy_daily_users.ipynb\n",
    "os.rename(\"./ezproxy_daily_users.py\", \"./py/ezproxy_daily_users.py\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FRESH ANACONDA INSTALL\n",
    "# export PATH=/anaconda3/bin:$PATH\n",
    "# conda config --add channels conda-forge\n",
    "# conda install -c conda-forge proj4\n",
    "# conda install -c anaconda mysql-connector-python\n",
    "# conda install -c conda-forge cartopy\n",
    "# conda install -c conda-forge tldextract\n",
    "# conda install -c conda-forge basemap\n",
    "# conda install -c conda-forge basemap-data-hires\n",
    "# conda install -c conda-forge ipywidgets\n",
    "# conda install -c conda-forge folium\n",
    "# conda install -c conda-forge pyzmq\n",
    "# conda install -c conda-forge jupyterlab\n",
    "# conda install -c conda-forge nodejs\n",
    "# conda install python=3.6.7\n",
    "# jupyter nbextension enable --py widgetsnbextension\n",
    "# jupyter labextension install @jupyter-widgets/jupyterlab-manager\n",
    "# jupyter lab build"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import random\n",
    "import re\n",
    "import sys\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib\n",
    "import mysql.connector\n",
    "import matplotlib.dates as mdates\n",
    "import cartopy.crs as ccrs\n",
    "os.environ['PROJ_LIB'] = '/anaconda3/share/proj'\n",
    "import requests\n",
    "import json\n",
    "import time\n",
    "import csv\n",
    "import tldextract\n",
    "from datetime import datetime, timedelta, date\n",
    "from ipywidgets import interact, interactive, interact_manual, Button, HBox, VBox, Layout, ButtonStyle\n",
    "import ipywidgets as widgets\n",
    "from IPython.display import display, clear_output, HTML\n",
    "from mpl_toolkits.basemap import Basemap\n",
    "pd.set_option('display.max_colwidth', -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def on_date(change):\n",
    "    global ddown\n",
    "    global audit\n",
    "    global thisDate2\n",
    "    global sessDate\n",
    "    global audits\n",
    "    global audits_dict\n",
    "    global thisDate\n",
    "    today = date.today()\n",
    "    today = int(today.strftime(\"%Y%m%d\"))\n",
    "    thisDate = aDates.value\n",
    "    thisDate2 = str(aDates.value)\n",
    "    thisDate3 = int(thisDate.strftime(\"%Y%m%d\"))\n",
    "    sessDate = thisDate.strftime(\"%Y%m%d\")\n",
    "    if thisDate3  > today:\n",
    "        thisDate = date.today()\n",
    "    thisDate = \"./data/\" + thisDate.strftime(\"%Y%m%d\") + \".txt\"\n",
    "    audit = pd.read_csv(thisDate,sep='\\t')\n",
    "    audit[\"is_duplicate\"] = audit.duplicated(['Username'])\n",
    "    audit = audit[audit.Event == \"Login.Success\"]\n",
    "    audit = audit[audit.is_duplicate == True]\n",
    "    del audit['Date/Time']\n",
    "    del audit['Other']\n",
    "    del audit['Event']\n",
    "    audit = audit[pd.notnull(audit['IP'])]\n",
    "    audits = audit.groupby('Username').size()\n",
    "    audits = pd.DataFrame({'Username':audits.index, 'Access':audits.values})\n",
    "    audits = audits[audits.Access > 1]\n",
    "    audits = audits.sort_values(by='Access',ascending=False)\n",
    "    #audits = audits.sort_values(by='Username',ascending=False)\n",
    "    audits['Action'] = audits.Username.map(str)+\" -- \"+audits.Access.map(str)\n",
    "    audits_dict = dict(zip(audits.Action,audits.Username,))\n",
    "    with outB:\n",
    "        clear_output()\n",
    "        ddown = widgets.Dropdown(\n",
    "            options = audits_dict,\n",
    "            description = 'Usernames',\n",
    "            disabled=False,\n",
    "            value=None,\n",
    "            rows=5\n",
    "        )\n",
    "        ddown.observe(on_user,names='value')\n",
    "        display(ddown)\n",
    "    with outC:\n",
    "        clear_output()\n",
    "    with outD:\n",
    "        clear_output()\n",
    "    with outE:\n",
    "        clear_output()\n",
    "    with outF:\n",
    "        clear_output()\n",
    "    with outG:\n",
    "        clear_output()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "global aDates\n",
    "now = datetime.utcnow() - timedelta(days=1)\n",
    "aDates = widgets.DatePicker(\n",
    "    description='Audit Date',\n",
    "    disabled=False,\n",
    "#    value=datetime(now.year,now.month,now.day)\n",
    ")\n",
    "aDates.observe(on_date,names='value')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def on_user(change):\n",
    "    global sessdown\n",
    "    global sessions\n",
    "    global ipaddresses\n",
    "    global ips\n",
    "    global users\n",
    "    global audit2\n",
    "    global dataZ2\n",
    "    global logZ\n",
    "    with outC:\n",
    "        clear_output()\n",
    "    with outF:\n",
    "        clear_output()\n",
    "    with outE:\n",
    "        clear_output()\n",
    "    with outD:\n",
    "        clear_output()\n",
    "    with outG:\n",
    "        clear_output()\n",
    "    #Audit data\n",
    "    thisUser = ddown.value\n",
    "    audit2 = audit[audit.Username == thisUser]\n",
    "    user = []\n",
    "    ipaddresses = []\n",
    "    sessions = []\n",
    "    #Log data\n",
    "    dataZ = pd.read_csv(\"./data_e/ezproxy\" + sessDate + \".log\", sep=\" \", header=None, error_bad_lines=False, warn_bad_lines=False)\n",
    "    dataZ.columns = [\"ipaddress\", \"sessionid\", \"url\", \"urlsessionid\", \"adate\", \"azone\", \"adomain\", \"astatus\", \"asize\"]\n",
    "    dataZ2 = pd.DataFrame(dataZ)\n",
    "    dataZ2 = dataZ2[dataZ2.urlsessionid == thisUser]\n",
    "    dataZ3 = dataZ2.groupby(['ipaddress'], as_index=False)['ipaddress'].agg(['count'])\n",
    "    ips2 = dataZ3.index.tolist()\n",
    "    ips2 = list(set(ips2))\n",
    "    for index, row in audit2.iterrows():\n",
    "        if row['IP'] != \"\" and row['is_duplicate'] is True:\n",
    "            aa = row['IP']\n",
    "            bb = row['Username']\n",
    "            cc = row['Session']\n",
    "            ipaddresses.append(aa)\n",
    "            user.append(bb)\n",
    "            sessions.append(cc)\n",
    "    ips = [x for x in ipaddresses if not pd.isnull(x)] \n",
    "    ips = list(set(ips))\n",
    "    #ips2 = [x for x in ips2 if x not in ips]\n",
    "    users = [x for x in user if not pd.isnull(x)]\n",
    "    users = list(set(users))\n",
    "    thisFile = \"./outputs/\" + users[0] + \"_\" + thisDate2 + \"_log.csv\"\n",
    "    exists = os.path.isfile(thisFile)\n",
    "    if exists:\n",
    "        myLog = \"Audit log CSV exists\"\n",
    "    else:\n",
    "        with open(thisFile, mode='w') as audit_file:\n",
    "            audit_writer = csv.writer(audit_file, delimiter=',', quotechar='\"', quoting=csv.QUOTE_MINIMAL)\n",
    "            audit_writer.writerow([\"IP\",\"lat\",\"lon\",\"city\",\"dsize\",\"continent_name\",\n",
    "                                   \"threat_is_tor\",\"threat_is_proxy\",\n",
    "                                   \"threat_is_anonymous\",\"threat_is_known_attacker\",\n",
    "                                   \"threat_is_known_abuser\",\"threat_is_threat\",\"threat_is_bogon\"])\n",
    "            z = 0\n",
    "            for x in ips:\n",
    "                url = \"http://ip-api.com/json/\" + x\n",
    "                #apikey = \"\"\n",
    "                #url = \"https://api.ipdata.co/\" + x + \"?api-key=\" + apikey\n",
    "                r = requests.get(url)\n",
    "                results = r.json()\n",
    "                if len(results) > 1:\n",
    "                    lat = results['lat']\n",
    "                    lon = results['lon']\n",
    "                    city = results['city']\n",
    "                    #lat = results['latitude']\n",
    "                    #lon = results['longitude']\n",
    "                    #continent_name = results['continent_name']\n",
    "                    #threat_is_tor = results['threat']['is_tor']\n",
    "                    #threat_is_proxy = results['threat']['is_proxy']\n",
    "                    #threat_is_anonymous = results['threat']['is_anonymous']\n",
    "                    #threat_is_known_attacker = results['threat']['is_known_attacker']\n",
    "                    #threat_is_known_abuser = results['threat']['is_known_abuser']\n",
    "                    #threat_is_threat = results['threat']['is_threat']\n",
    "                    #threat_is_bogon = results['threat']['is_bogon']\n",
    "                    dsize = \"\"\n",
    "                    continent_name = \"\"\n",
    "                    threat_is_tor = \"\"\n",
    "                    threat_is_proxy = \"\"\n",
    "                    threat_is_anonymous = \"\"\n",
    "                    threat_is_known_attacker = \"\"\n",
    "                    threat_is_known_abuser = \"\"\n",
    "                    threat_is_threat = \"\"\n",
    "                    threat_is_bogon = \"\"\n",
    "                    audit_writer.writerow([x,lat,lon,city,dsize,continent_name,\n",
    "                                           threat_is_tor,threat_is_proxy,threat_is_anonymous,\n",
    "                                           threat_is_known_attacker,threat_is_known_abuser,\n",
    "                                           threat_is_threat,threat_is_bogon])\n",
    "                z = z + 1\n",
    "                time.sleep(1.0)\n",
    "                #time.sleep(0.2)\n",
    "            for x in ips2:\n",
    "                url = \"http://ip-api.com/json/\" + x\n",
    "                #apikey = \"\"\n",
    "                #url = \"https://api.ipdata.co/\" + x + \"?api-key=\" + apikey\n",
    "                r = requests.get(url)\n",
    "                results = r.json()\n",
    "                if len(results) > 1:\n",
    "                    lat = results['lat']\n",
    "                    lon = results['lon']\n",
    "                    city = results['city']\n",
    "                    #lat = results['latitude']\n",
    "                    #lon = results['longitude']\n",
    "                    #continent_name = results['continent_name']\n",
    "                    #threat_is_tor = results['threat']['is_tor']\n",
    "                    #threat_is_proxy = results['threat']['is_proxy']\n",
    "                    #threat_is_anonymous = results['threat']['is_anonymous']\n",
    "                    #threat_is_known_attacker = results['threat']['is_known_attacker']\n",
    "                    #threat_is_known_abuser = results['threat']['is_known_abuser']\n",
    "                    #threat_is_threat = results['threat']['is_threat']\n",
    "                    #threat_is_bogon = results['threat']['is_bogon']\n",
    "                    dsize = \"\"\n",
    "                    continent_name = \"\"\n",
    "                    threat_is_tor = \"\"\n",
    "                    threat_is_proxy = \"\"\n",
    "                    threat_is_anonymous = \"\"\n",
    "                    threat_is_known_attacker = \"\"\n",
    "                    threat_is_known_abuser = \"\"\n",
    "                    threat_is_threat = \"\"\n",
    "                    threat_is_bogon = \"\"\n",
    "                    audit_writer.writerow([x,lat,lon,city,dsize,continent_name,\n",
    "                                           threat_is_tor,threat_is_proxy,threat_is_anonymous,\n",
    "                                           threat_is_known_attacker,threat_is_known_abuser,\n",
    "                                           threat_is_threat,threat_is_bogon])\n",
    "                z = z + 1\n",
    "                time.sleep(1.0)\n",
    "                #time.sleep(0.2)               \n",
    "        myLog = \"Audit log CSV created\"\n",
    "    logs = pd.read_csv(thisFile)\n",
    "    logZ = logs\n",
    "    os.remove(thisFile)\n",
    "    # Projection one\n",
    "    plt.figure(figsize=(18, 15))\n",
    "    m = Basemap(projection=\"lcc\", width=9E6, height=5E6, lat_0=logZ['lat'][0], lon_0=logZ['lon'][0])\n",
    "    m.shadedrelief()\n",
    "    lat = logZ['lat']\n",
    "    lon = logZ['lon']\n",
    "    for i in range(0,len(lat)-1):\n",
    "        x,y = m(lon[i],lat[i])\n",
    "        m.plot(x, y, 'or', markersize=15, alpha=0.4)\n",
    "    #plt.savefig('./imgs/ezproxy_intrusion_'+users[0]+'_'+thisDate2+'_log.png', bbox_inches = \"tight\")\n",
    "    with outD:\n",
    "        clear_output()\n",
    "        plt.show();\n",
    "    # Projection two\n",
    "    plt.figure(figsize=(17,17))\n",
    "    ax = plt.axes(projection=ccrs.PlateCarree())\n",
    "    ax.set_title('EzProxy User | ' + users[0] + ' | ' + thisDate2,y=1.08)\n",
    "    ax.set_global()\n",
    "    ax.coastlines(linewidth=0.6)\n",
    "    ax.stock_img()\n",
    "    ax.gridlines(xlocs=range(-180,181,40), ylocs=range(-80,81,20),draw_labels=False)\n",
    "    ax.gridlines(xlocs=range(-140,181,40), ylocs=range(-80,81,20),draw_labels=True)\n",
    "    ax.text(-0.05,0,'Latitude', transform=ax.transAxes, rotation='vertical', va='bottom')\n",
    "    ax.text(0,-0.07,'Longitude', transform=ax.transAxes, ha='left')\n",
    "    for index, row in logs.iterrows():\n",
    "        lat = row['lat']\n",
    "        lon = row['lon']\n",
    "        latx = lat - 1.5\n",
    "        lonx = lon + 3.5\n",
    "        city = row['city']\n",
    "        ds = row['dsize']\n",
    "        if ds != \"L\":\n",
    "            ax.plot(lon, lat, marker='o', markersize=10, markerfacecolor='#FF0000')\n",
    "        else:\n",
    "            msg = \"Do not plot\"\n",
    "        #ax.text(lonx, latx, city, fontsize=12, color='black')\n",
    "    #plt.savefig('./imgs/ezproxy_intrusion_'+users[0]+'_'+thisDate2+'_audit.png', bbox_inches = \"tight\")\n",
    "    with outC:\n",
    "        clear_output()\n",
    "        sessdown = widgets.Button(description=\"View Activity\",layout=Layout(width=\"270px\"))\n",
    "        display(sessdown)\n",
    "        sessdown.on_click(on_platform)\n",
    "    with outF:\n",
    "        clear_output()\n",
    "    with outE:\n",
    "        clear_output()\n",
    "    with outG:\n",
    "        clear_output()\n",
    "        plt.show();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def on_platform(b):\n",
    "    global errorsA\n",
    "    global errorsB\n",
    "    global errorsC\n",
    "    global data3\n",
    "    errorsA = []\n",
    "    errorsB = []\n",
    "    errorsC = []\n",
    "    thisUser = ddown.value\n",
    "    with outE:\n",
    "        clear_output()\n",
    "        print(\"\\t\" + str(sessDate) + \" / \" + thisUser + \" : This may take a minute ...\",end=\"\\n\\n\")\n",
    "    with outF:\n",
    "        clear_output()\n",
    "    try:\n",
    "        with outE:\n",
    "            print(\"\\tLoading log ...\",end=\"\\n\\n\")\n",
    "        data = pd.read_csv(\"./data_e/ezproxy\" + sessDate + \".log\", sep=\" \", header=None, error_bad_lines=False, warn_bad_lines=False)\n",
    "        data.columns = [\"ipaddress\", \"sessionid\", \"url\", \"urlsessionid\", \"adate\", \"azone\", \"adomain\", \"astatus\", \"asize\"]\n",
    "        #data2 = data\n",
    "        data2 = pd.DataFrame(data[data[\"sessionid\"].isin(sessions)])\n",
    "        data2.reset_index(drop=True, inplace=True)\n",
    "        with outE:\n",
    "            print(\"\\tParsing sessions ...\",end=\"\\n\\n\")\n",
    "        data3 = {\"ipaddress\":[], \"sessionid\":[], \"url\":[], \"time\":[], \"size\":[]}\n",
    "    except ValueError as e:\n",
    "        errorsA.append(\"Catch 1 E: \" + str(e))\n",
    "    except IOError as e:\n",
    "        errorsA.append(\"Catch 1a E: \" + str(e))\n",
    "    except:\n",
    "        errorsA.append(\"Catch 1b E: \", sys.exc_info()[0])\n",
    "    try:\n",
    "        with outE:\n",
    "            print(\"\\tIterate over rows ...\",end=\"\\n\\n\")\n",
    "        for i in data2.index:\n",
    "            adate = data2.get_value(i,'adate')\n",
    "            asize = data2.get_value(i,'asize')\n",
    "            sessionid = data2.get_value(i,'sessionid')\n",
    "            ipaddress = data2.get_value(i,'ipaddress')\n",
    "            domain = data2.get_value(i,'url')\n",
    "            matches = re.search(\"login\", domain)\n",
    "            matchesb = re.search(\"http[s]?://ezproxy.uws.edu.au\", domain)\n",
    "            adatex, datex = re.split(\"\\[\", adate)\n",
    "            datex = pd.to_datetime(datex, format=\"%d/%b/%Y:%H:%M:%S\")\n",
    "            datex = datex.strftime('%H:%M')\n",
    "            if domain is not None and domain != \"\" and domain != \"-\" and matches is None and matchesb is None:\n",
    "                try:\n",
    "                    dom, domy = re.split(\".ezproxy.uws.edu.au\", domain)\n",
    "                except ValueError as e:\n",
    "                    domain = re.sub(\"\\-\",\".\",str(domain))\n",
    "                    extracted = tldextract.extract(domain)\n",
    "                    domain = extracted.domain\n",
    "                    errorsB.append(str(domain))\n",
    "                    dom = None\n",
    "                except IOError as e:\n",
    "                    errorsB.append(\"Catch 2 E: \" + str(e))\n",
    "                    dom = None\n",
    "                except:\n",
    "                    errorsB.append(\"Catch 2a E: \", sys.exc_info()[0])\n",
    "                    dom = None\n",
    "            else:\n",
    "                if domain is not None and domain != \"\" and domain != \"-\":\n",
    "                    domain = re.sub(\"\\-\",\".\",str(domain))\n",
    "                    extracted = tldextract.extract(domain)\n",
    "                    domain = extracted.domain\n",
    "                    errorsB.append(str(domain))\n",
    "                dom = None\n",
    "            try:\n",
    "                if str(sessionid) != \"-\" and str(sessionid) != \"\" and str(dom) != \"\" and str(dom) != \"None\":\n",
    "                    dom = re.sub(\"\\-\",\".\",str(dom))\n",
    "                    extracted = tldextract.extract(dom)\n",
    "                    dom = extracted.domain\n",
    "                    data3[\"ipaddress\"].append(str(ipaddress))\n",
    "                    data3[\"sessionid\"].append(str(sessionid))\n",
    "                    data3[\"url\"].append(str(dom))\n",
    "                    data3[\"time\"].append(datex)\n",
    "                    data3[\"size\"].append(asize)\n",
    "                else:\n",
    "                    if matches is not None or matchesb is not None:\n",
    "                        errorsB.append(\"ezproxy\")\n",
    "                    else:\n",
    "                        domain = re.sub(\"\\-\",\".\",str(data2.get_value(i,'url')))\n",
    "                        extracted = tldextract.extract(domain)\n",
    "                        domain = extracted.domain\n",
    "                        errorsB.append(str(domain))\n",
    "            except ValueError as v:\n",
    "                errorsB.append(\"Catch 2b E: \" + str(v))\n",
    "            except IOError as v:\n",
    "                errorsB.append(\"Catch 2c E: \" + str(v))\n",
    "            except:\n",
    "                errorsB.append(\"Catch 2d E: sess \" + str(type(sessionid)) + \" dom \" + str(type(dom)) + \" error \" + str(sys.exc_info()[0]))\n",
    "    except ValueError as z:\n",
    "        errorsC.append(\"Catch 3 E: \" + str(z))\n",
    "    except IOError as z:\n",
    "        errorsC.append(\"Catch 3b E: \" + str(z))\n",
    "    except:\n",
    "        errorsC.append(\"Catch 3c E: \", sys.exc_info()[0])\n",
    "    with outE:\n",
    "        print(\"\\n\\n\\tGrouping and counting results\",end=\"\\n\\n\")\n",
    "    try:\n",
    "        df = pd.DataFrame(data3)\n",
    "        df2 = df[df.sessionid.isin(sessions)]\n",
    "        df3 = df2.groupby(['url'], as_index=False)['size'].agg(['sum','count'])\n",
    "    except ValueError as e:\n",
    "        errorsC.append(\"Catch 3d E: \" + str(e))\n",
    "    except IOError as e:\n",
    "        errorsC.append(\"Catch 3e E: \" + str(e))\n",
    "    except:\n",
    "        errorsC.append(\"Catch 3f E: \", sys.exc_info()[0])\n",
    "    with outE:\n",
    "        clear_output()\n",
    "        display(HTML('<h4>Vendor Resources</h4>'))\n",
    "        display(df3)\n",
    "    dfObj = pd.DataFrame(errorsB, columns = ['url'])\n",
    "    dfObj2 = dfObj.groupby(['url'], as_index=False)['url'].agg(['count'])\n",
    "    dfObj2.drop(dfObj2[dfObj2['count'] < 2].index, inplace=True)\n",
    "    dfObj2 = dfObj2.sort_values('count',ascending=False)\n",
    "    with pd.option_context('display.max_rows', None, 'display.max_columns', None):\n",
    "        with outF:\n",
    "            clear_output()\n",
    "            display(HTML('<h4>Other Sites<br />&nbsp;</h4>'))\n",
    "            display(dfObj2)\n",
    "            try:\n",
    "                if max(data3[\"time\"]) != \"\":\n",
    "                    display(HTML('<h4>Most Recent Access<br />&nbsp;</h4>'))\n",
    "                    display(max(data3[\"time\"]))\n",
    "                else:\n",
    "                    display(HTML('<h4>No Recent Access<br />&nbsp;</h4>'))\n",
    "            except:\n",
    "                myError = \"An error\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "779ba3f2e09f48cdba0d208816b5a152",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output(layout=Layout(border='0px solid #777777', height='2.3em', padding='0px', width='99%'))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6658f459603c46dfbfd7a3db24989fd4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(Output(layout=Layout(border='0px solid #777777', height='2.3em', padding='0px', width='310px'))…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "58660fd1d6f0463ea2d32dfecb93ba99",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output(layout=Layout(border='0px solid #777777', height='600px', padding='0px', top='25px', width='99%'))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0b5f3a8edbe1482b9b070c8e3cca27b8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output(layout=Layout(border='0px solid #777777', height='600px', padding='0px', top='25px', width='99%'))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c83b953c553547a4af1fa5d89d1a4e7e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(Output(layout=Layout(border='0px solid #777777', height='500px', left='30px', overflow_y='auto'…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "outZ = widgets.Output(layout={'border': '0px solid #777777', 'height':'2.3em', 'padding': '0px', 'width':'99%'})\n",
    "outA = widgets.Output(layout={'border': '0px solid #777777', 'height':'2.3em', 'padding': '0px', 'width':'310px'})\n",
    "outB = widgets.Output(layout={'border': '0px solid #777777', 'height':'2.3em', 'padding': '0px', 'width':'310px'})\n",
    "outC = widgets.Output(layout={'border': '0px solid #777777', 'height':'2.3em', 'padding': '0px', 'width':'310px', 'left': '30px'})\n",
    "outD = widgets.Output(layout={'border': '0px solid #777777', 'height':'600px', 'padding': '0px', 'width':'99%', 'top':'25px'})\n",
    "outG = widgets.Output(layout={'border': '0px solid #777777', 'height':'600px', 'padding': '0px', 'width':'99%', 'top':'25px'})\n",
    "outE = widgets.Output(layout={'border': '0px solid #777777', 'height':'500px', 'padding': '0px', 'width':'495px', 'top':'35px', 'overflow_y':'auto', 'left': '30px'})\n",
    "outF = widgets.Output(layout={'border-left': '1px solid #777777', 'height':'500px', 'padding': '0px', 'width':'495px', 'top':'35px', 'overflow_y':'auto'})\n",
    "interface = HBox([outA,outB,outC])\n",
    "interfaceb = HBox([outE,outF])\n",
    "display(outZ)\n",
    "display(interface)\n",
    "display(outG)\n",
    "display(outD)\n",
    "display(interfaceb)\n",
    "with outA:\n",
    "    clear_output()\n",
    "    display(aDates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}