ezproxy_intruders_daily.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# EzProxy Intruders : Daily"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***This runs best in Jupyter, either on a local machine or on a server you have file access to.*** \n",
    "\n",
    "To review all users failing to access EzProxy, make sure you place the audit logs into the /data folder and that they are named in the syntax of \"YYYYMMDD.txt\" (for example, \"20190314.txt\"). These audit files are usually in the /audit sub-folder of your EzProxy application folder on the server. Your audit logs will need to be in the following format:\n",
    "\n",
    "> **%h %{ezproxy-session}i %u %t \"%r\" %s %b**\n",
    "\n",
    "Please also maked sure that you place the EzProxy log files in the /data_e folder and that they are named in the syntax of \"ezproxyYYYYMMDD.log\" (for example, \"ezproxy20190101.log\"). These ezproxy files are usually in the /log sub-folder of your EzProxy application folder on the server. Your EzProxy logs will need to be in the following format, which is slightly different from the audit log format:\n",
    "\n",
    "> **%h %{ezproxy-session}i %U %u %t \"%r\" %s %b**\n",
    "\n",
    "Once you have some files in the approprate data folders, *run cells 1 through to 8*. If there are no warnings or errors, then you will be presented with a calendar dropdown menu, from which you can select the date for audting. Once you select a date, it will read the audit log for failed connections and then refresh with a 'username' dropdown, which presents users with the number of blocked intruder sessions they have held on the day in question. From this, you can select a user to see their location.\n",
    "\n",
    "Once you select a username, the program will take a little moment to visualise where the user attempted to log into EzProxy. In normal circumstances, under ten intruder sessions and from the same location can be normal behvaiour, signalling when someone who incorrectly remembers their password has been temporarily blocked by EzProxy after a lot of failed attempts. A high number of failed sessions, sometimes coupled with multiple locations, can be a flag for suspicious behaviour."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Activate all cells"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[NbConvertApp] Converting notebook ezproxy_intruders_daily.ipynb to script\n",
      "[NbConvertApp] Writing 14482 bytes to ezproxy_intruders_daily.py\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "!jupyter nbconvert --to script ezproxy_intruders_daily.ipynb\n",
    "os.rename(\"./ezproxy_intruders_daily.py\", \"./py/ezproxy_intruders_daily.py\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FRESH ANACONDA INSTALL\n",
    "# export PATH=/anaconda3/bin:$PATH\n",
    "# conda config --add channels conda-forge\n",
    "# conda install -c conda-forge proj4\n",
    "# conda install -c anaconda mysql-connector-python\n",
    "# conda install -c conda-forge cartopy\n",
    "# conda install -c conda-forge tldextract\n",
    "# conda install -c conda-forge basemap\n",
    "# conda install -c conda-forge basemap-data-hires\n",
    "# conda install -c conda-forge ipywidgets\n",
    "# conda install -c conda-forge folium\n",
    "# conda install -c conda-forge pyzmq\n",
    "# conda install -c conda-forge jupyterlab\n",
    "# conda install -c conda-forge nodejs\n",
    "# conda install python=3.6.7\n",
    "# jupyter nbextension enable --py widgetsnbextension\n",
    "# jupyter labextension install @jupyter-widgets/jupyterlab-manager\n",
    "# jupyter lab build"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import random\n",
    "import re\n",
    "import sys\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib\n",
    "import mysql.connector\n",
    "import matplotlib.dates as mdates\n",
    "import cartopy.crs as ccrs\n",
    "os.environ['PROJ_LIB'] = '/anaconda3/share/proj'\n",
    "import requests\n",
    "import json\n",
    "import time\n",
    "import csv\n",
    "import tldextract\n",
    "from datetime import datetime, timedelta, date\n",
    "from ipywidgets import interact, interactive, interact_manual, Button, HBox, VBox, Layout, ButtonStyle\n",
    "import ipywidgets as widgets\n",
    "from IPython.display import display, clear_output, HTML\n",
    "from mpl_toolkits.basemap import Basemap\n",
    "import folium\n",
    "from folium.plugins import MarkerCluster\n",
    "pd.set_option('display.max_colwidth', -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def on_date(change):\n",
    "    global ddown\n",
    "    global audit\n",
    "    global thisDate2\n",
    "    global sessDate\n",
    "    global audits\n",
    "    global audits_dict\n",
    "    global thisDate\n",
    "    today = date.today()\n",
    "    today = int(today.strftime(\"%Y%m%d\"))\n",
    "    thisDate = aDates.value\n",
    "    thisDate2 = str(aDates.value)\n",
    "    thisDate3 = int(thisDate.strftime(\"%Y%m%d\"))\n",
    "    sessDate = thisDate.strftime(\"%Y%m%d\")\n",
    "    if thisDate3  > today:\n",
    "        thisDate = date.today()\n",
    "    thisDate = \"./data/\" + thisDate.strftime(\"%Y%m%d\") + \".txt\"\n",
    "    audit = pd.read_csv(thisDate,sep='\\t')\n",
    "    audit = audit[audit['Event'].str.contains(\"Login.Intruder\")]\n",
    "    del audit['Date/Time']\n",
    "    del audit['Other']\n",
    "    del audit['Event']\n",
    "    audit = audit[pd.notnull(audit['IP'])]\n",
    "    audits = audit.groupby('Username').size()\n",
    "    audits = pd.DataFrame({'Username':audits.index, 'Access':audits.values})\n",
    "    audits = audits[audits.Access > 1]\n",
    "    audits = audits.sort_values(by='Access',ascending=False)\n",
    "    audits['Action'] = audits.Username.map(str)+\" -- \"+audits.Access.map(str)\n",
    "    audits_dict = dict(zip(audits.Action,audits.Username,))\n",
    "    with outB:\n",
    "        clear_output()\n",
    "        ddown = widgets.Dropdown(\n",
    "            options = audits_dict,\n",
    "            description = 'Usernames',\n",
    "            disabled=False,\n",
    "            value=None,\n",
    "            rows=5\n",
    "        )\n",
    "        ddown.observe(on_user,names='value')\n",
    "        display(ddown)\n",
    "    with outC:\n",
    "        clear_output()\n",
    "        sessdown = widgets.Button(description=\"View World\",layout=Layout(width=\"270px\"))\n",
    "        display(sessdown)\n",
    "        sessdown.on_click(on_world)\n",
    "    with outD:\n",
    "        clear_output()\n",
    "    with outG:\n",
    "        clear_output()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def on_world(b):\n",
    "    global audit3\n",
    "    global ipaudits\n",
    "    global logX\n",
    "    global ipr\n",
    "    global ipc\n",
    "    with outD:\n",
    "        clear_output()\n",
    "        ipr = []\n",
    "        audit3 = pd.DataFrame(audit.groupby(['IP'], as_index=False)['IP'].agg(['count']))\n",
    "        for row in audit3.index:\n",
    "            ipr.append(row)\n",
    "        ipr = list(set(ipr))\n",
    "        thisFile = \"./outputs/fail_world_\" + thisDate2 + \"_log.csv\"\n",
    "        with open(thisFile, mode='w') as audit_file:\n",
    "            audit_writer = csv.writer(audit_file, delimiter=',', quotechar='\"', quoting=csv.QUOTE_MINIMAL)\n",
    "            audit_writer.writerow([\"IP\",\"lat\",\"lon\",\"city\",\"dsize\",\"continent_name\",\n",
    "                                   \"threat_is_tor\",\"threat_is_proxy\",\n",
    "                                   \"threat_is_anonymous\",\"threat_is_known_attacker\",\n",
    "                                   \"threat_is_known_abuser\",\"threat_is_threat\",\"threat_is_bogon\"])\n",
    "            z = 0\n",
    "            for x in ipr:\n",
    "                url = \"http://ip-api.com/json/\" + x\n",
    "                #apikey = \"\"\n",
    "                #url = \"https://api.ipdata.co/\" + x + \"?api-key=\" + apikey\n",
    "                r = requests.get(url)\n",
    "                results = r.json()\n",
    "                if len(results) > 1:\n",
    "                    lat = results['lat']\n",
    "                    lon = results['lon']\n",
    "                    city = results['city']\n",
    "                    #lat = results['latitude']\n",
    "                    #lon = results['longitude']\n",
    "                    #continent_name = results['continent_name']\n",
    "                    #threat_is_tor = results['threat']['is_tor']\n",
    "                    #threat_is_proxy = results['threat']['is_proxy']\n",
    "                    #threat_is_anonymous = results['threat']['is_anonymous']\n",
    "                    #threat_is_known_attacker = results['threat']['is_known_attacker']\n",
    "                    #threat_is_known_abuser = results['threat']['is_known_abuser']\n",
    "                    #threat_is_threat = results['threat']['is_threat']\n",
    "                    #threat_is_bogon = results['threat']['is_bogon']\n",
    "                    dsize = \"10\"\n",
    "                    continent_name = \"\"\n",
    "                    threat_is_tor = \"\"\n",
    "                    threat_is_proxy = \"\"\n",
    "                    threat_is_anonymous = \"\"\n",
    "                    threat_is_known_attacker = \"\"\n",
    "                    threat_is_known_abuser = \"\"\n",
    "                    threat_is_threat = \"\"\n",
    "                    threat_is_bogon = \"\"\n",
    "                    audit_writer.writerow([x,lat,lon,city,dsize,continent_name,\n",
    "                                           threat_is_tor,threat_is_proxy,threat_is_anonymous,\n",
    "                                           threat_is_known_attacker,threat_is_known_abuser,\n",
    "                                           threat_is_threat,threat_is_bogon])\n",
    "                z = z + 1\n",
    "                time.sleep(1.0)\n",
    "                #time.sleep(0.2)                \n",
    "        logs = pd.read_csv(thisFile)\n",
    "        logZ = logs\n",
    "        os.remove(thisFile)\n",
    "        plt.figure(figsize=(17,17))\n",
    "        ax = plt.axes(projection=ccrs.PlateCarree())\n",
    "        ax.set_title('EzProxy Fails | World | ' + thisDate2,y=1.08)\n",
    "        ax.set_global()\n",
    "        ax.coastlines(linewidth=0.6)\n",
    "        ax.stock_img()\n",
    "        ax.gridlines(xlocs=range(-180,181,40), ylocs=range(-80,81,20),draw_labels=False)\n",
    "        ax.gridlines(xlocs=range(-140,181,40), ylocs=range(-80,81,20),draw_labels=True)\n",
    "        ax.text(-0.05,0,'Latitude', transform=ax.transAxes, rotation='vertical', va='bottom')\n",
    "        ax.text(0,-0.07,'Longitude', transform=ax.transAxes, ha='left')\n",
    "        for index, row in logZ.iterrows():\n",
    "            lat = row['lat']\n",
    "            lon = row['lon']\n",
    "            latx = lat - 1.5\n",
    "            lonx = lon + 3.5\n",
    "            city = row['city']\n",
    "            ds = row['dsize']\n",
    "            ax.plot(lon, lat, marker='o', markersize=10, markerfacecolor='#FF0000')\n",
    "        #plt.savefig('./imgs/ezproxy_fails_world_'+thisDate2+'_log.png', bbox_inches = \"tight\")\n",
    "        plt.show();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "global aDates\n",
    "now = datetime.utcnow() - timedelta(days=1)\n",
    "aDates = widgets.DatePicker(\n",
    "    description='Audit Date',\n",
    "    disabled=False,\n",
    "#    value=datetime(now.year,now.month,now.day)\n",
    ")\n",
    "aDates.observe(on_date,names='value')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def on_user(change):\n",
    "    global sessdown\n",
    "    global sessions\n",
    "    global ipaddresses\n",
    "    global ips\n",
    "    global users\n",
    "    global audit2\n",
    "    global dataZ2\n",
    "    global logZ\n",
    "    with outG:\n",
    "        clear_output()\n",
    "        thisUser = ddown.value\n",
    "        audit2 = audit[audit.Username == thisUser]\n",
    "        user = []\n",
    "        ipaddresses = []\n",
    "        sessions = []\n",
    "        for index, row in audit2.iterrows():\n",
    "            if row['IP'] != \"\":\n",
    "                aa = row['IP']\n",
    "                bb = row['Username']\n",
    "                cc = row['Session']\n",
    "                ipaddresses.append(aa)\n",
    "                user.append(bb)\n",
    "                sessions.append(cc)\n",
    "        ips = ipaddresses \n",
    "        ips = list(set(ips))\n",
    "        users = [x for x in user if not pd.isnull(x)]\n",
    "        users = list(set(users))\n",
    "        thisFile = \"./outputs/fail_\" + users[0] + \"_\" + thisDate2 + \"_log.csv\"\n",
    "        with open(thisFile, mode='w') as audit_file:\n",
    "            audit_writer = csv.writer(audit_file, delimiter=',', quotechar='\"', quoting=csv.QUOTE_MINIMAL)\n",
    "            audit_writer.writerow([\"IP\",\"lat\",\"lon\",\"city\",\"dsize\",\"continent_name\",\n",
    "                                   \"threat_is_tor\",\"threat_is_proxy\",\n",
    "                                   \"threat_is_anonymous\",\"threat_is_known_attacker\",\n",
    "                                   \"threat_is_known_abuser\",\"threat_is_threat\",\"threat_is_bogon\"])\n",
    "            z = 0\n",
    "            for x in ips:\n",
    "                url = \"http://ip-api.com/json/\" + x\n",
    "                #apikey = \"\"\n",
    "                #url = \"https://api.ipdata.co/\" + x + \"?api-key=\" + apikey\n",
    "                r = requests.get(url)\n",
    "                results = r.json()\n",
    "                if len(results) > 1:\n",
    "                    lat = results['lat']\n",
    "                    lon = results['lon']\n",
    "                    city = results['city']\n",
    "                    #lat = results['latitude']\n",
    "                    #lon = results['longitude']\n",
    "                    #continent_name = results['continent_name']\n",
    "                    #threat_is_tor = results['threat']['is_tor']\n",
    "                    #threat_is_proxy = results['threat']['is_proxy']\n",
    "                    #threat_is_anonymous = results['threat']['is_anonymous']\n",
    "                    #threat_is_known_attacker = results['threat']['is_known_attacker']\n",
    "                    #threat_is_known_abuser = results['threat']['is_known_abuser']\n",
    "                    #threat_is_threat = results['threat']['is_threat']\n",
    "                    #threat_is_bogon = results['threat']['is_bogon']\n",
    "                    dsize = \"\"\n",
    "                    continent_name = \"\"\n",
    "                    threat_is_tor = \"\"\n",
    "                    threat_is_proxy = \"\"\n",
    "                    threat_is_anonymous = \"\"\n",
    "                    threat_is_known_attacker = \"\"\n",
    "                    threat_is_known_abuser = \"\"\n",
    "                    threat_is_threat = \"\"\n",
    "                    threat_is_bogon = \"\"\n",
    "                    audit_writer.writerow([x,lat,lon,city,dsize,continent_name,\n",
    "                                           threat_is_tor,threat_is_proxy,threat_is_anonymous,\n",
    "                                           threat_is_known_attacker,threat_is_known_abuser,\n",
    "                                           threat_is_threat,threat_is_bogon])\n",
    "                z = z + 1\n",
    "                time.sleep(1.0)\n",
    "                #time.sleep(0.2)                \n",
    "        logs = pd.read_csv(thisFile)\n",
    "        logZ = logs\n",
    "        os.remove(thisFile)\n",
    "        c = logZ.shape[0]\n",
    "        if c > 0:\n",
    "            plt.figure(figsize=(18, 15))\n",
    "            m = Basemap(projection=\"lcc\", width=9E6, height=5E6, lat_0=logZ['lat'][0], lon_0=logZ['lon'][0])\n",
    "            m.shadedrelief()\n",
    "            lat = logZ['lat']\n",
    "            lon = logZ['lon']\n",
    "            for i in range(0,len(lat)):\n",
    "                x,y = m(lon[i],lat[i])\n",
    "                m.plot(x, y, 'or', markersize=15, alpha=0.8)\n",
    "            #plt.savefig('./imgs/ezproxy_fails_'+users[0]+'_'+thisDate2+'_log.png', bbox_inches = \"tight\")\n",
    "            plt.show();\n",
    "        else:\n",
    "            print(\"Private IP Range\")\n",
    "            print(ips)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "68c85ee4c9a84a4f91e2997054c6ec45",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output(layout=Layout(border='0px solid #777777', height='2.3em', padding='0px', width='99%'))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7595d683b1f34ab78fc711d909cd31a6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(Output(layout=Layout(border='0px solid #777777', height='2.3em', padding='0px', width='310px'))…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "628513266a204e5ea772269323683763",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output(layout=Layout(border='0px solid #777777', height='600px', padding='0px', top='25px', width='99%'))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "18c1c0775db44833b7f5024a0f7eb2e8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output(layout=Layout(border='0px solid #777777', height='600px', padding='0px', top='25px', width='99%'))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "outZ = widgets.Output(layout={'border': '0px solid #777777', 'height':'2.3em', 'padding': '0px', 'width':'99%'})\n",
    "outA = widgets.Output(layout={'border': '0px solid #777777', 'height':'2.3em', 'padding': '0px', 'width':'310px'})\n",
    "outB = widgets.Output(layout={'border': '0px solid #777777', 'height':'2.3em', 'padding': '0px', 'width':'310px'})\n",
    "outC = widgets.Output(layout={'border': '0px solid #777777', 'height':'2.3em', 'padding': '0px', 'width':'310px', 'left': '30px'})\n",
    "outG = widgets.Output(layout={'border': '0px solid #777777', 'height':'600px', 'padding': '0px', 'width':'99%', 'top':'25px'})\n",
    "outD = widgets.Output(layout={'border': '0px solid #777777', 'height':'600px', 'padding': '0px', 'width':'99%', 'top':'25px'})\n",
    "interface = HBox([outA,outB,outC])\n",
    "display(outZ)\n",
    "display(interface)\n",
    "display(outG)\n",
    "display(outD)\n",
    "with outA:\n",
    "    clear_output()\n",
    "    display(aDates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}