docs/notebooks/GeoIPLookups.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Title: msticpy - GeoIP Lookup\n",
    "\n",
    "## Introduction\n",
    "This module contains two classes that allow you to look up the Geolocation of IP Addresses.\n",
    "\n",
    "You must have msticpy installed to run this notebook:\n",
    "```\n",
    "%pip install --upgrade msticpy\n",
    "```\n",
    "\n",
    "\n",
    "### MaxMind GeoIPLite\n",
    "This product includes GeoLite2 data created by MaxMind, available from\n",
    "<a>https://www.maxmind.com</a>.\n",
    "\n",
    "This uses a local database which is downloaded first time when class object is instantiated. It gives very fast lookups but you need to download updates regularly. Maxmind offers a free tier of this database, updated monthly. For greater accuracy and more detailed information they have varying levels of paid service. Please check out their site for more details.\n",
    "\n",
    "The geoip module uses official maxmind pypi package - geoip2 and also has options to customize the behavior of local maxmind database.\n",
    "* ```db_folder``` : Specify custom path containing local maxmind city database. If not specified, download to .msticpy dir under user\\`s home dir.\n",
    "* ```force_update``` : can be set to True/False to issue force update despite of age check.\n",
    "* Check age of maxmind city database based on database info and download new if it is not updated in last 30 days.\n",
    "* ``auto_update``` : can be set to True/False Allow option to override auto update database if user is desired not to update database older than 30 days.\n",
    "\n",
    "### IPStack\n",
    "This library uses services provided by ipstack.\n",
    "<a>https://ipstack.com</a>\n",
    "\n",
    "IPStack is an online service and also offers a free tier of their service. Again, the paid tiers offer greater accuracy, more detailed information and higher throughput. Please check out their site for more details.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a></a>\n",
    "## Table of Contents\n",
    "- [Maxmind GeoIP Lookup](#geoip_lookups)\n",
    "- [IPStack GeoIP Lookup](#ipstack_lookups)\n",
    "- [Dataframe input](#dataframe_input)\n",
    "- [Creating your own GeoIP Class](#custom_lookup)\n",
    "- [Calculating Geographical Distances](#calc_distance)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-08T02:38:51.269375Z",
     "start_time": "2020-02-08T02:38:49.282504Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Imports\n",
    "import sys\n",
    "MIN_REQ_PYTHON = (3,6)\n",
    "if sys.version_info < MIN_REQ_PYTHON:\n",
    "    print('Check the Kernel->Change Kernel menu and ensure that Python 3.6')\n",
    "    print('or later is selected as the active kernel.')\n",
    "    sys.exit(\"Python %s.%s or later is required.\\n\" % MIN_REQ_PYTHON)\n",
    "\n",
    "\n",
    "from IPython.display import display\n",
    "import pandas as pd\n",
    "\n",
    "import msticpy.sectools as sectools\n",
    "from msticpy.nbtools import *\n",
    "from msticpy.nbtools.entityschema import IpAddress, GeoLocation\n",
    "from msticpy.sectools.geoip import GeoLiteLookup, IPStackLookup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a></a>[Contents](#contents)\n",
    "## Maxmind GeoIP Lite Lookup Class\n",
    "Signature:\n",
    "```\n",
    "iplocation.lookup_ip(ip_address: str = None, \n",
    "                     ip_addr_list: collections.abc.Iterable = None,\n",
    "                     ip_entity: msticpy.nbtools.entityschema.IpAddress = None)\n",
    "Docstring:\n",
    "Lookup IP location from GeoLite2 data created by MaxMind.\n",
    "\n",
    "Keyword Arguments:\n",
    "    ip_address {str} -- a single address to look up (default: {None})\n",
    "    ip_addr_list {Iterable} -- a collection of addresses to lookup (default: {None})\n",
    "    ip_entity {IpAddress} -- an IpAddress entity\n",
    "\n",
    "Returns:\n",
    "    tuple(list{dict}, list{entity}) -- returns raw geolocation results and\n",
    "        same results as IP/Geolocation entities\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-08T02:38:51.294362Z",
     "start_time": "2020-02-08T02:38:51.270375Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Raw result\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'continent': {'code': 'EU',\n",
       "   'geoname_id': 6255148,\n",
       "   'names': {'de': 'Europa',\n",
       "    'en': 'Europe',\n",
       "    'es': 'Europa',\n",
       "    'fr': 'Europe',\n",
       "    'ja': 'ヨーロッパ',\n",
       "    'pt-BR': 'Europa',\n",
       "    'ru': 'Европа',\n",
       "    'zh-CN': '欧洲'}},\n",
       "  'country': {'geoname_id': 2017370,\n",
       "   'iso_code': 'RU',\n",
       "   'names': {'de': 'Russland',\n",
       "    'en': 'Russia',\n",
       "    'es': 'Rusia',\n",
       "    'fr': 'Russie',\n",
       "    'ja': 'ロシア',\n",
       "    'pt-BR': 'Rússia',\n",
       "    'ru': 'Россия',\n",
       "    'zh-CN': '俄罗斯联邦'}},\n",
       "  'location': {'accuracy_radius': 1000,\n",
       "   'latitude': 55.7386,\n",
       "   'longitude': 37.6068,\n",
       "   'time_zone': 'Europe/Moscow'},\n",
       "  'registered_country': {'geoname_id': 2017370,\n",
       "   'iso_code': 'RU',\n",
       "   'names': {'de': 'Russland',\n",
       "    'en': 'Russia',\n",
       "    'es': 'Rusia',\n",
       "    'fr': 'Russie',\n",
       "    'ja': 'ロシア',\n",
       "    'pt-BR': 'Rússia',\n",
       "    'ru': 'Россия',\n",
       "    'zh-CN': '俄罗斯联邦'}},\n",
       "  'traits': {'ip_address': '90.156.201.97', 'prefix_len': 17}}]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IP Address Entity\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<h3>ipaddress</h3>{&nbsp;'AdditionalData':&nbsp;{},<br>&nbsp;&nbsp;'Address':&nbsp;'90.156.201.97',<br>&nbsp;&nbsp;'Location':&nbsp;{&nbsp;'AdditionalData':&nbsp;{},<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'CountryCode':&nbsp;'RU',<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'CountryName':&nbsp;'Russia',<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'Latitude':&nbsp;55.7386,<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'Longitude':&nbsp;37.6068,<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'Type':&nbsp;'geolocation',<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'edges':&nbsp;set()},<br>&nbsp;&nbsp;'ThreatIntelligence':&nbsp;[],<br>&nbsp;&nbsp;'Type':&nbsp;'ipaddress',<br>&nbsp;&nbsp;'edges':&nbsp;set()}"
      ],
      "text/plain": [
       "IpAddress(Address=90.156.201.97, Location={ 'AdditionalData': {},\n",
       "  'CountryCode': 'RU',\n",
       " ...)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "iplocation = GeoLiteLookup()\n",
    "loc_result, ip_entity = iplocation.lookup_ip(ip_address='90.156.201.97')\n",
    "\n",
    "print('Raw result')\n",
    "display(loc_result)\n",
    "\n",
    "print('IP Address Entity')\n",
    "display(ip_entity[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-08T02:38:51.311353Z",
     "start_time": "2020-02-08T02:38:51.296360Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Raw result\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'continent': {'code': 'EU',\n",
       "   'geoname_id': 6255148,\n",
       "   'names': {'de': 'Europa',\n",
       "    'en': 'Europe',\n",
       "    'es': 'Europa',\n",
       "    'fr': 'Europe',\n",
       "    'ja': 'ヨーロッパ',\n",
       "    'pt-BR': 'Europa',\n",
       "    'ru': 'Европа',\n",
       "    'zh-CN': '欧洲'}},\n",
       "  'country': {'geoname_id': 2017370,\n",
       "   'iso_code': 'RU',\n",
       "   'names': {'de': 'Russland',\n",
       "    'en': 'Russia',\n",
       "    'es': 'Rusia',\n",
       "    'fr': 'Russie',\n",
       "    'ja': 'ロシア',\n",
       "    'pt-BR': 'Rússia',\n",
       "    'ru': 'Россия',\n",
       "    'zh-CN': '俄罗斯联邦'}},\n",
       "  'location': {'accuracy_radius': 1000,\n",
       "   'latitude': 55.7386,\n",
       "   'longitude': 37.6068,\n",
       "   'time_zone': 'Europe/Moscow'},\n",
       "  'registered_country': {'geoname_id': 2017370,\n",
       "   'iso_code': 'RU',\n",
       "   'names': {'de': 'Russland',\n",
       "    'en': 'Russia',\n",
       "    'es': 'Rusia',\n",
       "    'fr': 'Russie',\n",
       "    'ja': 'ロシア',\n",
       "    'pt-BR': 'Rússia',\n",
       "    'ru': 'Россия',\n",
       "    'zh-CN': '俄罗斯联邦'}},\n",
       "  'traits': {'ip_address': '90.156.201.97', 'prefix_len': 17}}]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IP Address Entity\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<h3>ipaddress</h3>{&nbsp;'AdditionalData':&nbsp;{},<br>&nbsp;&nbsp;'Address':&nbsp;'90.156.201.97',<br>&nbsp;&nbsp;'Location':&nbsp;{&nbsp;'AdditionalData':&nbsp;{},<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'CountryCode':&nbsp;'RU',<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'CountryName':&nbsp;'Russia',<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'Latitude':&nbsp;55.7386,<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'Longitude':&nbsp;37.6068,<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'Type':&nbsp;'geolocation',<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'edges':&nbsp;set()},<br>&nbsp;&nbsp;'ThreatIntelligence':&nbsp;[],<br>&nbsp;&nbsp;'Type':&nbsp;'ipaddress',<br>&nbsp;&nbsp;'edges':&nbsp;set()}"
      ],
      "text/plain": [
       "IpAddress(Address=90.156.201.97, Location={ 'AdditionalData': {},\n",
       "  'CountryCode': 'RU',\n",
       " ...)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import tempfile\n",
    "from pathlib import Path\n",
    "tmp_folder = tempfile.gettempdir()\n",
    "iplocation = GeoLiteLookup(db_folder=str(Path(tmp_folder).joinpath('geolite')))\n",
    "loc_result, ip_entity = iplocation.lookup_ip(ip_address='90.156.201.97')\n",
    "\n",
    "print('Raw result')\n",
    "display(loc_result)\n",
    "\n",
    "print('IP Address Entity')\n",
    "display(ip_entity[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-08T02:38:54.589392Z",
     "start_time": "2020-02-08T02:38:51.312351Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "force_update is set to True. Attempting to download new database to C:\\Users\\Ian\\.msticpy\\GeoLite2\n",
      "Downloading and extracting GeoLite DB archive from MaxMind....\n",
      "Raw result\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "e:\\src\\microsoft\\msticpy\\msticpy\\sectools\\geoip.py:609: UserWarning: Error writing GeoIP DB file: C:\\Users\\Ian\\.msticpy\\GeoLite2\\GeoLite2-City.mmdb - [Errno 22] Invalid argument: 'C:\\\\Users\\\\Ian\\\\.msticpy\\\\GeoLite2\\\\GeoLite2-City.mmdb'\n",
      "  warnings.warn(f\"Error writing GeoIP DB file: {db_file_path} - {err}\")\n",
      "e:\\src\\microsoft\\msticpy\\msticpy\\sectools\\geoip.py:536: UserWarning: DB download failed\n",
      "  warnings.warn(\"DB download failed\")\n",
      "e:\\src\\microsoft\\msticpy\\msticpy\\sectools\\geoip.py:540: UserWarning: Continuing with cached database. Results may inaccurate.\n",
      "  \"Continuing with cached database. Results may inaccurate.\"\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'continent': {'code': 'EU',\n",
       "   'geoname_id': 6255148,\n",
       "   'names': {'de': 'Europa',\n",
       "    'en': 'Europe',\n",
       "    'es': 'Europa',\n",
       "    'fr': 'Europe',\n",
       "    'ja': 'ヨーロッパ',\n",
       "    'pt-BR': 'Europa',\n",
       "    'ru': 'Европа',\n",
       "    'zh-CN': '欧洲'}},\n",
       "  'country': {'geoname_id': 2017370,\n",
       "   'iso_code': 'RU',\n",
       "   'names': {'de': 'Russland',\n",
       "    'en': 'Russia',\n",
       "    'es': 'Rusia',\n",
       "    'fr': 'Russie',\n",
       "    'ja': 'ロシア',\n",
       "    'pt-BR': 'Rússia',\n",
       "    'ru': 'Россия',\n",
       "    'zh-CN': '俄罗斯联邦'}},\n",
       "  'location': {'accuracy_radius': 1000,\n",
       "   'latitude': 55.7386,\n",
       "   'longitude': 37.6068,\n",
       "   'time_zone': 'Europe/Moscow'},\n",
       "  'registered_country': {'geoname_id': 2017370,\n",
       "   'iso_code': 'RU',\n",
       "   'names': {'de': 'Russland',\n",
       "    'en': 'Russia',\n",
       "    'es': 'Rusia',\n",
       "    'fr': 'Russie',\n",
       "    'ja': 'ロシア',\n",
       "    'pt-BR': 'Rússia',\n",
       "    'ru': 'Россия',\n",
       "    'zh-CN': '俄罗斯联邦'}},\n",
       "  'traits': {'ip_address': '90.156.201.97', 'prefix_len': 17}}]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IP Address Entity\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<h3>ipaddress</h3>{&nbsp;'AdditionalData':&nbsp;{},<br>&nbsp;&nbsp;'Address':&nbsp;'90.156.201.97',<br>&nbsp;&nbsp;'Location':&nbsp;{&nbsp;'AdditionalData':&nbsp;{},<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'CountryCode':&nbsp;'RU',<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'CountryName':&nbsp;'Russia',<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'Latitude':&nbsp;55.7386,<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'Longitude':&nbsp;37.6068,<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'Type':&nbsp;'geolocation',<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'edges':&nbsp;set()},<br>&nbsp;&nbsp;'ThreatIntelligence':&nbsp;[],<br>&nbsp;&nbsp;'Type':&nbsp;'ipaddress',<br>&nbsp;&nbsp;'edges':&nbsp;set()}"
      ],
      "text/plain": [
       "IpAddress(Address=90.156.201.97, Location={ 'AdditionalData': {},\n",
       "  'CountryCode': 'RU',\n",
       " ...)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "iplocation = GeoLiteLookup(force_update=True)\n",
    "loc_result, ip_entity = iplocation.lookup_ip(ip_address='90.156.201.97')\n",
    "\n",
    "print('Raw result')\n",
    "display(loc_result)\n",
    "\n",
    "print('IP Address Entity')\n",
    "display(ip_entity[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-08T02:38:54.601359Z",
     "start_time": "2020-02-08T02:38:54.590367Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Raw result\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'continent': {'code': 'EU',\n",
       "   'geoname_id': 6255148,\n",
       "   'names': {'de': 'Europa',\n",
       "    'en': 'Europe',\n",
       "    'es': 'Europa',\n",
       "    'fr': 'Europe',\n",
       "    'ja': 'ヨーロッパ',\n",
       "    'pt-BR': 'Europa',\n",
       "    'ru': 'Европа',\n",
       "    'zh-CN': '欧洲'}},\n",
       "  'country': {'geoname_id': 2017370,\n",
       "   'iso_code': 'RU',\n",
       "   'names': {'de': 'Russland',\n",
       "    'en': 'Russia',\n",
       "    'es': 'Rusia',\n",
       "    'fr': 'Russie',\n",
       "    'ja': 'ロシア',\n",
       "    'pt-BR': 'Rússia',\n",
       "    'ru': 'Россия',\n",
       "    'zh-CN': '俄罗斯联邦'}},\n",
       "  'location': {'accuracy_radius': 1000,\n",
       "   'latitude': 55.7386,\n",
       "   'longitude': 37.6068,\n",
       "   'time_zone': 'Europe/Moscow'},\n",
       "  'registered_country': {'geoname_id': 2017370,\n",
       "   'iso_code': 'RU',\n",
       "   'names': {'de': 'Russland',\n",
       "    'en': 'Russia',\n",
       "    'es': 'Rusia',\n",
       "    'fr': 'Russie',\n",
       "    'ja': 'ロシア',\n",
       "    'pt-BR': 'Rússia',\n",
       "    'ru': 'Россия',\n",
       "    'zh-CN': '俄罗斯联邦'}},\n",
       "  'traits': {'ip_address': '90.156.201.97', 'prefix_len': 17}}]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IP Address Entity\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<h3>ipaddress</h3>{&nbsp;'AdditionalData':&nbsp;{},<br>&nbsp;&nbsp;'Address':&nbsp;'90.156.201.97',<br>&nbsp;&nbsp;'Location':&nbsp;{&nbsp;'AdditionalData':&nbsp;{},<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'CountryCode':&nbsp;'RU',<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'CountryName':&nbsp;'Russia',<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'Latitude':&nbsp;55.7386,<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'Longitude':&nbsp;37.6068,<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'Type':&nbsp;'geolocation',<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'edges':&nbsp;set()},<br>&nbsp;&nbsp;'ThreatIntelligence':&nbsp;[],<br>&nbsp;&nbsp;'Type':&nbsp;'ipaddress',<br>&nbsp;&nbsp;'edges':&nbsp;set()}"
      ],
      "text/plain": [
       "IpAddress(Address=90.156.201.97, Location={ 'AdditionalData': {},\n",
       "  'CountryCode': 'RU',\n",
       " ...)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "iplocation = GeoLiteLookup(auto_update=False)\n",
    "loc_result, ip_entity = iplocation.lookup_ip(ip_address='90.156.201.97')\n",
    "\n",
    "print('Raw result')\n",
    "display(loc_result)\n",
    "\n",
    "print('IP Address Entity')\n",
    "display(ip_entity[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-08T02:38:54.612353Z",
     "start_time": "2020-02-08T02:38:54.603359Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['151.101.128.223', '151.101.192.223', '151.101.0.223', '151.101.64.223']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[IpAddress(Address=151.101.128.223, Location={ 'AdditionalData': {},\n",
       "   'CountryCode': 'US',...),\n",
       " IpAddress(Address=151.101.192.223, Location={ 'AdditionalData': {},\n",
       "   'CountryCode': 'US',...),\n",
       " IpAddress(Address=151.101.0.223, Location={ 'AdditionalData': {},\n",
       "   'CountryCode': 'US',\n",
       "  ...),\n",
       " IpAddress(Address=151.101.64.223, Location={ 'AdditionalData': {},\n",
       "   'CountryCode': 'US',\n",
       " ...)]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import socket\n",
    "socket_info = socket.getaddrinfo(\"pypi.org\",0,0,0,0)\n",
    "\n",
    "ips = [res[4][0] for res in socket_info]\n",
    "print(ips)\n",
    "\n",
    "_, ip_entities = iplocation.lookup_ip(ip_addr_list=ips)\n",
    "display(ip_entities)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a></a>[Contents](#contents)\n",
    "## IPStack Geo-lookup Class\n",
    "\n",
    "#### Class Initialization\n",
    "\n",
    "Note - requires IPStack API Key, Optional parameter bulk_lookup allows multiple IPs in a single request. This is only available with the paid Professional tier and above.\n",
    "```\n",
    "Init signature: IPStackLookup(api_key: str, bulk_lookup: bool = False)\n",
    "Docstring:     \n",
    "GeoIP Lookup using IPStack web service.\n",
    "\n",
    "Raises:\n",
    "    ConnectionError -- Invalid status returned from http request\n",
    "    PermissionError -- Service refused request (e.g. requesting batch of addresses\n",
    "            on free tier API key)\n",
    "Init docstring:\n",
    "Create a new instance of IPStackLookup.\n",
    "\n",
    "Arguments:\n",
    "    api_key {str} -- API Key from IPStack - see https://ipstack.com\n",
    "    bulk_lookup {bool} -- For Professional and above tiers allowing you to\n",
    "        submit multiple IPs in a single request.\n",
    "        \n",
    "```\n",
    "\n",
    "#### lookup_ip method\n",
    "```\n",
    "Signature:\n",
    "iplocation.lookup_ip(\n",
    "    ['ip_address: str = None', 'ip_addr_list: collections.abc.Iterable = None', 'ip_entity: msticpy.nbtools.entityschema.IpAddress = None'],\n",
    ") -&gt; tuple\n",
    "Docstring:\n",
    "Lookup IP location from IPStack web service.\n",
    "\n",
    "Keyword Arguments:\n",
    "    ip_address {str} -- a single address to look up (default: {None})\n",
    "    ip_addr_list {Iterable} -- a collection of addresses to lookup (default: {None})\n",
    "    ip_entity {IpAddress} -- an IpAddress entity\n",
    "\n",
    "Raises:\n",
    "    ConnectionError -- Invalid status returned from http request\n",
    "    PermissionError -- Service refused request (e.g. requesting batch of addresses\n",
    "        on free tier API key)\n",
    "\n",
    "Returns:\n",
    "    tuple(list{dict}, list{entity}) -- returns raw geolocation results and\n",
    "        same results as IP/Geolocation entities\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a></a>[Contents](#contents)\n",
    "### You will need a IPStack API key\n",
    "You will get more detailed results and a higher throughput allowance if you have a paid tier. See IPStack website for more details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-08T02:38:54.643336Z",
     "start_time": "2020-02-08T02:38:54.613352Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "This library uses services provided by ipstack.\n",
       "<a href=\"https://ipstack.com\">https://ipstack.com</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fd2ab27181d94ec7be794001870018dc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HTML(value='To obtain an API key sign up here https://www.ipstack.com/')"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "iplocation = IPStackLookup()\n",
    "# Enter your IPStack Key here (if not set in msticpyconfig.yaml)\n",
    "ips_key = nbwidgets.GetEnvironmentKey(env_var='IPSTACK_AUTH',\n",
    "                           help_str='To obtain an API key sign up here https://www.ipstack.com/',\n",
    "                           prompt='IPStack API key:')\n",
    "if not iplocation.settings.args.get(\"AuthKey\"):\n",
    "    ips_key.display()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-08T02:38:54.761206Z",
     "start_time": "2020-02-08T02:38:54.646333Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Raw result\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[({'ip': '90.156.201.97',\n",
       "   'type': 'ipv4',\n",
       "   'continent_code': 'AS',\n",
       "   'continent_name': 'Asia',\n",
       "   'country_code': 'RU',\n",
       "   'country_name': 'Russia',\n",
       "   'region_code': 'MOW',\n",
       "   'region_name': 'Moscow',\n",
       "   'city': 'Moscow',\n",
       "   'zip': '115088',\n",
       "   'latitude': 55.712608337402344,\n",
       "   'longitude': 37.68056869506836,\n",
       "   'location': {'geoname_id': 524901,\n",
       "    'capital': 'Moscow',\n",
       "    'languages': [{'code': 'ru', 'name': 'Russian', 'native': 'Русский'}],\n",
       "    'country_flag': 'http://assets.ipstack.com/flags/ru.svg',\n",
       "    'country_flag_emoji': '🇷🇺',\n",
       "    'country_flag_emoji_unicode': 'U+1F1F7 U+1F1FA',\n",
       "    'calling_code': '7',\n",
       "    'is_eu': False}},\n",
       "  200)]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IP Address Entity\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<h3>ipaddress</h3>{&nbsp;'AdditionalData':&nbsp;{},<br>&nbsp;&nbsp;'Address':&nbsp;'90.156.201.97',<br>&nbsp;&nbsp;'Location':&nbsp;{&nbsp;'AdditionalData':&nbsp;{},<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'City':&nbsp;'Moscow',<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'CountryCode':&nbsp;'RU',<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'CountryName':&nbsp;'Russia',<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'Latitude':&nbsp;55.712608337402344,<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'Longitude':&nbsp;37.68056869506836,<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'State':&nbsp;'Moscow',<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'Type':&nbsp;'geolocation',<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'edges':&nbsp;set()},<br>&nbsp;&nbsp;'ThreatIntelligence':&nbsp;[],<br>&nbsp;&nbsp;'Type':&nbsp;'ipaddress',<br>&nbsp;&nbsp;'edges':&nbsp;set()}"
      ],
      "text/plain": [
       "IpAddress(Address=90.156.201.97, Location={ 'AdditionalData': {},\n",
       "  'City': 'Moscow',\n",
       "  'C...)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import os\n",
    "if not iplocation.settings.args.get(\"AuthKey\") and not ips_key.value:\n",
    "    raise ValueError(\"No Authentication key in config/environment or supplied by user.\")\n",
    "if ips_key.value:\n",
    "    iplocation = IPStackLookup(api_key=ips_key.value)\n",
    "\n",
    "if \"MSTICPY_SKIP_IPSTACK_TEST\" not in os.environ:\n",
    "    loc_result, ip_entity = iplocation.lookup_ip(ip_address='90.156.201.97')\n",
    "    print('Raw result')\n",
    "    display(loc_result)\n",
    "\n",
    "    print('IP Address Entity')\n",
    "    display(ip_entity[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-08T02:38:55.196405Z",
     "start_time": "2020-02-08T02:38:54.762206Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Raw results\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[({'ip': '151.101.128.223',\n",
       "   'type': 'ipv4',\n",
       "   'continent_code': 'NA',\n",
       "   'continent_name': 'North America',\n",
       "   'country_code': 'US',\n",
       "   'country_name': 'United States',\n",
       "   'region_code': 'CA',\n",
       "   'region_name': 'California',\n",
       "   'city': 'San Francisco',\n",
       "   'zip': '94107',\n",
       "   'latitude': 37.76784896850586,\n",
       "   'longitude': -122.39286041259766,\n",
       "   'location': {'geoname_id': 5391959,\n",
       "    'capital': 'Washington D.C.',\n",
       "    'languages': [{'code': 'en', 'name': 'English', 'native': 'English'}],\n",
       "    'country_flag': 'http://assets.ipstack.com/flags/us.svg',\n",
       "    'country_flag_emoji': '🇺🇸',\n",
       "    'country_flag_emoji_unicode': 'U+1F1FA U+1F1F8',\n",
       "    'calling_code': '1',\n",
       "    'is_eu': False}},\n",
       "  200),\n",
       " ({'ip': '151.101.192.223',\n",
       "   'type': 'ipv4',\n",
       "   'continent_code': 'NA',\n",
       "   'continent_name': 'North America',\n",
       "   'country_code': 'US',\n",
       "   'country_name': 'United States',\n",
       "   'region_code': 'CA',\n",
       "   'region_name': 'California',\n",
       "   'city': 'San Francisco',\n",
       "   'zip': '94107',\n",
       "   'latitude': 37.76784896850586,\n",
       "   'longitude': -122.39286041259766,\n",
       "   'location': {'geoname_id': 5391959,\n",
       "    'capital': 'Washington D.C.',\n",
       "    'languages': [{'code': 'en', 'name': 'English', 'native': 'English'}],\n",
       "    'country_flag': 'http://assets.ipstack.com/flags/us.svg',\n",
       "    'country_flag_emoji': '🇺🇸',\n",
       "    'country_flag_emoji_unicode': 'U+1F1FA U+1F1F8',\n",
       "    'calling_code': '1',\n",
       "    'is_eu': False}},\n",
       "  200),\n",
       " ({'ip': '151.101.0.223',\n",
       "   'type': 'ipv4',\n",
       "   'continent_code': 'NA',\n",
       "   'continent_name': 'North America',\n",
       "   'country_code': 'US',\n",
       "   'country_name': 'United States',\n",
       "   'region_code': 'CA',\n",
       "   'region_name': 'California',\n",
       "   'city': 'San Francisco',\n",
       "   'zip': '94107',\n",
       "   'latitude': 37.76784896850586,\n",
       "   'longitude': -122.39286041259766,\n",
       "   'location': {'geoname_id': 5391959,\n",
       "    'capital': 'Washington D.C.',\n",
       "    'languages': [{'code': 'en', 'name': 'English', 'native': 'English'}],\n",
       "    'country_flag': 'http://assets.ipstack.com/flags/us.svg',\n",
       "    'country_flag_emoji': '🇺🇸',\n",
       "    'country_flag_emoji_unicode': 'U+1F1FA U+1F1F8',\n",
       "    'calling_code': '1',\n",
       "    'is_eu': False}},\n",
       "  200),\n",
       " ({'ip': '151.101.64.223',\n",
       "   'type': 'ipv4',\n",
       "   'continent_code': 'NA',\n",
       "   'continent_name': 'North America',\n",
       "   'country_code': 'US',\n",
       "   'country_name': 'United States',\n",
       "   'region_code': 'CA',\n",
       "   'region_name': 'California',\n",
       "   'city': 'San Francisco',\n",
       "   'zip': '94107',\n",
       "   'latitude': 37.76784896850586,\n",
       "   'longitude': -122.39286041259766,\n",
       "   'location': {'geoname_id': 5391959,\n",
       "    'capital': 'Washington D.C.',\n",
       "    'languages': [{'code': 'en', 'name': 'English', 'native': 'English'}],\n",
       "    'country_flag': 'http://assets.ipstack.com/flags/us.svg',\n",
       "    'country_flag_emoji': '🇺🇸',\n",
       "    'country_flag_emoji_unicode': 'U+1F1FA U+1F1F8',\n",
       "    'calling_code': '1',\n",
       "    'is_eu': False}},\n",
       "  200)]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "IP Address Entities\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[IpAddress(Address=151.101.128.223, Location={ 'AdditionalData': {},\n",
       "   'City': 'San Francis...),\n",
       " IpAddress(Address=151.101.192.223, Location={ 'AdditionalData': {},\n",
       "   'City': 'San Francis...),\n",
       " IpAddress(Address=151.101.0.223, Location={ 'AdditionalData': {},\n",
       "   'City': 'San Francisco...),\n",
       " IpAddress(Address=151.101.64.223, Location={ 'AdditionalData': {},\n",
       "   'City': 'San Francisc...)]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "if \"MSTICPY_SKIP_IPSTACK_TEST\" not in os.environ:\n",
    "    loc_result, ip_entities = iplocation.lookup_ip(ip_addr_list=ips)\n",
    "    print('Raw results')\n",
    "    display(loc_result)\n",
    "\n",
    "    print('IP Address Entities')\n",
    "    display(ip_entities)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a></a>[Contents](#contents)\n",
    "## Taking input from a pandas DataFrame\n",
    "\n",
    "The base class for both implementations has a method that sources the ip addresses from a dataframe column and returns a new dataframe with the location information merged with the input frame\n",
    "```\n",
    "Signature: iplocation.df_lookup_ip(data: pandas.core.frame.DataFrame, column: str)\n",
    "Docstring:\n",
    "Lookup Geolocation data from a pandas Dataframe.\n",
    "\n",
    "Keyword Arguments:\n",
    "    data {pd.DataFrame} -- pandas dataframe containing IpAddress column\n",
    "    column {str} -- the name of the dataframe column to use as a source\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-08T02:38:55.262352Z",
     "start_time": "2020-02-08T02:38:55.197381Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>AllExtIPs</th>\n",
       "      <th>CountryCode</th>\n",
       "      <th>CountryName</th>\n",
       "      <th>State</th>\n",
       "      <th>City</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Asn</th>\n",
       "      <th>edges</th>\n",
       "      <th>Type</th>\n",
       "      <th>AdditionalData</th>\n",
       "      <th>IpAddress</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>65.55.44.109</td>\n",
       "      <td>US</td>\n",
       "      <td>United States</td>\n",
       "      <td>Virginia</td>\n",
       "      <td>Boydton</td>\n",
       "      <td>-78.3750</td>\n",
       "      <td>36.6534</td>\n",
       "      <td>None</td>\n",
       "      <td>{}</td>\n",
       "      <td>geolocation</td>\n",
       "      <td>{}</td>\n",
       "      <td>65.55.44.109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>13.71.172.128</td>\n",
       "      <td>CA</td>\n",
       "      <td>Canada</td>\n",
       "      <td>Ontario</td>\n",
       "      <td>Toronto</td>\n",
       "      <td>-79.4195</td>\n",
       "      <td>43.6644</td>\n",
       "      <td>None</td>\n",
       "      <td>{}</td>\n",
       "      <td>geolocation</td>\n",
       "      <td>{}</td>\n",
       "      <td>13.71.172.128</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13.71.172.130</td>\n",
       "      <td>CA</td>\n",
       "      <td>Canada</td>\n",
       "      <td>Ontario</td>\n",
       "      <td>Toronto</td>\n",
       "      <td>-79.4195</td>\n",
       "      <td>43.6644</td>\n",
       "      <td>None</td>\n",
       "      <td>{}</td>\n",
       "      <td>geolocation</td>\n",
       "      <td>{}</td>\n",
       "      <td>13.71.172.130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>40.124.45.19</td>\n",
       "      <td>US</td>\n",
       "      <td>United States</td>\n",
       "      <td>Texas</td>\n",
       "      <td>San Antonio</td>\n",
       "      <td>-98.4926</td>\n",
       "      <td>29.4221</td>\n",
       "      <td>None</td>\n",
       "      <td>{}</td>\n",
       "      <td>geolocation</td>\n",
       "      <td>{}</td>\n",
       "      <td>40.124.45.19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>104.43.212.12</td>\n",
       "      <td>US</td>\n",
       "      <td>United States</td>\n",
       "      <td>Iowa</td>\n",
       "      <td>Des Moines</td>\n",
       "      <td>-93.6127</td>\n",
       "      <td>41.6015</td>\n",
       "      <td>None</td>\n",
       "      <td>{}</td>\n",
       "      <td>geolocation</td>\n",
       "      <td>{}</td>\n",
       "      <td>104.43.212.12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>20.41.41.23</td>\n",
       "      <td>US</td>\n",
       "      <td>United States</td>\n",
       "      <td>Virginia</td>\n",
       "      <td>Boydton</td>\n",
       "      <td>-78.3750</td>\n",
       "      <td>36.6534</td>\n",
       "      <td>None</td>\n",
       "      <td>{}</td>\n",
       "      <td>geolocation</td>\n",
       "      <td>{}</td>\n",
       "      <td>20.41.41.23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>52.179.17.38</td>\n",
       "      <td>US</td>\n",
       "      <td>United States</td>\n",
       "      <td>Virginia</td>\n",
       "      <td>Washington</td>\n",
       "      <td>-78.1539</td>\n",
       "      <td>38.7095</td>\n",
       "      <td>None</td>\n",
       "      <td>{}</td>\n",
       "      <td>geolocation</td>\n",
       "      <td>{}</td>\n",
       "      <td>52.179.17.38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>157.55.134.142</td>\n",
       "      <td>US</td>\n",
       "      <td>United States</td>\n",
       "      <td>Virginia</td>\n",
       "      <td>Washington</td>\n",
       "      <td>-78.1539</td>\n",
       "      <td>38.7095</td>\n",
       "      <td>None</td>\n",
       "      <td>{}</td>\n",
       "      <td>geolocation</td>\n",
       "      <td>{}</td>\n",
       "      <td>157.55.134.142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>172.217.15.110</td>\n",
       "      <td>US</td>\n",
       "      <td>United States</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>-97.8220</td>\n",
       "      <td>37.7510</td>\n",
       "      <td>None</td>\n",
       "      <td>{}</td>\n",
       "      <td>geolocation</td>\n",
       "      <td>{}</td>\n",
       "      <td>172.217.15.110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>40.91.75.5</td>\n",
       "      <td>US</td>\n",
       "      <td>United States</td>\n",
       "      <td>Washington</td>\n",
       "      <td>None</td>\n",
       "      <td>-122.3412</td>\n",
       "      <td>47.6032</td>\n",
       "      <td>None</td>\n",
       "      <td>{}</td>\n",
       "      <td>geolocation</td>\n",
       "      <td>{}</td>\n",
       "      <td>40.91.75.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>87 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         AllExtIPs CountryCode    CountryName       State         City  \\\n",
       "0     65.55.44.109          US  United States    Virginia      Boydton   \n",
       "1    13.71.172.128          CA         Canada     Ontario      Toronto   \n",
       "2    13.71.172.130          CA         Canada     Ontario      Toronto   \n",
       "3     40.124.45.19          US  United States       Texas  San Antonio   \n",
       "4    104.43.212.12          US  United States        Iowa   Des Moines   \n",
       "..             ...         ...            ...         ...          ...   \n",
       "82     20.41.41.23          US  United States    Virginia      Boydton   \n",
       "83    52.179.17.38          US  United States    Virginia   Washington   \n",
       "84  157.55.134.142          US  United States    Virginia   Washington   \n",
       "85  172.217.15.110          US  United States        None         None   \n",
       "86      40.91.75.5          US  United States  Washington         None   \n",
       "\n",
       "    Longitude  Latitude   Asn edges         Type AdditionalData  \\\n",
       "0    -78.3750   36.6534  None    {}  geolocation             {}   \n",
       "1    -79.4195   43.6644  None    {}  geolocation             {}   \n",
       "2    -79.4195   43.6644  None    {}  geolocation             {}   \n",
       "3    -98.4926   29.4221  None    {}  geolocation             {}   \n",
       "4    -93.6127   41.6015  None    {}  geolocation             {}   \n",
       "..        ...       ...   ...   ...          ...            ...   \n",
       "82   -78.3750   36.6534  None    {}  geolocation             {}   \n",
       "83   -78.1539   38.7095  None    {}  geolocation             {}   \n",
       "84   -78.1539   38.7095  None    {}  geolocation             {}   \n",
       "85   -97.8220   37.7510  None    {}  geolocation             {}   \n",
       "86  -122.3412   47.6032  None    {}  geolocation             {}   \n",
       "\n",
       "         IpAddress  \n",
       "0     65.55.44.109  \n",
       "1    13.71.172.128  \n",
       "2    13.71.172.130  \n",
       "3     40.124.45.19  \n",
       "4    104.43.212.12  \n",
       "..             ...  \n",
       "82     20.41.41.23  \n",
       "83    52.179.17.38  \n",
       "84  157.55.134.142  \n",
       "85  172.217.15.110  \n",
       "86      40.91.75.5  \n",
       "\n",
       "[87 rows x 12 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "netflow_df = pd.read_csv(\"data/az_net_flows.csv\")\n",
    "netflow_df = netflow_df[[\"AllExtIPs\"]].drop_duplicates()\n",
    "iplocation = GeoLiteLookup()\n",
    "iplocation.df_lookup_ip(netflow_df, column=\"AllExtIPs\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a></a>[Contents](#contents)\n",
    "## Creating a Custom GeopIP Lookup Class\n",
    "\n",
    "You can derive a class that implements the same operations to use with a different GeoIP service.\n",
    "\n",
    "The class signature is as follows:\n",
    "```\n",
    "class GeoIpLookup(ABC):\n",
    "    \"\"\"Abstract base class for GeoIP Lookup classes.\"\"\"\n",
    "\n",
    "    @abstractmethod\n",
    "    def lookup_ip(self, ip_address: str = None, ip_addr_list: Iterable = None,\n",
    "                  ip_entity: IpAddress = None):\n",
    "        \"\"\"\n",
    "        Lookup IP location.\n",
    "\n",
    "        Keyword Arguments:\n",
    "            ip_address {str} -- a single address to look up (default: {None})\n",
    "            ip_addr_list {Iterable} -- a collection of addresses to lookup (default: {None})\n",
    "            ip_entity {IpAddress} -- an IpAddress entity\n",
    "\n",
    "        Returns:\n",
    "            tuple(list{dict}, list{entity}) -- returns raw geolocation results and\n",
    "                same results as IP/Geolocation entities\n",
    "\n",
    "        \"\"\"\n",
    "```\n",
    "You should override the lookup_ip method implementing your own method of geoip lookup."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a></a>[Contents](#contents)\n",
    "## Calculating Geographical Distances\n",
    "\n",
    "Use the geo_distance function from msticpy.sectools.geoip to calculated distances between two locations.\n",
    "I am indebted to Martin Thoma who posted this solution (which I've modified slightly) on Stackoverflow.\n",
    "\n",
    "\n",
    "```\n",
    "Signature: geo_distance(origin: Tuple[float, float], destination: Tuple[float, float]) -&gt; float\n",
    "Docstring:\n",
    "Calculate the Haversine distance.\n",
    "\n",
    "Author: Martin Thoma - stackoverflow\n",
    "\n",
    "Parameters\n",
    "----------\n",
    "origin : tuple of float\n",
    "    (lat, long)\n",
    "destination : tuple of float\n",
    "    (lat, long)\n",
    "\n",
    "Returns\n",
    "-------\n",
    "distance_in_km : float\n",
    "```\n",
    "\n",
    "\n",
    "Or where you have source and destination IpAddress entities, you can use the wrapper entity_distance.\n",
    "```\n",
    "Signature:\n",
    "entity_distance(\n",
    "    ['ip_src: msticpy.nbtools.entityschema.IpAddress', 'ip_dest: msticpy.nbtools.entityschema.IpAddress'],\n",
    ") -&gt; float\n",
    "Docstring:\n",
    "Return distance between two IP Entities.\n",
    "\n",
    "Arguments:\n",
    "    ip_src {IpAddress} -- Source IpAddress Entity\n",
    "    ip_dest {IpAddress} -- Destination IpAddress Entity\n",
    "\n",
    "Raises:\n",
    "    AttributeError -- if either entity has no location information\n",
    "\n",
    "Returns:\n",
    "    float -- Distance in kilometers.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-08T02:38:55.326316Z",
     "start_time": "2020-02-08T02:38:55.319320Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{ 'AdditionalData': {},\n",
      "  'Address': '90.156.201.97',\n",
      "  'Location': { 'AdditionalData': {},\n",
      "                'CountryCode': 'RU',\n",
      "                'CountryName': 'Russia',\n",
      "                'Latitude': 55.7386,\n",
      "                'Longitude': 37.6068,\n",
      "                'Type': 'geolocation',\n",
      "                'edges': set()},\n",
      "  'ThreatIntelligence': [],\n",
      "  'Type': 'ipaddress',\n",
      "  'edges': set()}\n",
      "{ 'AdditionalData': {},\n",
      "  'Address': '151.101.64.223',\n",
      "  'Location': { 'AdditionalData': {},\n",
      "                'CountryCode': 'US',\n",
      "                'CountryName': 'United States',\n",
      "                'Latitude': 37.751,\n",
      "                'Longitude': -97.822,\n",
      "                'Type': 'geolocation',\n",
      "                'edges': set()},\n",
      "  'ThreatIntelligence': [],\n",
      "  'Type': 'ipaddress',\n",
      "  'edges': set()}\n",
      "\n",
      "Distance between IP Locations = 8796.8km\n"
     ]
    }
   ],
   "source": [
    "from msticpy.sectools.geoip import geo_distance\n",
    "_, ip_entity1 = iplocation.lookup_ip(ip_address='90.156.201.97')\n",
    "_, ip_entity2 = iplocation.lookup_ip(ip_address='151.101.64.223')\n",
    "\n",
    "print(ip_entity1[0])\n",
    "print(ip_entity2[0])\n",
    "dist = geo_distance(origin=(ip_entity1[0].Location.Latitude, ip_entity1[0].Location.Longitude),\n",
    "                    destination=(ip_entity2[0].Location.Latitude, ip_entity2[0].Location.Longitude))\n",
    "print(f'\\nDistance between IP Locations = {round(dist, 1)}km')"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python (condadev)",
   "language": "python",
   "name": "condadev"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {
    "height": "318.996px",
    "width": "320.994px"
   },
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "165px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "position": {
    "height": "406.193px",
    "left": "1468.4px",
    "right": "20px",
    "top": "120px",
    "width": "456.572px"
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}