iat_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IAT: detecting micro-clusters on suspicious behavors\n",
    "\n",
    "A group of fraudsters always behave synchronously in a regular (fixed) pattern, which probably shows\n",
    "suspiciousness comparing to the normally behaving people.\n",
    "Therefore, we study the overall time intervals of users, and detect the suspicious micro-clusters that stands out of the majority distributions.\n",
    "It can be used with vision-guided detection algorithm, EagleMine.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import spartan as st"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load data by function ```loadTensor```.<br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "tensor_data = st.loadTensor(path = \"./inputData/wbcovid19_test.gz\", header=None, sep='\\x01')\n",
    "tensor_data.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "coords, data = tensor_data.do_map(hasvalue=False, mappers={0:st.TimeMapper(timeformat='%Y-%m-%d %H:%M:%S', timebin = 1, mints = 0)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use ```to_aggts``` function to extract time stamps in log files or edgelist tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "aggts = tensor_data.to_aggts(coords, time_col=0, group_col=[1])\n",
    "# print(aggts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IAT class\n",
    "\n",
    "calaggiat function：calculate iat dict **aggiat** (key:user, value: iat list)\n",
    "\n",
    "caliatcount function：calculate iat count dict **iatcount** (key:iat, value: frequency) and iat prob dict **iatprob** (key:iat; value:probability)\n",
    "\n",
    "caliatpaircount function：calculate iat dict **iatpaircount** (key:(iat1, iat2), value: frequency)\n",
    "\n",
    "get_user_iatpair_dict function：calculate iat dict **user_iatpair** (key:user, value: (iat1, iat2) list)\n",
    "\n",
    "get_iatpair_user_dict function：calculate iat dict **iatpair_user** (key:(iat1, iat2), value: user list)\n",
    "\n",
    "find_iatpair_user function: find users who have input iat pairs\n",
    "\n",
    "get_user_dict function: get users dict that have pairs in iatpairs ordered by decreasing frequency\n",
    "\n",
    "find_topk_user function: find Top-K users that have pairs in iatpairs ordered by decreasing frequency\n",
    "\n",
    "drawIatPdf: Plot Iat-Pdf line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "instance = st.IAT()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# calculate aggiat dict\n",
    "instance.calaggiat(aggts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "aggiat=instance.aggiat\n",
    "# print(aggiat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "instance.save_aggiat('./output/aggiat.dictlist.gz')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "instance.load_aggiat('./output/aggiat.dictlist.gz')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "xs, ys = instance.getiatpairs()\n",
    "len(xs), len(ys)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# invoke drawHexbin function\n",
    "hexfig = st.drawHexbin(xs, ys, gridsize=5, xlabel='IATn', ylabel='IATn+1',outfig='./images/iathexbin_demo.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is the result:\n",
    "<img src=\"images/iathexbin_demo.png\" width=\"400\"/> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# invoke drawRectbin function\n",
    "fig, hist = st.drawRectbin(xs, ys, gridsize=10, xlabel='IATn', ylabel='IATn+1', outfig='./images/iatrectbin_demo.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is the result:\n",
    "<img src=\"images/iatrectbin_demo.png\" width=\"400\"/> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## class RectHistogram\n",
    "1. draw function: draw 2D histogram with rectangular bin\n",
    "\n",
    "\n",
    "2. find_peak_range function: find the range of coordinates which bin with the largest number of samples in the range of \n",
    "\n",
    "    horizontal axis: [x-radius, x+radius]\n",
    "    \n",
    "    vertical axis: [y-radius, y+radius]\n",
    "\n",
    "    return xrange: the range of max bin along the x axis and yrange: the range of max bin along the y axis.\n",
    "    \n",
    "\n",
    "3. find_peak_rects function: find coordinate pairs in the max bin\n",
    "\n",
    "    return: (x,y) pairs in the bin that has the largest number of samples "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "recthistogram = st.RectHistogram(xscale='log', yscale='log', gridsize=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get iatpairs, you need to execute draw function first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "fig = recthistogram.draw(xs, ys, xlabel='IATn', ylabel='IATn+1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xrange, yrange = recthistogram.find_peak_range(x=100, y=100, radius=100)\n",
    "print(f\"the range of max bin along the x axis:\\n {xrange}\")\n",
    "print(f\"the range of max bin along the y axis:\\n {yrange}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "iatpairs = recthistogram.find_peak_rect(xrange, yrange)\n",
    "print(iatpairs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Find Top-k suspicious users"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "instance.get_iatpair_user_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "instance.get_user_dict(iatpairs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "usrlist = instance.find_topk_user(k=5) # default return all, k = -1\n",
    "print(f\"Top-5 user: \\n{usrlist}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "plot iat-pdf line by function `drawIatPdf`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "instance.caliatcount()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "fig = instance.drawIatPdf(usrlist, outfig='./images/iatpdf_demo.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is the result:\n",
    "<img src=\"images/iatpdf_demo.png\" width=\"400\"/> "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}