demo/demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bandicoot notebook\n",
    "\n",
    "_bandicoot_ is an open-source python toolbox to analyze mobile phone metadata. For more information, see: http://bandicoot.mit.edu/\n",
    "\n",
    "The source code of the notebook is available as ``demo.ipynb`` and a plain\n",
    "Python version as ``demo.py``. You can download them from our repository on Github at [https://github.com/yvesalexandre/bandicoot/tree/master/demo](https://github.com/yvesalexandre/bandicoot/tree/master/demo)\n",
    "\n",
    "\n",
    "<br />\n",
    "<div class=\"alert alert-info\" role=\"alert\">\n",
    "<p><strong>Try bandicoot on your phone!</strong></p>\n",
    "<p>If you want to try bandicoot with your own data, download our Android app at <a href=\"http://bandicoot.mit.edu/android\">bandicoot.mit.edu/android</a></p>\n",
    "</div>\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Input files\n",
    "\n",
    "<img src=\"mini-mockups-01.png\" width=\"80%\" style=\"border: 1px solid #aaa\" />\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "interaction,direction,correspondent_id,datetime,call_duration,antenna_id\r\n",
      "text,in,A,2014-03-02 07:13:30,,1\r\n",
      "text,in,E,2014-03-02 07:53:30,,1\r\n",
      "text,in,E,2014-03-02 08:22:30,,2\r\n",
      "text,out,D,2014-03-02 08:34:30,,3\r\n"
     ]
    }
   ],
   "source": [
    "# Records for the user 'ego'\n",
    "!head -n 5 data/ego.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "antenna_id,latitude,longitude\r\n",
      "1,42.366944,-71.083611\r\n",
      "2,42.386722,-71.138778\r\n",
      "3,42.3604,-71.087374\r\n",
      "4,42.353917,-71.105\r\n"
     ]
    }
   ],
   "source": [
    "# GPS locations of cell towers\n",
    "!head -n 5 data/antennas.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading a user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m[x]\u001b[0m 314 records from 2014-03-02 07:13:30 to 2014-04-14 12:04:37\n",
      "\u001b[32m[x]\u001b[0m 7 contacts\n",
      "\u001b[32m[ ]\u001b[0m No attribute stored\n",
      "\u001b[32m[x]\u001b[0m 27 antennas\n",
      "\u001b[32m[ ]\u001b[0m No recharges\n",
      "\u001b[32m[x]\u001b[0m Has home\n",
      "\u001b[32m[x]\u001b[0m Has texts\n",
      "\u001b[32m[x]\u001b[0m Has calls\n",
      "\u001b[32m[ ]\u001b[0m No network\n"
     ]
    }
   ],
   "source": [
    "import bandicoot as bc\n",
    "\n",
    "U = bc.read_csv('ego', 'data/', 'data/antennas.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualization\n",
    "\n",
    "Export and serve an interactive visualization using:\n",
    "\n",
    "```python\n",
    "bc.visualization.run(U)\n",
    "```\n",
    "\n",
    "or export only using:\n",
    "\n",
    "```python\n",
    "bc.visualization.export(U, 'my-viz-path')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully exported the visualization to /Volumes/Data/projects/bandicoot/demo/viz\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'/Volumes/Data/projects/bandicoot/demo/viz'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "viz_path = os.path.dirname(os.path.realpath(__name__)) + '/viz'\n",
    "\n",
    "bc.visualization.export(U, viz_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "hide_input": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"100%\"\n",
       "            height=\"700\"\n",
       "            src=\"/files/viz/index.html\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x10573fa50>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import IFrame\n",
    "IFrame(\"/files/viz/index.html\", \"100%\", 700)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Individual and spatial indicators\n",
    "\n",
    "Using bandicoot, compute aggregated indicators from ``bc.individual`` and ``bc.spatial``: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "    \"allweek\": {\n",
       "        \"allday\": {\n",
       "            \"callandtext\": {\n",
       "                \"mean\": 0.3244815064488733, \n",
       "                \"std\": 0.09659866096759165\n",
       "            }\n",
       "        }\n",
       "    }\n",
       "}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bc.individual.percent_initiated_conversations(U)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "    \"allweek\": {\n",
       "        \"allday\": {\n",
       "            \"mean\": 5.375, \n",
       "            \"std\": 1.8666480653835098\n",
       "        }\n",
       "    }\n",
       "}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bc.spatial.number_of_antennas(U)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "    \"allweek\": {\n",
       "        \"allday\": {\n",
       "            \"mean\": 1.4503807789208683, \n",
       "            \"std\": 0.8575480642906887\n",
       "        }\n",
       "    }\n",
       "}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bc.spatial.radius_of_gyration(U)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Let's play with indicators\n",
    "\n",
    "The signature of the ``active_days`` indicators is:\n",
    "\n",
    "```python\n",
    "bc.individual.active_days(user, groupby='week', interaction='callandtext', summary='default', split_week=False, split_day=False, filter_empty=True, datatype=None)\n",
    "```\n",
    "\n",
    "What does that mean?\n",
    "\n",
    "\n",
    "<hr />\n",
    "\n",
    "\n",
    "## The *‘groupby’* keyword\n",
    "\n",
    "<br />\n",
    "<div class=\"alert alert-info\" role=\"alert\">\n",
    "<p><strong>Weekly aggregation</strong></p>\n",
    "<p>By default, _bandicoot_ computes the indicators on a weekly basis and returns the average (mean) over all the weeks available and its standard deviation (std) in a nested dictionary.</p>\n",
    "\n",
    "<img src=\"mini-mockups-02.png\" width=\"80%\" style=\"border: 1px solid #aaa\" />\n",
    "\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "    \"allweek\": {\n",
       "        \"allday\": {\n",
       "            \"callandtext\": {\n",
       "                \"mean\": 5.5, \n",
       "                \"std\": 2.598076211353316\n",
       "            }\n",
       "        }\n",
       "    }\n",
       "}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bc.individual.active_days(U)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ``groupby`` keyword controls the aggregation:\n",
    "\n",
    "- ``groupby='week'`` to divide by week (by default),\n",
    "- ``groupby='month'`` to divide by month,\n",
    "- ``groupby=None`` to aggregate all values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "    \"allweek\": {\n",
       "        \"allday\": {\n",
       "            \"callandtext\": {\n",
       "                \"mean\": 5.5, \n",
       "                \"std\": 2.598076211353316\n",
       "            }\n",
       "        }\n",
       "    }\n",
       "}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bc.individual.active_days(U, groupby='week')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "    \"allweek\": {\n",
       "        \"allday\": {\n",
       "            \"callandtext\": {\n",
       "                \"mean\": 22.0, \n",
       "                \"std\": 8.0\n",
       "            }\n",
       "        }\n",
       "    }\n",
       "}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bc.individual.active_days(U, groupby='month')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "    \"allweek\": {\n",
       "        \"allday\": {\n",
       "            \"callandtext\": 44\n",
       "        }\n",
       "    }\n",
       "}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bc.individual.active_days(U, groupby=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The *‘summary’* keyword\n",
    "\n",
    "Some indicators such as *active\\_days* returns one number. Others, such as *duration\\_of\\_calls* returns a distribution.\n",
    "\n",
    "The summary keyword can take three values:\n",
    "\n",
    " - `summary='default'` to return mean and standard deviation,\n",
    " - `summary='extended'` for the second type of indicators, to return mean, sem, median, skewness and std of the distribution,\n",
    " - `summary=None` to return the full distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "    \"allweek\": {\n",
       "        \"allday\": {\n",
       "            \"call\": {\n",
       "                \"mean\": {\n",
       "                    \"mean\": 3776.7093501036775, \n",
       "                    \"std\": 1404.827412706482\n",
       "                }, \n",
       "                \"std\": {\n",
       "                    \"mean\": 1633.3931770157765, \n",
       "                    \"std\": 689.2035500056488\n",
       "                }\n",
       "            }\n",
       "        }\n",
       "    }\n",
       "}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bc.individual.call_duration(U)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "    \"allweek\": {\n",
       "        \"allday\": {\n",
       "            \"call\": {\n",
       "                \"mean\": {\n",
       "                    \"mean\": 3776.7093501036775, \n",
       "                    \"std\": 1404.827412706482\n",
       "                }, \n",
       "                \"std\": {\n",
       "                    \"mean\": 1633.3931770157765, \n",
       "                    \"std\": 689.2035500056488\n",
       "                }, \n",
       "                \"median\": {\n",
       "                    \"mean\": 3714.714285714286, \n",
       "                    \"std\": 1532.9148671064283\n",
       "                }, \n",
       "                \"skewness\": {\n",
       "                    \"mean\": 0.12925073170191398, \n",
       "                    \"std\": 0.48628300355189896\n",
       "                }, \n",
       "                \"kurtosis\": {\n",
       "                    \"mean\": 1.8063876957023484, \n",
       "                    \"std\": 0.8998073161683097\n",
       "                }, \n",
       "                \"min\": {\n",
       "                    \"mean\": 1330.857142857143, \n",
       "                    \"std\": 2200.2680634459994\n",
       "                }, \n",
       "                \"max\": {\n",
       "                    \"mean\": 6468.857142857143, \n",
       "                    \"std\": 519.0475972040188\n",
       "                }\n",
       "            }\n",
       "        }\n",
       "    }\n",
       "}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bc.individual.call_duration(U, summary='extended')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "    \"allweek\": {\n",
       "        \"allday\": {\n",
       "            \"call\": [\n",
       "                [\n",
       "                    374, \n",
       "                    1086, \n",
       "                    1099, \n",
       "                    1330, \n",
       "                    2456, \n",
       "                    3404, \n",
       "                    4472, \n",
       "                    5359, \n",
       "                    5413, \n",
       "                    6233\n",
       "                ], \n",
       "                [\n",
       "                    594, \n",
       "                    1927, \n",
       "                    2072, \n",
       "                    2258, \n",
       "                    2854, \n",
       "                    3286, \n",
       "                    3552, \n",
       "                    4202, \n",
       "                    4689, \n",
       "                    5142, \n",
       "                    5689, \n",
       "                    5752, \n",
       "                    6429, \n",
       "                    6891, \n",
       "                    7082, \n",
       "                    7123\n",
       "                ], \n",
       "                [\n",
       "                    403, \n",
       "                    539, \n",
       "                    2109, \n",
       "                    2726, \n",
       "                    2871, \n",
       "                    3609, \n",
       "                    3782, \n",
       "                    4154, \n",
       "                    4240, \n",
       "                    4666, \n",
       "                    5658, \n",
       "                    6392, \n",
       "                    6541, \n",
       "                    6674\n",
       "                ], \n",
       "                [\n",
       "                    154, \n",
       "                    267, \n",
       "                    706, \n",
       "                    1273, \n",
       "                    1890, \n",
       "                    3435, \n",
       "                    3454, \n",
       "                    3503, \n",
       "                    3531, \n",
       "                    3668, \n",
       "                    4832, \n",
       "                    4877, \n",
       "                    4929, \n",
       "                    5096, \n",
       "                    5184, \n",
       "                    6539, \n",
       "                    7038\n",
       "                ], \n",
       "                [\n",
       "                    140, \n",
       "                    231, \n",
       "                    620, \n",
       "                    1853, \n",
       "                    1937, \n",
       "                    5728\n",
       "                ], \n",
       "                [\n",
       "                    969, \n",
       "                    1309, \n",
       "                    1355, \n",
       "                    1999, \n",
       "                    2626, \n",
       "                    3210, \n",
       "                    4002, \n",
       "                    4146, \n",
       "                    4227, \n",
       "                    4451, \n",
       "                    5804\n",
       "                ], \n",
       "                [\n",
       "                    6682\n",
       "                ]\n",
       "            ]\n",
       "        }\n",
       "    }\n",
       "}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bc.individual.call_duration(U, summary=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splitting days and weeks\n",
    "\n",
    "- **``split_week``** divide records by 'all week', 'weekday', and 'weekend'.\n",
    "- **``split_day``** divide records by 'all day', 'day', and 'night'.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "    \"allweek\": {\n",
       "        \"allday\": {\n",
       "            \"callandtext\": {\n",
       "                \"mean\": 5.5, \n",
       "                \"std\": 2.598076211353316\n",
       "            }\n",
       "        }, \n",
       "        \"day\": {\n",
       "            \"callandtext\": {\n",
       "                \"mean\": 5.5, \n",
       "                \"std\": 2.598076211353316\n",
       "            }\n",
       "        }, \n",
       "        \"night\": {\n",
       "            \"callandtext\": {\n",
       "                \"mean\": 5.375, \n",
       "                \"std\": 2.54644359843292\n",
       "            }\n",
       "        }\n",
       "    }, \n",
       "    \"weekday\": {\n",
       "        \"allday\": {\n",
       "            \"callandtext\": {\n",
       "                \"mean\": 4.428571428571429, \n",
       "                \"std\": 1.3997084244475304\n",
       "            }\n",
       "        }, \n",
       "        \"day\": {\n",
       "            \"callandtext\": {\n",
       "                \"mean\": 4.428571428571429, \n",
       "                \"std\": 1.3997084244475304\n",
       "            }\n",
       "        }, \n",
       "        \"night\": {\n",
       "            \"callandtext\": {\n",
       "                \"mean\": 4.428571428571429, \n",
       "                \"std\": 1.3997084244475304\n",
       "            }\n",
       "        }\n",
       "    }, \n",
       "    \"weekend\": {\n",
       "        \"allday\": {\n",
       "            \"callandtext\": {\n",
       "                \"mean\": 1.8571428571428572, \n",
       "                \"std\": 0.34992710611188266\n",
       "            }\n",
       "        }, \n",
       "        \"day\": {\n",
       "            \"callandtext\": {\n",
       "                \"mean\": 1.8571428571428572, \n",
       "                \"std\": 0.34992710611188266\n",
       "            }\n",
       "        }, \n",
       "        \"night\": {\n",
       "            \"callandtext\": {\n",
       "                \"mean\": 1.7142857142857142, \n",
       "                \"std\": 0.45175395145262565\n",
       "            }\n",
       "        }\n",
       "    }\n",
       "}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bc.individual.active_days(U, split_week=True, split_day=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exporting indicators\n",
    "\n",
    "The function ``bc.utils.all`` computes automatically all indicators for a single user.\n",
    "\n",
    "You can use the same keywords to group by week/month/all time range, or return extended statistics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "features = bc.utils.all(U, groupby=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "    \"name\": \"ego\", \n",
       "    \"reporting\": {\n",
       "        \"antennas_path\": \"data/antennas.csv\", \n",
       "        \"attributes_path\": None, \n",
       "        \"recharges_path\": None, \n",
       "        \"version\": \"0.4.0\", \n",
       "        \"code_signature\": \"9f648eb0aa73d5e521c7e8a2bf8256697a467bdf\", \n",
       "        \"groupby\": None, \n",
       "        \"split_week\": false, \n",
       "        \"split_day\": false, \n",
       "        \"start_time\": \"2014-03-02 07:13:30\", \n",
       "        \"end_time\": \"2014-04-14 12:04:37\", \n",
       "        \"night_start\": \"19:00:00\", \n",
       "        \"night_end\": \"07:00:00\", \n",
       "        \"weekend\": [\n",
       "            6, \n",
       "            7\n",
       "        ], \n",
       "        \"number_of_records\": 314, \n",
       "        \"number_of_antennas\": 27, \n",
       "        \"number_of_recharges\": 0, \n",
       "        \"bins\": 1, \n",
       "        \"bins_with_data\": 1, \n",
       "        \"bins_without_data\": 0, \n",
       "        \"has_call\": true, \n",
       "        \"has_text\": true, \n",
       "        \"has_home\": true, \n",
       "        \"has_recharges\": false, \n",
       "        \"has_attributes\": false, \n",
       "        \"has_network\": false, \n",
       "        \"percent_records_missing_location\": 0.0, \n",
       "        \"antennas_missing_locations\": 0, \n",
       "        \"percent_outofnetwork_calls\": 0, \n",
       "        \"percent_outofnetwork_texts\": 0, \n",
       "        \"percent_outofnetwork_contacts\": 0, \n",
       "        \"percent_outofnetwork_call_durations\": 0, \n",
       "        \"ignored_records\": {\n",
       "            \"all\": 0, \n",
       "            \"interaction\": 0, \n",
       "            \"location\": 0, \n",
       "            \"correspondent_id\": 0, \n",
       "            \"call_duration\": 0, \n",
       "            \"direction\": 0, \n",
       "            \"datetime\": 0\n",
       "        }\n",
       "    }, \n",
       "    \"active_days\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": {\n",
       "                \"callandtext\": 44\n",
       "            }\n",
       "        }\n",
       "    }, \n",
       "    \"number_of_contacts\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": {\n",
       "                \"call\": 7, \n",
       "                \"text\": 7\n",
       "            }\n",
       "        }\n",
       "    }, \n",
       "    \"call_duration\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": {\n",
       "                \"call\": {\n",
       "                    \"mean\": 3557.2933333333335, \n",
       "                    \"std\": 2067.863501448348\n",
       "                }\n",
       "            }\n",
       "        }\n",
       "    }, \n",
       "    \"percent_nocturnal\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": {\n",
       "                \"call\": 0.4266666666666667, \n",
       "                \"text\": 0.4393305439330544\n",
       "            }\n",
       "        }\n",
       "    }, \n",
       "    \"percent_initiated_conversations\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": {\n",
       "                \"callandtext\": 0.3080357142857143\n",
       "            }\n",
       "        }\n",
       "    }, \n",
       "    \"percent_initiated_interactions\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": {\n",
       "                \"call\": 0.41333333333333333\n",
       "            }\n",
       "        }\n",
       "    }, \n",
       "    \"response_delay_text\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": {\n",
       "                \"callandtext\": {\n",
       "                    \"mean\": 2310.0, \n",
       "                    \"std\": 747.5961476626268\n",
       "                }\n",
       "            }\n",
       "        }\n",
       "    }, \n",
       "    \"response_rate_text\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": {\n",
       "                \"callandtext\": 0.025806451612903226\n",
       "            }\n",
       "        }\n",
       "    }, \n",
       "    \"entropy_of_contacts\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": {\n",
       "                \"call\": 1.8626732085630935, \n",
       "                \"text\": 1.8462551114653172\n",
       "            }\n",
       "        }\n",
       "    }, \n",
       "    \"balance_of_contacts\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": {\n",
       "                \"call\": {\n",
       "                    \"mean\": 0.05904761904761905, \n",
       "                    \"std\": 0.018662778992633737\n",
       "                }, \n",
       "                \"text\": {\n",
       "                    \"mean\": 0.04363419007770473, \n",
       "                    \"std\": 0.01828697972597532\n",
       "                }\n",
       "            }\n",
       "        }\n",
       "    }, \n",
       "    \"interactions_per_contact\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": {\n",
       "                \"call\": {\n",
       "                    \"mean\": 10.714285714285714, \n",
       "                    \"std\": 4.025429372458677\n",
       "                }, \n",
       "                \"text\": {\n",
       "                    \"mean\": 34.142857142857146, \n",
       "                    \"std\": 15.027865274012724\n",
       "                }\n",
       "            }\n",
       "        }\n",
       "    }, \n",
       "    \"interevent_time\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": {\n",
       "                \"call\": {\n",
       "                    \"mean\": 49024.86486486487, \n",
       "                    \"std\": 49455.92699110296\n",
       "                }, \n",
       "                \"text\": {\n",
       "                    \"mean\": 15683.474789915967, \n",
       "                    \"std\": 16816.128561460955\n",
       "                }\n",
       "            }\n",
       "        }\n",
       "    }, \n",
       "    \"percent_pareto_interactions\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": {\n",
       "                \"call\": 0.06666666666666667, \n",
       "                \"text\": 0.02092050209205021\n",
       "            }\n",
       "        }\n",
       "    }, \n",
       "    \"percent_pareto_durations\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": {\n",
       "                \"call\": 0.06666666666666667\n",
       "            }\n",
       "        }\n",
       "    }, \n",
       "    \"number_of_interactions\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": {\n",
       "                \"call\": 75, \n",
       "                \"text\": 239\n",
       "            }\n",
       "        }\n",
       "    }, \n",
       "    \"number_of_interaction_in\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": {\n",
       "                \"call\": 44, \n",
       "                \"text\": 166\n",
       "            }\n",
       "        }\n",
       "    }, \n",
       "    \"number_of_interaction_out\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": {\n",
       "                \"call\": 31, \n",
       "                \"text\": 73\n",
       "            }\n",
       "        }\n",
       "    }, \n",
       "    \"number_of_antennas\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": 10\n",
       "        }\n",
       "    }, \n",
       "    \"entropy_of_antennas\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": 1.5002835799997023\n",
       "        }\n",
       "    }, \n",
       "    \"percent_at_home\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": 0.4738562091503268\n",
       "        }\n",
       "    }, \n",
       "    \"radius_of_gyration\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": 1.8065370474407052\n",
       "        }\n",
       "    }, \n",
       "    \"frequent_antennas\": {\n",
       "        \"allweek\": {\n",
       "            \"allday\": 3\n",
       "        }\n",
       "    }, \n",
       "    \"churn_rate\": {\n",
       "        \"mean\": 0.07391866292877013, \n",
       "        \"std\": 0.05195576174609364\n",
       "    }\n",
       "}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exporting in CSV and JSON\n",
    "\n",
    "bandicoot supports exports in CSV and JSON format. Both ``to_csv`` and ``to_json`` functions require either a single feature dictionnary, or a list of dictionnaries (for multiple users)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully exported 1 object(s) to demo_export_user.csv\n",
      "Successfully exported 1 object(s) to demo_export_user.json\n"
     ]
    }
   ],
   "source": [
    "bc.to_csv(features, 'demo_export_user.csv')\n",
    "bc.to_json(features, 'demo_export_user.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name,reporting__antennas_path,reporting__attributes_path,reporting__recharges_path,reporting__version,reporting__code_signature,reporting__groupby,reporting__split_week,reporting__split_day,reporting__start_time,reporting__end_time,reporting__night_start,reporting__night_end,reporting__weekend,reporting__number_of_records,reporting__number_of_antennas,reporting__number_of_recharges,reporting__bins,reporting__bins_with_data,reporting__bins_without_data,reporting__has_call,reporting__has_text,reporting__has_home,reporting__has_recharges,reporting__has_attributes,reporting__has_network,reporting__percent_records_missing_location,reporting__antennas_missing_locations,reporting__percent_outofnetwork_calls,reporting__percent_outofnetwork_texts,reporting__percent_outofnetwork_contacts,reporting__percent_outofnetwork_call_durations,reporting__ignored_records__all,reporting__ignored_records__interaction,reporting__ignored_records__location,reporting__ignored_records__correspondent_id,reporting__ignored_records__call_duration,reporting__ignored_records__direction,reporting__ignored_records__datetime,active_days__allweek__allday__callandtext,number_of_contacts__allweek__allday__call,number_of_contacts__allweek__allday__text,call_duration__allweek__allday__call__mean,call_duration__allweek__allday__call__std,percent_nocturnal__allweek__allday__call,percent_nocturnal__allweek__allday__text,percent_initiated_conversations__allweek__allday__callandtext,percent_initiated_interactions__allweek__allday__call,response_delay_text__allweek__allday__callandtext__mean,response_delay_text__allweek__allday__callandtext__std,response_rate_text__allweek__allday__callandtext,entropy_of_contacts__allweek__allday__call,entropy_of_contacts__allweek__allday__text,balance_of_contacts__allweek__allday__call__mean,balance_of_contacts__allweek__allday__call__std,balance_of_contacts__allweek__allday__text__mean,balance_of_contacts__allweek__allday__text__std,interactions_per_contact__allweek__allday__call__mean,interactions_per_contact__allweek__allday__call__std,interactions_per_contact__allweek__allday__text__mean,interactions_per_contact__allweek__allday__text__std,interevent_time__allweek__allday__call__mean,interevent_time__allweek__allday__call__std,interevent_time__allweek__allday__text__mean,interevent_time__allweek__allday__text__std,percent_pareto_interactions__allweek__allday__call,percent_pareto_interactions__allweek__allday__text,percent_pareto_durations__allweek__allday__call,number_of_interactions__allweek__allday__call,number_of_interactions__allweek__allday__text,number_of_interaction_in__allweek__allday__call,number_of_interaction_in__allweek__allday__text,number_of_interaction_out__allweek__allday__call,number_of_interaction_out__allweek__allday__text,number_of_antennas__allweek__allday,entropy_of_antennas__allweek__allday,percent_at_home__allweek__allday,radius_of_gyration__allweek__allday,frequent_antennas__allweek__allday,churn_rate__mean,churn_rate__std\r",
      "\r\n",
      "ego,data/antennas.csv,,,0.4.0,9f648eb0aa73d5e521c7e8a2bf8256697a467bdf,,False,False,2014-03-02 07:13:30,2014-04-14 12:04:37,19:00:00,07:00:00,\"[6, 7]\",314,27,0,1,1,0,True,True,True,False,False,False,0.0,0,0,0,0,0,0,0,0,0,0,0,0,44,7,7,3557.29333,2067.8635,0.42667,0.43933,0.30804,0.41333,2310.0,747.59615,0.02581,1.86267,1.84626,0.05905,0.01866,0.04363,0.01829,10.71429,4.02543,34.14286,15.02787,49024.86486,49455.92699,15683.47479,16816.12856,0.06667,0.02092,0.06667,75,239,44,166,31,73,10,1.50028,0.47386,1.80654,3,0.07392,0.05196\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head demo_export_user.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "    \"ego\": {\r\n",
      "        \"name\": \"ego\",\r\n",
      "        \"reporting\": {\r\n",
      "            \"antennas_path\": \"data/antennas.csv\",\r\n",
      "            \"attributes_path\": null,\r\n",
      "            \"recharges_path\": null,\r\n",
      "            \"version\": \"0.4.0\",\r\n",
      "            \"code_signature\": \"9f648eb0aa73d5e521c7e8a2bf8256697a467bdf\",\r\n",
      "            \"groupby\": null,\r\n",
      "            \"split_week\": false,\r\n",
      "            \"split_day\": false,\r\n",
      "            \"start_time\": \"2014-03-02 07:13:30\",\r\n",
      "            \"end_time\": \"2014-04-14 12:04:37\",\r\n",
      "            \"night_start\": \"19:00:00\",\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 15 demo_export_user.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extending bandicoot\n",
    "\n",
    "You can easily develop your indicator using the ``@grouping`` decorator. You only need to write a function taking as input a list of records and returning an integer or a list of integers (for a distribution). The ``@grouping`` decorator wraps the function and call it for each group of weeks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from bandicoot.helper.group import grouping\n",
    "\n",
    "@grouping(interaction='call')\n",
    "def shortest_call(records):\n",
    "    in_durations = (r.call_duration for r in records)\n",
    "    return min(in_durations)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "    \"allweek\": {\n",
       "        \"allday\": {\n",
       "            \"call\": {\n",
       "                \"mean\": 1330.857142857143, \n",
       "                \"std\": 2200.2680634459994\n",
       "            }\n",
       "        }\n",
       "    }\n",
       "}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shortest_call(U)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "run_control": {
     "frozen": false,
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "    \"allweek\": {\n",
       "        \"allday\": {\n",
       "            \"call\": {\n",
       "                \"mean\": 1330.857142857143, \n",
       "                \"std\": 2200.2680634459994\n",
       "            }\n",
       "        }, \n",
       "        \"day\": {\n",
       "            \"call\": {\n",
       "                \"mean\": 1395.5714285714287, \n",
       "                \"std\": 2186.820187078088\n",
       "            }\n",
       "        }, \n",
       "        \"night\": {\n",
       "            \"call\": {\n",
       "                \"mean\": 1182.1666666666667, \n",
       "                \"std\": 908.7290422464895\n",
       "            }\n",
       "        }\n",
       "    }\n",
       "}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shortest_call(U, split_day=True)"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  },
  "toc": {
   "toc_cell": false,
   "toc_number_sections": false,
   "toc_threshold": 6,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}