essential_tools_python/essential_tools.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# ~~Batman's~~ Developer's Utility Belt\n",
    "<br><br><br><br>\n",
    "By Rajat Goyal\n",
    "\n",
    "@rajat404"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Shell Utilities\n",
    "<br>\n",
    "Some language agnostic utilities, which all developers can benefit from\n",
    "<br><br><br>\n",
    "\n",
    "To the terminal and beyond..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- [Tig](https://jonas.github.io/tig/)\n",
    "- [mycli](http://mycli.net/)/[pgcli](https://www.pgcli.com/)\n",
    "- [s](https://github.com/zquestz/s)\n",
    "- [howdoi](https://github.com/gleitz/howdoi)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## pip-tools\n",
    "\n",
    "https://github.com/jazzband/pip-tools\n",
    "\n",
    "```\n",
    "$ pip-compile  # To initialize `requirements.in`\n",
    "$ pip-compile requirements.in\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Jupyter Extensions\n",
    "\n",
    "\n",
    "- https://github.com/Jupyter-contrib/jupyter_nbextensions_configurator\n",
    "\n",
    "- http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## hr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-06-26T08:55:51.564314Z",
     "start_time": "2017-06-26T08:55:51.556412Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#########################################################################################\n",
      "it works!\n",
      "*****************************************************************************************\n"
     ]
    }
   ],
   "source": [
    "from hr import hr\n",
    "hr()\n",
    "print('it works!')\n",
    "hr('*')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Flower\n",
    "\n",
    "Celery monitoring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Arrow\n",
    "\n",
    "Lets you create, manipulate and format Time ~~& Space~~\n",
    "\n",
    "<br><br><br><br>\n",
    "https://arrow.readthedocs.io/en/latest/\n",
    "\n",
    "(examples from docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-06-26T08:55:53.200080Z",
     "start_time": "2017-06-26T08:55:53.180736Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arrow.utcnow()             2017-06-26T08:56:54.529540+00:00\n",
      "arrow.now()                2017-06-26T14:26:54.529930+05:30\n",
      "arrow.now(\"Asia/Kolkata\")  2017-06-26T14:26:54.531383+05:30\n",
      "arrow.now(\"US/Pacific\")    2017-06-26T01:56:54.532811-07:00\n"
     ]
    }
   ],
   "source": [
    "import arrow\n",
    "print('arrow.utcnow()            ', arrow.utcnow())\n",
    "print('arrow.now()               ', arrow.now())\n",
    "print('arrow.now(\"Asia/Kolkata\") ', arrow.now('Asia/Kolkata'))\n",
    "print('arrow.now(\"US/Pacific\")   ', arrow.now('US/Pacific'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Read from multiple formats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-06-26T08:55:53.778339Z",
     "start_time": "2017-06-26T08:55:53.760811Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arrow.get(1367900664) \n",
      " 2013-05-07T04:24:24+00:00 \n",
      "\n",
      "arrow.get('2013-05-05 12:30:45', 'YYYY-MM-DD HH:mm:ss') \n",
      " 2013-05-05T12:30:45+00:00 \n",
      "\n",
      "arrow.get('June was born in May 1980', 'MMMM YYYY') \n",
      " 1980-05-01T00:00:00+00:00 \n",
      "\n",
      "arrow.get('2013-09-30T15:34:00.000-07:00') \n",
      " 2013-09-30T15:34:00-07:00 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "from dateutil import tz\n",
    "from datetime import datetime\n",
    "print(\"arrow.get(1367900664)\", '\\n', arrow.get(1367900664), '\\n')\n",
    "print(\"arrow.get('2013-05-05 12:30:45', 'YYYY-MM-DD HH:mm:ss')\", '\\n', arrow.get('2013-05-05 12:30:45', 'YYYY-MM-DD HH:mm:ss'), '\\n')\n",
    "print(\"arrow.get('June was born in May 1980', 'MMMM YYYY')\", '\\n', arrow.get('June was born in May 1980', 'MMMM YYYY'), '\\n')\n",
    "# Some ISO-8601 compliant strings are recognized and parsed without a format string:\n",
    "print(\"arrow.get('2013-09-30T15:34:00.000-07:00')\", '\\n', arrow.get('2013-09-30T15:34:00.000-07:00'), '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Directly create Arrow objects\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arrow.get(2013, 5, 5) \n",
      " 2013-05-05T00:00:00+00:00 \n",
      "\n",
      "arrow.Arrow(2013, 5, 5) \n",
      " 2013-05-05T00:00:00+00:00 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"arrow.get(2013, 5, 5)\", '\\n', arrow.get(2013, 5, 5), '\\n')\n",
    "print(\"arrow.Arrow(2013, 5, 5)\", '\\n', arrow.Arrow(2013, 5, 5), '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Compatibility?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'arrow.arrow.Arrow'> 2017-06-26T14:26:54.576306+05:30\n",
      "<class 'datetime.datetime'> 2017-06-26 14:26:54.576306+05:30\n"
     ]
    }
   ],
   "source": [
    "a = arrow.now()\n",
    "print(type(a), a)\n",
    "print(type(a.datetime), a.datetime)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a.timestamp   1498467414\n",
      "a.naive       2017-06-26 14:26:54.576306\n",
      "a.tzinfo      tzlocal()\n",
      "a.year        2017\n",
      "a.day         26\n",
      "a.date()      2017-06-26\n"
     ]
    }
   ],
   "source": [
    "print('a.timestamp  ', a.timestamp)\n",
    "print('a.naive      ', a.naive)\n",
    "print('a.tzinfo     ', a.tzinfo)\n",
    "print('a.year       ', a.year)\n",
    "print('a.day        ', a.day)\n",
    "print('a.date()     ', a.date())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Manipulation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2017-06-26T14:26:54.597894+05:30\n"
     ]
    }
   ],
   "source": [
    "arw = arrow.now()\n",
    "print(arw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arw.replace(hour=6)   2017-06-26T06:26:54.597894+05:30\n",
      "arw.replace(hours=6)  2017-06-26T20:26:54.597894+05:30\n"
     ]
    }
   ],
   "source": [
    "print(\"arw.replace(hour=6)  \", arw.replace(hour=6))     # Replace the hour\n",
    "print(\"arw.replace(hours=6) \", arw.replace(hours=6))  # Adds to existing hours"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Excel/CSV\n",
    "\n",
    "<br><br><br><br><br><br><br>\n",
    "P.S: Uggghh! :("
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Tablib\n",
    "\n",
    "import, export, and manipulate tabular data sets\n",
    "\n",
    "<br><br><br><br>\n",
    "http://docs.python-tablib.org/\n",
    "\n",
    "(examples from docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import tablib\n",
    "data = tablib.Dataset()\n",
    "\n",
    "data.headers = ['Name', 'Psychological Disorders']\n",
    "\n",
    "data.append(['Gollum', 'unhealthy obsession with rings'])\n",
    "data.append(['Norman Osborn', 'Schizophrenia'])\n",
    "data.append(['Joker', 'plain crazy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name         |Psychological Disorders       \n",
      "-------------|------------------------------\n",
      "Gollum       |unhealthy obsession with rings\n",
      "Norman Osborn|Schizophrenia                 \n",
      "Joker        |plain crazy                   \n"
     ]
    }
   ],
   "source": [
    "print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{\"Name\": \"Gollum\", \"Psychological Disorders\": \"unhealthy obsession with rings\"}, {\"Name\": \"Norman Osborn\", \"Psychological Disorders\": \"Schizophrenia\"}, {\"Name\": \"Joker\", \"Psychological Disorders\": \"plain crazy\"}]\n"
     ]
    }
   ],
   "source": [
    "print(data.json)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name,Psychological Disorders\r\n",
      "Gollum,unhealthy obsession with rings\r\n",
      "Norman Osborn,Schizophrenia\r\n",
      "Joker,plain crazy\r\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(data.csv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<table>\n",
      "<thead>\n",
      "<tr><th>Name</th>\n",
      "<th>Psychological Disorders</th></tr>\n",
      "</thead>\n",
      "<tr><td>Gollum</td>\n",
      "<td>unhealthy obsession with rings</td></tr>\n",
      "<tr><td>Norman Osborn</td>\n",
      "<td>Schizophrenia</td></tr>\n",
      "<tr><td>Joker</td>\n",
      "<td>plain crazy</td></tr>\n",
      "</table>\n"
     ]
    }
   ],
   "source": [
    "print(data.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>Name</th>\n",
       "<th>Psychological Disorders</th></tr>\n",
       "</thead>\n",
       "<tr><td>Gollum</td>\n",
       "<td>unhealthy obsession with rings</td></tr>\n",
       "<tr><td>Norman Osborn</td>\n",
       "<td>Schizophrenia</td></tr>\n",
       "<tr><td>Joker</td>\n",
       "<td>plain crazy</td></tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "display(HTML(data.html))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Simply dump the dataset into a file\n",
    "with open('data.csv', 'w') as f:\n",
    "    f.write(data.csv) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Importing Data via Tablib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name         |Psychological Disorders       \n",
      "-------------|------------------------------\n",
      "Gollum       |unhealthy obsession with rings\n",
      "Norman Osborn|Schizophrenia                 \n",
      "Joker        |plain crazy                   \n"
     ]
    }
   ],
   "source": [
    "import tablib\n",
    "imported_data = tablib.Dataset().load(open('data.csv').read())\n",
    "print(imported_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Pandas\n",
    "\n",
    "The Backstory..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-06-26T08:55:24.384776Z",
     "start_time": "2017-06-26T08:55:24.307840Z"
    },
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Get excel sheet with all the names\n",
    "all_names = pd.read_excel('names.xlsx')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# logic to tabulate the frequency of people's name by alphabet\n",
    "def first_letter(row):\n",
    "    return row[0].lower()\n",
    "\n",
    "all_names['first'] = all_names['names'].apply(first_letter)\n",
    "group = all_names.groupby('first')\n",
    "count_by_letter = group.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# count_by_letter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "count_by_letter.to_excel('count_by_letter.xlsx')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Collections"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderedDict([('b', 'bilbo'), ('a', 'baggins')])\n"
     ]
    }
   ],
   "source": [
    "# collections.OrderedDict\n",
    "\n",
    "from collections import OrderedDict\n",
    "d = OrderedDict()\n",
    "d['b'] = 'bilbo'\n",
    "d['a'] = 'baggins'\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "d = defaultdict(list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Django"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- [django-import-export](https://django-import-export.readthedocs.io/en/latest/)\n",
    "- [cookiecutter-django](https://github.com/pydanny/cookiecutter-django)\n",
    "- [django-init](https://github.com/Fueled/django-init)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## How to make slides programmatically (like this one!)\n",
    "\n",
    "https://github.com/damianavila/RISE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Last but not the least\n",
    "\n",
    "- https://pyformat.info\n",
    "- https://awesome-python.com/\n",
    "- https://pymotw.com/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## That's all folks!"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}