deathbeds/2018-07-26-Importing-notebooks-with-proper-source.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Improving the interactivity of notebook modules\n",
    "\n",
    "A limitation of the current notebook importing systems is that we cannot view the code of imported modules.  Both [`importnb`]() and [The simplest path hook importer](2018-07-08-The-simplest-path-hook-importer-for-a-notebook.ipynb) have this problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "    Ø = __name__ == '__main__'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "    from . import __The_simplest_path_hook_importer_for_a_notebook; from inspect import getsource"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the module level we can recover the source, and it is in fact `json`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "    with open(__The_simplest_path_hook_importer_for_a_notebook.__file__) as file: assert __import__('json').load(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, at the __class__ or __function__ level Python is unable to discover the source.\n",
    "\n",
    "    Ø and getsource(__The_simplest_path_hook_importer_for_a_notebook.load_ipython_extension)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This ultimately effects the interactive experience with the IPython source inspector.\n",
    "\n",
    "    Ø and get_ipython().inspector.psource(__The_simplest_path_hook_importer_for_a_notebook.load_ipython_extension)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recovering the proper source for notebook modules\n",
    "\n",
    "The current problem we are facing in that the `getsource` discovery doesn't have the correct regular expressions to discover source code in json lines.  These lines are discovered in the `linecache` which stores information about files that are loaded in as modules. \n",
    "\n",
    "To return the source with `getsource` must replace assure that valid code is in the `linecache` rather than the raw `json`.   And, we must assure that the source code lines the original json file align with the lines in the `linecache`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "    import linecache"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Aligning a `json` string a sparse code blob."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our custom json decoder will record the slices from the original string where source code exists.  These slices allow us to create an aligned string later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "    def scanstring(s, end, strict=True, **kwargs):\n",
    "        s, id = py_scanstring(s, end, strict, **kwargs)\n",
    "        return (slice(end, id), s), id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `object_pairs_hook` filters the import parts of the decoded source for us. We are ultimately working with a list pairs of slices and source code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "    def object_pairs_hook(object) -> (slice, str):\n",
    "        object = dict(object)\n",
    "        if 'cells' in object: return object['cells']\n",
    "        \n",
    "        if 'cell_type' in object: \n",
    "            _, object['cell_type'] = object['cell_type']\n",
    "        \n",
    "        if 'source' in object:\n",
    "            if object['source']:\n",
    "                source = ''.join(_[1] for _ in object['source'])\n",
    "                if object['cell_type'] == 'markdown':\n",
    "                    source = \"'''\" + source + \"'''\"\n",
    "                    object['cell_type'] = 'code'\n",
    "                if object['cell_type'] == 'code':\n",
    "                    return slice(object['source'][0][0].start, object['source'][-1][0].stop), source\n",
    "        return slice(None), None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The `json` decoder\n",
    "\n",
    "This machinery supervises the decoding of a notebook on disk to source.  The source file is aligned with the code by filling in non code with whitespace.  The source code of a notebook will be very sparse."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "    from json.scanner import py_make_scanner\n",
    "    from json.decoder import JSONDecoder, WHITESPACE, WHITESPACE_STR, JSONObject, py_scanstring\n",
    "\n",
    "    class LineCacheDecoder(JSONDecoder):\n",
    "        def __init__(self, **kwargs):\n",
    "            super().__init__(**kwargs)\n",
    "            self.parse_string = scanstring\n",
    "            self.scan_once = py_make_scanner(self)\n",
    "\n",
    "        def decode(self, object, filename, transform=__import__('textwrap').dedent):\n",
    "            lines = []\n",
    "            linecache.updatecache(filename)\n",
    "            linecache.cache[filename] = *linecache.cache[filename][:2], lines, filename\n",
    "            last, new, old = slice(0, 0), 0, 0 \n",
    "            for current, cell in super().decode(object):\n",
    "                if cell:\n",
    "                    lines += ['\\n'] * (object[last.stop:current.start].splitlines().__len__() - 1 + (old-new))\n",
    "                    transformed = transform(cell)\n",
    "                    lines += list(map(\"{}\\n\".format, transformed.splitlines()))\n",
    "                    new, old = map(len, map(str.splitlines, (transformed, object[current])))\n",
    "                    if not lines[-1]: lines.pop()\n",
    "                    last = current\n",
    "            return ''.join(lines)\n",
    "\n",
    "    decoder = LineCacheDecoder(object_pairs_hook=object_pairs_hook)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This decoder will update the `linecache` with valid python source to be used for introspection. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Source File Loader\n",
    "\n",
    "... invokes the `decoder`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "    from importlib.machinery import SourceFileLoader; from importlib.util import decode_source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "    class FlatSourceLoader(SourceFileLoader):\n",
    "        def get_data(self, path):\n",
    "            return decoder.decode(\n",
    "                decode_source(super().get_data(self.path)), self.path, \n",
    "                get_ipython().input_transformer_manager.transform_cell)\n",
    "        \n",
    "        get_source = get_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install the loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "    def load_ipython_extension(ip=None):\n",
    "        __The_simplest_path_hook_importer_for_a_notebook.load_ipython_extension(None, FlatSourceLoader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "    Ø and load_ipython_extension()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Voila.\n",
    "\n",
    "The new decoder and source file loader makes the notebook source code inspectable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "    from pytest import fixture, raises"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "    @fixture\n",
    "    def module():  import Untitled; return __import__('importlib').reload(Untitled)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`black` can clean up the sparse source created by the decoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "    def _is_a_valid_module(module):\n",
    "        assert module.__file__.endswith('.ipynb')\n",
    "        assert __import__('ast').parse(__import__('black').format_str(getsource(module.test_thing), 100))\n",
    "        assert __import__('ast').parse(__import__('black').format_str(getsource(module), 100))\n",
    "        with raises(__import__('json').JSONDecodeError):\n",
    "            __import__('json').loads(getsource(module))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "29px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}