deathbeds/2018-07-08-The-simplest-path-hook-importer-for-a-notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reusing `importlib`, `nbconvert`, and `nbformat` to import notebooks\n",
    "\n",
    "A lot of code is written in Jupyter notebooks, but little of it is reused. Currently, there is ambiguity in [how authors design their computational essays](https://blog.jupyter.org/we-analyzed-1-million-jupyter-notebooks-now-you-can-too-guest-post-8116a964b536). \n",
    "\n",
    "At deathbeds, we are making an effort to compose our notebooks so they can be reused as modules.  This means that our essays can be loaded into Python using the import system.  To do this we rely on importnb which is a package dedicated to importing notebooks. \n",
    "\n",
    "In this post we focus on the most basic steps to add notebooks to the input system.  Most demonstrations of custom imports use [a meta finder](http://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Importing%20Notebooks.html4\n",
    "), here we define a path hook. The path hook finder will look for notebooks matching a qualified notebook any where along the sys path.  Any notebook in a Python package becomes a potential module. \n",
    "\n",
    "    %reload_ext deathbeds.__The_simplest__import_notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "    from importlib.machinery import SourceFileLoader, FileFinder\n",
    "    from importlib.util import decode_source, cache_from_source\n",
    "    import sys, inspect\n",
    "    from nbconvert.exporters.python import PythonExporter\n",
    "    from nbformat.v4 import reads\n",
    "    Ø = __name__ == '__main__'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Notebook Source File Loader\n",
    "\n",
    "The notebook loader is very similar to a Python loader, it just needs to convert the source json into valid Python. The standard source file loader has a [source_to_code](https://docs.python.org/3/library/importlib.html#importlib.abc.InspectLoader.source_to_code) method just for this purpose. To convert the notebook to python byte code we can use [nbconvert](https://github.com/jupyter/nbconvert)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "    class NotebookLoader(SourceFileLoader):\n",
    "        def source_to_code(self, data, path=None):\n",
    "            return compile(PythonExporter().from_notebook_node(\n",
    "                reads(decode_source(data))\n",
    "            )[0], path, 'exec')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the native Source file loader means that the  imported module will be cached and is reloadable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding new loader details to an existing FileFinder\n",
    "\n",
    "Creating a new meta finder is easy, adding a new path hook is not.  We are going to do some weird shit to discover the existing file finders for the existing Python imports."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "    def find_finder():\n",
    "        for i, hook in enumerate(sys.path_hooks):\n",
    "            try: \n",
    "                if issubclass(inspect.getclosurevars(hook).nonlocals['cls'], \n",
    "                              FileFinder):return i, hook\n",
    "            except: ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "    def new_loader(*details):\n",
    "        i, finder = find_finder()\n",
    "        sys.path_hooks[i] = FileFinder.path_hook(\n",
    "             *details,*(\n",
    "                 object for object in inspect.getclosurevars(find_finder()[1]).nonlocals['loader_details']\n",
    "                 if not any(map('.ipynb'.__eq__, object[1]))))\n",
    "        \n",
    "        sys.path_importer_cache.clear()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is essential to clear the path importer cache otherwise you will enter a waking nightmare while debugging."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The IPython extension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'NotebookLoader' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-1-f0f7e257c0c7>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[1;32mdef\u001b[0m \u001b[0mload_ipython_extension\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mip\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mloader\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mNotebookLoader\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m     \u001b[0mnew_loader\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mloader\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m(\u001b[0m\u001b[1;34m'.ipynb'\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m     \u001b[0mip\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mast_transformers\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;31m# This is a precaution for the demo\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[0mØ\u001b[0m \u001b[1;32mand\u001b[0m \u001b[0mload_ipython_extension\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mget_ipython\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'NotebookLoader' is not defined"
     ]
    }
   ],
   "source": [
    "    def load_ipython_extension(ip, loader=NotebookLoader): \n",
    "        new_loader((loader, ('.ipynb',)))\n",
    "        ip.ast_transformers = [] # This is a precaution for the demo\n",
    "    Ø and load_ipython_extension(get_ipython())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Demonstration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "    if Ø:\n",
    "        import __style__\n",
    "        assert isinstance(__style__.__loader__, NotebookLoader), \"Some other thing is loaded.\"\n",
    "        assert __import__('pathlib').Path(cache_from_source(__style__.__file__)).exists(), \"\"\"The module is not cached\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion\n",
    "\n",
    "* This approach works, but it does not handle line numbers.  Without line numbers, it becomes difficult to debug a computational essay.\n",
    "* Importing notebooks promotes an essay that will __restart and run all__, a condition essential for importing a Python module.\n",
    "* Notebooks generally have more permissive file names.  The standard Python finder has a limited search strategy for most human named notebooks.  `importnb` attends a solution for a wildcard finders.  That is why the deathbeds posts - which start with numbers & contain hyphens - may be imported."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       " <div id=\"disqus_thread\"></div>\n",
       " <script type=\"text/javascript\">\n",
       "     /* * * CONFIGURATION VARIABLES * * */\n",
       "     // Required: on line below, replace text in quotes with your forum shortname\n",
       "     var disqus_shortname = 'tonyfast';\n",
       "\n",
       "     /* * * DON'T EDIT BELOW THIS LINE * * */\n",
       "     (function() {\n",
       "         var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;\n",
       "         dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';\n",
       "         (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);\n",
       "     })();\n",
       " </script>\n",
       " <noscript>Please enable JavaScript to view the <a href=\"https://disqus.com/?ref_noscript\" rel=\"nofollow\">comments powered by Disqus.</a></noscript>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "    if Ø: import disqus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}