docs/example_notebooks/DockerExampleNB2KG.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example notebook - nb2kg\n",
    "\n",
    "This notebook is an example of connecting a jupyter notebook to IBM Analytics Engine (IAE) using [nb2kg](https://github.com/jupyter/kernel_gateway_demos/tree/master/nb2kg).  It is expected that you will run this notebook on a local jupyter environment (i.e. not from DSX).\n",
    "\n",
    "The notebook connecting to IAE is spun up in a docker container. This is because:\n",
    "\n",
    "- When you enable the nb2kg extension, all kernels run on the configured Kernel Gateway, instead of on the Notebook server host. The extension does not support local kernels.  If nb2kg is installed in your local notebook environment, it would prevent you from running local kernels. Thus using docker prevents me from corrupting your local notebook environment.\n",
    "- Setting up a notebook environment with all the dependencies for nb2kg can be tricky.  Docker allows me to provide you with an environment that is pre-configured.\n",
    "\n",
    "Python is used to interact with docker because this was easier to script in a notebook.  However, if you run docker with `sudo`, this notebook may not work for you.\n",
    "\n",
    "**WARNING:** This project is just a demo of connecting to IAE using nb2kg.  Your mileage may vary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "! pip install --quiet docker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the IBM Analytics Engine (IAE) credentials.  See [this example notebook](./CreateCluster.ipynb) for creating an IAE instance and saving the vcap.json credentials file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import json\n",
    "vcap = json.load(open('./vcap.json'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import docker\n",
    "client = docker.from_env()\n",
    "api_client = docker.APIClient()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The docker images can be quite large and take a long time to load.  Here we load the parent image and regularly print some output so you can see what is going on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "lines_processed = 0\n",
    "\n",
    "api_client = docker.APIClient()\n",
    "for line in api_client.pull('jupyter/minimal-notebook:fa77fe99579b', tag=None, stream=True):\n",
    "    if lines_processed % 25 == 0:\n",
    "        try:\n",
    "            status = json.loads(str(line)[2:-5]) # strip quotes and newline\n",
    "            if 'progressDetail' in status:\n",
    "                print(status['progressDetail']) \n",
    "        except:\n",
    "            pass\n",
    "    lines_processed += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have created a custom docker environment that uses nb2kg.  This environment has been kept as simple as possible to make it easy for you to adapt to to your own requirements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "image = client.images.build(\n",
    "    path='https://github.com/snowch/docker_jupyter_notebook_kg.git', \n",
    "    tag='docker_jupyter_notebook_kg')\n",
    "\n",
    "print(image.tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Delete any containers hanging around from a previous run of this notebook.\n",
    "\n",
    "**WARNING:** Ensure you backup any work you want to keep before running this command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from datetime import datetime as dt\n",
    "import dateutil.parser\n",
    "\n",
    "for cont in client.containers.list(filters={ 'name': 'iae_nb2kg_example' }):\n",
    "    created = dt.utcnow() - dateutil.parser.parse(cont.attrs['Created']).replace(tzinfo=None)\n",
    "    print(\"Name: {} | Status: {} | Age (H:M:S): {}\".format(\n",
    "                         cont.attrs['Name'],\n",
    "                         cont.attrs['State']['Status'],\n",
    "                         created\n",
    "                         ))\n",
    "    #print(json.dumps(cont.attrs, indent=4, sort_keys=True))\n",
    "    cont.kill()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the notebook.  You may need to change the LOCAL_PORT if this port is not free on your local machine.\n",
    "\n",
    "Change the LOCAL_NOTEBOOKS_FOLDER to a folder on your local machine where you want your notebooks in the 'work' folder to be saved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "LOCAL_PORT = 8899\n",
    "LOCAL_NOTEBOOKS_FOLDER = '/Users/snowch/Desktop/notebooks'\n",
    "\n",
    "container = client.containers.run(\n",
    "    image = image,\n",
    "    volumes = { LOCAL_NOTEBOOKS_FOLDER : '/home/jovyan/work' },\n",
    "    ports = {str(LOCAL_PORT)+'/tcp': LOCAL_PORT},\n",
    "    environment = {\n",
    "        'NB_PORT': LOCAL_PORT,\n",
    "        'KG_HTTP_USER': vcap['cluster']['user'],\n",
    "        'KG_HTTP_PASS': vcap['cluster']['password'],\n",
    "        'KG_URL': vcap['cluster']['service_endpoints']['notebook_gateway'],\n",
    "        'KG_WS_URL': vcap['cluster']['service_endpoints']['notebook_gateway_websocket'],\n",
    "        'KG_CONNECT_TIMEOUT': '50.0',\n",
    "        'KG_REQUEST_TIMEOUT': '40.0'\n",
    "    },\n",
    "    detach=True,\n",
    "    stdout=True,\n",
    "    stderr=True,\n",
    "    remove=True,\n",
    "    name='iae_nb2kg_example'\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next cell prints the log output.  Ensure there are no errors reported.\n",
    "\n",
    "You should see a url, e.g.\n",
    "\n",
    "```\n",
    "Copy/paste this URL into your browser when you connect for the first time,\n",
    "    to login with a token:\n",
    "        http://0.0.0.0:8899/?token=12345\n",
    "```\n",
    "\n",
    "Click on the url in the output to open it in your browser.  \n",
    "\n",
    "From here, you should be able to create a new notebook: <img src=\"./spark_kernels.png\" width=\"200px\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from IPython.lib import backgroundjobs as bg\n",
    "jobs = bg.BackgroundJobManager()\n",
    "\n",
    "def printlogs():\n",
    "    for line in container.logs(stream=True):\n",
    "        print(str(line)[2:-3]) # strip quotes and newline\n",
    "    sys.stdout.flush()\n",
    "\n",
    "jobs.new('printlogs()')\n",
    "jobs.status()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "finally, remove the notebook docker container"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "container.kill()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}