docs/tutorials/ml_workflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example ML Flow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    "We'll consider a typical (but simplified) machine learning flow.\n",
    "\n",
    "*(The code for this example is available in the Bionic repo at\n",
    "example/ml_workflow.py.  Run* ``pip install 'bionic[examples]'``\n",
    "*to ensure you have the required dependencies.)*\n",
    "\n",
    ".. literalinclude:: ../../example/ml_workflow.py\n",
    "    :language: python\n",
    "    :linenos:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {"nbsphinx": "hidden"},
   "outputs": [],
   "source": [
    "# Configure the PYTHONPATH for this notebook.\n",
    "import _tutorial_setup\n",
    "\n",
    "# Delete the Bionic cache directory, to make sure we're starting fresh.\n",
    "import shutil\n",
    "shutil.rmtree('./bndata', ignore_errors=True)\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    "Let's start by importing the flow into a notebook and visualizing it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from example.ml_workflow import flow\n",
    "\n",
    "flow.render_dag()"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    "This is still a toy example, but it demonstrates a classic supervised\n",
    "machine learning flow for binary classification:\n",
    "\n",
    "* Load data (``raw_frame``)\n",
    "* Clean it (``features_frame``)\n",
    "* Split into training and test sets (``train_frame`` and ``test_frame``)\n",
    "* Fit a model on the training data (``model``)\n",
    "* Evaluate the model on the test data (``predictions_frame``, ``precision_recall_frame``)\n",
    "* Visualize the model's performance (``all_hyperparams_pr_plot``)\n",
    "\n",
    "As in the previous example, we can access any of the entity values.\n",
    "However, this time we'll enable logging so we can see every entity that Bionic computes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging; logging.basicConfig(level='INFO', format='%(message)s')\n",
    "\n",
    "flow.get.train_frame().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flow.get.all_hyperparams_pr_plot()"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    "Note that in the logs, each entity is computed exactly once --\n",
    "every value is cached both in memory and on disk to avoid redundant computation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    "## Changing Inputs"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    "Now that we have our flow, we might want to experiment with\n",
    "different parameter settings.\n",
    "For example, we can try changing our regularization hyperparameter ``C``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flow.setting('hyperparams_dict', {'C': 10}).get.all_hyperparams_pr_plot()"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    "Bionic re-runs the flow with the changed parameter,\n",
    "generating a new plot.\n",
    "\n",
    "Since we've created a new copy of our flow,\n",
    "Bionic doesn't try to reuse its in-memory cache.\n",
    "However, we can see it that it only recomputed those\n",
    "entities whose values changed -- the others were loaded from disk.\n",
    "\n",
    "Similarly, we can try training our model on a subset of the available features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flow.setting('feature_inclusion_regex', 'mean.*').get.all_hyperparams_pr_plot()"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    "Naturally, reducing the number of features makes the model perform worse.\n",
    "\n",
    "We can also apply this flow to a completely different dataset,\n",
    "such as the UCI ML Wine dataset.\n",
    "(This is actually a multiclass dataset,\n",
    "but to match the original example we'll use binary labels.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "import pandas as pd\n",
    "\n",
    "wine_data = datasets.load_wine()\n",
    "wine_frame = pd.DataFrame(\n",
    "    data=wine_data['data'],\n",
    "    columns=wine_data['feature_names'])\n",
    "wine_frame['target'] = (wine_data['target'] == 0)\n",
    "\n",
    "wine_flow = flow.setting('raw_frame', wine_frame)\n",
    "wine_flow.get.all_hyperparams_pr_plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiplicity"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    "Now we have a flow for evaluating any given set of parameters.\n",
    "But usually we want to compare multiple variations at once,\n",
    "within a single flow.\n",
    "Bionic lets us do this by setting multiple values for a single entity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flow3 = flow.setting('hyperparams_dict', values=[{'C': 0.1}, {'C': 1}, {'C': 10}])\n",
    "flow3.get.all_hyperparams_pr_plot()"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    "Here we can see the PR curves for each of three hyperparameter configurations.\n",
    "But how did this work?\n",
    "Let's start by visualizing the dependency graph of our new flow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flow3.render_dag()"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    "When we specify multiple values for an entity, \n",
    "Bionic creates multiple instances of that entity *and of all downstream entities*.\n",
    "If there are three instances of ``hyperparams_dict``,\n",
    "there must also be three instances of ``model`` and ``precision_recall_frame``.\n",
    "\n",
    "However, we can also see that there is only one instance of ``all_hyperparams_plot_pr``.\n",
    "This is because we used the ``@gather`` decorator when defining this entity:\n",
    "Bionic has aggregated all the variations of ``hyperparameters_dict`` and ``precision_recall_frame``\n",
    "into a single frame, so they can be plotted together.\n",
    "\n",
    "Now, because we have three different model instances,\n",
    "it no longer makes sense to request \"the\" model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    flow3.get.model()\n",
    "except Exception as e:\n",
    "    caught_exception = e\n",
    "caught_exception"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    "If we want to access an entity with multiple values,\n",
    "we need to tell Bionic what data structure to use to\n",
    "aggregate them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flow3.get.model('series')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extended Multiplicity"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    "We can apply the same approach to any number of entities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flow6 = flow3.setting('feature_inclusion_regex', values=['.*', 'mean.*'])\n",
    "flow6.render_dag()"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    "Here we can see that each entity's multiplicity is propagated through the entire graph.\n",
    "For example, there are two ``train_frame`` nodes and two ``test_frame`` nodes,\n",
    "because those depend on ``feature_inclusion_regex`` but not on ``hyperparams_dict``.\n",
    "However, there are six ``model`` nodes, because there are six (two times three) combinations of\n",
    "``train_frame`` and ``hyperparams_dict``.\n",
    "\n",
    "We can also see two instances of ``all_hyperparams_pr_plot``,\n",
    "corresponding to the two instances of ``feature_inclusion_regex``.\n",
    "This illustrates another nuance of how we used ``@gather``:\n",
    "we asked Bionic to include *every* variation of ``hyperparams_dict``,\n",
    "*along with* the associated values of ``precision_recall_frame``.\n",
    "If we had used\n",
    "\n",
    ".. code-block:: python\n",
    "\n",
    "    @gather(\n",
    "        over=['hyperparams_dict', 'feature_inclusion_regex'],\n",
    "        also='precision_recall_frame',\n",
    "        into='gathered_frame')\n",
    "    def all_variations_pr_plot(...):\n",
    "        ...\n",
    "\n",
    "then all the values would be gathered into a single frame,\n",
    "and there would be one single plot node."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrapping Up"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "raw_mimetype": "text/restructuredtext"
   },
   "source": [
    "This tutorial has illustrated four topics:\n",
    "\n",
    "1. How to use Bionic to assemble a more complex data science flow.\n",
    "2. How Bionic uses caching to avoid redundant computation.\n",
    "3. How to repurpose an existing flow to use new parameters or data.\n",
    "4. How to use multiplicity to compare several variations within a single flow.\n",
    "\n",
    "These concepts should be enough to get started building real flows.\n",
    "The next section of the documentation will discuss the underlying concepts in Bionic."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}