tutorials/4-caching-persistence.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data persistence and data caching"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook presents data persistence and data caching features in steps.\n",
    "* Persistence helps to avoid re-running early steps of a pipeline when subsequent steps are changed\n",
    "* Caching makes it possible to run complex, multi-path pipelines without re-computing the results of early steps\n",
    "\n",
    "Note that the features presented here are different from *model persistence*, which saves the transformers as the steps are trained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.externals import joblib\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from steppy.base import Step, BaseTransformer\n",
    "from steppy.adapter import Adapter, E\n",
    "EXPERIMENT_DIR_A = './ex4a'\n",
    "EXPERIMENT_DIR_B = './ex4b'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil\n",
    "\n",
    "# By default pipelines will try to load previously trained models so we delete the cache to ba sure we're starting from scratch\n",
    "shutil.rmtree(EXPERIMENT_DIR_A, ignore_errors=True)\n",
    "shutil.rmtree(EXPERIMENT_DIR_B, ignore_errors=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time we'll have a look at text classification. We'll use the classic 20newsgroups dataset, but without the headers, footers or quotes which would make the task too easy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_20newsgroups\n",
    "\n",
    "newsgroups_train = fetch_20newsgroups(subset='train', remove=('headers', 'footers', 'quotes'))\n",
    "newsgroups_test = fetch_20newsgroups(subset='test', remove=('headers', 'footers', 'quotes'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, y_train = newsgroups_train.data, newsgroups_train.target\n",
    "\n",
    "X_fit, X_val, y_fit, y_val = train_test_split(X_train, y_train, test_size=0.1, stratify=y_train, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use a label encoder to ensure our labels are well-behaved"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "input_label_enc = LabelEncoder().fit(newsgroups_train.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time we have pre-defined training and test sets but we would like to have a hold-out set of training data available for ensembling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_fit = {'input':\n",
    "                {\n",
    "                     'text': X_fit,\n",
    "                     'label': input_label_enc.transform(y_fit),\n",
    "                }\n",
    "            }\n",
    "\n",
    "data_val = {'input':\n",
    "                {\n",
    "                     'text': X_val,\n",
    "                     'label': input_label_enc.transform(y_val),\n",
    "                }\n",
    "            }\n",
    "\n",
    "data_test = {'input':\n",
    "                {\n",
    "                     'text': newsgroups_test.data,\n",
    "                     'label': input_label_enc.transform(newsgroups_test.target),\n",
    "                }\n",
    "            }\n",
    "\n",
    "def print_data_summary(data, title):\n",
    "    print(title)\n",
    "    print('  Num. documents: {}'.format(len(data['input']['text'])))\n",
    "    print('  Num. categories: {}'.format(len(np.unique(data['input']['label']))))\n",
    "\n",
    "for data, title in [(data_fit, 'Model fitting data'), (data_val, 'Validation data'), (data_test, 'Testing data')]:\n",
    "    print_data_summary(data, title)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text processing transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define a transformer that does count vectorization on our documents - again, we can just wrap the one from sklearn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "class CountVecTransformer(BaseTransformer):\n",
    "    def __init__(self, max_features):\n",
    "        self.estimator = CountVectorizer(max_features=max_features)\n",
    "        \n",
    "    def fit(self, X):\n",
    "        self.estimator.fit(X)\n",
    "        return self\n",
    "\n",
    "    def transform(self, X, **kwargs):\n",
    "        X_tfm = self.estimator.transform(X)\n",
    "        return {'X': X_tfm}\n",
    "    \n",
    "    def persist(self, filepath):\n",
    "        joblib.dump(self.estimator, filepath)\n",
    "        \n",
    "    def load(self, filepath):\n",
    "        self.estimator = joblib.load(filepath)\n",
    "        return self"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly for the IDFs in our TF-IDF model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "\n",
    "class StepsTfidfTransformer(BaseTransformer):\n",
    "    def __init__(self):\n",
    "        self.estimator = TfidfTransformer()\n",
    "        \n",
    "    def fit(self, X):\n",
    "        self.estimator.fit(X)\n",
    "        return self\n",
    "\n",
    "    def transform(self, X, **kwargs):\n",
    "        X_tfm  = self.estimator.transform(X)\n",
    "        return {'X': X_tfm}\n",
    "    \n",
    "    def persist(self, filepath):\n",
    "        joblib.dump(self.estimator, filepath)\n",
    "        \n",
    "    def load(self, filepath):\n",
    "        self.estimator = joblib.load(filepath)\n",
    "        return self"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will give us a bunch of features to train on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a first attempt, we'll try to do our predictions with (sparse) logistic regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "class SparseLogRegProbaTransformer(BaseTransformer):\n",
    "    def __init__(self):\n",
    "        self.estimator = LogisticRegression(penalty='l1', multi_class='auto', solver='liblinear')\n",
    "        \n",
    "    def fit(self, X, y):\n",
    "        self.estimator.fit(X, y)\n",
    "        return self\n",
    "\n",
    "    def transform(self, X, **kwargs):\n",
    "        y_proba  = self.estimator.predict_proba(X)\n",
    "        return {'y_proba': y_proba}\n",
    "    \n",
    "    def persist(self, filepath):\n",
    "        joblib.dump(self.estimator, filepath)\n",
    "        \n",
    "    def load(self, filepath):\n",
    "        self.estimator = joblib.load(filepath)\n",
    "        return self"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "count_vec_step = Step(name='CountVec',\n",
    "                      transformer=CountVecTransformer(max_features=1000),\n",
    "                      input_data=['input'],\n",
    "                      adapter=Adapter({'X': E('input', 'text')}),\n",
    "                      experiment_directory=EXPERIMENT_DIR_A,\n",
    "                      is_fittable=True)\n",
    "\n",
    "tfidf_step = Step(name='TF-IDF',\n",
    "                  transformer=StepsTfidfTransformer(),\n",
    "                  input_steps=[count_vec_step],        \n",
    "                  experiment_directory=EXPERIMENT_DIR_A,\n",
    "                  persist_output=True,\n",
    "                  load_persisted_output=True,  # This breaks when switching from training data to val data or test data!\n",
    "                  is_fittable=True\n",
    "                  )\n",
    "\n",
    "logreg_step = Step(name='SparseLogReg',\n",
    "                   transformer=SparseLogRegProbaTransformer(),\n",
    "                   input_steps=[tfidf_step],\n",
    "                   input_data=['input'],\n",
    "                   adapter=Adapter({'X': E('TF-IDF', 'X'),\n",
    "                                    'y': E('input', 'label')\n",
    "                                   }),\n",
    "                   experiment_directory=EXPERIMENT_DIR_A,\n",
    "                   is_fittable=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we have passed `persist_output=True` to the `tfidf_step` constructor. This will make this step save its output so that once it's been computed once, it can later just be loaded from disk. Therefore, we will be able to work on the logistic regression classifier without having to re-compute the outputs of its ancestor nodes.  Additionally, we have also set `load_persisted_output=True`, which tells this step to load the previously computed and saved outputs instead of processing the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logreg_step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds_linear_fit = logreg_step.fit_transform(data_fit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "acc_linear_fit = accuracy_score(y_true=data_fit['input']['label'], y_pred=np.argmax(preds_linear_fit['y_proba'], axis=1))\n",
    "print('Model fitting accuracy: {:.4f}'.format(acc_linear_fit))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Bug workaround: manually delete saved output when switching datasets\n",
    "os.remove(tfidf_step.experiment_directory_output_step)\n",
    "preds_linear_val = logreg_step.transform(data_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "acc_linear_val = accuracy_score(y_true=data_val['input']['label'], y_pred=np.argmax(preds_linear_val['y_proba'], axis=1))\n",
    "print('Validation accuracy: {:.4f}'.format(acc_linear_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random forest model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an alternative, we'll also build a neural net model on top of the same TF-IDF features. We'll use the multi-layer perceptron (MLP) which is available in Scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "class RfClfTransformer(BaseTransformer):\n",
    "    def __init__(self, n_estimators, max_depth):\n",
    "        self.estimator = RandomForestClassifier(n_estimators=n_estimators, max_depth=max_depth)\n",
    "        \n",
    "    def fit(self, X, y):\n",
    "        self.estimator.fit(X, y)\n",
    "        return self\n",
    "\n",
    "    def transform(self, X, **kwargs):\n",
    "        y_proba  = self.estimator.predict_proba(X)\n",
    "        return {'y_proba': y_proba}\n",
    "    \n",
    "    def persist(self, filepath):\n",
    "        joblib.dump(self.estimator, filepath)\n",
    "        \n",
    "    def load(self, filepath):\n",
    "        self.estimator = joblib.load(filepath)\n",
    "        return self"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_step = Step(name='RF',\n",
    "               transformer=RfClfTransformer(n_estimators=200, max_depth=8),\n",
    "               input_steps=[tfidf_step],\n",
    "               input_data=['input'],\n",
    "               adapter=Adapter({'X': E('TF-IDF', 'X'),\n",
    "                                'y': E('input', 'label')\n",
    "                               }),\n",
    "               experiment_directory=EXPERIMENT_DIR_A,\n",
    "               is_fittable=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, so it was easy to add a different model on top of TF-IDF features. Indeed, this time we will be able to use the **saved** TF-IDF output, so we can get straight to fitting the random forest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Bug workaround: manually delete saved output when switching datasets\n",
    "os.remove(tfidf_step.experiment_directory_output_step)\n",
    "preds_rf_fit = rf_step.fit_transform(data_fit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "acc_rf_fit = accuracy_score(y_true=data_fit['input']['label'], y_pred=np.argmax(preds_rf_fit['y_proba'], axis=1))\n",
    "print('Model fitting accuracy: {:.4f}'.format(acc_rf_fit))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Bug workaround: manually delete saved output when switching datasets\n",
    "os.remove(tfidf_step.experiment_directory_output_step)\n",
    "preds_rf_val = rf_step.transform(data_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "acc_rf_val = accuracy_score(y_true=data_val['input']['label'], \n",
    "                            y_pred=np.argmax(preds_rf_val['y_proba'], axis=1))\n",
    "print('Validation accuracy: {:.4f}'.format(acc_rf_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ensembling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll do simple ensembling by averaging predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AvgTransformer(BaseTransformer):\n",
    "    def __init__(self):\n",
    "        pass\n",
    "        \n",
    "    def fit(self, y_proba_1, y_proba_2):\n",
    "        return self\n",
    "\n",
    "    def transform(self, y_proba_1, y_proba_2, **kwargs):\n",
    "        y_proba  = (y_proba_1 + y_proba_2) / 2\n",
    "        return {'y_proba': y_proba}\n",
    "    \n",
    "    def persist(self, filepath):\n",
    "        joblib.dump({}, filepath)\n",
    "        \n",
    "    def load(self, filepath):\n",
    "        self.estimator = joblib.load(filepath)\n",
    "        return self"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ens_step = Step(name='Ensembler',\n",
    "                transformer=AvgTransformer(),\n",
    "                input_steps=[logreg_step, rf_step],\n",
    "                adapter=Adapter({'y_proba_1': E('SparseLogReg', 'y_proba'),\n",
    "                                 'y_proba_2': E('RF', 'y_proba'),\n",
    "                                }),\n",
    "                experiment_directory=EXPERIMENT_DIR_A,\n",
    "                is_fittable=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ens_step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that for the TF-IDF step we set `cache_output` to `True`. What does this do? Note that the output of the TF-IDF step is used both by RF and SparseLogReg. This means that when we run the Ensemble node on some data, it will in turn call MLP and SparseLogReg, which will both call TF-IDF. Without caching, this would mean we're computing the output of the TF-IDF step twice, which is definitely a waste of precious compute time and could possibly lead to some inconsistencies in the data (e.g.  if the TF-IDF step was randomized in some way). Caching solves both problems without keeping anything in memory - the caching is done on disk, not in RAM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.remove(tfidf_step.experiment_directory_output_step)  # Bug workaround: manually delete saved output when switching datasets\n",
    "preds_ens_val = ens_step.fit_transform(data_val)\n",
    "\n",
    "os.remove(tfidf_step.experiment_directory_output_step)  # Bug workaround: manually delete saved output when switching datasets\n",
    "preds_ens_test = ens_step.transform(data_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "acc_ens_val = accuracy_score(y_true=data_val['input']['label'], y_pred=np.argmax(preds_ens_val['y_proba'], axis=1))\n",
    "print('Validation accuracy: {:.4f}'.format(acc_ens_val))\n",
    "\n",
    "acc_ens_test = accuracy_score(y_true=data_test['input']['label'], y_pred=np.argmax(preds_ens_test['y_proba'], axis=1))\n",
    "print('Test accuracy: {:.4f}'.format(acc_ens_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Caching: saving output within one run only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes you want to keep your output within one run of your pipeline but discard it at the end. This use case is handled by **caching**. Let's build a new pipeline that uses caching instead of saving to avoid re-computing results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_count_vec_step = Step(name='CountVec',\n",
    "                          transformer=CountVecTransformer(max_features=1000),\n",
    "                          input_data=['input'],\n",
    "                          adapter=Adapter({'X': E('input', 'text')}),\n",
    "                          experiment_directory=EXPERIMENT_DIR_B,\n",
    "                          is_fittable=True)\n",
    "\n",
    "new_tfidf_step = Step(name='TF-IDF',\n",
    "                      transformer=StepsTfidfTransformer(),\n",
    "                      input_steps=[new_count_vec_step],        \n",
    "                      experiment_directory=EXPERIMENT_DIR_B,\n",
    "                      cache_output=True,\n",
    "                      force_fitting=False,\n",
    "                      is_fittable=True)\n",
    "\n",
    "new_logreg_step = Step(name='SparseLogReg',\n",
    "                   transformer=SparseLogRegProbaTransformer(),\n",
    "                   input_steps=[new_tfidf_step],\n",
    "                   input_data=['input'],\n",
    "                   adapter=Adapter({'X': E('TF-IDF', 'X'),\n",
    "                                    'y': E('input', 'label')\n",
    "                                   }),\n",
    "                   experiment_directory=EXPERIMENT_DIR_B,\n",
    "                   is_fittable=True)\n",
    "\n",
    "new_rf_step = Step(name='RF',\n",
    "               transformer=RfClfTransformer(n_estimators=200, max_depth=8),\n",
    "               input_steps=[new_tfidf_step],\n",
    "               input_data=['input'],\n",
    "               adapter=Adapter({'X': E('TF-IDF', 'X'),\n",
    "                                'y': E('input', 'label')\n",
    "                               }),\n",
    "               experiment_directory=EXPERIMENT_DIR_B,\n",
    "               is_fittable=True)\n",
    "\n",
    "new_ens_step = Step(name='Ensembler',\n",
    "                transformer=AvgTransformer(),\n",
    "                input_steps=[new_logreg_step, new_rf_step],\n",
    "                adapter=Adapter({'y_proba_1': E('SparseLogReg', 'y_proba'),\n",
    "                                 'y_proba_2': E('RF', 'y_proba')\n",
    "                                }),\n",
    "                experiment_directory=EXPERIMENT_DIR_B,\n",
    "                is_fittable=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_ens_step"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_ens_step.clean_cache_upstream()\n",
    "new_preds_ens_fit = new_ens_step.fit_transform(data_fit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you look carefully at the training log above, you should see that when training the second branch, TF-IDF just loaded outputs instead of re-computing them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_ens_step.clean_cache_upstream()\n",
    "new_preds_ens_val = new_ens_step.transform(data_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_ens_step.clean_cache_upstream()\n",
    "new_preds_ens_test = new_ens_step.transform(data_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_acc_ens_fit = accuracy_score(y_true=data_fit['input']['label'], y_pred=np.argmax(new_preds_ens_fit['y_proba'], axis=1))\n",
    "print('New fitting accuracy: {:.4f}'.format(new_acc_ens_fit))\n",
    "\n",
    "new_acc_ens_val = accuracy_score(y_true=data_val['input']['label'], y_pred=np.argmax(new_preds_ens_val['y_proba'], axis=1))\n",
    "print('New validation accuracy: {:.4f}'.format(new_acc_ens_val))\n",
    "\n",
    "new_acc_ens_test = accuracy_score(y_true=data_test['input']['label'], y_pred=np.argmax(new_preds_ens_test['y_proba'], axis=1))\n",
    "print('New test accuracy: {:.4f}'.format(new_acc_ens_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you should be familiar with data persistence features. The next few notebooks will focus on building deep learning pipelines with steps."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}