dynamic-programming/Dynamic_Programming.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dynamic Programming\n",
    "\n",
    "In this notebook, you will write your own implementations of many classical dynamic programming algorithms.  \n",
    "\n",
    "While we have provided some starter code, you are welcome to erase these hints and write your code from scratch.\n",
    "\n",
    "---\n",
    "\n",
    "### Part 0: Explore FrozenLakeEnv\n",
    "\n",
    "We begin by importing the necessary packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import copy\n",
    "\n",
    "import check_test\n",
    "from frozenlake import FrozenLakeEnv\n",
    "from plot_utils import plot_values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the code cell below to create an instance of the [FrozenLake](https://github.com/openai/gym/blob/master/gym/envs/toy_text/frozen_lake.py) environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env = FrozenLakeEnv()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The agent moves through a $4 \\times 4$ gridworld, with states numbered as follows:\n",
    "```\n",
    "[[ 0  1  2  3]\n",
    " [ 4  5  6  7]\n",
    " [ 8  9 10 11]\n",
    " [12 13 14 15]]\n",
    "```\n",
    "and the agent has 4 potential actions:\n",
    "```\n",
    "LEFT = 0\n",
    "DOWN = 1\n",
    "RIGHT = 2\n",
    "UP = 3\n",
    "```\n",
    "\n",
    "Thus, $\\mathcal{S}^+ = \\{0, 1, \\ldots, 15\\}$, and $\\mathcal{A} = \\{0, 1, 2, 3\\}$.  Verify this by running the code cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the state space and action space\n",
    "print(env.observation_space)\n",
    "print(env.action_space)\n",
    "\n",
    "# print the total number of states and actions\n",
    "print(env.nS)\n",
    "print(env.nA)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dynamic programming assumes that the agent has full knowledge of the MDP.  We have already amended the `frozenlake.py` file to make the one-step dynamics accessible to the agent.  \n",
    "\n",
    "Execute the code cell below to return the one-step dynamics corresponding to a particular state and action.  In particular, `env.P[1][0]` returns the the probability of each possible reward and next state, if the agent is in state 1 of the gridworld and decides to go left."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.P[1][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each entry takes the form \n",
    "```\n",
    "prob, next_state, reward, done\n",
    "```\n",
    "where: \n",
    "- `prob` details the conditional probability of the corresponding (`next_state`, `reward`) pair, and\n",
    "- `done` is `True` if the `next_state` is a terminal state, and otherwise `False`.\n",
    "\n",
    "Thus, we can interpret `env.P[1][0]` as follows:\n",
    "$$\n",
    "\\mathbb{P}(S_{t+1}=s',R_{t+1}=r|S_t=1,A_t=0) = \\begin{cases}\n",
    "               \\frac{1}{3} \\text{ if } s'=1, r=0\\\\\n",
    "               \\frac{1}{3} \\text{ if } s'=0, r=0\\\\\n",
    "               \\frac{1}{3} \\text{ if } s'=5, r=0\\\\\n",
    "               0 \\text{ else}\n",
    "            \\end{cases}\n",
    "$$\n",
    "\n",
    "To understand the value of `env.P[1][0]`, note that when you create a FrozenLake environment, it takes as an (optional) argument `is_slippery`, which defaults to `True`.  \n",
    "\n",
    "To see this, change the first line in the notebook from `env = FrozenLakeEnv()` to `env = FrozenLakeEnv(is_slippery=False)`.  Then, when you check `env.P[1][0]`, it should look like what you expect (i.e., `env.P[1][0] = [(1.0, 0, 0.0, False)]`).\n",
    "\n",
    "The default value for the `is_slippery` argument is `True`, and so `env = FrozenLakeEnv()` is equivalent to `env = FrozenLakeEnv(is_slippery=True)`.  In the event that `is_slippery=True`, you see that this can result in the agent moving in a direction that it did not intend (where the idea is that the ground is *slippery*, and so the agent can slide to a location other than the one it wanted).\n",
    "\n",
    "Feel free to change the code cell above to explore how the environment behaves in response to other (state, action) pairs.  \n",
    "\n",
    "Before proceeding to the next part, make sure that you set `is_slippery=True`, so that your implementations below will work with the slippery environment!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 1: Iterative Policy Evaluation\n",
    "\n",
    "In this section, you will write your own implementation of iterative policy evaluation.\n",
    "\n",
    "Your algorithm should accept four arguments as **input**:\n",
    "- `env`: This is an instance of an OpenAI Gym environment, where `env.P` returns the one-step dynamics.\n",
    "- `policy`: This is a 2D numpy array with `policy.shape[0]` equal to the number of states (`env.nS`), and `policy.shape[1]` equal to the number of actions (`env.nA`).  `policy[s][a]` returns the probability that the agent takes action `a` while in state `s` under the policy.\n",
    "- `gamma`: This is the discount rate.  It must be a value between 0 and 1, inclusive (default value: `1`).\n",
    "- `theta`: This is a very small positive number that is used to decide if the estimate has sufficiently converged to the true value function (default value: `1e-8`).\n",
    "\n",
    "The algorithm returns as **output**:\n",
    "- `V`: This is a 1D numpy array with `V.shape[0]` equal to the number of states (`env.nS`).  `V[s]` contains the estimated value of state `s` under the input policy.\n",
    "\n",
    "Please complete the function in the code cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def policy_evaluation(env, policy, gamma=1, theta=1e-8):\n",
    "    V = np.zeros(env.nS)\n",
    "    \n",
    "    ## TODO: complete the function\n",
    "    \n",
    "    return V"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will evaluate the equiprobable random policy $\\pi$, where $\\pi(a|s) = \\frac{1}{|\\mathcal{A}(s)|}$ for all $s\\in\\mathcal{S}$ and $a\\in\\mathcal{A}(s)$.  \n",
    "\n",
    "Use the code cell below to specify this policy in the variable `random_policy`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_policy = np.ones([env.nS, env.nA]) / env.nA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the next code cell to evaluate the equiprobable random policy and visualize the output.  The state-value function has been reshaped to match the shape of the gridworld."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# evaluate the policy \n",
    "V = policy_evaluation(env, random_policy)\n",
    "\n",
    "plot_values(V)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code cell below to test your function.  If the code cell returns **PASSED**, then you have implemented the function correctly!  \n",
    "\n",
    "**Note:** In order to ensure accurate results, make sure that your `policy_evaluation` function satisfies the requirements outlined above (with four inputs, a single output, and with the default values of the input arguments unchanged)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_test.run_check('policy_evaluation_check', policy_evaluation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 2: Obtain $q_\\pi$ from $v_\\pi$\n",
    "\n",
    "In this section, you will write a function that takes the state-value function estimate as input, along with some state $s\\in\\mathcal{S}$.  It returns the **row in the action-value function** corresponding to the input state $s\\in\\mathcal{S}$.  That is, your function should accept as input both $v_\\pi$ and $s$, and return $q_\\pi(s,a)$ for all $a\\in\\mathcal{A}(s)$.\n",
    "\n",
    "Your algorithm should accept four arguments as **input**:\n",
    "- `env`: This is an instance of an OpenAI Gym environment, where `env.P` returns the one-step dynamics.\n",
    "- `V`: This is a 1D numpy array with `V.shape[0]` equal to the number of states (`env.nS`).  `V[s]` contains the estimated value of state `s`.\n",
    "- `s`: This is an integer corresponding to a state in the environment.  It should be a value between `0` and `(env.nS)-1`, inclusive.\n",
    "- `gamma`: This is the discount rate.  It must be a value between 0 and 1, inclusive (default value: `1`).\n",
    "\n",
    "The algorithm returns as **output**:\n",
    "- `q`: This is a 1D numpy array with `q.shape[0]` equal to the number of actions (`env.nA`).  `q[a]` contains the (estimated) value of state `s` and action `a`.\n",
    "\n",
    "Please complete the function in the code cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def q_from_v(env, V, s, gamma=1):\n",
    "    q = np.zeros(env.nA)\n",
    "    \n",
    "    ## TODO: complete the function\n",
    "    \n",
    "    return q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code cell below to print the action-value function corresponding to the above state-value function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Q = np.zeros([env.nS, env.nA])\n",
    "for s in range(env.nS):\n",
    "    Q[s] = q_from_v(env, V, s)\n",
    "print(\"Action-Value Function:\")\n",
    "print(Q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code cell below to test your function.  If the code cell returns **PASSED**, then you have implemented the function correctly!  \n",
    "\n",
    "**Note:** In order to ensure accurate results, make sure that the `q_from_v` function satisfies the requirements outlined above (with four inputs, a single output, and with the default values of the input arguments unchanged)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_test.run_check('q_from_v_check', q_from_v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 3: Policy Improvement\n",
    "\n",
    "In this section, you will write your own implementation of policy improvement. \n",
    "\n",
    "Your algorithm should accept three arguments as **input**:\n",
    "- `env`: This is an instance of an OpenAI Gym environment, where `env.P` returns the one-step dynamics.\n",
    "- `V`: This is a 1D numpy array with `V.shape[0]` equal to the number of states (`env.nS`).  `V[s]` contains the estimated value of state `s`.\n",
    "- `gamma`: This is the discount rate.  It must be a value between 0 and 1, inclusive (default value: `1`).\n",
    "\n",
    "The algorithm returns as **output**:\n",
    "- `policy`: This is a 2D numpy array with `policy.shape[0]` equal to the number of states (`env.nS`), and `policy.shape[1]` equal to the number of actions (`env.nA`).  `policy[s][a]` returns the probability that the agent takes action `a` while in state `s` under the policy.\n",
    "\n",
    "Please complete the function in the code cell below.  You are encouraged to use the `q_from_v` function you implemented above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def policy_improvement(env, V, gamma=1):\n",
    "    policy = np.zeros([env.nS, env.nA]) / env.nA\n",
    "    \n",
    "    ## TODO: complete the function\n",
    "\n",
    "    return policy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code cell below to test your function.  If the code cell returns **PASSED**, then you have implemented the function correctly!  \n",
    "\n",
    "**Note:** In order to ensure accurate results, make sure that the `policy_improvement` function satisfies the requirements outlined above (with three inputs, a single output, and with the default values of the input arguments unchanged).\n",
    "\n",
    "Before moving on to the next part of the notebook, you are strongly encouraged to check out the solution in **Dynamic_Programming_Solution.ipynb**.  There are many correct ways to approach this function!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_test.run_check('policy_improvement_check', policy_improvement)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 4: Policy Iteration\n",
    "\n",
    "In this section, you will write your own implementation of policy iteration.  The algorithm returns the optimal policy, along with its corresponding state-value function.\n",
    "\n",
    "Your algorithm should accept three arguments as **input**:\n",
    "- `env`: This is an instance of an OpenAI Gym environment, where `env.P` returns the one-step dynamics.\n",
    "- `gamma`: This is the discount rate.  It must be a value between 0 and 1, inclusive (default value: `1`).\n",
    "- `theta`: This is a very small positive number that is used to decide if the policy evaluation step has sufficiently converged to the true value function (default value: `1e-8`).\n",
    "\n",
    "The algorithm returns as **output**:\n",
    "- `policy`: This is a 2D numpy array with `policy.shape[0]` equal to the number of states (`env.nS`), and `policy.shape[1]` equal to the number of actions (`env.nA`).  `policy[s][a]` returns the probability that the agent takes action `a` while in state `s` under the policy.\n",
    "- `V`: This is a 1D numpy array with `V.shape[0]` equal to the number of states (`env.nS`).  `V[s]` contains the estimated value of state `s`.\n",
    "\n",
    "Please complete the function in the code cell below.  You are strongly encouraged to use the `policy_evaluation` and `policy_improvement` functions you implemented above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def policy_iteration(env, gamma=1, theta=1e-8):\n",
    "    policy = np.ones([env.nS, env.nA]) / env.nA\n",
    "    \n",
    "    ## TODO: complete the function\n",
    "\n",
    "    return policy, V"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the next code cell to solve the MDP and visualize the output.  The optimal state-value function has been reshaped to match the shape of the gridworld.\n",
    "\n",
    "**Compare the optimal state-value function to the state-value function from Part 1 of this notebook**.  _Is the optimal state-value function consistently greater than or equal to the state-value function for the equiprobable random policy?_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# obtain the optimal policy and optimal state-value function\n",
    "policy_pi, V_pi = policy_iteration(env)\n",
    "\n",
    "# print the optimal policy\n",
    "print(\"\\nOptimal Policy (LEFT = 0, DOWN = 1, RIGHT = 2, UP = 3):\")\n",
    "print(policy_pi,\"\\n\")\n",
    "\n",
    "plot_values(V_pi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code cell below to test your function.  If the code cell returns **PASSED**, then you have implemented the function correctly!  \n",
    "\n",
    "**Note:** In order to ensure accurate results, make sure that the `policy_iteration` function satisfies the requirements outlined above (with three inputs, two outputs, and with the default values of the input arguments unchanged)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_test.run_check('policy_iteration_check', policy_iteration)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 5: Truncated Policy Iteration\n",
    "\n",
    "In this section, you will write your own implementation of truncated policy iteration.  \n",
    "\n",
    "You will begin by implementing truncated policy evaluation.  Your algorithm should accept five arguments as **input**:\n",
    "- `env`: This is an instance of an OpenAI Gym environment, where `env.P` returns the one-step dynamics.\n",
    "- `policy`: This is a 2D numpy array with `policy.shape[0]` equal to the number of states (`env.nS`), and `policy.shape[1]` equal to the number of actions (`env.nA`).  `policy[s][a]` returns the probability that the agent takes action `a` while in state `s` under the policy.\n",
    "- `V`: This is a 1D numpy array with `V.shape[0]` equal to the number of states (`env.nS`).  `V[s]` contains the estimated value of state `s`.\n",
    "- `max_it`: This is a positive integer that corresponds to the number of sweeps through the state space (default value: `1`).\n",
    "- `gamma`: This is the discount rate.  It must be a value between 0 and 1, inclusive (default value: `1`).\n",
    "\n",
    "The algorithm returns as **output**:\n",
    "- `V`: This is a 1D numpy array with `V.shape[0]` equal to the number of states (`env.nS`).  `V[s]` contains the estimated value of state `s`.\n",
    "\n",
    "Please complete the function in the code cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def truncated_policy_evaluation(env, policy, V, max_it=1, gamma=1):\n",
    "    \n",
    "    ## TODO: complete the function\n",
    "    \n",
    "    return V"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, you will implement truncated policy iteration.  Your algorithm should accept five arguments as **input**:\n",
    "- `env`: This is an instance of an OpenAI Gym environment, where `env.P` returns the one-step dynamics.\n",
    "- `max_it`: This is a positive integer that corresponds to the number of sweeps through the state space (default value: `1`).\n",
    "- `gamma`: This is the discount rate.  It must be a value between 0 and 1, inclusive (default value: `1`).\n",
    "- `theta`: This is a very small positive number that is used for the stopping criterion (default value: `1e-8`).\n",
    "\n",
    "The algorithm returns as **output**:\n",
    "- `policy`: This is a 2D numpy array with `policy.shape[0]` equal to the number of states (`env.nS`), and `policy.shape[1]` equal to the number of actions (`env.nA`).  `policy[s][a]` returns the probability that the agent takes action `a` while in state `s` under the policy.\n",
    "- `V`: This is a 1D numpy array with `V.shape[0]` equal to the number of states (`env.nS`).  `V[s]` contains the estimated value of state `s`.\n",
    "\n",
    "Please complete the function in the code cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def truncated_policy_iteration(env, max_it=1, gamma=1, theta=1e-8):\n",
    "    V = np.zeros(env.nS)\n",
    "    policy = np.zeros([env.nS, env.nA]) / env.nA\n",
    "    \n",
    "    ## TODO: complete the function\n",
    "    \n",
    "    return policy, V"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the next code cell to solve the MDP and visualize the output.  The state-value function has been reshaped to match the shape of the gridworld.\n",
    "\n",
    "Play with the value of the `max_it` argument.  Do you always end with the optimal state-value function?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "policy_tpi, V_tpi = truncated_policy_iteration(env, max_it=2)\n",
    "\n",
    "# print the optimal policy\n",
    "print(\"\\nOptimal Policy (LEFT = 0, DOWN = 1, RIGHT = 2, UP = 3):\")\n",
    "print(policy_tpi,\"\\n\")\n",
    "\n",
    "# plot the optimal state-value function\n",
    "plot_values(V_tpi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code cell below to test your function.  If the code cell returns **PASSED**, then you have implemented the function correctly!  \n",
    "\n",
    "**Note:** In order to ensure accurate results, make sure that the `truncated_policy_iteration` function satisfies the requirements outlined above (with four inputs, two outputs, and with the default values of the input arguments unchanged)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_test.run_check('truncated_policy_iteration_check', truncated_policy_iteration)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 6: Value Iteration\n",
    "\n",
    "In this section, you will write your own implementation of value iteration.\n",
    "\n",
    "Your algorithm should accept three arguments as input:\n",
    "- `env`: This is an instance of an OpenAI Gym environment, where `env.P` returns the one-step dynamics.\n",
    "- `gamma`: This is the discount rate.  It must be a value between 0 and 1, inclusive (default value: `1`).\n",
    "- `theta`: This is a very small positive number that is used for the stopping criterion (default value: `1e-8`).\n",
    "\n",
    "The algorithm returns as **output**:\n",
    "- `policy`: This is a 2D numpy array with `policy.shape[0]` equal to the number of states (`env.nS`), and `policy.shape[1]` equal to the number of actions (`env.nA`).  `policy[s][a]` returns the probability that the agent takes action `a` while in state `s` under the policy.\n",
    "- `V`: This is a 1D numpy array with `V.shape[0]` equal to the number of states (`env.nS`).  `V[s]` contains the estimated value of state `s`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def value_iteration(env, gamma=1, theta=1e-8):\n",
    "    V = np.zeros(env.nS)\n",
    "    \n",
    "    ## TODO: complete the function\n",
    "    \n",
    "    return policy, V"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the next code cell to solve the MDP and visualize the output.  The state-value function has been reshaped to match the shape of the gridworld."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "policy_vi, V_vi = value_iteration(env)\n",
    "\n",
    "# print the optimal policy\n",
    "print(\"\\nOptimal Policy (LEFT = 0, DOWN = 1, RIGHT = 2, UP = 3):\")\n",
    "print(policy_vi,\"\\n\")\n",
    "\n",
    "# plot the optimal state-value function\n",
    "plot_values(V_vi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code cell below to test your function.  If the code cell returns **PASSED**, then you have implemented the function correctly!  \n",
    "\n",
    "**Note:** In order to ensure accurate results, make sure that the `value_iteration` function satisfies the requirements outlined above (with three inputs, two outputs, and with the default values of the input arguments unchanged)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_test.run_check('value_iteration_check', value_iteration)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}