getting_started/06_Redispatching_Curtailment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# In this notebook you will learn about the redispatching and curtailment capabilities offered by grid2op.\n",
    "Try me out interactively with: [![Binder](./img/badge_logo.svg)](https://mybinder.org/v2/gh/rte-france/Grid2Op/master)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Objectives**\n",
    "\n",
    "In the previous notebooks, we presented actions in a discrete action space. However, more actions are available in Grid2Op. Redispatching and curtailment are kind of continuous action that will be described here.\n",
    "\n",
    "This notebook will:\n",
    "\n",
    "- [I) Redispatching](#redispatching) details redispatching\n",
    "    - present what redispatching is\n",
    "    - show how it can be used in grid2op\n",
    "    - detail the redispatching actions\n",
    "    - show an example of a redispatching Agent\n",
    "- [II) Curtailment](#curtailment) explains the curtailment\n",
    "    - show the behaviour of the curtailment: how to perform it, what is does etc.\n",
    "    - expose some limits of this curtailment: it can break the grid easily !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://colab.research.google.com/assets/colab-badge.svg\" width=\"200\">\n",
    "Execute the cell below by removing the # character if you use google colab !\n",
    "\n",
    "Cell will look like:\n",
    "```python\n",
    "!pip install grid2op[optional]  # for use with google colab (grid2Op is not installed by default)\n",
    "```\n",
    "<img src=\"https://colab.research.google.com/assets/colab-badge.svg\" width=\"200\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install grid2op[optional]  # for use with google colab (grid2Op is not installed by default)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "import grid2op\n",
    "from grid2op.Agent import DoNothingAgent, BaseAgent\n",
    "from tqdm.notebook import tqdm  # for easy progress bar\n",
    "display_tqdm = False  # this is set to False for ease with the unitt test, feel free to set it to True\n",
    "import numpy as np\n",
    "max_iter = 100  # to make computation much faster we will only consider 100 time steps\n",
    "import pdb\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "res = None\n",
    "try:\n",
    "    from jyquickhelper import add_notebook_menu\n",
    "    res = add_notebook_menu()\n",
    "except ModuleNotFoundError:\n",
    "    print(\"Impossible to automatically add a menu / table of content to this notebook.\\nYou can download \\\"jyquickhelper\\\" package with: \\n\\\"pip install jyquickhelper\\\"\")\n",
    "res"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# I) Redispatching  <a class=\"anchor\" id=\"redispatching\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to implement redispatch actions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Having a suitable environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, some environments do not specify the cost of generators, their maximum and minimum production values, etc. In this case it is not possible to redispatch in grid2op.\n",
    "\n",
    "To know more about what is needed for using redispatching, it is advised to look at this help online : https://grid2op.readthedocs.io/en/latest/space.html#grid2op.Space.GridObjects.redispatching_unit_commitment_availble for the most recent documentation. When this notebook was created, the following were needed:\n",
    "\n",
    "- \"gen_type\": the type of generator\n",
    "- \"gen_pmin\": the minimum value a generator can produce\n",
    "- \"gen_pmax\" : the maximum value a generator can produce\n",
    "- \"gen_redispatchable\": whether this generator can be dispatched\n",
    "- \"gen_max_ramp_up\": the maximum increase of power a generator can have between two consecutive time steps\n",
    "- \"gen_max_ramp_down\": the maximum decrease of power a generator can have between two consecutive time steps\n",
    "- \"gen_min_uptime\": the minimum time a generator needs to be turned on (it's impossible to disconnect it if it's not connected for a least \"gen_min_uptime\" consecutive time step)\n",
    "- \"gen_min_downtime\": same as above, but for downtime\n",
    "- \"gen_cost_per_MW\": the generation cost. For each MW of electricity produced in a time step, this is the amount paid\n",
    "- \"gen_startup_cost\": the cost to start a generator\n",
    "- \"gen_shutdown_cost\": the cost to shutdown a generator\n",
    "\n",
    "We made available a dedicated environment, based on the IEEE case14 powergrid that has all this features. It is advised to use this small environment for testing and getting familiar with redispatching.\n",
    "\n",
    "This environment includes 5 generators, as the original case14 system. It has one solar generator and one wind generator (that cannot be dispatched), one nuclear powerplant (dispatchable) and 2 thermal generators (also dispatchable). Thus, redispatching is a problem of continuous control with 3 degress of freedom here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from lightsim2grid import LightSimBackend\n",
    "    bk_cls = LightSimBackend\n",
    "except ImportError as exc:\n",
    "    print(f\"Error: {exc} when importing faster LightSimBackend\")\n",
    "    from grid2op.Backend import PandaPowerBackend\n",
    "    bk_cls = PandaPowerBackend\n",
    "\n",
    "env_name = \"l2rpn_case14_sandbox\"\n",
    "env = grid2op.make(env_name, test=True, backend=bk_cls())\n",
    "\n",
    "print(\"Is this environment suitable for redispatching: {}\".format(env.redispatching_unit_commitment_availble))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, this environment is suitable for redispatching. It means that all quantities described above are set (and visible).\n",
    "\n",
    "In the L2RPN 2019 challenges, we rewarded participants based on the use of the powerlines. In next challenges, or for other uses of this platform where redispatching should be considered, it's better to consider the economic cost of the sytem as an evaluation metric. However, usually a cost should be minimized, while a reward should be maximized. To take this into account, a simple reward named \"EconomicReward\" has been created. It has the following properties:\n",
    "\n",
    "- it returns -1 if there has been a game over\n",
    "- otherwise (there has been no game over, no errors, etc) it is always strictly positive\n",
    "- maximizing this reward is equivalent to minimizing the cost\n",
    "\n",
    "Note that this reward doesn't take into account the cost to perform a redispatching action. This reward can be used to build what is called an \"economic dispatch\", a problem that is particularly interesting for electricity producers but of lower interest for Transmission System Operators (as opposed to the topology).\n",
    "\n",
    "Compared to standard \"economic dispatch\" problems, for now storages are not implemented (coming soon) and we don't fully take into account the startup cost, shutdown cost, as well as minimum downtime and minimum uptime (even though all of these features are implemented). Also, note that redispatching is implemented in differences, meaning that you first need to provide an economic dispatch, and then think in terms of variations from that point onward. This is the usage that will be explained in this notebook. For real unit commitment / economic dispatch problems, the key words \"injections\" / \"prod_p\" for the action would probably be more suited."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Redispatching implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike topological actions which are always feasible (this is an assumption that is made in this package), redispatching actions are limited by physical constraints on the generators. For example:\n",
    "\n",
    "- it is not possible for a generator to produce more (resp. less) than pmax (resp. pmin). Unlike the curent flow on the powerline, this is a strict physical constraint.\n",
    "- it is not possible for the same physical limitations to increase (or decrease) too much the value between two consecutive time steps.\n",
    "- redispatching actions stack with one another. It means that if you ask to increase the production of the generator 1 by $10MW$ at time step t, and by $20MW$ at time step $t+1$, it means that the set point at time step $t+1$ will be $+30MW$ higher than if no redispatching had been made (it would have been only $+10MW$ higher if the second redispatching action had not been performed).\n",
    "\n",
    "That being said, much can happen that makes redispatching a bit less trivial than topology changes:\n",
    "- When you do a redispatching action, you don't know what the time series of the environment looks like. For example, say the pmax of generator 1 is 100. The setpoint of this generator at time t is $60MW$, and you want to increase its value by another $40MW$. This action is legal: $60+40 \\leq pmax (=100) $. So at time $t$ everything is fine. Now let's suppose that the environment also moved the setpoint of this same generator from $60$ to be at $70MW$ at time $t+1$. With the redispatching action, this would mean that the setpoint would be $70+40 = 110 > pmax$. This is not possible. In this case, the redispatching action will be capped : instead of the desired redispatching of $+40$, only a smaller redispatching of $+30$ will be implemented on the powergrid. \n",
    "- Another problem can arise with the fundamental principle of power grid: power energy cannot be stored in a grid. At each time we then have $\\sum \\text{Prod} = \\sum \\text{Load} + \\text{Losses}$. In this competition, the data is generated such that this condition holds (approximately) for all time steps. This means that if you ask for a redispatching of +xxx MW on a given generator, then the other ones must compensate and \"absorb\" xxx MW such that it sums to 0 overall.\n",
    "\n",
    "Out of simplicity for the participants, there are some built-in control mechanisms and functions (\"automatons\") that automatically transform an proposed redispatching action into a valid redispatching action. \n",
    "\n",
    "These automatons basically ensure that the two above-mentionned conditions hold. It explains the differences between \"target_redispatching\" which is the desired setpoint enter by the agent, and the \"actual_redispatching\" which is the one that has actually been implemented on the powergrid after it has been corrected by the automatons."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More cases for ambiguous actions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.gen_redispatchable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above vector tells which generator is dispatchable and which is not. Any attempt to dispatch a generator that is not dispatchable leads to an ambiguous action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "act = env.action_space({\"redispatch\": [(2, +10)]})\n",
    "act.is_ambiguous()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we see, this action is ambiguous because we are `Trying to apply a redispatching action on a non redispatchable generator`. Indeed, as shown above, the generator 2 is not dispatchable.\n",
    "\n",
    "Generators also have physical constraints. You cannot ask them to change the active production value too fast, this would damage the generator, and breaking a nuclear plant is often a terrible idea. In grid2op it is implemented as an ambiguous action. Trying to go beyond this value will result in an ambiguous action.\n",
    "\n",
    "This value is called the \"ramp\" and it's available through the \"max_ramp_up\" attribute. On the next cell, you can see that the ramp is $5MW$ for the first generator and $10MW$ for the second one and the last one. For the other 2 it's irrelevant because they are not dispatchable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.gen_max_ramp_up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Any attempt to go beyond this value will raise an ambiguous error :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "act = env.action_space({\"redispatch\": [(0, +10)]})\n",
    "act.is_ambiguous()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous action, we asked the generator 0 to increase its setpoint by $10MW$. However, its maximum ramp up is only $5MW$. Thus, this action is ambiguous.\n",
    "\n",
    "Of course, there are some perfectly valid redispatching actions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "act = env.action_space({\"redispatch\": [(1, +10)]})\n",
    "act.is_ambiguous()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The (desired) setpoint is not the actual setpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As said in the preamble of this section, the target dispatching, what we want to achieve (the target), is not equal to the implemented dispatching. So that we can see the operations that are actually being performed, both of these values are present in the observation, as shown in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "observed = []\n",
    "# perform a valid redispatching action\n",
    "env.set_id(0)  # make sure to use the same environment input data.\n",
    "obs_init = env.reset()  # reset the environment\n",
    "act = env.action_space()\n",
    "act = env.action_space({\"redispatch\": [(0, -5)]})\n",
    "obs, reward, done, info = env.step(act)\n",
    "print(\"actual dispatch at time step 0: {}\".format(obs.actual_dispatch))\n",
    "observed.append(obs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The target dispatch is exactly what we wanted, e.g. the generator 0 has decreased its production by $5MW$. To compensate for this increase, both generator 1 has increase its production of 2MW and generator 5 of 3 MW.\n",
    "\n",
    "Let's plot the generators' base productions in this environment data (*ie* without any intervention from the agent), and what is implemented with the current redispatching :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a matplot figure\n",
    "redisp_fig = plt.figure()\n",
    "ax = redisp_fig.add_axes([0,0,1,1])\n",
    "# X axis is the generators\n",
    "x_gens = np.arange(obs.n_gen)\n",
    "# Y axis is the production in MW\n",
    "y_scenario_p = obs.gen_p - obs.actual_dispatch\n",
    "y_redisp_p = obs.gen_p\n",
    "# Blue bars for scenario productions\n",
    "ax.bar(x_gens - 0.2, y_scenario_p, width = 0.4)\n",
    "# Red bars for production with redispatch\n",
    "ax.bar(x_gens + 0.2, y_redisp_p, width = 0.4)\n",
    "# Set some legend to describe what's above\n",
    "ax.set_ylabel('MW')\n",
    "ax.set_xlabel('Generator ID')\n",
    "_ = ax.legend(labels=['Generation in scenario', 'Generation after redispatched'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following cell, we won't be performing any redispatching action. We will simply do nothing. This example is here to illustrate that grid2op will continue to apply the previous redispatching configuration over time, until the original redispatching action is removed (i.e. until the opposite command is sent)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "donothing = env.action_space()\n",
    "obs1, reward, done, info = env.step(donothing)\n",
    "print(\"actual dispatch at time step 1: {}\".format(obs1.actual_dispatch))\n",
    "observed.append(obs1)\n",
    "obs2, reward, done, info = env.step(donothing)\n",
    "print(\"actual dispatch at time step 2: {}\".format(obs2.actual_dispatch))\n",
    "observed.append(obs2)\n",
    "obs3, reward, done, info = env.step(donothing)\n",
    "print(\"actual dispatch at time step 2: {}\".format(obs2.actual_dispatch))\n",
    "observed.append(obs3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, the original redispatching action was to increase the setpoint of generator 0 by $5MW$. If we want to remove this, we need to decrease its setpoint by $5MW$ :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "act = env.action_space({\"redispatch\": [(0, 5)]})\n",
    "obs3, reward, done, info = env.step(act)\n",
    "print(\"actual dispatch at time step 3: {}\".format(obs3.actual_dispatch))\n",
    "observed.append(obs3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we see in the cell above, there are still residuals on the dispatch. \n",
    "\n",
    "This is because of the physical limit of the ramp of the generator 0. \n",
    "\n",
    "We wanted it to return to its original setpoint value, but at the same time step, the \"environment\" (representing a market for example) also modified the setpoint of this generator by +0.3MW. \n",
    "\n",
    "Therefore, the total desired increase for the setpoint of generator 0 was $5+0.4 = 5.4 > max\\_ramp\\_down$. The action (that would increase the production of 5.4 MW) could not completely performed. Grid2op \"capped\" the redispatch occuring at this timestep to $5MW$ (max\\_ramp\\_down) leaving a remaining of -0.4MW (instead of the initial -5.)\n",
    "\n",
    "That is why we can see a small part of the dispatch left. If we wait for another timestep and do nothing, the generator will likely be in order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "obs4, reward, done, info = env.step(donothing)\n",
    "print(\"actual dispatch at time step 4: {}\".format(obs4.actual_dispatch))\n",
    "observed.append(obs4)\n",
    "obs5, reward, done, info = env.step(donothing)\n",
    "print(\"actual dispatch at time step 5: {}\".format(obs5.actual_dispatch))\n",
    "observed.append(obs5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now everything is set up as it should be. The system is back to its original state. Let's see what happens if we ask to decrease again the value of this generator 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "act = env.action_space({\"redispatch\": [(0, -5.)]})\n",
    "obs6, reward, done, info = env.step(act)\n",
    "print(\"actual dispatch at time step 6: {}\".format(obs6.actual_dispatch))\n",
    "observed.append(obs6)\n",
    "obs7, reward, done, info = env.step(donothing)\n",
    "print(\"actual dispatch at time step 7: {}\".format(obs7.actual_dispatch))\n",
    "observed.append(obs7)\n",
    "obs8, reward, done, info = env.step(donothing)\n",
    "print(\"actual dispatch at time step 8: {}\".format(obs8.actual_dispatch))\n",
    "observed.append(obs8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time we see that the full (valid) redispatching action is not applied completely. \n",
    "\n",
    "This is due to the same phenomenon that occurred previously. The environment (*eg* a market or a central authority) decreased the value of this generator and at the same time, we (=an agent operating the grid in real time) also asked to decrease it by its \"max ramp up\" value. \n",
    "\n",
    "Consequently, our action was \"capped\" and only  4.4MW (out of 5MW) were indeed removed to the generator's production. \n",
    "\n",
    "At the next time step (step 7), the action is fully implemented.\n",
    "\n",
    "To conclude on redispatching, we saw that there is a difference between the value we ask for, and the value that is implemented by the environment. This is mainly because:\n",
    "\n",
    "- the implemented vector should sum to 0.\n",
    "- if a redispatching is close to the maximum value it can take (due to ramping limitation or hard limitations) and if at the same time the environment itself \"wants\" to increase this value, physical limitations have to be respected.\n",
    "\n",
    "Redispatching actions also last in time. One action must be explicitely canceled to be reset to 0. This cancellation can take a few time steps to be fully effective as of the limitations mentionned above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To summarize, we can look at the productions over time for generators 0 & 5:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a matplot figure\n",
    "redisp_fig = plt.figure(figsize=(16, 9))\n",
    "# X axis is the timesteps\n",
    "x_gens = np.arange(len(observed), dtype=np.int32)\n",
    "# Y axis is the production in MW\n",
    "y_scenario_p_0 = [o.gen_p[0] - o.actual_dispatch[0] for o in observed] \n",
    "y_redisp_p_0 = [o.gen_p[0] for o in observed]\n",
    "y_scenario_p_4 = [o.gen_p[5] - o.actual_dispatch[5] for o in observed] \n",
    "y_redisp_p_4 = [o.gen_p[5] for o in observed]\n",
    "# Blue lines for gen 0\n",
    "plt.plot(x_gens, y_scenario_p_0, 'b', x_gens, y_redisp_p_0, 'b--')\n",
    "# Red lines for gen 4\n",
    "plt.plot(x_gens, y_scenario_p_4, 'r', x_gens, y_redisp_p_4, 'r--')\n",
    "# Set some legend to describe what's above\n",
    "plt.ylabel('MW')\n",
    "plt.xlabel('Timesteps')\n",
    "_ = plt.legend(labels=['Scenario gen 0', 'Redispatched gen 0', 'Scenario gen 5', 'Redispatched gen 5'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example of use: economic dispatch problem\n",
    "\n",
    "**NB** This is an illustration of an economic dispatch problem that can use grid2op. The problems modeled in the L2RPN competitions are NOT economic dispatch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = DoNothingAgent(env.action_space)\n",
    "done = False\n",
    "reward = env.reward_range[0]\n",
    "\n",
    "env.set_id(0)  # make sure to evaluate the models on the same experiments\n",
    "obs = env.reset()\n",
    "cum_reward = 0\n",
    "nrow = env.chronics_handler.max_timestep() if max_iter <= 0 else max_iter\n",
    "gen_p = np.zeros((nrow, env.n_gen))\n",
    "gen_p_setpoint = np.zeros((nrow, env.n_gen))\n",
    "load_p = np.zeros((nrow, env.n_load))\n",
    "rho = np.zeros((nrow, env.n_line))\n",
    "i = 0\n",
    "with tqdm(total=max_iter, desc=\"step\", disable=not display_tqdm) as pbar:\n",
    "    while not done:\n",
    "        act = agent.act(obs, reward, done)\n",
    "        obs, reward, done, info = env.step(act)\n",
    "        data_generator = env.chronics_handler.real_data.data\n",
    "        gen_p_setpoint[i,:] = data_generator.prod_p[data_generator.current_index, :]\n",
    "        gen_p[i,:] = obs.gen_p\n",
    "        load_p[i,:] = obs.load_p\n",
    "        rho[i,:] = obs.rho\n",
    "        cum_reward += reward\n",
    "        i += 1\n",
    "        pbar.update(1)\n",
    "        if i >= max_iter:\n",
    "            break\n",
    "print(\"The cumulative reward with this agent is {:.0f}\".format(cum_reward))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try to redispatch as much production as possible to the cheapest generator, and leave the rest unchanged."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class GreedyEconomic(BaseAgent):\n",
    "    def __init__(self, action_space):\n",
    "        super().__init__(action_space)\n",
    "        self.do_nothing = action_space()\n",
    "        \n",
    "    def act(self, obs, reward, done):\n",
    "        act = self.do_nothing\n",
    "        if obs.gen_p[0] < obs.gen_pmax[0] - 1 and \\\n",
    "        obs.target_dispatch[0] < (obs.gen_pmax[0] - obs.gen_max_ramp_up[0]) - 1 and\\\n",
    "        obs.gen_p[0] > 0.:\n",
    "            # if the cheapest generator is significantly bellow its maximum cost\n",
    "            if obs.target_dispatch[0] < obs.gen_pmax[0]:\n",
    "                #in theory i can still ask for more\n",
    "                act = env.action_space({\"redispatch\": [(0, obs.gen_max_ramp_up[0])]})\n",
    "        return act\n",
    "    \n",
    "agent = GreedyEconomic(env.action_space)\n",
    "done = False\n",
    "reward = env.reward_range[0]\n",
    "\n",
    "env.set_id(0) # reset the env to the same id\n",
    "obs = env.reset()\n",
    "cum_reward = 0\n",
    "nrow = env.chronics_handler.max_timestep() if max_iter <= 0 else max_iter\n",
    "gen_p = np.zeros((nrow, env.n_gen))\n",
    "gen_p_setpoint = np.zeros((nrow, env.n_gen))\n",
    "load_p = np.zeros((nrow, env.n_load))\n",
    "rho = np.zeros((nrow, env.n_line))\n",
    "i = 0\n",
    "with tqdm(total=max_iter, desc=\"step\", disable=not display_tqdm) as pbar:\n",
    "    while not done:\n",
    "        act = agent.act(obs, reward, done)\n",
    "        obs, reward, done, info = env.step(act)\n",
    "        data_generator = env.chronics_handler.real_data.data\n",
    "        gen_p_setpoint[i,:] = data_generator.prod_p[data_generator.current_index, :]\n",
    "        gen_p[i,:] = obs.gen_p\n",
    "        load_p[i,:] = obs.load_p\n",
    "        rho[i,:] = obs.rho\n",
    "        cum_reward += reward\n",
    "        i += 1\n",
    "        pbar.update(1)\n",
    "        if i >= max_iter:\n",
    "            break\n",
    "            \n",
    "print(\"The cumulative reward with this agent is {:.0f}\".format(cum_reward))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, that second agent performed worse than the `DoNothing` agent, demonstrating that having the cheapest generators produce the most energy is not always the best solution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# II Curtailment  <a class=\"anchor\" id=\"curtailment\"></a>\n",
    "\n",
    "Quite like redispatching, curtailment is a type of action that can act on generators. As opposed to redispatching it acts on generators coming from renewable energy sources.\n",
    "\n",
    "The idea behind the curtailment is to limit the production of some generators to a certain upper bound. In real world application, such actions can be performed in case of emergency: when a powerline evacuating a wind farm is on overflow, it can be necassary to reduce the production the this wind farm even though we could have produced more.\n",
    "\n",
    "**NB** As for everything related to grid2op, we adopt the TSO point of view and model real time. This implies that the offer / demand equilibrium has been already computed. If the agent can not act, then the offer matches the demand already. Curtailment here is to understand as a real time security measure.\n",
    "\n",
    "If you want a detailed description of the modeling of the curtailment you can visit the official documentation [here](https://grid2op.readthedocs.io/en/latest/modeled_elements.html#generators)\n",
    "\n",
    "First let's see how you can apply a curtailment action in mainly 4 different ways. For the sake of the example, imagine we limit the production of the generator 2 at 10% of its maximum generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import grid2op\n",
    "from grid2op.Action import PlayableAction\n",
    "from grid2op.Parameters import Parameters\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "param = Parameters()\n",
    "param.NO_OVERFLOW_DISCONNECTION = True\n",
    "env1 = grid2op.make(\"educ_case14_storage\",\n",
    "                    test=True,\n",
    "                    action_class=PlayableAction,\n",
    "                    param=param,\n",
    "                    backend=bk_cls())\n",
    "\n",
    "gen_id = 2  # because we want to act on generator 2\n",
    "amount_ratio = 0.1  # that is the 7% of the limit\n",
    "\n",
    "# method 1\n",
    "curtail_act_1 = env1.action_space({\"curtail\": [(gen_id, amount_ratio)]})\n",
    "\n",
    "# method 2\n",
    "curtail_vect = np.zeros(env1.n_gen) - 1.\n",
    "curtail_vect[gen_id] = amount_ratio\n",
    "curtail_act_2 = env1.action_space({\"curtail\": curtail_vect})\n",
    "\n",
    "# method 3\n",
    "curtail_act_3 = env1.action_space()\n",
    "curtail_act_3.curtail = [(gen_id, amount_ratio)]\n",
    "\n",
    "\n",
    "# method 4\n",
    "curtail_act_4 = env1.action_space()\n",
    "curtail_act_4.curtail = curtail_vect\n",
    "\n",
    "# all these actions are equal\n",
    "print(curtail_act_4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One \"side effect\" of applying a curtailment action can be redispatching. Indeed, the balance between production and load should maintain. The data (before curtailment) meets this criteria. This means that whatever curtailment is performed should be absorbed somehow. \n",
    "\n",
    "We decided to have the dispatchable generators produce more to compensate the curtailment. This behaviour is showed in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env1.set_id(0)\n",
    "obs_init = env1.reset()\n",
    "obs, *_ = env1.step(curtail_act_1)\n",
    "print(f\"Amount of MW curtailed for generator {gen_id}: {obs.curtailment_mw[gen_id]:.2f} MW\")\n",
    "print(f\"Sum of redispatched generator: {obs.actual_dispatch.sum():.2f} MW\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, curtailment action will perdure on time. This means that if you perform a curtailment action on a step, unless you modify it later on. \n",
    "\n",
    "In this example above, the generator with id 2 will be constrained to produce less than 15% of its maximum production until you change that.\n",
    "\n",
    "We explain it in the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Curtailment setpoint for generator {gen_id}: {obs.curtailment_limit[gen_id]:.2f} \"\n",
    "      f\"({obs.curtailment_limit_mw[gen_id]:.2f} MW)\")\n",
    "\n",
    "# we perform a do nothing action\n",
    "obs0, *_ = env1.step(env1.action_space())\n",
    "print(f\"Curtailment setpoint for generator {gen_id} (after do nothing action): \"\n",
    "      f\"{obs0.curtailment_limit[gen_id]:.2f} \"\n",
    "      f\"({obs0.curtailment_limit_mw[gen_id]:.2f} MW)\")\n",
    "# we perform a do nothing action\n",
    "obs1, *_ = env1.step(env1.action_space())\n",
    "print(f\"Curtailment setpoint for generator {gen_id} (after a second do nothing action): \"\n",
    "      f\"{obs1.curtailment_limit[gen_id]:.2f} \"\n",
    "      f\"({obs1.curtailment_limit_mw[gen_id]:.2f} MW)\")\n",
    "# we perform a do nothing action\n",
    "obs2, *_ = env1.step(env1.action_space())\n",
    "print(f\"Curtailment setpoint for generator {gen_id} (after a third do nothing action): \"\n",
    "      f\"{obs2.curtailment_limit[gen_id]:.2f} \"\n",
    "      f\"({obs2.curtailment_limit_mw[gen_id]:.2f} MW)\")\n",
    "\n",
    "# we perform an action to remove the curtailment\n",
    "curtail_act_reset = env1.action_space({\"curtail\": [(gen_id, 1.0)]})\n",
    "obs3, *_ = env1.step(curtail_act_reset)\n",
    "print(f\"Curtailment setpoint for generator {gen_id} (after setting the curtailment to 100% again): \"\n",
    "      f\"{obs3.curtailment_limit[gen_id]:.2f} \"\n",
    "      f\"({obs3.curtailment_limit_mw[gen_id]:.2f} MW)\")\n",
    "\n",
    "# we perform another do nothing action\n",
    "obs4, *_ = env1.step(env1.action_space())\n",
    "print(f\"Curtailment setpoint for generator {gen_id} (after a third do nothing action): \"\n",
    "      f\"{obs4.curtailment_limit[gen_id]:.2f} \"\n",
    "      f\"({obs4.curtailment_limit_mw[gen_id]:.2f} MW)\")\n",
    "\n",
    "# and we can even plot it to see what's going on\n",
    "plt.figure(figsize=(15,10))\n",
    "plt.vlines(list([1, 5]), ymin=0., ymax=18, label=\"an action occured just before this step\", alpha=0.5)\n",
    "plt.plot([obs_init.gen_p[gen_id],\n",
    "          obs.gen_p[gen_id],\n",
    "          obs0.gen_p[gen_id],\n",
    "          obs1.gen_p[gen_id],\n",
    "          obs2.gen_p[gen_id],\n",
    "          obs3.gen_p[gen_id],\n",
    "          obs4.gen_p[gen_id],],\n",
    "         label=\"generation limit\", linestyle=\"-.\")\n",
    "plt.legend()\n",
    "plt.ylim([0., 18])\n",
    "plt.xlabel(\"step\")\n",
    "plt.title(f\"Evolution of the generation of generator {gen_id}\")\n",
    "_ = plt.ylabel(f\"Generation for generator {gen_id} (MW)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An example of the behaviour of this feature is provided bellow. In this example, we perform 4 curtailment actions at 4 different steps (showed by vertical bars in the final graphs) and show, for generator 2:\n",
    "- generation limit (this is the maximum allowed by the curtailment)\n",
    "- effective curtailment (this is the total amount of MW that has been curtailed)\n",
    "- reference generation (generation there would have been without curtailment)\n",
    "- actual generation (the generation observed for this generator, the power that is really injected on the grid)\n",
    "- sum of dispatch (total power absorbed by the dispatchable generator to compensate for the curtailment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env1 = grid2op.make(\"educ_case14_storage\", \n",
    "                    test=True,\n",
    "                    action_class=PlayableAction,\n",
    "                    param=param,\n",
    "                    backend=bk_cls())\n",
    "\n",
    "gen_id = 2\n",
    "ratios_max = [0.25, 0.24, 0.23, 0.25]  # don't change it\n",
    "n_steps = 40 # don't change it\n",
    "acts = []\n",
    "prod1 = []\n",
    "prod2 = []\n",
    "curtailment_limit = []\n",
    "curtail_gen1 = []\n",
    "sum_disp = []\n",
    "li_id_xplt = set()\n",
    "for ratio in ratios_max:\n",
    "    acts.append(env1.action_space({\"curtail\": [(gen_id, ratio)]}))\n",
    "    \n",
    "for step in range(n_steps):\n",
    "    if step % 10 == 0:\n",
    "        # execute a curtailment action\n",
    "        li_id = step // 10\n",
    "        li_id_xplt.add(step)\n",
    "        ratio_max = ratios_max[li_id]\n",
    "        obs1_3, reward1_3, done1_3, info1_3 = env1.step(acts[li_id])\n",
    "    else:\n",
    "        obs1_3, reward1_3, done1_3, info1_3 = env1.step(env1.action_space())\n",
    "        \n",
    "    if done1_3:\n",
    "        raise RuntimeError(\"Environment did game over, this is not normal\")\n",
    "    if info1_3[\"exception\"]:\n",
    "        raise RuntimeError(\"Exception for info1_3\")\n",
    "\n",
    "    prod1.append(obs1_3.gen_p[gen_id])\n",
    "    prod2.append(obs1_3.gen_p_before_curtail[gen_id])\n",
    "    curtail_gen1.append(obs1_3.curtailment_mw[gen_id])\n",
    "    curtailment_limit.append(obs1_3.curtailment_limit_mw[gen_id])\n",
    "    sum_disp.append(obs1_3.actual_dispatch.sum())\n",
    "    \n",
    "plt.figure(figsize=(15,10))\n",
    "plt.vlines(list(li_id_xplt), ymin=0., ymax=18., label=\"an action occured at this step\", alpha=0.5)\n",
    "plt.plot(curtailment_limit, label=\"generation limit\", linestyle=\"-.\")\n",
    "plt.plot(curtail_gen1, label=\"effective curtailment\")\n",
    "plt.plot(prod2, label=\"reference generation (without curt.)\", linestyle=\"--\")\n",
    "plt.plot(prod1, label=\"actual generation (with curt.)\", linestyle=\":\", linewidth=3)\n",
    "plt.plot(sum_disp, label=\"sum of dispatch\", linestyle=\":\", linewidth=3)\n",
    "plt.legend()\n",
    "plt.xlabel(\"step\")\n",
    "plt.title(f\"Evolution of the generation of generator {gen_id}\")\n",
    "_ = plt.ylabel(f\"Generation for generator {gen_id} (MW)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Important limit to the curtailment\n",
    "\n",
    "As always in grid2op, we suppose that the \"time series data\" (the one the agent has no access to, present in the environment and representing the \"optimal\" generator setpoint from the \"market\" - or a central authority) are balanced. This means that, without any intervention from any agent, the total generation exactly match the total demand plus the losses.\n",
    "\n",
    "When an agent perform a redispatching or a curtailment, it will affect this equilibrium. This is why grid2op \"compensate\" the action of the agent by ensuring that the \"global action\" (*ie* the sum of all modifications made by the agent) sums up to 0.\n",
    "\n",
    "To do this, the environment will modify the **controllable** generators. One key consequence is that you can take curtailment action that would \"overload\" the grid and lead directly to game over.\n",
    "\n",
    "For example, say an agent curtails 50MW of renewable energy, but the available \"ramp up\" of controlable generators is only 40MW (*ie* you can increase the controlable generators of at most 40MW between two consecutive steps). In this case the environment cannot simply create the missing 10 MW and it directly lead to a game over, as showed below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.set_id(0)\n",
    "obs = env.reset()\n",
    "act = env.action_space({\"curtail\": [(el, 0.) for el in range(env.n_gen) if env.gen_renewable[el]]})\n",
    "obs, reward, done, info = env.step(act)\n",
    "print(f\"Is the environment done: {done}\")\n",
    "info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same phenomenon occurs when you \"undo\" the curtailment, for example when you start from a situation where curtailment is really strong (you limited the curtailment to a really low level) and you suddently remove all curtailment, you might end up with issues due to ramps down, as shown bellow:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.set_id(0)\n",
    "obs = env.reset()\n",
    "# gradually decrease the total generation from renewable sources\n",
    "act = env.action_space({\"curtail\": [(el, 0.13) for el in range(env.n_gen) if env.gen_renewable[el]]})\n",
    "obs, reward, done, info = env.step(act)\n",
    "act = env.action_space({\"curtail\": [(el, 0.0) for el in range(env.n_gen) if env.gen_renewable[el]]})\n",
    "obs, reward, done, info = env.step(act)\n",
    "\n",
    "# increase it \"at once\"\n",
    "act = env.action_space({\"curtail\": [(el, 1.) for el in range(env.n_gen) if env.gen_renewable[el]]})\n",
    "obs, reward, done, info = env.step(act)\n",
    "print(f\"Is the environment done: {done}\")\n",
    "info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To alleviate this issue, you have two possibilities:\n",
    "- make the environment more tolerant to these \"bad\" actions. In this case, the environment will \"limit\" the amount of MW curtailed in your action (using a heuristic not detailed here) and preventing you to \"break\" the grid. This can be used only if you create yourself your environment (so this is not suitable at the evaluation time of the l2rpn competitions) and is particularly useful if you envisage \"curiculum learning\" (*ie* start learning with an easier environment and when your agent achieves good performances change the environment parameters closer to the evaluation parameters)\n",
    "- do not change the environment at all, but call the `act.limit_curtail_storage(...)` to... limit the curtailment (and the storage) part of your action and keep it \"manageable\" (hopefully) by the environment.\n",
    "\n",
    "We explain the two methods in the sub sub sections bellow.\n",
    "\n",
    "#### modify the parameters of the environment\n",
    "\n",
    "As you see here, setting the `LIMIT_INFEASIBLE_CURTAILMENT_STORAGE_ACTION` to ``True`` solves the issue. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param = env.parameters\n",
    "param.LIMIT_INFEASIBLE_CURTAILMENT_STORAGE_ACTION = True\n",
    "env.change_parameters(param)\n",
    "\n",
    "env.set_id(0)\n",
    "obs = env.reset()\n",
    "act = env.action_space({\"curtail\": [(el, 0.) for el in range(env.n_gen) if env.gen_renewable[el]]})\n",
    "obs, reward, done, info = env.step(act)\n",
    "print(f\"Is the environment done: {done}\")\n",
    "\n",
    "# set back the parameters to orignal value\n",
    "param = env.parameters\n",
    "param.LIMIT_INFEASIBLE_CURTAILMENT_STORAGE_ACTION = False\n",
    "env.change_parameters(param)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this method you should be guaranteed to have a valid action that will not lead to a game over (because of curtailment, you can still \"game over\" because of every other reasons)\n",
    "\n",
    "#### modify the action\n",
    "\n",
    "We developed an method of an action that can limit its impact and keep it \"small enough\" so that it does not overflow.\n",
    "\n",
    "To use it you need:\n",
    "- an observation: it will be used to compute the amount of MW that can be absorbed / produced by the controlable generators\n",
    "- a \"margin\": when the limitation is done by the environment, the environment knows exactly the amount of MW the generators can increase or decrease (this is not exactly the ramps because even if the agent does nothing, the generator will increase / decrease their production). If you put the \"margin\" to 0. (which we do not recommend), the \"limit the action\" will act as if the generators will not increase / decrease their production. Basically you can think of it of a \"safety margin\": the higher it is the more your action will be impacted but the less likely"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if you don't limit the action, it breaks\n",
    "env.set_id(0)\n",
    "obs = env.reset()\n",
    "act = env.action_space({\"curtail\": [(el, 0.) for el in range(env.n_gen) if env.gen_renewable[el]]})\n",
    "obs, reward, done, info = env.step(act)\n",
    "print(f\"Is the environment done (without the limit_curtail_storage(...) ): {done}\")  # of course it is...\n",
    "\n",
    "# if you limit the action, it's ok\n",
    "env.set_id(0)\n",
    "obs = env.reset()\n",
    "act = env.action_space({\"curtail\": [(el, 0.) for el in range(env.n_gen) if env.gen_renewable[el]]})\n",
    "act.limit_curtail_storage(obs, margin=10.)\n",
    "obs, reward, done, info = env.step(act)\n",
    "print(f\"Is the environment done (with limit_curtail_storage(...) ): {done}\")  # of course it is...\n",
    "\n",
    "# if limit the action, but does not take enough \"margin\", it breaks again\n",
    "env.set_id(0)\n",
    "obs = env.reset()\n",
    "act = env.action_space({\"curtail\": [(el, 0.) for el in range(env.n_gen) if env.gen_renewable[el]]})\n",
    "act.limit_curtail_storage(obs, margin=0.)\n",
    "obs, reward, done, info = env.step(act)\n",
    "print(f\"Is the environment done (with limit_curtail_storage(...) and not enough margin ): {done}\")  # of course it is..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nothing prevents you to use the `act.limit_curtail_storage(obs, margin=...)` in your code, when training your agent (for example as a \"heuristic\" like in the `GymEnvWithRecoWithDN` showed in notebook [04_TrainingAnAgent](./04_TrainingAnAgent.ipynb)). In this case you only need to properly set the `margin` parameter as shown above."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}