examples/citylearn_rlem23_tutorial.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a href=\"https://colab.research.google.com/github/intelligent-environments-lab/CityLearn/blob/master/examples/citylearn_rlem23_tutorial.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "13i7KQ9t-CV8"
      },
      "source": [
        "# CityLearn: A Tutorial on Reinforcement Learning Control for Grid-Interactive Efficient Buildings and Communities\n",
        "---\n",
        "\n",
        "Authors:\n",
        "*   [Kingsley Nweye](https://kingsleynweye.com), The University of Texas at Austin, [nweye@utexas.edu](mailto:nweye@utexas.edu)\n",
        "*   [Allen Wu](https://www.linkedin.com/in/allenjeffreywu), The University of Texas at Austin, [allen.wu@utexas.edu](mailto:allen.wu@utexas.edu)\n",
        "* [Hyun Park](), The University of Texas at Austin, [hyun_0421@utexas.edu](mailto:hyun_0421@utexas.edu)\n",
        "* [Yara Almilaify](https://www.linkedin.com/in/yara-almilaify), The University of Texas at Austin, [yara.m@utexas.edu](mailto:yara.m@utexas.edu)\n",
        "*   [Zoltan Nagy](https://www.caee.utexas.edu/people/faculty/faculty-directory/nagy), The University of Texas as Austin, [nagy@utexas.edu](mailto:nagy@utexas.edu)\n",
        "\n",
        "This tutorial notebook will help users get acquainted with the [CityLearn](https://www.citylearn.net) OpenAI Gym environment, developed for easy implementation and benchmarking of control algorithms, e.g., rule-based control, model predictive control or deep reinforcement learning control in the demand response, building energy and grid-interactive community domain. By the end of the tutorial, participants will learn how to design their own simple or advanced control algorithms to provide energy flexibility, and acquire familiarity with the CityLearn environment for extended use in personal projects."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "baBrylkqrRhH"
      },
      "source": [
        "# Glossary\n",
        "***\n",
        "\n",
        "- AI - artificial intelligence\n",
        "- API - application programming interface\n",
        "- DER - distributed energy resource\n",
        "- ESS - energy storage system\n",
        "- EV - electric vehicle\n",
        "- GEB - grid-interactive efficient building\n",
        "- GHG - greenhouse gas\n",
        "- HVAC - heating, ventilation and air conditioning\n",
        "- KPI - key performance indicator\n",
        "- MPC - model predictive control\n",
        "- PV - photovoltaic\n",
        "- RBC - rule-based control\n",
        "- RLC - reinforcement learning control\n",
        "- SoC - state of charge\n",
        "- TES - thermal energy storage\n",
        "- ToU - time of use\n",
        "- ZNE - zero net energy"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QH81wjfsJsv1"
      },
      "source": [
        "<a name=\"overview\"></a>\n",
        "# Overview\n",
        "---\n",
        "\n",
        "This workshop will consist of a presentation of the underlying background for CityLearn and its relevance to climate change mitigation. We provide a walk-through tutorial on how to set up and interact with the simulation environment using a real-world dataset from a grid-interactive residential community in Fontana, California.\n",
        "\n",
        "This notebook will provide a guide on how to use a simple rule-based control architecture and advanced reinforcement control algorithms to manage batteries in each home in the community with a goal of minimizing the cost of purchased electricity, carbon emissions associated with consuming electricity from the grid, as well as improve on other grid-level KPIs e.g. peak demand, load factor and ramping that are critical for the long-term sustainability of decarbonizing existing power generation sources on the supply side and electrification of buildings on the demand side."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_2EFretjvwqQ"
      },
      "source": [
        "## Contributions\n",
        "\n",
        "The primary contribution of this tutorial is to introduce the software tool, CityLearn, to model and benchmark simple and advanced control strategies in grid-interactive smart communities, e.g., demand response and load shaping in buildings. The secondary contribution is to bring awareness to datasets for controls research that are provided within the CityLearn environment."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jt6XDw2Cv9BJ"
      },
      "source": [
        "## Learning Outcomes\n",
        "\n",
        "<img src=\"https://media.giphy.com/media/PLHdpauwfN2MvHcHxL/giphy.gif\" height=200></img>\n",
        "\n",
        "The primary learning outcome for participants is to gain familiarity with CityLearn environment, its application programming interface (API) and dataset offerings for extended use in academic research or personal projects. Other secondary outcomes are to:\n",
        "\n",
        "1. Understand how electrification, distributed energy resources e.g. batteries, photovoltaic (PV) systems and smart controls provide a promising pathway to decarbonization and energy flexibility.\n",
        "2. Learn how to design and optimize their own rule-based control (RBC) agent for battery management using readily available knowledge of a building's energy use.\n",
        "3. Identify the challenges surrounding the generalizability of an RBC agent and how reinforcement learning (RL) can mitigate these challenges.\n",
        "4. Train their own RL Tabular Q-Learning algorithm.\n",
        "5. Evaluate the performance of a standard model-free deep RL algorithm in optimizing key performance indicators (KPIs) that are targeted at quantifying energy flexibility, environmental and economic costs.\n",
        "6. Learn the effect of different control algorithms and their parameters in improving these KPIs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "99jkSa_KmrDH"
      },
      "source": [
        "<a name=\"climate-impact\"></a>\n",
        "# Climate Impact\n",
        "---\n",
        "\n",
        "The residential building stock in the United States is responsible for 21% ([Energy Information Administration, 2022](https://www.eia.gov/totalenergy/data/monthly/archive/00352211.pdf)) of energy consumption and 20% of greenhouse gas emissions ([Goldenstein et al., 2020](https://doi.org/10.1073/pnas.1922205117)). Electrification of end-uses, as well as decarbonizing the electrical grid through renewable energy sources such as solar and wind, constitutes a pathway to zero-emission buildings ([Leibowicz et al., 2018](https://doi.org/10.1016/j.apenergy.2018.09.046)). However, electrifying fossil-fueled building end-uses in fact could increase the demand on the existing electricity power grid and if power generation sources are not decarbonized at a similar rate as electrification happening on the demand side, will in fact result in adverse effects of increased greenhouse gas (GHG) emissions. Also, new grid infrastructure constitutes a significant capital investment and requires extensive planning and execution.\n",
        "\n",
        "Through distributed energy resources (DERs), buildings can provide flexibility services to the existing grid infrastructure in demand response events. On-site solar photovoltaic (PV) systems can also reduce dependence on the grid through self-generation. Yet, the intermittency of renewable energy sources introduces additional challenges of grid instability due to a mismatch between electricity generation (supply) and demand ([Suberu et al., 2014](https://doi.org/10.1016/j.rser.2014.04.009)). The California duck-curve is a good illustration of the mismatch in renewable supply and demand, and shows the effect of increased PV penetration that may lead to high ramp rate after the sun sets in the evening which the electric power grid may not be able to handle ([Denholm et al., 2008](https://www.nrel.gov/docs/fy08osti/42305.pdf)).\n",
        "\n",
        "<figure class=\"image\">\n",
        "  <img src=\"https://github.com/intelligent-environments-lab/CityLearn/blob/master/assets/images/california_duck_curve.png?raw=true\"  width=\"600\" alt=\"Load shapes in California with various PV penetration scenarios in the United States western grid.\">\n",
        "  <figcaption>Figure: Load shapes in California highlighting the duck curve caused by various PV penetration scenarios in the western United States grid (modified from <a href=\"https://www.nrel.gov/docs/fy08osti/42305.pdf\">Denholm et al., 2008</a>).</figcaption>\n",
        "</figure>\n",
        "\n",
        "Advanced control algorithms including model-predictive control (MPC) and reinforcement learning control (RLC) are thus, critical to properly manage DERs and even out the supply-demand imbalance brought about by renewable energy sources. However, a major challenge for the adoption of reinforcement learning (RL) in grid-interactive efficient buildings (GEBs) is the ability to benchmark control algorithm performance to accelerate their deployment on live systems.\n",
        "\n",
        "CityLearn ([Vazquez-Canteli et al., 2019](https://doi.org/10.1145/3360322.3360998)) provides a platform for such benchmarking tasks. The different stakeholder in the energy and real-estate market including, utility companies, grid-operators, home-owners and policy makers can utilize CityLearn to make decisions on what control policies are viable and provide the best performance as more controllable DERs become available."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o5sbM_JPpdMR"
      },
      "source": [
        "<a name=\"target-audience\"></a>\n",
        "# Target Audience\n",
        "---\n",
        "\n",
        "The target audience for this tutorial includes the following:\n",
        " - Academic, private and commercial researchers or professionals that are interested in sustainable artificial intelligence (AI)-related pathways to electrification and building decarbonization.\n",
        " - AI enthusiasts with at least beginner level of expertise in programming or data science whom may have or pick interest in solving control problems or are interested in learning about a new reinforcement learning (RL) environment that deviates from popular simpler problems e.g. the [Atari](https://www.gymlibrary.dev/environments/atari/), [MuJoCo](https://www.gymlibrary.dev/environments/mujoco/) and [Box2D](https://www.gymlibrary.dev/environments/box2d/) Gym environments, to real-world problems with urgency like climate change mitigation and decarbonization of the electric grid infrastructure and building end-uses."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gQgijl46pYzn"
      },
      "source": [
        "<a name=\"prereqs\"></a>\n",
        "# Prerequisites\n",
        "---\n",
        "\n",
        "> ⚠️ **NOTE**:\n",
        "> This CityLearn tutorial has a fairly low entry level and participants do not need to have prior experience in reinforcement learning (RL) nor use of a Gym environment. However, participants need to have at least, beginner knowledge in Python or other similar high-level scripting language.\n",
        "\n",
        "The [Building Optimization Testing (BOPTEST) tutorial](https://colab.research.google.com/drive/1WeA_3PQeySba0MMRRte_oZTF7ptlP_Ra?usp=sharing#scrollTo=Ae18iXNKWV5I) provides a __very__ good overview on some of the topics and methods that are discussed and applied in this tutorial. While we will briefly provide background on some of these topics and methods, it is encouraged but not required, that participants in this tutorial first read up the following sections in the [BOPTEST tutorial](https://colab.research.google.com/drive/1WeA_3PQeySba0MMRRte_oZTF7ptlP_Ra?usp=sharing#scrollTo=Ae18iXNKWV5I):\n",
        "\n",
        "- [Introduction to Reinforcement Learning](https://colab.research.google.com/drive/1WeA_3PQeySba0MMRRte_oZTF7ptlP_Ra#scrollTo=Fas232CyMX6_)\n",
        "- [The OpenAI-Gym Standard](https://colab.research.google.com/drive/1WeA_3PQeySba0MMRRte_oZTF7ptlP_Ra#scrollTo=7YnuNAQdM_L2)\n",
        "\n",
        "Other prerequisites that are technical in nature are:\n",
        "- Beginner knowledge in Python\n",
        "- Basic knowledge about [Object-Oriented Programming (OOP) in Python 3](https://realpython.com/python3-object-oriented-programming/) and the use of inheritance\n",
        "- Familiarity with the Jupyter Notebook environment"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LCeaK9CEngV5"
      },
      "source": [
        "<a name=\"background\"></a>\n",
        "# Background\n",
        "---\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3RN4VIYf0_Zs"
      },
      "source": [
        "## Grid-Interactive Efficient Buildings and their Energy Flexibility\n",
        "\n",
        "As buildings become electrified and the penetration of renewable energy source increases, a smart approach to manage and control building loads to ensure that these developments do not bring about grid insecurity is needed. The Department of Energy (DOE), introduced the __Grid-Interactive Efficient Building (GEB)__ Initiative to promote the integration of distributed energy resources (DERS) such as photovoltaic (PV) systems, electric vehicles (EVs), active and passive storage systems e.g. batteries, thermal storage tanks and thermal mass (walls) in buildings that can provide the grid with energy flexibility. A GEB is defined by [Neukomm et al., 2019](https://www1.eere.energy.gov/buildings/pdfs/75470.pdf)] as:\n",
        "\n",
        "> [...] an energy-efficient building that uses smart technologies and on-site DERs to provide demand flexibility while co-optimizing for energy cost, grid services, and occupant needs and preferences, in a continuous and integrated way.\n",
        "\n",
        "GEBs are characterized by their energy efficiency, interconnectivity, smartness and energy flexibility as highlighted in the figure below:\n",
        "\n",
        "<figure class=\"image\">\n",
        "  <img src=\"https://github.com/intelligent-environments-lab/CityLearn/blob/master/assets/images/grid_interactive_building-neukomm.png?raw=true\"  width=\"600\" alt=\"The characteristics of a Grid-Interactive Efficient Building (GEB)\">\n",
        "  <figcaption>Figure: The characteristics of a Grid-Interactive Efficient Building (GEB) (<a href=\"https://www1.eere.energy.gov/buildings/pdfs/75470.pdf\">Neukomm et al., 2019</a>).</figcaption>\n",
        "</figure>\n",
        "\n",
        "The energy efficiency of GEBs by means of tighter envelopes and high-quality construction that minimize heat losses lead to reduced loads without the need to make changes to a building's function or for changes in occupant preferences. PV adoption in GEBs provides self-generation capabilities which reduce the net load (difference between building end-use load and renewable energy generation) that will otherwise be satisfied by the electric grid alone. To then solve the duck curve problem that is introduced by renewable power generation, GEBs are able to shed and/or shift their loads. In load shedding, GEBs reduce their electricity use for a short time period and typically on short notice during peak demand periods. Load shifting in GEBs entails intentional and planned change in the timing of electricity use to reduce the demand on the grid during typical peak periods. Load shifting helps take advantage of cheaper and cleaner electricity. Load shedding  an result in shifting the shedded load.\n",
        "\n",
        "The figure below shows the changes in a building profile as it becomes more efficient, incorporates self generation and provides load shedding and shifting grid services.\n",
        "\n",
        "<figure class=\"image\">\n",
        "  <img src=\"https://github.com/intelligent-environments-lab/CityLearn/blob/master/assets/images/grid_interactive_building_profile-neukomm.png?raw=true\"  width=\"600\" alt=\"GEB load profiles\">\n",
        "  <figcaption>Figure: GEB load profiles (<a href=\"https://www1.eere.energy.gov/buildings/pdfs/75470.pdf\">Neukomm et al., 2019</a>).</figcaption>\n",
        "</figure>\n",
        "\n",
        "Particularly, load shifting services could be achieved through control of active storage systems in combination with renewable power generation. Smart controls can adapt to changes in occupant behavior, weather conditions or respond to external signals e.g. temperature set-point as in the case of demand-response, while maintaining energy efficiency and without causing discomfort to occupants. This adaptability is often referred to as __energy flexibility__ of a building ([Jensen et al., 2017](https://doi.org/10.1016/j.enbuild.2017.08.044))."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z6mw91EAvQ-v"
      },
      "source": [
        "## Control Theories for Grid-Interactive Efficient Buildings\n",
        "Rule-based control (RBC) is a popular control theory for building energy management. It makes use of simple if-else statements in their decision making process e.g. \"if outdoor dry-bulb temperature is < 20<sup>o</sup>C and hour is 10 PM, charge battery by 5% of its capacity\". Advanced control systems such as model predictive control (MPC) ([Drgona et al., 2020](https://doi.org/10.1016/j.arcontrol.2020.09.001)) and reinforcement learning control (RLC) ([Wang et al., 2020](https://doi.org/10.1016/j.apenergy.2020.115036)) can be a major driver for executing grid services by automating the operation of energy systems, while adapting to individual characteristics of occupants and buildings. Comparatively, RBC and RLC are inexpensive to implement as they have a lower entry bar for domain knowledge of the systems to be controlled. RBC and vanilla RLC may perform sub-optimally compared to MPC as they are not modeled after the system under control although there exists a branch of RLC that includes a model of the controlled system. However, RLC is a data-driven solution and as more training data become available, it _learns_ the model of the building or system under control and achieves comparable performance as MPC. The greatest strength of RLC is its ability to adapt to changes in the system or building it controls as the dynamics e.g. thermal, occupancy change. We provide more context on reinforcement learning in the following section."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1d05ZUl11HUz"
      },
      "source": [
        "## Reinforcement Learning Control for Grid-Interactive Efficient Buildings\n",
        "\n",
        "Here we provide a simple description of reinforcement learning (RL) in the context of grid-interactive efficient buildings (GEBs) without as little theoretical jargon as possible 🙂. For a more detailed description of RL, refer to the [BOPTEST Tutorial Introduction to Reinforcement Learning](https://colab.research.google.com/drive/1WeA_3PQeySba0MMRRte_oZTF7ptlP_Ra#scrollTo=Fas232CyMX6_) and for a thorough introduction, refer to [Sutton and Barto, 2018](http://www.incompleteideas.net/book/the-book.html).\n",
        "\n",
        "<figure class=\"image\">\n",
        "  <img src=\"https://github.com/intelligent-environments-lab/CityLearn/blob/master/assets/images/supervised_vs_unsupervised_vs_rl.png?raw=true\"  width=\"250\" alt=\"Overview of the three machine learning techniques.\">\n",
        "  <figcaption>Figure: Overview of the three machine learning techniques (<a href=\"https://www.taylorfrancis.com/chapters/edit/10.4324/9781315142074-37/reinforcement-learning-intelligent-environments-zoltan-nagy-june-young-park-josé-ramón-vázquez-canteli\">Nagy et al., 2018</a>).</figcaption>\n",
        "</figure>\n",
        "\n",
        "Machine learning techniques are generally characterized as either supervised or unsupervised learning. In supervised learning, the observations, otherwise called samples, and their response, otherwise called target are known when training the model. Supervised learning is further classified into regression and classification models where the former learns to predict a continuous variable and the latter learns to predict a discrete class. The supervised machine learning model must then learn the mapping of samples and targets by minimizing a loss function that evaluates the error between predicted targets and actual targets. An example of regression modelling in the context of GEBs is the prediction of a building's electricity consumption (target) when the the weather conditions e.g. dry-bulb temperature and relative humidity, as well as their associated timestamps (samples) are known. An example of a classification model in the context of GEBs is the prediction of a binary variable such as if building is \"occupied\" or \"not occupied\" (target) given the same samples as the regression example. It so happens that these kinds of problems are common in the building energy domain where predicting a building's electricity consumption and occupancy can help with scheduling and occupant-centric control to balance energy efficiency and comfort.\n",
        "\n",
        "Unsupervised learning on the other hand differs from supervised learning in the sense that the associated targets for the samples are unknown but models can be used to find common patterns amongst samples. This is important especially in exploratory data analysis to draw preliminary conclusions about a new dataset, and in situations where the class labels are unknown. An unsupervised model can then be used to infer data-driven class labels that can be used as targets in supervised classification models. An example of such application in buildings is the classification of customer load profiles to determine and allocate appropriate tariff plans.\n",
        "\n",
        "RL is similar to both supervised and unsupervised learning such that it follows the same idea of learning from observations. However in contrast to supervised learning, there are no known labeled targets. Instead, a model (agent/controller) influences the targets by acting in an environment that rewards actions that have desirable outcomes but penalizes those that lead to adverse outcomes. An example of RL in GEBs is charging/discharging of a battery (action) every hour (control time step) with the objective of minimizing the sum of electricity consumption of a building over a control horizon e.g. a year, given the same sample variables in the supervised learning example (observations). These actions affect the hourly electricity consumption of the building (reward). RL is thought of as a trial an error approach because typically, the agent starts off with no prior knowledge of the environment it acts in but learns from experience to associate observations with actions that maximize its rewards. The association is called a control policy.\n",
        "\n",
        "<figure class=\"image\">\n",
        "  <img src=\"https://github.com/intelligent-environments-lab/CityLearn/blob/master/assets/images/reinforcement_learning.png?raw=true\"  width=\"400\" alt=\"Basic structure of reinforcement learning.\">\n",
        "  <figcaption>Figure: Basic structure of reinforcement learning (<a href=\"https://www.taylorfrancis.com/chapters/edit/10.4324/9781315142074-37/reinforcement-learning-intelligent-environments-zoltan-nagy-june-young-park-josé-ramón-vázquez-canteli\">Nagy et al., 2018</a>).</figcaption>\n",
        "</figure>\n",
        "\n",
        "The experiential learning characteristic of RL is very promising for GEBs because it provides a scalable control solution as it can adapt to each building's unique conditions and occupant interactions that influence observations. It also means that the agent need not have a building's model that can involve a level of complexity that is expensive to design for. [Vazquez-Canteli et al., 2019](https://doi.org/10.1016/j.apenergy.2018.11.002) provide a comprehensive overview of reinforcement learning in the built environment."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6jM2IsZ91UnJ"
      },
      "source": [
        "## CityLearn\n",
        "\n",
        "[CityLearn](https://doi.org/10.1145/3360322.3360998) is an open source OpenAI Gym environment targeted at the easy implementation and benchmarking of simple and advanced control algorithms e.g. rule-based control (RBC) model predictive control (MPC) or reinforcement learning control (RLC). It has been utilized in [demand response, energy management, voltage regulation and algorithm benchmarking applications](https://www.citylearn.net/#applications). CityLearn is used in [The CityLearn Challenge](https://www.citylearn.net/citylearn_challenge/index.html) which is an opportunity to compete in investigating the potential of AI and distributed control systems to tackle multiple problems in the built-environment. It attracts a multidisciplinary audience including researchers, industry experts, sustainability enthusiasts and AI hobbyists as a means of crowd-sourcing solutions to these multiple problems."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RGDJpFfdvltb"
      },
      "source": [
        "### Environment\n",
        "\n",
        "<figure class=\"image\">\n",
        "  <img src=\"https://github.com/intelligent-environments-lab/CityLearn/blob/master/assets/images/ccai_tutorial_citylearn_building_schematic.png?raw=true\"  width=\"300\" alt=\"An overview of the heating, ventilation and air conditioning systems, energy storage systems, on-site electricity sources and grid interaction in buildings in the CityLearn environment.\" style=\"background-color:white;margin:20px;padding:5px\">\n",
        "  <figcaption>Figure: An overview of the heating, ventilation and air conditioning systems, energy storage systems, end uses, on-site electricity sources and grid interaction in buildings in the CityLearn environment.</figcaption>\n",
        "</figure>\n",
        "\n",
        "The CityLearn environment includes simplified energy models of buildings that are made up of heating, ventilation and air conditioning (HVAC) systems including air-to-water heat pumps and electric heaters. Buildings can optionally include energy storage systems (ESSs) for load shifting including batteries and thermal energy storage (TES) systems. These ESSs may be used during peak, high carbon intensity or expensive periods to meet space cooling, space heating, domestic hot water and non-shiftable (plug) loads. A collection of buildings make up a district (alias neighborhood or community).\n",
        "\n",
        "In each building, space cooling, space heating and domestic hot water heating end-use loads may be independently satisfied through air-to-water heat pumps. Alternatively, space heating and domestic hot water heating loads are satisfied through electric heaters. Each ESS is charged by the HVAC system (heat pump or electric heater) that satisfies the end-use its stored energy services. All HVAC systems and non-shiftable plug loads consume electricity from any of the available electricity sources as well as the grid. Photovoltaic (PV) systems may be included as an electricty source in a building to offset all or part of the electricity consumption from the grid. Excess PV generation is wasted as there is no energy flow between buildings."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6iVGND1_wQTQ"
      },
      "source": [
        "### Control\n",
        "\n",
        "A rule-based control (RBC), reinforcement learning control (RLC) or model predictive control (MPC) agent manages energy storage in a building by determining how much energy to store or release at any given time. The agent receives [observations](https://www.citylearn.net/overview/observations.html) from the environment and returns [actions](https://www.citylearn.net/overview/actions.html) based on some control policy. CityLearn guarantees that, at any time, the space cooling, space heating, domestic hot water heating and non-shiftable plug loads are satisfied irrespective of the control actions by:\n",
        "\n",
        "1. utilizing pre-computed or pre-measured ideal loads as input data. (ideal load refers to the energy that must be provided by an energy system to meet a control setpoint e.g. an air conditioner providing cooling energy to meet a cooling temperature setpoint of 23C in a room);\n",
        "2. sizing the heating, ventilation and air conditioning (HVAC) systems for peak ideal loads; and\n",
        "3. prioritizing the satisfaction of buildings loads before storing energy.\n",
        "\n",
        "In the case of energy discharge from the energy storage systems (ESSs), a backup controller guarantees that discharged energy does not exceed the building loads.\n",
        "\n",
        "<figure class=\"image\">\n",
        "  <img src=\"https://github.com/intelligent-environments-lab/CityLearn/blob/master/assets/images/ccai_tutorial_citylearn_centralized_control_schematic.png?raw=true\"  width=\"300\" alt=\"An overview of the power sources, energy systems and end-uses in CityLearn.\" style=\"background-color:white;margin:50px;padding:5px;\">\n",
        "  \n",
        "  <img src=\"https://github.com/intelligent-environments-lab/CityLearn/blob/master/assets/images/ccai_tutorial_citylearn_decentralized_control_schematic.png?raw=true\"  width=\"290\" alt=\"An overview of centralized (left) and decentralized with optional information sharing (right) control strategies in CityLearn.\" style=\"background-color:white;margin:50px;padding:5px;\">\n",
        "  <figcaption>Figure: An overview of centralized (left) and decentralized with optional information sharing (right) control strategies in CityLearn.</figcaption>\n",
        "</figure>\n",
        "\n",
        "CityLearn allows for centralized or decentralized control strategies. In a centralized control strategy, one agent manages the energy storage systems (ESSs) in the entire district whereas in a decentralized control strategy a one-controller-to-one-building approach is taken such that an independent controller manages the ESSs in a specific building. In the centralized control strategy, the controller receives district-level observations that may include individual building observations e.g. a building's net electricity consumption as well as observations taht are common in all buildings e.g. hour of the day. The central controller then prescribes actions for each building's ESSs. In the decentralized control strategy, each agent receives observations and returns actions for one building. The agents in the decentralized strategy may however choose to share information to achieve cooperative or competitive objectives. An example of a decentralized strategy with information sharing is the [MARLISA algorithm](https://doi.org/10.1145/3408308.3427604) developed in tandem with CityLearn."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0g5oX3xVwUyK"
      },
      "source": [
        "### Datasets\n",
        "\n",
        "The CityLearn environment makes use of [datasets](https://www.citylearn.net/overview/dataset.html#dataset) to define the simulation environment as well as provide the control agent with observations. The data files include a `schema.json` that is used to initialize the environment and comma-separated variable (CSV) files containing time series data that provide the agent with observations such as building space ideal cooling loads, ideal heating loads, ideal domestic hot water heating loads, plug loads, weather conditions, carbon intensity, time of use (ToU) electricity pricing, e.t.c. that are independent of control actions (i.e. observations that are not a function of the control actions). [See the CityLearn documentation](https://www.citylearn.net/overview/dataset.html#dataset) for a detailed explanation of CityLearn datasets and examples."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kv0D8BrywZ8_"
      },
      "source": [
        "### Other Environments\n",
        "\n",
        "Please, refer to [Application of Reinforcement Learning in buildings in BOPTEST Tutorial](https://colab.research.google.com/drive/1WeA_3PQeySba0MMRRte_oZTF7ptlP_Ra#scrollTo=6G1nWECgbmuW) for a list of other building control environments."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Il0wEGpK1fV4"
      },
      "source": [
        "## Other References\n",
        "- [OpenAI: Key Concepts in AI](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html#part-1-key-concepts-in-rl)\n",
        "- [OpenAI: Kinds of RL Algorithms](https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html#part-2-kinds-of-rl-algorithms)\n",
        "- [OpenAI: Intro to Policy Optimization](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#part-3-intro-to-policy-optimization)\n",
        "- [CityLearn Documentation](https://www.citylearn.net/index.html)\n",
        "- [CityLearn Related References](https://www.citylearn.net/references.html)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W_Qa6khFwg-N"
      },
      "source": [
        "# Hands-On Experiments\n",
        "---\n",
        "\n",
        "<img src=\"https://media.giphy.com/media/KPn24x701Asus/giphy.gif\" height=200></img>\n",
        "\n",
        "The previous sections have motivated the purpose of this tutorial as well as provided background theoretical information on reinforcement learning control (RLC) and the CityLearn environment while highlighting their importance in harnessing energy flexibility in grid-interactive efficient buildings (GEBs). The following sections will be hands-on as we will go over the experiments that this tutorial covers.\n",
        "\n",
        "Using the `citylearn_challenge_2022_phase_all` dataset, we will __learn how to__ design a controller for battery management in up to 15 buildings for load shifting flexibility where each building also has a photovoltaic (PV) system for self-generation. This way, the battery-PV system provide flexibility in the district of buildings.\n",
        "\n",
        "We will implement three types of control algorithms namely:\n",
        "1. Rule-based control (RBC)\n",
        "2. Tabular Q-Learning (TQL)\n",
        "3. Soft-Actor Critic (SAC)\n",
        "\n",
        "Rule-based control is a simple control theoryy where rules or statements that are usually in the form of `if X do Y else ...` are used to prescribe an appropriate control action at a given point in time. Tabular Q-Learning and SAC are both reinforcement learning (RL) algorithms where the difference between the two is that while in Tabular Q-Learning, Q-values are updated through exhaustive exploration and exploitation of the environment, SAC uses a function approximator to estimate the Q-values to accelerate learning. More information about Tabular Q-Learning, Q-values and SAC will be provided in the following sections so do no worry if they are not familiar terms _yet_ 🙂.\n",
        "\n",
        "By using these three control algorithms, we will highlight some of their strengths and weaknesses. We will evaluate their ability to solve the control problem using a set of key performance indicators (KPIs)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rSRCNgYzUwaf"
      },
      "source": [
        "<a name=\"software-requirements\"></a>\n",
        "# Software Requirements\n",
        "---\n",
        "\n",
        "This section installs and imports the software packages that will be used in the remainder of the tutorial. We start off by comparing the Python version of this current environment. CityLearn and its dependencies will work with `python>=3.7.x`\n",
        "\n",
        "The Python version of this environment is:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "6SbrWM_mnEDd"
      },
      "outputs": [],
      "source": [
        "!python --version"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6YClXjMknEDe"
      },
      "source": [
        "The following Python packages are required:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-eazCDbmNQqo"
      },
      "outputs": [],
      "source": [
        "# %%capture\n",
        "\n",
        "# The environment we will be working with\n",
        "!pip install CityLearn==1.8.0\n",
        "\n",
        "# For participant interactions (buttons)\n",
        "!pip install ipywidgets==7.7.2\n",
        "\n",
        "# To generate static figures\n",
        "!pip install matplotlib==3.5.3\n",
        "!pip install seaborn==0.12.2\n",
        "\n",
        "# Provide standard RL algorithms\n",
        "!pip install stable-baselines3==2.1.0\n",
        "\n",
        "# Enable gym compatibility with later stable-baselines3 versions\n",
        "!pip install shimmy==0.2.1\n",
        "\n",
        "# Results submission\n",
        "!pip install requests==2.27.1\n",
        "!pip install beautifulsoup4==4.11.2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wEW_ldpKOXDA"
      },
      "source": [
        "We can now import the relevant modules, classes and functions used in the tutorial:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xVzk4V7qUu2R"
      },
      "outputs": [],
      "source": [
        "# System operations\n",
        "import inspect\n",
        "import os\n",
        "import uuid\n",
        "\n",
        "# Date and time\n",
        "from datetime import datetime\n",
        "\n",
        "# type hinting\n",
        "from typing import List, Mapping, Tuple\n",
        "\n",
        "# Data visualization\n",
        "import matplotlib.colors as mcolors\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.ticker as ticker\n",
        "import seaborn as sns\n",
        "\n",
        "# User interaction\n",
        "from IPython.display import clear_output\n",
        "from ipywidgets import Button, FloatSlider, HBox, HTML\n",
        "from ipywidgets import IntProgress, Text, VBox\n",
        "\n",
        "# Data manipulation\n",
        "from bs4 import BeautifulSoup\n",
        "import math\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import random\n",
        "import re\n",
        "import requests\n",
        "import simplejson as json\n",
        "\n",
        "# CityLearn\n",
        "from citylearn.agents.rbc import HourRBC\n",
        "from citylearn.agents.q_learning import TabularQLearning\n",
        "from citylearn.citylearn import CityLearnEnv\n",
        "from citylearn.data import DataSet\n",
        "from citylearn.reward_function import RewardFunction\n",
        "from citylearn.wrappers import NormalizedObservationWrapper\n",
        "from citylearn.wrappers import StableBaselines3Wrapper\n",
        "from citylearn.wrappers import TabularQLearningWrapper\n",
        "\n",
        "# baseline RL algorithms\n",
        "from stable_baselines3 import SAC\n",
        "from stable_baselines3.common.callbacks import BaseCallback"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JyCH2neUw4IN"
      },
      "source": [
        "Here we include some global settings we want applied for the remainder of the notebook:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lDAe5ZeVw7_q"
      },
      "outputs": [],
      "source": [
        "# set all plotted figures without margins\n",
        "plt.rcParams['axes.xmargin'] = 0\n",
        "plt.rcParams['axes.ymargin'] = 0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jXoiLncsU3pe"
      },
      "source": [
        "<a name=\"data-description\"></a>\n",
        "# Dataset Description\n",
        "---\n",
        "\n",
        "The `citylearn_challenge_2022_phase_all` dataset used in this tutorial is from [17 zero net energy (ZNE) single-family homes in the Sierra Crest Zero Net Energy community in Fontana, California](https://www.calmac.org/publications/CSIRDD_Sol4_EPRI_Grid-Integration-of-ZNE-Communities_FinalRpt_2017-01-27.pdf), which is pictured below. The buildings were studied for grid integration of zero net energy communities as part of the California Solar Initiative program specifically exploring the [impact of high penetration PV generation and on-site electricity storage](https://www.aceee.org/files/proceedings/2016/data/papers/10_1237.pdf). This dataset is the same as that used in [The CityLearn Challenge 2022](https://www.aicrowd.com/challenges/neurips-2022-citylearn-challenge) and is represents a one-year period between August 1, 2016 and July 31, 2017.\n",
        "\n",
        "<figure class=\"image\">\n",
        "  <img src=\"https://github.com/intelligent-environments-lab/CityLearn/blob/master/assets/images/fontana_elevations.png?raw=true\"  width=\"400\" alt=\"Building elevations in Sierra Crest Zero Net Energy community in Fontana, California.\">\n",
        "  <figcaption>Figure: Building elevations in Sierra Crest Zero Net Energy community in Fontana, California.</figcaption>\n",
        "</figure>\n",
        "\n",
        "Each building in the dataset is a single-family archetype that was constructed in the mid to late 2010s and has a gross floor area between 177 m<sup>2</sup> and 269 m<sup>2</sup>. The figure below shows the envelope and system characteristics of the buildings in the Sierra Crest Zero Net Energy community. The building envelope is made from high-performance materials for improved insulation and the buildings are equipped with high-efficiency appliances, electric heating and water heating systems. The buildings also have circuit-level monitoring that provide 1-minute power time series data as well as home energy management systems.\n",
        "\n",
        "<figure class=\"image\">\n",
        "  <img src=\"https://github.com/intelligent-environments-lab/CityLearn/blob/master/assets/images/fontana_building_systems.png?raw=true\"  width=\"400\" alt=\"Building elevations in Sierra Crest Zero Net Energy community in Fontana, California.\">\n",
        "  <figcaption>Figure: Building systems in Sierra Crest Zero Net Energy community in Fontana, California.</figcaption>\n",
        "</figure>\n",
        "\n",
        "In the as-built community, eight of the 17 buildings are equipped with 6.4 kWh capacity batteries that have a 5 kW power rating, 90\\% round-trip efficiency, and 75\\% depth-of-discharge. The installed PV capacity is 4 kW or 5 kW for homes with or without batteries, respectively.\n",
        "\n",
        "The `citylearn_challenge_2022_phase_all` dataset is a transformed version of the as-built buildings and original circuit-level data that were provided by Electric Power Research Institute (EPRI), whom were stakeholders in the Sierra Crest Zero Net Energy community and a sponsor in The CityLearn Challenge 2022. This transformed version fixed data quality issues and addressed privacy issues towards making the dataset open-source. The transformed dataset has the following modifications:\n",
        "\n",
        "1. The power time series has been transformed to hourly energy data in kWh.\n",
        "2. Data quality issues such as outliers and gaps were addressed prior using inter-quartile range (IQR) outlier detection, interpolation and supervised learning prediction modeling.\n",
        "3. All buildings have 6.5 kWh capacity batteries with 5 kW power rating, and 90% round-trip efficiency.\n",
        "4. Space cooling, space heating, domestic water heating and plug loads have been coupled together under non-shiftable loads by directly using the main meter.\n",
        "\n",
        "The following projects have made use of this transformed dataset:\n",
        "1. [MERLIN: Multi-agent offline and transfer learning for occupant-centric energy flexible operation of grid-interactive communities using smart meter data and CityLearn](https://doi.org/10.48550/arXiv.2301.01148)\n",
        "2. [Combining Forecasting and Multi-Agent Reinforcement Learning Techniques on Power Grid Scheduling Task](https://doi.org/10.1109/EEBDA56825.2023.10090669)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2gRR9HOBxOR2"
      },
      "source": [
        "## Loading the Data\n",
        "\n",
        "The dataset is included in the CityLearn library installation which we will now read into memory. To read the dataset, all we need is the `schema.json` that defines it:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9SuxbmkixQ2z"
      },
      "outputs": [],
      "source": [
        "DATASET_NAME = 'citylearn_challenge_2022_phase_all'\n",
        "schema = DataSet.get_schema(DATASET_NAME)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cb-xiU9qxUUl"
      },
      "source": [
        "> ⚠️ **NOTE**: To get the names of all datasets in CityLearn execute the `citylearn.data.Dataset.get_names` method:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "p8W88P96xXiR"
      },
      "outputs": [],
      "source": [
        "print('All CityLearn datasets:', sorted(DataSet.get_names()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "41IxMp9rxb5W"
      },
      "source": [
        "### Preview a Building Data File\n",
        "We can now preview the data files for one of the buildings in the `citylearn_challenge_2022_phase_all` dataset. The schema includes a `root_directory` key-value where all files that are relevant to this dataset are stored as well as the name of each file. We will use this root directory and filenames to read a building file:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_0LorchUxg8v"
      },
      "outputs": [],
      "source": [
        "root_directory = schema['root_directory']\n",
        "\n",
        "# change the suffix number in the next code line to a\n",
        "# number between 1 and 17 to preview other buildings\n",
        "building_name = 'Building_1'\n",
        "\n",
        "filename = schema['buildings'][building_name]['energy_simulation']\n",
        "filepath = os.path.join(root_directory, filename)\n",
        "building_data = pd.read_csv(filepath)\n",
        "display(building_data.head())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nWgMhWFjxl-z"
      },
      "source": [
        "This building file has 12 fields describing the hourly building loads and indoor environmental conditions. Descriptive statistics of these fields are reported below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WMq9RpWUxoq-"
      },
      "outputs": [],
      "source": [
        "display(building_data.describe(include='all'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LtCpbXiNxvM_"
      },
      "source": [
        "The `Month`, `Hour`, `Day Type` and `Daylight Savings Status` define the temporal dimension of the building loads. `Indoor Temperature [C]`, `Average Unmet Cooling Setpoint Difference [C]` and `Indoor Relative Humidity [%]` are null values in the entire time series since they are not provided in the original dataset from the real-world building. However, they are included in the file to maintain compatibility with the methods used to construct a CityLearn environment. For the same reason, `DHW Heating [kWh]`, `Cooling Load [kWh]` and `Heating Load [kWh]` have zero values throughout the time series as they have coupled with the non-shiftable `Equipment Electric Power [kWh]`.\n",
        "\n",
        "Thus, the goal with this dataset is to learn to use battery-PV system to satisfy these non-shiftable loads. The `Equipment Electric Power [kWh]` and `Solar Generation [W/kW]` time series for the building are plotted below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WieiE91Fx0xr"
      },
      "outputs": [],
      "source": [
        "fig, axs = plt.subplots(1, 2, figsize=(18, 2))\n",
        "x = building_data.index\n",
        "y1 = building_data['Equipment Electric Power [kWh]']\n",
        "y2 = building_data['Solar Generation [W/kW]']\n",
        "axs[0].plot(x, y1)\n",
        "axs[0].set_xlabel('Time step')\n",
        "axs[0].set_ylabel('Equipment Electric Power\\n[kWh]')\n",
        "axs[1].plot(x, y2)\n",
        "axs[1].set_xlabel('Time step')\n",
        "axs[1].set_ylabel('Solar Generation\\n[W/kW]')\n",
        "fig.suptitle(building_name)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kChHIRCcx7HS"
      },
      "source": [
        "### Preview Weather File\n",
        "\n",
        "Other supplemental data in the dataset include [TMY3 weather data from the Los Angeles International Airport weather station](https://energyplus.net/weather-location/north_and_central_america_wmo_region_4/USA/CA/USA_CA_Los.Angeles.Intl.AP.722950_TMY3) that is representative of a typical meteorological year in the Los Angeles International Airport location. All buildings in this dataset reference the same weather file as previewed:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "x7hovT6HyAfb"
      },
      "outputs": [],
      "source": [
        "filename = schema['buildings'][building_name]['weather']\n",
        "filepath = os.path.join(root_directory, filename)\n",
        "weather_data = pd.read_csv(filepath)\n",
        "display(weather_data.head())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vrVYBj_EyFwE"
      },
      "source": [
        "The weather file has fields that describe the outdoor dry-bulb temperature, relative humidity, diffuse and direct solar radiation, as well as their 6 hour, 12 hour and 24 hour forecasts. In this dataset, the forecasts are perfect forecasts for example, the 6 hour outdoor dry-bulb temperature forecast at a certain time step is equal to the temperature 6 hours later. The summary statistics for the weather fields are provided below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OSuE0q0ayJ9l"
      },
      "outputs": [],
      "source": [
        "display(weather_data.describe(include='all'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jIST0gHIyM90"
      },
      "source": [
        "We can also plot this weather data on an axes to understand it better:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mwpuHpIUyPpk"
      },
      "outputs": [],
      "source": [
        "columns = [\n",
        "    'Outdoor Drybulb Temperature [C]', 'Relative Humidity [%]',\n",
        "    'Diffuse Solar Radiation [W/m2]', 'Direct Solar Radiation [W/m2]'\n",
        "]\n",
        "fig, axs = plt.subplots(2, 2, figsize=(18, 5))\n",
        "x = weather_data.index\n",
        "\n",
        "for ax, c in zip(fig.axes, columns):\n",
        "    y = weather_data[c]\n",
        "    ax.plot(x, y)\n",
        "    ax.set_xlabel('Time step')\n",
        "    ax.set_ylabel(c)\n",
        "\n",
        "fig.align_ylabels()\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N3deLJEwyWNy"
      },
      "source": [
        "### Preview Electricity Price Data\n",
        "\n",
        "The electricity rate-plan for the dataset is that of the community's utility provider, [Southern California Edison](https://www.sce.com/residential/rates/Time-Of-Use-Residential-Rate-Plans). We adopt their _TOU-D-PRIME_ rate plan summarized in table below, which is designed for customers with residential batteries where electricity is cheapest in the early morning and late at night, and cheaper during off-peak months of October-May. Meanwhile, electricity is cheaper on weekends for peak hours of 4 PM-9 PM in June-September.\n",
        "\n",
        "Table: Time-Of-Use rate plan ($/kWh).\n",
        "\n",
        "| | June-September |  | October-May |  |\n",
        "|---|---|---|---|---|\n",
        "| **Time** | **Weekday** | **Weekend** | **Weekday** | **Weekend** |\n",
        "| 8 AM-4 PM | 0.21 | 0.21 | 0.20 | 0.20 |\n",
        "| 4 PM-9 PM | 0.54 | 0.40 | 0.50 | 0.50 |\n",
        "| 9 PM-8 AM | 0.21 | 0.21 | 0.20 | 0.20 |\n",
        "\n",
        "The electricity pricing time series is shown below. It has four fields including perfect forecast of the pricing 6, 12 and 24 hours ahead."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Hv2Bc6Qoyr2X"
      },
      "outputs": [],
      "source": [
        "filename = schema['buildings'][building_name]['pricing']\n",
        "filepath = os.path.join(root_directory, filename)\n",
        "pricing_data = pd.read_csv(filepath)\n",
        "display(pricing_data.head())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NKq_lE1MyuTB"
      },
      "source": [
        "### Preview Carbon Intensity Data\n",
        "\n",
        "Another supplementary data in the `citylearn_challenge_2022_phase_all` dataset is the grid carbon intensity time series descring the CO<sub>2</sub> equivalent of greenhouse gases that are emitted for every unit kWh of energy consumption. This carbon intensity data were provided by EPRI and the time series is shown with its summary statistics below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mXMSuDcOyv_6"
      },
      "outputs": [],
      "source": [
        "filename = schema['buildings'][building_name]['carbon_intensity']\n",
        "filepath = os.path.join(root_directory, filename)\n",
        "carbon_intensity_data = pd.read_csv(filepath)\n",
        "display(carbon_intensity_data.head())\n",
        "display(carbon_intensity_data.describe(include='all').T)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dfFUem5My_QN"
      },
      "source": [
        "We also preview the carbon intensity time series on a pair of axes:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Q771yVRfzAXA"
      },
      "outputs": [],
      "source": [
        "fig, ax = plt.subplots(1, 1, figsize=(9, 2))\n",
        "x = carbon_intensity_data.index\n",
        "y = carbon_intensity_data['kg_CO2/kWh']\n",
        "ax.plot(x, y)\n",
        "ax.set_xlabel('Time step')\n",
        "ax.set_ylabel('kg_CO2/kWh')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PoF-BxSM5Jkc"
      },
      "source": [
        "## Data Preprocessing\n",
        "\n",
        "Now that we are familiar with the CityLearn `citylearn_challenge_2022_phase_all` dataset, we will make minor changes to its schema that will improve our learning experience in this tutorial. These changes are as follows:\n",
        "\n",
        "1. We want to use a subset of the buildings so that we are not overwhelmed by the amount of data to analyze during the tutorial 🙂. Since CityLearn is primarily designed for district level energy management and coordination we should use more than 1 building, although a 1-building environment is possible. An considerable building count for tutoring purposes is 2-3.\n",
        "2. We want to use only a one-week period from the entire one-year period for this tutorial for the same reason of ease of analysis.\n",
        "3. Instead of using the [full observation space](https://www.citylearn.net/overview/observations.html) that will take a while to explore and converge in RL implementations, we will narrow down the space to only one observations: `hour`. This is not the best set up because the hour alone does not explain the state transitions in the environment that the agent is observing, nevertheless, it will help highlight the strengths and weaknesses of different control algorithms.\n",
        "4. CityLearn allows for two control strategies: centralized and decentralized as earlier discussed. In this tutorial we will make use of the former.\n",
        "\n",
        "We will make these modifications directly in the schema. To keep things interesting, the buildings and one-week period will be pseudo-randomly selected but for reproducibility, we will set the random generator seed. This seed can be changed to any value to select another pseudorandom set of buildings and time period. Also, we will provide a method to set the observations we want to use in our simulations so that later down the line, it will be easy to switch and utilize other observations. We will define three functions to help us make these decisions:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2o6iEzE_zP_U"
      },
      "outputs": [],
      "source": [
        "def set_schema_buildings(\n",
        "schema: dict, count: int, seed: int\n",
        ") -> Tuple[dict, List[str]]:\n",
        "    \"\"\"Randomly select number of buildings to set as active in the schema.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    schema: dict\n",
        "        CityLearn dataset mapping used to construct environment.\n",
        "    count: int\n",
        "        Number of buildings to set as active in schema.\n",
        "    seed: int\n",
        "        Seed for pseudo-random number generator\n",
        "\n",
        "    Returns\n",
        "    -------\n",
        "    schema: dict\n",
        "        CityLearn dataset mapping with active buildings set.\n",
        "    buildings: List[str]\n",
        "        List of selected buildings.\n",
        "    \"\"\"\n",
        "\n",
        "    assert 1 <= count <= 15, 'count must be between 1 and 15.'\n",
        "\n",
        "    # set random seed\n",
        "    np.random.seed(seed)\n",
        "\n",
        "    # get all building names\n",
        "    buildings = list(schema['buildings'].keys())\n",
        "\n",
        "    # remove buildins 12 and 15 as they have pecularities in their data\n",
        "    # that are not relevant to this tutorial\n",
        "    buildings_to_exclude = ['Building_12', 'Building_15']\n",
        "\n",
        "    for b in buildings_to_exclude:\n",
        "        buildings.remove(b)\n",
        "\n",
        "    # randomly select specified number of buildings\n",
        "    buildings = np.random.choice(buildings, size=count, replace=False).tolist()\n",
        "\n",
        "    # reorder buildings\n",
        "    building_ids = [int(b.split('_')[-1]) for b in buildings]\n",
        "    building_ids = sorted(building_ids)\n",
        "    buildings = [f'Building_{i}' for i in building_ids]\n",
        "\n",
        "    # update schema to only included selected buildings\n",
        "    for b in schema['buildings']:\n",
        "        if b in buildings:\n",
        "            schema['buildings'][b]['include'] = True\n",
        "        else:\n",
        "            schema['buildings'][b]['include'] = False\n",
        "\n",
        "    return schema, buildings"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TcfYt2eH0sP8"
      },
      "outputs": [],
      "source": [
        "def set_schema_simulation_period(\n",
        "    schema: dict, count: int, seed: int\n",
        ") -> Tuple[dict, int, int]:\n",
        "    \"\"\"Randomly select environment simulation start and end time steps\n",
        "    that cover a specified number of days.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    schema: dict\n",
        "        CityLearn dataset mapping used to construct environment.\n",
        "    count: int\n",
        "        Number of simulation days.\n",
        "    seed: int\n",
        "        Seed for pseudo-random number generator.\n",
        "\n",
        "    Returns\n",
        "    -------\n",
        "    schema: dict\n",
        "        CityLearn dataset mapping with `simulation_start_time_step`\n",
        "        and `simulation_end_time_step` key-values set.\n",
        "    simulation_start_time_step: int\n",
        "        The first time step in schema time series files to\n",
        "        be read when constructing the environment.\n",
        "    simulation_end_time_step: int\n",
        "        The last time step in schema time series files to\n",
        "        be read when constructing the environment.\n",
        "    \"\"\"\n",
        "\n",
        "    assert 1 <= count <= 365, 'count must be between 1 and 365.'\n",
        "\n",
        "    # set random seed\n",
        "    np.random.seed(seed)\n",
        "\n",
        "    # use any of the files to determine the total\n",
        "    # number of available time steps\n",
        "    filename = schema['buildings'][building_name]['carbon_intensity']\n",
        "    filepath = os.path.join(root_directory, filename)\n",
        "    time_steps = pd.read_csv(filepath).shape[0]\n",
        "\n",
        "    # set candidate simulation start time steps\n",
        "    # spaced by the number of specified days\n",
        "    simulation_start_time_step_list = np.arange(0, time_steps, 24*count)\n",
        "\n",
        "    # randomly select a simulation start time step\n",
        "    simulation_start_time_step = np.random.choice(\n",
        "        simulation_start_time_step_list, size=1\n",
        "    )[0]\n",
        "    simulation_end_time_step = simulation_start_time_step + 24*count - 1\n",
        "\n",
        "    # update schema simulation time steps\n",
        "    schema['simulation_start_time_step'] = simulation_start_time_step\n",
        "    schema['simulation_end_time_step'] = simulation_end_time_step\n",
        "\n",
        "    return schema, simulation_start_time_step, simulation_end_time_step"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "A04LckaH0uuS"
      },
      "outputs": [],
      "source": [
        "def set_active_observations(\n",
        "    schema: dict, active_observations: List[str]\n",
        ") -> dict:\n",
        "    \"\"\"Set the observations that will be part of the environment's\n",
        "    observation space that is provided to the control agent.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    schema: dict\n",
        "        CityLearn dataset mapping used to construct environment.\n",
        "    active_observations: List[str]\n",
        "        Names of observations to set active to be passed to control agent.\n",
        "\n",
        "    Returns\n",
        "    -------\n",
        "    schema: dict\n",
        "        CityLearn dataset mapping with active observations set.\n",
        "    \"\"\"\n",
        "\n",
        "    active_count = 0\n",
        "\n",
        "    for o in schema['observations']:\n",
        "        if o in active_observations:\n",
        "            schema['observations'][o]['active'] = True\n",
        "            active_count += 1\n",
        "        else:\n",
        "            schema['observations'][o]['active'] = False\n",
        "\n",
        "    valid_observations = list(schema['observations'].keys())\n",
        "    assert active_count == len(active_observations),\\\n",
        "        'the provided observations are not all valid observations.'\\\n",
        "          f' Valid observations in CityLearn are: {valid_observations}'\n",
        "\n",
        "    return schema"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "te2L4gwOzUQo"
      },
      "source": [
        "### Setting your Random Seed\n",
        "\n",
        "Begin by setting a random seed. You can set the seed to any integer including your birth day, month or year. Perhaps lucky number 😁. Choose wisely because we will use this random seed moving forward 😉?!?):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vfnO0QBszXcS"
      },
      "outputs": [],
      "source": [
        "RANDOM_SEED = 0\n",
        "print('Random seed:', RANDOM_SEED)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TX5rxfNdz3lE"
      },
      "source": [
        "### Setting the Buildings, Time Periods and Observations to use in Simulations from the Schema\n",
        "\n",
        "Now we can pseudo-randomly select buildings and time periods as well as set the active observations we will use:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6C6S46xmz50t"
      },
      "outputs": [],
      "source": [
        "# edit next code line to change number of buildings in simulation\n",
        "BUILDING_COUNT = 2\n",
        "\n",
        " # edit next code line to change number of days in simulation\n",
        "DAY_COUNT = 7\n",
        "\n",
        "# edit next code line to change active observations in simulation\n",
        "ACTIVE_OBSERVATIONS = ['hour']\n",
        "\n",
        "schema, buildings = set_schema_buildings(schema, BUILDING_COUNT, RANDOM_SEED)\n",
        "schema, simulation_start_time_step, simulation_end_time_step =\\\n",
        "    set_schema_simulation_period(schema, DAY_COUNT, RANDOM_SEED)\n",
        "schema = set_active_observations(schema, ACTIVE_OBSERVATIONS)\n",
        "\n",
        "print('Selected buildings:', buildings)\n",
        "print(\n",
        "    f'Selected {DAY_COUNT}-day period time steps:',\n",
        "    (simulation_start_time_step, simulation_end_time_step)\n",
        ")\n",
        "print(f'Active observations:', ACTIVE_OBSERVATIONS)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6xL8ayLYz90X"
      },
      "source": [
        "Lastly, the choice between either control strategy is set using the `central_agent` parameter in CityLearn, which is a key-value in the `schema`. We set the `central_agent` key-value in the schema to `True` to define an environment that uses one agent to control many buildings (centralized control strategy):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Qh5FKi6Nopbr"
      },
      "outputs": [],
      "source": [
        "schema['central_agent'] = True"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bSt6h_Q-oqjK"
      },
      "source": [
        "# Initialize a CityLearn Environment\n",
        "***\n",
        "\n",
        "We will now initialize an example instance of the CityLearn environment that we will use in this tutorial. To initialize an environment, all that needs to be done is call the `citylearn.citylearn.CityLearnEnv.__init__` method and parse the `schema` to it:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0aBJ5aLZosk-"
      },
      "outputs": [],
      "source": [
        "env = CityLearnEnv(schema)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Co9Zh6dk0UIt"
      },
      "source": [
        "The `env` object has a number of properties and methods that can be learned about in the [docs](https://www.citylearn.net/api/citylearn.citylearn.html#citylearn.citylearn.CityLearnEnv). We will interact with some of its feature to learn about the current state of the environment:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QYT8ouXg0WeH"
      },
      "outputs": [],
      "source": [
        "print('Current time step:', env.time_step)\n",
        "print('environment number of time steps:', env.time_steps)\n",
        "print('environment uses central agent:', env.central_agent)\n",
        "print('Common (shared) observations amogst buildings:', env.shared_observations)\n",
        "print('Number of buildings:', len(env.buildings))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pxErndp-0at7"
      },
      "source": [
        "The buildings in the environment are objects of the `citylearn.building.Building` class and the class properties and methods are detailed in the [docs](https://www.citylearn.net/api/citylearn.building.html#citylearn.building.Building). We will interact with some of these features:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gT64404w0dlu"
      },
      "outputs": [],
      "source": [
        "# electrical storage\n",
        "print('Electrical storage capacity:', {\n",
        "    b.name: b.electrical_storage.capacity for b in env.buildings\n",
        "})\n",
        "print('Electrical storage nominal power:', {\n",
        "    b.name: b.electrical_storage.nominal_power for b in env.buildings\n",
        "})\n",
        "print('Electrical storage capacity history:', {\n",
        "    b.name: b.electrical_storage.capacity_history for b in env.buildings\n",
        "})\n",
        "print('Electrical storage loss_coefficient:', {\n",
        "    b.name: b.electrical_storage.loss_coefficient for b in env.buildings\n",
        "})\n",
        "print('Electrical storage initial_soc:', {\n",
        "    b.name: b.electrical_storage.initial_soc for b in env.buildings\n",
        "})\n",
        "print('Electrical storage soc:', {\n",
        "    b.name: b.electrical_storage.soc for b in env.buildings\n",
        "})\n",
        "print('Electrical storage efficiency:', {\n",
        "    b.name: b.electrical_storage.efficiency for b in env.buildings\n",
        "})\n",
        "print('Electrical storage efficiency history:', {\n",
        "    b.name: b.electrical_storage.efficiency_history for b in env.buildings\n",
        "})\n",
        "print('Electrical storage electricity consumption:', {\n",
        "    b.name: b.electrical_storage.electricity_consumption\n",
        "    for b in env.buildings\n",
        "})\n",
        "print('Electrical storage capacity loss coefficient:', {\n",
        "    b.name: b.electrical_storage.loss_coefficient for b in env.buildings\n",
        "})\n",
        "print()\n",
        "# pv\n",
        "print('PV nominal power:', {\n",
        "    b.name: b.pv.nominal_power for b in env.buildings\n",
        "})\n",
        "print()\n",
        "\n",
        "# active observations and actions\n",
        "with pd.option_context(\n",
        "    'display.max_rows', None,\n",
        "    'display.max_columns', None,\n",
        "    'display.width', None\n",
        "):\n",
        "    print('Active observations:')\n",
        "    display(pd.DataFrame([\n",
        "        {**{'building':b.name}, **b.observation_metadata}\n",
        "        for b in env.buildings\n",
        "    ]))\n",
        "    print()\n",
        "    print('Active actions:')\n",
        "    display(pd.DataFrame([\n",
        "        {**{'building':b.name}, **b.action_metadata}\n",
        "        for b in env.buildings\n",
        "    ]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b2jH6nki0hpG"
      },
      "source": [
        "# Key Performance Indicators for Evaluation\n",
        "---\n",
        "\n",
        "We evaluate the control agents' performance using six key performance indicators (KPIs) that are to be minimized: electricity consumption, cost, carbon emissions, average daily peak, ramping, and (1 - load factor). average daily peak, ramping and (1 - load factor) are district-level KPIs that are calculated using the aggregated district-level hourly net electricity consumption (kWh), $E_h^{\\textrm{district}}$. electricity consumption, cost, and carbon emissions are building-level KPIs that are calculated using the building-level hourly net electricity consumption (kWh), $E_h^{\\textrm{building}}$, and are reported at the grid level as the average of the building-level values.\n",
        "\n",
        "Electricity consumption is defined as the sum of imported electricity $E_h^{\\textrm{building}}$ as the objective is to minimize the energy consumed but not profit from the excess generation.\n",
        "\n",
        "$$\n",
        "    \\textrm{electricity consumption} = \\sum_{h=0}^{n-1}{\\textrm{max} \\left (0,E_h^{\\textrm{building}} \\right)}\n",
        "$$\n",
        "\n",
        "Cost is defined as the sum of building-level imported electricity cost, $E_h^{\\textrm{building}} \\times T_h$ (\\$), where $T_h$ is the electricity rate at hour $h$.\n",
        "\n",
        "$$\n",
        "    \\textrm{cost} = \\sum_{h=0}^{n-1}{\\textrm{max} \\left (0,E_h^{\\textrm{building}} \\times T_h \\right )}\n",
        "$$\n",
        "\n",
        "Carbon emissions is the sum of building-level carbon emissions (kg<sub>CO<sub>2</sub>e</sub>), $E_h^{\\textrm{building}} \\times O_h$, where $O_h$ is the carbon intensity (kg<sub>CO<sub>2</sub>e</sub>/kWh) at hour $h$.\n",
        "\n",
        "$$\n",
        "    \\textrm{carbon emissions} = \\sum_{h=0}^{n-1}{\\textrm{max} \\left (0,E_h^{\\textrm{building}} \\times O_h \\right )}\n",
        "$$\n",
        "\n",
        "Average daily peak, is defined as the mean of the daily $E_h^{\\textrm{district}}$ peak where $d$ is the day index and $n$ is the total number of days.\n",
        "\n",
        "$$\n",
        "    \\textrm{average daily peak} = \\frac{\n",
        "        {\\sum}_{d=0}^{n - 1} {\\sum}_{h=0}^{23} {\\textrm{max} \\left (E_{24d + h}^{\\textrm{district}}, \\dots, E_{24d + 23}^{\\textrm{district}} \\right)}\n",
        "    }{n}\n",
        "$$\n",
        "\n",
        "Ramping is defined as the absolute difference of consecutive $E_h^{\\textrm{district}}$. It represents the smoothness of the district’s load profile where low ramping means there is gradual increase in grid load even after self-generation becomes unavailable in the evening and early morning. High ramping means abrupt change in grid load that may lead to unscheduled strain on grid infrastructure and blackouts as a result of supply deficit.\n",
        "\n",
        "$$\n",
        "    \\textrm{ramping} = \\sum_{h=0}^{n-1}  \\lvert E_{h}^{\\textrm{district}} - E_{h - 1}^{\\textrm{district}} \\rvert\n",
        "$$\n",
        "\n",
        "Load factor is defined as the average ratio of monthly average and peak $E_{h}^{\\textrm{district}}$ where $m$ is the month index, $d$ is the number of days in a month and $n$ is the number of months. load factor represents the efficiency of electricity consumption and is bounded between 0 (very inefficient) and 1 (highly efficient) thus, the goal is to maximize the load factor or in the same fashion as the other KPIs, minimize (1 - load factor).\n",
        "\n",
        "$$\n",
        "    \\textrm{1 - load factor}  = \\Big(\n",
        "        \\sum_{m=0}^{n - 1} 1 - \\frac{\n",
        "            \\left (\n",
        "                \\sum_{h=0}^{d - 1} E_{d \\cdot m + h}^{\\textrm{district}}\n",
        "            \\right ) \\div d\n",
        "        }{\n",
        "            \\textrm{max} \\left (E_{d \\cdot m}^{\\textrm{district}}, \\dots, E_{d \\cdot m + d - 1}^{\\textrm{district}} \\right )\n",
        "    }\\Big) \\div n\n",
        "$$\n",
        "\n",
        "For the remainder of the paper, the KPIs are reported as normalized values with respect to the baseline outcome where the baseline outcome is when buildings are not equipped with batteries i.e., no control.\n",
        "\n",
        "$$\n",
        "    \\textrm{KPI} = \\frac{{\\textrm{KPI}_{control}}}{\\textrm{KPI}_{baseline (no\\ battery)}}\n",
        "$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qa9Iuu2GU52Z"
      },
      "source": [
        "# Convenience Functions to Display Simulation Results\n",
        "---\n",
        "\n",
        "CityLearn itself is able to report the key performance indicators (KPIs) during simulation using the `citylearn.citylearn.CityLearnEnv.evaluate` (see [docs](https://www.citylearn.net/api/citylearn.citylearn.html#citylearn.citylearn.CityLearnEnv.evaluate)) method however, let us go ahead and define some convenience functions to help us report the KPIs illustratively. The first function helps us calculate and return the KPIs in a table:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aG2SsVYq0sye"
      },
      "outputs": [],
      "source": [
        "def get_kpis(env: CityLearnEnv) -> pd.DataFrame:\n",
        "    \"\"\"Returns evaluation KPIs.\n",
        "\n",
        "    Electricity consumption, cost and carbon emissions KPIs are provided\n",
        "    at the building-level and average district-level. Average daily peak,\n",
        "    ramping and (1 - load factor) KPIs are provided at the district level.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    env: CityLearnEnv\n",
        "        CityLearn environment instance.\n",
        "\n",
        "    Returns\n",
        "    -------\n",
        "    kpis: pd.DataFrame\n",
        "        KPI table.\n",
        "    \"\"\"\n",
        "\n",
        "    kpis = env.evaluate()\n",
        "\n",
        "    # names of KPIs to retrieve from evaluate function\n",
        "    kpi_names = [\n",
        "        'electricity_consumption', 'cost', 'carbon_emissions',\n",
        "        'average_daily_peak', 'ramping', '1 - load_factor'\n",
        "    ]\n",
        "    kpis = kpis[\n",
        "        (kpis['cost_function'].isin(kpi_names))\n",
        "    ].dropna()\n",
        "\n",
        "    # round up the values to 3 decimal places for readability\n",
        "    kpis['value'] = kpis['value'].round(3)\n",
        "\n",
        "    # rename the column that defines the KPIs\n",
        "    kpis = kpis.rename(columns={'cost_function': 'kpi'})\n",
        "\n",
        "    return kpis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RBCwwMKb0uLC"
      },
      "source": [
        "> ⚠️ **NOTE**:\n",
        "> You do not need to understand the content of the next code cell where a plotting function is defined.\n",
        "\n",
        "The next function, `plot_building_kpis` plots the KPIs at the building level in bar charts and can be used to compare different control strategies by providing it with a dictionary that maps a control agent name to the environment the agent acted on:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TVPVuSdL0xOu"
      },
      "outputs": [],
      "source": [
        "def plot_building_kpis(envs: Mapping[str, CityLearnEnv]) -> plt.Figure:\n",
        "    \"\"\"Plots electricity consumption, cost and carbon emissions\n",
        "    at the building-level for different control agents in bar charts.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    envs: Mapping[str, CityLearnEnv]\n",
        "        Mapping of user-defined control agent names to environments\n",
        "        the agents have been used to control.\n",
        "\n",
        "    Returns\n",
        "    -------\n",
        "    fig: plt.Figure\n",
        "        Figure containing plotted axes.\n",
        "    \"\"\"\n",
        "\n",
        "    kpis_list = []\n",
        "\n",
        "    for k, v in envs.items():\n",
        "        kpis = get_kpis(v)\n",
        "        kpis = kpis[kpis['level']=='building'].copy()\n",
        "        kpis['building_id'] = kpis['name'].str.split('_', expand=True)[1]\n",
        "        kpis['building_id'] = kpis['building_id'].astype(int).astype(str)\n",
        "        kpis['env_id'] = k\n",
        "        kpis_list.append(kpis)\n",
        "\n",
        "    kpis = pd.concat(kpis_list, ignore_index=True, sort=False)\n",
        "    kpi_names= kpis['kpi'].unique()\n",
        "    column_count_limit = 3\n",
        "    row_count = math.ceil(len(kpi_names)/column_count_limit)\n",
        "    column_count = min(column_count_limit, len(kpi_names))\n",
        "    building_count = len(kpis['name'].unique())\n",
        "    env_count = len(envs)\n",
        "    figsize = (3.0*column_count, 0.3*env_count*building_count*row_count)\n",
        "    fig, _ = plt.subplots(\n",
        "        row_count, column_count, figsize=figsize, sharey=True\n",
        "    )\n",
        "\n",
        "    for i, (ax, (k, k_data)) in enumerate(zip(fig.axes, kpis.groupby('kpi'))):\n",
        "        sns.barplot(x='value', y='name', data=k_data, hue='env_id', ax=ax)\n",
        "        ax.axvline(1.0, color='black', linestyle='--', label='Baseline')\n",
        "        ax.set_xlabel(None)\n",
        "        ax.set_ylabel(None)\n",
        "        ax.set_title(k)\n",
        "\n",
        "        if i == len(kpi_names) - 1:\n",
        "            ax.legend(\n",
        "                loc='upper left', bbox_to_anchor=(1.3, 1.0), framealpha=0.0\n",
        "            )\n",
        "        else:\n",
        "            ax.legend().set_visible(False)\n",
        "\n",
        "        for s in ['right','top']:\n",
        "            ax.spines[s].set_visible(False)\n",
        "\n",
        "        for p in ax.patches:\n",
        "            ax.text(\n",
        "                p.get_x() + p.get_width(),\n",
        "                p.get_y() + p.get_height()/2.0,\n",
        "                p.get_width(), ha='left', va='center'\n",
        "            )\n",
        "\n",
        "    plt.tight_layout()\n",
        "    return fig"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "slUb1kOY0zjI"
      },
      "source": [
        "> ⚠️ **NOTE**:\n",
        "> You do not need to understand the content of the next code cell where a plotting function is defined.\n",
        "\n",
        "The `plot_district_kpis` function plots the KPIs at the district level in a bar chart and can be used to compare different control agents:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wXlXXqtI04Cm"
      },
      "outputs": [],
      "source": [
        "def plot_district_kpis(envs: Mapping[str, CityLearnEnv]) -> plt.Figure:\n",
        "    \"\"\"Plots electricity consumption, cost, carbon emissions,\n",
        "    average daily peak, ramping and (1 - load factor) at the\n",
        "    district-level for different control agents in a bar chart.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    envs: Mapping[str, CityLearnEnv]\n",
        "        Mapping of user-defined control agent names to environments\n",
        "        the agents have been used to control.\n",
        "\n",
        "    Returns\n",
        "    -------\n",
        "    fig: plt.Figure\n",
        "        Figure containing plotted axes.\n",
        "    \"\"\"\n",
        "\n",
        "    kpis_list = []\n",
        "\n",
        "    for k, v in envs.items():\n",
        "        kpis = get_kpis(v)\n",
        "        kpis = kpis[kpis['level']=='district'].copy()\n",
        "        kpis['env_id'] = k\n",
        "        kpis_list.append(kpis)\n",
        "\n",
        "    kpis = pd.concat(kpis_list, ignore_index=True, sort=False)\n",
        "    row_count = 1\n",
        "    column_count = 1\n",
        "    env_count = len(envs)\n",
        "    kpi_count = len(kpis['kpi'].unique())\n",
        "    figsize = (6.0*column_count, 0.225*env_count*kpi_count*row_count)\n",
        "    fig, ax = plt.subplots(row_count, column_count, figsize=figsize)\n",
        "    sns.barplot(x='value', y='kpi', data=kpis, hue='env_id', ax=ax)\n",
        "    ax.axvline(1.0, color='black', linestyle='--', label='Baseline')\n",
        "    ax.set_xlabel(None)\n",
        "    ax.set_ylabel(None)\n",
        "\n",
        "    for s in ['right','top']:\n",
        "        ax.spines[s].set_visible(False)\n",
        "\n",
        "    for p in ax.patches:\n",
        "        ax.text(\n",
        "            p.get_x() + p.get_width(),\n",
        "            p.get_y() + p.get_height()/2.0,\n",
        "            p.get_width(), ha='left', va='center'\n",
        "        )\n",
        "\n",
        "    ax.legend(loc='upper left', bbox_to_anchor=(1.3, 1.0), framealpha=0.0)\n",
        "    plt.tight_layout()\n",
        "\n",
        "    return fig"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xu8kBq5i06QO"
      },
      "source": [
        "> ⚠️ **NOTE**:\n",
        "> You do not need to understand the content of the next code cell where a plotting function is defined.\n",
        "\n",
        "The `plot_building_load_profiles` function plots the building-level net electricity consumption profiles for the baseline (no battery) and control scenario with battery. It can also be used to compare different control agents:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "roq__q_M0_Yx"
      },
      "outputs": [],
      "source": [
        "def plot_building_load_profiles(envs: Mapping[str, CityLearnEnv]) -> plt.Figure:\n",
        "    \"\"\"Plots building-level net electricty consumption profile\n",
        "    for different control agents.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    envs: Mapping[str, CityLearnEnv]\n",
        "        Mapping of user-defined control agent names to environments\n",
        "        the agents have been used to control.\n",
        "\n",
        "    Returns\n",
        "    -------\n",
        "    fig: plt.Figure\n",
        "        Figure containing plotted axes.\n",
        "    \"\"\"\n",
        "\n",
        "    building_count = len(list(envs.values())[0].buildings)\n",
        "    column_count_limit = 4\n",
        "    row_count = math.ceil(building_count/column_count_limit)\n",
        "    column_count = min(column_count_limit, building_count)\n",
        "    figsize = (4.0*column_count, 1.75*row_count)\n",
        "    fig, _ = plt.subplots(row_count, column_count, figsize=figsize)\n",
        "\n",
        "    for i, ax in enumerate(fig.axes):\n",
        "        for k, v in envs.items():\n",
        "            y = v.buildings[i].net_electricity_consumption\n",
        "            x = range(len(y))\n",
        "            ax.plot(x, y, label=k)\n",
        "\n",
        "        y = v.buildings[i].net_electricity_consumption_without_storage\n",
        "        ax.plot(x, y, label='Baseline')\n",
        "        ax.set_title(v.buildings[i].name)\n",
        "        ax.set_xlabel('Time step')\n",
        "        ax.set_ylabel('kWh')\n",
        "        ax.xaxis.set_major_locator(ticker.MultipleLocator(24))\n",
        "\n",
        "        if i == building_count - 1:\n",
        "            ax.legend(\n",
        "                loc='upper left', bbox_to_anchor=(1.0, 1.0), framealpha=0.0\n",
        "            )\n",
        "        else:\n",
        "            ax.legend().set_visible(False)\n",
        "\n",
        "\n",
        "    plt.tight_layout()\n",
        "\n",
        "    return fig"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kRPhywwC1ElB"
      },
      "source": [
        "> ⚠️ **NOTE**:\n",
        "> You do not need to understand the content of the next code cell where a plotting function is defined.\n",
        "\n",
        "The `plot_district_load_profiles` function plots the district-level net electricity consumption profiles for the baseline (no battery) and control scenario with battery. It can also be used to compare different control agents."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3dA40P1O1Ho3"
      },
      "outputs": [],
      "source": [
        "def plot_district_load_profiles(envs: Mapping[str, CityLearnEnv]) -> plt.Figure:\n",
        "    \"\"\"Plots district-level net electricty consumption profile\n",
        "    for different control agents.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    envs: Mapping[str, CityLearnEnv]\n",
        "        Mapping of user-defined control agent names to environments\n",
        "        the agents have been used to control.\n",
        "\n",
        "    Returns\n",
        "    -------\n",
        "    fig: plt.Figure\n",
        "        Figure containing plotted axes.\n",
        "    \"\"\"\n",
        "\n",
        "    figsize = (5.0, 1.5)\n",
        "    fig, ax = plt.subplots(1, 1, figsize=figsize)\n",
        "\n",
        "    for k, v in envs.items():\n",
        "        y = v.net_electricity_consumption\n",
        "        x = range(len(y))\n",
        "        ax.plot(x, y, label=k)\n",
        "\n",
        "    y = v.net_electricity_consumption_without_storage\n",
        "    ax.plot(x, y, label='Baseline')\n",
        "    ax.set_xlabel('Time step')\n",
        "    ax.set_ylabel('kWh')\n",
        "    ax.xaxis.set_major_locator(ticker.MultipleLocator(24))\n",
        "    ax.legend(loc='upper left', bbox_to_anchor=(1.0, 1.0), framealpha=0.0)\n",
        "\n",
        "    plt.tight_layout()\n",
        "    return fig"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "po_TEviJ1MWQ"
      },
      "source": [
        "> ⚠️ **NOTE**:\n",
        "> You do not need to understand the content of the next code cell where a plotting function is defined.\n",
        "\n",
        "The `plot_battery_soc_profiles` function plots the building-level battery state of charge (SoC) profiles can also be used to compare different control agents:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "unYJBsZB1N-X"
      },
      "outputs": [],
      "source": [
        "def plot_battery_soc_profiles(envs: Mapping[str, CityLearnEnv]) -> plt.Figure:\n",
        "    \"\"\"Plots building-level battery SoC profiles fro different control agents.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    envs: Mapping[str, CityLearnEnv]\n",
        "        Mapping of user-defined control agent names to environments\n",
        "        the agents have been used to control.\n",
        "\n",
        "    Returns\n",
        "    -------\n",
        "    fig: plt.Figure\n",
        "        Figure containing plotted axes.\n",
        "    \"\"\"\n",
        "\n",
        "    building_count = len(list(envs.values())[0].buildings)\n",
        "    column_count_limit = 4\n",
        "    row_count = math.ceil(building_count/column_count_limit)\n",
        "    column_count = min(column_count_limit, building_count)\n",
        "    figsize = (4.0*column_count, 1.75*row_count)\n",
        "    fig, _ = plt.subplots(row_count, column_count, figsize=figsize)\n",
        "\n",
        "    for i, ax in enumerate(fig.axes):\n",
        "        for k, v in envs.items():\n",
        "            soc = np.array(v.buildings[i].electrical_storage.soc)\n",
        "            capacity = v.buildings[i].electrical_storage.capacity_history[0]\n",
        "            y = soc/capacity\n",
        "            x = range(len(y))\n",
        "            ax.plot(x, y, label=k)\n",
        "\n",
        "        ax.set_title(v.buildings[i].name)\n",
        "        ax.set_xlabel('Time step')\n",
        "        ax.set_ylabel('SoC')\n",
        "        ax.xaxis.set_major_locator(ticker.MultipleLocator(24))\n",
        "\n",
        "        if i == building_count - 1:\n",
        "            ax.legend(\n",
        "                loc='upper left', bbox_to_anchor=(1.0, 1.0), framealpha=0.0\n",
        "            )\n",
        "        else:\n",
        "            ax.legend().set_visible(False)\n",
        "\n",
        "\n",
        "    plt.tight_layout()\n",
        "\n",
        "    return fig"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OSBpXJTg1R0s"
      },
      "source": [
        "> ⚠️ **NOTE**:\n",
        "> You do not need to understand the content of the next code cell where a plotting function is defined.\n",
        "\n",
        "The last function, `plot_simulation_summary` is a convenience function used to plot all figures in one go:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TyZrdT5a1UJM"
      },
      "outputs": [],
      "source": [
        "def plot_simulation_summary(envs: Mapping[str, CityLearnEnv]):\n",
        "    \"\"\"Plots KPIs, load and battery SoC profiles for different control agents.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    envs: Mapping[str, CityLearnEnv]\n",
        "        Mapping of user-defined control agent names to environments\n",
        "        the agents have been used to control.\n",
        "    \"\"\"\n",
        "\n",
        "    _ = plot_building_kpis(envs)\n",
        "    print('Building-level KPIs:')\n",
        "    plt.show()\n",
        "    _ = plot_building_load_profiles(envs)\n",
        "    print('Building-level load profiles:')\n",
        "    plt.show()\n",
        "    _ = plot_battery_soc_profiles(envs)\n",
        "    print('Battery SoC profiles:')\n",
        "    plt.show()\n",
        "    _ = plot_district_kpis(envs)\n",
        "    print('District-level KPIs:')\n",
        "    plt.show()\n",
        "    print('District-level load profiles:')\n",
        "    _ = plot_district_load_profiles(envs)\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yWXsiZ5freTG"
      },
      "source": [
        "# Build your Custom Rule-Based Controller\n",
        "---\n",
        "\n",
        "With our convenience functions defined, we are ready to start solving our earlier described control problem.\n",
        "\n",
        "We will start simple with a rule-based control (RBC) agent that you will build yourself_! RBC is a popular control strategy that is used in most systems e.g. HVAC, batteries, etc because of their level of simplicity. They are  best described as a set of rules expressed as if-else statements and conditions that guide their decision making. An example of such statement is `if outdoor dry-bulb temperature is 20 degrees Celcius and hour 10 PM, charge battery with 5% of capacity`. Now the actual implementation of this statement is open-ended as a designer can choose to program it using any programming language e.g. Python (as used in CityLearn) or a proprietary language that the battery manufacturer uses. Nevertheless, at a high-level, it simplifies to a set of statements and conditions that are easily understood and mappable (think decision tree in supervised learning).\n",
        "\n",
        "The RBC you will be designing here, is a set of if-else statements that use the `hour` observation to determine the amount of energy to charge or discharge a battery. Remember we are using a centralized control strategy thus, the if-else statements you define will apply to all batteries in all buildings.\n",
        "\n",
        "We will use widgets for an interactive RBC tuning experience. You will design a custom RBC that inherits from an existing RBC in CityLearn called the [HourRBC](https://www.citylearn.net/api/citylearn.agents.rbc.html#citylearn.agents.rbc.HourRBC). Inheritance, allows us to copy existing properties and methods in the parent class, `HourRBC`, into our custom class. The `HourRBC` class allows one to define a custom `action_map` using the `hour` as the if condition and the battery capacity proportion as the action where negative proportions imply discharging and positive proportions imply charging.\n",
        "\n",
        "We begin by initializing the environment we will work with:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "unHiw8HH1bzD"
      },
      "outputs": [],
      "source": [
        "rbc_env = CityLearnEnv(schema)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BXyGFcYE1d5j"
      },
      "source": [
        "Now let us define the custom RBC class we will use. All agent classes in CityLearn inherit from the [citylearn.agents.base.Agent](https://www.citylearn.net/api/citylearn.agents.base.html#citylearn.agents.base.Agent) class. This base class has 4 methods that are important to note when defining a new class that inherits from it. namely:\n",
        "\n",
        "1. `__init__` - Used to initialize a new agent with a `citylearn.citylearn.CityLearnEnv` object.\n",
        "2. `learn`: Used to train the initialized object on its environment object.\n",
        "3. `predict`: Used to select actions at each simulation timestep using a defined policy that may be rule-based, reinforcement learning-based or model predictive control-based. The base class selects random actions.\n",
        "4. `update`: Used to update replay buffers, networks and policies at least every timestep. The base class does not perform any updates.\n",
        "5. `next_time_step`: Used to proceed to the next timestep and is called inside `predict`. This function is where class values or custom values that need to collected or updated are best manipulated.\n",
        "\n",
        "In our case with the RBC, we want to include an `action_map` class instance that is a `dict` type. This `action_map` has `int` keys that define hours and `float` values that define charge/discharge action for the hour key that maps them.\n",
        "\n",
        "We also want to include a loader variable to help us visualize the simulation progress. The loader is an `IntProgress` ipywidgets object. We will update the loader's value each timestep the `next_time_step` method is called in the RBC class.\n",
        "\n",
        "The RBC class is defined below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "M-qNSqYSuiY-"
      },
      "outputs": [],
      "source": [
        "class CustomRBC(HourRBC):\n",
        "   def __init__(\n",
        "       self, env: CityLearnEnv, action_map: Mapping[int, float] = None,\n",
        "       loader: IntProgress = None\n",
        "    ):\n",
        "      r\"\"\"Initialize CustomRBC.\n",
        "\n",
        "      Parameters\n",
        "      ----------\n",
        "      env: Mapping[str, CityLearnEnv]\n",
        "         CityLearn environment instance.\n",
        "      action_map: Mapping[int, float]\n",
        "         Mapping of hour to control action.\n",
        "      loader: IntProgress\n",
        "         Progress bar.\n",
        "      \"\"\"\n",
        "\n",
        "      super().__init__(env=env, action_map=action_map)\n",
        "      self.loader = loader\n",
        "\n",
        "   def next_time_step(self):\n",
        "      r\"\"\"Advance to next `time_step`.\"\"\"\n",
        "\n",
        "      super().next_time_step()\n",
        "\n",
        "      if self.loader is not None:\n",
        "         self.loader.value += 1\n",
        "      else:\n",
        "         pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KFj5ovew1uqG"
      },
      "source": [
        "We can now initialize the RBC by setting all actions to 0 for every hour:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2cr_3VaD1wbe"
      },
      "outputs": [],
      "source": [
        "action_map = {i: 0.0 for i in range(1, 25)}\n",
        "rbc_model = CustomRBC(env=rbc_env, action_map=action_map)\n",
        "print('default RBC action map:', action_map)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ubLZV4k4H0H2"
      },
      "source": [
        "We also need to define a convenience function to set and return a loader i.e. a progress bar as we will use this visualization a number of times to track our learning progress:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ST3YhSkRIDLT"
      },
      "outputs": [],
      "source": [
        "def get_loader(**kwargs):\n",
        "    \"\"\"Returns a progress bar\"\"\"\n",
        "\n",
        "    kwargs = {\n",
        "        'value': 0,\n",
        "        'min': 0,\n",
        "        'max': 10,\n",
        "        'description': 'Simulating:',\n",
        "        'bar_style': '',\n",
        "        'style': {'bar_color': 'maroon'},\n",
        "        'orientation': 'horizontal',\n",
        "        **kwargs\n",
        "    }\n",
        "    return IntProgress(**kwargs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SIOhohXk12Vo"
      },
      "source": [
        "With our custom RBC now defined, we can set up the interactive widgets.\n",
        "\n",
        "> ⚠️ **NOTE**:\n",
        "> You do not need to understand the content of the next code cell where the widget is defined. Instead wait for the widgets to load and interact with it using the instructions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wdmxDMHJuiY_"
      },
      "outputs": [],
      "source": [
        "action_step = 0.05\n",
        "hour_step = 2\n",
        "hours = list(range(1, 25, hour_step))\n",
        "default_loader_description = 'Waiting'\n",
        "questions = \"\"\"\n",
        "<h1>Custom RBC Tuner</h1>\n",
        "<p>Use this interactive widget to tune your custom RBC!\n",
        "Reference the building load profiles above and the questions below when\n",
        "deciding on how to charge/discharge your rule-based controlled batteries.</p>\n",
        "\n",
        "<h3>Some considerations when tuning your custom RBC:</h3>\n",
        "<ul>\n",
        "    <li>What happens when actions for all hours are set to 0?</li>\n",
        "    <li>How can we set the RBC so that it takes advantage\n",
        "    of solar generation?</li>\n",
        "    <li>Can you spot the duck curve?</li>\n",
        "    <li>What settings work best for a specific building?</li>\n",
        "    <li>What settings work best for the entire district?</li>\n",
        "    <li>Can you tune the RBC to target improvements in any one of\n",
        "    the evaluation KPIs?</li>\n",
        "    <li>What challenges can you identify from this RBC tuning process?</li>\n",
        "</ul>\n",
        "\n",
        "<h3>Interact with the controls to tune your RBC:</h3>\n",
        "\n",
        "<p>Use the sliders to set the hourly charge and discharge rate\n",
        "of the batteries. Positive values indicate charging\n",
        "and negative values indicate discharging the batteries</p>\n",
        "\"\"\"\n",
        "html_ui = HTML(value=questions, placeholder='Questions')\n",
        "sliders = [FloatSlider(\n",
        "    value=0.0,\n",
        "    min=-1.0,\n",
        "    max=1.0,\n",
        "    step=action_step,\n",
        "    description=f'Hr: {h}-{h + hour_step - 1}',\n",
        "    disabled=False,\n",
        "    continuous_update=False,\n",
        "    orientation='vertical',\n",
        "    readout=True,\n",
        "    readout_format='.2f',\n",
        ") for h in hours]\n",
        "reset_button = Button(\n",
        "    description='Reset', disabled=False, button_style='info',\n",
        "    tooltip='Set all hour actions to 0.0', icon=''\n",
        ")\n",
        "random_button = Button(\n",
        "    description='Random', disabled=False, button_style='warning',\n",
        "    tooltip='Select random hour actions', icon=''\n",
        ")\n",
        "simulate_button = Button(\n",
        "    description='Simulate', disabled=False, button_style='success',\n",
        "    tooltip='Run simulation', icon='check'\n",
        ")\n",
        "sliders_ui = HBox(sliders)\n",
        "buttons_ui = HBox([reset_button, random_button, simulate_button])\n",
        "\n",
        "# run simulation so that the environment has results\n",
        "# even if user does not interact with widgets\n",
        "sac_episodes = 1\n",
        "rbc_model.learn(episodes=sac_episodes)\n",
        "\n",
        "loader = get_loader(description=default_loader_description)\n",
        "\n",
        "def plot_building_guide(env):\n",
        "    \"\"\"Plots building load and generation profiles.\"\"\"\n",
        "\n",
        "    column_count_limit = 4\n",
        "    building_count = len(env.buildings)\n",
        "    row_count = math.ceil(building_count/column_count_limit)\n",
        "    column_count = min(column_count_limit, building_count)\n",
        "    figsize = (4.0*column_count, 1.75*row_count)\n",
        "    fig, _ = plt.subplots(row_count, column_count, figsize=figsize)\n",
        "\n",
        "    for i, (ax, b) in enumerate(zip(fig.axes, env.buildings)):\n",
        "        y1 = b.energy_simulation.non_shiftable_load\n",
        "        y2 = b.pv.get_generation(b.energy_simulation.solar_generation)\n",
        "        x = range(len(y1))\n",
        "        ax.plot(x, y1, label='Load')\n",
        "        ax.plot(x, y2, label='Generation')\n",
        "        ax.set_title(b.name)\n",
        "        ax.set_xlabel('Time step')\n",
        "        ax.set_ylabel('kWh')\n",
        "        ax.xaxis.set_major_locator(ticker.MultipleLocator(24))\n",
        "\n",
        "        if i == building_count - 1:\n",
        "            ax.legend(\n",
        "                loc='upper left', bbox_to_anchor=(1.0, 1.0), framealpha=0.0\n",
        "            )\n",
        "        else:\n",
        "            ax.legend().set_visible(False)\n",
        "\n",
        "    plt.tight_layout()\n",
        "    return fig\n",
        "\n",
        "def on_reset_button_clicked(b):\n",
        "    \"\"\"Zeros sliders and loader values.\"\"\"\n",
        "\n",
        "    loader.value = 0\n",
        "    loader.description = default_loader_description\n",
        "\n",
        "    for s in sliders:\n",
        "        s.value = 0.0\n",
        "\n",
        "def on_random_button_clicked(b):\n",
        "    \"\"\"Zeros loader value and sets sliders to random values.\"\"\"\n",
        "\n",
        "    loader.value = 0\n",
        "    loader.description = default_loader_description\n",
        "    options = np.arange(-1.0, 1.0, action_step)\n",
        "\n",
        "    for s in sliders:\n",
        "        s.value = round(random.choice(options), 2)\n",
        "\n",
        "def on_simulate_button_clicked(b):\n",
        "    \"\"\"Runs RBC simulation using selected action map.\"\"\"\n",
        "\n",
        "    loader.description = 'Simulating'\n",
        "    loader.value = 0\n",
        "    clear_output(wait=False)\n",
        "\n",
        "    # plot building profiles\n",
        "    _ = plot_building_guide(rbc_env)\n",
        "    plt.show()\n",
        "\n",
        "    display(html_ui, sliders_ui, buttons_ui, loader)\n",
        "    reset_button.disabled = True\n",
        "    random_button.disabled = True\n",
        "    simulate_button.disabled = True\n",
        "\n",
        "    for s in sliders:\n",
        "        s.disabled = True\n",
        "\n",
        "    action_map = {}\n",
        "\n",
        "    for h, s in zip(hours, sliders):\n",
        "        for i in range(hour_step):\n",
        "            action_map[h + i] = s.value\n",
        "\n",
        "    loader.max = rbc_env.time_steps*sac_episodes\n",
        "    rbc_model.action_map = action_map\n",
        "    rbc_model.learn(episodes=sac_episodes)\n",
        "\n",
        "    loader.description = 'Finished'\n",
        "    plot_simulation_summary({'RBC': rbc_env})\n",
        "\n",
        "    reset_button.disabled = False\n",
        "    random_button.disabled = False\n",
        "    simulate_button.disabled = False\n",
        "\n",
        "    for s in sliders:\n",
        "        s.disabled = False\n",
        "\n",
        "reset_button.on_click(on_reset_button_clicked)\n",
        "random_button.on_click(on_random_button_clicked)\n",
        "simulate_button.on_click(on_simulate_button_clicked)\n",
        "\n",
        "# plot building profiles\n",
        "_ = plot_building_guide(rbc_env)\n",
        "plt.show()\n",
        "\n",
        "# preview of building load profile\n",
        "display(html_ui, sliders_ui, buttons_ui, loader)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6KHS2--GuiY_"
      },
      "source": [
        "# An Introduction to Tabular Q-Learning Algorithm as an Adaptive Controller\n",
        "---\n",
        "\n",
        "Tuning your RBC must have revealed that it is a cumbersome and labor intensive process, especially as the number of buildings, time period and variance in load profiles increase. What we will be ideal is an adaptive controller that can adjust to different occupant preferences and behaviors in each building that influence load profiles and adjust to different weather conditions that affect the seasonal variance in load profiles.\n",
        "\n",
        "Moreover, we want a controller that is able to learn with little to no knowledge about the environment model it is controlling unlike the RBC tuning process where you probably chose your charge and discharge proportion by visually inspecting the building load and generation profiles. Instead, we want a controller that can learn those patterns in a data-driven fashion."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y8nW-IMQ2Oah"
      },
      "source": [
        "## Q-Learning Background\n",
        "[Tabular Q-Learning](https://link.springer.com/article/10.1007/BF00992698) is a popular model-free reinforcement learning technique due to its simplicity. In simple tasks with small finite state sets, and discrete actions, all transitions can be represented using a table, hence the name Tabular Q-Learning, which stores the state-action values, i.e., Q-values.\n",
        "\n",
        "After taking an action $a$, given a state $s$, and observing the immediate reward $r$ for taking $a$ at $s$, learning is achieved through updating $Q(s, a)$ ([Bellman equation](https://en.wikipedia.org/wiki/Bellman_equation)) as:\n",
        "\n",
        "$$\n",
        "Q(s, a) = Q(s, a) + \\alpha [r + \\gamma \\max_{a'} Q(s', a') - Q(s, a)]\n",
        "$$\n",
        "\n",
        "where $Q(s, a)$ is the Q-value for taking action $a$ in state $s$, $\\alpha ∈ [0, 1]$ is the learning rate, which explicitly defines to what degree new knowledge overrides old knowledge: for $\\alpha = 0$, no learning happens, while for $\\alpha = 1$, all prior knowledge is lost. $\\gamma$ is the discount factor which allow to balance between an agent that considers only immediate rewards ($\\gamma$ = 0) and one that strives towards long term rewards ($\\gamma$ = 1). $\\max_{a'} Q(s', a')$ is the maximum Q-value for all actions $a'$ in the next state $s'$ that is reached after taking action $a$ in state $s$.\n",
        "\n",
        "In other words, the optimal policy, $\\pi$, results from taking those actions $a$ that maximize the respective Q-values in each state, $s$. In order for the algorithm to converge to the optimal policy, the requirement is that each state-action pair $(s, a)$ be visited infinitely many times, such that the Q-values have converged."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-mtY8kPK2RgT"
      },
      "source": [
        "### Algorithm\n",
        "\n",
        "The general Q-Learning algorithm is as follows:\n",
        "\n",
        "> 1. Initialize the Q-table for all state-action pairs.\n",
        "> 2. Set the learning rate $\\alpha$ ($0 < \\alpha < 1$) and the discount factor $\\gamma$ ($0 < \\gamma < 1$).\n",
        "> 3. Repeat the following steps for each episode:\n",
        ">     - Observe the initial state $s$.\n",
        ">     - Choose an action $a$ based on the epsilon-greedy policy (a random action is chosen with probability epsilon, $\\epsilon$ and the action with the highest Q-value is chosen with probability $1 - \\epsilon$).\n",
        ">     - Take the action $a$ and observe the next state $s'$ and the reward $r$.\n",
        ">     - Update the Q-value of the state-action pair $(s,a)$ using the Bellman equation.\n",
        ">     - Set $s = s'$.\n",
        "> 4. Repeat step 3 for a large number of episodes or until convergence is reached."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pI_mZrJ-2WXJ"
      },
      "source": [
        "### Action Selection\n",
        "\n",
        "In Q-learning, the process of accumulating knowledge happens through the trade-off between exploiting known, high-reward, actions, and exploring other, unknown, actions that have not been executed yet under that state. The $\\epsilon$-greedy approach which we use here, selects a random action with probability epsilon, $\\epsilon$ (exploration), and the action with the highest expected return with probability $1 - \\epsilon$ (exploitation). This balancing allows the agent to avoid local minima (exploration), while striving towards convergence (exploitation). In practice, $\\epsilon$ is set relatively large in the beginning of the learning process, and then reduced progressively. The choice of the initial value and the reduction strategy is domain specific and task of the designer."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3JCbSW0t2aYa"
      },
      "source": [
        "## CityLearn Tabular Q-Learning Implementation\n",
        "\n",
        "CityLearn has a Tabular Q-learning implementation in its `citylearn.agents.q_learning.TabularQLearning` class (see [docs](https://www.citylearn.net/api/citylearn.agents.base.html#citylearn.agents.q_learning.TabularQLearning)). This Q-Learning implementation is inspired by the [BOPTEST Tutorial](https://colab.research.google.com/drive/1WeA_3PQeySba0MMRRte_oZTF7ptlP_Ra#scrollTo=9U81QUVcUfoW&line=17&uniqifier=1) but follows the general algorithm in the literature that we have described. However, a caveat of making use of this agent is that it requires discrete observations and actions in order to update the Q-Table whereas the default CityLearn environment provides continuous observations and actions.\n",
        "\n",
        "CityLearn provides an environment wrapper, `TabularQLearningWrapper` (see [docs](https://www.citylearn.net/api/citylearn.wrappers.html#citylearn.wrappers.DiscreteSpaceWrapper)) used to discretize observations and actions before passing to an agent. All we need to do is define the number of bins to use to discretize the observations and actions using the wrapper's `observation_bin_sizes` and `action_bin_sizes` initialization variables.\n",
        "\n",
        "We begin by initializing a new environment:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_j2GAa7YuiZA"
      },
      "outputs": [],
      "source": [
        "tql_env = CityLearnEnv(schema)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o8L28axZ4NVj"
      },
      "source": [
        "We will discretize the hour into 24 bins and the action into 12 bins. Hour is an observation shared by all buildings thus, its values are the same in all buildings at each time step. For this reason, one of the dimensions of our Q-Table will equal hour bin count. The action space for controlling the batteries has the same size as number of buildings thus when discretized, the other Q-Table dimension will equal the `electrical_storage` action raised to the power of building count:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_6HotiSW4Pe8"
      },
      "outputs": [],
      "source": [
        "# define active observations and actions and their bin sizes\n",
        "observation_bins = {'hour': 24}\n",
        "action_bins = {'electrical_storage': 12}\n",
        "\n",
        "# initialize list of bin sizes where each building\n",
        "# has a dictionary in the list definining its bin sizes\n",
        "observation_bin_sizes = []\n",
        "action_bin_sizes = []\n",
        "\n",
        "for b in tql_env.buildings:\n",
        "    # add a bin size definition for the buildings\n",
        "    observation_bin_sizes.append(observation_bins)\n",
        "    action_bin_sizes.append(action_bins)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zFBOCDVU2nGT"
      },
      "source": [
        "Can you think of a way to choose more appropriate bin sizes? How does the choice of bin size affect the learning process?\n",
        "\n",
        "Now we wrap the environment to make sure we are exchanging discrete observations and actions between the environment and agent:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1rkt9jnNuiZE"
      },
      "outputs": [],
      "source": [
        "tql_env = TabularQLearningWrapper(\n",
        "    tql_env.unwrapped,\n",
        "    observation_bin_sizes=observation_bin_sizes,\n",
        "    action_bin_sizes=action_bin_sizes\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IqiqVp0b21zG"
      },
      "source": [
        "We can now go ahead to initialize our Q Learner. We will modify the CityLearn `TabularQLearning` class like we did the `HourRBC` so that we are able to visually track the learning process as well as keep tabs on its cummulative reward as training episode go by. We also provide a `random_seed` instance variable that we set to the random seed you defined earlier. This random seed will ensure that each time this notebook is run, the epsilon-greedy action selections are reproducible. The modifications to the `TabularQLearning` class are done below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5s9klu5nuiZF"
      },
      "outputs": [],
      "source": [
        "class CustomTabularQLearning(TabularQLearning):\n",
        "    def __init__(\n",
        "        self, env: CityLearnEnv, loader: IntProgress,\n",
        "        random_seed: int = None, **kwargs\n",
        "    ):\n",
        "        r\"\"\"Initialize CustomRBC.\n",
        "\n",
        "        Parameters\n",
        "        ----------\n",
        "        env: Mapping[str, CityLearnEnv]\n",
        "            CityLearn environment instance.\n",
        "        loader: IntProgress\n",
        "            Progress bar.\n",
        "        random_seed: int\n",
        "            Random number generator reprocucibility seed for\n",
        "            eqsilon-greedy action selection.\n",
        "        kwargs: dict\n",
        "            Parent class hyperparameters\n",
        "        \"\"\"\n",
        "\n",
        "        super().__init__(env=env, random_seed=random_seed, **kwargs)\n",
        "        self.loader = loader\n",
        "        self.reward_history = []\n",
        "\n",
        "    def next_time_step(self):\n",
        "        if self.env.time_step == 0:\n",
        "            self.reward_history.append(0)\n",
        "\n",
        "        else:\n",
        "            self.reward_history[-1] += sum(self.env.rewards[-1])\n",
        "\n",
        "        self.loader.value += 1\n",
        "        super().next_time_step()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uXHQVD_427-s"
      },
      "source": [
        "With our Tabular Q-Learning agent set up, it is time to train it on our environment. We will use the following hyperparameters:\n",
        "\n",
        "- `epsilon` ($\\epsilon$) = 1.0\n",
        "- `minimum_epsilon` ($\\epsilon_{\\textrm{min}}$) = 0.01\n",
        "- `epsilon_decay` ($\\epsilon_{\\textrm{decay}}$) = 0.0001\n",
        "- `discount_factor` ($\\gamma$) = 0.99\n",
        "- `learning_rate` ($\\alpha$) = 0.005\n",
        "\n",
        "The agent is trained for $\\frac{m \\times n \\times i}{t}$ episodes where $m$ and $n$ are the observation and action space sizes respectively, $i$ is an arbitrary integer and t is the number of time steps in one episode. That way, we increase the probability that we at least visit each state-action combination once."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uS7RzQUyuiZF"
      },
      "outputs": [],
      "source": [
        "# ----------------- CALCULATE NUMBER OF TRAINING EPISODES -----------------\n",
        "i = 3\n",
        "m = tql_env.observation_space[0].n\n",
        "n = tql_env.action_space[0].n\n",
        "t = tql_env.time_steps - 1\n",
        "tql_episodes = m*n*i/t\n",
        "tql_episodes = int(tql_episodes)\n",
        "print('Q-Table dimension:', (m, n))\n",
        "print('Number of episodes to train:', tql_episodes)\n",
        "\n",
        "# ------------------------------- SET LOADER ------------------------------\n",
        "loader = get_loader(max=tql_episodes*t)\n",
        "display(loader)\n",
        "\n",
        "# ----------------------- SET MODEL HYPERPARAMETERS -----------------------\n",
        "tql_kwargs = {\n",
        "    'epsilon': 1.0,\n",
        "    'minimum_epsilon': 0.01,\n",
        "    'epsilon_decay': 0.0001,\n",
        "    'learning_rate': 0.005,\n",
        "    'discount_factor': 0.99,\n",
        "}\n",
        "\n",
        "# ----------------------- INITIALIZE AND TRAIN MODEL ----------------------\n",
        "tql_model = CustomTabularQLearning(\n",
        "    env=tql_env,\n",
        "    loader=loader,\n",
        "    random_seed=RANDOM_SEED,\n",
        "    **tql_kwargs\n",
        ")\n",
        "_ = tql_model.learn(episodes=tql_episodes)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jU_PUps73Gdm"
      },
      "source": [
        "We now evaluate the trained model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-uG900pOuiZH"
      },
      "outputs": [],
      "source": [
        "observations = tql_env.reset()\n",
        "\n",
        "while not tql_env.done:\n",
        "    actions = tql_model.predict(observations, deterministic=True)\n",
        "    observations, _, _, _ = tql_env.step(actions)\n",
        "\n",
        "# plot summary and compare with other control results\n",
        "plot_simulation_summary({'RBC': rbc_env, 'TQL': tql_env})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lpbit9J6uiZK"
      },
      "source": [
        "The figures plotted for the Tabular Q-Learning are compared against the baseline and your tuned RBC. The Q-Learning agent has performed worse than the baseline and RBC in terms of the building-level and district-level KPIs. The net electricity consumption profile as a result of the Q-Learning agent shows unstable and spiky consumption. The reason for this behavior is seen in the battery SoC curves where the changes in SoC are abrupt. This highlights an issue with our discretized action space having too large steps as trade off for maintaining a reasonably-sized Q-Table. We also see that the agent did not learn the unique day-to-day building needs as the SoC profile is identical every 24 time steps. This is a consequence of using a single observation, hour to learn.\n",
        "\n",
        "For the buildings 2 and 7 selected when the `RANDOM_SEED` = 0, we see that agent learned to charge the battery in building 2 in the early morning just after midnight and slightly charged and discharges during the day before completely depleting charge by midnight and into the early early hours if the next day. Building 7 on the other hand has 2 charge-discharge cycles each day that are split around noon.\n",
        "\n",
        "Since the Q-Table is 2 dimensional, we can visualize and spot the the state-action combinations that maximize the Q-value below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "v3OX9GNVnEDt"
      },
      "outputs": [],
      "source": [
        "def plot_table(\n",
        "    ax: plt.Axes, table: np.ndarray, title: str, cmap: str,\n",
        "    colorbar_label: str, xlabel: str, ylabel: str\n",
        ") -> plt.Axes:\n",
        "    \"\"\"Plot 2-dimensional table on a heat map.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    ax: plt.Axes\n",
        "        Figure axes\n",
        "    table: np.ndarray\n",
        "        Table array\n",
        "    title: str\n",
        "        axes title\n",
        "    cmap: str\n",
        "        Colormap\n",
        "    colorbar_label: str\n",
        "        Colorbar name\n",
        "    xlabel: str\n",
        "        Heat map x-axis label\n",
        "    ylabel: str\n",
        "        Heat map y-axis label\n",
        "\n",
        "    Returns\n",
        "    -------\n",
        "    ax: plt.Axes\n",
        "        Plotted axes\n",
        "    \"\"\"\n",
        "\n",
        "    x = list(range(table.shape[0]))\n",
        "    y = list(range(table.shape[1]))\n",
        "    z = table.T\n",
        "    pcm = ax.pcolormesh(\n",
        "        x, y, z, shading='nearest', cmap=cmap,\n",
        "        edgecolors='black', linewidth=0.0\n",
        "    )\n",
        "    _ = fig.colorbar(\n",
        "        pcm, ax=ax, orientation='horizontal',\n",
        "        label=colorbar_label, fraction=0.025, pad=0.08\n",
        "    )\n",
        "    ax.set_xlabel(xlabel)\n",
        "    ax.set_ylabel(ylabel)\n",
        "    ax.set_title(title)\n",
        "\n",
        "    return ax"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hdSMspcn3jFG"
      },
      "outputs": [],
      "source": [
        "cmap = 'coolwarm'\n",
        "figsize = (12, 8)\n",
        "fig, axs = plt.subplots(1, 3, figsize=figsize, sharey=True)\n",
        "axs[0] = plot_table(\n",
        "    axs[0], tql_model.q[0], 'Q-Table',\n",
        "    cmap, 'Q-Value', 'State (Hour)', 'Action Index'\n",
        ")\n",
        "axs[1] = plot_table(\n",
        "    axs[1], tql_model.q_exploration[0], 'Q-Table Exploration',\n",
        "    cmap, 'Count', 'State (Hour)', None\n",
        ")\n",
        "axs[2] = plot_table(\n",
        "    axs[2], tql_model.q_exploitation[0], 'Q-Table Exploitation',\n",
        "    cmap, 'Count', 'State (Hour)', None\n",
        ")\n",
        "\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2VAlscKD3iKO"
      },
      "source": [
        "The Q-Table shows (left) shows that for each hour, the Q-Values for most action indices are similar and very low (dark blue) asides the one action index that has been exploited. The middle heat map shows how many times each state-action pair was explored i.e. randomly chosen using $\\epsilon$, and we see that while most pairs have been visited at least once, some pairs have the monopoly. The figure on the right shows how many times state-action pairs were exploited. For each state, only one action was ever an exploitation candidate. This shows that the algorithm spent much time exploring randomly and the first discovered exploitation candidate for each state remained till learning was terminated. We can tell the exploration-exploitation balance through $\\epsilon$:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "N2Alb3JD3o6b"
      },
      "outputs": [],
      "source": [
        "print(\n",
        "    f'Current Tabular Q-Learning epsilon after {tql_episodes}'\\\n",
        "        f' episodes and {tql_model.time_step} time steps:', tql_model.epsilon\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XYmw5oz13pmy"
      },
      "source": [
        "Epsilon is still high so there is a higher probability of random exploration. The Q-Learning agent updates epsilon using the following exponential decay formula:\n",
        "\n",
        "$$\n",
        "\\epsilon = \\textrm{max}(\\epsilon_{\\textrm{minimum}}, \\epsilon_{0} \\cdot e^{-\\epsilon_{\\textrm{decay}}*\\textrm{episode}})\n",
        "$$\n",
        "\n",
        "where $\\epsilon_{0}$ is $\\epsilon$ at time step 0. Thus with the current decay rate, $\\epsilon_{\\textrm{decay}}$ we can visualize the number of episodes needed to get to at least 50-50 probability of exploration-exploitation:  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JDjatcra3vPG"
      },
      "outputs": [],
      "source": [
        "fig, ax = plt.subplots(1, 1, figsize=(4, 2))\n",
        "y = np.array([max(\n",
        "    tql_model.minimum_epsilon,\n",
        "    tql_model.epsilon_init*np.exp(-tql_model.epsilon_decay*e)\n",
        ") for e in range(100_000)])\n",
        "ref_x = len(y) - len(y[y <= 0.5]) - 1\n",
        "ref_y = y[ref_x]\n",
        "ax.plot(y)\n",
        "ax.axvline(ref_x, color='red', linestyle=':')\n",
        "ax.axhline(ref_y, color='red', linestyle=':')\n",
        "ax.axvline(tql_episodes, color='green', linestyle=':')\n",
        "ax.set_xlabel('Episode')\n",
        "text = f'{ref_x} training episodes needed to get\\nat least 50%'\\\n",
        "    ' exploitation probability.'\n",
        "ax.text(ref_x + 1000, ref_y + 0.05, text, color='red')\n",
        "ax.text(\n",
        "    tql_episodes + 1000,\n",
        "    ref_y - 0.1,\n",
        "    f'Current training episodes = {tql_episodes}',\n",
        "    va='bottom', color='green'\n",
        ")\n",
        "ax.set_ylabel(r'$\\epsilon$')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PUTniQTj3ycU"
      },
      "source": [
        "Now that we have experimented with the Tabular Q-Learning algorithm, what issues can you identify with this control approach? Ponder on these questions:\n",
        "\n",
        "1. How do the observations we use affect learning?\n",
        "2. How does the table dimension affect learning?\n",
        "4. What can we do to ensure that there is enough exploration of all state-action pairs?\n",
        "6. In what building control applications/examples could Tabular Q-Learning work well?\n",
        "7. In what building applications/examples will Tabular Q-Learning most likely fail?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yNC34scp32zW"
      },
      "source": [
        "## Replacing the Q-Table with a Function Approximator\n",
        "\n",
        "Tabular Q-Learning is affected by the curse of dimensionality: as the size of the state space increases due to, e.g., continuous sensor inputs, the size of the Q-table has to necessarily increase is well. In particular for building control, the curse of dimensionality is significant, considering the potentially large number of sensors measuring various quantities (temperature, humidity, energy consumption, etc.) continuously. This means that the agent has an exponentially increasing number of state-action pairs to explore before it can converge to an optimal solution. Function approximators, e.g., linear regression or artificial neural networks ([Haykin (2009)](https://www.pearson.com/en-us/subject-catalog/p/neural-networks-and-learning-machines/P200000003278/9780133002553)), have been proposed as solutions that allow generalization by directly mapping the state-action pairs, $(s, a)$, to their respective Q-value, $Q(s, a)$. Refer to [Reinforcement learning for intelligent environments](https://www.taylorfrancis.com/chapters/edit/10.4324/9781315142074-37/reinforcement-learning-intelligent-environments-zoltan-nagy-june-young-park-josé-ramón-vázquez-canteli) for more information on how to make use of function approximators to improve learning in reinforcement learning control (RLC).\n",
        "\n",
        "In the next section, we will introduce the soft-actor critic (SAC) algorithm, which is a model-free Q-Learning algorithm, that uses a neural network to approximate the Q-values thus, reducing the cost of training compared to Tabular Q-Learning."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q74Y_l8a369T"
      },
      "source": [
        "# Optimize a Soft-Actor Critic Reinforcement Learning Controller\n",
        "---\n",
        "\n",
        "To control an environment like CityLearn that has continuous states and actions, tabular Q-learning is not practical, as it suffers from the _curse of dimensionality_. Actor-critic reinforcement learning (RL) methods use artificial neural networks to generalize across the state-action space. The actor network maps the current states to the actions that it estimates to be optimal. Then, the critic network evaluates those actions by mapping them, together with the states under which they were taken, to the Q-values.\n",
        "\n",
        "<figure class=\"image\">\n",
        "  <img src=\"https://github.com/intelligent-environments-lab/CityLearn/blob/master/assets/images/sac_schematic.png?raw=true\"  width=\"350\" alt=\"SAC networks overview.\">\n",
        "  <figcaption>Figure: SAC networks overview (adopted from <a href=\"https://doi.org/10.1145/3408308.3427604\">Vazquez-Canteli et al., 2020</a>).</figcaption>\n",
        "</figure>\n",
        "\n",
        "Soft actor-critic (SAC) is a model-free off-policy RL algorithm. As an off-policy method, SAC can reuse experience and learn from fewer samples. SAC is based on three key elements: an actor-critic architecture, off-policy updates, and entropy maximization for efficient exploration and stable training. SAC learns three different functions: the actor (policy), the critic (soft Q-function), and the value function.\n",
        "\n",
        "This tutorial does not dive into the theory and algorithm of SAC but for interested participants please, refer to [Soft Actor-Critic Algorithms and Applications](https://doi.org/10.48550/arXiv.1812.05905).\n",
        "\n",
        "We will now initialize a new environment and plug it to an SAC agent to help us solve our control problem. Luckily, we do not have to write our own implementation of the SAC algorithm. Instead, we can make use of Python libraries that have standardized the implementation of a number of RL algorithms. One of such libraries that we will use is [Stable Baselines3](https://stable-baselines3.readthedocs.io/en/master/index.html). At the time of writing, there are [13 different RL algorithms](https://stable-baselines3.readthedocs.io/en/master/guide/algos.html#rl-algorithms) including SAC implemented between Stable Baselines3 and Stable-Baselines3 - Contrib (contrib package for Stable-Baselines3 - experimental reinforcement learning (RL) code).\n",
        "\n",
        "The new environment is initialized below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "n9A8-38t390y"
      },
      "outputs": [],
      "source": [
        "sac_env = CityLearnEnv(schema)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EfKkwMPG4Aff"
      },
      "source": [
        "Before our environment is ready for use in Stable Baselines3, we need to take a couple of preprocessing steps in the form of wrappers. Firstly, we will wrap the environment using the `NormalizedObservationWrapper` (see [docs](https://www.citylearn.net/api/citylearn.wrappers.html#citylearn.wrappers.NormalizedObservationWrapper)) that ensure all observations that are served to the agent are [min-max normalized](https://www.codecademy.com/article/normalization) between [0, 1] and cyclical observations e.g. hour, are encoded using the [sine and cosine transformation](https://www.avanwyk.com/encoding-cyclical-features-for-deep-learning/)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cBH83tFY4DhV"
      },
      "outputs": [],
      "source": [
        "sac_env = NormalizedObservationWrapper(sac_env)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LYyxUFcO4Erv"
      },
      "source": [
        "Next, we wrap with the `StableBaselines3Wrapper` (see [docs](https://www.citylearn.net/api/citylearn.wrappers.html#citylearn.wrappers.StableBaselines3Wrapper)) that ensures observations, actions and rewards are served in manner that is compatible with Stable Baselines3 interface:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3Yq5edYr4JXo"
      },
      "outputs": [],
      "source": [
        "sac_env = StableBaselines3Wrapper(sac_env)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vjx0GjAk4Kux"
      },
      "source": [
        "Now we can go ahead and initialize the SAC model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WLkcTNQ34NLy"
      },
      "outputs": [],
      "source": [
        "sac_model = SAC(policy='MlpPolicy', env=sac_env, seed=RANDOM_SEED)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3HQNnT8O4Qmi"
      },
      "source": [
        "In order to track the progress of learning, we will use a loader as we have done before. Stable Baselines3 makes use of callbacks to help with performing user-defined actions and procedures during learning. However, you do not need to know the specifics of the code below beyond being aware that it is used to update the loader value and store aggregated rewards at each time step."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LtnL5S394TJB"
      },
      "outputs": [],
      "source": [
        "class CustomCallback(BaseCallback):\n",
        "    def __init__(self, env: CityLearnEnv, loader: IntProgress):\n",
        "        r\"\"\"Initialize CustomCallback.\n",
        "\n",
        "        Parameters\n",
        "        ----------\n",
        "        env: Mapping[str, CityLearnEnv]\n",
        "            CityLearn environment instance.\n",
        "        loader: IntProgress\n",
        "            Progress bar.\n",
        "        \"\"\"\n",
        "\n",
        "        super().__init__(verbose=0)\n",
        "        self.loader = loader\n",
        "        self.env = env\n",
        "        self.reward_history = [0]\n",
        "\n",
        "    def _on_step(self) -> bool:\n",
        "        r\"\"\"Called each time the env step function is called.\"\"\"\n",
        "\n",
        "        if self.env.time_step == 0:\n",
        "            self.reward_history.append(0)\n",
        "\n",
        "        else:\n",
        "            self.reward_history[-1] += sum(self.env.rewards[-1])\n",
        "\n",
        "        self.loader.value += 1\n",
        "\n",
        "        return True"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gLSppNHb4ViE"
      },
      "source": [
        "We will train the model for a fraction of the episodes we used to train the Tabular Q-Learning agent:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Hpytx_Rz4onF"
      },
      "outputs": [],
      "source": [
        "# ----------------- CALCULATE NUMBER OF TRAINING EPISODES -----------------\n",
        "fraction = 0.25\n",
        "sac_episodes = int(tql_episodes*fraction)\n",
        "print('Fraction of Tabular Q-Learning episodes used:', fraction)\n",
        "print('Number of episodes to train:', sac_episodes)\n",
        "sac_episode_timesteps = sac_env.time_steps - 1\n",
        "sac_total_timesteps = sac_episodes*sac_episode_timesteps\n",
        "\n",
        "# ------------------------------- SET LOADER ------------------------------\n",
        "sac_loader = get_loader(max=sac_total_timesteps)\n",
        "display(sac_loader)\n",
        "\n",
        "# ------------------------------- TRAIN MODEL -----------------------------\n",
        "sac_callback = CustomCallback(env=sac_env, loader=sac_loader)\n",
        "sac_model = sac_model.learn(\n",
        "    total_timesteps=sac_total_timesteps,\n",
        "    callback=sac_callback\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ErTZIqzS4zgO"
      },
      "source": [
        "With the SAC model trained, we will evaluate it for 1 episode using deterministic actions i.e. actions that maximized the Q-values during training as in the Tabular Q-Learning approach."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5SxBBofg5pgL"
      },
      "outputs": [],
      "source": [
        "observations = sac_env.reset()\n",
        "sac_actions_list = []\n",
        "\n",
        "while not sac_env.done:\n",
        "    actions, _ = sac_model.predict(observations, deterministic=True)\n",
        "    observations, _, _, _ = sac_env.step(actions)\n",
        "    sac_actions_list.append(actions)\n",
        "\n",
        "# plot summary and compare with other control results\n",
        "plot_simulation_summary({'RBC': rbc_env, 'TQL': tql_env, 'SAC-1': sac_env})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tk-fLPpx5tZi"
      },
      "source": [
        "<img src=\"https://media.giphy.com/media/80TEu4wOBdPLG/giphy.gif\" height=200></img>\n",
        "\n",
        "The figures show that the SAC agent pretty much did not learn anything! The KPIs remain unchanged compared to the baseline and the battery SoCs are 0 all the time. What might be the case here? Let us have a look a the actions the SAC agent prescribed:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tLIKfCXO5xiD"
      },
      "outputs": [],
      "source": [
        "def plot_actions(actions_list: List[List[float]], title: str) -> plt.Figure:\n",
        "    \"\"\"Plots action time series for different buildings\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    actions_list: List[List[float]]\n",
        "        List of actions where each element with index, i,\n",
        "        in list is a list of the actions for different buildings\n",
        "        taken at time step i.\n",
        "    title: str\n",
        "        Plot axes title\n",
        "\n",
        "    Returns\n",
        "    -------\n",
        "    fig: plt.Figure\n",
        "        Figure with plotted axes\n",
        "\n",
        "    \"\"\"\n",
        "\n",
        "    fig, ax = plt.subplots(1, 1, figsize=(6, 1))\n",
        "    columns = [b.name for b in sac_env.buildings]\n",
        "    plot_data = pd.DataFrame(actions_list, columns=columns)\n",
        "    x = list(range(plot_data.shape[0]))\n",
        "\n",
        "    for c in plot_data.columns:\n",
        "        y = plot_data[c].tolist()\n",
        "        ax.plot(x, y, label=c)\n",
        "\n",
        "    ax.legend(loc='upper left', bbox_to_anchor=(1.0, 1.0), framealpha=0.0)\n",
        "    ax.set_xlabel('Time step')\n",
        "    ax.set_ylabel(r'$\\frac{kWh}{kWh_{capacity}}$')\n",
        "    ax.xaxis.set_major_locator(ticker.MultipleLocator(24))\n",
        "    ax.set_title(title)\n",
        "\n",
        "    return fig\n",
        "\n",
        "fig = plot_actions(sac_actions_list, 'SAC-1 Actions')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mT5LI1iz51jd"
      },
      "source": [
        "<img src=\"https://media.giphy.com/media/b8RfbQFaOs1rO10ren/giphy.gif\" height=200></img>\n",
        "\n",
        "The SAC agent was calling for discharge all the time! To give it away, the reason for this behavior is the reward function that we have used to train the agent 😅.\n",
        "\n",
        "Recall that the Bellman equation uses a reward, $r$, to update the Q-values hence the Q-Table is sensitive to the way the $r$ changes for $(s, a, s')$ tuple. That is to say, we need to make sure the reward we calculate after an action, $a$, is taken at state, $s$, quantifies how-well that action actually causes desirable next state, $s'$. If we define a poor reward function, we risk not learning quickly, or undesirable outcomes. See this example of the [implication of a poorly designed reward function](https://openai.com/research/faulty-reward-functions) where an agent learns to maximize a game score but with dangerous actions!\n",
        "\n",
        "The reward function is a variable in the CityLearn environment. The [docs](https://www.citylearn.net/api/citylearn.reward_function.html) provides information on in-built reward functions that can be used in simulation. The reward function used at run time is that which is defined in the schema and used to construct the environment. It can be overridden by parsing an alternative reward function that inherits from the `citylearn.reward_function.RewardFunction` class (see [docs](https://www.citylearn.net/api/citylearn.reward_function.html#citylearn.reward_function.RewardFunction)). Let us see what the current reward is:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "57tLgSp858N3"
      },
      "outputs": [],
      "source": [
        "help(sac_env.reward_function)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TWprvdo46EYE"
      },
      "source": [
        "The current reward functions is the electricity consumption from the grid at the current time step returned as a negative value. While this reward will penalize high electricity consumption, it might not be ideal for all KPIs we are trying to optimize. As you would imagine, the best way to minimize electricity consumption is to try to move all loads to the battery hence, the insistence of the agent to continue to discharge the batteries!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dt77XG7I6Fd6"
      },
      "source": [
        "## Defining a Custom Reward Function\n",
        "\n",
        "We want to reduce electricity consumption but also reduce its cost and emissions. Likewise, we want to reduce the peaks and ramping, and increase the load factor. One way to achieve this is to teach the agent to charge the batteries when electricity is cheap after 9 PM and before 4 PM, which typically coincides with when the grid is cleaner (lower emissions). But recall that each building is able to generate power provided there is solar radiation. So, we can take advantage of self-generation in the late morning to late afternoon to charge for free and discharge the rest of the day thus reducing electricity consumption, cost and emissions at the very least. Also, by shifting the early morning and evening peak loads to the batteries we can improve on our peak and load-factor KPIs.\n",
        "\n",
        "We should also teach our agent to ensure that renewable solar generation is not wasted by making use of the PV to charge the batteries while they are charged below capacity. On the flip side, the agent should learn to discharge when there is net positive grid load and the batteries still have stored energy.\n",
        "\n",
        "Given these learning objectives, we can now define a reward function that closely satisfies the criteria for which the agent will learn good rewards:\n",
        "\n",
        "$$\n",
        "    r = \\sum_{i=0}^n \\Big(p_i \\times |C_i|\\Big)\n",
        "$$\n",
        "\n",
        "$$\n",
        "    p_i = -\\left(1 + \\textrm{sign}(C_i) \\times \\textrm{SOC}^{\\textrm{battery}}_i\\right)\n",
        "$$\n",
        "\n",
        "The reward function, $r$, is designed to minimize electricity cost, $C$. It is calculated for each building, $i$ and summed to provide the agent with a reward that is representative of all $n$ buildings. It encourages net-zero energy use by penalizing grid load satisfaction when there is energy in the battery as well as penalizing net export when the battery is not fully charged through the penalty term, $p$. There is neither penalty nor reward when the battery is fully charged during net export to the grid. Whereas, when the battery is charged to capacity and there is net import from the grid the penalty is maximized.\n",
        "\n",
        "Now we define this custom reward below and set it as the reward for the SAC agent."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oPK08TkI6Jsi"
      },
      "outputs": [],
      "source": [
        "class CustomReward(RewardFunction):\n",
        "    def __init__(self, env: CityLearnEnv):\n",
        "        r\"\"\"Initialize CustomReward.\n",
        "\n",
        "        Parameters\n",
        "        ----------\n",
        "        env: Mapping[str, CityLearnEnv]\n",
        "            CityLearn environment instance.\n",
        "        \"\"\"\n",
        "\n",
        "        super().__init__(env)\n",
        "\n",
        "    def calculate(self) -> List[float]:\n",
        "        r\"\"\"Returns reward for most recent action.\n",
        "\n",
        "        The reward is designed to minimize electricity cost.\n",
        "        It is calculated for each building, i and summed to provide the agent\n",
        "        with a reward that is representative of all n buildings.\n",
        "        It encourages net-zero energy use by penalizing grid load satisfaction\n",
        "        when there is energy in the battery as well as penalizing\n",
        "        net export when the battery is not fully charged through the penalty\n",
        "        term. There is neither penalty nor reward when the battery\n",
        "        is fully charged during net export to the grid. Whereas, when the\n",
        "        battery is charged to capacity and there is net import from the\n",
        "        grid the penalty is maximized.\n",
        "\n",
        "        Returns\n",
        "        -------\n",
        "        reward: List[float]\n",
        "            Reward for transition to current timestep.\n",
        "        \"\"\"\n",
        "\n",
        "        reward_list = []\n",
        "\n",
        "        for b in self.env.buildings:\n",
        "            cost = b.net_electricity_consumption_cost[-1]\n",
        "            battery_capacity = b.electrical_storage.capacity_history[0]\n",
        "            battery_soc = b.electrical_storage.soc[-1]/battery_capacity\n",
        "            penalty = -(1.0 + np.sign(cost)*battery_soc)\n",
        "            reward = penalty*abs(cost)\n",
        "            reward_list.append(reward)\n",
        "\n",
        "        reward = [sum(reward_list)]\n",
        "\n",
        "        return reward"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lolPaXje6Mlk"
      },
      "source": [
        "Let us repeat all the previous steps we took in the former SAC simulation where the only difference in the workflow here is the use of our new custom reward function:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "38i9ZAnp6Ns7"
      },
      "outputs": [],
      "source": [
        "# ----------------- INITIALIZE ENVIRONMENT -----------------\n",
        "sacr_env = CityLearnEnv(schema)\n",
        "\n",
        "# -------------------- SET CUSTOM REWARD -------------------\n",
        "sacr_env.reward_function = CustomReward(sacr_env)\n",
        "\n",
        "# -------------------- WRAP ENVIRONMENT --------------------\n",
        "sacr_env = NormalizedObservationWrapper(sacr_env)\n",
        "sacr_env = StableBaselines3Wrapper(sacr_env)\n",
        "\n",
        "# -------------------- INITIALIZE AGENT --------------------\n",
        "sacr_model = SAC(policy='MlpPolicy', env=sacr_env, seed=RANDOM_SEED)\n",
        "\n",
        "\n",
        "# ----------------------- SET LOADER -----------------------\n",
        "print('Number of episodes to train:', sac_episodes)\n",
        "sac_modr_loader = get_loader(max=sac_total_timesteps)\n",
        "display(sac_modr_loader)\n",
        "\n",
        "# ----------------------- TRAIN AGENT ----------------------\n",
        "sacr_callback = CustomCallback(env=sacr_env, loader=sac_modr_loader)\n",
        "sacr_model = sacr_model.learn(\n",
        "    total_timesteps=sac_total_timesteps,\n",
        "    callback=sacr_callback\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hsMNFhJX6RWb"
      },
      "source": [
        "Finally, evaluate the trained model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "R9uB1Fr56TmN"
      },
      "outputs": [],
      "source": [
        "observations = sacr_env.reset()\n",
        "sacr_actions_list = []\n",
        "\n",
        "while not sacr_env.done:\n",
        "    actions, _ = sacr_model.predict(observations, deterministic=True)\n",
        "    observations, _, _, _ = sacr_env.step(actions)\n",
        "    sacr_actions_list.append(actions)\n",
        "\n",
        "plot_simulation_summary(\n",
        "    {'RBC': rbc_env, 'TQL': tql_env, 'SAC-1': sac_env, 'SAC-2': sacr_env}\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "46jpY4Or6X4I"
      },
      "source": [
        "Finally, we have results that have improved the baseline KPIs all thanks to our custom reward function! The agent has learned to take advantage of the solar generation to charge the batteries and discharge the stored energy during the evening peak.\n",
        "\n",
        "Let us now have a look at the actions that the agent predicted in the deterministic simulation:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0s2C5gOf6aSO"
      },
      "outputs": [],
      "source": [
        "fig = plot_actions(sacr_actions_list, 'SAC Actions using Custom Reward')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "geSmAWJ76ePi"
      },
      "source": [
        "The agent learned the different building needs as building 7 begins to charge later than building 2 daily (selected buildings when `RANDOM_SEED` = 0). The agent discharges the batteries differently as well."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OJniwtkc6gz9"
      },
      "source": [
        "## Evaluate the Episode Rewards for RL Algorithms\n",
        "\n",
        "We can also investigate the convergence rate in training by looking at the sum of rewards in each episode. We expect to see the reward sum increase as we train on more episodes and eventually plateau when exploitation increases or performance can not be further improved. We will look at the reward trajectory for the Tabular Q-Learning, SAC with and without custom reward models:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4D-Qlfv-6kNI"
      },
      "outputs": [],
      "source": [
        "def plot_rewards(ax: plt.Axes, rewards: List[float], title: str) -> plt.Axes:\n",
        "    \"\"\"Plots rewards over training episodes.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    rewards: List[float]\n",
        "        List of reward sum per episode.\n",
        "    title: str\n",
        "        Plot axes title\n",
        "\n",
        "    Returns\n",
        "    -------\n",
        "    ax: plt.Axes\n",
        "        Plotted axes\n",
        "    \"\"\"\n",
        "\n",
        "    ax.plot(rewards)\n",
        "    ax.set_xlabel('Episode')\n",
        "    ax.set_ylabel('Reward')\n",
        "    ax.set_title(title)\n",
        "\n",
        "    return ax"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VZOaEMwQ6lqa"
      },
      "outputs": [],
      "source": [
        "rewards = {\n",
        "    'Tabular Q-Learning': tql_model.reward_history[:tql_episodes],\n",
        "    'SAC-1': sac_callback.reward_history[:sac_episodes],\n",
        "    'SAC-2': sacr_callback.reward_history[:sac_episodes]\n",
        "}\n",
        "fig, axs = plt.subplots(1, 3, figsize=(15, 2))\n",
        "\n",
        "for ax, (k, v) in zip(fig.axes, rewards.items()):\n",
        "    ax = plot_rewards(ax, v, k)\n",
        "\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CZ451oYn6qEI"
      },
      "source": [
        "Some questions to ponder on:\n",
        "1. What do you notice in the reward trajectories for the three models?\n",
        "2. Which model converged?\n",
        "3. Which model did not learn anything?\n",
        "4. Which model needs to train some more?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4EtYMAY-6s7o"
      },
      "source": [
        "# Tune your SAC Agent\n",
        "---\n",
        "\n",
        "Thus far, you have learned to manage battery charge/discharge for a district of buildings by 1) tuning your own rule-based control (RBC) agent, 2) training a Tabular Q-Learning agent, 3) implementing the soft-actor critic (SAC) off-policy reinforcement learning (RL) algorithm with and crude reward function and a better tailored reward function.\n",
        "\n",
        "When each control agent is evaluated on the a set of building-level and district-level KPIs, we find that if carefully tuned, your RBC will improve the baseline albeit a painstaking effort. The Tabular Q-Learning agent has the potential to adapt to unique building properties but suffers from the curse of dimensionality affecting its convergence to an optimal solution for the battery management. We also find that the SAC agent is sensitive to the reward function design and with a custom reward that is tailored towards achieving our evaluation KPIs, we can achieve a performance that is better than the baseline case and potentially better than an averagely tuned RBC.\n",
        "\n",
        "However, we find that the SAC + custom reward case did not converge after our set number of training episodes. Also, the improvements it provides beyond the baseline are not very large. Hence, there is still room for improvement.\n",
        "\n",
        "In the next cells, you will improve the SAC model by:\n",
        "1. Revising the custom reward function with a function you deem more appropriate towards optimizing the KPIs. Perhaps, you can design a reward function that targets a specific KPI. You can also keep the current custom reward function.\n",
        "2. Changing the length of training i.e. episodes.\n",
        "3. Optimizing the SAC hyperparameters. In our previous models, we used the default Stable Baselines3 hyperparameters. Hyperparameter tuning is an _art_ of its own. Refer to the [Stable Baselines3 SAC docs](https://stable-baselines3.readthedocs.io/en/master/modules/sac.html#stable_baselines3.sac.SAC) to learn about the SAC hyperparameters. Also, refer to [Training with Soft-Actor Critic](https://github.com/yosider/ml-agents-1/blob/master/docs/Training-SAC.md#training-with-soft-actor-critic) for a more elaborate description of what the hyperparameters mean, their typical values and appropriate values for different case scenarios.\n",
        "\n",
        "You may also choose to update the active observations. Recall that thus far, we have only used the `hour` observation. Refer to the [CityLearn Observation docs](https://www.citylearn.net/overview/observations.html) to discover other available environment observations."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nunxV4ev6wWb"
      },
      "source": [
        "## Set Environment, Agent and Reward Function\n",
        "\n",
        "The next cell is a __recipe__ for your tuned SAC and custom environment:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zjkvrnpS6yhL"
      },
      "outputs": [],
      "source": [
        "# -------------------- CUSTOMIZE **YOUR** ENVIRONMENT --------------------\n",
        "# Include other observations if needed.\n",
        "# See https://www.citylearn.net/overview/observations.html\n",
        "# for table of observations that you can include\n",
        "# NOTE: More active observations could mean longer trainer time.\n",
        "your_active_observations = [\n",
        "    'hour',\n",
        "    # 'day_type'\n",
        "]\n",
        "\n",
        "# ------------------ SET **YOUR** AGENT HYPERPARAMETERS ------------------\n",
        "# try out different hyperparameter value combinations to see\n",
        "# which one provides you with the best KPIs. See\n",
        "# https://github.com/yosider/ml-agents-1/blob/master/docs/Training-SAC.md#training-with-soft-actor-critic\n",
        "# for a guide on how to select hyperparameter values.\n",
        "your_agent_kwargs = {\n",
        "    'learning_rate': 0.0003,\n",
        "    'buffer_size': 1000000,\n",
        "    'learning_starts': 100,\n",
        "    'batch_size': 256,\n",
        "    'tau': 0.005,\n",
        "    'gamma': 0.99,\n",
        "    'train_freq': 1,\n",
        "}\n",
        "\n",
        "# --------------- SET **YOUR** NUMBER OF TRAINING EPISODES ---------------\n",
        "your_episodes = sac_episodes\n",
        "\n",
        "# --------------- DEFINE **YOUR** CUSTOM REWARD FUNCTION -----------------\n",
        "class YourCustomReward(CustomReward):\n",
        "    def __init__(self, env: CityLearnEnv):\n",
        "        r\"\"\"Initialize CustomReward.\n",
        "\n",
        "        Parameters\n",
        "        ----------\n",
        "        env: Mapping[str, CityLearnEnv]\n",
        "            CityLearn environment instance.\n",
        "        \"\"\"\n",
        "\n",
        "        super().__init__(env)\n",
        "\n",
        "    def calculate(self) -> List[float]:\n",
        "        r\"\"\"Returns reward for most recent action.\n",
        "\n",
        "        <Provide a description for your custom reward>.\n",
        "\n",
        "        Parameters\n",
        "        ----------\n",
        "        reward: List[float]\n",
        "            Reward for transition to current timestep.\n",
        "        \"\"\"\n",
        "\n",
        "        # comment the next line of code and define your custom reward otherwise,\n",
        "        # leave as is to use the previously defined custom reward function.\n",
        "        reward = super().calculate()\n",
        "\n",
        "        return reward"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M212veso630o"
      },
      "source": [
        "## Train\n",
        "\n",
        "Here we define one function that performs all the procedures we took to train the SAC agent from selecting buildings, simulation period and active observations to initializing and wrapping the environment, initializing the agent, training it a nd reporting it's results:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UJc3Rvsf668Y"
      },
      "outputs": [],
      "source": [
        "def train_your_custom_sac(\n",
        "    agent_kwargs: dict, episodes: int, reward_function: RewardFunction,\n",
        "    building_count: int, day_count: int, active_observations: List[str],\n",
        "    random_seed: int, reference_envs: Mapping[str, CityLearnEnv] = None,\n",
        "    show_figures: bool = None\n",
        ") -> dict:\n",
        "    \"\"\"Trains a custom soft-actor critic (SAC) agent on a custom environment.\n",
        "\n",
        "    Trains an SAC agent using a custom environment and agent hyperparamter\n",
        "    setup and plots the key performance indicators (KPIs), actions and\n",
        "    rewards from training and evaluating the agent.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    agent_kwargs: dict\n",
        "        Defines the hyperparameters used to initialize the SAC agent.\n",
        "    episodes: int\n",
        "        Number of episodes to train the agent for.\n",
        "    reward_function: RewardFunction\n",
        "        A base or custom reward function class.\n",
        "    building_count: int\n",
        "        Number of buildings to set as active in schema.\n",
        "    day_count: int\n",
        "        Number of simulation days.\n",
        "    active_observations: List[str]\n",
        "        Names of observations to set active to be passed to control agent.\n",
        "    random_seed: int\n",
        "        Seed for pseudo-random number generator.\n",
        "    reference_envs: Mapping[str, CityLearnEnv], default: None\n",
        "        Mapping of user-defined control agent names to environments\n",
        "        the agents have been used to control.\n",
        "    show_figures: bool, default: False\n",
        "        Indicate if summary figures should be plotted at the end of\n",
        "        evaluation.\n",
        "\n",
        "    Returns\n",
        "    -------\n",
        "    result: dict\n",
        "        Results from training the agent as well as some input variables\n",
        "        for reference including the following value keys:\n",
        "\n",
        "            * random_seed: int\n",
        "            * env: CityLearnEnv\n",
        "            * model: SAC\n",
        "            * actions: List[float]\n",
        "            * rewards: List[float]\n",
        "            * agent_kwargs: dict\n",
        "            * episodes: int\n",
        "            * reward_function: RewardFunction\n",
        "            * buildings: List[str]\n",
        "            * simulation_start_time_step: int\n",
        "            * simulation_end_time_step: int\n",
        "            * active_observations: List[str]\n",
        "            * train_start_timestamp: datetime\n",
        "            * train_end_timestamp: datetime\n",
        "    \"\"\"\n",
        "\n",
        "    # get schema\n",
        "    schema = DataSet.get_schema('citylearn_challenge_2022_phase_all')\n",
        "\n",
        "    # select buildings\n",
        "    schema, buildings = set_schema_buildings(\n",
        "        schema, building_count, random_seed\n",
        "    )\n",
        "    print('Selected buildings:', buildings)\n",
        "\n",
        "    # select days\n",
        "    schema, simulation_start_time_step, simulation_end_time_step =\\\n",
        "        set_schema_simulation_period(schema, day_count, random_seed)\n",
        "    print(\n",
        "        f'Selected {day_count}-day period time steps:',\n",
        "        (simulation_start_time_step, simulation_end_time_step)\n",
        "    )\n",
        "\n",
        "    # set active observations\n",
        "    schema = set_active_observations(schema, active_observations)\n",
        "    print(f'Active observations:', active_observations)\n",
        "\n",
        "    # initialize environment\n",
        "    env = CityLearnEnv(schema, central_agent=True)\n",
        "\n",
        "    # set reward function\n",
        "    env.reward_function = reward_function(env=env)\n",
        "\n",
        "    # wrap environment\n",
        "    env = NormalizedObservationWrapper(env)\n",
        "    env = StableBaselines3Wrapper(env)\n",
        "\n",
        "    # initialize agent\n",
        "    model = SAC('MlpPolicy', env, **agent_kwargs, seed=random_seed)\n",
        "\n",
        "    # initialize loader\n",
        "    total_timesteps = episodes*(env.time_steps - 1)\n",
        "    print('Number of episodes to train:', episodes)\n",
        "    loader = get_loader(max=total_timesteps)\n",
        "    display(loader)\n",
        "\n",
        "    # initialize callback\n",
        "    callback = CustomCallback(env=env, loader=loader)\n",
        "\n",
        "    # train agent\n",
        "    train_start_timestamp = datetime.utcnow()\n",
        "    model = model.learn(total_timesteps=total_timesteps, callback=callback)\n",
        "    train_end_timestamp = datetime.utcnow()\n",
        "\n",
        "    # evaluate agent\n",
        "    observations = env.reset()\n",
        "    actions_list = []\n",
        "\n",
        "    while not env.done:\n",
        "        actions, _ = model.predict(observations, deterministic=True)\n",
        "        observations, _, _, _ = env.step(actions)\n",
        "        actions_list.append(actions)\n",
        "\n",
        "    # get rewards\n",
        "    rewards = callback.reward_history[:episodes]\n",
        "\n",
        "    # plot summary and compare with other control results\n",
        "    if show_figures is not None and show_figures:\n",
        "        env_id = 'Your-SAC'\n",
        "\n",
        "        if reference_envs is None:\n",
        "            reference_envs = {env_id: env}\n",
        "        else:\n",
        "            reference_envs = {env_id: env, **reference_envs}\n",
        "\n",
        "        plot_simulation_summary(reference_envs)\n",
        "\n",
        "        # plot actions\n",
        "        plot_actions(actions_list, f'{env_id} Actions')\n",
        "\n",
        "        # plot rewards\n",
        "        _, ax = plt.subplots(1, 1, figsize=(5, 2))\n",
        "        ax = plot_rewards(ax, rewards, f'{env_id} Rewards')\n",
        "        plt.tight_layout()\n",
        "        plt.show()\n",
        "\n",
        "    else:\n",
        "        pass\n",
        "\n",
        "    return {\n",
        "        'random_seed': random_seed,\n",
        "        'env': env,\n",
        "        'model': model,\n",
        "        'actions': actions_list,\n",
        "        'rewards': rewards,\n",
        "        'agent_kwargs': agent_kwargs,\n",
        "        'episodes': episodes,\n",
        "        'reward_function': reward_function,\n",
        "        'buildings': buildings,\n",
        "        'simulation_start_time_step': simulation_start_time_step,\n",
        "        'simulation_end_time_step': simulation_end_time_step,\n",
        "        'active_observations': active_observations,\n",
        "        'train_start_timestamp': train_start_timestamp,\n",
        "        'train_end_timestamp': train_end_timestamp,\n",
        "    }"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ga5ExjjdHkRx"
      },
      "source": [
        "Now, we shall train!!\n",
        "\n",
        "<img src=\"https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExMWU5NzcxNGQzODRiYmI0MzQwNDVlYWU1NjhjODI0ZDhhZDhlNzM3NCZjdD1n/KGYmNdjOUxkFO8JVbM/giphy.gif\" height=200></img>\n",
        "\n",
        "Note that you can use a for loop to train on different `agent_kwargs` combinations in order to find which hyperparameters give the best results."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kFLBpXVA7HQ2"
      },
      "outputs": [],
      "source": [
        "your_results = train_your_custom_sac(\n",
        "    agent_kwargs=your_agent_kwargs,\n",
        "    episodes=your_episodes,\n",
        "    reward_function=YourCustomReward,\n",
        "    building_count=BUILDING_COUNT,\n",
        "    day_count=DAY_COUNT,\n",
        "    active_observations=your_active_observations,\n",
        "    random_seed=RANDOM_SEED,\n",
        "    reference_envs={\n",
        "        'RBC': rbc_env,\n",
        "        # 'TQL': tql_env,\n",
        "        'SAC-1': sac_env,\n",
        "        'SAC-2': sacr_env\n",
        "    },\n",
        "    show_figures=True,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JC5e_3AW8a0w"
      },
      "source": [
        "## Submit\n",
        "\n",
        "You may choose to submit __your results__ to the [scoreboard](https://docs.google.com/spreadsheets/d/1wI1mz7fFiNNc1eZvZfKu_Id23y3QAzL_joVmiqUHm2U/edit?resourcekey#gid=939604299). To this we will programmatically submit your results to a Google Form that live updates the scoreboard in a Google Sheet.\n",
        "\n",
        "Run the following cell to set the function that helps us with the submission.\n",
        "\n",
        "> ⚠️ **NOTE**:\n",
        "> You do not need to understand the content of the next code cell where the result submission function is defined."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-5xzZ4nt8bRL"
      },
      "outputs": [],
      "source": [
        "def post_results(tag: str, results: dict) -> Tuple[dict, requests.Response]:\n",
        "    \"\"\"Submit your trained SAC model results to public scoreboard.\n",
        "\n",
        "    Submits trained SAC model results to a Google Form and results\n",
        "    are displayed and ranked in Google Sheets.\n",
        "\n",
        "    Parameters\n",
        "    ----------\n",
        "    tag: str\n",
        "        A name to use to identify submitted results in scoreboard.\n",
        "        Avoid including personal identifiers in the tag.\n",
        "    results: dict\n",
        "        Mapping of results from your simulation. It is the variable returned\n",
        "        by the :code:`train_your_custom_sac` function.\n",
        "\n",
        "    Returns\n",
        "    -------\n",
        "    payload: dict\n",
        "        Submitted results.\n",
        "    response: requests.Response\n",
        "        Form post request response.\n",
        "    \"\"\"\n",
        "\n",
        "    # submission for ID\n",
        "    form_id = '1FAIpQLSc69VR3t5z7ag6ydvv11mDpdBS8ruhz4yBfWD_81IUZ2IYtEw'\n",
        "\n",
        "    # url to get and fill the form\n",
        "    get_url = f'https://docs.google.com/forms/d/e/{form_id}/viewform?usp=sf_link'\n",
        "\n",
        "    # url to submit the form\n",
        "    post_url = f'https://docs.google.com/forms/u/1/d/e/{form_id}/formResponse'\n",
        "\n",
        "    # get KPIs\n",
        "    kpis = get_kpis(results['env']).pivot(\n",
        "        index='kpi', columns='name', values='value'\n",
        "    ).to_dict()\n",
        "    kpis = {k: {\n",
        "        k_: float(v_) for k_, v_ in v.items() if not math.isnan(v_)\n",
        "    } for k, v in kpis.items()}\n",
        "\n",
        "    # set payload\n",
        "    datetime_fmt = '%Y-%m-%d %H:%M:%S'\n",
        "    buildings = [int(b.split('_')[-1]) for b in results['buildings']]\n",
        "    buildings = sorted(buildings)\n",
        "    buildings = ', '.join([str(b) for b in buildings])\n",
        "    payload = {\n",
        "        'uid': uuid.uuid4().hex,\n",
        "        'create_timestamp': datetime.utcnow().strftime(datetime_fmt),\n",
        "        'train_start_timestamp': results['train_start_timestamp'].strftime(datetime_fmt),\n",
        "        'train_end_timestamp': results['train_end_timestamp'].strftime(datetime_fmt),\n",
        "        'tag': '' if tag is None else tag,\n",
        "        'random_seed': results['random_seed'],\n",
        "        'buildings': buildings,\n",
        "        'simulation_start_time_step': int(results['simulation_start_time_step']),\n",
        "        'simulation_end_time_step': int(results['simulation_end_time_step']),\n",
        "        'episodes': results['episodes'],\n",
        "        'active_observations': ', '.join(sorted(results['active_observations'])),\n",
        "        'agent_name': str(results['model'].__class__),\n",
        "        'agent_kwargs': results['agent_kwargs'],\n",
        "        'reward_function_calculate': inspect.getsource(results['reward_function'].calculate),\n",
        "        'kpis': kpis,\n",
        "        'district_electricity_consumption': kpis['District']['electricity_consumption'],\n",
        "        'district_cost': kpis['District']['cost'],\n",
        "        'district_carbon_emissions': kpis['District']['carbon_emissions'],\n",
        "        'district_ramping': kpis['District']['ramping'],\n",
        "        'district_average_daily_peak': kpis['District']['average_daily_peak'],\n",
        "        'district_load_factor': kpis['District']['1 - load_factor'],\n",
        "    }\n",
        "\n",
        "    # get form question IDs\n",
        "    response = requests.get(get_url)\n",
        "    soup = BeautifulSoup(response.text, 'html.parser')\n",
        "    pattern = re.compile('var FB_PUBLIC_LOAD_DATA_ = (.*?);')\n",
        "    string = soup.findAll(\n",
        "        'script', string=pattern\n",
        "    )[0].string.split(' = ')[-1][:-1]\n",
        "    questions = json.loads(string)[1][1]\n",
        "    questions = {q[1]: q[4][0][0] for q in questions}\n",
        "\n",
        "    # set form question answers\n",
        "    payload = {k: json.dumps(payload[k]) for k, v in questions.items()}\n",
        "    parsed_payload = {f'entry.{questions[k]}': v for k, v in payload.items()}\n",
        "\n",
        "    # submit form\n",
        "    response = requests.post(post_url, data=parsed_payload)\n",
        "\n",
        "    return payload, response"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ljADPiVk8j1U"
      },
      "source": [
        "Finally, run the following cell to set up the submission interface.\n",
        "\n",
        "> ⚠️ **NOTE**:\n",
        "> You do not need to understand the content of the next code cell where the result submission user interface is defined."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lqSZRoZ48lJg"
      },
      "outputs": [],
      "source": [
        "# instructions\n",
        "instructions = \"\"\"\n",
        "<h1>Submit your Results</h1>\n",
        "<p>Use this interactive widget to submit the results of your tuned SAC\n",
        "agent!</p>\n",
        "\n",
        "<p style=\"color:yellow\"><strong>NOTE:</strong> The scoreboard\n",
        "is merely an informational tool. Please, we urge participants\n",
        "to adhere to fair use practices including but not limited to:\n",
        "\n",
        "<ul style=\"color:yellow\">\n",
        "    <li>Do not spam the scoreboard.</li>\n",
        "    <li>Make only one submission for every custom agent\n",
        "    and environment set up.</li>\n",
        "    <li>Do not make alterations to the\n",
        "    <code>post_results</code> function.</li>\n",
        "</ul>\n",
        "\n",
        "</p>\n",
        "\n",
        "<p>Your results are displayed in the\n",
        "<a href=\"https://docs.google.com/spreadsheets\n",
        "/d/1wI1mz7fFiNNc1eZvZfKu_Id23y3QAzL_joVmiqUHm2U/\n",
        "edit?resourcekey#gid=939604299\" target=\"_blank\">scoreboard</a>.</p>\n",
        "\n",
        "\n",
        "<p><strong>Provide a tag (avoid personal identifiers)\n",
        "for your submission and hit the <strong>Submit</strong> button:</strong></p>\n",
        "\"\"\"\n",
        "instructions_html_ui = HTML(value=instructions, placeholder='Instructions')\n",
        "\n",
        "\n",
        "# tag text input\n",
        "tag_text_ui = Text(\n",
        "    value='',\n",
        "    placeholder='Provide a submission tag',\n",
        "    description='Tag:',\n",
        "    disabled=False\n",
        ")\n",
        "\n",
        "# submit button\n",
        "submit_button_ui = Button(\n",
        "    description='Submit',\n",
        "    disabled=True,\n",
        "    button_style='success',\n",
        "    tooltip='Submit your Results',\n",
        "    icon='check'\n",
        ")\n",
        "interactions_ui = HBox([tag_text_ui, submit_button_ui])\n",
        "\n",
        "# post-submission html\n",
        "post_submission_html_ui = HTML(value='', placeholder='Post submission report')\n",
        "\n",
        "def on_tag_value_change(change):\n",
        "    \"\"\"Activate/deactivate submit button based on tag value.\"\"\"\n",
        "\n",
        "    value = tag_text_ui.value.strip(' ')\n",
        "\n",
        "    if len(value) > 0:\n",
        "        submit_button_ui.disabled = False\n",
        "    else:\n",
        "        submit_button_ui.disabled = True\n",
        "\n",
        "def on_submit_button_ui_clicked(b):\n",
        "    \"\"\"Submit your results when submit button is clicked.\"\"\"\n",
        "\n",
        "    # set UI pre-submission states\n",
        "    tag_text_ui.disabled = True\n",
        "    submit_button_ui.disabled = True\n",
        "    current_submit_button_description = submit_button_ui.description\n",
        "    submit_button_ui.description = 'Submitting ...'\n",
        "    tag = tag_text_ui.value.strip()\n",
        "    post_submission_html_ui.value = ''\n",
        "\n",
        "    # make submission\n",
        "    payload, response = post_results(tag, your_results)\n",
        "\n",
        "    # confirm successful submission\n",
        "    try:\n",
        "        assert response.status_code == 200\n",
        "        assert 'Your response has been recorded' in response.text\n",
        "        post_submission_html = f\"\"\"\n",
        "        <p style=\"color:green\">Your last submission\n",
        "        on \"{payload['create_timestamp'].strip('\"')} UTC\"\n",
        "        with tag: {payload['tag']}\n",
        "        and unique ID: {payload['uid']}\n",
        "        was successful!</p>\n",
        "        \"\"\"\n",
        "\n",
        "    except AssertionError:\n",
        "        post_submission_html = f\"\"\"\n",
        "        <p style=\"color:red\">Your last submission\n",
        "        on \"{payload['create_timestamp'].strip('\"')} UTC\"\n",
        "        with tag: {payload['tag']}\n",
        "        was unsuccessful!</p>\n",
        "        \"\"\"\n",
        "\n",
        "\n",
        "    # set UI post-submission states\n",
        "    submit_button_ui.description = current_submit_button_description\n",
        "    tag_text_ui.value = ''\n",
        "    tag_text_ui.disabled = False\n",
        "    submit_button_ui.disabled = False\n",
        "    post_submission_html_ui.value = post_submission_html\n",
        "\n",
        "\n",
        "# callbacks\n",
        "tag_text_ui.observe(on_tag_value_change, names='value')\n",
        "submit_button_ui.on_click(on_submit_button_ui_clicked)\n",
        "\n",
        "# show UI\n",
        "ui = VBox([instructions_html_ui, interactions_ui, post_submission_html_ui])\n",
        "display(ui)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZhkmytKNU_Z2"
      },
      "source": [
        "# Next Steps\n",
        "---\n",
        "\n",
        "Now that you are a _CityLearner_, here are some next steps and ideas (asides the awesome ideas you probably already have of course 😉):\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sc1_SxLlLoqU"
      },
      "source": [
        "## The CityLearn Challenge 2023\n",
        "\n",
        "[The CityLearn Challenge 2023](https://www.aicrowd.com/challenges/neurips-2023-citylearn-challenge) addresses this multi-faceted nature of advanced control in buildings by blending the challenges of control algorithm design, forecast quality and grid-resilience. The CityLearn Challenge 2023 presents a control track as done in previous challenges as well as introduces an independent forecast track where, both tracks are run in parallel and utilize the same dataset.\n",
        "\n",
        "In the control track, participants will develop energy management agent(s) and an optional custom reward function (in RLC solutions) to manage electrical and domestic hot water energy storage systems, and heat pump power in a synthetic single-family neighborhood under normal grid-operation and power outages. Whereas, in the forecast problem, participants will design regression models to predict the 48-hour-ahead end-use load profiles for each building in the neighborhood as well as the neighborhood-level 48-hour-ahead solar generation and carbon intensity profiles."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-PEwzcNdLYsK"
      },
      "source": [
        "## Other Ideas\n",
        "\n",
        "- Rerun the entire tutorial with a new [RANDOM_SEED](#scrollTo=vfnO0QBszXcS&line=1&uniqifier=1), [number of buildings](#scrollTo=6C6S46xmz50t&line=2&uniqifier=1) (between 1 - 15), [number of days](#scrollTo=6C6S46xmz50t&line=5&uniqifier=1) (1 - 365) and/or [observations](#scrollTo=6C6S46xmz50t&line=8&uniqifier=1). Remember to [set the number of discretization bins](#scrollTo=_6HotiSW4Pe8&line=2&uniqifier=1) for Tabular Q-Learning if you use other observations in your simulations.\n",
        "- How does the Tabular Q-Learning agent perform with a different set of hyperparameters and/or active observations?\n",
        "- How well does the Tabular Q-Learning learn if we use the custom reward function we defined? Are there any improvements compared to the original reward function?\n",
        "- Try to train the SAC agent on all the buildings and the full one-year period in the `citylearn_challenge_2022_phase_all` dataset.\n",
        "- Can you still improve some KPIs without self-generation in the buildings i.e. no photovoltaic (PV) system?\n",
        "- In our hand-on experiments here, we trained and tested on the same days. In reality, when an RL agent is deployed, it may experience states and state transitions that were not seen during training. Try to evaluate your trained agent on a different sequence of days and see if your trained agent generalizes well.\n",
        "- Try out the other datasets in CityLearn.\n",
        "- Bring your own dataset to CityLearn!\n",
        "- \\<Insert __YOUR__ ideas 🙂\\>\n",
        "\n",
        "<img src=\"https://media.giphy.com/media/3ohs86vZAWiJXWvQI0/giphy.gif\" height=200></img>"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "private_outputs": true,
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "examples_env",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.7"
    },
    "vscode": {
      "interpreter": {
        "hash": "aef885a76ce31739e452d1e6967b400907b14827afd25732d0e38ec88d4e0d05"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}