markov_decision_process/deep_q_learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8d4f2d8f",
   "metadata": {},
   "source": [
    "# Deep Q-learning Tutorial\n",
    "\n",
    "In this series of notebooks, I have introduced various policy iteration algorithms to solve Markov Decision Processes (MDPs). While these methods, which utilize tabular estimates of state/action values, work well in simpler settings, they face challenges in complex environments due to the great number of states and actions, making it impractical to learn with tabular estimates.\n",
    "\n",
    "One solution to this problem is to employ a sophisticated function to efficiently estimate values. In this notebook, I will introduce one such approach known as Deep Q-learning (DQN). DQN utilizes a neural network to efficiently estimate state values. To address challenges when applying a neural network to estimate the value function, DQN incorporates techniques such as experience replay and a target Q-network.\n",
    "\n",
    "We will learn these crucial conceptual elements with the implementation. So, let's get started"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c84dc2f6",
   "metadata": {},
   "source": [
    "___"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d371fb91",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import packages\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "from torch.autograd import Variable\n",
    "from typing import List, Tuple\n",
    "\n",
    "import collections\n",
    "from collections import namedtuple, deque\n",
    "import tqdm\n",
    "import matplotlib.pyplot as plt\n",
    "import random\n",
    "import gymnasium as gym\n",
    "\n",
    "from IPython.display import clear_output\n",
    "from IPython import display\n",
    "\n",
    "%matplotlib inline\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6e3646f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# set random seeds\n",
    "torch.manual_seed(0)\n",
    "np.random.seed(0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d981828",
   "metadata": {},
   "source": [
    "### Prepare environment"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d55f390",
   "metadata": {},
   "source": [
    "In this notebook, we utilize the CartPole-v0 environment in Gymnasium.\n",
    "\n",
    "[Learn more about the CartPole-v0 environment](https://gymnasium.farama.org/environments/classic_control/cart_pole/)\n",
    "\n",
    "The goal of this environment is to balance a pole by applying forces in the left and right directions on the cart. It has a discrete action space:\n",
    "- 0: Push cart to the left\n",
    "- 1: Push cart to the right\n",
    "\n",
    "Upon taking an action, either left or right, an agent observes a 4-dimensional state consisting of:\n",
    "- Cart Position\n",
    "- Cart Velocity\n",
    "- Pole Angle\n",
    "- Pole Angular Velocity\n",
    "\n",
    "A reward of +1 is granted to the agent at each step while the pole is kept upright. The maximum reward an agent can earn in a single episode is 200.\n",
    "\n",
    "The episode ends under the following conditions:\n",
    "- Termination: Pole Angle is greater than ±12°\n",
    "- Termination: Cart Position is greater than ±2.4 (center of the cart reaches the edge of the display)\n",
    "- Truncation: Episode length exceeds 200 steps\n",
    "\n",
    "In the code below, I provide an example of the agent randomly exploring this environment over 100 time steps.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "391e4be9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create CartPole environment\n",
    "env = gym.make('CartPole-v0', render_mode='rgb_array')\n",
    "state, _ = env.reset()\n",
    "\n",
    "# Run the environment for 100 steps\n",
    "for i in range(100):\n",
    "    # Display the current state of the environment\n",
    "    plt.imshow(env.render())\n",
    "    display.display(plt.gcf())\n",
    "    display.clear_output(wait=True)\n",
    "    \n",
    "    # Choose a random action from the action space\n",
    "    action = env.action_space.sample()\n",
    "    \n",
    "    # Take the chosen action and observe the next state, reward, and termination status\n",
    "    state, reward, terminated, truncated, info = env.step(action)\n",
    "    \n",
    "    # If the episode is terminated or truncated, reset the environment\n",
    "    if terminated or truncated:\n",
    "        state, info = env.reset()\n",
    "\n",
    "# Close the environment after exploration\n",
    "env.close()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "825ceeb8",
   "metadata": {},
   "source": [
    "### Define Q-Network\n",
    "\n",
    "Let's begin by preparing a model to estimate a value function. Here, we assume that the environment dynamics is unknow and the agent needs to learn the optimal policy by interacting with the environment. It's important to note that the state space of CartPole-v0 is continuous, and the number of possible states is infinitely large. Therefore, applying a tabular estimate strategy, as done in Monte Carlo Methods or TD-learning, is not feasible. This is why we use a model to estimate the value based on the observed state.\n",
    "\n",
    "For Deep Q-Network (DQN), we employ a deep learning model as a value function estimator. The estimated value function is referred to as the Q-value, and the neural network used for this purpose is known as the Q-network. This network takes a state ($\\mathbb{R}^{Nx4}$) as an input and outputs Q values of the actions available to the agent ($\\mathbb{R}^{Nx2}$). To keep it simple, we will utilize a fully connected neural network with ReLU activation. We use the Adam optimizer for training.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b8ab7512",
   "metadata": {},
   "outputs": [],
   "source": [
    "class FullyConnectedModel(nn.Module):\n",
    "    def __init__(self, input_size, output_size):\n",
    "        super(FullyConnectedModel, self).__init__()\n",
    "\n",
    "        # Define layers with ReLU activation\n",
    "        self.linear1 = nn.Linear(input_size, 16)\n",
    "        self.activation1 = nn.ReLU()\n",
    "        self.linear2 = nn.Linear(16, 16)\n",
    "        self.activation2 = nn.ReLU()\n",
    "        self.linear3 = nn.Linear(16, 16)\n",
    "        self.activation3 = nn.ReLU()\n",
    "\n",
    "        # Output layer without activation function\n",
    "        self.output_layer = nn.Linear(16, output_size)\n",
    "\n",
    "        # Initialization using Xavier uniform (a popular technique for initializing weights in NNs)\n",
    "        nn.init.xavier_uniform_(self.linear1.weight)\n",
    "        nn.init.xavier_uniform_(self.linear2.weight)\n",
    "        nn.init.xavier_uniform_(self.linear3.weight)\n",
    "        nn.init.xavier_uniform_(self.output_layer.weight)\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        # Forward pass through the layers\n",
    "        x = self.activation1(self.linear1(inputs))\n",
    "        x = self.activation2(self.linear2(x))\n",
    "        x = self.activation3(self.linear3(x))\n",
    "        x = self.output_layer(x)\n",
    "        return x\n",
    "\n",
    "    \n",
    "class QNetwork:\n",
    "    def __init__(self, env, lr, logdir=None):\n",
    "        # Define Q-network with specified architecture\n",
    "        self.net = FullyConnectedModel(4, 2)\n",
    "        self.env = env\n",
    "        self.lr = lr \n",
    "        self.logdir = logdir\n",
    "        self.optimizer = optim.Adam(self.net.parameters(), lr=self.lr)\n",
    "\n",
    "    def load_model(self, model_file):\n",
    "        # Load pre-trained model from a file\n",
    "        return self.net.load_state_dict(torch.load(model_file))\n",
    "\n",
    "    def load_model_weights(self, weight_file):\n",
    "        # Load pre-trained model weights from a file\n",
    "        return self.net.load_state_dict(torch.load(weight_file))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "848de8bc",
   "metadata": {},
   "source": [
    "Note: The Xavier initialization (or Glorot initialization) is a popular technique for initializing weights in a neural network. For more information, you can check [this article](https://365datascience.com/tutorials/machine-learning-tutorials/what-is-xavier-initialization/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b8800e5",
   "metadata": {},
   "source": [
    "### Define Replay Memory\n",
    "\n",
    "Next, we introduce one of the key components of DQN, Experience Replay.\n",
    "\n",
    "While deep learning models or other complex ML models for estimating a value function may seem appealing, they did not enjoy success in RL for a long time until Experience Replay was introduced. This is due to a correlation within the training dataset. In supervised learning, a crucial assumption is that all training data is independent and identically distributed (i.i.d). However, consecutive samples collected during the agent's interactions with the environment in RL are highly correlated, breaking this assumption and causing issues in fitting a machine learning model.\n",
    "\n",
    "Experience Replay mitigates this temporal correlation by randomly sampling experiences from a replay memory. This approach helps decorrelate training samples, promoting better convergence. By storing and reusing past experiences, Experience Replay enables the agent to learn from a diverse set of transitions, making more efficient use of the collected data. This diversity aids in better exploration and can lead to more stable and faster convergence of the Q-network.\n",
    "\n",
    "Specifically, we first define a replay memory with a specified memory size. On each iteration, we store a new observation in this replay memory and randomly sample data from this memory to train a model. This random sampling from a replay memory breaks the correlational structure in the training data.\n",
    "\n",
    "Because of Replay Memory, DQN is classified as an off-policy method. On-Policy methods use the same policy for both collecting samples and updating the value function. In contrast, Off-Policy methods use two distinct policies for collecting samples and updating a value function, respectively. For DQN, experience samples are collected with an epsilon-greedy policy by using the Q-network estimate at the time. The accumulated experience in the memory buffer is then used to update the Q-network, consequently updating the policy by taking greedy action baed on Q-network estimate. The data accumulated in the memory buffer does not follow the same policy as the current policy because we keep updating the Q-network. Thus, the data used to update our target policy is collected with different policies (i.e., Off-policy).\n",
    "\n",
    "Let's set up the replay memory.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "05af9a76",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ReplayMemory:\n",
    "    def __init__(self, env, memory_size=50000, burn_in=10000):\n",
    "        # Initializes the replay memory, which stores transitions recorded from the agent taking actions in the environment.\n",
    "        self.memory_size = memory_size\n",
    "        self.burn_in = burn_in\n",
    "        self.memory = collections.deque([], maxlen=memory_size)\n",
    "        self.env = env\n",
    "\n",
    "    def sample_batch(self, batch_size=32):\n",
    "        # Returns a batch of randomly sampled transitions to be used for training the model.\n",
    "        return random.sample(self.memory, batch_size)\n",
    "\n",
    "    def append(self, transition):\n",
    "        # Appends a transition to the replay memory.\n",
    "        self.memory.append(transition)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5052bb56",
   "metadata": {},
   "source": [
    "### Q-Value Function Update\n",
    "\n",
    "As we discussed in the above section, to address the correlation problem in training data, we leverage a replay buffer. Samples extracted from the memory buffer are used to train our model. Let's learn the detailed process of updating a model.\n",
    "\n",
    "Firstly, we utilize the Temporal Difference (TD) error to update the value of a state and action pair, following the principles of Q-learning, which is why this approach is termed Deep \"Q-learning\". Q-learning is an Off-policy TD learning algorithm.\n",
    "\n",
    "The TD error is given by the formula:\n",
    "\n",
    "$$ (R_{t+1} + \\gamma \\max_a Q(S_{t+1}, a)) - Q(S_t, A_t) $$\n",
    "\n",
    "A crucial difference between an original Q-learning and DQN is that the current estimate of the Q-value $Q(S_t, A_t)$ and $Q(S_{t+1}, a)$ is obtained from a deep learning model. The model updates the prediction to minimize this TD error, meaning it uses the mean squared error between the estimated Q-value and the sum of the immediate reward and the value of the best next state-aciton pair.\n",
    "\n",
    "However, there is a challenge. While our aim is to update the Q-value estimate by minimizing the TD error, the computation of the next state value $Q(S_{t+1}, a)$ within TD error equation also relies on the network, which we update at each iteration. As our value estimate for the next state fluctuates every iteration due to the network update, the error fluctuates too, leading to instability in the Q-network fitting.\n",
    "\n",
    "To overcome this issue, DQN introduces another neural network called the target network. This network is updated more slowly compared to the policy Q-network, which we use to define the optimal policy based on the value estimation. Unlike the policy network, the target network is not updated every iteration. Instead, it is updated once in a while to stabilize the learning process.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "feef98c3",
   "metadata": {},
   "source": [
    "### Define DQN Agent\n",
    "\n",
    "Now that we have covered all the essential components (Deep Learning Model, Replay Memory, and Target Network), we can proceed with the implementation to train a DQN agent. The training of DQN in the below implementation roughly follows these steps:\n",
    "\n",
    "1. Initialize Q-network and Target Network.\n",
    "2. Initialize Replay Memory using burn-in with a random action policy.\n",
    "3. Initialize the environment.\n",
    "4. For each episode from 1 to M:\n",
    "    1. Choose an action based on the current Q-value estimate using an epsilon-greedy policy.\n",
    "    2. Take a step and observe the next state and reward.\n",
    "    3. Store the new experience into the replay memory.\n",
    "    4. Sample a minibatch with size N from the replay memory.\n",
    "    5. Train the network with the batched dataset.\n",
    "    6. Go back to step 1.\n",
    "5. Every 10 episodes, perform 20 test episodes to evaluate the performance of the current agent.\n",
    "\n",
    "Let's explore the implementation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "aee0c0bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "Transition = namedtuple('Transition', ('state', 'action', 'next_state', 'reward'))\n",
    "\n",
    "class DQN_Agent:\n",
    "\n",
    "    def __init__(self, environment_name, lr=5e-4, render=False):\n",
    "        # Initialize the DQN Agent.\n",
    "        self.env = gym.make(environment_name)\n",
    "        self.lr = lr\n",
    "        self.policy_net = QNetwork(self.env, self.lr)\n",
    "        self.target_net = QNetwork(self.env, self.lr)\n",
    "        self.target_net.net.load_state_dict(self.policy_net.net.state_dict())  # Copy the weight of the policy network\n",
    "        self.rm = ReplayMemory(self.env)\n",
    "        self.burn_in_memory()\n",
    "        self.batch_size = 32\n",
    "        self.gamma = 0.99\n",
    "        self.c = 0\n",
    "\n",
    "    def burn_in_memory(self):\n",
    "        # Initialize replay memory with a burn-in number of episodes/transitions.\n",
    "        cnt = 0\n",
    "        terminated = False\n",
    "        truncated = False\n",
    "        state, _ = self.env.reset()\n",
    "        state = torch.tensor(state, dtype=torch.float32).unsqueeze(0)\n",
    "\n",
    "        # Iterate until we store \"burn_in\" buffer\n",
    "        while cnt < self.rm.burn_in:\n",
    "            # Reset environment if terminated or truncated\n",
    "            if terminated or truncated:\n",
    "                state, _ = self.env.reset()\n",
    "                state = torch.tensor(state, dtype=torch.float32).unsqueeze(0)\n",
    "            \n",
    "            # Randomly select an action (left or right) and take a step\n",
    "            action = torch.tensor(random.sample([0, 1], 1)[0]).reshape(1, 1)\n",
    "            next_state, reward, terminated, truncated, _ = self.env.step(action.item())\n",
    "            reward = torch.tensor([reward])\n",
    "            if terminated:\n",
    "                next_state = None\n",
    "            else:\n",
    "                next_state = torch.tensor(next_state, dtype=torch.float32).unsqueeze(0)\n",
    "                \n",
    "            # Store new experience into memory\n",
    "            transition = Transition(state, action, next_state, reward)\n",
    "            self.rm.memory.append(transition)\n",
    "            state = next_state\n",
    "            cnt += 1\n",
    "\n",
    "    def epsilon_greedy_policy(self, q_values, epsilon=0.05):\n",
    "        # Implement an epsilon-greedy policy. \n",
    "        p = random.random()\n",
    "        if p > epsilon:\n",
    "            with torch.no_grad():\n",
    "                return self.greedy_policy(q_values)\n",
    "        else:\n",
    "            return torch.tensor([[self.env.action_space.sample()]], dtype=torch.long)\n",
    "\n",
    "    def greedy_policy(self, q_values):\n",
    "        # Implement a greedy policy for test time.\n",
    "        return torch.argmax(q_values)\n",
    "        \n",
    "    def train(self):\n",
    "        # Train the Q-network using Deep Q-learning.\n",
    "        state, _ = self.env.reset()\n",
    "        state = torch.tensor(state, dtype=torch.float32).unsqueeze(0)\n",
    "        terminated = False\n",
    "        truncated = False\n",
    "\n",
    "        # Loop until reaching the termination state\n",
    "        while not (terminated or truncated):\n",
    "            with torch.no_grad():\n",
    "                q_values = self.policy_net.net(state)\n",
    "\n",
    "            # Decide the next action with epsilon greedy strategy\n",
    "            action = self.epsilon_greedy_policy(q_values).reshape(1, 1)\n",
    "            \n",
    "            # Take action and observe reward and next state\n",
    "            next_state, reward, terminated, truncated, _ = self.env.step(action.item())\n",
    "            reward = torch.tensor([reward])\n",
    "            if terminated:\n",
    "                next_state = None\n",
    "            else:\n",
    "                next_state = torch.tensor(next_state, dtype=torch.float32).unsqueeze(0)\n",
    "\n",
    "            # Store the new experience\n",
    "            transition = Transition(state, action, next_state, reward)\n",
    "            self.rm.memory.append(transition)\n",
    "\n",
    "            # Move to the next state\n",
    "            state = next_state\n",
    "\n",
    "            # Sample minibatch with size N from memory\n",
    "            transitions = self.rm.sample_batch(self.batch_size)\n",
    "            batch = Transition(*zip(*transitions))\n",
    "            non_final_mask = torch.tensor(tuple(map(lambda s: s is not None, batch.next_state)), dtype=torch.bool)\n",
    "            non_final_next_states = torch.cat([s for s in batch.next_state if s is not None])\n",
    "            state_batch = torch.cat(batch.state)\n",
    "            action_batch = torch.cat(batch.action)\n",
    "            reward_batch = torch.cat(batch.reward)\n",
    "\n",
    "            # Get current and next state values\n",
    "            state_action_values = self.policy_net.net(state_batch).gather(1, action_batch) # extract values corresponding to the actions Q(S_t, A_t)\n",
    "            next_state_values = torch.zeros(self.batch_size)\n",
    "            \n",
    "            with torch.no_grad():\n",
    "                # no next_state_value update if an episode is terminated (next_satate = None)\n",
    "                # only update the non-termination state values (Ref: https://gymnasium.farama.org/tutorials/gymnasium_basics/handling_time_limits/)\n",
    "                next_state_values[non_final_mask] = self.target_net.net(non_final_next_states).max(1)[0] # extract max value\n",
    "                \n",
    "            # Update the model\n",
    "            expected_state_action_values = (next_state_values * self.gamma) + reward_batch\n",
    "            criterion = torch.nn.MSELoss()\n",
    "            loss = criterion(state_action_values, expected_state_action_values.unsqueeze(1))\n",
    "            self.policy_net.optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            self.policy_net.optimizer.step()\n",
    "\n",
    "            # Update the target Q-network in each 50 steps\n",
    "            self.c += 1\n",
    "            if self.c % 50 == 0:\n",
    "                self.target_net.net.load_state_dict(self.policy_net.net.state_dict())\n",
    "\n",
    "    def test(self, model_file=None):\n",
    "        # Evaluates the performance of the agent over 20 episodes.\n",
    "\n",
    "        max_t = 1000\n",
    "        state, _ = self.env.reset()\n",
    "        rewards = []\n",
    "\n",
    "        for t in range(max_t):\n",
    "            state = torch.from_numpy(state).float().unsqueeze(0)\n",
    "            with torch.no_grad():\n",
    "                q_values = self.policy_net.net(state)\n",
    "            action = self.greedy_policy(q_values)\n",
    "            state, reward, terminated, truncated, _ = self.env.step(action.item())\n",
    "            rewards.append(reward)\n",
    "            if terminated or truncated:\n",
    "                break\n",
    "\n",
    "        return np.sum(rewards)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff675006",
   "metadata": {},
   "source": [
    "### Train and Test DQN Agent\n",
    "\n",
    "Let's train a DQN agent to evaluate its performance on the CartPole-v0 environment. In this environment, the maximum achievable reward for an agent in each episode is 200. We will train the model over 200 episodes, evaluating its performance every 10 episodes with an additional 20 test episodes. This training process will be repeated five times, allowing us to assess the average performance of the agent across episodes.\n",
    "\n",
    "Please note that this section may take a few minutes to complete.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "1d0bae4c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/minaiyuki/anaconda3/envs/py37/lib/python3.7/site-packages/gymnasium/envs/registration.py:524: DeprecationWarning: \u001b[33mWARN: The environment CartPole-v0 is out of date. You should consider upgrading to version `v1`.\u001b[0m\n",
      "  f\"The environment {env_name} is out of date. You should consider \"\n",
      "  0%|                                                     | 0/5 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 0\n",
      "The test reward for episode 0 is 58.55 with a standard deviation of 26.261140493131673.\n",
      "Episode: 10\n",
      "The test reward for episode 10 is 9.15 with a standard deviation of 0.7921489758877431.\n",
      "Episode: 20\n",
      "The test reward for episode 20 is 9.5 with a standard deviation of 0.6708203932499369.\n",
      "Episode: 30\n",
      "The test reward for episode 30 is 9.35 with a standard deviation of 0.5722761571129799.\n",
      "Episode: 40\n",
      "The test reward for episode 40 is 9.5 with a standard deviation of 0.7416198487095663.\n",
      "Episode: 50\n",
      "The test reward for episode 50 is 9.4 with a standard deviation of 0.7348469228349535.\n",
      "Episode: 60\n",
      "The test reward for episode 60 is 9.2 with a standard deviation of 0.8717797887081347.\n",
      "Episode: 70\n",
      "The test reward for episode 70 is 19.9 with a standard deviation of 2.4269322199023198.\n",
      "Episode: 80\n",
      "The test reward for episode 80 is 61.7 with a standard deviation of 38.1.\n",
      "Episode: 90\n",
      "The test reward for episode 90 is 80.7 with a standard deviation of 20.149689823915402.\n",
      "Episode: 100\n",
      "The test reward for episode 100 is 129.5 with a standard deviation of 42.94589619509645.\n",
      "Episode: 110\n",
      "The test reward for episode 110 is 182.0 with a standard deviation of 30.602287496198713.\n",
      "Episode: 120\n",
      "The test reward for episode 120 is 195.0 with a standard deviation of 14.484474446799926.\n",
      "Episode: 130\n",
      "The test reward for episode 130 is 192.4 with a standard deviation of 18.014993755202916.\n",
      "Episode: 140\n",
      "The test reward for episode 140 is 194.15 with a standard deviation of 12.709346954112158.\n",
      "Episode: 150\n",
      "The test reward for episode 150 is 199.7 with a standard deviation of 1.3076696830622017.\n",
      "Episode: 160\n",
      "The test reward for episode 160 is 193.55 with a standard deviation of 15.75904502182794.\n",
      "Episode: 170\n",
      "The test reward for episode 170 is 191.15 with a standard deviation of 20.42125118595822.\n",
      "Episode: 180\n",
      "The test reward for episode 180 is 196.35 with a standard deviation of 5.952100469582145.\n",
      "Episode: 190\n",
      "The test reward for episode 190 is 199.85 with a standard deviation of 0.6538348415311009.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 20%|█████████                                    | 1/5 [00:23<01:33, 23.31s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 0\n",
      "The test reward for episode 0 is 9.55 with a standard deviation of 0.739932429347437.\n",
      "Episode: 10\n",
      "The test reward for episode 10 is 9.75 with a standard deviation of 0.82915619758885.\n",
      "Episode: 20\n",
      "The test reward for episode 20 is 9.4 with a standard deviation of 0.7348469228349535.\n",
      "Episode: 30\n",
      "The test reward for episode 30 is 9.3 with a standard deviation of 0.714142842854285.\n",
      "Episode: 40\n",
      "The test reward for episode 40 is 8.95 with a standard deviation of 0.5894913061275798.\n",
      "Episode: 50\n",
      "The test reward for episode 50 is 9.3 with a standard deviation of 0.7810249675906654.\n",
      "Episode: 60\n",
      "The test reward for episode 60 is 9.35 with a standard deviation of 0.6538348415311012.\n",
      "Episode: 70\n",
      "The test reward for episode 70 is 9.4 with a standard deviation of 0.66332495807108.\n",
      "Episode: 80\n",
      "The test reward for episode 80 is 9.25 with a standard deviation of 0.698212002188447.\n",
      "Episode: 90\n",
      "The test reward for episode 90 is 17.0 with a standard deviation of 2.3021728866442674.\n",
      "Episode: 100\n",
      "The test reward for episode 100 is 42.8 with a standard deviation of 12.694880858046679.\n",
      "Episode: 110\n",
      "The test reward for episode 110 is 143.0 with a standard deviation of 51.69139193328034.\n",
      "Episode: 120\n",
      "The test reward for episode 120 is 188.8 with a standard deviation of 19.18749592833833.\n",
      "Episode: 130\n",
      "The test reward for episode 130 is 199.4 with a standard deviation of 2.034698994937581.\n",
      "Episode: 140\n",
      "The test reward for episode 140 is 199.4 with a standard deviation of 2.6153393661244038.\n",
      "Episode: 150\n",
      "The test reward for episode 150 is 196.1 with a standard deviation of 8.791473141629906.\n",
      "Episode: 160\n",
      "The test reward for episode 160 is 195.9 with a standard deviation of 8.659676668328904.\n",
      "Episode: 170\n",
      "The test reward for episode 170 is 196.7 with a standard deviation of 8.672369918309528.\n",
      "Episode: 180\n",
      "The test reward for episode 180 is 183.8 with a standard deviation of 15.806960492137632.\n",
      "Episode: 190\n",
      "The test reward for episode 190 is 158.65 with a standard deviation of 17.23158437288922.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 40%|██████████████████                           | 2/5 [00:42<01:02, 20.77s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 0\n",
      "The test reward for episode 0 is 10.15 with a standard deviation of 0.8529361054615991.\n",
      "Episode: 10\n",
      "The test reward for episode 10 is 9.55 with a standard deviation of 0.8046738469715539.\n",
      "Episode: 20\n",
      "The test reward for episode 20 is 9.25 with a standard deviation of 0.5361902647381804.\n",
      "Episode: 30\n",
      "The test reward for episode 30 is 9.45 with a standard deviation of 0.6689544080129826.\n",
      "Episode: 40\n",
      "The test reward for episode 40 is 9.4 with a standard deviation of 0.7348469228349535.\n",
      "Episode: 50\n",
      "The test reward for episode 50 is 9.55 with a standard deviation of 0.6689544080129826.\n",
      "Episode: 60\n",
      "The test reward for episode 60 is 9.3 with a standard deviation of 0.8426149773176357.\n",
      "Episode: 70\n",
      "The test reward for episode 70 is 12.7 with a standard deviation of 1.3820274961085257.\n",
      "Episode: 80\n",
      "The test reward for episode 80 is 38.75 with a standard deviation of 4.836062447901185.\n",
      "Episode: 90\n",
      "The test reward for episode 90 is 116.9 with a standard deviation of 17.469688033848804.\n",
      "Episode: 100\n",
      "The test reward for episode 100 is 177.6 with a standard deviation of 19.67587355112855.\n",
      "Episode: 110\n",
      "The test reward for episode 110 is 193.75 with a standard deviation of 11.202120335008011.\n",
      "Episode: 120\n",
      "The test reward for episode 120 is 192.75 with a standard deviation of 10.760460027340839.\n",
      "Episode: 130\n",
      "The test reward for episode 130 is 192.7 with a standard deviation of 11.606463716395274.\n",
      "Episode: 140\n",
      "The test reward for episode 140 is 191.65 with a standard deviation of 14.616001505199703.\n",
      "Episode: 150\n",
      "The test reward for episode 150 is 175.2 with a standard deviation of 21.371944226017433.\n",
      "Episode: 160\n",
      "The test reward for episode 160 is 178.35 with a standard deviation of 24.66835016777571.\n",
      "Episode: 170\n",
      "The test reward for episode 170 is 191.2 with a standard deviation of 15.907859692617357.\n",
      "Episode: 180\n",
      "The test reward for episode 180 is 196.95 with a standard deviation of 9.216696805255124.\n",
      "Episode: 190\n",
      "The test reward for episode 190 is 190.6 with a standard deviation of 17.48256274120016.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 60%|███████████████████████████                  | 3/5 [01:06<00:44, 22.39s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 0\n",
      "The test reward for episode 0 is 9.65 with a standard deviation of 0.6538348415311012.\n",
      "Episode: 10\n",
      "The test reward for episode 10 is 9.35 with a standard deviation of 0.7262919523166975.\n",
      "Episode: 20\n",
      "The test reward for episode 20 is 9.25 with a standard deviation of 0.7664854858377946.\n",
      "Episode: 30\n",
      "The test reward for episode 30 is 9.5 with a standard deviation of 0.5916079783099616.\n",
      "Episode: 40\n",
      "The test reward for episode 40 is 9.1 with a standard deviation of 0.7681145747868608.\n",
      "Episode: 50\n",
      "The test reward for episode 50 is 33.15 with a standard deviation of 6.828433202426454.\n",
      "Episode: 60\n",
      "The test reward for episode 60 is 81.1 with a standard deviation of 24.306172055673432.\n",
      "Episode: 70\n",
      "The test reward for episode 70 is 117.75 with a standard deviation of 35.71116772103651.\n",
      "Episode: 80\n",
      "The test reward for episode 80 is 180.6 with a standard deviation of 24.432355596626376.\n",
      "Episode: 90\n",
      "The test reward for episode 90 is 197.0 with a standard deviation of 9.268225288586807.\n",
      "Episode: 100\n",
      "The test reward for episode 100 is 183.1 with a standard deviation of 20.96163161588334.\n",
      "Episode: 110\n",
      "The test reward for episode 110 is 174.85 with a standard deviation of 24.28224660116934.\n",
      "Episode: 120\n",
      "The test reward for episode 120 is 197.15 with a standard deviation of 7.458384543585829.\n",
      "Episode: 130\n",
      "The test reward for episode 130 is 193.8 with a standard deviation of 13.735355838128113.\n",
      "Episode: 140\n",
      "The test reward for episode 140 is 194.95 with a standard deviation of 12.812006088041013.\n",
      "Episode: 150\n",
      "The test reward for episode 150 is 194.25 with a standard deviation of 10.133484099755622.\n",
      "Episode: 160\n",
      "The test reward for episode 160 is 199.45 with a standard deviation of 1.6575584454250776.\n",
      "Episode: 170\n",
      "The test reward for episode 170 is 200.0 with a standard deviation of 0.0.\n",
      "Episode: 180\n",
      "The test reward for episode 180 is 200.0 with a standard deviation of 0.0.\n",
      "Episode: 190\n",
      "The test reward for episode 190 is 199.6 with a standard deviation of 1.7435595774162693.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      " 80%|████████████████████████████████████         | 4/5 [01:34<00:24, 24.47s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 0\n",
      "The test reward for episode 0 is 9.15 with a standard deviation of 0.7262919523166975.\n",
      "Episode: 10\n",
      "The test reward for episode 10 is 9.6 with a standard deviation of 0.7999999999999999.\n",
      "Episode: 20\n",
      "The test reward for episode 20 is 9.2 with a standard deviation of 0.6782329983125269.\n",
      "Episode: 30\n",
      "The test reward for episode 30 is 9.2 with a standard deviation of 0.6782329983125268.\n",
      "Episode: 40\n",
      "The test reward for episode 40 is 9.55 with a standard deviation of 0.739932429347437.\n",
      "Episode: 50\n",
      "The test reward for episode 50 is 9.4 with a standard deviation of 1.1575836902790224.\n",
      "Episode: 60\n",
      "The test reward for episode 60 is 9.4 with a standard deviation of 0.8602325267042626.\n",
      "Episode: 70\n",
      "The test reward for episode 70 is 21.25 with a standard deviation of 3.0639027399707057.\n",
      "Episode: 80\n",
      "The test reward for episode 80 is 46.7 with a standard deviation of 9.34933152690608.\n",
      "Episode: 90\n",
      "The test reward for episode 90 is 77.4 with a standard deviation of 31.355063386955543.\n",
      "Episode: 100\n",
      "The test reward for episode 100 is 194.35 with a standard deviation of 10.84562123624092.\n",
      "Episode: 110\n",
      "The test reward for episode 110 is 164.5 with a standard deviation of 16.847848527334286.\n",
      "Episode: 120\n",
      "The test reward for episode 120 is 154.9 with a standard deviation of 11.063001401066531.\n",
      "Episode: 130\n",
      "The test reward for episode 130 is 151.15 with a standard deviation of 15.982099361473134.\n",
      "Episode: 140\n",
      "The test reward for episode 140 is 145.65 with a standard deviation of 17.65863811283305.\n",
      "Episode: 150\n",
      "The test reward for episode 150 is 164.5 with a standard deviation of 22.66164159984885.\n",
      "Episode: 160\n",
      "The test reward for episode 160 is 169.0 with a standard deviation of 19.45507645834372.\n",
      "Episode: 170\n",
      "The test reward for episode 170 is 167.95 with a standard deviation of 20.2422207279735.\n",
      "Episode: 180\n",
      "The test reward for episode 180 is 191.15 with a standard deviation of 12.830724843125582.\n",
      "Episode: 190\n",
      "The test reward for episode 190 is 175.05 with a standard deviation of 20.998749962795404.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████████| 5/5 [01:56<00:00, 23.25s/it]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Avg. Return')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkEAAAG2CAYAAAB4e1KRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAABppElEQVR4nO3deXxTVfo/8M9N0iTdS0tXaMsmu+xQcUERFBBBhVFAGHHEdUAdUAf5jRvqDCgzMuMyOs5XxHFBZUZRmBFlX6RshYogVMCWrRu0dG+z3fP7I21s7EKXJPcm9/N+vSLNvTe5zzVN8vSc85wjCSEEiIiIiDRGp3QAREREREpgEkRERESaxCSIiIiINIlJEBEREWkSkyAiIiLSJCZBREREpElMgoiIiEiTDEoHoGayLCM3Nxfh4eGQJEnpcIiIiKgFhBAoLy9HUlISdLqm23uYBDUjNzcXycnJSodBREREbXDmzBl07ty5yf1MgpoRHh4OwPk/MSIiQuFoiIiIqCXKysqQnJzs+h5vCpOgZtR1gUVERDAJIiIi8jOXGsrCgdFERESkSUyCiIiISJOYBBEREZEmMQkiIiIiTWISRERERJrEJIiIiIg0iUkQERERaRKTICIiItIkJkFERESkSUyCiIiISJNUmQQtWbIEw4cPR3h4OOLi4nDrrbciKyvL7ZiamhrMnTsXMTExCAsLw9SpU1FQUOB2zOnTpzFx4kSEhIQgLi4OTzzxBOx2uy8vhYiIiFRKlUnQtm3bMHfuXOzevRsbNmyAzWbDjTfeiMrKStcx8+fPx9q1a7F69Wps27YNubm5mDJlimu/w+HAxIkTYbVasWvXLrz33ntYuXIlnnnmGSUuiYiIiFRGEkIIpYO4lPPnzyMuLg7btm3DqFGjUFpaitjYWHz00Uf41a9+BQA4duwY+vTpg/T0dFxxxRX46quvcPPNNyM3Nxfx8fEAgLfeegsLFy7E+fPnYTQaL3nesrIyREZGorS0lAuoEhER+YmWfn+rsiXol0pLSwEA0dHRAICMjAzYbDaMHTvWdUzv3r2RkpKC9PR0AEB6ejouv/xyVwIEAOPGjUNZWRmOHDnS6HksFgvKysrcbkRERGokhIAs++YmhPstUBiUDuBSZFnG7373O1x11VXo378/ACA/Px9GoxFRUVFux8bHxyM/P991TP0EqG5/3b7GLFmyBIsXL/bwFRARBT5ffDHWnULUO5+ot0/U3vtlKI3t+/lxPx8sSZLzX9f92n9rt9TdRxP767ZJbsc478lCuOJwxVDvft01iboY6+2T620Xzh1QUxoiNbNB+sVeSXI/rEOIETpdg2fwGdUnQXPnzsXhw4exc+dOr59r0aJFWLBgget+WVkZkpOTvX5eIiJ/YXfIsMsCdlnA4RCwyzIcslDVlzL5VoPXXtT/8ZcZ6SUe62OqToLmzZuHdevWYfv27ejcubNre0JCAqxWK0pKStxagwoKCpCQkOA6Zu/evW7PV1c9VnfML5lMJphMJg9fBRGR/2GyQ1qgyjFBQgjMmzcPn3/+OTZv3oyuXbu67R86dCiCgoKwadMm17asrCycPn0aI0eOBACMHDkS33//PQoLC13HbNiwAREREejbt69vLoSISOXsDhk1NgcqLHaUVtlQVGFBYVkNiiqtKK22odJiR43dATsTIApAqmwJmjt3Lj766CN88cUXCA8Pd43hiYyMRHBwMCIjIzFnzhwsWLAA0dHRiIiIwMMPP4yRI0fiiiuuAADceOON6Nu3L37961/j5ZdfRn5+Pp566inMnTuXrT1EpDls2SFqSJUl8tIvR5/Vevfdd3H33XcDcE6W+Nhjj2HVqlWwWCwYN24c/v73v7t1dZ06dQoPPfQQtm7ditDQUMyePRtLly6FwdCy3I8l8kT+o8pqh80uYNBLMOglBOl0ig64VJrNIcNql53/OuQGg4WJ1KBjmAl6L7xPW/r9rcokSC2YBBH5hxqbA6XVtgbbdZKEIL0Eg16HoABPjGwO2ZX4MOkhf6F0EqTK7jAiopayO2SUNZIAAc7SYotdwGKXXdvqJ0YGnYQgvc4rH8LexqSHqP2YBBGR35JlgZJqW6vGtTSWGEkSEKTTObvR9DpVJkb22m4tJj1EnsMkiIj8Vmm1DQ65/dmAEHAmGA4AcABomBgZdBIkSYJOanrcoifVJT02u4DF4WDSQ+QFTIKIyC+V1dhgdciXPrCNGkuM6pNq/yNBgiQ5u9kk1M0Y7Jw2uG72YJ0kuba7Zhl2e4wEIQSTHlKFwvIabP/xAlbvP4MzF6sxulcsXri1P3Q+SP59jUkQEfmdaqsD1daGiYkvidr/1C1j4GCxOfmxnAuV2PbjeWz78TyO5Lqvm7nxaCGKKg7gDxP7IDk6RKEIvYNJEBH5FatdRnlN4wOhiahlZCFwNK/MmfhknUdOUZVrnwTg8s6RiA41IuPURdRYHTh4pgQz/28PHrquO+4Ylqy6MXNtxSSIiPyGQxYoqbayzYWoDewOGQdOl2BrViG2H7+A8+UW1z6DTsLwLtG4tlcsRl3WETFhP08qnFtSjT/+9yj2n7qIv248jk1HC/HUxD7o0jFUicvwKM4T1AzOE0SkHkIIFFdaYffAQGgirai2OrD7pyJs/fE8vj1xAeU1dte+EKMeV3aPwbU9Y3Fl944IMzfdLiKEwJrMXLy66TiqrA4Y9TrcP6obZqQlw6Br+wpcSs8TxCSoGUyCiNSjpMrqVtZORI0rqbJix/EL2PbjeezNLnZ733QICcKonrG4tmcshnXpAJNB36rnzi+twZKvjmL3T8UAgL6JEXhqYh90jwtrU6xMglSMSRCROlRY7Ki02C99IJFG5ZVWY1uWc2Bz5pkS1G8wTYoy47qecbi2Vywu7xTZ7qRDCIH/fp+H5RuOo8Jih0EnYc7VXXHXyFQY9K1rFWISpGJMgoiU19SSGERaZXfIOFdSjZwLVfixoBw7TlxAVn652zGXxYXhul6xuLZXLHrEhnllbqvz5Ra8tP4Ydhy/AADoGR+Gp2/ui57x4S1+DiZBKsYkiEhZNoeMi5UcCE3aVG114FRxJXIuVCHnQiWyiyqRc6ESZy9WNxgbJwEYlByFa3s5u7qSooJ9EqMQAl8fKcBfNmShrNoOvU7C7JGpuOfqrghqQasQkyAVYxJEpBxZFiiqtELmRxQFuIuVVuQUVSKnyJns5BQ5E5/8spomH2MO0iE1JhSniipRY5MRF27C2oev9mHU7ooqLHj56yxszToPAOgeG4qnb+6LPonNf3cqnQSxRJ6IVEcI55pgTIAoUMhCIL+0xpXg5NS26uQUVTXb3RsVHIQuHUPRJSak9t9QdOkYgvgIM3SShM8OnMV7u05h9pWpPryahmLCTFg65XJsOlqIZV9n4eT5SsxZuR8zr0jBvdd0bfUAbF9hS1Az2BJEpIzSahtqbMrOCE3UUrIQKKmy4UKFBcWVVlyosOBChRVFFRZ8d6YUJ89XAECz0zskRpp/TnZiQl0/R4UYfXUZHnOx0oo/f5OFjUcLAQBdYkLw1M19cXmnyAbHKt0SxCSoGUyCiHyv0mJHBSvBSAUsdkdtUuNMaIoqnAlOUaX1558rrCiutMLRgq9Sg05CSnRIg5ad1JgQmIPU2VLSHluzCvHS+iwUV1qhk4DpI1LwwKhubtfKJEjFmAQR+ZbF7kBJFSvBPE0IAYcQkGVnq4VDdt7qfpYFfnG/9pgWPMYhBIQQMBn0CDXpEWo0IKT232CjXjWLbtocMipqnAl23ZQLdT+XVttQVGH9RZJjQVlNy5NxCUBUSBBiwkzoGGZETKgJMWFGnLtYjYzTF3H70M64+6ou7ZpY0B+VVtuwfMOP+OpwPgAgOToYT03si0HJUQCYBKkakyAi37E7ZBRXWbl6uoddqLDg3vf2I6+06UG23iIBroQo1GRAiFGPUJMBoXX/1v4c0si2UJMBYbWPMRn0qLLaUWlxuBIXt2SmNrmptDp//uVxlRZ7myfaDNJL6BhmQnSoER1rE5yff3YmOjFhRkSHGFs9R46W7DxxAUv/dwznKyyQANw+rDN+e10PJEeHMAlSKyZBRL4hywLFVVY4uCSGR5VUWfHgBweQfaGy2eP0kgSdDtBJEvQ6500nSdBJcLtf/9/6jzlZWAGbLKCTgDCTAZUWR4u6h5QQUi/BCjMZkFVQDqtdRohRj3uu7oqOYUZ0DK1LbkyIMBu8MseOFpXX2PDqphP48rtcAM5JHF+eOhBXX9bR4+diEuQBTIKIfONipRVWB5fE8KTyGhvmfngQWQXlCDMZYA7SYVZaKm4ZnOSW0OgktPtLvn6F0pQhnSGEgMUuo9LibJWptDpbYyqtjtpt7j9XWeuOqd1W+3OV1Y4a28+/FyFGvSt5CTMb3JKZuluoSY8wc/377j//stXhl7GT9+3+qQhL/nfMNQXAzLQU/PG2yz16DiZBHsAkiMj7ympsqLayEsyTqqx2PLIqE9+fK0WHkCC8NWuo3674vTrjDN5PP4W7RnbBr4YySQkUFRY7bn51J6ptDiRFmbHryTEefX7OE0REqldtdTAB8rAamwOPrz6E78+VIsJswKszBvttAgQAtw9Nxu1Dk5UOQ7OC9DrYvNBKG2Yy4JExPfCv9FP47XU9PP78LcUkiIgUYbXLKK9hJZgnWe0ynvzse2ScuogQox5/nT6oVes4EdUXZjIgOEiPCxUWryxdM2VIZ9w/qrtXBka3FIeyE5HPOWSBkmquCeZJdlnG018cRvrJIpgMOrxyx0D0S2o4OR1RSwTXDiDX6SSEmAK3vYRJEBH5lBACJSyF9yiHLPDC2qPYmnUeQXoJy24fgMEpHZQOi/yUUa9DhDnIdT/UqEegFsgxCSIinyqttjW7fAC1jhACL60/hvVH8qHXSfjTbZcjrWuM0mGRn9LrJEQGB7ltkyQJYQHaGsQkiIh8pqIdk9ZRQ0IILN94HF9k5kInAYsn98OonrFKh0V+SpKADiFG6BoZoxMcpJ7Zvz2JSRAR+YQQApVcE8yj3tr2Ez7ZdwYA8IeJfXBD33iFIyJ/JQGICjY2OUhZkiSEmwOvNYhJEBH5BCdD9KyV3+Zg5a4cAMAT43rh5gFJygZEfi0iOAhGQ/MpgTlID4OClVzewCSIiHzC7uA4IE9Ztfc03tx2EgAw7/oenESQ2iXUZGjxKvahATY2iEkQEfmENyZc06I1B8/hrxuPAwDuvborfn1FqsIRkT8zG/StGvRsDtIjKIAWig2cKyEiVWN3WPt9dTgPS786BsC53tK913RVOCLyZ0F6HSKCW9+yE0iVYkyCiMjrHLLgvEDttOVYIV5YexQCwNQhnfDw9T24ujm1mV4nISo4qE2/Q0aDDqZLjB/yF6q8iu3bt2PSpElISkqCJElYs2aN235Jkhq9LVu2zHVMly5dGuxfunSpj6+EiAB2hbXXtycu4Kk1h+EQAhMHJOLxcb2YAFGbSRIQFRzUaCl8SwVKa5Aqk6DKykoMHDgQb7zxRqP78/Ly3G4rVqyAJEmYOnWq23HPP/+823EPP/ywL8Inol9gEtR2+3OKseiz72GXBcb2icMfbuoTkPO1kG9IACKDg2Bo57geg17X4sHUaqbKVG7ChAmYMGFCk/sTEhLc7n/xxRcYPXo0unXr5rY9PDy8wbFE5Hs2Voa1yaGzJXh89SFY7DKuuawjFk/up+hik+T/ws1BMBk8k7yEmQyw2Bx+vQagKluCWqOgoAD//e9/MWfOnAb7li5dipiYGAwePBjLli2D3d78RG0WiwVlZWVuNyJqPztbglrtWH4ZfvdJJqptDozoGo0/3ta/3X+9k7aFGPUINnqu9Uavkzz6fEpQZUtQa7z33nsIDw/HlClT3LY/8sgjGDJkCKKjo7Fr1y4sWrQIeXl5eOWVV5p8riVLlmDx4sXeDplIU2wO2a//UlTCycIKPLIqE5UWBwZ2jsTLUwd47K930iaTQYdwc9ClD2ylUKMB1TaH3xY+SEKoO3RJkvD555/j1ltvbXR/7969ccMNN+C1115r9nlWrFiBBx54ABUVFTCZTI0eY7FYYLFYXPfLysqQnJyM0tJSREREtPkaiLSsympHeQ2Xy2ip08VVeOD9DBRXWtE3MQKv3Tk4YAahkjIMOgnRoUavDaavtNhR0cYlcTqGmbzSxVtWVobIyMhLfn/79Ttrx44dyMrKwieffHLJY9PS0mC325GTk4NevXo1eozJZGoyQSKituF4oJbLK63GvI8OoLjSih5xYfjr9EFMgKhddJKEDiHeS4AAZzdbldUBWd1tKo3y6w7md955B0OHDsXAgQMveWxmZiZ0Oh3i4uJ8EBkR1WFlWMucL7dg7ocHUVBmQWp0CF6bMRiRwZ7vviDtkABEhbSvFL5F55Ekv03WVRl1RUUFTpw44bqfnZ2NzMxMREdHIyUlBYCzqWv16tX4y1/+0uDx6enp2LNnD0aPHo3w8HCkp6dj/vz5mDVrFjp06OCz6yDSOlkWcMj+99ehr12stGLeRwdwrqQaSVFmvD5zMKJDjUqHRX4uMiTIZ0tcBBv1qLTa/e79rsokaP/+/Rg9erTr/oIFCwAAs2fPxsqVKwEAH3/8MYQQmDFjRoPHm0wmfPzxx3juuedgsVjQtWtXzJ8/3/U8ROQbNpmtQJcihMD972fgdHEVwk0GvHHnEMSFm5UOi/xcuNng88H0YSYDSqttPj1ne6l+YLSSWjqwiogaV2Gxo7KNAya1Yk92ER5ZlQkA6BhmxH8fuUbZgMjvBRv1iPBCJVhLFFdaW9UFrvTAaL8eE0RE6sb5gZonhMA/tv0EwDm4dM7VXBCV2sdk0CmWAAH+t5yGf0VLRH6FK8c3b+eJCziSWwZzkA7/fnAkYsJYnUptZ9BJig+mr1tc1WL3j/c+W4KIyCu4cnzz5HqtQHcMS2YCRO2ikyREebkUvqVC/ag1yH8iJSK/wtL45m05VojjhRUIMeoxKy1V6XDIC3SSBFOQDjpJgiwEhOxMfp03Z3eoJ/5OqCuFV8u6ckG1i6vW2BxKh3JJTIKIyCuYBDXNIQu8vd3ZCnTniBREhnA+oEBRl/iYDXoYDZfubBG1CVFdciRcP6PFiVNEsO9K4VvKXxZXZRJERF7BmaKb9vWRfOQUVSHCbMCMESlKh0Pt1NrEpz5JkqCXAD1a3opTP3ECoLoECHAurmo26lFtVXdrEJMgIvIKVoY1zu6Q8X87sgEAs65IRZiZH8P+qD2JT3u1JXFSQpjRgBqruluD+O4jIo/jyvFNW3coD+dKqtEhJAh3DEtWOhxqBUkCzEF6RRIff6TTSQgxGVQ9VxiTICLyOI4HapzF7sA7O52tQHdf2QXBRt/O6Eutx8SnfUKNelRZ7aqtFGUSREQex/FAjVtzMBeF5RbEhptw25BOSodDTWDi4zl1i6uW16izNYhJEBF5HFuCGqq2OrByVw4A4J6ruvh8XSdqHhMf7wkxGlBldahycVUmQUTkUVw5vnH/zjiL4korkqLMmDQwSelwqJYE52rrTEq9S62LqzLdJSKP4srxDVVY7PjX7hwAwL1Xd1NlSbNWRQQzAfIFc5AeBpVM5lgf34lE5FEcD9TQx3tPo6zajtToEIzrH690OFTLbNDDHMQEyFfUOB0EkyAi8ijOD+SutNqGj/aeBgDcN6obDDp+7KqBTpIQEay+L+VAZjLoYVRZK6i6oiEiv8eV4919uOcUKi0O9IgLw5g+cUqHQ7Uig4NUsdio1qitNYhJEBF5DFeOd1dcacUn+84AAB4Y1Q06fumqQqjJwAowhQTpnbNsqwV/C4jIY1ga7+5f6TmoscnomxiBay7rqHQ4BOeXcJhJXa0RWhNqUs+CH0yCiMhj2BX2s8LyGvwn4xwA4IFru7HrRQUkydkNRsoy6HUwq2S2dCZBROQxdlaGuaz8NgdWh4xByVFI6xqtdDgEIMIcBL0Ky7S1KMxoUEVrEJMgIvIIIQQrw2rlllRjTWYuAOBBtgKpgjmI5fBqUre4qtKUj4CIAoJdFlw5vtb/7cyGQxYY0TUag1M6KB2O5ul1EiJUVpVEzsVVlcbfCiLyCA6KdjpVVImvvs8D4GwFIuVFmFkOr0ZqeE3YHUZEHmGzsx0IAP65IxuyAK65rCP6JUUqHY7msRyemsPfDCLyCK4ZBhwvLMeGHwoAAPePYiuQ0lgOT5fCJIiI2o0rxzv9c3s2AGBsnzj0jA9XOBptYzk8tQSTICJqN7YCAUfzyrDtx/PQScC917AVSGksh6eWYBJERO3GleOBf2z7CQAwvn8CunYMVTgabWM5PLUUO0uJqN20Pj9Q5pkSpP9UBL1OwpyruyodTgNmgx5Ggw5WuwyLwxHQ67uxHJ5ag78pRNRuWl4uQwiBf2w7CQCYNCARnTuEKByRuxCjHuFm59iYYKMeQBCsdhlWhwyrXQ6oqQ0kcHV4ah0mQUTULlpfOX5/zkUcOF2CIL2Ee1TWChRuNiDE2PBj3mjQOcvGTc5B7VaHDIvdmRTJfvxihpoMCNJzlAe1HJMgImqXQGpJaC0hBN6sbQW6bXAnxEeYFY7ISQIQERzUonExOp0Es+7nMTR2h7OVyGJzthL5S0pk1OsQynJ4aiX+xhBRu2i5K+zbE0U4klsGc5AOd1/ZRelwADhLw6OCjW2eINCg18Gg1yHE6Ezy6rrNLHZZtdMgSJIz6SNqLVW2G27fvh2TJk1CUlISJEnCmjVr3PbffffdkCTJ7TZ+/Hi3Y4qLizFz5kxEREQgKioKc+bMQUVFhQ+vgkgbtLpyvCwE/rHd2Qp0x7BkxISZFI4I0EkSokPangD9kiRJMBmcY4o6hpnQMcyECHMQzAY91DTshuXw1FaqTIIqKysxcOBAvPHGG00eM378eOTl5bluq1atcts/c+ZMHDlyBBs2bMC6deuwfft23H///d4OnUhTtLxy/JZjhfixoAIhRj1mpaUqHQ4MOgkxoUYYvDgmRq+TEGzUIzIkCHHhZnQIMSo+Dofl8NQequwOmzBhAiZMmNDsMSaTCQkJCY3uO3r0KNavX499+/Zh2LBhAIDXXnsNN910E/785z8jKSnJ4zETaZFWV453yAJvb3fOC3TniBREhijbFWMy6BSpiqo/wNrukFFlc6DG5rsSfJbDU3upsiWoJbZu3Yq4uDj06tULDz30EIqKilz70tPTERUV5UqAAGDs2LHQ6XTYs2dPk89psVhQVlbmdiOipml1UPQ3P+Qjp6gKEWYDZoxIUTQWc5AeUSFGxcvCDXodIsxBiK3tMvN26xDL4ckT/DIJGj9+PP71r39h06ZNeOmll7Bt2zZMmDABDocDAJCfn4+4uDi3xxgMBkRHRyM/P7/J512yZAkiIyNdt+TkZK9eB5G/0+LK8XaHjP/b4VwjbNYVqQhTsCUi1GRQ3fpYkuTsMosONSI61AhzkB7eSFOU7oajwOCX7YjTp093/Xz55ZdjwIAB6N69O7Zu3YoxY8a0+XkXLVqEBQsWuO6XlZUxESJqhhbXDPvv93k4e7EaHUKCcMcwZT4fWlMCr6QgvQ6RwToIswHVNgeqrQ7YPVBhxnJ48pSASKO7deuGjh074sSJEwCAhIQEFBYWuh1jt9tRXFzc5DgiwDnOKCIiwu1GRI3T4srxVruMd3Y6W4FmX9mldgZm35IARIaoPwGqT5IkhBgNiAkzoUOI0Vld1ubnYjk8eU5AJEFnz55FUVEREhMTAQAjR45ESUkJMjIyXMds3rwZsiwjLS1NqTCJAooWW4G+yDyHgjILYsNNmDKkk8/Pr5MkdAg1wmTwnwTol4wGHSJDnCX3YSZDq0vbWQ5PnqTK9sSKigpXqw4AZGdnIzMzE9HR0YiOjsbixYsxdepUJCQk4OTJk/j973+PHj16YNy4cQCAPn36YPz48bjvvvvw1ltvwWazYd68eZg+fTorw4g8RGsrx9fYHHj32xwAwD1XdfF5IqLXSegQYgyYBECnkxBqMiDUZIDF7uwqs9ibT6yDjSyHJ89SZUvQ/v37MXjwYAwePBgAsGDBAgwePBjPPPMM9Ho9Dh06hMmTJ6Nnz56YM2cOhg4dih07dsBk+nmysg8//BC9e/fGmDFjcNNNN+Hqq6/G22+/rdQlEQUc2yW+sALNvzPOoqjSiqQoMyYN9O0fU0a9DtEBlAD9ksngrHDrGGZCqMkAXSMVX3qdhHCOAyIPk4Tw49XyvKysrAyRkZEoLS3l+CCiXygsr9HMwqkVFjum/H0XSqtteObmvpg4INFn5zYb9IgINmiuFLymdiC11SFDAtAh1MhqMGqxln5/M60molazO2TNJEAAsObgOZRW26DXSaiy2X123hCjc8kKLaqbCdohC9hlmQkQeQV/q4io1TxR5uxPdhy/AMA5U/QH6ad9cs5ws0GzCVB9ep3k1wPBSd3YEkREraallePLa2z4/mwpACA2zITZV3p3nTB/mQOIKBAwCSKiVtPSyvH7ci7CIQRSo0Pw6YMjvXouSQKigj23CjwRNY9JEBG1itZWjk8/6VyXcGT3GK+eR6+TEBUc5NVV4InIHd9tRNQqNod2Vo4XQiD9J98kQRFmJkBEvsZ3HBG1il1DM0WfPF+J8+UWmAw6DE6J8uq5DAE6BxCRmjEJIqJW0dLK8XWtQENSO3i1QkmSnDMoE5FvMQkiolbR0pphdeOBruzm5fFAGpsIkUgtmAQRUYtpaeX4Sosd350pAQBc4eXxQAYdP4qJlMB3HhG1mJZagfafugi7LNC5QzBSokO8ei7mQETK4FuPiFpMSyvH767tCrvCy11hAFuCiJTCdx4RtZhWVo73ZWk8wJYgIqXwrUdELaaV7rBTRVXIK61BkF7C0JQOXj8fW4KIlMF3HhG1iJZWjq9rBRqc0gHBRu+u4SXBOVs0Efleu5fNsNvtKCoqgsViafKYlJSU9p6GiBSmpZXjXUtl+GA8EOcHIlJOm5OgjRs34sUXX8Tu3bths9maPE6SJNjt9raehohUQisrx9fYHDh4ugQAcEW3aK+fj3MEESmnTUnQunXrcNttt8HhcKBDhw7o2rUrwsPDPR0bEamIVgZFZ5y6CKtDRkKEGV07hnr9fHo9kyAipbQpCVq8eDFkWcby5csxb9486PXe7TMnImUJoZ1JEnf/VFcaHw3JB600bAkiUk6bkqAjR45g5MiRePTRRz0dDxGpkJZWjt9Vt1RG944+OR8HRRMpp03VYWFhYRzsTKQhWlk5/kxxFc5erIZeJ2FoF++XxgNMgoiU1KYkaOzYsdi/f7+nYyEildLKyvF1XWEDO0cizNTu4tkWMTAJIlJMm5Kgl156CWVlZVi4cCErv4g0QCuVYb6cJRoAJAk+GXdERI1r05867777LiZMmIA///nP+M9//oPrrrsOnTt3hq6RWU8lScLTTz/d7kCJSBmyLCBrYJZEi92B/TkXAfguCeJM0UTKkoRo/aebTqeDJEloyUMlSYLD4WhTcEorKytDZGQkSktLERERoXQ4RIqosTlQWt30XGCBYk92ER5ZlYnYMBPWPnyVT1pozAY9IkOCvH4eIq1p6fd3m1qCVqxYwSZcIo3QykzRdbNEX9HdN6XxAOcIIlJam5KgKVOmQJIkTpBIpAFamSTRl0tl1OEcQUTKalOHdIcOHXDjjTd6OhYiUiEtrByfV1qNnKIq6CUJI7p6f6mMOiyPJ1JWm5KgiIgIdOvWzdOxEJHKaGXl+LpWoH6dIhBu9t0YHSZBRMpqUxI0ePBgnDx50tOxEJHK2BwayIAA7P6pGIBvu8IkMAkiUlqbkqCFCxdi3759+Pe//+3peIhIRbTQFWZzyNiXU5sE+ag0HgB0TICIFNemgdHBwcG49957MW3aNNx8882YNGkSUlJSYDabGz1+1KhR7QqSiJShhUHRh86WosrqQIeQIPRK8F2xB2eKJlJem5Kg6667zjVP0Nq1a7Fu3bpmj/fXeYKItEwrK8e7SuO7xUDnw2ottgQRKa9NSdBdd93l1Xk0tm/fjmXLliEjIwN5eXn4/PPPceuttwIAbDYbnnrqKfzvf//DTz/9hMjISIwdOxZLly5FUlKS6zm6dOmCU6dOuT3vkiVL8OSTT3otbqJAopWV4329VEYdtgQRKa9NSdDKlSs9HIa7yspKDBw4EPfccw+mTJnitq+qqgoHDhzA008/jYEDB+LixYt49NFHMXny5AaLuj7//PO47777XPc5rxFRy9k0sF5YYXkNThRWQAKQ5sPSeAA+bXUiosb5ZpnkVpowYQImTJjQ6L7IyEhs2LDBbdvrr7+OESNG4PTp00hJSXFtDw8PR0JCgldjJQpUdg1UhtVVhfVNikBUiNGn52ZLEJHyAmL1vtLSUkiShKioKLftS5cuRUxMDAYPHoxly5ZdcsV7i8WCsrIytxuRVmlh5XglZomuw/J4IuW1qSXonnvuafGxkiThnXfeactpWqSmpgYLFy7EjBkz3BZJe+SRRzBkyBBER0dj165dWLRoEfLy8vDKK680+VxLlizB4sWLvRYrkb/QwsrxdlnG3mxnS9AVPh4PpJMkrr9IpAJtXkX+kk9cWz3W3lXkJUlyGxhdn81mw9SpU3H27Fls3bq12ZViV6xYgQceeAAVFRUwmUyNHmOxWGCxWFz3y8rKkJyczFXkSXO0sHJ85pkSPPB+BiKCDVj/6CiftswE6XWIDvVt9xuRlnh1FfktW7Y0ul2WZZw5cwbffPMNPv74Y8yfPx+TJk1qyykuyWaz4Y477sCpU6ewefPmSyYpaWlpsNvtyMnJQa9evRo9xmQyNZkgEWmJFlaO313bFZbWNcbnXVPsCiNShzYlQddee22z+++66y5MnDgRs2fPxuTJk9sUWHPqEqDjx49jy5YtiIm5dFN2ZmYmdDod4uLiPB4PUaDRwiSJdaXxV/q4KwxgEkSkFl6rDpsxYwZefvllPPfcc9i8eXOrHltRUYETJ0647mdnZyMzMxPR0dFITEzEr371Kxw4cADr1q2Dw+FAfn4+ACA6OhpGoxHp6enYs2cPRo8ejfDwcKSnp2P+/PmYNWsWOnTo4NHrJApEgb5cRlGFBcfyywH4vjQeYGUYkVp4tUT+sssuw/r161v9uP3792P06NGu+wsWLAAAzJ49G8899xy+/PJLAMCgQYPcHrdlyxZcd911MJlM+Pjjj/Hcc8/BYrGga9eumD9/vut5iKhpWlg5fk/tgOheCeGICfN9FzjnCCJSB68lQbIs49ChQy0aRP1L1113HZobr32psdxDhgzB7t27W31eItLGyvFKlsYDbAkiUguPzxNUVVWFzMxMzJgxA8ePH7/k+CEiUpdA7wpzyMLVEnRFN993hUngumFEatGmliC9Xn/JY4QQiI2NxbJly9pyCiJSSKAPij6aV4bSahvCTAZc3jnS5+fnoGgi9WhTEpScnNzkRF9GoxGJiYm49tprMXfuXFZjEfkRIUTAl8fvrq0KG96lAwxt6K5vLyZBROrRpiQoJyfHw2EQkRpoYTzQrpN1pfEdFTk/kyAi9QiItcOIyDMCfeX40iobfsh1rgmYpsB4IIBJEJGatCkJ6tatGxYuXHjJ4xYtWoTu3bu35RREpIBAT4J2ZxdBAOgRG4b4CLMiMTAJIlKPNiVBOTk5OH/+/CWPu3DhArvOiPyEEALWAB8UXTce6IruyrQCAYCecwQRqYZXu8MqKysRFBTkzVMQkYfU2GQE8oggWQjF5wcC2BJEpCZemSxRlmVkZWVhy5YtSElJ8cYpiMjDqm0OpUPwqh8LynGxyoYQox4Dk6MUiUEnSU1W1hKR77W4JUiv17tuAPDee++5bat/CwoKQv/+/VFQUIAZM2Z4LXgi8gyHLAJ/PNBJ5wSJw7p0QJBemZoQzhRNpC4tbgmqPzfQ6dOnERISgo4dGy8xNRqNSEpKwuTJk/HII494JlIi8ppAbwUCgF0nLwBQtiuMM0UTqUuLk6D6A5x1Oh1uv/12rFixwhsxEZGP1QR4ElReY8Phc87S+CsUTILYEkSkLm0aE7RlyxYkJCR4OhYiUoDVLsMR4LNE780uhkMIdIkJQVJUsGJxcFA0kbq0KQn65aKoFosFxcXFMJlMiI5WrvSUiFqvxh7YrUAAsPunugVTlWsFApgEEalNu0YHvv322xg8eDBCQ0PRuXNnPP744659n332GaZMmYITJ060O0gi8g4hRMB3hQkhkF47P9DI7gonQawMI1KVNiVBDocDt912Gx566CEcPXoUffr0gRDuzekDBw7EmjVr8Mknn3gkUCLyPItdhgjsnjCcPF+J8+UWmAw6DE6JUiwOSeLAaCK1aVMS9Prrr+OLL77AhAkTcOrUKXz//fcNjunevTt69OiBr776qt1BEpF3BHorEADXBIlDUzvAZNArFgdbgYjUp01J0MqVKxEfH49PPvkE8fHxTR7Xt29fnDp1qs3BEZH3yLKAJcCXyQDwc1eYwuOBDDquV02kNm16V2ZlZSEtLQ2hoaHNHhcaGtqiNcaIyPe0MCC60mLHd2dKACg/Hog5EJH6tOltGRQUhJqamksed/r0aYSHh7flFETkZdXWwE+C9p+6CLss0LlDMJKjQxSNhS1BROrTpndlv379kJGRgfLy8iaPKSwsRGZmJgYNGtTW2IjIS+wOGfYAnxsIgCoWTK3DHIhIfdr0tvz1r3+NoqIiPPjgg7BarQ32OxwOzJ07F1VVVZg9e3a7gyQiz9LCMhlCCOyuHQ90hcJdYQBbgojUqE2TJd5///1YvXo1Vq1ahV27dmHcuHEAgO+++w6PPvoo1q1bh+zsbNx4442YOXOmRwMmovarsQX+gOicoirkldbAqNdhaEoHRWORwIkSidSoTX+a6PV6/O9//8NDDz2E3NxcvP322wCAgwcP4rXXXsPp06dx3333Yc2aNa5FV4lIHSx2B+RAnxwIcLUCDUqJQrBRudJ4gPMDEalVm1qCAMBsNuONN97Ac889h61btyInJweyLKNz584YPXo0kpKSPBknEXlIjTXwW4GAn8cDXamCrjDOEUSkTm1OgurExsbi9ttvb3Sf1WrFihUr8OCDD7b3NETkAUIIWDRQGl9jc+Dg6RIAyq8XBgB6PZMgIjXyyki9qqoq/OUvf0HXrl0xd+5cb5yCiNqgxiYj8DvCgIxTF2F1yEiMNKNLjLKl8QBbgojUqlUtQbt27cLXX3+NwsJCxMXFYfz48Rg5cqRrf2VlJZYvX46//e1vKC4uhhACQ4cO9XjQRNQ2WqgKA37uCruiW4wqxiVyUDSROrU4Cbrnnnvw3nvvAXA2qUuShBdffBHz5s3D3/72N2zcuBGzZ89Gfn4+hBAYPHgwnnvuOUyaNMlrwRNRyzlkAZtDI+OBVLJURh0mQUTq1KIk6L333sPKlSsBAOPHj0e/fv1QXl6OjRs34vXXX0dCQgIWL14Mq9WKfv364cUXX8Qtt9zizbiJqJW00gp0prgKZy9WQ6+TMKyLsqXxdQxMgohUqUVJ0LvvvgtJkvD5559j8uTJru12ux3Tpk3DU089BQB45JFH8Je//AV6vbLlqETUkBZWjAd+Lo0f2DkSoaZ21360myRBFV1yRNRQiwZGf//99xg+fLhbAgQABoMBL774IoQQSE1NxfLly5kAEamQ1S7DoYFlMgBgl6s0vqPCkThxpmgi9WrRu7O0tBSXXXZZo/vqtg8fPpx/7RCplBZWjAecE0FmnLoIALiie7TC0TixMoxIvVqUBMmyjKCgoEb3GQzO5ubQ0FCPBbV9+3ZMmjQJSUlJkCQJa9ascdsvhMAzzzyDxMREBAcHY+zYsTh+/LjbMcXFxZg5cyYiIiIQFRWFOXPmoKKiwmMxEvkLIYRmusIOni6BxS4jNsyEHrFhSocDgHMEEamZKttpKysrMXDgQLzxxhuN7n/55Zfx6quv4q233sKePXsQGhqKcePGoaamxnXMzJkzceTIEWzYsAHr1q3D9u3bcf/99/vqEohUw2KXoYFVMgCg3oKp0appmWZLEJF6SUJc+uNRp9MhLCwMHTs23sd+6tQphIaGNrpfkiScPHmy7QHWDsi+9dZbATj/qk1KSsJjjz2Gxx9/HICzuy4+Ph4rV67E9OnTcfToUfTt2xf79u3DsGHDAADr16/HTTfdhLNnz7Z4SY+ysjJERkaitLQUERERbb4GIiWVVFlhsWujNH7aP9KRU1SFP93WH2P6xCsdDgCgQ4gRRoMq/94kClgt/f5ucelERUVFs91JTe339F9j2dnZyM/Px9ixY13bIiMjkZaWhvT0dEyfPh3p6emIiopyJUAAMHbsWOh0OuzZswe33XZbo89tsVhgsVhc98vKyjwaO5GvybLQTAKUW1KNnKIq6CUJI7qqYzwQwDmCiNSsRUlQdna2t+Nosfz8fABAfLz7X3nx8fGuffn5+YiLi3PbbzAYEB0d7TqmMUuWLMHixYs9HDGRcrQyIBr4uSusf6cIhJsbH8PoaxKYBBGpWYuSoNTUVG/HoQqLFi3CggULXPfLysqQnJysYERE7VNt1U4S5JolWgWrxtfRMQEiUjW/66hOSEgAABQUFLhtLygocO1LSEhAYWGh23673Y7i4mLXMY0xmUyIiIhwuxH5K7tDhl0jcwPZHDL25zhL49WUBHGmaCJ187skqGvXrkhISMCmTZtc28rKyrBnzx7XYq4jR45ESUkJMjIyXMds3rwZsiwjLS3N5zETKUEry2QAwJ7sYlRZHZAk4PA59YzlY0sQkbopP6d8IyoqKnDixAnX/ezsbGRmZiI6OhopKSn43e9+hxdffBGXXXYZunbtiqeffhpJSUmuCrI+ffpg/PjxuO+++/DWW2/BZrNh3rx5mD59eosrw4j8XY1NGwOiAeCTvWcAAEIA76efwq+GdlY4Iie2BBGpmyqToP3792P06NGu+3XjdGbPno2VK1fi97//PSorK3H//fejpKQEV199NdavXw+z2ex6zIcffoh58+ZhzJgx0Ol0mDp1Kl599VWfXwuREix2B2SNTA50orACe3OKIQHoGGbC7CvVM4ZRxzmCiFStRfMEaRXnCSJ/VVpl00xl2AvrfsC6Q3kY0zsOf5pyudLhuIkJNcKg97tRB0R+r6Xf33x3EgUYIQQsGkmAiios+PqIc9qLGSNSFI6mIZbHE6kbkyCiAFNjk6GV5t3/HDgHm0Ogf6cIXN45Uulw3OgkSTVLdxBR45gEEQUYrSyWWmNz4D8ZZwEAd7IViIjagEkQUQBxyAJWhzaqwr4+ko+SahsSI824tles0uE0wCSISP28lgT17dsXer0eBoMqC9CIApJW5gYSQuCjPacBAHcMS4ZBp76/55gEEamf1zIUWZYhhACLz4h8RytdYbt/KkZOURVCjHpMHqjOub84RxCR+nktCdq8eTNsNpu3np6IfsFql+HQyDIZq/Y6W4EmD0xCmFmdrc2cI4hI/bz26cGZmYl8SyvzAp0srMCe7GLoJGDacPUucMyWICL1U19HOhG1mhBCM11hH+9zLpFxbc9YJEUFKxxN4yRw3TAif8AkiCgAWOwytDD8rqjCgvWHnZMj3pmmvrL4OhwUTeQf2tQddv3117foOKPRiJiYGAwaNAjTp09HcrJ6m66J/JlWWoE+O3AOVoeMfkkRuLyTuiZHrI9JEJF/aNPaYbraclRJkpqs/vrlvqCgILz00kv43e9+17ZIFcC1w8gfyLLAhQpLwM8SbbE7cMvr3+JilQ0v3tofN/SNVzqkJoUY9Qg3BykdBpFmeXXtsOzsbDz66KMwGAyYOXMmvvzyS2RmZiIzMxNr167FrFmzYDAY8PDDD2Pnzp3405/+BLPZjMceewzffPNNmy+KiBqqsTsCPgECgK8PF+BilQ0JEWaM7q2+yRHrY0sQkX9oU3fY7t278dprr+Grr77CDTfc4LZvwIABmDhxIn7961/jpptuwhVXXIEnn3wSaWlpGDNmDF577TXceOONHgmeiIBqa+B3hQkhXGXxdwzvrMrJEetjEkTkH9rUHTZ8+HCEhYVhy5YtzR43evRolJeXY//+/QCAwYMHIzc3FwUFBW2L1sfYHUZqZ3fIKKq0Kh2G1+3+qQiPfpyJEKMea+ddrdq5gerEhBph0Ks7USMKZF7tDjt69GiL5gFKSkrCsWPHXPcvu+wylJSUtOWURNQIrSyTUdcKNEnFkyPWx5YgIv/QpiQoJCQE+/fvb3ZJDCEE9u/fj5CQENe2mpoatqgQeVCNLfAXS/3pfAV2/1Q7OeIw9VeY6iQJEmeLJvILbUqCxo4dixMnTuDhhx9GVVVVg/3V1dV49NFHceLECbfxP8ePH2eZPJGHWOwOyBqYHKhucsRRPWPRqYM6J0esjzNFE/mPNrUrL1myBBs3bsSbb76JVatWYfz48a7k5syZM/j6669x8eJFxMbG4o9//CMAZxdaVlYWnnjiCc9FT6RhNdbAbwUqrrTiq+9rJ0ccod7JEevjTNFE/qNNSVBqairS09PxwAMPYPPmzVi1alWDY8aMGYM333wTqampAIBu3bohLy8PkZHqneCMyF8IIWDRwFphnx0465occUBn//jsYEsQkf9o8wjD7t27Y+PGjTh58iS+/fZb5OXlAQASExNx5ZVXokePHm7Hm0wmxMerd3IzIn9SY5MDfm4gi92Bf2ecBQBMH57sN+NsOCiayH+0u8yie/fu6N69uydiIaIW0sIyGV8fcU6OGB9hwvW945QOp8WYBBH5jzYNjH788cfx3XffeToWImoBhyxgdQT2eCAhBD6uLYu/fViyX825o/eTFisiamMS9Morr2DIkCHo378/lixZglOnTnk6LiJqghbmBtqbU4yT5ysRHKTHrYMuPSeZWkgSB0YT+ZM2JUF/+9vfMHz4cPzwww/4wx/+gG7dumHUqFH4xz/+gYsXL3o6RiKqRwtdYav2OsviJw1M9KuFSNkKRORf2pQEPfzww9i9ezdOnDiB5557Dj169MDOnTvx29/+FomJibj11luxevVqWCwWT8dLpGlWuwyHHNhDon86X4H0k0WQAEwb7l/ziql9TTMicteud2y3bt3wzDPPICsrC/v27cMjjzyC6OhofPnll5g+fTri4+Nxzz33eCpWIs3TQll83eSI1/aMRecOIZc4Wl2YAxH5F4+9ZYcOHYrly5fj7Nmz+OabbzBt2jSUlZXhvffe89QpiDTPYg/sAdEXK61Yf9g5OeKMNP+YHLE+tgQR+RePv2O3b9+OTz/9FF9//bWnn5pI0xyyCPiusM8OnoPFLqNPYjgG+snkiPUxByLyLx5ZjjkzMxMffvghPv74Y+Tm5kIIgfDwcNx1112YOXOmJ05BpHnWAG8Fstpl1+SIM0ak+M3kiPWxJYjIv7Q5CcrOzsZHH32Ejz76CMeOHYMQAkFBQZg4cSJmzpyJW265BWaz2ZOxEmlaoCdB3/yQj+JKK+LCTRjjR5Mj1pHAiRKJ/E2bkqCRI0di7969ELUrWF955ZWYOXMmpk2bhujoaI8GSEROFkfgDooWQmDVHueA6Dv8bHLEOpwfiMj/tCkJ2rNnD3r37o2ZM2di5syZ6NKlS5PHyrIMHZuIidrFapchAng40P6cizhxvgLBQXrc4keTI9bHOYKI/E+bspOMjAzXRIlNJUAHDx7EggUL0Llz5/bE16QuXbpAkqQGt7lz5wIArrvuugb7HnzwQa/EQuRtgb5Mxke1S2RMGpiIiGD/mRyxPr2eSRCRv2lTS9DgwYMb3X7mzBl8+OGH+OCDD3D06FEIIbw2uHHfvn1w1OseOHz4MG644Qbcfvvtrm333Xcfnn/+edf9kBD/mnOEqI4lgGeJzr5QiV1+OjlifWwJIvI/7a4OKy8vx+rVq/HBBx9g+/btEEJACIFOnTph2rRpmDFjhifibCA2Ntbt/tKlS9G9e3dce+21rm0hISFISEjwyvmJfEWWBewBXBr/Se3kiKP8cHLE+jgomsj/tCkJcjgcWL9+Pd5//32sXbsWNTU1rkHSkiRh69atuOaaa3xW4mq1WvHBBx9gwYIFbuesa5VKSEjApEmT8PTTTzfbGmSxWNyW+igrK/Nq3EQtEchdYSVVVvzv+zwAwIwR/tsKBDAJIvJHrUqC9u3bh/fffx+ffPIJLly44CqLnzx5MmbNmoWXX34Z+/fvx6hRo7wVb6PWrFmDkpIS3H333a5td955J1JTU5GUlIRDhw5h4cKFyMrKwmeffdbk8yxZsgSLFy/2QcRELWexBW4S9NkB5+SIvRPCMSg5Sulw2sXAJIjI70hCXLrm5MUXX8SHH36IH3/80a0sftasWbjjjjtcZfHXXHMNdu3a5TZWxxfGjRsHo9GItWvXNnnM5s2bMWbMGJw4cQLdu3dv9JjGWoKSk5NRWlqKiIgIj8dN1BKF5TUBWRlmtcu49Y1vUVRpxfO39MO4fv7bdS1JQFw450UjUouysjJERkZe8vu7RS1BzzzzDCRJQkJCAn77299esizel06dOoWNGzc228IDAGlpaQDQbBJkMplgMpk8HiNRW9kcgVsav+GHAhRVWhHrp5Mj1seZoon8U4vfuUII5Ofn4+uvv8aGDRtQUlLixbBa7t1330VcXBwmTpzY7HGZmZkAgMTERB9EReQZgbpgqhDCVRZ/x7DOfjk5Yn2sDCPyTy365NmzZw/mzp2LmJgY7Ny5Ew8++CASExMxdepUfPbZZ7DZbN6Os1GyLOPdd9/F7NmzYTD83Kh18uRJvPDCC8jIyEBOTg6+/PJL3HXXXRg1ahQGDBigSKxEbRGoS2VknLqIE4UVMAfpcOugTkqH026cI4jIP7UoCRo+fDhee+015Obm4osvvsCvfvUrSJKEzz//HLfffjsSExPxwAMPoKCgwNvxutm4cSNOnz6Ne+65x2270WjExo0bceONN6J379547LHHMHXq1GbHDBGpjSwL2AK0MqyuFejmAUl+OzlifWwJIvJPLRoY3ZiysjKsXr0a77//Pnbs2OE2MeITTzyB6dOnY9CgQZ6M1edaOrCKyBtqbA6UVivTyupNp4oqccc/dkMC8OmDI5ES7b9zA9XpEGKE0eDfXXpEgaSl399tftdGRERgzpw52Lp1K3JycvDHP/4RvXv3hhACy5Ytw9ChQ9GnTx+88MILbT0FkaYF6nigj/c6J0e8pmfHgEiAAM4RROSv2twS1JQDBw7g/fffx8cff4yCggJIkuTzknlPYUsQKel8uQVygJWGlVbZMOn1nbDYZbw5cwiGpHZQOqR2kwDERbA8nkhNvN4S1JQhQ4Zg+fLlOHfuHP773/9i+vTpnj4FUcCzO+SAS4AA4PODzskReyWEY3BKlNLheISOrUBEfqvda4c1RafTYcKECZgwYYK3TkEUsAKxK8xql/HpfmdX2J0jUny2rI63caZoIv/FkXxEKhSIpfEbj9ZOjhhmwpg+/j05Yn1sCSLyX0yCiFRGiMArjRdC4M2tJwEAfRLDEeTnkyPWx5YgIv8VOJ9ERAHCYpcRaKOBDp4uQWG5c12+Y/nlCkfjWboA6dYj0iImQUQqYw2wViAA+KR2LFBwkB6/uaqLssF4GFuCiPyX1wZGE1HbBNp4oNySamz/8TwAYMXdw9AtNkzhiDyLcwQR+S+2BBGpiN0hwyEHVmfYvzPOQhbAiK7RAZcA6SQpYKrciLSISRCRigRaV1i11YEvv8sFAEwblqxwNJ7HViAi/8YkiEhFAq0r7KvDeSivsaNzh2Bc2SNG6XA8jkkQkX9jEkSkEkKIgEqChBD4ZJ9zQPQdw5IDsoqKSRCRf2MSRKQSVkdglcbvyS5GTlEVQox6TByQqHQ4XsHKMCL/xiSISCUCqRUIgGuJjJsHJCLMFJiFqIHYukWkJUyCiFQikNYLO11chW9PFEGCsyssULEliMi/MQkiUgGHLAKqNH51bSvQVT06Ijk6ROFovEMC1w0j8ndMgohUIJC6wipq7Fh3KA8AcMfwzgpH4z0cFE3k/5gEEamAxe5QOgSPWXcoF1VWB7p2DMWILtFKh+M1TIKI/B+TICIVCJRJEh2ywOqMswCAO4Z1DujZlJkEEfk/JkFECrPaZYgAGQ606+QFnL1YjXCzARP6B2ZZfB0mQUT+j0kQkcICqSusbnLEWwYlIdioVzga72ISROT/mAQpJJAGwlL7BMrvwk/nK7Av5yJ0EvCroYE7ILqOPoC7+oi0gkmQQgJlDAi1jywL2AOkNL6uFejanrFIjAxWOBrvM+j58Unk7/guVohzHEhgfPlR2wVKMlxabcNXh/MBANOGB+7kiHU4UzRRYGASpBAhRMB8AVLbWWyB8TvwZWYuLHYZPePDMCg5SulwvI4zRRMFBiZBCgqUsSDUdhaH/w+KtssyVmc4u8KmDU8O6LL4OpwpmigwMAlSEJMgbbM5AqM0flvWeRSUWdAhJAg39I1XOhyfYEsQUWBgEqQguywgB8igWGq9QFkwtW5A9K2DO8FkCOyy+DosjycKDEyCFMZxQdoVCC2Bx/LL8N3ZUuh1EqYOCfyy+DpMgogCA5MghQXKwFhqHVkWsAVAAvzpPucSGWP7xCE23KRwNL7DOYKIAgOTIIUFwsBYar1AaAEsqrDgmx+cZfF3DAv8svg6ksSB0USBgkmQwoRAQLQIUOsEwnigzw+eg80h0C8pAv07RSodjs+wFYgocPhlEvTcc89BkiS3W+/evV37a2pqMHfuXMTExCAsLAxTp05FQUGBghE3LxDGhlDr+PtrbnPI+OzAOQDamByxPoPOLz82iagRfvtu7tevH/Ly8ly3nTt3uvbNnz8fa9euxerVq7Ft2zbk5uZiypQpCkbbPH//QqTWsTlkyH5eG7/paCGKKq2IDTNhTO84pcPxKeZARIHDoHQAbWUwGJCQkNBge2lpKd555x189NFHuP766wEA7777Lvr06YPdu3fjiiuu8HWol+ScL0ZoYpI58v+kVwjhKoufMqST5tbQYksQUeDw23fz8ePHkZSUhG7dumHmzJk4ffo0ACAjIwM2mw1jx451Hdu7d2+kpKQgPT292ee0WCwoKytzu/mCQGCMEaGW8fck6HBuGX7IK4NRr8NtgzspHY7PMQciChx++XZOS0vDypUrsX79erz55pvIzs7GNddcg/LycuTn58NoNCIqKsrtMfHx8cjPz2/2eZcsWYLIyEjXLTnZd2MdAqFaiC5NCP8vja9rBbqxXzw6hBoVjsb32BJEFDj8sjtswoQJrp8HDBiAtLQ0pKam4tNPP0VwcHCbn3fRokVYsGCB635ZWZnPEiF/bx2glrHYZfjzaKDC8hpsPlYIQHsDogFAAidKJAokAfEnTVRUFHr27IkTJ04gISEBVqsVJSUlbscUFBQ0OoaoPpPJhIiICLebrzhkAQeX0Ah4/t7i95+Mc3DIAoOTo9AzPlzpcHyO8wMRBZaASIIqKipw8uRJJCYmYujQoQgKCsKmTZtc+7OysnD69GmMHDlSwSgvja1Bgc+fZwivsTnw+UFtlsXX4RxBRIHFL7vDHn/8cUyaNAmpqanIzc3Fs88+C71ejxkzZiAyMhJz5szBggULEB0djYiICDz88MMYOXKkKivD6rPYHQg2amMBSi2y+3lp/Dc/FKC02obESDOu6dlR6XAUodczCSIKJH6ZBJ09exYzZsxAUVERYmNjcfXVV2P37t2IjY0FACxfvhw6nQ5Tp06FxWLBuHHj8Pe//13hqC/N37tKqHn+/PrWL4ufOrSzZgcHsyWIKLBIQvjxn6ZeVlZWhsjISJSWlnp8fFBRhQX2RsYAdQgxwmjQ5hdMoCupsvrtVAgZpy7itx8egDlIh7XzrkZEcJDSISkiMjgI5iC21hKpXUu/v/ltqzL+3FpATRNC+PWYr09rW4Fu6p+o2QQIYGUYUaBhEqQy/vxFSU2zOvy3ND63pBrbj58HANw+rLPC0SjLwCSIKKAwCVIZm0OGzFL5gOPPye3qjLOQBTCiazS6xYYpHY5iJAlc2oYowDAJUiF2iQUefx0LVGW148vMXADaLYuvo9XB4ESBjO9qFfLXL0xqnD9PhPnV9/mosNjRuUMwruweo3Q4imJlGFHgYRKkQv7cdUIN+evrKQuBT/c7B0TfMSwZOo0nAUEGbV8/USBiEqRCshCws0ssYFjsDqVDaJO92cXIKapCiFGPiQMSlQ5HUTpJQjBL44kCDpMgleK4oMDhry1BdZMjThqYhDCTX86r6jGhJj0HRRMFICZBKuXPa0zRz6x+umr86aIq7DpZBAnA7UO1XRbPViCiwMUkSKVsDhmczNv/+WtXWN1YoKt6dERydIjC0SgrzGRgKxBRgGISpFIC7BILBP7YFVZRY8d/v88DwLJ4vU7iosZEAYxJkIr54xco/UyWRaPrw6nd2kO5qLI60LVjKIZ36aB0OIrS+lgookDHJEjFmAT5N3+c78khC6zefxYAcMewzpruBjLoJC6WShTgmASpmN2PJ9kj/0xivz1xAedKqhFhNuCmy7VdFh9mZisQUaBjEqRy/vhFSk4Wh/8Niq4bEH3LoE6abgUJ0utgMmj3+om0gkmQyjEJ8k9Wuwx/K+47WViBfTkXoZOAX2m8LJ5jgYi0gUmQyvljawL5X2WfzSHjqS8OAwB6xIUhIdKscETKMRl0MBr40UikBXynq5wQzi8o8i/+1IJXY3PgidWH8NP5SgBAUYVV4YiUFcpWICLN4LvdD1jsMoL0zFf9hSwLv0lcy6pteGz1dzh0thQGnYRwswH3XtNV6bAUYzbo+V4j0hAmQX7AapcBk9JRUEv5S1dYUYUFj3yciROFFQg3G/DKHQMxoHOU0mEpKtTEwdBEWsIkyA/Ya5fQ0PKcLf7EH+YHyi2pxsOrDuLsxWpEhxrx6oxBuCwuXOmwFGUO0sPAViAiTWES5AcEnF+sWi5Z9idqHw/00/kKPLIqE+crLEiKMuO1GYPRuYO21weTwIowIi3iu95PWB1MgvyB1S5DVnFt/OFzpZj/aSbKqu3o1jEUr84YjNhw9rWajXrodWxpJdIaJkF+wmKTAe1WLfuNCotd6RCatDe7GL//9yFU2xzolxSB5dMGITI4SOmwFCcBCDPyo5BIi/jO9xOyELA7ZI5ZULEam0O1VWFbjhXi6S8Ow+YQGNElGi/96nKE8IsfABBs1EPHViAiTeKnoB+xMglSLSEEymvU2Qr05Xe5WPK/o5AFMLpXLJ6/pT8nA6wlSUAok0EizeK7349Y7TJCjEpHQY2ptDpUORbog92n8NrmEwCAyQOT8OSE3hz7Uk+o0cBWICINYxLkR5zrUbFUXm0cskCVysYCCSHw5raTeG/XKQDArCtSMG90D/7u1CNJQIiRxQZEWsYkyI8IADaHgNHALzI1qaixQ01tQA5ZYNnXWfj84DkAwNzR3XHXyC7KBqVCYSYDk0IijWMS5GcsdgfHc6iI1S6jxq6eRW5tDhnPfXkEG48WQgKwcEJv3Da4k9JhqY5OkhDMKSeINI9JkJ9R+0R8WlNeY1M6BJcamwNP/ud7pP9UBINOwuLJ/TC2b7zSYakSW4GICGAS5HfssoAsCw7mVIFqqwN2WR0dYfUXQjUH6fDS1AG4oluM0mGpkl4nIZhjgYgITIL8ktUhw6zjh7iShBCqmRiRC6G2DpfHIKI6fjm4ZMmSJRg+fDjCw8MRFxeHW2+9FVlZWW7HXHfddZAkye324IMPKhSxZ/nDAp2BTi0l8bkl1bj//QycKKxAdKgRb84awgSoGQadxOVniMjFL5Ogbdu2Ye7cudi9ezc2bNgAm82GG2+8EZWVlW7H3XfffcjLy3PdXn75ZYUi9iyOC1KWWkrifzpfgfv/lYGzF6uRFGXGP+8aqvmV4C8llK1ARFSPX34irF+/3u3+ypUrERcXh4yMDIwaNcq1PSQkBAkJCb4Oz+tkIWBzyAji7NGKUENJPBdCbb0gvY6tQETkJiC+RUtLSwEA0dHRbts//PBDdOzYEf3798eiRYtQVVXV7PNYLBaUlZW53dSKrUHKUENJ/N7sYsz76CDKqu3olxSBt349lAlQC3AsEBH9kt9/KsiyjN/97ne46qqr0L9/f9f2O++8E6mpqUhKSsKhQ4ewcOFCZGVl4bPPPmvyuZYsWYLFixf7Iux2s9plhPJ7z+eULonnQqhtY9TrOL8WETUgCaGC0Z3t8NBDD+Grr77Czp070blz5yaP27x5M8aMGYMTJ06ge/fujR5jsVhgsVhc98vKypCcnIzS0lJERER4NO6iCku7yqslALHhJs514kPVVgfKFEqCHLLA++mn8Na2kxAAesaH4Z3Zw/nF3kLRoUZ2HxNpSFlZGSIjIy/5/e3Xf0LOmzcP69atw/bt25tNgAAgLS0NAJpNgkwmE0wm/2heEXCWypsMHOPgC0qWxOeWVOO5L4/gu7Olrm1l1TYmQC1kMuiYABFRo/wyCRJC4OGHH8bnn3+OrVu3omvXrpd8TGZmJgAgMTHRy9H5jsXOJMhXlCiJF0Jg/ZF8LPs6C5UWB0KMelzXKxYHTl3E7Cu7+DQWf8axQETUFL/8dJg7dy4++ugjfPHFFwgPD0d+fj4AIDIyEsHBwTh58iQ++ugj3HTTTYiJicGhQ4cwf/58jBo1CgMGDFA4es/h4GjfUKIkvqzahpfWH8PGo4UAgAGdI/HcpH7o1CHYp3H4O3OQHga2AhFRE/xyTFBT42Deffdd3H333Thz5gxmzZqFw4cPo7KyEsnJybjtttvw1FNPtWpsT0v7FNuivWOC6nQMM0HPJTS8qrTK5tOKsP05xVi89gcUlluglyTce01X3HVlKgw6fpm3hgQghu8PIk0K6DFBl8rbkpOTsW3bNh9FoyyrXeY6SF7ky5J4q13GW9tO4qM9pyEAJEcHY/HkfuiXFOmT8wcas1HPBIiImuWXSRD9jEmQd/mqJP6n8xV45osjOF5YAQC4ZVASfjf2Mpa/t5EEIJT/74joEvgpoYAvv8vF3uwiPHht93ZXrVgcDgBBngmM3PhilXghBFbvP4vXt5yAxS4jKjgI/29iH1zbM9ar5w10wWwFIqIWYBLkYyVVVjyx+jtY7DI2/lCIv88c0q7BrkI4W4NYLu1ZviiJv1BhwQvrfsDun4oBACO7x+DpiX0QE+Yf0zSoFVuBiKil+M3pY1EhRtcijvllNfj1ij3Y8ENBu57T6mCVmKd5uyR+a1Yh7vznHuz+qRgmgw6P39gTy+8YyATIA0JMBujYCkRELcAkSAELbuiJuHATOkWZUWlx4Kk1h/Gn/x1Fja1tA3BZKu9Z3iyJr7La8cf/HsXC/3yP0mobesWH4717RuD2Ycmc/dsDJAkI5Rg5ImohvyyR9xVvl8jX2B34vx3ZWPltDgSAbh1D8cfb+qNbbFirny82zMS/fj3EWyXxh8+V4tkvj+DsxWpIAH49MhX3j+rG2Yw9KMxkcLW0EpF2tfT7m5++CjLodHjw2u54bcZgxIQa8dOFStz97j6sOXjuktMA/BK7xDzDGyXxdlnG/+34Cff/KwNnL1YjPsKEv88cgrmjezAB8iC9TkIIW4GIqBX4CawCw7tG44N703BFt2hY7DKWfHUMT605jIqalnfJWNgl5hGeLok/e7EKD75/AP/ckQ2HEBjXLx4f3puGIakdPHoeLZMAhJoMiAk1skuRiFqF7cYqER1qxPJpg/DhntN4c+tJbDxaiB/yyvDirf1bNFkexwW1nydL4oUQWHcoD69s+BFVVgfCTAb8fnwvjOuX4JHnJyezQY8ws4Hl8ETUJkyCVEQnSfj1FakYnByFp9YcRm5JDe77Vwbmju6OGSNSoGvmr1xZCNgdMtdJaiNPlsSXVtnwp6+OYmvWeQDA4OQoPDu5LxIjue6Xpxh0EsLNQZwagojahZ8gKtS/UyTenzMCY3rHwSELvLrpBBZ8+h0uVlqbfRzHBbVdhcXe7pJ4IQS+PXEBd/7fbmzNOg+DTsK80T3wxswhTIA8RCdJiDAHISbMxASIiNqN1WHNUHoBVSEE1mTmYvmGH2Gxy+gYZsTiyf0wrEt0o8cb9Tp0CDV6NE4tcMgCRRUWtPWNcLq4Cl8fzsf6I/k4e7EaABAdGoTl0wahd4Jnf2+0SoJzFugwk4HjfojokgJ6AVWtkCQJtw3uhMs7ReIPn3+PnKIqzPvoIH5zVRfMuaZrg1XFbQ4ZQgh+SbRSRY291QnQxUorNh4twFeH83Ekt6zB/iC9jgmQh3DcDxF5C5MgP9AjLgwrfzMCr2z4EV9+l4sV3+bgwOkSPH9LP8RHmF3HCTi7xEwGlgm3VGtK4mtsDuw4fgHrD+cj/aciOGpb8vSShBHdojGhfwIuVlqxau8ZzL4y1ZthawLH/RCRt7E7rBlKd4c15usj+Vj61TFUWR2ICDbgmZv74prLfl5sM8SoR7iZC6q21KVeB4cscPD0RXx1OB+bjxWiyvpzwtQnMRzj+yXghr7xXO7CgyQJCDcFIZhz/hBRG7E7LECN65eAfkkReGrNYRzNK8fjqw9h2vBkzBvdA0aDjqXyrdBcSfzxwnKsP5yPr48U4Hy5xbU9MdKM8f0TML5fArp0DPVVqJrAcT9E5GtsCWqGGluC6tgcMv6+5SQ+2nsaANArIRwv3tofKdEhXEKjBYQQuFBhdasIKyirwTc/FGD94XycKKxwbY8wGzCmTzzG90/AgM6RzU5VQG1jMugQZjJwigci8oiWfn8zCWqGmpOgOjtPXMDza39AabUNIUY9Fo7vjTuGJbMr4RLKa2yosjpQYbFjy7FCrD+cj4xTF10DpIP0Eq7q0RE39U/EyO4xHJfiJXqdhHCzgePYiMijmAR5gD8kQQBQWF6DZ784ggOnSwAAkwcmYfHkfgg26qGTJOh1EnQS2MVQq8bmwH8P5eGrw3nYcfyC25Ijg5OjML5/Aq7vHYeIYI6t8hZJci52GmJkjzwReR6TIA/wlyQIcA7gfffbbLyzMxtNPa1OQm1CJP3iX/ftje3T6yToJQk6nYSLlVbkl9UgIcKMmDAThBA/l5gLQECg7rdKABC12+D6ufYHt/1OJVVWlFTZEBUShKgQo3M2z9oEToLzy1OCMy7U/uzcVv+Y+tt+PgYACsstyCmqRP3f+i4xIZjQPxHj+sdzUkMvkwCYjXqEGQ3ssiUir2ES5AHeTIIqLXY4vPC/fn9OMe57bz8cfFWbpZOA6SNScFP/BPROCIeuds6lusTJ+XNtJgW4kirnz1K9n2uPq/e4S2nsZReNzFTU+HFwO3edS8Xgdk0NttU/TsIvPxLq36u/65cxN/frXLcvSC9x3A8ReR2rw1Qu1OSd//XX947Hc5P74c2tJzHnmm741dDOkGUBhxCufx2ygCzj559r/63/s/NfuO+vfY5NtWNoJvRPwHW94n5ucflF8vDLpKH+/vpJRf0WnQ0/FOCLzHO4ZVAnjO0T72pVEvVamJytR+LnfwUa2f6Ln2sfu+P4BWw+VoBHrr8Md13ZxSuvgf9jCw0RaQNbgprhzZYgIiIi8o6Wfn+zXZqIiIg0iUkQERERaRKTICIiItIkJkFERESkSUyCiIiISJOYBBEREZEmMQkiIiIiTWISRERERJrEJIiIiIg0iUkQERERaVLAJ0FvvPEGunTpArPZjLS0NOzdu1fpkIiIiEgFAjoJ+uSTT7BgwQI8++yzOHDgAAYOHIhx48ahsLBQ6dCIiIhIYQGdBL3yyiu477778Jvf/AZ9+/bFW2+9hZCQEKxYsULp0IiIiEhhAZsEWa1WZGRkYOzYsa5tOp0OY8eORXp6eqOPsVgsKCsrc7sRERFRYDIoHYC3XLhwAQ6HA/Hx8W7b4+PjcezYsUYfs2TJEixevLjBdiZDRERE/qPue1sI0exxAZsEtcWiRYuwYMEC1/1z586hb9++SE5OVjAqIiIiaovy8nJERkY2uT9gk6COHTtCr9ejoKDAbXtBQQESEhIafYzJZILJZHLdDwsLw5kzZxAeHg5JkjwWW1lZGZKTk3HmzBlERER47HnVSkvXy2sNXFq6Xl5r4NLK9QohUF5ejqSkpGaPC9gkyGg0YujQodi0aRNuvfVWAIAsy9i0aRPmzZvXoufQ6XTo3Lmz12KMiIgI6F/CX9LS9fJaA5eWrpfXGri0cL3NtQDVCdgkCAAWLFiA2bNnY9iwYRgxYgT++te/orKyEr/5zW+UDo2IiIgUFtBJ0LRp03D+/Hk888wzyM/Px6BBg7B+/foGg6WJiIhIewI6CQKAefPmtbj7y1dMJhOeffZZt/FHgUxL18trDVxaul5ea+DS2vVeiiQuVT9GREREFIACdrJEIiIiouYwCSIiIiJNYhJEREREmsQkiIiIiDSJSZAC3njjDXTp0gVmsxlpaWnYu3ev0iG125IlSzB8+HCEh4cjLi4Ot956K7KystyOue666yBJktvtwQcfVCjitnvuuecaXEfv3r1d+2tqajB37lzExMQgLCwMU6dObTBzuT/p0qVLg+uVJAlz584F4N+v6/bt2zFp0iQkJSVBkiSsWbPGbb8QAs888wwSExMRHByMsWPH4vjx427HFBcXY+bMmYiIiEBUVBTmzJmDiooKH15FyzR3rTabDQsXLsTll1+O0NBQJCUl4a677kJubq7bczT2u7B06VIfX0nLXOq1vfvuuxtcy/jx492OCYTXFkCj719JkrBs2TLXMf702noSkyAf++STT7BgwQI8++yzOHDgAAYOHIhx48ahsLBQ6dDaZdu2bZg7dy52796NDRs2wGaz4cYbb0RlZaXbcffddx/y8vJct5dfflmhiNunX79+btexc+dO17758+dj7dq1WL16NbZt24bc3FxMmTJFwWjbZ9++fW7XumHDBgDA7bff7jrGX1/XyspKDBw4EG+88Uaj+19++WW8+uqreOutt7Bnzx6EhoZi3LhxqKmpcR0zc+ZMHDlyBBs2bMC6deuwfft23H///b66hBZr7lqrqqpw4MABPP300zhw4AA+++wzZGVlYfLkyQ2Off75591e64cfftgX4bfapV5bABg/frzbtaxatcptfyC8tgDcrjEvLw8rVqyAJEmYOnWq23H+8tp6lCCfGjFihJg7d67rvsPhEElJSWLJkiUKRuV5hYWFAoDYtm2ba9u1114rHn30UeWC8pBnn31WDBw4sNF9JSUlIigoSKxevdq17ejRowKASE9P91GE3vXoo4+K7t27C1mWhRCB87oCEJ9//rnrvizLIiEhQSxbtsy1raSkRJhMJrFq1SohhBA//PCDACD27dvnOuarr74SkiSJc+fO+Sz21vrltTZm7969AoA4deqUa1tqaqpYvny5d4Pzgsaud/bs2eKWW25p8jGB/Nrecsst4vrrr3fb5q+vbXuxJciHrFYrMjIyMHbsWNc2nU6HsWPHIj09XcHIPK+0tBQAEB0d7bb9ww8/RMeOHdG/f38sWrQIVVVVSoTXbsePH0dSUhK6deuGmTNn4vTp0wCAjIwM2Gw2t9e4d+/eSElJCYjX2Gq14oMPPsA999zjtqhwoLyu9WVnZyM/P9/ttYyMjERaWprrtUxPT0dUVBSGDRvmOmbs2LHQ6XTYs2ePz2P2pNLSUkiShKioKLftS5cuRUxMDAYPHoxly5bBbrcrE6AHbN26FXFxcejVqxceeughFBUVufYF6mtbUFCA//73v5gzZ06DfYH02rZUwM8YrSYXLlyAw+FosGxHfHw8jh07plBUnifLMn73u9/hqquuQv/+/V3b77zzTqSmpiIpKQmHDh3CwoULkZWVhc8++0zBaFsvLS0NK1euRK9evZCXl4fFixfjmmuuweHDh5Gfnw+j0djgiyM+Ph75+fnKBOxBa9asQUlJCe6++27XtkB5XX+p7vVq7P1aty8/Px9xcXFu+w0GA6Kjo/369a6pqcHChQsxY8YMt0U2H3nkEQwZMgTR0dHYtWsXFi1ahLy8PLzyyisKRts248ePx5QpU9C1a1ecPHkS/+///T9MmDAB6enp0Ov1AfvavvfeewgPD2/QRR9Ir21rMAkij5s7dy4OHz7sNk4GgFtf+uWXX47ExESMGTMGJ0+eRPfu3X0dZptNmDDB9fOAAQOQlpaG1NRUfPrppwgODlYwMu975513MGHCBCQlJbm2BcrrSk42mw133HEHhBB488033fYtWLDA9fOAAQNgNBrxwAMPYMmSJX63DMP06dNdP19++eUYMGAAunfvjq1bt2LMmDEKRuZdK1aswMyZM2E2m922B9Jr2xrsDvOhjh07Qq/XN6gUKigoQEJCgkJReda8efOwbt06bNmyBZ07d2722LS0NADAiRMnfBGa10RFRaFnz544ceIEEhISYLVaUVJS4nZMILzGp06dwsaNG3Hvvfc2e1ygvK51r1dz79eEhIQGRQ12ux3FxcV++XrXJUCnTp3Chg0b3FqBGpOWlga73Y6cnBzfBOhF3bp1Q8eOHV2/t4H22gLAjh07kJWVdcn3MBBYr21zmAT5kNFoxNChQ7Fp0ybXNlmWsWnTJowcOVLByNpPCIF58+bh888/x+bNm9G1a9dLPiYzMxMAkJiY6OXovKuiogInT55EYmIihg4diqCgILfXOCsrC6dPn/b71/jdd99FXFwcJk6c2OxxgfK6du3aFQkJCW6vZVlZGfbs2eN6LUeOHImSkhJkZGS4jtm8eTNkWXYlg/6iLgE6fvw4Nm7ciJiYmEs+JjMzEzqdrkG3kT86e/YsioqKXL+3gfTa1nnnnXcwdOhQDBw48JLHBtJr2yylR2ZrzccffyxMJpNYuXKl+OGHH8T9998voqKiRH5+vtKhtctDDz0kIiMjxdatW0VeXp7rVlVVJYQQ4sSJE+L5558X+/fvF9nZ2eKLL74Q3bp1E6NGjVI48tZ77LHHxNatW0V2drb49ttvxdixY0XHjh1FYWGhEEKIBx98UKSkpIjNmzeL/fv3i5EjR4qRI0cqHHX7OBwOkZKSIhYuXOi23d9f1/LycnHw4EFx8OBBAUC88sor4uDBg66KqKVLl4qoqCjxxRdfiEOHDolbbrlFdO3aVVRXV7ueY/z48WLw4MFiz549YufOneKyyy4TM2bMUOqSmtTctVqtVjF58mTRuXNnkZmZ6fYetlgsQgghdu3aJZYvXy4yMzPFyZMnxQcffCBiY2PFXXfdpfCVNa656y0vLxePP/64SE9PF9nZ2WLjxo1iyJAh4rLLLhM1NTWu5wiE17ZOaWmpCAkJEW+++WaDx/vba+tJTIIU8Nprr4mUlBRhNBrFiBEjxO7du5UOqd0ANHp79913hRBCnD59WowaNUpER0cLk8kkevToIZ544glRWlqqbOBtMG3aNJGYmCiMRqPo1KmTmDZtmjhx4oRrf3V1tfjtb38rOnToIEJCQsRtt90m8vLyFIy4/b7++msBQGRlZblt9/fXdcuWLY3+3s6ePVsI4SyTf/rpp0V8fLwwmUxizJgxDf4fFBUViRkzZoiwsDAREREhfvOb34jy8nIFrqZ5zV1rdnZ2k+/hLVu2CCGEyMjIEGlpaSIyMlKYzWbRp08f8ac//cktaVCT5q63qqpK3HjjjSI2NlYEBQWJ1NRUcd999zX4YzQQXts6//jHP0RwcLAoKSlp8Hh/e209SRJCCK82NRERERGpEMcEERERkSYxCSIiIiJNYhJEREREmsQkiIiIiDSJSRARERFpEpMgIiIi0iQmQURERKRJTIKIiIhIk5gEEZFXSZJ0ydvdd9/d7vPcfffdkCQJW7dubfdzeYoaYyKinxmUDoCItGH27NlN7rv66qt9GAkRkROTICLyiZUrV3r1+ZcsWYInn3wSKSkpXj0PEQUOJkFEFBASExORmJiodBhE5Ec4JoiIVEeSJHTp0gVWqxXPPvssunfvDrPZjG7duuGZZ55BTU1Ng8c0Nf7m/PnzePLJJ9G3b1+EhYUhMjISPXv2xF133YW9e/c2eJ4ffvgBM2fORGJiIoxGIzp16oS77roLWVlZTca7YsUKDBo0CMHBwUhISMDdd9+N/Pz8Zq+xuLgYixYtQt++fREcHIzIyEhcf/31WLduXcv+JxFRuzEJIiJVEkJg6tSpWLZsGfr27YuJEyeiuLgYL7zwAm6++WY4HI5LPkd5eTnS0tLw0ksvoaKiAjfccANuvPFGdOjQAR9//DH+97//uR2/adMmDBs2DB999BESExMxdepUxMXF4f3338ewYcOwY8eOBud48sknMWfOHPzwww8YNWoURo0aha+++gppaWkoLi5uNK4ff/wRgwYNwtKlS1FdXY1x48Zh2LBh2LNnDyZNmoQ///nPbfufRkStI4iIvAiAaO1HTd1jOnfuLE6ePOnaXlhYKPr37y8AiOXLl7s9Zvbs2QKA2LJli2vbihUrBAAxefJk4XA43I4vLCwU33//vet+RUWFiI+PFwDE66+/7nbsK6+84oqnurratT09PV1IkiQiIyPFgQMHXNvLy8vF9ddf77qO+jHZ7XZx+eWXCwDi5Zdfdovr+PHjomvXrkKv17vFRkTewZYgIvKJ5krk16xZ0+hjnnnmGXTr1s11PzY2FsuWLQMAvP7665c85/nz5wEA119/PXQ694+72NhY9O/f33X/008/RUFBAUaOHIm5c+e6HTt//nwMHToUZ8+exX/+8x/X9jfffBNCCDz66KMYPHiwa3tYWBhee+01SJLUIKa1a9fi+++/x9SpU/HEE0+4xdWjRw/85S9/gcPhwD//+c9LXh8RtQ8HRhORTzRXIt9URdf06dMbbBs/fjw6dOiAkydPIi8vr9nB0EOHDgUALFu2DPHx8Zg4cSLCw8MbPbauq2vmzJmN7p81axYyMjKwY8cO1zF1j2kszr59+2LgwIHIzMx02/7NN98AAKZMmdLoea655hoAaHS8EhF5FpMgIvKJ1pbId+jQocmEJTU1FRcvXkRubm6zSdCYMWMwf/58/PWvf8WMGTNgMBgwZMgQ3HDDDbjnnnvcWplyc3MBAF26dGn0ueq2nzt3rsFjUlNTm3zML5OgnJwcAM5kq6mECwAuXLjQ5D4i8gwmQUQU0F555RU88MAD+OKLL7Bx40Z8++232Lt3L15++WWsWrUKU6dObdHzNNa11RayLANwtmjFx8c3eVzHjh09cj4iahqTICJSpYsXL6K8vLzR1qDTp08DAJKSklr0XL169cLvf/97/P73v0dNTQ1ef/11PPHEE3jooYdcSVDdc506darR56hrwenUqZNrW2JiInJycnDq1Cn06dOnwWMae67OnTsDAO69994WJ2BE5B0cGE1EqvXpp5822PbNN9+guLgY3bp1a9PkiGazGY8//jgSExNx/vx5FBYWAvh5LM6qVasafdwHH3zgdlz9nxuL89ixYw26wgDghhtuAAB8/vnnrY6diDyLSRARqdbixYtdLTCAc5zME088AQANKrgas2bNGuzevbvB9oyMDBQUFCAsLAxRUVEAgDvuuAPx8fHYuXMn3n77bbfjX331Vezfvx+dOnVya7158MEHAQB//etf8d1337m2V1ZW4uGHH4YQosG5p06dir59++LDDz/ECy+8AIvF4rZfCIFvv/0W33777SWvj4jah91hROQTza0Un5KSgueff77BtgEDBqBfv34YM2YMgoKCsHnzZpSUlGD06NF45JFHLnnOrVu34m9/+xs6deqEwYMHIyIiArm5udixYwdkWcbixYthNBoBAKGhofjwww8xadIkPPDAA3j77bfRs2dPHDt2DAcPHkRYWBhWrVoFs9nsev4rr7wSjz/+OP785z9j+PDhuP766xEZGYlt27bBZDJh0qRJWLt2rVtMBoMBa9aswbhx4/DMM8/g9ddfx4ABAxAXF4cLFy4gMzMThYWFWL58Oa666qpW/B8molZTeJ4iIgpwqJ0wsLnbwIEDGzwmNTVV1NTUiP/3//6f6NKlizAajSI1NVX84Q9/EFVVVQ3O09hkiQcPHhSPPfaYGD58uIiLixMmk0mkpqaKSZMmiY0bNzYa7+HDh8WMGTNEfHy8CAoKEomJiWLWrFni2LFjTV7jP//5TzFgwABhMplEXFycmDVrljh37lyjMdUpKSkRL774ohgyZIgICwsTZrNZdOnSRYwbN0688cYb4vz58y36/0tEbScJ0Uh7LRGRgiRJQmpqqltXGBGRp3FMEBEREWkSkyAiIiLSJCZBREREpEmsDiMi1eFQRSLyBbYEERERkSYxCSIiIiJNYhJEREREmsQkiIiIiDSJSRARERFpEpMgIiIi0iQmQURERKRJTIKIiIhIk/4/GWRwEs13RNgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Set environment and training parameters\n",
    "env_name = 'CartPole-v0'\n",
    "num_episodes_train = 200\n",
    "num_episodes_test = 20\n",
    "learning_rate = 5e-4\n",
    "\n",
    "# Create the environment\n",
    "env = gym.make(env_name)\n",
    "action_space_size = env.action_space.n\n",
    "state_space_size = 4\n",
    "\n",
    "# Plot average performance of 5 trials\n",
    "num_seeds = 5\n",
    "l = num_episodes_train // 10\n",
    "res = np.zeros((num_seeds, l))\n",
    "gamma = 0.99\n",
    "\n",
    "# Loop over multiple seeds\n",
    "for i in tqdm.tqdm(range(num_seeds)):\n",
    "    reward_means = []\n",
    "\n",
    "    # Create an instance of the DQN_Agent class\n",
    "    agent = DQN_Agent(env_name, lr=learning_rate)\n",
    "\n",
    "    # Training loop\n",
    "    for m in range(num_episodes_train):\n",
    "        agent.train()\n",
    "\n",
    "        # Evaluate the agent every 10 episodes during training\n",
    "        if m % 10 == 0:\n",
    "            print(\"Episode: {}\".format(m))\n",
    "\n",
    "            # Evaluate the agent's performance over 20 test episodes\n",
    "            G = np.zeros(num_episodes_test)\n",
    "            for k in range(num_episodes_test):\n",
    "                g = agent.test()\n",
    "                G[k] = g\n",
    "\n",
    "            reward_mean = G.mean()\n",
    "            reward_sd = G.std()\n",
    "            print(f\"The test reward for episode {m} is {reward_mean} with a standard deviation of {reward_sd}.\")\n",
    "            reward_means.append(reward_mean)\n",
    "\n",
    "    res[i] = np.array(reward_means)\n",
    "\n",
    "# Plotting the average performance\n",
    "ks = np.arange(l) * 10\n",
    "avs = np.mean(res, axis=0)\n",
    "maxs = np.max(res, axis=0)\n",
    "mins = np.min(res, axis=0)\n",
    "\n",
    "plt.fill_between(ks, mins, maxs, alpha=0.1)\n",
    "plt.plot(ks, avs, '-o', markersize=1)\n",
    "\n",
    "plt.xlabel('Episode', fontsize=15)\n",
    "plt.ylabel('Avg. Return', fontsize=15)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8031648e",
   "metadata": {},
   "source": [
    "The plot above illustrates the average return, representing the total reward received by the agent in an episode, across 5 different runs. It is evident that the agent progressively learns a more effective strategy to achieve higher returns with an increasing number of episodes.\n",
    "\n",
    "Now, let's observe the behavior of the trained agent in the environment.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f1b093e9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAF7CAYAAAD4/3BBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAAArJklEQVR4nO3df3RU9Z3/8ddMkhkSwkwIkEwiCaKgECHYAoapraVLSkB0ZY3nqGUFuxz5yiaeaqzFdK2K3WNc3bP+6Cp8z9ldcc9XSmuPaKWCIkhYa0CMpPzSVFjaYMkkKGYmCWSSzHy+f7jMOgokEyaZO/B8nHPPydz7njvv+zmRvLxz7+fajDFGAAAAFmJPdAMAAABfRUABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWk9CA8uyzz+riiy/WsGHDVFJSovfeey+R7QAAAItIWED51a9+paqqKj300EP64IMPNG3aNJWVlam1tTVRLQEAAIuwJephgSUlJZo5c6b+9V//VZIUDodVUFCgu+66S/fff38iWgIAABaRmogP7e7uVn19vaqrqyPr7Ha7SktLVVdX97X6YDCoYDAYeR0Oh3X8+HGNGjVKNpttSHoGAADnxhij9vZ25efny24/+5c4CQkon376qUKhkHJzc6PW5+bm6qOPPvpafU1NjVauXDlU7QEAgEF05MgRjR079qw1CQkosaqurlZVVVXktd/vV2FhoY4cOSKXy5XAzgAAQH8FAgEVFBRoxIgRfdYmJKCMHj1aKSkpamlpiVrf0tIij8fztXqn0ymn0/m19S6Xi4ACAECS6c/lGQm5i8fhcGj69OnasmVLZF04HNaWLVvk9XoT0RIAALCQhH3FU1VVpSVLlmjGjBm66qqr9NRTT6mzs1M//OEPE9USAACwiIQFlJtvvlnHjh3Tgw8+KJ/PpyuvvFKbNm362oWzAADgwpOweVDORSAQkNvtlt/v5xoUAACSRCx/v3kWDwAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsBwCCgAAsJy4B5SHH35YNpstapk0aVJke1dXlyoqKjRq1ChlZmaqvLxcLS0t8W4DAAAksUE5g3LFFVeoubk5srzzzjuRbffcc49ee+01vfTSS6qtrdXRo0d14403DkYbAAAgSaUOyk5TU+XxeL623u/369///d+1du1a/dVf/ZUk6fnnn9fkyZO1Y8cOzZo1azDaAQAASWZQzqB8/PHHys/P1yWXXKJFixapqalJklRfX6+enh6VlpZGaidNmqTCwkLV1dWdcX/BYFCBQCBqAQAA56+4B5SSkhKtWbNGmzZt0qpVq3T48GF95zvfUXt7u3w+nxwOh7KysqLek5ubK5/Pd8Z91tTUyO12R5aCgoJ4tw0AACwk7l/xzJ8/P/JzcXGxSkpKNG7cOP36179Wenr6gPZZXV2tqqqqyOtAIEBIAQDgPDbotxlnZWXpsssu08GDB+XxeNTd3a22traompaWltNes3KK0+mUy+WKWgAAwPlr0ANKR0eHDh06pLy8PE2fPl1paWnasmVLZHtjY6Oamprk9XoHuxUAAJAk4v4Vz49//GNdf/31GjdunI4ePaqHHnpIKSkpuvXWW+V2u7V06VJVVVUpOztbLpdLd911l7xeL3fwAACAiLgHlE8++US33nqrPvvsM40ZM0bf/va3tWPHDo0ZM0aS9OSTT8put6u8vFzBYFBlZWV67rnn4t0GAABIYjZjjEl0E7EKBAJyu93y+/1cjwIAQJKI5e83z+IBAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWQ0ABAACWE3NA2b59u66//nrl5+fLZrPplVdeidpujNGDDz6ovLw8paenq7S0VB9//HFUzfHjx7Vo0SK5XC5lZWVp6dKl6ujoOKcDAQAA54+YA0pnZ6emTZumZ5999rTbH3/8cT3zzDNavXq1du7cqeHDh6usrExdXV2RmkWLFmn//v3avHmzNmzYoO3bt2vZsmUDPwoAAHBesRljzIDfbLNp/fr1WrhwoaQvzp7k5+fr3nvv1Y9//GNJkt/vV25urtasWaNbbrlFH374oYqKirRr1y7NmDFDkrRp0yZde+21+uSTT5Sfn9/n5wYCAbndbvn9frlcroG2DwAAhlAsf7/jeg3K4cOH5fP5VFpaGlnndrtVUlKiuro6SVJdXZ2ysrIi4USSSktLZbfbtXPnztPuNxgMKhAIRC0AAOD8FdeA4vP5JEm5ublR63NzcyPbfD6fcnJyoranpqYqOzs7UvNVNTU1crvdkaWgoCCebQMAAItJirt4qqur5ff7I8uRI0cS3RIAABhEcQ0oHo9HktTS0hK1vqWlJbLN4/GotbU1antvb6+OHz8eqfkqp9Mpl8sVtQAAgPNXXAPK+PHj5fF4tGXLlsi6QCCgnTt3yuv1SpK8Xq/a2tpUX18fqdm6davC4bBKSkri2Q4AAEhSqbG+oaOjQwcPHoy8Pnz4sBoaGpSdna3CwkLdfffd+sd//EdNnDhR48eP189+9jPl5+dH7vSZPHmy5s2bpzvuuEOrV69WT0+PKisrdcstt/TrDh4AAHD+izmgvP/++/re974XeV1VVSVJWrJkidasWaOf/OQn6uzs1LJly9TW1qZvf/vb2rRpk4YNGxZ5z4svvqjKykrNmTNHdrtd5eXleuaZZ+JwOAAA4HxwTvOgJArzoAAAkHwSNg8KAABAPBBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5RBQAACA5cQcULZv367rr79e+fn5stlseuWVV6K233777bLZbFHLvHnzomqOHz+uRYsWyeVyKSsrS0uXLlVHR8c5HQgAADh/xBxQOjs7NW3aND377LNnrJk3b56am5sjyy9/+cuo7YsWLdL+/fu1efNmbdiwQdu3b9eyZcti7x4AAJyXUmN9w/z58zV//vyz1jidTnk8ntNu+/DDD7Vp0ybt2rVLM2bMkCT94he/0LXXXqt//ud/Vn5+fqwtAQCA88ygXIOybds25eTk6PLLL9fy5cv12WefRbbV1dUpKysrEk4kqbS0VHa7XTt37jzt/oLBoAKBQNQCAADOX3EPKPPmzdN//ud/asuWLfqnf/on1dbWav78+QqFQpIkn8+nnJycqPekpqYqOztbPp/vtPusqamR2+2OLAUFBfFuGwAAWEjMX/H05ZZbbon8PHXqVBUXF+vSSy/Vtm3bNGfOnAHts7q6WlVVVZHXgUCAkAIAwHls0G8zvuSSSzR69GgdPHhQkuTxeNTa2hpV09vbq+PHj5/xuhWn0ymXyxW1AACA89egB5RPPvlEn332mfLy8iRJXq9XbW1tqq+vj9Rs3bpV4XBYJSUlg90OAABIAjF/xdPR0RE5GyJJhw8fVkNDg7Kzs5Wdna2VK1eqvLxcHo9Hhw4d0k9+8hNNmDBBZWVlkqTJkydr3rx5uuOOO7R69Wr19PSosrJSt9xyC3fwAAAASZLNGGNiecO2bdv0ve9972vrlyxZolWrVmnhwoXavXu32tralJ+fr7lz5+rnP/+5cnNzI7XHjx9XZWWlXnvtNdntdpWXl+uZZ55RZmZmv3oIBAJyu93y+/183QMAQJKI5e93zAHFCggoAAAkn1j+fvMsHgAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkEFAAAYDkxPywQAHBujDE6tHm1UhwZSk0fobRhI5SaMUJpw1xKTc9UWrpLaekjZLOnJLpVIGEIKAAwxHq7OtTWtFcmHJbNbpfNZpdsdtnsKbLZ7JF1ReX/oLR0njeGCxMBBQCGWHdnm2SMZMIyobDO9MTWJHyWKxA3XIMCAEOs50RbolsALI+AAgBD7PPDu2XC4US3AVgaAQUAhlhny39LZ/xi5wuugitkT3EMTUOABRFQAMCC0kdeJHsKd/HgwkVAAQALSstwSTb+icaFi99+ABhCJnzmu3a+LC3d9cXtx8AFit9+ABhCoe6TUjjUZ12qc7hksw1BR4A1EVAAYAj1dLUr3I+A8sWkbQQUXLgIKAAwhHpPdsiEexPdBmB5BBQAGEJdgVaFe7sT3QZgeQQUABhC7X/5UKHgibPWODJHKdWZMUQdAdZEQAEAixmec7HShmclug0goQgoAGAxKY4M2VPSEt0GkFAEFAAYIsYY9ecBxanODNlTmeYeFzYCCgAMkXBPl8K9wT7rUhzpsqWkDkFHgHURUABgiIS6uxTu6TugyGZjDhRc8AgoADBEQj0nFerHGRQAMQaUmpoazZw5UyNGjFBOTo4WLlyoxsbGqJquri5VVFRo1KhRyszMVHl5uVpaWqJqmpqatGDBAmVkZCgnJ0f33XefenuZuAjA+a3dd1AnPm1KdBtAUogpoNTW1qqiokI7duzQ5s2b1dPTo7lz56qzszNSc8899+i1117TSy+9pNraWh09elQ33nhjZHsoFNKCBQvU3d2td999Vy+88ILWrFmjBx98MH5HBQAWFO7tkQmd/X/GbPYU7uABJNmM6c815ad37Ngx5eTkqLa2Vtdcc438fr/GjBmjtWvX6qabbpIkffTRR5o8ebLq6uo0a9Ysbdy4Udddd52OHj2q3NxcSdLq1au1YsUKHTt2TA5H31euBwIBud1u+f1+uVyugbYPAEPKt3eLjrz7q7PWZIwZp/Gzb1dG9kVD1BUwdGL5+31O16D4/X5JUnZ2tiSpvr5ePT09Ki0tjdRMmjRJhYWFqqurkyTV1dVp6tSpkXAiSWVlZQoEAtq/f/9pPycYDCoQCEQtAHA+Skl1KsWRnug2gIQbcEAJh8O6++67dfXVV2vKlCmSJJ/PJ4fDoaysrKja3Nxc+Xy+SM2Xw8mp7ae2nU5NTY3cbndkKSgoGGjbAJAQxhipH08xtqc6CCiAziGgVFRUaN++fVq3bl08+zmt6upq+f3+yHLkyJFB/0wAiCcT6lVv98k+6+ypaUpJGzYEHQHWNqCZgCorK7VhwwZt375dY8eOjaz3eDzq7u5WW1tb1FmUlpYWeTyeSM17770Xtb9Td/mcqvkqp9Mpp9M5kFYBwBLCoR71dnX2XSjmQAGkGM+gGGNUWVmp9evXa+vWrRo/fnzU9unTpystLU1btmyJrGtsbFRTU5O8Xq8kyev1au/evWptbY3UbN68WS6XS0VFRedyLABgWSbUo1CwPwEFgBTjGZSKigqtXbtWr776qkaMGBG5ZsTtdis9PV1ut1tLly5VVVWVsrOz5XK5dNddd8nr9WrWrFmSpLlz56qoqEi33XabHn/8cfl8Pj3wwAOqqKjgLAmA81bPyXZ1HvtTotsAkkZMAWXVqlWSpNmzZ0etf/7553X77bdLkp588knZ7XaVl5crGAyqrKxMzz33XKQ2JSVFGzZs0PLly+X1ejV8+HAtWbJEjzzyyLkdCQBYWKj7pIKBY2etsacNk2vs5CHqCLC2c5oHJVGYBwVAsmlv/lgf/faJs9akpo/QpaX/R678y4aoK2BoDdk8KACA+LHZUpSWPiLRbQCWQEABgEFmjJHpxxwoNrtdqQQUQBIBBQAGnzHqOdn3DNg2m12pjowhaAiwPgIKAAw6o55Of99lNptsdv5ZBiQCCgAMOhMOK9jxWaLbAJIKAQUABlk41K3WfVsT3QaQVAgoAGARw8dcnOgWAMsgoACAFdhsGnnpjER3AVgGAQUABls/p8N0DM8a1DaAZEJAAYBBZIxRONTdj0qbHMNHDno/QLIgoADAIOvubOtXHbPIAv+LgAIAg6y74/NEtwAkHQIKAAyy7k4CChArAgoADLKWPW/1WWOz2YagEyB5EFAAYJCFgp191uRMmSMRUoAIAgoAWIAjc6QkAgpwCgEFACyAW4yBaAQUABhE4d7ufs3TxiRtQDQCCgAMop4Tfsn0HVFSh2UOQTdA8iCgAMAg6j7hV7/murfZuJMH+BICCgAMop7ONhkTTnQbQNIhoADAIAp88qFMOJToNoCkQ0ABgEF08vgnfV6D4iqYopS0YUPUEZAcCCgAkGDpI/NkT01LdBuApRBQAGCQmH7cvSNJacNGyGZLGeRugORCQAGAQRLu7e7XBbKp6SNksxNQgC8joADAIOkNdsqEevusS3EM4zk8wFcQUABgkPR2dSoc7jug2Gx25kABvoKAAgCDpOeEX+HenkS3ASQlAgoADJK2PzWop/PzRLcBJCUCCgAkkCNzFM/hAU4jpoBSU1OjmTNnasSIEcrJydHChQvV2NgYVTN79mzZ/ueZEqeWO++8M6qmqalJCxYsUEZGhnJycnTfffept7fv72kB4HyTMbpQzhGjE90GYDmpsRTX1taqoqJCM2fOVG9vr376059q7ty5OnDggIYPHx6pu+OOO/TII49EXmdkZER+DoVCWrBggTwej9599101Nzdr8eLFSktL06OPPhqHQwKAxOvvHCgpjmGypzoGuRsg+cQUUDZt2hT1es2aNcrJyVF9fb2uueaayPqMjAx5PJ7T7uPNN9/UgQMH9NZbbyk3N1dXXnmlfv7zn2vFihV6+OGH5XDwHyqA5GfCvQqH+r5ANtWZIXsa/+4BX3VO16D4/X5JUnZ2dtT6F198UaNHj9aUKVNUXV2tEydORLbV1dVp6tSpys3NjawrKytTIBDQ/v37T/s5wWBQgUAgagEAKwt1dynU09VnnT3VKXsK09wDXxXTGZQvC4fDuvvuu3X11VdrypQpkfU/+MEPNG7cOOXn52vPnj1asWKFGhsb9fLLL0uSfD5fVDiRFHnt8/lO+1k1NTVauXLlQFsFgCEX7gkq3BNMdBtA0hpwQKmoqNC+ffv0zjvvRK1ftmxZ5OepU6cqLy9Pc+bM0aFDh3TppZcO6LOqq6tVVVUVeR0IBFRQUDCwxgFgCIR6uhTqPpnoNoCkNaCveCorK7Vhwwa9/fbbGjt27FlrS0pKJEkHDx6UJHk8HrW0tETVnHp9putWnE6nXC5X1AIAVnby+F904rMjiW4DSFoxBRRjjCorK7V+/Xpt3bpV48eP7/M9DQ0NkqS8vDxJktfr1d69e9Xa2hqp2bx5s1wul4qKimJpBwAsKxzq6fM5PE53jtwF/LsHnE5MX/FUVFRo7dq1evXVVzVixIjINSNut1vp6ek6dOiQ1q5dq2uvvVajRo3Snj17dM899+iaa65RcXGxJGnu3LkqKirSbbfdpscff1w+n08PPPCAKioq5HQ643+EAGBR9hSHUpzD+y4ELkAxnUFZtWqV/H6/Zs+erby8vMjyq1/9SpLkcDj01ltvae7cuZo0aZLuvfdelZeX67XXXovsIyUlRRs2bFBKSoq8Xq/+9m//VosXL46aNwUAkpkxRurHNCj21DSlOjL6LgQuQDGdQelr4qGCggLV1tb2uZ9x48bp9ddfj+WjASBpGBPu5y3GDqU4CSjA6fAsHgCIMxPqVW9XR591NnsKs8gCZ0BAAYA4M+H+BRRJstlsg9wNkJwIKAAQZ6HuLp08/kmi2wCSGgEFAOKsN9ipjpb/TnQbQFIjoABAAthTHRp58ZWJbgOwLAIKACSALSVV6dkXJboNwLIIKAAQR1/MgRLus85msystncd2AGdCQAGAOOvp6uy7yGZXagYBBTgTAgoAxJVRz4m2PqtsNptS0oYNfjtAkiKgAEA8GaOeTn+/SpkDBTgzAgoAxJExRsH2TxPdBpD0CCgAEEcmHNKnH73TZ13a8KzBbwZIYgQUAEiA0Zd9K9EtAJZGQAGABHBkjkx0C4ClEVAAIJ6M6VdZWkbW4PYBJDkCCgDEUc/J/t3B4+QMCnBWBBQAiKPujs/7VZfiyBjkToDkRkABgDjq7mxLdAvAeYGAAgBxdKwftxgD6FtqohsAACswxigUCp3zfro+P9pnTdhIvaFe2TWwmWRTU/mnG+c/fssBQFJ9fb28Xu857+d3Nbdo5Ij0s9b8vzf/oO/+yK1QuH93/HyZ2+3WsWPHmCYf5z0CCgDoizMovb29cdhP3zWtn3equ6env3ckR4lHj0AyIKAAQJyk2G2y2aSecJpau8fpRNglu0IakXpcOY4jkbpj/k5pAOEEuJAQUAAgTtzDh8koVR8EytQeGqke45RNRk77CX3mOKzJmTskSZ+2nSCfAH0goABAnIzOGqGdgRslp0f6nwtgjaSu8Aj9uesKpdh6NTGjXieCPQntE0gGBBQAiBNv6cMyDo9sp7k7xyhFh05+Q8NT2jh7AvQD86AAQJzYbLY+7q7hzhugvwgoAADAcggoADCE3t1/RP6OrkS3AVgeAQUA4uSbrjc1zN6u099DHNa4YfsU/PwDnexmLhOgLzEFlFWrVqm4uFgul0sul0ter1cbN26MbO/q6lJFRYVGjRqlzMxMlZeXq6WlJWofTU1NWrBggTIyMpSTk6P77ruPiYcAnBeOth5TQdf/1cn2IzKhk5IJy6awHLaTGuv8oyYPf1eBjk719oYT3SpgeTHdxTN27Fg99thjmjhxoowxeuGFF3TDDTdo9+7duuKKK3TPPffod7/7nV566SW53W5VVlbqxhtv1O9//3tJUigU0oIFC+TxePTuu++qublZixcvVlpamh599NFBOUAAGCqPrX1HOVnDNS7/fXkKvyu7c4xcw1N1aXaHsnKadTItQ20dXeqJwzN/gPOdzZiBTLb8v7Kzs/XEE0/opptu0pgxY7R27VrddNNNkqSPPvpIkydPVl1dnWbNmqWNGzfquuuu09GjR5WbmytJWr16tVasWKFjx47J4XD06zMDgYDcbrduv/32fr8HAM7m2LFjWr9+fdz3m5nu0ChXukZnZWi0K0MH/nxMR1oDA96fw+HQ7bffHr8GgSHU3d2tNWvWyO/3y+VynbV2wPOghEIhvfTSS+rs7JTX61V9fb16enpUWloaqZk0aZIKCwsjAaWurk5Tp06NhBNJKisr0/Lly7V//3594xvfOO1nBYNBBYPByOtA4Iv/uG+77TZlZmYO9BAAIOLDDz8clIDScbJbHSe79ecWf1z2l5aWpqVLl8ZlX8BQ6+jo0Jo1a/pVG3NA2bt3r7xer7q6upSZman169erqKhIDQ0NcjgcysrKiqrPzc2Vz+eTJPl8vqhwcmr7qW1nUlNTo5UrV35t/YwZM/pMYADQH8nydODU1FTNnDkzafoFvuzUCYb+iPkunssvv1wNDQ3auXOnli9friVLlujAgQOx7iYm1dXV8vv9keXIkSN9vwkAACStmM+gOBwOTZgwQZI0ffp07dq1S08//bRuvvlmdXd3q62tLeosSktLizwejyTJ4/Hovffei9rfqbt8TtWcjtPplNPpjLVVAACQpM55HpRwOKxgMKjp06crLS1NW7ZsiWxrbGxUU1OTvF6vJMnr9Wrv3r1qbW2N1GzevFkul0tFRUXn2goAADhPxHQGpbq6WvPnz1dhYaHa29u1du1abdu2TW+88YbcbreWLl2qqqoqZWdny+Vy6a677pLX69WsWbMkSXPnzlVRUZFuu+02Pf744/L5fHrggQdUUVHBGRIAABARU0BpbW3V4sWL1dzcLLfbreLiYr3xxhv6/ve/L0l68sknZbfbVV5ermAwqLKyMj333HOR96ekpGjDhg1avny5vF6vhg8friVLluiRRx6J71EBAICkds7zoCTCqXlQ+nMfNQD0x65du3TVVVcluo0+ud1uff7559zFg6QUy99vnsUDAAAsh4ACAAAsh4ACAAAsh4ACAAAsZ8DP4gGA88nIkSO1cOHCRLfRp4yMjES3AAwJAgoASJowYcKgPCwQwMDwFQ8AALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALAcAgoAALCcmALKqlWrVFxcLJfLJZfLJa/Xq40bN0a2z549WzabLWq58847o/bR1NSkBQsWKCMjQzk5ObrvvvvU29sbn6MBAADnhdRYiseOHavHHntMEydOlDFGL7zwgm644Qbt3r1bV1xxhSTpjjvu0COPPBJ5T0ZGRuTnUCikBQsWyOPx6N1331Vzc7MWL16stLQ0Pfroo3E6JAAAkOxsxhhzLjvIzs7WE088oaVLl2r27Nm68sor9dRTT522duPGjbruuut09OhR5ebmSpJWr16tFStW6NixY3I4HP36zEAgILfbLb/fL5fLdS7tAwCAIRLL3+8BX4MSCoW0bt06dXZ2yuv1Rta/+OKLGj16tKZMmaLq6mqdOHEisq2urk5Tp06NhBNJKisrUyAQ0P79+8/4WcFgUIFAIGoBAADnr5i+4pGkvXv3yuv1qqurS5mZmVq/fr2KiookST/4wQ80btw45efna8+ePVqxYoUaGxv18ssvS5J8Pl9UOJEUee3z+c74mTU1NVq5cmWsrQIAgCQVc0C5/PLL1dDQIL/fr9/85jdasmSJamtrVVRUpGXLlkXqpk6dqry8PM2ZM0eHDh3SpZdeOuAmq6urVVVVFXkdCARUUFAw4P0BAABri/krHofDoQkTJmj69OmqqanRtGnT9PTTT5+2tqSkRJJ08OBBSZLH41FLS0tUzanXHo/njJ/pdDojdw6dWgAAwPnrnOdBCYfDCgaDp93W0NAgScrLy5Mkeb1e7d27V62trZGazZs3y+VyRb4mAgAAiOkrnurqas2fP1+FhYVqb2/X2rVrtW3bNr3xxhs6dOiQ1q5dq2uvvVajRo3Snj17dM899+iaa65RcXGxJGnu3LkqKirSbbfdpscff1w+n08PPPCAKioq5HQ6B+UAAQBA8okpoLS2tmrx4sVqbm6W2+1WcXGx3njjDX3/+9/XkSNH9NZbb+mpp55SZ2enCgoKVF5ergceeCDy/pSUFG3YsEHLly+X1+vV8OHDtWTJkqh5UwAAAM55HpREYB4UAACSz5DMgwIAADBYCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByCCgAAMByUhPdwEAYYyRJgUAgwZ0AAID+OvV3+9Tf8bNJyoDS3t4uSSooKEhwJwAAIFbt7e1yu91nrbGZ/sQYiwmHw2psbFRRUZGOHDkil8uV6JaSViAQUEFBAeMYB4xl/DCW8cE4xg9jGR/GGLW3tys/P192+9mvMknKMyh2u10XXXSRJMnlcvHLEgeMY/wwlvHDWMYH4xg/jOW56+vMySlcJAsAACyHgAIAACwnaQOK0+nUQw89JKfTmehWkhrjGD+MZfwwlvHBOMYPYzn0kvIiWQAAcH5L2jMoAADg/EVAAQAAlkNAAQAAlkNAAQAAlpOUAeXZZ5/VxRdfrGHDhqmkpETvvfdeoluynO3bt+v6669Xfn6+bDabXnnllajtxhg9+OCDysvLU3p6ukpLS/Xxxx9H1Rw/flyLFi2Sy+VSVlaWli5dqo6OjiE8isSrqanRzJkzNWLECOXk5GjhwoVqbGyMqunq6lJFRYVGjRqlzMxMlZeXq6WlJaqmqalJCxYsUEZGhnJycnTfffept7d3KA8loVatWqXi4uLIJFder1cbN26MbGcMB+6xxx6TzWbT3XffHVnHePbPww8/LJvNFrVMmjQpsp1xTDCTZNatW2ccDof5j//4D7N//35zxx13mKysLNPS0pLo1izl9ddfN//wD/9gXn75ZSPJrF+/Pmr7Y489Ztxut3nllVfMH/7wB/PXf/3XZvz48ebkyZORmnnz5plp06aZHTt2mP/6r/8yEyZMMLfeeusQH0lilZWVmeeff97s27fPNDQ0mGuvvdYUFhaajo6OSM2dd95pCgoKzJYtW8z7779vZs2aZb71rW9Ftvf29popU6aY0tJSs3v3bvP666+b0aNHm+rq6kQcUkL89re/Nb/73e/MH//4R9PY2Gh++tOfmrS0NLNv3z5jDGM4UO+99565+OKLTXFxsfnRj34UWc949s9DDz1krrjiCtPc3BxZjh07FtnOOCZW0gWUq666ylRUVEReh0Ihk5+fb2pqahLYlbV9NaCEw2Hj8XjME088EVnX1tZmnE6n+eUvf2mMMebAgQNGktm1a1ekZuPGjcZms5m//OUvQ9a71bS2thpJpra21hjzxbilpaWZl156KVLz4YcfGkmmrq7OGPNFWLTb7cbn80VqVq1aZVwulwkGg0N7ABYycuRI82//9m+M4QC1t7ebiRMnms2bN5vvfve7kYDCePbfQw89ZKZNm3babYxj4iXVVzzd3d2qr69XaWlpZJ3dbldpaanq6uoS2FlyOXz4sHw+X9Q4ut1ulZSURMaxrq5OWVlZmjFjRqSmtLRUdrtdO3fuHPKercLv90uSsrOzJUn19fXq6emJGstJkyapsLAwaiynTp2q3NzcSE1ZWZkCgYD2798/hN1bQygU0rp169TZ2Smv18sYDlBFRYUWLFgQNW4Sv5Ox+vjjj5Wfn69LLrlEixYtUlNTkyTG0QqS6mGBn376qUKhUNQvgyTl5ubqo48+SlBXycfn80nSacfx1Dafz6ecnJyo7ampqcrOzo7UXGjC4bDuvvtuXX311ZoyZYqkL8bJ4XAoKysrqvarY3m6sT617UKxd+9eeb1edXV1KTMzU+vXr1dRUZEaGhoYwxitW7dOH3zwgXbt2vW1bfxO9l9JSYnWrFmjyy+/XM3NzVq5cqW+853vaN++fYyjBSRVQAESqaKiQvv27dM777yT6FaS0uWXX66Ghgb5/X795je/0ZIlS1RbW5votpLOkSNH9KMf/UibN2/WsGHDEt1OUps/f37k5+LiYpWUlGjcuHH69a9/rfT09AR2BinJ7uIZPXq0UlJSvnYVdUtLizweT4K6Sj6nxups4+jxeNTa2hq1vbe3V8ePH78gx7qyslIbNmzQ22+/rbFjx0bWezwedXd3q62tLar+q2N5urE+te1C4XA4NGHCBE2fPl01NTWaNm2ann76acYwRvX19WptbdU3v/lNpaamKjU1VbW1tXrmmWeUmpqq3NxcxnOAsrKydNlll+ngwYP8XlpAUgUUh8Oh6dOna8uWLZF14XBYW7ZskdfrTWBnyWX8+PHyeDxR4xgIBLRz587IOHq9XrW1tam+vj5Ss3XrVoXDYZWUlAx5z4lijFFlZaXWr1+vrVu3avz48VHbp0+frrS0tKixbGxsVFNTU9RY7t27Nyrwbd68WS6XS0VFRUNzIBYUDocVDAYZwxjNmTNHe/fuVUNDQ2SZMWOGFi1aFPmZ8RyYjo4OHTp0SHl5efxeWkGir9KN1bp164zT6TRr1qwxBw4cMMuWLTNZWVlRV1Hjiyv8d+/ebXbv3m0kmX/5l38xu3fvNn/+85+NMV/cZpyVlWVeffVVs2fPHnPDDTec9jbjb3zjG2bnzp3mnXfeMRMnTrzgbjNevny5cbvdZtu2bVG3Ip44cSJSc+edd5rCwkKzdetW8/777xuv12u8Xm9k+6lbEefOnWsaGhrMpk2bzJgxYy6oWxHvv/9+U1tbaw4fPmz27Nlj7r//fmOz2cybb75pjGEMz9WX7+IxhvHsr3vvvdds27bNHD582Pz+9783paWlZvTo0aa1tdUYwzgmWtIFFGOM+cUvfmEKCwuNw+EwV111ldmxY0eiW7Kct99+20j62rJkyRJjzBe3Gv/sZz8zubm5xul0mjlz5pjGxsaofXz22Wfm1ltvNZmZmcblcpkf/vCHpr29PQFHkzinG0NJ5vnnn4/UnDx50vz93/+9GTlypMnIyDB/8zd/Y5qbm6P286c//cnMnz/fpKenm9GjR5t7773X9PT0DPHRJM7f/d3fmXHjxhmHw2HGjBlj5syZEwknxjCG5+qrAYXx7J+bb77Z5OXlGYfDYS666CJz8803m4MHD0a2M46JZTPGmMScuwEAADi9pLoGBQAAXBgIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHIIKAAAwHL+P+4bb1723pEAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create CartPole environment\n",
    "env = gym.make('CartPole-v0', render_mode='rgb_array')\n",
    "state, _ = env.reset()\n",
    "\n",
    "# Run the environment for 100 steps\n",
    "for i in range(100):\n",
    "    # Display the current state of the environment\n",
    "    plt.imshow(env.render())\n",
    "    display.display(plt.gcf())\n",
    "    display.clear_output(wait=True)\n",
    "    \n",
    "    # Choose an action based on the learned Q-network\n",
    "    state = torch.from_numpy(state).float().unsqueeze(0)\n",
    "    with torch.no_grad():\n",
    "        q_values = agent.policy_net.net(state)\n",
    "    action = agent.greedy_policy(q_values).detach().numpy()\n",
    "    \n",
    "    # Take the chosen action and observe the next state, reward, and termination status\n",
    "    state, reward, terminated, truncated, _ = env.step(action)\n",
    "    \n",
    "    # If the episode is terminated or truncated, reset the environment\n",
    "    if terminated or truncated:\n",
    "        state, info = env.reset()\n",
    "\n",
    "# Close the environment after exploration\n",
    "env.close()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0870ba75",
   "metadata": {},
   "source": [
    "It properly choose the right or left action to maintain the pole position upright!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b299e721",
   "metadata": {},
   "source": [
    "### Ref\n",
    "- Mnih, Volodymyr, et al. \"Human-level control through deep reinforcement learning.\" nature 518.7540 (2015): 529-533.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84cb070c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7 (py37)",
   "language": "python",
   "name": "py37"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}