mario_ai_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a5411683",
   "metadata": {},
   "source": [
    "# RL for Super Mario Bros\n",
    "\n",
    "Refer to YouTube: Build an Mario AI Model with Python | Gaming Reinforcement Learning: https://youtu.be/2eeYqJ0uBKE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34f51310",
   "metadata": {},
   "source": [
    "## Setup Mario\n",
    "\n",
    "Setup a model to take random actions in each game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1f9be100-99fe-462a-a15a-6da79a4b1766",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting package metadata (current_repodata.json): done\n",
      "Solving environment: failed with initial frozen solve. Retrying with flexible solve.\n",
      "Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.\n",
      "Collecting package metadata (repodata.json): done\n",
      "Solving environment: done\n",
      "\n",
      "## Package Plan ##\n",
      "\n",
      "  environment location: /home/james/miniconda3/envs/cmpe260_dev\n",
      "\n",
      "  added / updated specs:\n",
      "    - pytorch\n",
      "    - pytorch-cuda=11.7\n",
      "    - torchaudio\n",
      "    - torchvision\n",
      "\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "  blas               pkgs/main/linux-64::blas-1.0-mkl \n",
      "  brotlipy           pkgs/main/linux-64::brotlipy-0.7.0-py310h7f8727e_1002 \n",
      "  cffi               pkgs/main/linux-64::cffi-1.15.1-py310h5eee18b_3 \n",
      "  charset-normalizer pkgs/main/noarch::charset-normalizer-2.0.4-pyhd3eb1b0_0 \n",
      "  cryptography       pkgs/main/linux-64::cryptography-38.0.4-py310h9ce1e76_0 \n",
      "  cuda               nvidia/linux-64::cuda-11.7.1-0 \n",
      "  cuda-cccl          nvidia/linux-64::cuda-cccl-11.7.91-0 \n",
      "  cuda-command-line~ nvidia/linux-64::cuda-command-line-tools-11.7.1-0 \n",
      "  cuda-compiler      nvidia/linux-64::cuda-compiler-11.7.1-0 \n",
      "  cuda-cudart        nvidia/linux-64::cuda-cudart-11.7.99-0 \n",
      "  cuda-cudart-dev    nvidia/linux-64::cuda-cudart-dev-11.7.99-0 \n",
      "  cuda-cuobjdump     nvidia/linux-64::cuda-cuobjdump-11.7.91-0 \n",
      "  cuda-cupti         nvidia/linux-64::cuda-cupti-11.7.101-0 \n",
      "  cuda-cuxxfilt      nvidia/linux-64::cuda-cuxxfilt-11.7.91-0 \n",
      "  cuda-demo-suite    nvidia/linux-64::cuda-demo-suite-12.0.76-0 \n",
      "  cuda-documentation nvidia/linux-64::cuda-documentation-12.0.76-0 \n",
      "  cuda-driver-dev    nvidia/linux-64::cuda-driver-dev-11.7.99-0 \n",
      "  cuda-gdb           nvidia/linux-64::cuda-gdb-12.0.90-0 \n",
      "  cuda-libraries     nvidia/linux-64::cuda-libraries-11.7.1-0 \n",
      "  cuda-libraries-dev nvidia/linux-64::cuda-libraries-dev-11.7.1-0 \n",
      "  cuda-memcheck      nvidia/linux-64::cuda-memcheck-11.8.86-0 \n",
      "  cuda-nsight        nvidia/linux-64::cuda-nsight-12.0.78-0 \n",
      "  cuda-nsight-compu~ nvidia/linux-64::cuda-nsight-compute-12.0.0-0 \n",
      "  cuda-nvcc          nvidia/linux-64::cuda-nvcc-11.7.99-0 \n",
      "  cuda-nvdisasm      nvidia/linux-64::cuda-nvdisasm-12.0.76-0 \n",
      "  cuda-nvml-dev      nvidia/linux-64::cuda-nvml-dev-11.7.91-0 \n",
      "  cuda-nvprof        nvidia/linux-64::cuda-nvprof-12.0.90-0 \n",
      "  cuda-nvprune       nvidia/linux-64::cuda-nvprune-11.7.91-0 \n",
      "  cuda-nvrtc         nvidia/linux-64::cuda-nvrtc-11.7.99-0 \n",
      "  cuda-nvrtc-dev     nvidia/linux-64::cuda-nvrtc-dev-11.7.99-0 \n",
      "  cuda-nvtx          nvidia/linux-64::cuda-nvtx-11.7.91-0 \n",
      "  cuda-nvvp          nvidia/linux-64::cuda-nvvp-12.0.90-0 \n",
      "  cuda-runtime       nvidia/linux-64::cuda-runtime-11.7.1-0 \n",
      "  cuda-sanitizer-api nvidia/linux-64::cuda-sanitizer-api-12.0.90-0 \n",
      "  cuda-toolkit       nvidia/linux-64::cuda-toolkit-11.7.1-0 \n",
      "  cuda-tools         nvidia/linux-64::cuda-tools-11.7.1-0 \n",
      "  cuda-visual-tools  nvidia/linux-64::cuda-visual-tools-11.7.1-0 \n",
      "  ffmpeg             pytorch/linux-64::ffmpeg-4.3-hf484d3e_0 \n",
      "  flit-core          pkgs/main/noarch::flit-core-3.6.0-pyhd3eb1b0_0 \n",
      "  freetype           pkgs/main/linux-64::freetype-2.12.1-h4a9f257_0 \n",
      "  gds-tools          nvidia/linux-64::gds-tools-1.5.0.59-0 \n",
      "  giflib             pkgs/main/linux-64::giflib-5.2.1-h7b6447c_0 \n",
      "  gmp                pkgs/main/linux-64::gmp-6.2.1-h295c915_3 \n",
      "  gnutls             pkgs/main/linux-64::gnutls-3.6.15-he1e5248_0 \n",
      "  idna               pkgs/main/linux-64::idna-3.4-py310h06a4308_0 \n",
      "  intel-openmp       pkgs/main/linux-64::intel-openmp-2021.4.0-h06a4308_3561 \n",
      "  jpeg               pkgs/main/linux-64::jpeg-9e-h7f8727e_0 \n",
      "  lame               pkgs/main/linux-64::lame-3.100-h7b6447c_0 \n",
      "  lcms2              pkgs/main/linux-64::lcms2-2.12-h3be6417_0 \n",
      "  lerc               pkgs/main/linux-64::lerc-3.0-h295c915_0 \n",
      "  libcublas          nvidia/linux-64::libcublas-11.10.3.66-0 \n",
      "  libcublas-dev      nvidia/linux-64::libcublas-dev-11.10.3.66-0 \n",
      "  libcufft           nvidia/linux-64::libcufft-10.7.2.124-h4fbf590_0 \n",
      "  libcufft-dev       nvidia/linux-64::libcufft-dev-10.7.2.124-h98a8f43_0 \n",
      "  libcufile          nvidia/linux-64::libcufile-1.5.0.59-0 \n",
      "  libcufile-dev      nvidia/linux-64::libcufile-dev-1.5.0.59-0 \n",
      "  libcurand          nvidia/linux-64::libcurand-10.3.1.50-0 \n",
      "  libcurand-dev      nvidia/linux-64::libcurand-dev-10.3.1.50-0 \n",
      "  libcusolver        nvidia/linux-64::libcusolver-11.4.0.1-0 \n",
      "  libcusolver-dev    nvidia/linux-64::libcusolver-dev-11.4.0.1-0 \n",
      "  libcusparse        nvidia/linux-64::libcusparse-11.7.4.91-0 \n",
      "  libcusparse-dev    nvidia/linux-64::libcusparse-dev-11.7.4.91-0 \n",
      "  libdeflate         pkgs/main/linux-64::libdeflate-1.8-h7f8727e_5 \n",
      "  libiconv           pkgs/main/linux-64::libiconv-1.16-h7f8727e_2 \n",
      "  libidn2            pkgs/main/linux-64::libidn2-2.3.2-h7f8727e_0 \n",
      "  libnpp             nvidia/linux-64::libnpp-11.7.4.75-0 \n",
      "  libnpp-dev         nvidia/linux-64::libnpp-dev-11.7.4.75-0 \n",
      "  libnvjpeg          nvidia/linux-64::libnvjpeg-11.8.0.2-0 \n",
      "  libnvjpeg-dev      nvidia/linux-64::libnvjpeg-dev-11.8.0.2-0 \n",
      "  libpng             pkgs/main/linux-64::libpng-1.6.37-hbc83047_0 \n",
      "  libtasn1           pkgs/main/linux-64::libtasn1-4.16.0-h27cfd23_0 \n",
      "  libtiff            pkgs/main/linux-64::libtiff-4.5.0-hecacb30_0 \n",
      "  libunistring       pkgs/main/linux-64::libunistring-0.9.10-h27cfd23_0 \n",
      "  libwebp            pkgs/main/linux-64::libwebp-1.2.4-h11a3e52_0 \n",
      "  libwebp-base       pkgs/main/linux-64::libwebp-base-1.2.4-h5eee18b_0 \n",
      "  lz4-c              pkgs/main/linux-64::lz4-c-1.9.4-h6a678d5_0 \n",
      "  mkl                pkgs/main/linux-64::mkl-2021.4.0-h06a4308_640 \n",
      "  mkl-service        pkgs/main/linux-64::mkl-service-2.4.0-py310h7f8727e_0 \n",
      "  mkl_fft            pkgs/main/linux-64::mkl_fft-1.3.1-py310hd6ae3a3_0 \n",
      "  mkl_random         pkgs/main/linux-64::mkl_random-1.2.2-py310h00e6091_0 \n",
      "  nettle             pkgs/main/linux-64::nettle-3.7.3-hbbd107a_1 \n",
      "  nsight-compute     nvidia/linux-64::nsight-compute-2022.4.0.15-0 \n",
      "  numpy              pkgs/main/linux-64::numpy-1.23.5-py310hd5efca6_0 \n",
      "  numpy-base         pkgs/main/linux-64::numpy-base-1.23.5-py310h8e6c178_0 \n",
      "  openh264           pkgs/main/linux-64::openh264-2.1.1-h4ff587b_0 \n",
      "  pillow             pkgs/main/linux-64::pillow-9.3.0-py310hace64e9_1 \n",
      "  pycparser          pkgs/main/noarch::pycparser-2.21-pyhd3eb1b0_0 \n",
      "  pyopenssl          pkgs/main/noarch::pyopenssl-22.0.0-pyhd3eb1b0_0 \n",
      "  pysocks            pkgs/main/linux-64::pysocks-1.7.1-py310h06a4308_0 \n",
      "  pytorch            pytorch/linux-64::pytorch-1.13.1-py3.10_cuda11.7_cudnn8.5.0_0 \n",
      "  pytorch-cuda       pytorch/noarch::pytorch-cuda-11.7-h67b0de4_1 \n",
      "  pytorch-mutex      pytorch/noarch::pytorch-mutex-1.0-cuda \n",
      "  requests           pkgs/main/linux-64::requests-2.28.1-py310h06a4308_0 \n",
      "  six                pkgs/main/noarch::six-1.16.0-pyhd3eb1b0_1 \n",
      "  torchaudio         pytorch/linux-64::torchaudio-0.13.1-py310_cu117 \n",
      "  torchvision        pytorch/linux-64::torchvision-0.14.1-py310_cu117 \n",
      "  typing_extensions  pkgs/main/linux-64::typing_extensions-4.4.0-py310h06a4308_0 \n",
      "  urllib3            pkgs/main/linux-64::urllib3-1.26.14-py310h06a4308_0 \n",
      "  zstd               pkgs/main/linux-64::zstd-1.5.2-ha4553b6_0 \n",
      "\n",
      "\n",
      "\n",
      "Downloading and Extracting Packages\n",
      "\n",
      "Preparing transaction: done\n",
      "Verifying transaction: done\n",
      "Executing transaction: done\n"
     ]
    }
   ],
   "source": [
    "# PyTorch\n",
    "!conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia -y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "6d6e3453-eec1-47e0-ade4-0524a431165e",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting package metadata (current_repodata.json): done\n",
      "Solving environment: done\n",
      "\n",
      "# All requested packages already installed.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# might need to install tensorboardx before pytorch.\n",
    "# !conda install -c conda-forge tensorboardx -y\n",
    "!pip install tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "7e5e42ce-e5ad-42d1-9a0e-7a71d8cf641b",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting package metadata (current_repodata.json): done\n",
      "Solving environment: done\n",
      "\n",
      "# All requested packages already installed.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# !conda install -c conda-forge protobuf -y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1ca8c254-aa45-4340-b2e7-7dc66065b4c8",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting package metadata (current_repodata.json): done\n",
      "Solving environment: done\n",
      "\n",
      "## Package Plan ##\n",
      "\n",
      "  environment location: /home/james/miniconda3/envs/cmpe260_dev\n",
      "\n",
      "  added / updated specs:\n",
      "    - stable-baselines3\n",
      "\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "  bottleneck         pkgs/main/linux-64::bottleneck-1.3.5-py310ha9d4c09_0 \n",
      "  cloudpickle        conda-forge/noarch::cloudpickle-2.2.1-pyhd8ed1ab_0 \n",
      "  cycler             conda-forge/noarch::cycler-0.11.0-pyhd8ed1ab_0 \n",
      "  gym                conda-forge/linux-64::gym-0.21.0-py310hfdc917e_2 \n",
      "  kiwisolver         pkgs/main/linux-64::kiwisolver-1.4.4-py310h6a678d5_0 \n",
      "  matplotlib-base    conda-forge/linux-64::matplotlib-base-3.4.3-py310h23f4a51_2 \n",
      "  numexpr            pkgs/main/linux-64::numexpr-2.8.4-py310h8879344_0 \n",
      "  packaging          conda-forge/noarch::packaging-23.0-pyhd8ed1ab_0 \n",
      "  pandas             pkgs/main/linux-64::pandas-1.5.2-py310h1128e8f_0 \n",
      "  pyparsing          conda-forge/noarch::pyparsing-3.0.9-pyhd8ed1ab_0 \n",
      "  python-dateutil    conda-forge/noarch::python-dateutil-2.8.2-pyhd8ed1ab_0 \n",
      "  python_abi         conda-forge/linux-64::python_abi-3.10-2_cp310 \n",
      "  pytz               conda-forge/noarch::pytz-2022.7.1-pyhd8ed1ab_0 \n",
      "  stable-baselines3  conda-forge/noarch::stable-baselines3-1.5.0-pyhd8ed1ab_0 \n",
      "  tornado            conda-forge/linux-64::tornado-6.1-py310h5764c6d_3 \n",
      "\n",
      "The following packages will be SUPERSEDED by a higher-priority channel:\n",
      "\n",
      "  ca-certificates    pkgs/main::ca-certificates-2023.01.10~ --> conda-forge::ca-certificates-2022.12.7-ha878542_0 \n",
      "  certifi            pkgs/main/linux-64::certifi-2022.12.7~ --> conda-forge/noarch::certifi-2022.12.7-pyhd8ed1ab_0 \n",
      "\n",
      "\n",
      "\n",
      "Downloading and Extracting Packages\n",
      "\n",
      "Preparing transaction: done\n",
      "Verifying transaction: done\n",
      "Executing transaction: done\n"
     ]
    }
   ],
   "source": [
    "# For RL Stuff\n",
    "# pip install breaks gym and super_mario_bros\n",
    "# !pip install stable-baselines3[extra]\n",
    "!conda install -c conda-forge stable-baselines3 -y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "87fdb845",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\n",
      "Collecting gym_super_mario_bros==7.3.0\n",
      "  Downloading gym_super_mario_bros-7.3.0-py2.py3-none-any.whl (198 kB)\n",
      "\u001b[2K     \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m198.6/198.6 kB\u001b[0m \u001b[31m2.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting nes_py\n",
      "  Downloading nes_py-8.2.1.tar.gz (77 kB)\n",
      "\u001b[2K     \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.7/77.7 kB\u001b[0m \u001b[31m7.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: gym>=0.17.2 in /home/james/miniconda3/envs/cmpe260_dev/lib/python3.10/site-packages (from nes_py) (0.21.0)\n",
      "Requirement already satisfied: numpy>=1.18.5 in /home/james/miniconda3/envs/cmpe260_dev/lib/python3.10/site-packages (from nes_py) (1.23.5)\n",
      "Collecting pyglet<=1.5.21,>=1.4.0\n",
      "  Downloading pyglet-1.5.21-py3-none-any.whl (1.1 MB)\n",
      "\u001b[2K     \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m3.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m[31m3.4 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0mm\n",
      "\u001b[?25hCollecting tqdm>=4.48.2\n",
      "  Downloading tqdm-4.64.1-py2.py3-none-any.whl (78 kB)\n",
      "\u001b[2K     \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.5/78.5 kB\u001b[0m \u001b[31m14.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: cloudpickle>=1.2.0 in /home/james/miniconda3/envs/cmpe260_dev/lib/python3.10/site-packages (from gym>=0.17.2->nes_py) (2.2.1)\n",
      "Building wheels for collected packages: nes_py\n",
      "  Building wheel for nes_py (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for nes_py: filename=nes_py-8.2.1-cp310-cp310-linux_x86_64.whl size=48185 sha256=fedff7642ce79719368f78334628a5f20e0ceb006a06687d347521ada178e61e\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-0awcmbcu/wheels/bc/1a/6e/27afc1393514b773012ff8d89f77ed4fa02055aa36f39d43df\n",
      "Successfully built nes_py\n",
      "Installing collected packages: pyglet, tqdm, nes_py, gym_super_mario_bros\n",
      "Successfully installed gym_super_mario_bros-7.3.0 nes_py-8.2.1 pyglet-1.5.21 tqdm-4.64.1\n"
     ]
    }
   ],
   "source": [
    "!pip install opencv-python gym_super_mario_bros==7.3.0 nes_py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ef6f82ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the game\n",
    "import gym\n",
    "import gym_super_mario_bros\n",
    "# Import the Joypad wrapper\n",
    "from nes_py.wrappers import JoypadSpace\n",
    "# Import the SIMPLIFIED controls\n",
    "from gym_super_mario_bros.actions import SIMPLE_MOVEMENT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "88907c44",
   "metadata": {},
   "outputs": [],
   "source": [
    "# setup game\n",
    "env = gym_super_mario_bros.make(\"SuperMarioBros-v0\")\n",
    "env = JoypadSpace(env, SIMPLE_MOVEMENT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f00e70e9",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/james/miniconda3/envs/cmpe260_dev/lib/python3.10/site-packages/gym_super_mario_bros/smb_env.py:148: RuntimeWarning: overflow encountered in ubyte_scalars\n",
      "  return (self.ram[0x86] - self.ram[0x071c]) % 256\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[3], line 10\u001b[0m\n\u001b[1;32m      8\u001b[0m     env\u001b[38;5;241m.\u001b[39mreset()\n\u001b[1;32m      9\u001b[0m \u001b[38;5;66;03m# Do random actions\u001b[39;00m\n\u001b[0;32m---> 10\u001b[0m state, reward, done, info \u001b[38;5;241m=\u001b[39m \u001b[43menv\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mstep\u001b[49m\u001b[43m(\u001b[49m\u001b[43menv\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43maction_space\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msample\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     11\u001b[0m \u001b[38;5;66;03m# done = terminated or truncated\u001b[39;00m\n\u001b[1;32m     12\u001b[0m \u001b[38;5;66;03m# Show the game on the screen\u001b[39;00m\n\u001b[1;32m     13\u001b[0m env\u001b[38;5;241m.\u001b[39mrender()\n",
      "File \u001b[0;32m~/miniconda3/envs/cmpe260_dev/lib/python3.10/site-packages/nes_py/wrappers/joypad_space.py:74\u001b[0m, in \u001b[0;36mJoypadSpace.step\u001b[0;34m(self, action)\u001b[0m\n\u001b[1;32m     59\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m     60\u001b[0m \u001b[38;5;124;03mTake a step using the given action.\u001b[39;00m\n\u001b[1;32m     61\u001b[0m \n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     71\u001b[0m \n\u001b[1;32m     72\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m     73\u001b[0m \u001b[38;5;66;03m# take the step and record the output\u001b[39;00m\n\u001b[0;32m---> 74\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43menv\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mstep\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_action_map\u001b[49m\u001b[43m[\u001b[49m\u001b[43maction\u001b[49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/envs/cmpe260_dev/lib/python3.10/site-packages/gym/wrappers/time_limit.py:18\u001b[0m, in \u001b[0;36mTimeLimit.step\u001b[0;34m(self, action)\u001b[0m\n\u001b[1;32m     14\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mstep\u001b[39m(\u001b[38;5;28mself\u001b[39m, action):\n\u001b[1;32m     15\u001b[0m     \u001b[38;5;28;01massert\u001b[39;00m (\n\u001b[1;32m     16\u001b[0m         \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_elapsed_steps \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m     17\u001b[0m     ), \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCannot call env.step() before calling reset()\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m---> 18\u001b[0m     observation, reward, done, info \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43menv\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mstep\u001b[49m\u001b[43m(\u001b[49m\u001b[43maction\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     19\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_elapsed_steps \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m \u001b[38;5;241m1\u001b[39m\n\u001b[1;32m     20\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_elapsed_steps \u001b[38;5;241m>\u001b[39m\u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_max_episode_steps:\n",
      "File \u001b[0;32m~/miniconda3/envs/cmpe260_dev/lib/python3.10/site-packages/nes_py/nes_env.py:300\u001b[0m, in \u001b[0;36mNESEnv.step\u001b[0;34m(self, action)\u001b[0m\n\u001b[1;32m    298\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcontrollers[\u001b[38;5;241m0\u001b[39m][:] \u001b[38;5;241m=\u001b[39m action\n\u001b[1;32m    299\u001b[0m \u001b[38;5;66;03m# pass the action to the emulator as an unsigned byte\u001b[39;00m\n\u001b[0;32m--> 300\u001b[0m \u001b[43m_LIB\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mStep\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_env\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    301\u001b[0m \u001b[38;5;66;03m# get the reward for this step\u001b[39;00m\n\u001b[1;32m    302\u001b[0m reward \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mfloat\u001b[39m(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_get_reward())\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "# Create a flag restart or not\n",
    "done = True\n",
    "# Loop through each frame in the game\n",
    "for step in range(100000):\n",
    "    # Start the game to begin with\n",
    "    if done:\n",
    "        # Start the game\n",
    "        env.reset()\n",
    "    # Do random actions\n",
    "    state, reward, done, info = env.step(env.action_space.sample())\n",
    "    # done = terminated or truncated\n",
    "    # Show the game on the screen\n",
    "    env.render()\n",
    "# Close the game\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "dd7d65e3-5853-4fda-8275-425457438b57",
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66fe565b-972e-4f87-a364-f821d3c2bb9c",
   "metadata": {},
   "source": [
    "## Preprocess Environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "7c679ce8-dbec-47bc-b251-c31bf147ebb2",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\n",
      "Collecting opencv-python\n",
      "  Downloading opencv_python-4.7.0.68-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.8 MB)\n",
      "\u001b[2K     \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.8/61.8 MB\u001b[0m \u001b[31m4.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m[36m0:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: numpy>=1.19.3 in /home/james/miniconda3/envs/cmpe260_dev/lib/python3.10/site-packages (from opencv-python) (1.23.5)\n",
      "Installing collected packages: opencv-python\n",
      "Successfully installed opencv-python-4.7.0.68\n"
     ]
    }
   ],
   "source": [
    "# !pip install opencv-python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "62598d55-f3a6-4971-9086-e777b952556e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Import FrameStack Wrapper and GrayScaling Wrapper\n",
    "from gym.wrappers import GrayScaleObservation\n",
    "# Import Vectorization Wrappers\n",
    "from stable_baselines3.common.vec_env import VecFrameStack, DummyVecEnv\n",
    "# Import Matplotlib to show the impact of frame stacking\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "d9241245-5330-4a22-901e-14531ef3b1d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. Create the base environment\n",
    "env = gym_super_mario_bros.make(\"SuperMarioBros-v0\")\n",
    "\n",
    "# 2. Simplify the controls\n",
    "env = JoypadSpace(env, SIMPLE_MOVEMENT)\n",
    "\n",
    "# 3. Grayscale\n",
    "env = GrayScaleObservation(env, keep_dim=True)\n",
    "\n",
    "# 4. Wrap inside the Dummy Environment (place our env in another array)\n",
    "env = DummyVecEnv([lambda: env])\n",
    "\n",
    "# 5. Pass env through frame stack, Stack the frames up to 4\n",
    "env = VecFrameStack(env, 4, channels_order='last')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "5b2d7cab-d5cf-4e96-9871-2c3b6a26a510",
   "metadata": {},
   "outputs": [],
   "source": [
    "state = env.reset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "ee518b70-32d0-43eb-bce7-3e9907ae9b2b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['NOOP'],\n",
       " ['right'],\n",
       " ['right', 'A'],\n",
       " ['right', 'B'],\n",
       " ['right', 'A', 'B'],\n",
       " ['A'],\n",
       " ['left']]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SIMPLE_MOVEMENT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "ca31f834-cdb8-4096-bbbe-dc2e55b78109",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 240, 256, 4)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# since env placed in array via DummyVecEnv, we have 1 in front\n",
    "# since we pass env in frame stack, last is 4\n",
    "state.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "6d345e13-335a-4a85-8dcd-0952eb52936e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# each step gets us another stacked frame, keep doing eventually get 4 stacked\n",
    "# note: 1st step is 1 frame\n",
    "# Instead of random action [env.action_space.sample()], jump [5] action\n",
    "state, reward, done, info = env.step([5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "e73184d3-7cc5-495c-9d45-0860ff4068e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f72c25e65c0>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAccAAAGiCAYAAACSxQ1bAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAA9hAAAPYQGoP6dpAACfd0lEQVR4nOzdZ3gc13nw/f9sL9hd9EUHSRAEe+8iqU6KqpbsSJasZiu2HNlKZNmPHjt2Ejnxa8XKYztxlCix41jdktVlVapSoij2TpEgAQJE732xfef9MOKCWwACIMB6/66Ll4Td2TlnzpR75swpiqqqKkIIIYSI0p3uDAghhBBnGgmOQgghRBwJjkIIIUQcCY5CCCFEHAmOQgghRBwJjkIIIUQcCY5CCCFEHAmOQgghRBwJjkIIIUQcCY5CCCFEnNMaHP/zP/+TiRMnYrFYWLBgAZ988snpzI4QQggBnMbg+Nxzz3Hffffx4x//mJ07d7Jy5UrWrl1LTU3N6cqSEEIIAYByugYeX7JkCfPnz+fRRx+NfjZt2jS+9KUv8dBDD52OLAkhhBAAGE5HooFAgO3bt/PDH/4w5vPVq1ezcePGhOX9fj9+vz/6dyQSoaOjg4yMDBRFGff8CiGEOPuoqkpvby95eXnodCOrKD0twbGtrY1wOIzb7Y753O1209TUlLD8Qw89xE9/+tNTlT0hhBDnkNraWgoKCkb0m9MSHI+Jf+pTVTXpk+CPfvQj7r///ujf3d3dFBUV8a1v/QmTyTbu+RRCCHH2CQT6+e1vb8ThcIz4t6clOGZmZqLX6xOeEltaWhKeJgHMZjNmsznhc5PJhtlsH7d8CiGEOPuN5vXbaWmtajKZWLBgAe+++27M5++++y7Lly8/HVkSQgghok5bter999/PbbfdxsKFC1m2bBm//e1vqamp4dvf/vbpypIQQggBnMbgeNNNN9He3s4//uM/0tjYyMyZM3nzzTcpLi4+XVkSQgghgNPcIOeee+7hnnvuOZ1ZEEIIIRLI2KpCCCFEHAmOQgghRBwJjkIIIUQcCY5CCCFEHAmOQgghRBwJjkIIIUQcCY5CCCFEHAmOQgghRBwJjkIIIUQcCY5CCCFEHAmOQgghRBwJjkIIIUQcCY5CCCFEHAmOQgghRBwJjkIIIUQcCY5CCCFEnNM62fG5JBJRqK42Ew7rUBSV4mIfRqMa/b6/X0d9vQUAszlMUZE/5vdtbQY6O03RvxVFZcIEHwbDwDo8Hh0NDZYh86HXR5gwwY9ON/C77m49LS3mIX9nMoUpLtbyFAopVFdbUFUl6bIOR5CcnOCQ6xPnppoaM36/HoD8fB82WyT6XTCocPSodtwcfxyqqnZuhELJ78UtljCFhdqxFwgoHD1qHTIPyc4vr1dHXd3g59fxurr0tLYOfj7Eb9epdPSomUBAP+QybrcfpzMMJF43krHZQuTnB6ivN9Hfr13yc3J8OBwD2xgOa+d8JKJErz39/Tqam4e+bhyTne3H5QoPa9mzhQTHMRIKwXvvzaC/34bBEOLWW7eQmjpwsLS1GXnzzfmoqkJ2dhs333wQRRk4uQ8cSGPHjrLo3wZDiNtu2xo9CQBaW0288cYCgJjfHs9m6+f223diPu6Yrq218v77c6N/Hwt6x68jI6OToqLPURQVn09h3brZBIPGpGlMmXKUtWtrh1Eq4lzz6acTaW7OQlFUrr9+a0wQ6u/X8fbbcwmFDNhs/dxxxw7MZu3c+OCDafT12ZOuMze3hcLCQ4B2A/j223MJh/WDHuM6XYRbb91MWtrAudHeboieX1lZ7dxyy4FBf19VlcL69bOBxPNIVRWuv35L9EbxVIpEFNavL6WzMzUmPxCbz8sv38X06X0AlJensm3b1CGXLy6uJz+/ii1bCjh6NB+AK67YwdSpnugygYDCu+/OxOezYDQGuf32LdTWWnn33XkJ60vm4ot3M3t272g2+4wlwXEchMN63nijjEmTmlm2rJ13382jsTE96ZOYx6Pjrbcm0dnpxGQKcNVVuzlyJJXdu0t5/fUplJQ0s2RJR8LvMjI6ueyyQzGf7diRQ2VlIS+9VMasWXXMnKmdQCUl/WRmbgagrs7Ohg2zAMjM7ODSSw8DYDCoKIrKtm2pHDqUSyhkYNasCqZPb4+uPxJReOutGdTUuHnuORuXX15JenpobApNnJMOHLCze3cB/f1WJk2qZdGihpjvP/hgMm1taTz3XBkXXFCF3T4Q8CwWH1ddtSem9uTw4VR27CjjzTfLmDixmeXL2zkZxcX1LF1aB0BTk5X162ezfn0pubkdXH55wwl+PbZ0OpUrrzxIKKRdJ957bwptbekArFy5l7y8foCYJ7Q5czooKdHO7UOHBm6wj98usznxKXjTphIqK7tYu7aWPXscHDiQRyAw+BNoXl4zK1dWD/r98Tfx5woJjuNAVRVaWzOwWALU1vbR2JhOR0dawnIdHQZaW000NmbjcPSRm9tObm4An6+XtrYmmpqysFr91NX14XbHVmMajaGEqk2bLUA4rKe5OYsJE1oBLTharRGsVu0E8XgG7ohNpsTq0e5uKy0tmQA4HP6Y7yMRBYMhRG9vCj6fhWDwyOgLSZwTAgGFlhbjoDdJfX0mmpqyAbBaAwnHm8kUJBDQzgGfryYmOOr1YdzuYEz1aXOzP3p+paX1AicXHI/Pk6JAYWEjLS3pNDRkUFvbTkZG8JRWsWZmDpSjyTRQVmlp/qSvMlJSIqSkaPlrbBw4ty2WoV99dHW5CIX01NY2U1+fSnNz1pD5MpvPv1cp0iBnHNXV5fDSS4uSBkaAbdvcvP32fMJhPbNnH+Xqq49iNKpMmdLPl75Uic3mpaYmj5deWkRX19DvIYQ4HTo7Dbz44mJqa4d+F342cLuD3HBDJW53O52dqbz00qJzYrsG09eXwiuvLKKiouh0Z+WMJE+O42iwBi0jWUZVlRPW9/f16fjggyLa2lJHkj0hRk1VFTZunBBt+LVt2wTMZq3mYjyEQgrvvZdPa6trXNafzHDO37PdcLexqSmDV19NXDYlxccllzSd8Bp1NpLgOA4URSU9vZNQyEBPj4O0tC4iER1dXYOf2H19ZlpbB3ZHOKwQDg/+YB8MGqLL9/QYOXo0D5vNS3Z2GwB2e2CMtkaI5I5VlwK0tmaM+fojER2trUaMxgjBoI6jR3NQFDV6jDud3jFP83zhdPZiNAbp6EjDbvdgsQTo6EglEkl+zQmH9Xg8iU/Rev3padV7KkhwHAd6fZhrrjlIc7OZt9+ez+rV5QQCOl5+edGgd2o7d05h167ElnODaW9P449/XBKz3IIFlcyZo7UYOxfv5MT5xeu18sILi6J/q6rC7NmHueiiFkCO8ZOxbNlh8vJ8PPnkYubPP8LUqb088cQCfL7k1ch5ea1ce211wufn8j6Q4DhOFAVyc/2sWbOT1NQwLS1Dv96dMuUoEyYMtEqNRBQ2bJiK15u8z5fL1cOSJZXs3l0QvYM/eDCXri4rK1a0YJA9K04Bl6uHpUsr2LOngMbG7BP/YATMZj8rVhygujoj+l6stjaL997Ts3x5E3b72F6Y29sNbN/upr3dFd2u3NxT36XjVFAUsNkiXH75brKyAignqF1VFPWcDoTJyCV0HDmdYZxOz4kXBNzuHqZNG1g2FFLYvDmEd5CaI6vVz7RpHpqbu/B4bPT12Wlqyqa728HMme04HGHMZu1gDgYVvF4tOB/rBAwQDuvo6dHeEel0KikpEazWAA5HH319dnw+Q/R7gEhEq+oyGoPY7f3o9efXySJi2e0eMjO7mDbNQ0tLJ3192nF4fI2H2RzC6eylr89OMBh7PAGEQnoMhhB2ez9GY2wVncEQoqzMQzCop6Uljb4+Ox0daXR3O5k6tR2dLhhthX087bjWJVzwFUUlJSX2mD0+Ty0tZg4cmIjN1h/drjOd36/g92vnts83cG4Hg/rodhkMatIWt0ajSlmZ1j3k2PVhMMn23fEslggm07l1PZDgeJZbtaqFmTPb+eMfFxMO6/H5LDz77CIWLSqP9o88dMjOhx9qfRuPv3A1N2fxxBNaP6qMjE6++tVyli7tYMaMLp56ahE7d05h9+7Ykyoc1lNaepQ1a+okOJ7HFEXl8ss/jw4CsHJlKzNndvDMM4tjGuXMmtXHlCl7ePrpORw+XERlZUHMesJhPTk5rXz5y4fQ66GrK/EiPXduD1On7uGpp+bi8diJRHS8+up8Jk+u5Yor6hKW7+xM5cknFyd8brH4uPXW3TGfHTlSQHV1HqCdG6qqcNlln5+WQQBGY8eOdLZvnwLEnttVVQUcPaptV1FRY9Iq0ZGor3fzxBODd/e46KK90X7V5woJjmNEr4elSw8RCml3rBZLbFBJSwuycuVeQBvOSVFUysrayMzUDqiCgv6E9S1eXEEgoEdRtP5MZnOQVav2AGC3a32OdDoVhyPMihX7UY+LVbm5A+tzu30sX/75kPm3WkNfVJ1o1S0XXPA5kUjyupa0NL8ExvPUvHm19Pc3ApCREYi2VtVqHgaOQ4MhgsGgBVGTCZYsOTzosGh2ezDa0d9qVaPrMBoj6PXaOsxmWLr0MMHgQPBMSxsIYKmpoej5NRi9XsVgUMnP74+eR8lkZgZihl88XebMqWPy5GYA0tOT9zEsLu7FbB763HY6tcZ5s2Y1Ulys9QvNzo4N/kajyrJl5YTDCjqdisWikpPjG7KcjpeT4xvWcmcTRVXV038UjFBPTw8ul4vvfvd1zObkQ1Kdbn6/EnMnp9OpQ1Y7hEJKdGSMY8zmwev5VVXB749d3mhUhwxaZ2KeToVjQV67yVAJBiEwho15w2GFYPDkyslgUGNGgjmXHX8cmkzqmAei48tXC6znR7mO5DgMBhXC4YFlT1ROkYhCIBC77vHYd2PN7/fwyCNX093djdPpHNFv5clxHEQiCi+9VEZ3tyP6WWFhM1ddVTPob7ZuTWf37knRvw2GEF/96u6EdyTHtLYaePnlOTHBbsWKA4NWbaiqwquvltLRMdCdJDe3jeuuqxo0Tzt3prJ9++To33p9mBtv3D3oAMPt7XpefHFuTJ6WLStnzpyeQdMYb8GgQl2dGVXV7r5/8pNZfPrpIT76qP/EPx6m/ftT2LhxYHxLRVH58pd3xYx2crzeXh3PPTcnpvpxwYIKFi3qHLM8nalUVeG11ybT3p6Koqhce+1ucnPHduSVjg49L744h0hEh8Ph4aabDpwXNx6ff57Cp5/GHoc33LCLrKzE4/CTT9wcOlQY/dtq9fHVr+4bNEDW1Zl4883ZMZ+tXr2XSZPOvSfGYyQ4jrGWFiM1NXaKi1uJRLT+WAcOFNHW5mL79lRKS3tjxiEMBBT27XOiKCozZ2qBqqkplcbGbHbvTic/v48JE2IPwEOHbHR3m5kxoxoAv9/I/v2TqKrKIBzWMXNmb8zTWlubgerqFAoK2snL095DlpcX0tHhZPv2VEpKemMGSQ8GtTyFw0o0Ty0tLurqcti7N428vL6Ek+LwYRtdXQN5CgQM7NtXQnV1BqoKs2b1nvInSI9Hh9erIxzWnlT6+/W8++4BLBYdq1ZZ2bTJe1JPkOGwwr59Dnw+Q7Sc2tsdVFfns39/Onl5fZSWxgbhqioLbW1Wpk+vQVW1Bin79k2ipiYDgyHCzJk9McOlnUuOHYddXQ58PguKog5adT9aFRVWGhoceL1WVFWJGYLtXHXsODx6ND2mK4aiqAndwfr7dRw44MRqDUaP2bq6DNrb09i1K42Cgj7y82NPis8/T6Gvzxhd3uMxc/DgRA4fzsDr7Wb6dM852ZJVho8bQ+GwQl2djU2bpjFjRhcrVrSxYkUbqam99Pam8Mkns2htNUWrMyIRhb4+PRs3TsdkCkeXLylpRaeLsG3bVMrLM6JVm6qqVXPu2ZPHkSPZ0eUXLGjBaAxSVVXA5s2l+P1K9KITDivU19v47LPpTJ3aHf1NWloPHo+NDRtm0tJijsmTx6Nj48Zp6PVqdPnS0hb0+jDbt0/lwIHshDzt25fD4cM5XHBBOytWtLFoUQtms5+jR/PYvHkKXq8y5hfCE+np0dPZaYpeIDweA08/rcfni7B69ZyYmUuO35ZkQqHYKulIRMHrVdi0aQqBgD5aTlOnNmMwhNi1awr79uXE/C4UUjhwIIv9+wtZulQrpyVLWjGb/TQ0ZLNx4zQ8Ht0pL6dTpaXFwubNU+nvt41bGuXlWezZUzJu6z/TRCIKPp92HFZX52M0BgcNVOGwQleXgQ0bZpKW5oses4WF7aiqwqZN0zlyxBVzbgeDCjt2FNPU5IouP3duCwZDiPLyCWzbNpFg8NwcTUiC4xgJBBT+9KcytmyZkvDdVVdVctlle1AUlfffn87rr08AYMOGLF56aVbCkFszZvRwyy1bsds9VFYW8OSTs+noMFBfb+SJJ+bQ1BTbaszpjHDrrduYOrUan8/CH/84h127nIRCCi+8MIVNmxLzdMUVVVxxxS4APvpoGq+9NhFVVfjsswxefHE2oVBspcLUqX187WtbcTp7qa7O48knZ9PWZqCxUctTfb07ZvmUFJVbbtnBjBlH8PvNPPfcHLZvTx1hqZ5a1dVmnnxydsxIRaANEP/kk7OprBy4K9++PZXnnpuD3x8bYUtKvNx221bS0zu/aOE3h8ZGI21t2jqqqvJjlrdYVG6+eRfz5h0iFDLw4ouz+eyzsR9t5kxQWurh1lu3kpbWNW5pXHJJHddcs+OcfJJJZscOF88+qx2H06cf4Wtf24rDkfzVynvv5fHGGzMTPl+4sJ2bbtqKyRRg376JPPPMDDweHRUVVp58cg6dnbEje2Vmhrj99i0UFzfQ0+PgqafmUF4+fjc8p4tUq46h3l47Pp8FgyG2jt9mi2CzaZ/199vwePpRVQWfz0hfX0rCerQX3RF0OpVAwEQoZCAchlBIR0+PI2F5rcWqiskUIhLR0dubgt9vQFWht9eG12tFr499T6jlKRzNU1+fVk3q8xnp7U3Mk9Go4nRG0OkiBINGenoMXwxxp9Dbm5Jw56goWp7M5uBxeTqzB0+32cIUFLRSX2+jv99PcbGfmhoznZ1mCgpasdsH9qvfr09aTgaDisMRQa+PEAoZ6O1NiZZTT48jaTmlpKiYzWFUVflixpPk82ie7QaOoRMHrsZGI52dQ0+063QGKCiIrQK0WiMxM3ucq0IhhYoKG/X1afT3WyktraGwsBuHQztHk+nvN+PxJDZgNJu1Ps7aXK4WIhEdkQgEAsmvN3q9dm4bjSHCYT09PQ4CgXPvOUuCoxBfcLuDXH55A3/60xQMhjBFRUfYtSsXj8fCV79aft48jZwJDhzIYM+e0iGXmTKlmoKC83PSba9X4f33ZxEMGrFYfFx4YT1Wq0o47r7g7OuLcOaQ4ChEnNWrK2lpMfPMM9OYP/8oubk+CYzijBUImHjxxWnodGq09uGYt9+eSmFhK5dc0nQac3h2kuA4RnQ6KC5uoqUlla4uF7W1VgIBX0Iz6pycFnJzO1EUlaysPgoLG6mvd9PRYaOmxkdBQWwHZIejj+zsDsxmFVUNM2lSLY2Nmfj9RqqqLGRnB7DbB6pRDIYQBQVNpKX50OmgqKiFlhZtxP26OiuhkI/s7NgWfG53Kzk5WjeCzEwPRUUN1NXl0NFh4+jRfgoLY/OUktJHdnYnZnMEvV5l4sQ6mpoyCASMVFWZyc4ORCdgBa0LSEFBE+npZ0ez7/5+PX6/Hoejn2BQh8ejj2nNC5Ce7qO4uJ66uhy6u21UV1soKPDHdBmw2frJyWnHag1jMGjl1NKSRiikp7raQlZWIKblsk4XoaCgiczMM3/YsvGWmelh0qShnwqzs3tPUW7OPAYDTJjQkDBzj6oqdHc7ou0YbDYfVqt2vufkdBMK6WlocNPaaiMlJUh+fmwDntTUbrKyujAYwOEIMnFiHfX12fT3m6mqspCbG4gZ4MRkCpCf34LTee61CpbgOEYMBpU1a+rZtauX9evTeP/9uZSVVScMb7V06dFo14x587opKenjiSey2L+/hMrKfO64YwcWy8DBWljYwurV9YA2Vuu111bz0kt6amtzefXVRVx11faY7gJms58rr6yOdu5fvbqevXt7+OCDND78cA4lJbVcffXRmDwtXnyUkhJtENc5c3q+yFMmBw9O5MiRfO68cxtW60Ce8vPbWLt24MJ17bXVvPqqQnV1Pq+9tog1a3bEjEtpMgVYu7Y6YdSgU+H45uzDffr75JOJGI0hbrihkldfnci+fRZuvvlgzO+nT+9j0qR+Hn88ncrKQqqr87j99i0xwc7t7ogZtuvaa6t5+22tld+f/7yQiy7azdy5A31ADYYQq1dXxdxYnK9mz+5l9uzzN/idiNUaSdpvOhxWePJJR3R6vAsvrI7eDC9b1k5LSw9//KObHTvKqKjo5fbb96A/rilAaWkDF1ygdUErKvJTWFjNM89YaW7O4rXXsvjyl7fEvOdNSfFwzTXVZ/xgAKMhI+SMsb4+HR0dRtatm04koiM9vRuA1FQP06a1k5kZjOloGw4rNDcb2bXLTWVlIW53W7Qxzty59aSlBUlLi31qaWsz0NBg5cMP55Ce3onVqg0FNXFiG3l5HnJyQjEXco9Hy9O7704jGDSQkdEFaPPhzZjRSkZGKCZwHcvT3r1ZlJdPICenNZqn2bMbSE8Pkp4e+0Tc3m6gsdHC++/PJS2tC5tNuwEoLm6noKAPtzt0yk+gSy6xY7EYeOQRP+GwDpcryH33pdPe3k9Dg5eDB0MEj7vhraszsXlzETNmNJKZ6SczM0R7u4H2dhN79+axcGFtzJibkYhCc7OBgwfT2bOnlJycVvT6MIoCs2Y1kJERICMjtpw6Ogy0tppYt24uDkcfKSnajU1+fifFxb243cHTPqLQWAuHFd55p4D+fq2BTXNzJsGg8Yvak/ZoX8Rly2oS+tgNV2urgfXrJwDaINktLZmoqoLBEMLtbvtiGLsQV1xx9JwbIPt4WnCcRVeXC0VRufnmzTE1RcGgQkuLkS1bCqivd0fLxmgMM3duA2lpoZgbPBjou/3ppzPJzOzAbNb20ZQpzbjdPtzuM/epUUbIOYOkpESw2QLk5bXR12eNTh5qNIaTnvh6vUpeXoDW1m76+rTpqSIRHYoSIScnkHQ0/czMEDqdl9zclujyAC6XP+loI3b7QJ56emzH5Sk0ZJ46Orrp7m49Lo0IOTn+pE82GRkh9HoveXnNqKoSTcPpTJ6nU8FqNZCaamXq1G4iEbDbIS0thdraHqqqQkkaL2gTTGdn+6PBX9surbN6fEtTnU4lNzdIT08vra0D+0KbkNefUBULkJ4ewmSKkJvb8kWrQK2cHA4/eXnn7gTVx29rVlZHwndw8o1Hjq1Hr49Ezw1tvcoXx6RyzjdQURTIzu7EZvN9EfRiz1WjUSU/P0BeXhfBoCFaNhAmLy+QdACK7Owg4bAn4XqTnu4/owPjyZInR3HOUhRwu+F731uJXq+nt7eX//qv7fT0gO+L159n39EvhBgueXIUIglVha4ueOqpz1AUCIVUurv5YkSP0507IcSZTIKjOKf5fLB3b/IBwIUQYjDn3rAGQgghxEmS4CiEEELEkeAohBBCxJHgKIQQQsSR4CiEEELEkeAohBBCxJHgKIQQQsSR4CiEEELEkeAohBBCxJHgKIQQQsSR4CiEEELEkeAohBBCxJHgKIQQQsSR4CiEEELEkeAohBBCxJHgKIQQQsSR4CiEEELEkeAohBBCxJHgKIQQQsSR4CiEEELEkeAohBBCxJHgKIQQQsQxnO4MCCHOLWYzGJJcWUIh8PtPfX6EGI0xD44PPvggP/3pT2M+c7vdNDU1AaCqKj/96U/57W9/S2dnJ0uWLOE//uM/mDFjxlhnRQhxGlx6qYtly6YmfL59+2FeeaXjNORIiJEblyfHGTNm8N5770X/1uv10f9/+OGH+dWvfsVjjz3GlClT+NnPfsbll19OeXk5DodjPLIjhDgFbDZYvNhGdXUfO3ZsTvg+NVXPRRfZ2Latn76+05BBIUZgXIKjwWAgJycn4XNVVfnXf/1XfvzjH3PDDTcA8Pjjj+N2u3nmmWe4++67xyM7QohxotMNVKE6nXDhhdP427/dwaOPrkxY9o47NvKb38ygomIbPp9KKHSKMyvECIxLcDx8+DB5eXmYzWaWLFnCz3/+cyZNmkRVVRVNTU2sXr06uqzZbObCCy9k48aNgwZHv9+P/7iXFT09PeORbSHECOXkKHz963NQFIWammYWL26luXlO0mVffHE2n3zSyjPPZBGJqPzpT62nOLdCDN+YB8clS5bwxBNPMGXKFJqbm/nZz37G8uXL2b9/f/S9o9vtjvmN2+3m6NGjg67zoYceSniPKYQ4vcrK9Dgcel57bR8ALS0hqquXEgwaky7f15eCz2fBaq0FQFFg6lTtlcvBg2FU9dTkW4jhGPPguHbt2uj/z5o1i2XLllFSUsLjjz/O0qVLAVAUJeY3qqomfHa8H/3oR9x///3Rv3t6eigsLBzjnAshhktR4KKLCunu7mf58lIikeH3CguHI+h0yhfrKEKn03HoUCWRCBIgxRlj3Lty2O12Zs2axeHDh/nSl74EQFNTE7m5udFlWlpaEp4mj2c2mzGbzeOdVSHEOAuH9dx8s54LL6zn7/5uGv/3/1YQicDDD0/l9dcPcehQ5HRnUQjgFAwC4Pf7OXDgALm5uUycOJGcnBzefffd6PeBQID169ezfPny8c6KEOI00Oki5OU1k57eiaoqlJdPYPfuNHbsqGHPnkx2785m585aUlL0lJToGKISSYhTZsyD4w9+8APWr19PVVUVmzdv5itf+Qo9PT3ccccdKIrCfffdx89//nNefvll9u3bx5133onNZuOWW24Z66wIIcbBYMFLUdSEfwB6fZgLLqhm6tT66OebN0/nuuvms2/fZA4cmMR1183HYtHz5S9PQyfjdokzwJhXq9bV1XHzzTfT1tZGVlYWS5cuZdOmTRQXFwPwwAMP4PV6ueeee6KDAKxbt076OApxligsVLj66on8+tdVbN6chapq0dJm6+eCCw6h06kEAjo++WQGwaCRcFjPRx9NJju7i9Wrd7Jx4xR6e1NO81YIMbQxD47PPvvskN8risKDDz7Igw8+ONZJCyHGkaJAbq5CWpqB5uZutm3LYMeOMgBcrh6cTg86nRr9d0wkoqO1NQOLJYBO13Waci/EyMjYqkKIYVEUuOmmKezf38CqVdNivpszp5rMTB+vvrpg0JardXU51NUlDg4ixJlIgqMQIqkZM/QsXDjQilxV4eGHD7F7tytalXrMgQP5mExBIhFdwncDvx+6pc2vfmXjvfcO8K1v5bJ1awvl5WEA5s41Mn16Oi+/3IzXe5IbJcQwSXAUQsTQ6SAjA1wuI0bjwCUiEonw4Yd5fP75pITftLZmnHS6mzbNoLv7CL/5zUSOHOlErw+TkQFOpwGTyUBWlkJnp0pv70knJcQJSbswIUQMiwXuuWcBNTV+5s3Li/5bsKCAAwcmnrJ8pKTAvfcuYdcuL5demspf/MVULr00/ZSlL85v8uQohIiaNcvAhAkpPPjgDrZsSRt0KLjx0tSUzr33bmTZMh0XXODiRz/awmefZREKGdDpQuh00glSnBoSHIUQUWlpRrKzHTz5ZAEdHWmnPP3OzlR+97sLmDjxE2bPtvHYY5Po60vB5RqYbEBRtKfKcBj6+095FsV5QqpVhRBRv/ylhWXLMunsTD2t+fjHf1zERRfl4PHYE77T6eAv/7KM664bfMhJIU6WPDkKcZ6bNctAXp4VgPp6D59+evo76Pt8lri/zfy//7eb0lIdl17q4Omny7HZFNascbB5cy9dXacnn+LcJU+OQpyndDqt8U1pqYslS0pZsqSUvLzg6c5WUn6/mV//eiU7d8KCBRP5wx+K2LDByOWXLyA1Vd5DirEnwVGI81R2NjzwwHyefbaD6dM9TJ/u4fe/n3u6szWkZ56Zy5w5fqqq8k93VsQ5TqpVhThP6XQKDoeD/n4dTU3Zpzs7w+Lx2JO+hxRirMmToxDnKVVVCYfD6HRgMgWis2ic6RRFxWgMotdr+ZcJksV4kOAoxHmqtRV++csN3HijnbfeKsdi8Z3uLA2LwRDilVf28c1v2vnlLz+hoUGioxh7EhyFOE+FQlqA7OkJoNMpXH/9bqZOrYp+73L1UFxcj9F4co10dLoIhYWNZGZ2nGyWmTy5hi9/eTsmk57e3gAtLRA8M9sQibOcBEchznMbN/r4+ONOfv/7xVx7bR06XQRFUSkoaGXlyhosFn9MlWv8ZMZDURQVvT7MkiU1TJnScNJ5Xb26hsceW8K2bd18/LGMACDGjwRHIQS9vfDooxtZvNjKm2/uxeXqoaoql3ffncySJYeZM+dwdNnMzA6uvHLHsJ4ES0pqueyyPXz2WTF790446Xy+9NIULr10PytXZrJ2reuk1yfEYKS1qhCCUAhqalTy8iJkZ6ewcuVhDhzIpqoqn0CgEZMpRG5uCwCZmT1kZvpxuztRVYX29rSY6aiczl7sdm1uKbe7h4yMACZTCJ0ukjRtkylARkYXXV0OvF7rkPlsasqmvT2NQ4e2kZpqpLRUR11dRKayEmNOnhyFEFGbN/t54YUGnn56FnffXUM4rOeTT2bR3p7C6tUVrF5dwfz5WpBcsKCFpUuPJKxj6tTa6LKTJ/eg16usWlXDjBk1SdNMTe1h9eoKsrOH904yGDRy111LeeaZMHffvQq3WwYBEGNPnhyFEFHJukWoqkJjYybvvx87Q8e8efU4nSEuvXRXzOepqYHo/1dXp1BdncGyZbXk53u47LKdCev3eo28//5E2ttTR5BPhaoqF//zP5/Q3CytVcXYk+AohIihqtDY2IrZrDBvXnnMd+XlhfT321AUlcLCDsCL1RomJSWEyaRVm/b36+noMAHQ0pJCfb2b9vYWXK4A+fmx9Z+9vQZ8PgP19TkxVbODycpqp6CgDYCMDC8HD4bHYIuFSCTBUQgRw+eD//7vKlautLF589To55FIhOXLD7FjRxmqqrBp0/Tod6tW7WXChD4A9u/PSpgU+b335lJaepRly2JbrG7cOIHm5qxhBUaAr3zlAP/2bysB2LPnME891TeqbRTiRCQ4CiFiqKr2r7y8H49nU8x3f/M3Knv2fMIvf7kyJqAdPJhLXZ32VNje7sLh6OOf/3kPH3xg4IUXlgDQ1JTBhg2xzRy6ux0x68nLa+af/ukITz9t5YMP5gJw/fVbuO467alUpzPwpz9peersDI3thgtxHAmOQoikmpqgqWng/aFOB3feWUBaWicvvVQHgN9vpLExm+bmLJqbteUyMjqYNKmJNWsm4fFUsn27tmxvr43KyqKEdCwWH7m5WlXphAntrF1bxuef76eqSvvd4sV+rrxyDgDbtx/hnXd6x22bhThGgqMQYlhUFZ54oo7SUj1792oTDe/YcZiLLoqtFv3hDz/n9ttn8cgje1i0yB5d9vHHt/Cd71yQsN65cyt5991SFEWhrc3Eb36zn2uuSeenP9V+99FHrfzLv+wGIJK8N4gQY06CoxBiWFRV6w/Z3Bzmvff2AODzhfnFLzbGLJeVZWD9+kP4fFBZ2U8koi2bkqLnX/5lQ8J6LRaF99/fC0B/f5hAAD7/vIuuLu131dVeGSJOnHISHIUQI9LRAR99pA3dlp+v8K1vLURRBp4c33xzNxs3aoOYHz2qcvSotuyCBSbuvntRwvpqapr4/e9rYrqRHDoU4dAhGR5OnD4SHIUQo9bUpPKLX2yN+Wywp7zduwMcOLAl4XOpKhVnIgmOQohRC4ehf5gPeKGQ9k+Is4EMHyeEEELEkeAohBBCxJHgKIQQQsSR4CiEEELEkeAohBBCxJHgKIQQQsSR4CiEEELEkeAohBBCxJHgKIQQQsSR4CiEEELEkeAohBBCxJHgKIQQQsSR4CiEEELEkeAohBBCxJHgKIQQQsSR4CiEEELEkeAohBBCxJHgKIQQQsSR4CiEEELEkeAohBBCxJHgKIQQQsSR4CiEEELEkeAohBBCxJHgKIQQQsSR4CiEEELEkeAohBBCxJHgKIQQQsSR4CiEEELEkeAohBBCxJHgKIQQQsSR4CiEEELEkeAohBBCxJHgKIQQQsSR4CiEEELEkeAohBBCxJHgKIQQQsSR4CiEEELEkeAohBBCxBlxcPz444+55ppryMvLQ1EUXnnllZjvVVXlwQcfJC8vD6vVykUXXcT+/ftjlvH7/dx7771kZmZit9u59tprqaurO6kNEUIIIcbKiIOjx+Nhzpw5PPLII0m/f/jhh/nVr37FI488wtatW8nJyeHyyy+nt7c3usx9993Hyy+/zLPPPsuGDRvo6+vj6quvJhwOj35LhBBCiDFiGOkP1q5dy9q1a5N+p6oq//qv/8qPf/xjbrjhBgAef/xx3G43zzzzDHfffTfd3d38/ve/58knn+Syyy4D4KmnnqKwsJD33nuPNWvWnMTmCCGEECdvTN85VlVV0dTUxOrVq6Ofmc1mLrzwQjZu3AjA9u3bCQaDMcvk5eUxc+bM6DLx/H4/PT09Mf+EEEKI8TKmwbGpqQkAt9sd87nb7Y5+19TUhMlkIi0tbdBl4j300EO4XK7ov8LCwrHMthBCCBFjXFqrKooS87eqqgmfxRtqmR/96Ed0d3dH/9XW1o5ZXoUQQoh4Yxocc3JyABKeAFtaWqJPkzk5OQQCATo7OwddJp7ZbMbpdMb8E0IIIcbLmAbHiRMnkpOTw7vvvhv9LBAIsH79epYvXw7AggULMBqNMcs0Njayb9++6DJCCCHE6TTi1qp9fX1UVFRE/66qqmLXrl2kp6dTVFTEfffdx89//nNKS0spLS3l5z//OTabjVtuuQUAl8vFXXfdxfe//30yMjJIT0/nBz/4AbNmzYq2XhVCCCFOpxEHx23btnHxxRdH/77//vsBuOOOO3jsscd44IEH8Hq93HPPPXR2drJkyRLWrVuHw+GI/ubXv/41BoOBG2+8Ea/Xy6WXXspjjz2GXq8fg00SQgghTo6iqqp6ujMxUj09PbhcLr773dcxm+2nOztCCCHOQH6/h0ceuZru7u4Rt1WRsVWFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4EhyFEEKIOBIchRBCiDgSHIUQQog4Iw6OH3/8Mddccw15eXkoisIrr7wS8/2dd96Joigx/5YuXRqzjN/v59577yUzMxO73c61115LXV3dSW2IEEIIMVZGHBw9Hg9z5szhkUceGXSZK664gsbGxui/N998M+b7++67j5dffplnn32WDRs20NfXx9VXX004HB75FgghhBBjzDDSH6xdu5a1a9cOuYzZbCYnJyfpd93d3fz+97/nySef5LLLLgPgqaeeorCwkPfee481a9aMNEtCCCHEmBqXd44fffQR2dnZTJkyhW9+85u0tLREv9u+fTvBYJDVq1dHP8vLy2PmzJls3Lgx6fr8fj89PT0x/4QQQojxMubBce3atTz99NN88MEH/PKXv2Tr1q1ccskl+P1+AJqamjCZTKSlpcX8zu1209TUlHSdDz30EC6XK/qvsLBwrLMthBBCRI24WvVEbrrppuj/z5w5k4ULF1JcXMwbb7zBDTfcMOjvVFVFUZSk3/3oRz/i/vvvj/7d09MjAVIIIcS4GfeuHLm5uRQXF3P48GEAcnJyCAQCdHZ2xizX0tKC2+1Oug6z2YzT6Yz5J4QQQoyXcQ+O7e3t1NbWkpubC8CCBQswGo28++670WUaGxvZt28fy5cvH+/sCCGEECc04mrVvr4+Kioqon9XVVWxa9cu0tPTSU9P58EHH+TLX/4yubm5VFdX87d/+7dkZmZy/fXXA+Byubjrrrv4/ve/T0ZGBunp6fzgBz9g1qxZ0darQgghxOk04uC4bds2Lr744ujfx94F3nHHHTz66KPs3buXJ554gq6uLnJzc7n44ot57rnncDgc0d/8+te/xmAwcOONN+L1ern00kt57LHH0Ov1Y7BJQgghxMlRVFVVT3cmRqqnpweXy8V3v/s6ZrP9dGdHCCHEGcjv9/DII1fT3d094rYqMraqEEIIEUeCoxBCCBFHgqMQQggRR4KjEEIIEUeCoxBCCBFHgqMQQggRR4KjEEIIEUeCoxBCCBFHgqMQQggRR4KjEEIIEUeCoxBCCBFHgqMQQggRR4KjEEIIEUeCoxBCCBFHgqMQQggRR4KjEEIIEUeCoxBCCBFHgqMQQggRR4KjEEIIEUeCoxBCCBFHgqMQQggRR4KjEEIIEUeCoxBCCBFHgqMQQggRR4KjEEIIEUeCoxBCCBFHgqMQQggRx3C6MyCS0+mgpESH0agMuVx9fZju7lOUKSGEOE9IcDxDKHEx0GCAG2+cTVpa6pC/e+qpT9m9O5jwuaqOYeaEEOI8I8HxDLFsmYW5c/Oif0ciKtu2VRIMRob83cyZmSxfbo3+3d/v549/rMfvH7esCiHEOU+C42mUlQV2u/LF/1tJSRkIcuFwhLY2P/394SHXkZfnjPmdwaCnqEhHMKiiqlBfrxIKjU/+hRDiXCXB8TS67DI3CxZMA2DLlv28/vrBEa9jx44Wduxoif5tter5+teXYjKZ8Pv9PPzwZ/JOUgghRkiC42mQkQFXXZVHX5+fDz7YCUBHh29U7wmP/01ZmYvcXBcbN35OJKKiKHDttblUVHTy2We+Mcq9EEKc+6QrxymWmgput56iIjf9/UGqqvqoquqju3v0dZ96vUJqqpHcXBcFBTn09PhpauqnpsaD251Gfn4K2dlaIx8hhBAnJsHxFLvhhgKuvXY6r7yyl5oaz5isMyVFz3XXLaSkpBiz2cw11yxm2rQMwmGVt94qJxyO8L3vXUBGxpgkJ4QQ5zx5ljhF3G5YsSKDri4vTU29RCLqmHa30OkG7nMURUFRFFRVq3Ztbu5n69aDrFiRQUODR6pYhRDiBOTJcZwpCqSkgNttYM6cErq7fVRV9Z3SfoidnUHKy7uYONHN5MlpuFxSxSqEEEOR4HgK3H77JC66aCIvvriTpqbT89SmqrBu3SHa2/t54IFl5OcPPfKOEEKcz+T5YRzl5sLs2Q7a2nrxeoMEAkN36B8ORYHSUicOhwWAI0c6iURO/BiqqhAMqnR2+tm//wizZrnIyfGyZYtfRtMRQog48uQ4DhQFzGYoKDBx4YWzaGzso6Kid8zWP3lyDnPnljF3bhlpaWYAgsEgkUgk+v/hcPJA3NUVZNeuNkpL3cyfn4fFAnr9mGVNCCHOCRIcx4FeD9/61mTmzs3hxRe30dERGNf0entDvPTSdioqjuLz+Xj11W3s398+5G8++ugIFRWtPPDAAkpLJToKIcTxpFp1jOXlKUyaZKGry0NPj++Ew7+NhUgE+vvD1NV14fMF8XjChEKD15WqKvh8EXp6gtTVtTBhgh2r1cuuXUGpYhVCCOTJccwoivbEOHmyhauvXsChQx0cOtRzSvNQVdXH1q3NQwbG4/X0hNi4sYHJk7O59NJSjEZtqiwhhDjfyaVwjJjNcM89kykqSuWVV7bi8Yz/E+NY2bSphp07j3LffTOYPdt4urMjhBCnnVSrjoHcXMjLM+H3B+nt9Z3UUHDD0djYgc+nvcf0eBLnchwpjyeMokBfnxe328KMGREOHgwTPnviuxBCjCkJjifh2ATFc+c6WblyJi+8sBWv9+S7awxFVWHnzjagbUzX29cX5sMPj7JiRT6LF0/m4Yd3EonIpMlCiPOTVKuehJQU+Pa3J+JwmHn77Z34/eMbGE+F3bub+eyzw3zzm5NZuNB0urMjhBCnhQTHUXK7YeJEAyaTEa83SGtrgMjZHxvp7Q3R0eHHYNCTlWWlpEQnQ80JIc47EhxH6cILs/jyl+fx4YdHqKwcuw7+ZwKvN8K6dRVkZNj4+tcXY7We7hwJIcSpJc8EI5SeDtdfX0hnZz8ff/w54fDYzq5xplBVOHy4g9ZWD3/xFwUcPNjGxo0ym4cQ4vwgT44jkJEBeXkGsrJS8XqDNDT4zomq1MF0d4doaOgnNTUFt9tGfr4iVaxCiPOCBMcRuOaafK69dgavv36Ao0fHZqLiM10opPL224fR63Xce+8FpKWd7hwJIcT4k+eAYcjOhksvddPd7aWpqWrMJyo+kx3bzvr6PgKB/Vx6aTa1tT18+qlUsQohzl3y5DgERQGnU5uouKyskN7eAEePes6bwHi8rq4gR470kp+fQXGxi4wMmTBZCHHukuB4ArfcMpHLL5/Miy/uorHRe7qzc1pFIirvvHOYnh4/P/jBBeTmyoTJQohzk9z7DyI3FxYuTP1iouKOc7ZV6kioKoTDKu3tPnbuPMT8+akUFHjZtMl33peNEOLcIsExjqKA1Qq5uUaWLp3GunW7aWnxn/R69XoFo1EhGFQJh8+MSKLTgcmkG3GeurqCdHd3sGbNZLKzvezZU4PfD6HxHVJWiHOKyQTGJOP8h8PgG4NX+ooCFkvymXbG6nzV6Ri0H7TXy1ndml+CYxydDu66azLBYJgXXthGIDA2ezc728yll87mww/3Ul9/ZlTPZmaaWb16DuvX76O2tn/Ev//ggyO43RZ++MMlPPnkVg4dOovPBCFOsRUr7Fx88cyEz/ftq+K551rGJI077phIfn52wucvvLCD3btPftKCjAz4zncWotfHTpju9wf4zW920nNqZ+0bUxIcj5Ofr1BWZqO9vZe+vgA+38lf7BUFJk92kpPjxGw2U1KSid3ezeHDPWNeFakNFD74SvV6JZqnkhIHbreWp0mTMrHZujl0qHvYeVJVCAQi9PQEqKioZfJkBy6Xl23bAlLFKsQQzGZYssRKf3+I55/fnuR7HRdfbGfHDg/d3aNLIycHpk2zs2VLHaFQbcL3DoeBZcv0bNniG/XsOzNnGkhPN/Lyy7sTrjuKojBvno2mJj/l5Wfn9D4SHNGChcEAEyaYufzyubz22na6uk7+ruqY6dMLyMzMAKC0dALp6R0cPrx/zNZ/TCSiDjrRsaIo6HTatqoqlJbm4HZnATB5cjGZmd0cPrxnxIGtuzvEli3NXHbZRKZM0bNnTznB4NldnSLEWFOUgSpUux0uumgaTz+9i//938yEZS++uIV//Mf51NRsob8/Qig0vNlx9HrtH0B+volLLpnJt761g4MH0xOW/bu/62XZsons3v05fj/DDpAGw0A17Zw5mdjtZu68s49gMPbJ0WYL8swz2VRXt1JV1UEwePbN8CPBEa3u/9vfnkJXVx8vvbSV/v6z605HVSEYPHGDoUBAW0BRYN26CnJz61izZuGY5OHTT4+SlmbiBz+Yy5//vI89e+QFpBDHZGTAN785G71eT1dXL9/85n5aWx1Jl920KY2bbtrDT36SyfLlep56qnFYaSxYYGb16ukAbNx4iJtu2ktbmzPpsv/+70aKi8v5+c9nsH59BZs3n7hdhaLAV7+ay4QJOQA88shONm5MIRSyJSzr8xm4++46Lrusn+9/fza/+90e2sZ2lr1xd94Hx/x8hcJCE729/XR2eunrO7sCI2jB0eeLjYz9/Sr9X7xGtFi06bU6O1VMJsjK0pGVZSYryz5mefB6I+j1QVpbO8nLs6EoXvbuDcoTpDjvTZ6sIyPDyMaNFSiKNvNNfX0Kfn/yy6/Xa8TnM2I2G0hJsQBQWqrDbNaxf38o4SbYYIBZs4wYDAobNpQDsHdvP42Nie8aj+nqspCSEiQ11YXFosdo1NbR2hqitjbxLjs1FUpLTRw92kVdnVbXe/iwgdbWxMAIEIkoNDfb8fk8pKWlotdr41KXlJg4eDBA71kwV8N5GxwVRfs3Y0YKF100ixdf3DYmT4zH1gunrhohElHxeGITa2jQUV+vnVjZ2X6Ki8McOmQiNTVEbq7CggXF0WrVsdLXF+aTT+pYtaqQ+fNdHDy4m0Dg7KtOEWIsXXBBDg6HjVtu6UyofhxKJKISiagoCqxYkUdaWgoHDx4kHI49p0wm+NKX5vLaa7t56KFjT4opJ1y/qkLki7tXiwWuu24umzcfor5ea3twfBp5eXpuumk5f/VXH7NtW2JV8FBphMNhFAUKCgz8xV8s5dFHP8HjUc/4G+fzdhAAmw3uuWcy6elW3nhjBz7f2DwxlpQ4uO66mVx33UwWLnSPyTqH0tsbobk5wuefm+joUAiH4eBBIy0tyScqbmszsX69jp/97CD//u8fj0ueduxoYPPmCu65p0wmTBZilP7xHzt5/PEqvve9abzySh2/+tVB7r23jBkzxuaZpqXFxte/vgOPJ8TXvjaRv/mbHRw50sd3v1uKI3mN74h98IGTb3xjC5ddlovbbeG22z5j/vx0/uIvsqMPEWeq8/LJMTcXcnKMqKqKxxOgo2PsGt+YzQbS07XRubu6tLqD5uZOVFUlKyuT1tY2Wlu7xiy9cFglGASvV08opG2H0agSCkUIBrV7H79fR1dXGLs9jNcL3d1GuruNKIqXbds+JzXVTjA4du8I+/rChMMqoVCYrCwrZWVhKivD0g9SiCEoCuTl9RII6GlttVFb68RsjlBZ2UxFhYGODhOVlS04HAZKSyNUVkZG/PRlNofIyemnvd1CX5+JiopUKitbSEtr5fBhJ5mZnZSVtTFxolbF2tAw8mqf1FQfTmeAhoYUuros9PaaqKzspLk5wOHDmVRWduJ2W5g2TU9tbfiMrWI9r54cj1V5LluWwQ03zGfDhhoqKsZ3z6gqbNrUyPbt1QDs3HmUzz5rHLeqRr0eSkpCZGQMRKKeHgMVFVZyc8Pk5QVQFBVFUamttfCzn/Xy3HNVbNzYMKbVHF5vhPfeqyIry87XvrYAs5kz/k5RiLF0/CuWZJ8f/w9Ap4uwcqWXefP6op9XVKTyve8Z2LcvnYaGFL7/fSNGo8JNN83BYBj8nEq2fgCXy8/atUHy8jzRz195JZuf/MRGf7+R99/P5sc/htWrp7ByZVZ0XSNJY/LkPi6/PIDZHEJRIBzW8c//7OQPf8hEVeHf/i2NF1/08I1vrKSoaPjVzKfaiILjQw89xKJFi3A4HGRnZ/OlL32J8vLymGVUVeXBBx8kLy8Pq9XKRRddxP79sd0W/H4/9957L5mZmdjtdq699lrq6upOfmtOwOWCb31rAooC77+/l1Do1FV6t7X5efPNrbS2jv1sFiYTlJV5CQQUysuNlJcbaW0dGHojNTXI1KleTCatYU5ZmZeyMi8zZ3pZvtxLevr43SMdPNjOJ598zq23TmDZMsu4pSPEmcbthm9/eyLvvtvIgw9WEwppgcBsDnHllS1ce20LV17Zgtms3chGIjo++MBOQ4OBa65pIS3txIOFLFli5qtfLeAnP9nG888PnMf5+b1ce62WxtKlrdHg1dVl4bXXzGRnh7j00hb0+qGvgTod3HhjNoWFNr797Y8pLx+ob124sC2axoQJAx0yy8sdrFtnYtWqLmbPbh92eZ1pRnRVXL9+Pd/5znfYtGkT7777LqFQiNWrV+PxDMxt+PDDD/OrX/2KRx55hK1bt5KTk8Pll19O73HPzvfddx8vv/wyzz77LBs2bKCvr4+rr76a8Gh7ow5DdjYUFhpwOGx4vSGam/2n9IWw3x+hsdE3JgMLHM9gUDCbtZZgigI9PUZ6eoz4/Yl3ZP392rBUiqLdKGRkKGRk6DGZxu+RrqcnREuLH6vVTFaWleJimTBZnNsUReuEn51twOPxc/iwwv796agqOJ1+srP70eu1wHP80G6qCi0tdjo7jej1oCiDVy8ZDHqKi3VYrQZ6enzs3u2kttaJokBWVj+pqcHo+o9PIxDQ09CQgs+nJB1W7ngpKWaKixVCoTDNzT62b8+kt9eMyRQmJ6cPq1WNrv/4J8feXjNNTfZoWZytRnSZevvtt2P+/sMf/kB2djbbt29n1apVqKrKv/7rv/LjH/+YG264AYDHH38ct9vNM888w9133013dze///3vefLJJ7nssssAeOqppygsLOS9995jzZo1Y7Rpsdas0frnvPzyXiKRc6f5ZEqKjpQvGqY1NiYPvF1dRrq6YgdxvOACL07nqalVDwYjrFtXwaxZGXz728v5l3/5lI6OU5K0EKfF9ddPxOPx881vBlHVgdadc+b0MHEiPPtsxqAtVxsaHLzyytAtYlJSUrj77lX8x3+s56c/tQNmAPT6CJdf7qGlReWVVwa6csS/xtm+/cQtTqdNK2HKlAncdttmKioGZjlPT/fypS/5efddI598kpr0t6GQnrffHrwrydngpO7hu78Y2yg9XRuBoaqqiqamJlavXh1dxmw2c+GFF7Jx40buvvtutm/fTjAYjFkmLy+PmTNnsnHjxqTB0e/34/cPdFLtGcGAfVlZsHZtHr29PjZtOjzuExXX1vbS378TgL6+wPgldJxIBHp6wjgcUFraj80GwSAEAgO3bYqi4nAMjKDhcJy6183Hyvvo0W76+vZx5ZW5VFd3sWHDmTHGrBAno7RUx/LlOTGfvfpqNYcPK6hqZsz15sABOzU1IUIh3aDXoRNdn154IcTGjR8BUFlpjlk+HNaxYYMZn08/5HqG+i4Q0PPQQ59jsWjXysbG1Jjlu7osrFsXpqXFOuptOHLEzg9/+BEXXmhn8mQTf/5zJ5GINnrQl76Uw549bezde3pb8I06OKqqyv3338+KFSuYOVMbPLepqQkAtzu2C4Pb7ebo0aPRZUwmE2lpaQnLHPt9vIceeoif/vSnI8qfomgdV3NyDBQVudmypYK6uvG/GPf0hOjp6Rv3dI6JRLTRcXw+7d2jxQJOp4Lfr+L1akfosaoPl0vH8eMDh8NqdLzVU6G7O0Rfn4eyslyCwTBut5eODi2QC3G2URRt5JuMDCMOR2xn+D179OzfnzhsW1ub7aRHijlwII0DB5J/p6pQU5N8VJzhCod1fPrp4H2gfT4DR464TiqNzk4r779v5aqrgpSW5qIonTid2qseh8NKerqJrKwQ7e2nbyjKUQfH7373u+zZs4cNGzYkfKfEVTSrqprwWbyhlvnRj37E/fffH/27p6eHwsLCE+bxxhsn4HTaeeWVc6sq9Xg9PeETTm9jNILNptDdHYkZQ9FkgvT0U9ta7NiEyZMnO/ne91bwm998Oqrm4kKcbkYjfOtbc9iw4RC33NIZ810olBgYxdBWr86kuDiLr32tgdtvD/Dd7y7iX/5lK32n7lkjxqiC47333strr73Gxx9/TEFBQfTznBytaqGpqYnc3Nzo5y0tLdGnyZycHAKBAJ2dnTFPjy0tLSxfvjxpemazGbPZPOz85ebC8uUZtLb2UlPTMe5VqadDOKyNijPYgL5G48AL92P/jR/1IhSCnp4INpuCwXBqniCP5aGlxcuWLQdYtiydpiYPn3469q14hRgvU6fqmTbNxRNP7KG8XCEYHKNe8+eZF17oYvLkLq67LoPy8i7eeqsNr1d7H6o7UYuhcTai1FVV5bvf/S4vvfQSH3zwARMnToz5fuLEieTk5PDuu+9GPwsEAqxfvz4a+BYsWIDRaIxZprGxkX379g0aHIdLUbSuCjk5RubMKaGjw0t1teecC4ygjaLf1aXi9Q5MWnqsv1Hgi1edJpP2T6fTJjdVlIFAGQppLVe7ulSCwVNfQF1dQQ4e7KK4OIspUzJxuZBWrOKskZZmpLg4i3fftY1oODURa+PGLD7+2ERJSQ4VFSHefDObUCg2LJnN4HQmn7R5PI3ocvSd73yHZ555hldffRWHwxF9R+hyubBarSiKwn333cfPf/5zSktLKS0t5ec//zk2m41bbrkluuxdd93F97//fTIyMkhPT+cHP/gBs2bNirZeHS1Fga9/fTKqCi++uPOU9mM81fr7I+zbZ0FVwWKJMG1aAJtNQVFg0yYTBQUBJk3Slm1qUikvt7B4sQ+zWcHjUTlyxEhfn1alunChb9DZvMfbe+9VkJdn44EHlvGHP2ymouLc3Wfi3PHSS2F+85sWfL7kA2+L4aupcXLTTfUEAhlJv1+40MKaNbP5t3/bQvsp7DY5ouD46KOPAnDRRRfFfP6HP/yBO++8E4AHHngAr9fLPffcQ2dnJ0uWLGHdunU4jhus79e//jUGg4Ebb7wRr9fLpZdeymOPPZYwm/RI5OcrzJrloKWlB48nQCBw7l9kw2EFVVXw+7WBxvPyIrhcCkVFflwuAK2q1OGACRN8mEwKXq9KQ4MOn09HOKzdip2uJ+tjU211dfnZv/8IU6c6ycz0snmz/5x82hdnt6lT9RQXa8Fw/fpeDh0ynuAXYjgiEQWPJ3YM5i1bIoRC21i2LIW+vhBPP72duXPttLb6T9l0eIo61NTxZ6ienh5cLhff/e7rWCx2TCaYN8/Mtdcu4I03dozpWKlnqq6uMJ98YkVVFRRFRa9XmTLFR0HB0O8OW1tV9u61RAMrwOLF/eTknN5hnBQFLrtsEpGIyu9/X0EgMPwJWIUYTzqd9nrisstSWbSoFICnn97O734n1anjyeHw8+STObz3XiW//a2LP/0pk5aWbp5+OnmvhmT8fg+PPHI13d3dOJ0ja8V71r/lMRjgr/6qjJ4eDy++uG3MZtc4m1itYRYuDAxrpJuMDIUVK/zs3Gmgu/vMufNVVfj442oyMkw88MACXnxxN/v2yUjl4vRLTYV77pnHU0/t4he/0PpQeL0n111CnJjHY+Ib32jA7z/x9Fvj4awOjvn5CtOmWeno6KWzs39M5mM8W5jNChMn+lBVMJlULJbYwKjTKTGtVMPhgT6PJhPk5wdJT9eCj812Zozx5PdH6O0NUVfXQmGhDbPZx86dgTN+3jdxbtPpICXFTjCo9VMUp0YkotDRoTWGMJlO/bX9rA6OkydbufLK+bz00lb6+s6fwAhgter4YuwFVFULLMcoihYYj3XPCIdVIhGF42vQJ03Sn/LWX8PR2xti48YGLrqoiNmz7ezbt49g8PR1BBbi+Al7jcYwodDQo8+IsaXXRzAaw4TDkVPaX/2sfue4fv0WWlpC9PaGztuLZ09PBL9/oB+nTqd17I8PfKoKnZ3hmG4fRiOkpp6ZU8bYbHpcLiMLF07g/fcPs337qRmKT4h4er02sP/8+Q5ychz89V/309cnk3ifKldf3cLtt5fw1ltVtLVFGMHooefvO8cjR7o5cGD4m2C3wwUXOGlp8dHaGqShIfngALm5CsYvXsc1N6scN6zrCRUX65g61UFlZR+dneGkTY9NJsjJ0Z7qQiFobBzZIAWLFpmx2QxUVfUTDKoxkwiHQlBZGcbpBJdLS6OvT6WjQ9v++KDp80VobdX6Sx5jscDKlU7a2320tGjllOzmIydHwfTFNaKlRT3hSD3Hy89XmDnTSVWVVk6trfFLhHE6AxQVdREIRNDroaHBOKJymjcvSGqqniNHwrS366JdV46XkhImPV3bOI9Hob19+MeT0aiycqVKb2+EpiaVhgYD4XBiFXV2dgiLRct4W5ue/v7hP7JnZ4eYP99AdXWQzk6F5ubE/On1Krm5IXQ67Qm7ocE4opvFmTODuN16KirCdHbq6OlJLCebLUJmplY74/UqtLYOv5z0eq2cvN4IDQ0qTU0GgsHEcsrMDGGzaeXU0ZF8fw0mPT3E4sUGamqCdHRAU1Pi+3SdDnJzg+j1fDFmqIFQaHivFOrrYc4cleLiHPLz99LQoNLVldj/yWoNkpXVD4Dfr6e5efjvyxRFZcWKFsJhhbo6M83Ndvz+xHJOT+8nJUVrdNjVZaanZ/hTwTmdPpYt66KhwUB7u5nGxpRow7zj85GX14ter+2LxsaUQQdJT2by5E4mTfJTUWGhq8tER0diVbTZHMLt1mZzCgZ1NDYOPohCW5vCgQNN7NkDfX3afl22LIRer1BTE6GlRY/Pl3hOpaaGGcHYMQnO6uD45psR9u8/8TByx0yY0MHf/u1s3ntvF/v3B3j77VyCwdgi0OkirF1bj8t1rJFIKq2twx/94tZba7nwwtkcPPgphw/r2bgxL2GZtDQPV1yhTUvh8cCbb+ZHu1UMxw03eCkoyGDnzvKEYOH3w9tv51JW1sHs2VqXiIoKA9u3u1m7tp7jb55CIejsVPnsMwd1danRz93uHn74w6ls3lzO5593sW6dG58v8U55zZo6MjK0wP7pp04aG4c/3uJ119Vy771zqazcwJEjOtavz09YxuHwEonUo9NpT7l792Ym7K+hfP3r3cyZU8xvf7uXffssVFQk9qOaMqWNwkIvqgqtrTp27sxJsqbkUlP7+fWvizh4sJZ33mnjwIF0PJ7Es/GSS+rIyNCOp7o6K0ePpiVZW3KXXFLHrbdewBNPbKC1NZw0f1ZrgEmTmjEYtAEg9u9PT7q/BvOVr7Rx8cVT+a//2kl5uYkDBxLH1ZwwoYPiYm1Aje5uZUTlZLUG+MUvMmhq6uTPf26kvNxFd3fiBXPlynqysrTjaf/+5PtrMIsXN3LrrRfw3HMb6egIsGuXO+GibzKFmDChEbNZO/YPHEijr2/4gWXNmiYMBgNz5waJRLw0Nyc+iWRmeli0SDue2toUamqGv6+NxhD33TcJny/Aiy/W8u67Fnp6EgNwUVE/JSUhVBV2746M6PqUnd3Jz39+Ia+9toXPPvNQU5OacO0xGMLMmtWP3a7dbL39tm1E5XTRRUH+8i+X8+ijn7FzZyjpdcHl8kXLqbsbjh5NS9hfx3zwQQEffDDwt6Ko/PCHKna7haefrqS62p40jfnzm5gwYdjZTnBWB8dgUMfq1cOfJDk7Wxu/9eBBD5995mLVqibiu1bqdHDrrXkcOdLJr35loaysh3nzupOvMImpUy1EIhE2bDDQ369Pmr/sbJXbbivj0UfL2bbNycUXN4zo/V96ejHd3R7WrUtj9uwu3O6BCGk2wyWXNLF0qYmFCydx//2N6PUql13WgN2efH1z5/YyffrAAIapqSp6vZ6Kil4+/jiVZctaMRpjo7CiwNe+lk1bm4f/7/8zMnlyL7Nm9cavelAzZxpRVZVNm6ClxZK0nNLSVO68cwpPPnlo0P01lOzsXPr6vLz3nou8vP6kacyfb2DVqsk88EAtfv/Ijie7XcVonMTRoz18+GEqCxe2YTYnPtreeGM6wWCEv/s7mDChl7Ky4acxc6Z2YGzdGqay0p40fykpKnfcMZlXX63klVdcSffXUPLyMvH5Arz3npPMTF/SNKZP17F2bRk//nEl7e3mEZWT1mAsj/r6Ht57L43ZszuxWhPnLLvuOhd2u4kHHgiQn+8ZURqTJ2v/3bkzwN69Di67rD5hLkGLReX22yfy4Yc1PP20gwUL2pPur8EUF6cSCAT58EM7dnsoaf5KSuD668v4p386TE2NbUTboNer2O0zOHq0nXXr0igr62H+/K6E5a64wk5urpPvf79v0P01mGNDUu/d62X79uTXHqNR5fbbi9i+vZHf/jaFWbOS76/BlJTYCYfDrF9vRlFImr/CQrjppjL+3/87xOefpyTdX4PR5qMtpbm5i3Xr0igt7Ul67bn4YgtlZVP42c+GnfUYZ3VwjEQUMjKGf3CnpupQFG2EmK4uC2lp3dHq02P0esjLy6ClpY/29hTs9q4RpWG3ayvs7DR9MWp/4m+zsvTk5+cQiZTT3W0hPb1n2Bd97V2hAa83QFtbCn5/DxCO+T49XcXttpKTk0l3dwdOZ2DIbdDGZxj43ulUUBSF/v4InZ1W0tK6EqonFAVyc9MIBsO0t1uZNat7ROXkcGjl1NVlxOczJP1tRoaOvDw3cGjQ/TUUs9lIKBSmrS2FoiJP0jSys83k5bnp7W1CVZPvr8HYbAo6nQ6vN0JHhx2XqyvhBuRYOfX29tPermPKlJ4RpeF0aqdod7cBj8eY9LdOp0J+vhu9vpLOTiupqV1Yhn+jj8ViJBKJ0NFhJTvbO0g5mcjPz6G/vxqfTz+ibdCGMFTw+cK0t9txODq/GKQiVk6OE5PJQEeHj+LivhGl4XJpB0ZPj46+PhMZGWrCxdZqhby8LEymWtrb7TidXdG5UIfDYjEQiURob7dis/UOejzl5+fi81UOur8Go9eDXq8nEAjT3p466LUnJ8dBVpaLjo4wWVm+EV4D9SiKQl8f9PSYycjoSQiOJpN2w/T55y20t9tJSekkNXXYSWC1aje+HR0WXK7k156sLCP5+bkEAofp7U2+vwajKNpkz9q1J2XQa4/bbSc3151kDcNzBrZXFEIIIU4vCY5CCCFEnPM2OOp0Km63Dodj/DrAWywhcnL0I6oKHKnU1DBZWbphV0mMlE6nkpWl4HSOXzmZzWFycvQn1bLsRFyuMNnZunHt25mVpZCaOn7lZDJp5TSSKtORcjjCuN3j2wc2MxPS0savnAyGCDk5+nEd3CIlRSunkxgO+oTS01XS08fv3NbrI7jdOuz28Ssnm027Bo7njDtpaREyM8e+nM7b4GizBbnttnnMn586bmmUlPi4664VZGWN3xl0ySUpfOUrM8ftYmYyhfja12axbNngM4OfrMJCL3fdtYK8vPG7i7jgAhNf/erccTtJDYYwN944hVWrck+88Ci53Vo5TZgwfncRixYp3HrrwoQRl8aKTqdy/fXFXHbZ8FuZj1R6uo9vfGM5paXjN9XMnDlh7rhjCSkp4xdYrrkmh6uumjRu63c4Atx55xJmzhy/4dmmTg3w9a8vJzV1/ELNFVekc911ZWMeHM/qBjmjNXGikQULujAajZSV5ZGaOnASqSps3XqEXbs8wPC7JhxPURTmzetjwgStAdCqVUV4vQOdJQOBMH/+8xZqa0cfDFJSLCxbdpTMzAwcjhSuuSY/ZgScri4vb7yxG5/PgdM5ug70hYVmFi1qx2QqZtKkHK67LvZw2bu3jv37PUBB8hWcgKIozJ7tIzVVawC0fHkRs2cPdLgMBsO8+eZ2jhwZ/c2FzWZm6dIa3G4HNpuNq64qIHJcJ8CeHh+vv76dvj4rdvvoBqzPy7OyZEkLFksWRUVurrsu9vvDh5s5dKgPVR1dOYHWFzES0cps8eIiyso80e/CYZV33tlJefmoV4/JZGTx4lby822YTCbWri0gFBpo6NXX5+e117bQ1TX6wJydbWXp0ibs9sk4HHauuy527Nz6+i6qqjxEIsPvJhJv6lSV7GwvOp2OefMKmTBhoBVjJKLy4Yf72b9/9GP2Go0GFi9up6jIjNFo5PLLCwkGB9bX3x/gtdc209Y2+nJKT7eybFkDTmcRmZlpXHttbHewtjYPu3e3Eg6P/oZ18mQ90Iter2fmzAJycgZa5KsqbNhQzq5do5+AXK/Xs3BhFyUlRvR6PZdcUojfP3B++XwhXnttM83Nox9MweXSyik1NYe0NBfXXBNbTh0d/bz55rZRr/+8DI4pKQbS0oI0NrZiMhlITx/oJxSJqOza1UJb2+hvQxRFIT0djEaF+vpGrFYTVuvAQdDV5aG21ofHM/r6MaPRQFZWmL6+AK2t7aSlxd79tbZ6qKsLjKj/ZDy73UhGho+mpnZsNnNMOakq7N3bSmvryQ2wlJamYDJp5RS/L/r6vNTVtdLXN/rD1GDQk5UVxucL0dzcSmpqbHPSri4vtbV+QqFB+rkMg9VqICMjQmtrF06nLWYbAA4caKe5+eTKKTVVh8+nUl/fiF6vi0nD6w1QV9ectPP+cOn1erKyIgSDEZqaWnA6Y/shejwBamt9BAIWjMbRDdVotRrJzAzT3q4FrPhyqqjooKnp5IaBdLn0BIMh6uoa0emUmDQCgRANDS10do6+nHQ6HZmZKpGISkNDEw5H7NOp399JTY0Pv3/4fQ/jWSxGsrLCdHZ6sFhMCeVUXd1FQ0No0H6Bw+F0GnC5wtTXN2Ew6GPSCIcjbNvWTEfH6K8diqKQkaG1LK2vb8Rut2C3D1zv2tp6qK314fWOfqxak0krp54eH+3tnQnlVF/fQ13d6AP8eRkc9+3z8vzz+bz0Uj+KEnvRMhgiPP10AW53G2+/Pbr1RyIRPvrITm2ti3//98Q+kmVlHTz++DJqajZSXz+6k6irq4833sjnrbci6HSJadx1Vw933TWP1147Mqr1A5SX9/HSS/m8+qofRYk9yBRF5X//N5PiYj+vvTa69auqyiefmDh0KIPf/jZxGwoLe3j22UW0t2/m8OHRpdHb28+bb+byxhskLacbb+zkb/5mCW+9tX90CQBHjvTy2mv5vPFGCEVJTOM//iOFKVPSeeml0QfIzz7TsX27m8ceS1x/VpaHF16YRyCwg927R7d+r9fH229n89pr+qTltHZtGw8+eAEffbSdrq7R3e3X1PTw+uvaMZusnP75n43cdFMOzz8/+qECt26N8PHH+Tz9dOIYYw6HnxdfnI7B8DmbNo1u/YFAkHfeyeDFF83o9YnbsHJlK7/85QVs3bqJ6urRnduNjT288UY+b7+dvJx+8pMIt95axvPPd41q/QC7dvl5/fV8nn++L+E7iyXI889Pxums5qOPRrf+UCjEe++5aGuz88tfJm7D/Pmt/Nd/XcD+/Rvp7BzdQ0JbW++Q5XTffT6+8Y2l/J//M6rVn5/BUVW1i/vUqe3RIZKO0etVbLY09PqTqyMPhxUcDl9Mtc4xEyYEMRqNJ/WeUBsMWUdhYRdpaYkXE7fbhOEkX7Adq6UtK2vHYEgcBMDhKMTrPbm5MyMRBZstQElJ4sGdkzN25ZSX101mZuI4gHl5BgwGA4oy+kmfVVXbjqlTOzCZEsdtczrdMVW5oxEOK1gsQaZM6Uz4Li0thMlkOun3zuGwQlaWB7fbm/BdQYHui3IafYA/ti9KS9uxWhOfENPSTv68i0S09+RlZR0J76Ds9jBm8yR0utE/camqSjiskJHhIS+vP+H74mKix9Po09DKqaSkA7s9sQo4I8M+JuWk10eYOrUDnS52n5rNESyW/JMqJ9COp9RUL4WFiQF44sTwGJSTSjisY8KETpzOxOtQVpb5pK6B52VwBK0BxZQpgaSDAJjNYzOosNMZYOrUxJ02lhML5+f7KCpKvNBkZaWOyfp1ugilpYGkgwBYLGNTTikpycspI0N7ZzsWcnJ8TJ6ceKFxu8emMYJOp1JS4k86CIDdbqG3N/FCOlJWazBpOTmd2mAEYyE725s0jby8sWncoigqkyb5kg4C4HCMTTNcsznE1KnBpIMA6MeoeWlGhi9pORUUjF1jqeJiH5mZiTcjx7eROBkmU4ipUwNJBwEwGscmNKSm+pOWU1HR2DW+Kyz0kZubePOZnj66NiPHnLetVYUQQojBSHAUQggh4pzVwdFsDmM0Mqx6a4NBqzL1+Xxf1LcPPZafXq9gsQQwGlUMhhOncWx+RFVV8fl86HRqQl1+PKNRe/ltNDKszsTaBMYQCoUJBEIYDOEh3wEpioLZHMJkGn45aeM7auUUDmvvJYbO0+jKCbQ0FIUTlpPBoA2oMJJyMhq1riCBQBCDIXzC93Fmczh6PA2nhvJYOfn9AcJhdRjlpMNiCWIyqcPaF9r4kdp/tXI6cRoGgzKi4+nYvgiFwvj9wROeEwAmU2TE5WQwKAQCQUKhyLiXk5bm8MrJZGJY/V6PlVMkouL3B9Dr1RNuu8mkYjaHhl1Ox47ZQCBIMKiV01DbrigKFksIszkyomugNsatD1U9cTlp18CRl5N2DfR/UU5Dn9vaoPTjV04n46ye7PjIkSqcTidPP72Xnp6hN2PtWjc2m4n77+8kL89LTk4Ya5Kqe70evvnNWaSk2Ons7MZiMdPU1M5LL9UO2WDDbIbbbpvOhx9W8Ic/mJk5sw+7XU066ktOjp677lpBd3cPPp8fq9XMp58eYscOT+LCx5kwwcg118zkwQd3UVVlYdo0LxZL8gvhypWprFo1h5aWNnQ6BYPBwB//uIeOjqFPiEsuySAnx8Vf/3UzmZl+8vJC2JK0tlYU+PrXp5KZmU57eydms4mOjh7+9KeqIecSNBjg1lvL2L69hkceMTJjRi8OR/JyysjQ8a1vXYDH46G/34fVambr1ko2bRp6ttO8PD1f/vJsHn54J3v22Jg5sx+zOfkJvmhRCmvWLKSlRZtQ0mw28fzze07YpWDFilRKS93ce+9RbLYwRUVBrNbEi5Q2e8lk8vPdtLV1YDab6Ovr5+mnDxMeIgmdDm6+uYTKylZ+8QuYOrUXl0tNOjqO06nwV3+1HJ/PR19fP1arhX37jvLRR0PPpJCZqeOrX53Df/7nTj791M6sWR7MZhLewwPMmmXluuuW0NbWTiQSwWw28eqre6mpGbrP4MKFKcyfX8xf//VhFAUmTgwkLSeAr3yliMmTi2htbcdkMuL3B3jqqYMEhmi8qija79rbPfzDPwQpLe0jPT2StJysVvirv1qMqqr09PRhtVo4dKiOdesSJhONkZqqcMsts3niiT28846dWbP6sFqTl9OUKWZuvHEZ7e0dhEJhLBYTb721n4qKoVvgzp5tZcWKKXzve/vxevVMnuzHYkkeMK65JpeZMyfT2tqOwaAnEonw5JP7Y+ZkTea66/IIBsP88If9TJjQT1ZW8mugyQTf/vZ8jEYjXV09WK0Wjh5t4s9/bhxy/Xa7wm23zeTllz/nhRdszJrVi81GdM7X4xUXG7nttgvo6OgkEAhitZr54IMD7N8/dNeLsjIzl18+nR/+cDetrSbKynyYzcmvgZdfnsm0aUW4XK7zb7LjjRsrsdtTmDhxYA8fOtSf9CCx2UxYrWYaGx1kZ/uSHhTH6+zsZtOmKuBYZ3UtQvh8YQ4d8icESkUBpzOFcFilsdHJ/Pm9JxwOrbKygZoarZWmXq8wZ46WRmurn4aGxCun0ah8saP1dHRYsNuHPhsikQg7dx7F6w2hKAqFhZbolDUVFV48nsRob7UasdutNDWl4HAEkwbG43V397B5czWqqk0HNmuW9oNgMMLBg76EQKm1ctWWaWhwMWdOzwnLqbq6icpK7UKv0w2UU0dHgNraxIuzwaCVU2+v8sUMCv1D3l1GIhF2766hry+Ioii43WbcXwzmX1XlTXrjZbEYSEmx0dxsJyfHg802dKvdvj5PTDnNnKltQyiklVN8oFQUSEmxotMpNDQ4mT6994TDxtXVtXDwoHahP76curuDVFcn5k+vV3C5nPT3Q0uLHavVc8Inzr17a+jq0lr9pqYaSUvTrnw1NV46OxPLyWzW43Ck0NqqDbJgsw0dJHw+H1u2VBOJaOU0bZq2DZGIysGDXoJJijklxUp3t5eGBhelpX0nLKfGxjb27tUu9MeXU29vkCNHEhPQ6bRy8vlUmptTWLy474RDQh44UEdrq9YIy2bTR9Oor/fR1pZ492gy6XE6HbS3mwmFFGy2oWdYDwQCbN1aTSgUQVEUpkzR1q+qKuXl3qQTtNvtFrzeAA0NLoqK+k94DWxpaWfHjrpoGRzbhv7+EBUVgYRroE4HTqeDQCBCU5ODhQt7kwbG4x06VE9jo9aa9fhrYFOTn+bmxGugyaTD5XLS2Wmkq8uMzTb6fownclYHx3/6pwxSUsy88koqeXlac/nm5s/w+xMLNRJRiUSG/wi+f389P/5xFooCCxc28oc/LAegtbWdysr9SS/64XD4i24iw0vjhReaee21QhRF5fvfb+SOOy4AYNOm/TQ1tSf9zbEuAcNJw+/384tfGGhpScNgCPPCCyZKSooAeOqpjXi9iYFFVY+V0/CmkKmsbOLHP84EYNq0Vp59dgE6nY7u7h4qKnYRiktCUbR9oapaS9jheP31Bp5+Wiunb32rnu9+dyUAu3aVU1/fnLC8oigjKqdwOMyvf61y9GgWOl2EJ5+MMHt2KQDPP7+Jvr7EC7rWjHz45VRX18JPfpKBqioUFXXy8sszMZlMeDwejhzZnnAxGygnddjdJ95/v45HH9VG4bn55lr+9m9XAXDgwBFqahLn1BvpMauqKv/5nz4+/zwbRVH57//uZ9mymQC89tpWuruT36yNpBtLS0sHf//3aYRCerKze3nttRLsdjt+v5/q6i2Ew4llMdJy+vTTWh5+WBvm7+qra3noIa2cjhyppbq6KmH5Y+Wk/f/w0vif/+lh69ZcFEXll7/s4vLL5wOwbt0OOjoSuzbA8WmceP1dXT08+KADn8+Ey9XPn/+cT1paKqFQiPr6TQSDiWU+cG4P7wK1fXsdf/d32ig8F15YzyOPrACgvr6JyspDSWtJBs674ZXTk0928NFH+SiKyoMPtnLDDUsAWL9+N62tyefRHWkao3VWB8cJEzqYMUPP/fd3EQodRVHgxz/OpKAgM2HZ3/1uP3v2mLnkkgB2+/AKVVFg6dIGrFaVr3xlCwB5eQF+8pPpCX2A+vv9fPe7u9HpYPXq+hM+cR1jtQZYtaqFjz/W8+c/a2msXh3gL/9yRsKye/fW8pWvbCE3N8KyZYn93QZTWNjJjBkefvITE8FgEwA/+IGLK65IHKbrqac+57PPuli5cvjlBNoNhNMZ5sYbteGaMjKC/MM/lGEyxR5igUCIBx7Yi9+vY82a+i/mkjwxkynEhRc2sXu3LrovVqzwJy2nioomvvKVLbjdCitWtA/7ZiU3t5u5c3v4xS9M+P1a+X7nOxYuvrg0YdkXXzzAL3/pYdEiPzbb8MtpzpxmsrKCfO1ru1BVcDhC/PSnk7HZYh+fw+EI//RPB+jq0nPFFT3DLieDIcyqVY3U1Oij5bRggY9vfjOxnOrq2rjppm1kZuq46KKWYQ+inZnZx6JFnfznfxr55S+1NO66S590X7z1VjmPPLKXWbNGVk4zZrSQnx/gzjv3o6pgsUT46U+Lcbli+8qoKjz88Oc0Nhq54oq2Yc/PqNNFWLmyke7ugeNp+nQff/3XidvQ2trNrbdux+XSc8klzcMenzc1tZ9ly9p56ikj//3fWho336wmLaf16yu4+eadX3QHGn45lZW1MmGCn7vv9hCJgMGg8g//kEdWVmI3ht/8Zj+VlSbWrAkMOvF5MitW1ANEy2niRD8PPDAj4bzq7vbwjW/swGrVcfnljcMeID8lxceKFW38+c96nnlGS+O660JJy2nLlir+4i+2kp8fYOLEoZ+uT9ZZHRzNZhWXS+XQIT0+nwGdTqWioove3oFCczotzJs3lY4OhaNHHcyY0TaiztIORwSjUaW3V6sf6OwMc+BAfUxwnDgxi4wMF0eOOMnM9FJSMvyxG3U6bRu6uoim0djo48CB+ugyiqKwYMFkjEY95eVZ5OfXjWiSVrM5QmqqSmWlDo/HiKKoHDnSQzA48IRtt5tYsGAa3d0qVVWpzJgx/IsAQEpKBLtdpa/PiKoq6PUqBw82YjQOFHZhYTr5+W6qq+2YzWGmTBn+AAI6XQSXS6W/f6Ccmppiywlg7tyJWCwGysuzyM4efvAFMBojpKZCVZUumkZ1tQeDYSANi8XAokXT8XgiVFamMX164wmrjo5nt4dxOLRyikQUIhGF8vImLJaBws7NdVFSUkRdXTX9/Yak/cQGoyja8dTRMXDMtrT4E8pp1qxi7HYzhw5l4HI143QO/+nOaAyTmgo1NQPldPRoHykp9ccto2fx4mkEAiqHD2dQVjb8G0YAm00rJ4/HQDisIxAIc+hQKykpXdFlsrJSmD59Mo2NRhobbUyfPrKRdZzOCP39SnQb2toCCeU0bVo+TqeNw4fTmTmzDZdr+MPbGQza8dTQMJBGbW1vTBp6vY7Fi6cSDqscOpRFaWndiMrJYongdKr09xsIBnUYDBEOH26nrW3g6TQtzcbs2VNobtZTV5fCjBlDv4s+nqKoOBwRIpGBbejoCHHgQH1McCwtzSE1NYWKilQmTuxm4sThl5Ner12fWlsH0qivjz23dTqFRYvKUBSF8vIsJkwY2TVwNM7q4AhaPfeSJVrdvqrCv/1bBl1dA0fXnDnNPPPM8NY12BOGywUXXqg1AunuhnvvdcSMa/jXf13JLbfMO6k0Jk0KMWmSlsaBA0ZeeGHgqq7TRXj++cTqw5GsX6se9gJa1dfvfpdGW9vA0TV5cjsvvTS8C+RgaaSkwKpV2ohAHg/cf39mzNiuX//6Ee69d3gzcw+WRlFRmKIirZwqKw3cc09s9HvyyYaTWj/AvHk+QHuX8eyzThoaBtLIy+vm9deHF6wGS8NigZUrtXLy++H//t9cgsGBU/ErXznK3/990UmlkZcXIS9PK6eaGn1COT36qDYY+8mkMXu2H9BuRF9/PYX/+I+B9aWne3jzzRO0EDnB+k0mWLFCu8gHg/D3f+/G5xu4E1m9uo5f/3rySaWRna2Sna2VU2OjLqGcHn64mtmz808qDS1oa4H744+t/P73A2nY7X7eeGPoBmYnWr/BAMuXa+UUicBDD2XR1zfw2LZ8eQO/+92Uk0ojPV2NXgPb2xW+853Ycvq7vzvCpZeeXBpTpgSjN8w7d5p5+umBNEymEK+9NrygPlatV8/64Hg8RYElSzoIhQYKceLE4f9+yhQzS5YU4XIN3qopJQUuuST27nLixOHfwlxySQYTJ7qHrPcvLg6SkTHwjkinA4djCi0tiUPRxbNY4LrrJp5whJyFC7sIBruif+fmqsN+F1FcbGTVqglkZqZTU9OWdBmrFS6+uCHmpf1IphBauTKNyZPdQ45okp8f4tJLY9+lpaZOpKtr6Fa/oLU0vO66IrKz04dcbu7cHqZNGyj3jAx12KPR5OXpufTSSWRnZ9DbW5N0GZMJLrqoKeYd9rRpw38UXbLEybRpuUMOk+V2hxPKKTNzeBd8gwGuvbYAt3vocpo5s4/JkwfKPSVFxWBIrI5OJitLx5o1k8jKSqepKfnxZDDAypUtMeU0e/bwq4DmzbMze3YBliHq+jIyIgnllJOTPaz163Rw1VW55OVlDLlcWZmXoqKBNCwWFbN5eLO1pKYqXHVVCVlZafT0JH9vqdPB8uVtMQ28pk4dfrSYMcPKggWF2Ieod3W51IRyKihIG9b6FQWuuMKd9NXX8SZP9pObO5CGwQBW69C/OSYlReHaayeRmZnKyXTGOKuDo9sdoLAw9g7mWGvMY1wuI3V1jfT3KxgMEQoK9Oj1yQ+WwkIHRUX5NDe30tsbIDOzj7w8Fbc7tpgmTIj9nc1mpKGhlUhEweEIUVg4eLEWFWWSkZH2xewKkJ3toaBAj8Ew9AHc2+ulrc0LpJCVpQ6ahsWio7g4D5/PR0NDKy6Xn+zsQMLy8eVkt+upr2/C49GqOfLz9ZhMyfNUVJRCUVE+LS2tdHf7yMwMk5cXoaAgNo3i4tjfuVxm6uubCYcV7PYTlVMG2dmZ1NdrrQqzsrRyGixPx/T3+2lp8QAmMjMjg6ZhNCoUF+d90YChGYcjiNUaPmE5WSw6Ghtb6O3Vnujz8nRYrckv0vn5NoqK8mlra6ejo5/MTMjLS0yjKO4hMSPDTF1dI8GgLmmeYn+bRm6um8bGZsJhyMz0kJ+vw2YbOnAEg2Ha2jyAjfT0wfeFXq9QVJSDTqejrq4Ruz2M0egbMk+glW9zczvd3WF0ugi5uQpOZ/IbnawsM8XFBbS3d9DW1ktGRoDc3MQ8xZdTdrZWToGADrP5RMdTKvn5OTQ1tRAIqGRl9VFQoOBwDP2iVVVVmpq6UFU9qamDp6EoUFTkxmq1UFfXiMUSITu7n4KCoccP1esV2tq66OwMoSgqbrdCRkbyPKWlGSkuzqerq5uWli4yMvrJzQ2e8JhNTzdRV9eI36/DaAwPmaeiIicFBbk0N7fi9YbJzOwjP18hPT02T/EPHQaDjsbGDiIRBZfrRPsiC6fTQV1dIyaT+kU56Yccy1WnU+js7KW9XaupyM5Wyc1NnkZKioHi4nx6e3tpaBhejVsyZ3VwvOaaFK65ZsWQy9TXN/LYY+VUV6fgcvm57balGIdohx2JRHjllQO0tUVYvdrHzTdPobAwb9DlAT76aBfPPFOD15vDtGlB7rjjoiGXb25u5fHHy7FY4Ior+rnjjguGvPNXVZXf/34DO3dqd0Fr1mSyZMnMIdPYvPkA69d3sHAhzJ9v58orFw25fGtrG//zP/s4dMiG3R7ga19bgO0ELz/eeOMgDQ1hLr8cvvrViZSUFA+5/KZN+3jqqSP09WWzdGn3Ccupo6OTJ58sJxKBK67wcPvtSzGfoN/HE09sYPv2MJDGxRc7ufTS+UMuv23b57zzTguzZsGMGRauv37pkMt3d3fzX/+1k/37LZjNIb761Tmkpg49huO6dQepqgpy2WVw/fUFzJgxdFXgzp0HeeKJCjo7M5g/v5M77lg15PK9vb08/bTWH3DNmh5uu20hKSd4IfPccxvZujWAqtq54IITb/e+fYd59dV6Jk+GkhITN9889HnX39/Po49uYccOEwZDhL/4i2knfAr76KNDHDzo45JL4Mor3cyfP23I5Q8cqOTxx8tpbU1j0qRe7rhj6Dz5fD6effYg/f0qq1d3c+ut80647159dTOffeYlHM5n0SI9t98+dBqHD1fzpz9VU1AAS5eGufPOoZcPBAI8+uhn7Nih/2Ii6ElMnDj0RNCffnqI3bv7WbVKq4lavnzWkMsfOVLD44+XU1/vIjPTxx13XDBkLVEwGOT55w/Q06OyenUXX/vaDLKzh5478u23t/H6620Eg3nMmRPhjjsuHHL5mpp6nnzyMBkZMG2anzvuWDFkjUw4HOa///tTdu5UUBSVq6/OZ+bMoWsmtm6tZP36oftmDuWsDo7r1/fw/vvrh1zG61U4ejSFtjY7brdWFbFrVzkHDrSzfXuAUCj2IFEUSE1Voy33fvObQwSDQ8+X1NpqoLXVQTCo/SgSifDuuzuoqurn4MHEx3qDQSU1Vfv/jg6V//N/Pj3hjBBVVdaYiWY7Ojr58MNyDh700dqaeKDbbGr0xf7GjR7WrRu6nPx+haqqFDo6bKSk+FEUhf37K9i3r4Xt2wMEAolppKaq0UY7//3fVQQC1UOm0dFhoLnZgc83cHPy/vs7OHLEw/79ie879XpI+6K2prdX5Yc/3HTCcjp61EJX10Bg6Onp4b33PufwYR+NjYnbYLWq0ZZ7O3f6WL9+6HIKBhWOHEmhs9MSHf3j8OFqdu1qZPt2Pz5fYhoulxrtF/fEE3X4/fUJyxyvq0tPY6OD/n5TtLn6+vW7OHKkl927E8tJp4P0L2o9fT74+7/fPuRADAB1dWY6OlzRd+f9/f2sW7eXigovdXWJ22CxqNEGEAcPBrjvvqHLKRyGysqUmGO2urqObdtq2bnTj8eTmIbTqUYbNz3/fDNPPNEyZBq9vTrq61Po6bEAWtX3xo17qajoZufOcNJ+eGlpWtebUAj+6Z92DTkQA0BDg4mODieRiDYIvt/v5513dnHkiI/q6sTlTSaVY33Nq6tDJyynSASOHLHFlFN9fSObNh1l924fPT2J5XT8oBmvv97On/40dBoej47a2hS6uixkZGjv0rdu/ZxDhzrZvj2UtFtaWpo2CpDWGnh/wnUyXnOzkbY2Z7SNQSgU4p13dlJV1U9FReLyRqMaHXy+sTHC97//yZDntqrCkSNWuroGqsVbWlr55JNK9u3z0dmZmD+7fXjdrAZzVgfHo0dN1NUNXcd/jNEYJiVF6wxfXd3F5s1Btm1LT5gMWKdTufbadgwGaG6GPXsc9PQMb6T9lBQ/FouCqqrs3t3HgQMGDh5MTVguPd3L1Vd7aWuL0NamsH17OpHI8Pai0+nFYNDT19fPli0+Dhyw0dqa+IQ3e3YHhYVQWxuhttZIRcXwRqjX6yM4HNpL8bq6Y+WUFg38x7vqqjbsdoXGRpV9+1KGPS+b3e7HatWeiPfu7WXfPh379yfuR6fTz3XXeejsjNDerrB1a9qwJ292Or0YjXq8Xj9bt/ooL7fQ2Jj4NDVtWheLFkWoq4vQ0JB8fyWjDfrgR1GgsbGbzZv9bN+eis+XeEqtXt2Gw6FQX69y8GDy/ZWM1RrEatWuGAcO9LBrF+zenVhONluQ66/vobdX/eJ4SiUQGF6/DKfTi8mkJxAIsnWrl0OHzNTVJTbUmTSpm5Urw9TXR2hq0rNv3/DeMYE2j6JOp9DSopXTjh0u+vsTa28uuqiNrCyFujqVw4eT769kLJYgNpt2hS8v72LbNpWdO9MTJgM2mcLccEMXPp9Kayvs2OFKur+S0cpJIRQKsX27l0OHTFRXJ7ZNKCzs5ZJLgjQ2Rmhu1rF799DvamPT8KHX2+no6GXzZh+7dye/9lxwQRtut1ZOR46Yqa0dXsMqkymM3a7dDVRWdrF5cyjptcdgiPClL3UQiUBLC+za5cTjGd57cIfDh8WiIxwOs3Onh/Ly5NeenBwPa9b4aWqK0NamY8eOtGFP3ux0etHrbXR3e9i82TfotWfRonYmThz9CKlndXAMBHRceWXyF/jJZGTo0Ol0fP65lw0b3Fx5ZXPCSBc6HXz96zPYvbuG733PxaWX1pOZeeKGMMdMn57yxWTH2omdLH9ut4677rqAn/zkEzZvzmTt2tZh9zHTtmMyXV0e3nwzn2XLmli0KHE6pBUrXCxaVMq11x7G4QiOqJycTm24ufJyDx995Gbt2qakfZbuvLOM2to2vvlNK6tWNbBsWfJGAsnMmWNFVVU2bDDT3W1Kmr/0dIVvfnM5//Ivn/LJJ9lJ99dQ3O4i+vr6eeutXObObWXevMQ0Fi60c8kls7nhhj2oqjKicrJawWQyUVXVy7vv5rF2bUPS/mO33DIRj8fP176msHRp8v01mLIy7eK4aZOO6uqUpPlzOBT+8i+X8D//s5k//SmHK68cfh8zgIKCXLxeH++846asrCNpGjNnWrjmmiXccstWOjvNIyonbQzhSdTV9fLOO/msXl2fdMqqL3+5ALPZyJe/7GPOnOT7azATJmgHxrZtKrt3p3LFFYn9Wy0WuOuuhbz44i6eeSabK65oGFF3gAkTMgkEgrz7bgb5+X1Jy6C01MSNN67gG9/YSHV1CmvXtg376UWnA7u9gIMHm3jrrXwuuaQh6bXn6qtzyM528aUvdVJa2jWifZGbq0dRFHbvDrBlSyZXXNGWcO0xGuGuu+bx/vsHeOKJDFavbiA1dfgNW0pL0wiHw3zwgQunM5A0f0VFBm67bQV//dcb2LkzjTVr2ofdxU5RwOXK4ujRVt58s2DQa89ll2Uwbdoc7r9/2FmPcVYHRxhZs91jy6oq0VFB4n+v02mDHyuKgqoqSZcZOg0lmsZgv9XpBubfG2kaxy8XiegSPotP49jd2GjKSUtDQadLPmaoTqeMSTmdaBuOLTeaNLSJiIdXTiMZ3ejYb4/lbbAyUBRtDkFF4aT2RSQyUM6J+RiYp3C05XTst4PlT6dT0Ov1MefOcB276B2bFHrw8+JkzruBNAb7/bFzG4bOx+BpKF+MMjP4fow/t0fSp/r4ZbVq3KHTGCofJ0rjWDklO7e1clKix+xwR4GKN/Qxe+y844vRskZ2DRw4Zoc6nk5untOzelYOIYQQYjxIcBRCCCHinLfB0WQKs2KFI/quYjxkZ/u48MI0HI4xGrIhienTIyxe7BhR9c1IGAwRli2zM3nyCMZIG6H0dD8XXphGWtr4ldOUKSGWLXOOaEi8kdDpIixebGHq1OE13hoNl0srp6ys8TttJ00KcsEFrhG92x0JRVFZuNDEzJnDHxBipOz2AKtWpZKTM4IX+SNUVBRg5crUEb3bHal58/TMnWsbsxFf4lksIVascCX0Tx5LeXk+Vq1Kw24fv3N71ixYsMA+5uV0XgZHnU5r4bd8+VRKSpyYzRz3TyEYDMWMOzoaBkOEnJwQK1fOISPDEJOG0ag1CT9RM/KhKIo2pNL06SksWlSCxaLEpAHahKAnM1unTqe1BFy6dDJlZelx5aRNknvy5aSSnR1g5co5ZGWZEsopEAicdDkZjaEvRj+agtUaW07HmuefbDmZzSEWLy5m+vSsmPWbTGNXTunpWjnl5JjH/HgCrUV3SYmBZcumkZISW046nZbGcFtVJ6PTacfsggX5zJyZk3A8RSIRAoHhj0ucjNZNKsDKlbMpKLDG7QuFYDBIKDT6na0oCkZjhAkTYPnyGTgcupg09PqTL6djx+zcuVnMmVOQpJxUAoHhj7ebjF4PKSkBVqyYwYQJKUnKKXRS5QTaWMUFBRFWrJiFyxVbTgbDsXIa/fq1fRFi9uw05s+fkFBOoPUjHa2zvkHOaMyaZUOn68ZgMLBw4VRmzx4owEgkwksv7WbnzjAw/GbYx9PpdFxyiZf8fG0PffnL82Om7Wlr6+S//3szhw6N/u45Lc3B1VeXk5eXT2qqi7vvXhjz/c6dlfzudzvxeLJxuUZ3gEyd6uCKKzowm0uYObOEKVMGOierqsprr+1m584gMLoRgBVFYdWqAE6ndhhee+286LQ9AN3dvfzP/2xh797RP405HDauuqqK4uJs7HY7f/mXseW0f381v/3tdrq6MkhNHd3ccCUlTtaubcVmy8ftzqK4ODfm+zff3MOuXX5UdejO3UNZvlyNBvArrpjLZZcNBBGPp5/HH9/Otm2jP52tVgtr17ZRUpKGxWLh619fFDP01uHDdTz66Gba2lIxGkcXhYuKXFx1VTNO5wwyMtK5557Y4cDee28fO3d6iUSGN7RdMosWGejuDqAoChdfPIuVKwfKye8P8Mc/7mbr1tEHLpPJyBVXdDB1qgOj0chtty2MKaejRxt59NFNMePxjlRenourr64jLa2U/Pwc/uqvYpv2fvzx57zxRhOh0NCDkwxl3jwzGRn96HQ6li+fzqJFA8E2FArx/PN72L79JIZeMxi4/PIepk61otPpuPnmhTHXwMbGNh59dBPV1SOYHiROVpaDq6+uIitrAllZGfzVX8UOdLJpUzn/8z9bRr8No/7lWayzM0hFhZ7nntuaMGyboihkZZnJzz/x+JyDUVWVxkaV5mY/odCnCd8bDDrKyuzs2OGjYXjjZCfw+wPU1BhYt66BPXsSO0tbrXrKyqyjvpABdHcHqKrS8/zzOxOGbVMUBZfLRFHRyd3pNzWpVFUF0esTy0mvVygttbNv3/C7PsQLBkPU1Rn48MMWDh9OnObLYtFTVmbDZBp9OfX2Bjh61MBLL+1NOpSc02mkuPjkKmlaWsI0Nys880xiOel0CiUldioqRn/MhsNh6ur0fPxxJ7W1iWmYzXqmTrVhsYQIh0cXXPr6/NTUGHjllQOkpCRWeTocRiZNOrlLUltbiMpKhT/+cWPCdzqdQnGxlaNHR19OkUiEujodoVAvra2J5WQy6Zg2zc4HH4To7h5dtW5/f4DaWgOvv15Jamp1wvcpKQYmTx4YIGI02tuDHDqk8OyzmxNeySiKQn6+hYaGkyun+nqFnh4vfX2J5WQ0auW0cWOQ7u7R3fz6fEFqaw289dZRMjMTB9aw2w3RSaBH47wMjjU1QTZuLOSzzxInSDUYIrz4ohWTqY3HHhvd+rXO7SnU1KTxxhuJ9QZTp7bx3HPzWb/+Yw4cGF0aHo+fbdty2b49+SSvd9/dwDe+sYhf//rQ6BIA6uv9bNpUwObNiWkoCjzzDGRkWPmv/xrd+lVV5fPPLRw8mMU77ySWU3FxFy+/PIcdOz5l27bRpeH1Bti2zc2WLUrScrrllnp+8IML+I//2D3qqtWmJh9btuSydWvyffG73/kpLk7lN78Z/cXs4EEj27fn8MEHieWUnd3Hn/9cRkXFVj78cHTrDwSC7NiRyWefGZNuw9VX1/Pzn1/I//7vJjo7R3cxa2vzsXVrLtu2JS+nX/2qhxUrivjlL0c/u/vhwwrr1+ezfn1iOTmdPl5/vZCWln28/vro1h8Mhti5M53eXmvSybovuqiBf//3ZfzxjxtGfdHv7PQOeW7/4z+2c+GFJfzLvwx/6ql4VVURPvmkkE8/TdwGiyXIn/+cidd7hD/9aXTrj0Qi7N7torXVwauvJqaxcGEzf/jDMl577WPqEufhHpaenqHL6f/8n2a+9KWhh48cynkZHAHM5iAXX9yMwRBbqFpdfOIkm6ORl9fN/PmJ09Hk5SnDngHjRObNayYvL/H9w5Qpo6/WOZ7BEObiixsxmxODo8s1hb6+k59wNDu7l8WLuxI+z8zkpPopHW/mzBaKixOrl2fMGJuGITpdhIsuakg6oW9m5kS83pMvp7Q0DxdckHhBdDoZcmzekZg6tY2SksTgNHfu2DTIUhSVVasacDgSL5i5uaOvTj2ew+Fl1arEzvfaoA2TxiSNyZPbmTo1sUZj+vSxu6QuX95IWlpijUZh4fBmCjkRm83PRRe1JDw5aoM2jM2+KC7uZNasxA76kyaNXXOXRYuayM5OrMGaOHH4ozglc94GR0VRSUlRE1rl6fVjd0E2GiNJJ9u128fuwLBak6dhNo9NSz1tstOBsRwHPtcmah0LBkPybUhJGbtysljGu5y0aZriR8hRFDAYxiYNvV5Nug0Ox9jdbJnN4aRpWCxj1/LTbk++L8a6nJIFx7EqJ5MpeTlZrWNXTjZb8nIyGscmDZ1OK6f4y53JNDDAxcka7Bpos41/OZlMJ5fGedlaVQghhBiKBEchhBAizlldrTp9uo/i4lTq64OETtBoMjtbR2qqkS1bPqe/X5s8d6ja0/R0K0uWNFJaqlUpNTUlToFzPJ0OCgoMBAJhtmw5QEpKMOF9ZrySEj09PW1MnGiguztMd/fQy9tsCm63gerqNtravLjdFiyWwTsK6fV65s/vxmZTmTDBSENDkBN1+8nM1JGWZmDbtoN0d6tkZ/cNWU4ul1ZOZWUqKSl6Gv//9s49qOkz3ePfX+4XEpIAIYRrBCmgSAEBr4ioeCletue09DZrte6sndWuW/fM6Z5Oh+6cmdN2d+vs7PTiTmu77Wx77Ha27W6P1o5WQZDiDSwoCKjITS6GWwK5J+/546dJQ+CXgKdaju9nhplAXvK8v2+evM/7e/O+z9MbXKfYWAEIIThzpgkSiRs8Hvf3cUlJQuTl3cScOQKMjbkxPMytk0QCxMQI0d09grExB7Raobdqw+R94iEra8xrq7fXCXuQrwg1Glan+vorMBpd0GrNnMnjw8KkKCi4jrQ0NyIiBOjpcXHqxDCAXs+HUMjD6dMXIRR6oNFwb1SJi5MgP38Ac+bwYbcTDA5yHyITiQC9Xoj+fjN6e02IjGQgl3P/z/z5doyN2WEwCNHX54TVytkcKhUDjUaAxsbr6O21IzrazJmIQSoVIz+/Aykpbuh0rE5cZ+EYBtDp+JDJ+Dh9+iIAICLCynkgPCZGioKCPqSk8OF2E9y8yX3NQiEQGyvE8LAVFy5cR0SECwoF9+7m9HQX1OoRJCUJcfOmC+Pj3D6rVDKIiBCgqakLnZ0WREfzIBRO/T9isQh5eTcRH88W9O7pcQU956rT8aFUCnD69EU4nQwiI7l3gWu1sltjIA88HoP+fm4DAgGr0/i4A+fOtUGlskOh4B6YH3iAgMcbgsEgxOCgC2Yzt05hYQyiogRoa+tDV9cYoqNdAfsh/q+Y1cHxN79ZDplMhv37T2FkhFug5ctjIZWKsWXLOLKyBFi2jLuCRE5OOt57j318uzAnFyIR8K//moPPPqvHb36jxkMPDQXN+L9jx3Lv42PH6lBbG7h55/vExgpRVrYE27dX48oVBUpKAo8mfB+xWIzXX2eLjno8HrzzTnXQgaCgQAu9PgIPPWTE3LkiFBZyVyRJT0/Ge+8lA2CLOB84cIlz0OfzgS1bMlFRcRk7dkiwbp3RW9tyKp54YimeeIJ9XF3dgIoK7l16Wq0ATz65FM89V4lz5zRYvz6wQoN/n/h45ZUi7+/vv1+Fnh7ugSAnR4P09Hhs2NAJnU6CoiLu985giMeBA+w5x+HhEfz5zxc4J3Q8HlBamo7Gxi5s3y7F6tWDMBi4I9GWLQXYsoV9fO5cE44c4a6HqFbz8eSTS/DSS5X4+mstSksHOAM8wzAoLy/y/v7xx6dw7Rr3YfR585RYtCgNpaWXIZdLsXLlKGf7mJhovPNONABgbGwMb799LuhEZd26FPT0DGHbNglWrADy87kH/ZKSXJSUsI8vXmzDF19w19dUKHh4/PFF+P3vq/C3v+mwcWNf0AxC//7vRd7Hf/97LZqbuSc2qalyrF79ILZsqYfLJUVxMbdOGo0a+/ezRbBtNhvefvt00ABcXJwIq9WBsjJg0SIeFi/mPqqxbNkCLLtVq/nKlQ4cPNjO2V4qZVBWlod3363Fn/8cjtLSYUiD7Hd77jlfUeRDh86ivp67T0lJEmzenI/HHquF0SgLqtOdMKuD4549pyCTyfHMM9FQKqUgBPj66+tB7yxC4cKFFuzf3w8AiIvz4PHHDQCA0VELjhzpv6PMDrf5y1+qUVvLvlBRkcBro7m5HxcuzPxs323sdjvKy7/FyAgPPB7w9NMRiIhgI/axYx1BA2UoXL58DX/8I7sXW6v14Kc/NYBh2KMmhw/f4AwAodZv++STGpw4wb7Q4sU8r05Xrxpx5kzo5cSmwu1247e/rcLAAHuL/NRTKhQWsgevKys7cePGHaaeAVvo97XXroEQQKXyYPv2RPD5PNhsThw61B30jj4UvvzyDA4dYgfh7Gx4dersHMKpU3c+iBBC8MorlejsZHV65JEwFBTEAQBqarrR0XFnWVsAoK9vAP/5n5fhdgNhYR4880w8RCIBnE43Dh3qDHqnGgrffFOHTz9lJ8cZGcSrU2/vKCorh+4oW9Jt9u2rRGsr69+bN0u9Ns6d60Fb252/2cPDIygv/w4OBwOJhGDHDj2kUhHcbg+++qoz6B1YKNTUNOLDD9kJeHKybww0Gs04dsw4pU5kGgK+9dZJNDSwj9euFXltNDT04tKlwAnFD5VKbzJmdXCsq4uERiPA+LgDEgk7ldNohBCLAwczo3EMbrcZ4eFMSLfhRuM4qqpioVBY4XCMeNNaEQJER/MDHEMgYHDlSjdGRlxQqy0h5TptaXHh1KlYKJU2pKcPeW1IJIJJ80IKhTw0N18Fj0egVAb/gLndbpw+rcLIiBRKpQ1jYw7vModaLQSfHxi5RkdtsFr7EB7u4lyyvc3IiAVVVbEIC7MhOXkUDocTDMPA4yHQavkBkwgeD2hv74PR6IBGMx5SHcurVx2oro5DeLgVSUmDXp1EIv6kOkkkfDQ3XwUhbEHiYHg8Hpw7F4YbNxRQKOwwmeyIiGBtqFTCSSdCY2MOtLX1QKGwQyYLHjzNZguqqvSQyRyIizPBbndCIODD7fZAqxUEpOpiGKCz8yb6+61Qq/kh5YTt6LCiqioO4eEWREX5dBIIeJPqJJfzcfnyNTidgEplC2ngqa8Xo7U1AgqFDWvWWLw2lEoBdLpAoex2N1pbOyGTOSGTBU8YYbHYUF2tg0jkQnT0OGw2NuC63R5ERQngcATq1NMzhBs3xqBWuzmXIm/T3c36bHi4BWFhPp34fAbR0ZP5Ew8tLe2w2QCVinvJ9jYNDXzU1UVDqbRi+XKz10ZYmBA6XaC/uN0Ely+3QyIJbSJms9lx6pQWDANoNBZYrU7w+TwQQhARIZh0eby/fxSjo3ao1SKIRMF16utjx0Cl0goezzc+MQwDnS5wDBSJGLS2dmB8nEClsoY0Bl66BNTUxECptCI3d9RrQyqdfAwEgObmqxAKPQgLu/PJGBezOjgmJpqQnc3H3r1RsFg84PEI/vY3HVJTDQFt/+3fKlFTE4F160IvqgkABQWDIAR49FHW2VJSTPj008Xe+nm3MZvN2LixHRER4qDLnd9HKnWipGQQVVUSvPsua2PbNjN+9avCgLZVVQ149FEPioo8WLQo9OwV8fGjWLjQgpdeioDJxNr44AMVsrPTAtr+9reVOHxYiw0bBqeVpDs3dxBSKUFZGQFAoNeP4R//yIN4whkQu92OzZsvQSQSY+3a0A8xi8VOrFo1iO++E+Gjj9hr+Jd/GUF5+YqAtnV1l/Hoox4sXcoEXT7/PjExZixZMobXXlNjaIi18eabUvzkJwUBbfftq8THH0uxYcNYwDEXLrKyBqHRuPHUU2xQ1mis+PLLBVBM2IvudrvxyCPnYbGIsW5d6DoJhS6sXDmIzk6B12fXrh3G738fqFNrazsefdSFhQv5WLGCe1n4+0RFjaGoyIS331aiv5+18dprPDzzzLKAtu++W40XXlBgwwYLZNNIVpKRMYS4OBe2b1fC5fJALrfjyy/TEBkZEdD2qadq0Ncnm5ZOPJ4HhYWDMBr5Xp2WLh3G228vD2jb1XUDP/mJCRkZwmkt46nVFqxePYy//jUMr77K2njxReekOn3yybd44glg3ToblMqQTSA1dRgpKQ48+6wMdrsHIpELn3+ehPj4wNRyO3dWobk5HCUlQ9OonUiwdKkRFgvPq9OCBSP4618Dr8FoHMTGjX1IShJhzZrQx8CwMBtKSobwP/8jw5/+xNp47jkrduwItHH48DmUlRGsWeNCWhoNjpzw+UB29iBcLvbdfvddNwSCTu/zCQk8PPtsIQhhZpQMmGEAhYKgoKDvlj2Cl16q9mtTUhKOhQtTvMU9Z0JCgh0qFWujvR34j/+o9OvD7t3pAHyFe6d7DTwesGDBMJxOtn8ff+zGp5/2e9tERzN47rnl3oK0M7Ehl/t04vGAl1+u9fsQFhbKUVy8YMY6MQwQG+uAXM7aMBp5fjoBwM9/ngKGmZlOt23Mnz8Ku50NFv/4hwdffeWzoVYDzz+/zFtEeSbLPFIpkJfX77X3X/9V5/c6+fkSbNqU5y0EO5Nr0Olc3vfCag3U6emnEwDgjnw2LW0MSUnsJO3oUQ+qqnw25HLg179edMvGzHxWJAIWLhzwFlf+wx+MfhPbzEwhnnhi6Yw/2wAQFeX289mJOpWVxSAiQnFH/jR37jhiY9n14NpaD777zmdDLAb27s0F4CvAPF0EAiAnx+hd4XjjjX7w+b49EnPn8rF9+3LO4sPBUKs9Xp0EAhKg0+bNkUhO1sHj4c34s20wWBEVxa70XLzo8bMhEAC/+lUWgJl/tqfLrA+ODMN+J3ibigolTCZfHZnMzMGQX0ssZg+eT7wrlEiApCR2ucNsBo4fj/JzgNTUQSz0z2c9JWFhDBSKwOUCjYZAo2FtXLkiwOnTvqTnPJ4H27aF9mULj8fufJNIAt/a2FifTtXVYRga8k3l58wZwe7doY3EIhEglzMBh7ZFIp9OVitw/Hgk3G6fI+v1N1FcHJIJyOWsThMPbatUgErF2ujo4OPMGV+2EIYhKCsL7Y6aYVidZLLAnRUxMT6dTp+Wob/ft7NKrzfjl78MbelLKGTf74k6CQQ+nRwO4JtvIuB0+tooFAPYtCkkE5DJGCiVk+UphXdH5Y0bPHzzjX9WldLSMURGBk8Yz04OGchkgf70/WXU8+el6Onx3f2q1Vb88pehzewFAtbGxMPtfD6QmMheg8sFHD+uhs32/feLe8PR95FKgfBwfkCCD7kckMtZGzdvMgE6FRWZERERPNvUbZ3k8kCdtFoCgLXR2ChGe7svkbhc7sDu3aHpxOezNkQifxs8HpCQwL6+xwOcOKHC+Lgvo9H4+M2QXh9gxzqFghegk1Tq89nh4UCdFi40ITk5+OuzyTKYSXPrRkYSREayNpqbhWhr82W4EYlc2LkztLSCt3USi++s7tqsD44TKSw0gRDfElF8fOhZEjIy5Fi7NpszFZdCATz00E2/GX1KSuhpijZsSEBycgJnpo7kZBcMBt8Hnw140ejtDb6kI5Uy2LZtIaRSKWy2qZ1pyZIxEOJbctRqeSFnD0lJkWDTpoUQCAS4dq1/0jZSKbB+vf+X9unpoa8XrV4dg4yMZM5sRQkJbsTF+XRiv3+ZC6Mx+BKhWAz89KcPQqFQ+FUCmUhengWE+DZHqVRMwORp6v6J8Mgj+eDz+RgennzjkEgErF076KfTggWhrz8WFkYhO/sBzj7FxHjw0EP+gUSrTfKrkjAVAgHw5JOZUKtVnO1ycqx48EHfBE4uB4QhFoXU6QR46qlF4PP5aG/vmrIfq1cP++k0b17o69kFBRosXjyPU6eoKBKgk14fWuULhgEeeSQN0dFRnO3mz7cjI8Nngy2vlBKSjYgIHrZtWwyBQIC+vsknBjwesHLliJ9OycmhB4nsbCWKirI4dVKrA3VKTAw9pd2WLcmIi4vhbJOW5kRqqs+GQADIZAkhvb5SyWDHjkUQiUQwm2e+YW9WB8eCAjtWreIWzO0mOHasDv39AshkTqxapYFAMPmAq9WGg8/no7b2Ijo7x5GdbcWKFWLo9dwzbLvdjYqKZjgcciQk2LBmTeSUbbVaDSwWC86cabu1bGTE6tVT9wlgl9aamrrR2GgCEIbcXD7S01WTthUK+ZBIJOjo6EFLSz/mzrVizhw3Vq6cuk8AWyPuxIkL6OnhQyJxYuVK1aR3nwAQEaGAQCDA2bNNaGsbRXb2KAoLBUhK4g5+Docbx483wGYTIylpnFMnvT4SDocD3357GYQAublGrFqlDpru7erVAbS1mQHEIisLyMmZ3Aafz4NMJkNPTx+am3uRlGRBdLSHs08Aq1NlZQM6OhgIhW4UFiqhUEyed1SlkkEgEKCurhmtrcN48EGCwkIeUlO5J1NOpwfffHMBY2NCREdbOfuUkBAFl8uFqqpG2GwEOTk3UVwcDrmce0Ds6RlBZ+cYCNFh3jwXli2b3AbDMAgLk2NgwIiLF7uh1zswd649qE6EAKdONeHKFTf4fA+WL5dDo5l8X39YmAQCgQCNja1obR1CVpYby5YRZGZy23C5PDh2rA6jo0JoNDasXh055ZJhfHwkCCE4efI7jIy4kZPTj+JiBcLDuQPs0JAFly4Z4fFEIjXVyXndKpUSQ0PDuHChA5GRLkRFDXP2CWB1OnOmDZcvO8DjESxeLEVMzORlnGQyEYRCIZqbr6K11YjMTCfy8z3IzQ1Np6EhIRQKB1atipgyPZxerwHDMKiubkB/vwPZ2f1YuVKOyEjuMxnj4w5UVV2By6VCcjK3f0REqGA2j+HcuatQKj3IyxvC6tVT9wlgdaqvb8fFi+NgGDXy84UwGCYfb8RiIUQiEa5c6UBTU+ekbUJhVgbH21uFk5N5SE/nDo43bvThq6860d8vh0AwgrS0BZwz2tHRUdTWdsFo9MBgANLS2JpqXFRVNeDUqVHYbDyoVGNIT8/mbN/ffxOVlb3weIA5cxikp8/nvFslhODDD79FQwOB221GQoKA87rHx8fR2tqF6uphaLVASoosqE6Dg0P4y18uoadHCh5vFGlpqZByHFIymUw4e5Y95pCUBDzwQCLmzOG2cfZsE06eHITFEgWl0oT09CzO9v39Azh5kj0OwuqUFrDBZyL//d/for7eDbfbjNhYcF631WrF1avdOHnyJjQa9o44mE6jo6N4770GdHZKAIwiNTUOKlX4lO1NJhPOn+9CR4cTSUlAaqo+qI3vvmtFZWU/TCYNkpNHkZ6eydneaDTi5MkeOByAwcCePZVPTPI6gb///TTq6hxwu+XQ6dycfbLb7bh+vQcVFb1QKACDQRT0GiwWCw4cOI9r10QATEhJ0XDeVZlMJly40IWWFhsSEoDUVG1QGy0t1/DVV70YGVEhKWkMGRkZnO2HhoZw6lQPxsfJrc92Iud7BwCHDp3DmTNWOJ1SREc7OPvkdDrR1XUDFRW9kEjYpCAZGdzX4HA4cODAGbS08OHxmDFnjgJJSXFTtjeZTGhs7EBDgxVxccDcuZqgOrW3d+HIkS4YjUpoNGPIyMjjXCUaHh5GTU03TKbbOqUjKoo7AB89Wofa2nE4HCJoNBbOPrHlrHpRWdkLPp9NQp6R8SBnn9xuN957rxZNTQzcbjOSkqScNsxmM5qaOlFdza5sTed4yW0YMpP/usd0d3cjPn7mhWMpFAqFcv/Q1dWFuLipJx2TMSuDo8fjQUtLCzIyMtDV1QXldPY+38eYTCbEx8dTzaYB1Wz6UM2mD9Vs+oSiGSEEZrMZer1+2tWWZuWyKo/HQ2wsW29MqVRSZ5omVLPpQzWbPlSz6UM1mz7BNAsP5146nwpalYNCoVAolAnQ4EihUCgUygRmbXAUi8UoLy8PunuR4oNqNn2oZtOHajZ9qGbT54fWbFZuyKFQKBQK5Ydk1t45UigUCoXyQ0GDI4VCoVAoE6DBkUKhUCiUCdDgSKFQKBTKBGZtcHzrrbdgMBggkUiQm5uLqqqqe92lHwUvv/wyGIbx+9HpfLlhCSF4+eWXodfrIZVKUVRUhEuXLt3DHt99Tp48iY0bN0Kv14NhGHzxxRd+z4eikd1ux+7duxEZGQm5XI5Nmzahu7v7Ll7F3SWYZk8//XSA3y1atMivzf2m2SuvvIK8vDwoFApotVps2bIFLS0tfm2or/kTimZ3y9dmZXD85JNPsGfPHrz44ouor6/H8uXLsX79enR2zjwD+/8n5s2bh97eXu9PY2Oj97nf/e532LdvH9544w2cPXsWOp0Oa9asuaPSLrON8fFxZGVl4Y033pj0+VA02rNnDz7//HMcPHgQ1dXVGBsbQ2lpKWf5q9lMMM0AYN26dX5+d/jwYb/n7zfNKisr8Ytf/AK1tbU4evQoXC4XSkpKMD7uqzlKfc2fUDQD7pKvkVlIfn4+2blzp9/f0tLSyAsvvHCPevTjoby8nGRlZU36nMfjITqdjrz66qvev9lsNhIeHk72799/l3r44wIA+fzzz72/h6LRyMgIEQqF5ODBg942PT09hMfjkSNHjty1vt8rJmpGCCFbt24lmzdvnvJ/7nfNCCFkYGCAACCVlZWEEOproTBRM0Lunq/NujtHh8OB8+fPo6SkxO/vJSUlqKmpuUe9+nHR1tYGvV4Pg8GAxx57DNeuXQMAtLe3o6+vz087sViMFStWUO1uEYpG58+fh9Pp9Guj1+sxf/78+1rHiooKaLVapKam4mc/+xkGBnzFaqlmbLkzANBoNACor4XCRM1uczd8bdYFR6PRCLfbjejoaL+/R0dHo6+v7x716sdDQUEBPvzwQ3z99dd455130NfXhyVLlmBwcNCrD9VuakLRqK+vDyKRCGq1eso29xvr16/HRx99hOPHj+P111/H2bNnUVxcDLvdDoBqRgjB888/j2XLlmH+/PkAqK8FYzLNgLvna7OyKgeAgMKYhBDOYpn3C+vXr/c+zszMxOLFi5GcnIwPPvjA+6U11S44M9HoftaxrKzM+3j+/PlYuHAhEhMTcejQITz88MNT/t/9otmuXbvQ0NCA6urqgOeor03OVJrdLV+bdXeOkZGR4PP5ATOAgYGBgBkYBZDL5cjMzERbW5t31yrVbmpC0Uin08HhcGB4eHjKNvc7MTExSExMRFtbG4D7W7Pdu3fjn//8J06cOOFXcJf62tRMpdlk/FC+NuuCo0gkQm5uLo4ePer396NHj2LJkiX3qFc/Xux2O5qbmxETEwODwQCdTuenncPhQGVlJdXuFqFolJubC6FQ6Nemt7cXFy9epDreYnBwEF1dXYiJiQFwf2pGCMGuXbvw2Wef4fjx4zAYDH7PU18LJJhmk/GD+VrIW3d+RBw8eJAIhUJy4MAB0tTURPbs2UPkcjm5fv36ve7aPWfv3r2koqKCXLt2jdTW1pLS0lKiUCi82rz66qskPDycfPbZZ6SxsZE8/vjjJCYmhphMpnvc87uH2Wwm9fX1pL6+ngAg+/btI/X19aSjo4MQEppGO3fuJHFxceTYsWOkrq6OFBcXk6ysLOJyue7VZf2gcGlmNpvJ3r17SU1NDWlvbycnTpwgixcvJrGxsfe1Zs8++ywJDw8nFRUVpLe31/tjsVi8baiv+RNMs7vpa7MyOBJCyJtvvkkSExOJSCQiOTk5flt972fKyspITEwMEQqFRK/Xk4cffphcunTJ+7zH4yHl5eVEp9MRsVhMCgsLSWNj4z3s8d3nxIkTBEDAz9atWwkhoWlktVrJrl27iEajIVKplJSWlpLOzs57cDV3By7NLBYLKSkpIVFRUUQoFJKEhASydevWAD3uN80m0wsAef/9971tqK/5E0yzu+lrtGQVhUKhUCgTmHXfOVIoFAqF8kNDgyOFQqFQKBOgwZFCoVAolAnQ4EihUCgUygRocKRQKBQKZQI0OFIoFAqFMgEaHCkUCoVCmQANjhQKhUKhTIAGRwqFQqFQJkCDI4VCoVAoE6DBkUKhUCiUCdDgSKFQKBTKBP4XNNo6VohxDQ4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# since env is placed in array, we need to get by index\n",
    "# Due to VecFrameStack, we have 4 diff imgs stacked together\n",
    "plt.imshow(state[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "6db5e97a-9a8a-45fc-b302-0985be48bb8a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAz8AAADOCAYAAAAQc7xiAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAA9hAAAPYQGoP6dpAABM5UlEQVR4nO3deXxc1WH3/8+9d1aN9sXavWCMWUwgmD3si4kTlpQ0ISFJoc1CFmj5kZQnlOfXuFtI06dJnl9p02ZpQkhT0iyQpCEpJoAJa8DYGBviBcu2ZEnWPqNl9nt+f4w88iDJsmzJGmm+79drEs/M0Zlzj+4XnXPmLpYxxiAiIiIiIrLA2XPdABERERERkeNBkx8RERERESkImvyIiIiIiEhB0ORHREREREQKgiY/IiIiIiJSEDT5ERERERGRgqDJj4iIiIiIFARNfkREREREpCBo8iMiIiIiIgVBkx8RERERESkIczr5+Zd/+ReWLVtGIBBg9erV/Pa3v53L5ojkJeVEZGrKicjUlBOROZz8/PCHP+TOO+/k3nvvZdOmTVx88cWsXbuWffv2zVWTRPKOciIyNeVEZGrKiUiGZYwxc/HB5513HmeddRZf//rXs6+dcsopvOc97+G+++477M+6rkt7ezslJSVYljXbTRWZNmMMg4ODNDQ0YNtHv8ZwLDkBZUXym3IiMjXlRGRq08mJ5zi1KUcikWDjxo18/vOfz3l9zZo1PPfcc+PKx+Nx4vF49vn+/fs59dRTZ72dIseqtbWVpqamo/rZ6eYElBWZn5QTkakpJyJTO5KczMnkp6enh3Q6TW1tbc7rtbW1dHZ2jit/33338Vd/9VfjXr+Id+HBO2vtFDlaKZI8w6OUlJQcdR3TzQkoKzK/5FtOGv7h89jBwFG3RWQ2uNEY7X/+JeVE5DCmk5M5mfwc9NavTY0xE36Ves8993DXXXdln0ciEZqbm/HgxWNpQCd5aPRg0pk4NOBIcwLKiswzeZYTOxjQoE7ylnIiMrUjycmcTH6qq6txHGfcakNXV9e4VQkAv9+P3+8/Xs0TyQvTzQkoK1J4lBORqSknImPm5GpvPp+P1atXs379+pzX169fz4UXXjgXTRLJO8qJyNSUE5GpKSciY+bssLe77rqLj3zkI5x99tlccMEFfOMb32Dfvn188pOfnKsmieQd5URkasqJyNSUE5GMOZv83HTTTfT29vLXf/3XdHR0sGrVKh599FGWLFkyV00SyTvKicjUlBORqSknIhlzdp+fYxGJRCgrK+MybtBJ3JKXUibJU/yMcDhMaWnpnLVDWZF8lm85abp/nU7klrzjRmO03b5OORE5jOnkZE7O+RERERERETneNPkREREREZGCMKf3+REREZnv7ECaq076PY/vOBk35nDOyS3s6qumv6uEusZ+3lbVzgvtS4n0hGhu7uWUirHLDT/bdgLDfUFOWNrFiaXd2def3nsiHk+aCxtbABhO+Xn29ROP+7bNOgsuPnUHmzqbGOotYumSbjy2y669tVxx6u/x2GlgfH8AbOurZ39bJeU1Q6ys7uLFN07Ivvf2k/ZSExgC4JWuZno6p3e4WKAsziVLdmWf7wgvYk9bNVecsj3bpuGUn23ddZxSfYDn31iO5XO5+uQ3eKb1BC5q3p1T39G0YaFRTo6BcjKj9M2PiIjIMQgEE/xb0/P843k/4rozX+XBZb9mdW0bjU19fP3kHzCYDPDvZzxAcdUI1zVu4YNVL/J8+1KiaS8/OuubWF6Xtt5yXGPzjw1P8vKBZtIpm5+c9U0GkwGeb1/KH9f+lrVnbJ3rTZ15Fvyfpl+xoqobT1GKmxpf5iONz/Ohs17kXZVbeL59KV7L5YGz/53TFnXypfrf8LuOJWztq+dHp36P0uphzqzdz981/Tyn2nubf8mJRV3sCC/im6c9SGVtZFrNike9vHygma82bCDlOrT3lYFr8VJnM++v+h3vKN3J5s5GTqk+wANLH+f9q1/mb877Gf/W9DxLKvv5t6bnaR2u4Pn2pTzfvpT+SNFM9tq8pJwcA+VkRmnyIyIiMgPeExri/2t4Cb/lxbZcHjjle/zfA1fx/BvL+dXg2/jS6T8FYMAtYrA3xJklbSzxeMCCxKCP1uFy0hj6e0tIp2yWeHy0DZUz2BMiZrw0+fvneAtnz/trX+Y/Lvgm5wQzK9bL/N0MpIsY7Amxb6SCFZ4kAEkMA73FLCnpp95TjN+bmrTO/fEK9uyt4cG+C/jn0/5zWu0xCZv+vmJcXNqGy0kM+sDAYE+IgXSInlQpw31BALyWw9/XbuZDJb05dVxavZPLmnZxas0B0iM60OYg5eToKSczQ2kUERGZATe3XE6lb4TPLvpN9rVP1T7BRWU7uTC4m92pSgDO83fyncv+Ha+V4uJNH8YktQ75gZJ+9qXiLPYUsy0+ebky28f9l3yf0309XLTlj+jpLoWqzsl/wDHU+cO8mVw0840e1ZEa4i87rmE45eMHy57Mvv5418m0D5SSTuv3eyjl5OgpJzNDe5KIiMgMeLWzgV++ejoxM/an9esHruBvn72Wn0TOyr72YryOv2m5lrf7Utyx4imwDStP6ODmhhcJWB7ed8ZGQiUxvh9p5sOLX+TGt79CnRPhsc5T5mCrjo9vh+t498ZP8Lt4ZuX6J51ncXqglRvf/go31b3Et8Kn4xqLsJvg/339BmLG4u9P+jHeYKZ8tePwh2dt5Ma3v0JDUx8A7yp7lU+f8xQr/R385UvXH3MbLY/hhrdv5nR/BxeFtrPmbduwLUMSeHzrKbzc2pxTfm3tNt659A3evXwbtQ0Dx/z5C4VycvSUk5mh+/yIzIJ8u3+JsiL5KN9yctT3L7EhUBLH601hAcmUQzzqxeNL4/OlsC1D2rVJxD24ro0/kMDnSRPyJ+iJhIgP+vEEUyyv68YYi9b+cmLDPrCgqbafYl+c1oFyhnozx8M7RSlCRXGiUR/J4enn2vIYTMLGitmYovRhCkJp1TCGzGEsBwXLY3g9aQbDwRlZjfeXxknEvJiEjVOU6cPUiAd/aZzq0szndxwoBwt8gSTxiB9vKInjcYkP+7Acl2AoweKKftrCZQwPBXA8aRwnM7xJxD24Mefo2xb1ZrbTgkBpnJrSIYp98cxnDQbwB5N4PZl+TCQ9xEe8+IuS1JVHCHqStA6UMzLsP+o2LJj7/Cgnx0Q5OTzd50dEROR4caGqZJiLGlv44qqHuWrpdty4gzFwefNObl7+Mn97+s9IJR3SMYdLF7/JBQ17+L8rH8LvS4GBlQ0HWFLcx18t+xkn1vRgkjZlZSO8rbKd99e/zB+veAEssP1prl25lXcueYNvnfcA3uLEtJpqeV3q6/rxl8eofsWGwwzKzjixlXPr9/GtM77HyhM6AKisjXDZkl3cfcpjfOztzx5Ttx0Uj/gxiUw70iMeUqPH/ft8Kb558ve5efFLmKSNSdjEI34AksNeYmE/JmXhxh0uaX6TR1c+igW4MYfkkI9Y2E8sfPSDqWzbDvaRgYAvybdO/j6PrnyUkkAcN+6wqq6Dc+v38e9nPEBdeQSTtCkvHuGU8gN8oulp3rX09WNqw4KhnBwT5WTmaPIjIiJyDAJlcX5x2vfZsG85f/r8B7mt+mnOO2U33zn3uwymAvzry5fQmqji3nMe5f1nvcyVZa/zq1dX8Tf7ruMXZ32Tk5Z18u/Lf8z6N07h1pdv5d9P+AknLjvAL97+Lbb0NfDXL17LqkArN575Cl845xdUeEb4r1fO5hcDb+db535vWm21PYZib4IlVf04CYOVtMCGtWdsxSlKcdqJ+6lYNMgJS7u4f9lPePz1k7l14x/znyt+xAlLu/jVmd/hle4m/t8Xb+Ci4u28+4zXZqlXYSgcZPtbzkEoqohSWRvJPiyfC8Dz7UtnrR0HlVYP8623fY/P730PSZNZwT59RRt/2/xzHt96Ch/b8kc8cspDLF3SzaOrHuSZ/cv47Avv44PlL3LxaTtmvX35TjmZHcrJ9GnyIyIicgxs26XCKcKywI072BhKPHFqnCg2BtIW4XSQGk+ESs8wjpUZiIQTAaptH0FPkiLLyVzNKualyPIS9CSpcTKrtyZpk8ShwjvCIs9g5kMNDCSLqLFHjn0DXHiiZQU3rNzCcNJHf1cJfidFyLKxLIjHvBTZmTZV2AEsK3M4UNI4lHtn4PMnYZI2kXTu4VV/ffov+N1ZD2Uf1VWZ/hiJ+WatHQfdfcpjOBjOq9iDjcUHmjdS7IkTsDKHDQ1HfRRbfvxOKrM/kNkfAEo9hzk7vUAoJ7NDOZk+TX5ERESOwUg4yIf3XMZX3vZf1DYM8B8D5/Gb36/kQ6/9MXfU/oYTlx1gMB3g7k3v5RuvXoTXSnHdma9SExzixh1/wJaWJv6m+3z+8tz/pmHRAH9x4EK2tjTyBztu4MsrfkxDUx+/G17Od7ZcwF2b38fa0lc55+QWvHaaD79267Tamh72sD9cRs9IEcYiMwqw4MRFPWwZaOS08k6cYJrf763na33n8PnzfkVjzQCfbb+I1/c08N5d7+b/rPwRdY39PD10Mv/52tmz0aVgQU19mDpPmGZvL9V1mfuP3P2793Lyhj/JProPlOENJVm+qAeAldVd+Eqmd4jTkfrLl67nfc/fxoM7z8XF8N+dp/PCrmX8OPI2/uz8x2muHuATrZewY18dH2i5gq+cntkfHo6cxaOvnzYrbZpPlJNZoJwcFV3wQGQW5NuJ3MqK5KN8y8lRn8g96pTl7ZT6YvzuzaXZY/Or6yIsr+hh64H6sftdFCc4q7mN9qEyWlurMj9sw+oVe3CNxaadS2D0L/OSxT3UhSK8sq85e9J2cdUIpy3qZGdfNX0HptdvdsRD1RYLO2UIn2gTa0yCbahr6KdzfwX+0szKazziB8dwzoo9pFw7p01Ll3RTWzTIy3sXz959OUb7w2NnVv8TaSenDYc62B8H7eitob+rZHbaBdl+2dzaRHLYi+V1OffEPQwl/Wzb1ZgtdtKyTioCI7zUsuSYzmVYMBc8GKWczCDlJGs6OdHkR2QW5NugTlmRfJRvOTnWQd28kbCx0hYm4II174YABWehTX7mDeVkXplOTnSTUxERkULicydaGBaRQyknC5bO+RERERERkYKgyY+IiIiIiBQETX5ERERERKQgaPIjIiIiIiIFQZMfEREREREpCJr8iIiIiIhIQdDkR0RERERECsKMT37WrVuHZVk5j7q6uuz7xhjWrVtHQ0MDwWCQyy67jG3bts10M0TymnIiMjXlRGRqyonI9MzKNz+nnXYaHR0d2cdrr72Wfe/LX/4yX/nKV7j//vt56aWXqKur4+qrr2ZwcHA2miKSt5QTkakpJwXGtajY6KXqRS9Fb/rmujXzhnJSYJSTY+KZlUo9npxVh4OMMXzta1/j3nvv5cYbbwTggQceoLa2lh/84Afcdttts9EckbyknIhMTTkpEMbCHnYo3WnT9IHdlPtGeGb7CoI7/USXJsEyc93CvKacFAjlZEbMyjc/O3fupKGhgWXLlvGBD3yA3bt3A9DS0kJnZydr1qzJlvX7/Vx66aU899xzk9YXj8eJRCI5D5H5bqZzAsqKLDzKSWEo2u3F32uz5mPP8fMVv+Z7S55m95pv4+8HX6cHp987103Ma8pJYVBOZsaMT37OO+88vve97/E///M/fPOb36Szs5MLL7yQ3t5eOjs7Aaitrc35mdra2ux7E7nvvvsoKyvLPpqbm2e62SLH1WzkBJQVWViUk8JRvTXFjz76j/x97ebsa/3pEd73yd+ABb4BC3toVg5WmfeUk8KhnMyMGZ/8rF27lve+972cfvrpXHXVVfzyl78EMl+zHmRZVs7PGGPGvXaoe+65h3A4nH20trbOdLNFjqvZyAkoK7KwKCeFwen30rPKQ43j8usRP78cCZA0ab4xcAYfLNvI9j/5Og99/Cs0/cad66bmJeWkMCgnM2fWL3UdCoU4/fTT2blzZ/Z41LeuNnR1dY1blTiU3++ntLQ05yGykMxETkBZkYVNOVlAjIXT78XT48UXtvjeJ75Ge8pDb7qYU709eC2HT1ZsodJxAAhYaeKlNiRtrLgDKd2pYzLKyQKinMyKWe+VeDzOG2+8QX19PcuWLaOuro7169dn308kEmzYsIELL7xwtpsikreUE5GpKScLR+hNL04CjAOPfeLLrPb7ONPv50MlvSzzFgNQZgcps4MAnOQNMXDDMCU7PZTscCh73QPm8N9cFCrlZOFQTmbHjE9+Pve5z7FhwwZaWlp48cUX+cM//EMikQi33HILlmVx55138sUvfpGHH36YrVu3cuutt1JUVMTNN988000RyVvKicjUlJOFydPnJboqyqZb/i+7PvR1mjzFR/RzPzn3G8SqDX9z+3fxRQzWiANJrWwrJwuTcjJ7ZvysqLa2Nj74wQ/S09NDTU0N559/Pi+88AJLliwB4O677yYajfLpT3+a/v5+zjvvPB577DFKSkpmuikieUs5EZmacrLwhHb6SBYbdl7xHWB69yc5zRdkx61fB+Bz7xym5IVirLRN+G2FfYlf5WThUU5ml2WMmXc9EYlEKCsr4zJuwGPpsn6Sf1ImyVP8jHA4PKfHSSsrks/yLSdN96/DDgbmrB0Lna/DS7LM5WtrH+T60Mgx1RU3STbFbf7sC7fTdUlqQQ/q3GiMttvXKScFQjk5OtPJib4HExERkVmXChmK9juc5us65rr8lpeH+s8jXqHzGWRhUU5mnyY/IiIiMuvc0hQjDS7X/+vdPBtz+fveFfSnRxhyY/Snc1e4dySH+Ua4YdK63kiM8NR3zyWy3F3Qq9lSeJST2afJj4iIiBwXJpQiXu1y27/dzgWhnfxz/1l8unUNn9p3LSNugqRJ81fdp3LbjpsJWInsz/3XUBlpM3b/klN8RfzBx56ibKetq1nJgqOczC7dBlZERESOm3RZiuBWL1/a+y6qA0PcU/9rHgqfw4d3v5uO4VIqAlE+tvgZHtx/Pv+yO0iJP869S3+JY42t1w65MboTJQyclQDd01EWIOVk9mjyIyIiIsdVz4Up0j9cQlcS3n3piWy/8pt0VLzIr4dPYn+igjeiDXz1hB/xxMhKTva3845Akj9tP58VwS7eGKnnqZ+dRajdYK8yuKWpud4ckVmhnMwOTX5ERETk+LIM/WcnwVgEdgW44uFPU/KnrQScJC0/XIGTMDxcfTFvv+51nmc5d/70VLxDhhdikApCfKVLdKkGc7LAKSezQpMfERERmRuWIbY4QVuTRfmPlmK5hoG3J8ExeHq9vPHdUwCInuQyVK5BnBQo5WRGafIjIiIic8s2mfMSDpGqStJXNUftEclHysmM0NXeRERERESkIGjyIyIiIiIiBUGTHxERERERKQia/IiIiIiISEHQ5EdERERERAqCJj8iIiIiIlIQNPkREREREZGCoMmPiIiIiIgUBE1+RERERESkIGjyIyIiIiIiBUGTHxERERERKQia/IiIiIiISEHQ5EdERERERAqCJj8iIiIiIlIQpj35efrpp7nuuutoaGjAsiweeeSRnPeNMaxbt46GhgaCwSCXXXYZ27ZtyykTj8e54447qK6uJhQKcf3119PW1nZMGyKSTwboAWDlypXKicgklBORqcV37QGUE5GZMu3Jz/DwMGeccQb333//hO9/+ctf5itf+Qr3338/L730EnV1dVx99dUMDg5my9x55508/PDDPPTQQzzzzDMMDQ1x7bXXkk6nj35LRPJImsy+/A//8A8Tvq+ciCgnIkfCjScB5URkpnim+wNr165l7dq1E75njOFrX/sa9957LzfeeCMADzzwALW1tfzgBz/gtttuIxwO8+1vf5sHH3yQq666CoDvf//7NDc38/jjj3PNNdeMqzcejxOPx7PPI5HIdJstclxVUQvA9ddfP+692coJKCsyvygnIlMLnrYCUE5EZsqMnvPT0tJCZ2cna9asyb7m9/u59NJLee655wDYuHEjyWQyp0xDQwOrVq3Klnmr++67j7Kysuyjubl5JpstclzNVk5AWZGFQzkRmZpyIjJ9Mzr56ezsBKC2tjbn9dra2ux7nZ2d+Hw+KioqJi3zVvfccw/hcDj7aG1tnclmixxXs5UTUFZk4VBORKamnIhM37QPezsSlmXlPDfGjHvtrQ5Xxu/34/f7Z6x9IvlgpnMCyoosPMqJyNSUE5EjN6Pf/NTV1QGMW0no6urKrkrU1dWRSCTo7++ftIzIQqaciExNORGZmnIiMn0zOvlZtmwZdXV1rF+/PvtaIpFgw4YNXHjhhQCsXr0ar9ebU6ajo4OtW7dmy4gsZMqJyNSUE5GpKSci0zftw96GhobYtWtX9nlLSwubN2+msrKSxYsXc+edd/LFL36RFStWsGLFCr74xS9SVFTEzTffDEBZWRkf/ehH+exnP0tVVRWVlZV87nOf4/TTT89ehURkvkuRAmDLli2AciIyEeVEZGru6BXXlBORmTHtyc/LL7/M5Zdfnn1+1113AXDLLbfw3e9+l7vvvptoNMqnP/1p+vv7Oe+883jssccoKSnJ/sxXv/pVPB4P73//+4lGo1x55ZV897vfxXGcGdgkkbk3yAAAF198MaCciExEORGZWmJvO6CciMwUyxhj5roR0xWJRCgrK+MybsBjeee6OSLjpEySp/gZ4XCY0tLSOWuHsiL5LN9y0nT/OuxgYM7aITIRNxqj7fZ1yonIYUwnJzN6zo+IiIiIiEi+0uRHREREREQKgiY/IiIiIiJSEDT5ERERERGRgqDJj4iIiIiIFARNfkREREREpCBo8iMiIiIiIgVBkx8RERERESkImvyIiIiIiEhB0ORHREREREQKgiY/IiIiIiJSEDT5ERERERGRgqDJj4iIiIiIFARNfkREREREpCBo8iMiIiIiIgVBkx8RERERESkImvyIiIiIiEhB0ORHREREREQKgiY/IiIiIiJSEDT5ERERERGRgqDJj4iIiIiIFARNfkREREREpCBMe/Lz9NNPc91119HQ0IBlWTzyyCM57996661YlpXzOP/883PKxONx7rjjDqqrqwmFQlx//fW0tbUd04aI5JMBegBYuXKlciIyCeVEZGrxXXsA5URkpkx78jM8PMwZZ5zB/fffP2mZd77znXR0dGQfjz76aM77d955Jw8//DAPPfQQzzzzDENDQ1x77bWk0+npb4FIHkqT2Zf/4R/+YdIyyokUOuVEZGpuPAkoJyIzxTPdH1i7di1r1649bBm/309dXd2E74XDYb797W/z4IMPctVVVwHw/e9/n+bmZh5//HGuueaacT8Tj8eJx+PZ55FIZLrNFjmuqqgF4Prrr5+0zEznBJQVmV+UE5GpBU9bASgnIjNlVs75eeqpp1i0aBEnnXQSH//4x+nq6sq+t3HjRpLJJGvWrMm+1tDQwKpVq3juuecmrO++++6jrKws+2hubp6NZoscVzOdE1BWZOFRTkSmppyIHLkZn/ysXbuW//iP/+CJJ57gH//xH3nppZe44oorsqsHnZ2d+Hw+Kioqcn6utraWzs7OCeu85557CIfD2Udra+tMN1vkuJqNnICyIguLciIyNeVEZHqmfdjbVG666absv1etWsXZZ5/NkiVL+OUvf8mNN9446c8ZY7Asa8L3/H4/fr9/ppsqMmdmIyegrMjCopyITE05EZmeWb/UdX19PUuWLGHnzp0A1NXVkUgk6O/vzynX1dVFbW3tbDdHJC8pJyJTU05EpqaciBzerE9+ent7aW1tpb6+HoDVq1fj9XpZv359tkxHRwdbt27lwgsvnO3miOQl5URkasqJyNSUE5HDm/Zhb0NDQ+zatSv7vKWlhc2bN1NZWUllZSXr1q3jve99L/X19ezZs4e/+Iu/oLq6mj/4gz8AoKysjI9+9KN89rOfpaqqisrKSj73uc9x+umnZ69CIjLfpUgBsGXLFkA5EZmIciIyNXf03B3lRGRmTHvy8/LLL3P55Zdnn991110A3HLLLXz961/ntdde43vf+x4DAwPU19dz+eWX88Mf/pCSkpLsz3z1q1/F4/Hw/ve/n2g0ypVXXsl3v/tdHMeZgU0SmXuDDABw8cUXA8qJyESUE5GpJfa2A8qJyEyxjDFmrhsxXZFIhLKyMi7jBjyWd66bIzJOyiR5ip8RDocpLS2ds3YoK5LP8i0nTfevww4G5qwdIhNxozHabl+nnIgcxnRyMuvn/IiIiIiIiOQDTX5ERERERKQgaPIjIiIiIiIFQZMfEREREREpCJr8iIiIiIhIQdDkR0RERERECsK07/MjIiIi02NFHey4lX2eLnLB585hi0Tyj3Iix4MmPyIiIrOkaLeP4jZDsDeFN5zMvh5b5CdR7KX/ZEguSh6mBpGFTzmR40mTHxERkRlmjTgs/XmKVHGKWLlDeJmXVHDsRsPeQYNvyND8eJJYlYf2y41WuKXgKCcyF3TOj4iIyAyyBz00P2boO9VP/woP0RqLVDC3TLLYonz7ML7eERIlNg1PWpC2Jq5QZAFSTmSuaPIjIiIyQ6xhD01PuISXeUj7D1POBWvbm5htO0mUWMQqbeqfcLDizvFrrMgcUU5kLmnyIyIiMgPsIQ/N60cHdIFJChmoeTVBoNfkvJwotUiUWNQ/hVa2ZUFTTmSuafIjIiJyjOwhD02PH8FKtgH/s28Q7EszuPZ0omvPwh09xSFeYRGtsmn4jVa2ZWFSTiQfaPIjIiJyDHIGdJOtZL+FsSF8gkPfyR44ZAE7WWIRL9XKtiw8yonkC01+REREjpIdGR3QnXBkAzpjw4GPvI3wsskvtpqzsh3TyrbMf8qJ5BNNfkRERI7CkZ60/VbJEit7CM/hysRLLeo3AK5WtmX+Uk4k32jyIyIiMk12JHPS9sDyIz+EZ7riFRbRapuG9bZWtmVeUk4kH2nyIyIiMg2HrmS7vtn9rGSxRbzM1sq2zDvKieQrTX5ERESOkDMwupJ94uytZL/VwZXtxse0si3zg3Ii+UyTHxERkSNgDXtofMolstQz+bkIBirfSIGZ5P0p2Emo2Dn+55PFFrFym4an0NWtZpGv08viR2wWP2ITaDvM1xXGovq5TNmGxzyH/Z04YU+2zuIdPjAL+/ennCx88z0nmvyIiIhM4dCV7FRw8nKLNsUJPLqRqm3JKet0PZD2jz58YByo2J6k6JebKd7vjisfr7AYqbFp1LkNM84acSje4cOJWjTes5PGe3YC4N8/fmDn3++jfJOXwXcO0XjPTgKfaKfqZc/430nSJrTTR6jVztYZrXUpapniLP55TDlZ2BZKTjT5EREROYwjXcm2XLBTLrhp7PQUS9oWJEsNyWJD4pQosbo0w4tTuD4Lk0pijR/TAaMr2xVa2Z5RrkXVZpvhRpe1N77AyuIDrCw+wJXv3kjtSynKX/FBMjNcCu7xYScgfkWED67cyMriA1xSs4vICVDc4lC6dWzFuvolh2idy9kf2JKt80NX/ZbqLSlKt/qwhya/jPN8pJwscAsoJ9Oa/Nx3332cc845lJSUsGjRIt7znvewffv2nDLGGNatW0dDQwPBYJDLLruMbdu25ZSJx+PccccdVFdXEwqFuP7662lrazv2rRHJAy3m92xkAwDLly9XTkQmMF9y4vR7MyvZKw6/kl2zOU7zT9qIVfpwSkuJlx1+xTlRarCXD5FuiLOsrgfvoigVTWEG/niQ9GVvJxWcfMAWL9fK9owy4Bt0ueWypyn1xAA4kCgl7npovzlB+LwYlZsd7EEPgV5D7KQYN614Jfvjr4YbuW7NiyTKDJGTUxRvz4z8Q50p/uiKp1kc7AMgkgrw5nAN0U/1Ezk5RcmbNp6+qVe3I//zWwAaGxvzduylnBSAPM/JdExr8rNhwwY+85nP8MILL7B+/XpSqRRr1qxheHg4W+bLX/4yX/nKV7j//vt56aWXqKur4+qrr2ZwcDBb5s477+Thhx/moYce4plnnmFoaIhrr72WdDo9c1smMkcG6KaBZQA88sgjyonIBOZDTqxhD40b0pmV7CkWH+20gXiCgRMdet9zGpGlUw+2ElEvRcXxzGdZhv62MgZ7Q+y72k+05vCr1VrZniGuRflrXjquTTLi+nhg0wW81L+EpOuwYfeJvO+UV7jlzBfoP92ldGdmyHTmslae6VnOA5vPx8WiLxbi2QMn8MEbNnDrBc9QuT1F8e997LvGIWkcHtx2Lr/pXEkam9/tW8y5i/Zx6wXPkLg0QvkbUzcxvmsPAI8//nhejr2UkwIwD3IyHZYx5ihPN4Pu7m4WLVrEhg0buOSSSzDG0NDQwJ133sn/+l//C8isNNTW1vL3f//33HbbbYTDYWpqanjwwQe56aabAGhvb6e5uZlHH32Ua665ZtznxONx4vF49nkkEqG5uZnLuAGPtXCPnZX5K2WSPMXPCIfDxOPx45KTg/UoKzJf5FtOmu5fhx3MXJrK6ffSuCHNwIqpB3QAZbvTFHXG6biw6Mg23oLoIhfvkmEcx8V1LRJxL86eAL6BIx+keYYhdCBN++VgAloYOVLWiEPJmw52CsLnxvjImS/y3/tO47y6fVR6h/FaaeKuB6+dxsbwn9tXU/LrYs751CZCTpzftJ3ElU07KHbiuFgkXQe/nQLgJ9+7jKGlaW6++DmePnAii0v6WRzsw2+ncup88sBJDP1XPb1np/F1OyRq0uAZfxyXG43Rdvs6wuEwpaWlcz72Uk4Kx3zOyeEc0zk/4XAYgMrKSgBaWlro7OxkzZo12TJ+v59LL72U5557DoCNGzeSTCZzyjQ0NLBq1apsmbe67777KCsryz6am5uPpdkix9XxygkoKzJ/5VNOprOSnW3/Cc6RD+gge2iQ+/tiYm+WktpRgrMnAAbMNP4yp0IQrRpd2U7pNN4jkrKp2GoTOTXJyCVD3HLmC9gYrl+8lVpfBK+VGRz77RT26OXEPrhyI9f+2QZqfRGKnTg3LHmNYiczMbAx2QEdwHv/6CluueS3eK00V9ZtZ0WoK/v+oXVeXruDRKlFqMWDZ8Si5oUjOzQrX8ZeyskCN89zcjhHvQcYY7jrrru46KKLWLVqFQCdnZ0A1NbW5pStra3NvtfZ2YnP56OiomLSMm91zz33EA6Hs4/W1tajbbbIcXU8cwLKisxP+ZQTT5+XxY+lpzx34VikQhBbksj8u9jgHbRwfQbXA4maFNG6Sc7inkSiLHNuQ9Njls5tOALBNg/L/2gHt57/LDef/PKctuXqD71AvMLwzj98gVBnKnPZ4MMMzvNl7KWcLHzzOSdTOepLKNx+++1s2bKFZ555Ztx7lpX7VaQxZtxrb3W4Mn6/H7/ff7RNFZkzn/vc545bTkBZkfkpX3JiDXto/G2K8DLvEa9kH41U0HDSkk52WLV4DviwDFgpC+M1NC7ppbO3DNqnN6JMFluMpG0annLZf5U94WEhBS9p0/i4xf41SU4r6Zjr1gBQ6onx4bUbcLHYt8bB3ws1zzt0Xzzx7y8fxl7KyQK3AHIylaOaNt1xxx38/Oc/58knn6SpqSn7el1dHcC4VYSurq7sikRdXR2JRIL+/v5Jy4gsFL/61a+UE5Ep5EtOmp5I07fSO2sr2Qc5cdi5fxGBvX68QxZWCryDFr4Bm/0t1bi9h7lp4GFoZfvwGp6w8Nx2gA+e++JcN2UcG8MHr3iWNTf+jtCB1IRl/vzP/zwvxl7KycI233NyZPVMgzGG22+/nZ/+9Kc88cQTLFu2LOf9ZcuWUVdXx/r167OvJRIJNmzYwIUXXgjA6tWr8Xq9OWU6OjrYunVrtozIfGaMYQdbAPjFL36hnIhMIB9z0neKl2Rx5jyDVDBzU0U7lfn/VBDSgcx5BgefH/rAyrx38Lnrzdyc8a3lUkHw91v4dwdIBQxpv8EygA2pkMHb7xDsdEgFMzdzhEPaM3qDx4nqtNKZz4zWWESWONQ+a4ENls/NeWADSRvLO/qax4CxxpWzfG7mnh3WWB3AhHVaHgOJQ+r0uuBaY88PrTNl53weFjmfkX14Xay4g+UxY2WNlfM8+7prQdrK3UbG19l5gUX/LxsAcCwXx3KJpAIkjZN9fvAxlPbTHi/Llk0ah/5UUc7PHny0xSoYcX05dcZdz7hycddDW6wiW4eLlVNnwE7yiw1ns/8ST/aQHsvngjedzUk+jL2UE+UkH3OS7fsjMK0vLD/zmc/wgx/8gJ/97GeUlJRkVxnKysoIBoNYlsWdd97JF7/4RVasWMGKFSv44he/SFFRETfffHO27Ec/+lE++9nPUlVVRWVlJZ/73Oc4/fTTueqqq6bTHJG8tJ1NHCBzbHRxcbFyIjKBfMxJzRX78YTGDvPpGiwm9FApnj85gNdJ0zdcRPTVChad14nXyb1iVN9Pmxg4I8kJyw8AsG9jI9aSYZqrB3LKJdMO1v017LvO4EQczNIR3K3FlF7QRff2aqw0jJyY5ITFXbT/tom039B87n4AWrbXYwJpTljSNa7t6a/WEvtMPyFf5hyJzt80UVffT/Ho84Pe/H0DxW861LyrE9syvLm/Bitqs2zZAWwr9+Kv0a83MPjhYRaVDAHQtqGZ4nN7KA9Gc8rt7a6g/EkfRR/qwuuk6QiXkmgvpfHsLjx27oBk4D+a6Ls8xgkNPZl+eqEJz8kRGsojOeUG4368P6vCfKKfoDfJUMLHwAu11FzYgd+Tu+Lb+etmRhpdTjgxsw/t3tII1fHsZxzkNlikH6/l6QMnEvImcLHY/2QzlRd1EvK+pZ9eacYXsWi8rBUbw46WOkjZnLSifVzfR/+pgdjH+6kIZPql9anFlF14gFJfPKfcztZayl/0sfu9+/FYLnt6Kkm3FbHszP3Zk7uXvq0d2zIMfbuR/vdEaa4a4I1/eAyAb33rW3kx9lJOlJNsP+VRTlLDcY70LOdpTX6+/vWvA3DZZZflvP6d73yHW2+9FYC7776baDTKpz/9afr7+znvvPN47LHHKCkpyZb/6le/isfj4f3vfz/RaJQrr7yS7373uziOvn6U+a+N3dl/n3TSSdl/KyciY/IxJ62v1WMHAtnnTtQiWm0R6SrHOuDHO2jjG4TW9kqc3txLxzfsT2G9M8LezZkV06anU4x8Opp9fpDlWtQUGYqqBolHSjB7QpTsNQy8LcjoxZNw+jzs7Wugco+h58o4e7Y0YLlQ+wr0Xpdk76sN8JabVDS5aRzbzX5e42spRt7h0LKpMadcSavN8GKXlu312FGLst024VPTtLxej53IPfdjSX8Sb2gkW8fi5+KYi+Pj6vSFbUbqoLu1Gqffg7/XxuOBPS2LcCK5v4emjhSe6shYnRuSpM4eGVennbIoK4PwQDFuWxFO3KK409DeUw4dueehLNrnkjxnmJbNjWCg4TlD+I9i4+oEaPC5jCS8dG5dBMai7vdpImf7M88PUbYL+t6e4s2tjdhJqH7doufiBLu2NGV/TwctHkpS5E3y5qbMoWjNv0uQfIfJPj+oqDvTT/276nCGbIrabaK1hl3b63FGcg/EWdyVxF8+SMumRjp+lblZ6bvf/e6cMnOVFeVEOTkon3LixmLjtmMy05r8HMktgSzLYt26daxbt27SMoFAgH/6p3/in/7pn6bz8SLzwlXWH+bcv2Si680rJ1Lo5lNOvLuDpEJTH1IxHPUT7LSP6CpUsX0l2Edw00WTsinZZzPcNHWd7e2V+GIW6cAUf6sNFO92GKk/gjr7yvCFbRJlU5cN7vaRqJi63MBQEYFum1jNERymsicERzA2TyY8FLfZDDdOXWf//jK8SWvqk/VdKNlzZHW2dVfgGbaPaD8JtXiOaNsPhEvw99ks+bu/Ze+9//uw9y/Jh6woJ8rJVGYzJ9Ejv4L60V/tTUREZKELdaYpaTN4oi6ux2LvjZMPmJzXiml4epBkuZ9YhYfK4Ah9VI0rF+hPUb3JS/H+BK7Hou1K76TjltB2H/XPDpKo8JMocairjNA+wRWuvEMpyjYFqd4aw/XaHFjtY7KxQEmLTc3mEVIhD6mgjbtmkMF94wfVdsrFs7mYht+OkC7yEF7uo9yaePuL21yCvWks1+B6LVo/mIKR8UMMy4D1agkNTw+TKvEyUuOh/C2H0mS3vSuFd8Qh0JvEeGz2XmtPeqJy4PUg9c9k+ile5lBbMkQLZePK+cIpKjf5Kd+VwPVadFzoY7LT5kt2eah7fohkqY9kyKG2Nkz3QPW4ck7cJbgpSN1LUdJ+h95TfZRMUB9A8T5D+e44rtcm7bdJnB8l1TP+92m5YG0uofGpYWKBNHsnqS9fKCfKyXzKiSY/IiIikxiuc1h20056osV09pVS8nKIwVXxCctWX9xB9GKAIZI/rKNvkqXIWIWHFZ98g32DFUSTXip+XEVk6cSfHztjhNiVg7hmiPBv6on3TzxcSBZ7OO8jm3ijP3Plr/JvLSLxjonrjCx3qb6xl1TKQ/uWOtz9E3+b4HpsLnzPq2y/LHOoS/Abi3DNxKvwQ0021R/fSyQeoLW9ksCOILGm5LhyxoITr97NwMWjA5nv1TKSnHhYNVLjYenHdtA5XEr/SJDyx8qIXDTxNjnn9jNyaQwYYuTn9QwOFk9YLlHq4cw/eY2d4RrSxqLigUUMnzFxnYMnJ6laGybt2vQ8X0/sQNmEg8q03+bim15h69X1AJR8Y9EEpTKGFltUf+gAI0kfrW/W4GkphpL0uHLGhjPf9QZtl5STGo7DbyatMi8oJ8rJfMqJJj8iIiKH8erepuy5DG89h+BQQ3EfkR2ZqxQ1HUgxcpg6N3U0Em8pwUpZVI7/m57lpmxat9ZhuVDT5hJ/++SHAG3tq+fA6DH5jVGXideJM6vKb/6+ATtqUdJiET518s/fFamm/bXMpZAXh5OH23y2vdmI0++hqNfGHGZ00T0SontbTabO7hSTXbDWWLC5rTF7LsPhLuaUSHhof7MWDNR3pAlPVtCCTV2Nmd+TsahLHqZS12Lv6/XYSajYZ4gtmbzo6/112X5qHkoxfjg7ZvuORpwhm5LRcxkm0z5cRvtrtdM6l2EuKSfKyXzJydHfHlVERGQB8fXn/kks2Qfhk9O4w2MnbS/aFKPqmdwVWCdqMVzrMDh0yGEZBlK/qKaoI7fO4r0W3W+3GBkYK1u+c5jix4qx3LEBm5WyMA5Y9uglfg+W/Z8iSlpy6wztt+la7aWze+zwleCBKO4vqnDiuYNA36AhXZGC1Njr9RugYltunf5em+4z/LR1V2Rf8wynGHikEW8kt2xRh2F4WSrnnim1LyWp+l3uyM47ZBNZ6qF3YGy12UobBn9WT6D7Lf20B/pXGZKRsZO2K7dFKX8ikBntjbKTFokSi1Qq94Ao33+XE2p7S537bLrOchjoG/v84r0j+H5Zjn1If1iuhZ0EK5jCOmTMV/u4l7Iduf0ZPGDTtTq3n3z9cYZ/Xodn5C1luw3x+iRW8pC+fy5N1aa3tH3Apu8UP+294w9HygfKyRjlZH7mRJMfERERwDcI1ZszAxp/r40/bKha1o8zNPanMrzUT/jKKE1PpAm1ZcoFuy0Gl0EqkjvYG7wkStoHtb8bqzN0IE3pab04/WMDnpH6IOnr+ql/Lk3J7ky5QK9FrNrCpK2cVfSec1xG6g31z5psnUWdLpwZwTowNgBKlPnwvaeLyq2G8t9b2bLGAX9xHPuQJde+0xz6V7k0PuUS6D5Yp2HobXHcA2NX9XL9Dv53dxHqMFS+Nland9hQXh/Bjo4NVgaWe+m7OE7TE2mKOka3qQvCJxlSfWN1GgsSV4axU1CzcayfinpcKlb25VwFa6g5wMjaQRo3pCneO1pnt8Vwg0V6xJPTT70XJkmUQd3zY/0U6kjjf9sAds/YID1WHcC6oZdFL7uU7Ty4TRbJkIXtdXOuWNV7hkVkuaFxg0ugZ6yf4meMQOdY36dDXgLXHaD0Tah4fayfrDQU1wxjHzLQ7l/poffsFE1PpAkeGN2fugzh01Kkumb5LqJHSTlRTuZ7TjT5ERERAaIXDNG1JoHrhfJdaTqvj9O3ozKnTLzCwhiL2O39jNQbXC9UbE9QdWZXzuAPoKJsmPiqKD3vHcH1QrDL0PEOi/69FTnlEsU28aSH6McGiJzk4nqh7oUo1Vfvx+7OHSh6KmMkGxOEb4ngesEThcElNtGBQE65dMAhlXYYuSlM79lpXC/UPx/D/94DpPaFcrepOo0pTjHyyQFSh5x+4fGncgY1rsci6E0ycs0g3Zcmcb1Q/VqKgRuHGdxVnltnJWBB7PZ+YtWZfirZn6JseX/O4A/boqp4hOjqEbqvi2XKtbrsv9ql983cvk+UWqRSNtFP9TO0NNNPizbFqL2oHWcgd/W8pHKY+PIY/TcP4XrBP2DoeZtNpDP3XJBUyCaecoje0k//6Zk665+LUn5dO7Tn9mmqJkG6Ksngx8OkA5kbeybKLNKp3MO80j4bxzJE3xOm54IUrhdqNyZIvbeP6Ju5543EqgyW32XkMwMkykxm0B02FNUM5wy884lyopzM95zonB8RERHAu7kYXyhAKmhIFNs4jouvy84e0tH4mzBtV5VhtQUYfjWIpwhcryFW6SE8HCTYlRnU+QYMoW2ddAxVEHw1iOtAqsiQDlq4wTSBDg+e0fsf1v4uRmSpn2hXEc6eUnzB0TqrfcTi/mydTgwqn23lwJpFFL3hx7X8uCFD2m+RLDZ4+rwEejKDpaptSXCht7eY0LYAQT+kfYZ4pZdEypOtExfqH+sgfPIigrv8RH/vh1DmKlSJMotk2E/xaNnSPS6+/hgtfWX4NhYT9EI6YIiX2di2S6DLzh521PTrPvbcWIm31c/w1kXYITAeQ6zcITI01k+BbkNwVw+7IyUENhdl+ykZsrD8LsG9XuzREx3qfztM9+oQ6Y4ioq8U4z3YT1VeBkfG6vQMQelL++j4w2pCWwO4TgC3yJAKWqRCLr4eB18k0081mxMkSh0Gu4sp3unFHxjr+3jSm63TSsGiJ9roOa+Woj1eEq4ft9iQ9o1eBrjfly1bvjONZyRNW18pgc1FBH2Q9hvi5ZnhVrbvgaZfdbPz1mr8e/xEX6/BKs4MfuPlFiORQLbvAy3k1dXelBPlZL7nxDJHcvOePBOJRCgrK+MybsBjeaf+AZHjbKr7lxwvyorks3zLyQn/+++yN29M1KRZfdpuRm6rZN/11SRDBqzMQMaJW3DIMe4Vq7uJPLeIqtfTdL0984fYeDNXIrIPueBVstRw9QWvsu8Ty2i/vJxYZebPbzposJMW1iFnNDunDBLrDLHs4RT7L/VhLDAecD2ZAd5Brh8uu3QLLZ9fSe+pAUbqRusMgJ0G65BV0dTSGL5AiiVfhr1rS3C9mRVU12dwDlllNg687eKdtP/ziSRCFoNLRz/LBxhyVloTdSlOOqED509D7L2+ilTRIf0Uyz0UqeH8dvp+2Uio06X3tMznuV7grf1U6XLp6tc58NEGWtdWkSg7pJ8SVs4qe+D0AaK/L6fh6RSd53sy/eQF18ntp3SRYc0lm3nzjpM4cG4x0ZpD+ilFTt+bE0dIxT2c+M00rVcX4TqT9JMXzr90G3v/7mQGmxyGG8d+J7hv6aemBDWLIlT9hYe911WQ9huwMwO+nH6yYPk79tL1H0twEjCwAtxYjN1/e69yopwoJzOUE01+RGZBvg3qlBXJR/mWk6av/TV28JA71w/blO2wGLxshGTYjyfiEOixiJ45Qvot5y00/MZi4ANDjHRnDpWpe9om/oF++tvfcjKuCw1PWgx+KMJg5+hVrDbbxN4VYbgr9zCbylcc+s5NQtoCY1G5ySF8aXTcZwMs+Zkh/qd9dO7N3C+l/kmb1B/10t2ae+hQaI+HRJkhuSgJSZtQi4eR5jQmkIZU7uFISx92Sf4/fezfk7lnR9NjFtZtXbS21OSU8/Y7BLsshlZHcYe8+Hod7IRFfHkM85Z7mDT/CoY+Gs72S8NvLNw/6cm2+yArYVHzks3gDYNEe4qwYzYlb9pELxoi0Z97mM2iZx163xkjPZj5b1zNCw7D10Wyv4u3fn704wP0tJWDgdpnbeLvG2CgI3f/K33Dw+DyNCboQtqi9A0PkbclMifAv+Uyxksfdknf1Zvtl8bHLfhYd7bfDgp0eLBSEDsphol6CLR7SIUMqZokJHL7/tDfpxuN0XbnXyonyolyMkM50WFvIiIiZAZxdnrsj6sTtXA9FqmkgzNk4xnOrDqn48648xachIvPkyY++ro/nCbtuOPKWa6FsQ3ptI0zZGOlLJw4pFL2+DrjYPlcrB4flpu5ApVtGRie6FLCmWXeg3V4h12SxhpfZxTcGoMVdbBjFt7B0XaNZAZih7JTabxOeqzOSBJjj98mz4hF2gdubLSfhiyMB0xsfD/ZiTR+byr7uj+cIjVBnXYqc6jMwb534hZOwpBOTdT3Bo83BUOZk6n9EZfYBH0/+gsg7Y72i7HwxFxGJugnT8xgfAZ72MFKZfoe22CPeHJW1AGsdBrbMmP9NJi5hO+4dsYgVQQm6sEZsvEOQSoExGyckbdsfzKJf7TvrVh+nZ6tnCgn2W2apznJr0SJiIjkEU/U0PBzL6HWzNWFEuWTHywR3ltG8/oUoTYbY1v4nIlvTOIkDIH/KaVkt02o3SJeMfk9SewuH8t+kSDUZmMnDf7AxGf32kmXyNO1VLxuEWqziVU4E5YD8A7aLPtppp2+QYMJTXIHEQNtGxtY9DKE2mySJZOvl3qHYfEjNqE2m2CPIVE6cT9ZBg7sqKFxg0uozc6c9GxPfA8RT8xQ8WgRxXttiqbop2R7iCW/SmYu22vA75m47+2EIf1EFWU7LUJtFrHyyYdB3n6HZY9k+t4TNXgDE/eT5ULnC/VUbcn0faLs8P209OHMtvv7DakJbtx4UOvWOuqeNxTtz/+hmnKinMynnOibHxERkUk4CcP+qwzOkIsTs0gHJx/UmYBL69UewMUTdSidZFBnJw19FyRwer1YaQvv0OSDFYzFnmt9YFwwDj5PiugExSwX4qePEK8KkFnutpn4fvCZ8wb2XO/FSrmkgjYef5r04PjhgGUMLI7S5QmMbpNNuTXx9jtxw753gTPi4h2ySU82UAQoS7L/Ui/g4h12qJhkUOckDJ1XpnAGPNgpa9xK8qGMY9i7NlOnnXQo8kz8+XbKMLQ6mrncsYFgt8VkvW8noeV6H5br4npsfP7J+t6QPjFKb1Gm7+2kzfgDiUa3KQZ7r3Ww4y7JYhsCLkywYm0ZYFGczgv8uLHD3GAyTygnysl8ykn+LyeIiIjMkXiZjbc8hhs0GJvDri5W1odxgwY3aAh1phlOTDysSoZsKqqGMmX9hpI9hxko1sVIB13coME/YHJvEHmItN+muaY/+/nlb04+qIpXu7ihNG7Q4IlCus8/YTljW5xQ25Ots3RPEtdMPASKVVrYpUncoMkMljomXls1FjTUjbWzeH+KeGrisvFSm6KKaKasA8X7Jt0k/HUj2TqLul0GYxNvU6rIZlF1JFM24FK2e/KRYnxRGrco0/e+QcNwODBhOdeT209luye/9m6sCkxxCjdosFPg6Z74XExjwdK63tF25v+p2cqJcjKfcqJvfkRERCaQOe8AUgkPdtTCiVuZy/mmJxjYWOC6dvbeHHbKJZlyMn+dD10FzixME09m6rRSFpZrcNMTDxaNsbBjmcsIO0mD605wHoOxwIZo0pv9fMt1SaTGH9JjmdEmxW3smIWdyGzPuIVqY2Eci3jac8g2GWLpCQ4TMqPbH3dwopk6jQ24E/dTKqefRuucqJ9sSB7s+0Rm1d5NWROu2hoD9ujVoOyUydzJ/i11Zn6fFrHEaD8ZC8t1x931Pqe5cRs7OXpFqpSdcyf7bD95bKKpQ/vekJignywz+j9xB3u0nyw3065x2+NYpNKZcsQO841HHlBOlJP5lhN98yMiIgKU7bQIdI/9Wax7IYnvnd1YPWMr0w1PDbBog5eS3Xb2xGd/n03vKR4GuovHKjMQerCMijegqGOszoZn0sQ/0E9s39hNBCs391P5yyClb1p4RkYHOwmLtB9sx+SMdZp+4KFyq0WodazORS8b2j+coHvH2FWTgq2DBH9cRukuC194tKyxCHW6BJYM5tw9/YSHE1RtsihpsbNXaCrbCS032uzbXpst540kSH9/EWU7Mndjz37+piTBi3qww2PrqYsfG2TRs06mn1KZOoMHbDou9HBg39hNGa2UwX6gmvLtmfez/fTbFPaNPaQ7x1bwa57vZ9F6HyW77cxlbwHPsM1wvU0i6s0Z7FZ/v4iK1zPnYBxU97xL/4eHiLSUZ18r2T5A6SPFlL5p4R0cvV9J2sIzAr6KWPb+KQDLfmSo3GJRvHeszqrXYO8HXNp+P9ZPgQMj2P+ZOV/C1z9WtmJHiuAZ/TjDY68t/cUINb+zKWmxsUYnC8V7bfZd46H1zdyrheUL5UQ5gfmdE33zIyIiAvSdnaS4PcjiXyex04a2y/3Eu0pxDlnF3H9FOf4regjvraB+g4W/L0kq5ND23hTWwCGHZljg/2QH+1tr8LX5WPzrJJYLXWf7GR4IYR9yFEnvWZWU3trGru311PzOobgtCbZFy01A2ItzyGDlwK0xEjEPVmcgU6eBvpV+0mk75x4cI4tLabptFxu3nUD5Fg91z2e2afd7vZieopw6d7/Pi6dqmFRXkMW/crGTLkONPuziJHSNHRaTKPNxwqe28/y2Eyne4c32075r/CQP5PbTvneW4Fndz9D+Uhofz1zZKVHuoeOGBHbvWJ3GY1H96T289mYTgT1j/dRxgZ9YT0lOP3W9o4LgHxzgwO5qap+1KTqQxPXZ7Hu/C4O5h8VEPzZAb28xng5/tp96TvcTHfbl9FPklHKqP7aXbW80U/WKQ2lL5vN332RDfyCnn/Z8yMX2xDBdfhb/OoWdMoRP8GF53Jw6Y7VFmX7aeiKlr3upezGJnTK0XO/D7SrJqbPlhiK8yzP3qmn+tcGJpYnWeBk8L47dPfEhSXNNOVFO5ntO9M2PiIgIYA97iJ4cw727l/Z3BEieEMPuHxssFHVk/hr39pRAGmIf6cO9uxfXY1FbE86ucDtxi0BXjMG4H3vAS6IuhXt3Ly03eBmpdyEyVmewK3M4T2tvOVbKYuT6CO7dvSRKHZYu7s5e2tVKW4Q6EqRTNvT6SRence/uZe+fuCRLM5fPPcjfa+MkXPaGK7HiNsOXDOHe3UtksZ/6E7txBsfKFnekwUCqJ5g51ObPehn40yFcLziOmz18xRux8Q4l6RgpxR5xGDkzint3L91nBnCWD+EMjK2lhtoz/TTYXQxpiH+sj9Sf9+E6FmVlI9kBkBO18PVGGYgFsSMeYksTuHf30nq1j+jiJAyMfZNQ1JE5VKeztwwrZRF9Xxj37l7SAZvFDb2ZQ3nIXPo32BkjnnKw+nwkKzJ9v/sDNrEqgzt8SN93W9gpw76BcqyEzfCaTD9FF3lZsqw7ewley7UIdaQwroXb48f4DOm7eun4VJxUwMK4Yzdf9PXbeEbS7BuswIo5jJw3gnt3L/0r/ZSd2I8TPqTv2zOdG+0uAhcSn+kl+tkBjAOBYCJ74rpnoksRzyHlRDmZ7zmZl9/8HLwva4rkBNdwF5l7mavXj+2rc0VZkXyWbzlZ/MN+PI4f8FFZMUjktCGqf+jHSmf++Hr297L7Txaz6GGX0P7RG3/go/v8JCOvlnDCj/sAsKNJEsV+el8uZukv+rLlmooH6fponOoHgvgHMrdq97T20PqBpQR/4aHxjf5s2cjKBH1vhlj+UObnrZSbuerU7lKWPtKfucIUPpr9MVo+Eqfmf3yUtkQydbb3cWBNM6lHi1n2u7HPtxtH6O22WTJaJ67BGRiBs2tY/KsETiwFePDZ0PIHUUK/dagd/XlPzyDhMxbR/1gVS58cq7O4aojBc4eofygx2ibwdPTT9/EmGn6cJtAbAyzAy4HLE6Q3FrHkl5mfdwbjRGuLaX+2nKWPjtVZWzZI30dHqPl+ACeayvbTnluWUvGIobRlrJ/6z0gw8HoJJ45ukx1P43pthraGWPbIWJ1LgsO03Zqm7pEARR3R7O+z/folWP8dZNnmsbKppTG6Wv3ZfrJcgz2cgI4Klvx8GDvlAl5qPSl23zRExRMOVVtH+6krTO8FDQz/upxlz4zV6a8dJjyUHOv70bIHTm2g+UdJvJEE4GAsi31r4zgv+Fky2s+mt583UU6UE+VkpnJimblO01Foa2ujubl5rpshMqXW1laamprm7PN3797N8uXL5+zzRY6EciIyNeVEZGpHkpN5+c1PQ0MDr7/+Oqeeeiqtra2UlpbOdZPmRCQSobm5uaD7APKzH4wxDA4O0tDQMKftqKzMnDC5b98+ysrK5rQtcykf95HjLR/7QDnJL/m4j8yFfOsH5SS/5Nv+MVfyrR+mk5N5OfmxbZvGxkYASktL86LT55L6ICPf+iEf/jjYduYY2LKysrzqm7mSb/vIXMi3PlBO8k++7SNzJZ/6QTnJP/m0f8ylfOqHI81Jfp1FJyIiIiIiMks0+RERERERkYIwbyc/fr+fL3zhC/j9+Xkd/ONBfZChfpic+iZD/aA+OBz1TYb6IUP9MDH1S4b6IWM+98O8vNqbiIiIiIjIdM3bb35ERERERESmQ5MfEREREREpCJr8iIiIiIhIQdDkR0RERERECoImPyIiIiIiUhDm5eTnX/7lX1i2bBmBQIDVq1fz29/+dq6bNKOefvpprrvuOhoaGrAsi0ceeSTnfWMM69ato6GhgWAwyGWXXca2bdtyysTjce644w6qq6sJhUJcf/31tLW1HcetODb33Xcf55xzDiUlJSxatIj3vOc9bN++PadMIfTDsVBOFv7+oZwcO+Vk4e8fysnMWMhZUU4yCiYrZp556KGHjNfrNd/85jfN66+/bv7sz/7MhEIhs3fv3rlu2ox59NFHzb333mt+8pOfGMA8/PDDOe9/6UtfMiUlJeYnP/mJee2118xNN91k6uvrTSQSyZb55Cc/aRobG8369evNK6+8Yi6//HJzxhlnmFQqdZy35uhcc8015jvf+Y7ZunWr2bx5s3n3u99tFi9ebIaGhrJlCqEfjpZyUhj7h3JybJSTwtg/lJNjt9CzopxkFEpW5t3k59xzzzWf/OQnc147+eSTzec///k5atHsemsIXdc1dXV15ktf+lL2tVgsZsrKysy//uu/GmOMGRgYMF6v1zz00EPZMvv37ze2bZtf//rXx63tM6mrq8sAZsOGDcaYwu2HI6WcFOb+oZxMj3JSmPuHcjJ9hZQV5WTMQs3KvDrsLZFIsHHjRtasWZPz+po1a3juuefmqFXHV0tLC52dnTl94Pf7ufTSS7N9sHHjRpLJZE6ZhoYGVq1aNW/7KRwOA1BZWQkUbj8cCeWkcPcP5eTIKSeFu38oJ9NT6Fkp5P1joWZlXk1+enp6SKfT1NbW5rxeW1tLZ2fnHLXq+Dq4nYfrg87OTnw+HxUVFZOWmU+MMdx1111cdNFFrFq1CijMfjhSyklh7h/KyfQoJ4W5fygn01foWSnU/WMhZ8Uz1w04GpZl5Tw3xox7baE7mj6Yr/10++23s2XLFp555plx7xVSP0yXclJY+4dycnSUk8LaP5STo1foWSm0/WMhZ2VeffNTXV2N4zjjZo5dXV3jZqELVV1dHcBh+6Curo5EIkF/f/+kZeaLO+64g5///Oc8+eSTNDU1ZV8vtH6YDuWk8PYP5WT6lJPC2z+Uk6NT6FkpxP1joWdlXk1+fD4fq1evZv369Tmvr1+/ngsvvHCOWnV8LVu2jLq6upw+SCQSbNiwIdsHq1evxuv15pTp6Ohg69at86afjDHcfvvt/PSnP+WJJ55g2bJlOe8XSj8cDeWkcPYP5eToKSeFs38oJ8em0LNSSPtHwWTl+FxXYeYcvNzit7/9bfP666+bO++804RCIbNnz565btqMGRwcNJs2bTKbNm0ygPnKV75iNm3alL2k5Je+9CVTVlZmfvrTn5rXXnvNfPCDH5zwMoNNTU3m8ccfN6+88oq54oor8uoyg1P51Kc+ZcrKysxTTz1lOjo6so+RkZFsmULoh6OlnBTG/qGcHBvlpDD2D+Xk2C30rCgnGYWSlXk3+THGmH/+5382S5YsMT6fz5x11lnZS/AtFE8++aQBxj1uueUWY0zmUoNf+MIXTF1dnfH7/eaSSy4xr732Wk4d0WjU3H777aaystIEg0Fz7bXXmn379s3B1hydibYfMN/5zneyZQqhH46FcrLw9w/l5NgpJwt//1BOZsZCzopyklEoWbGMMWbmv08SERERERHJL/PqnB8REREREZGjpcmPiIiIiIgUBE1+RERERESkIGjyIyIiIiIiBUGTHxERERERKQia/IiIiIiISEHQ5EdERERERAqCJj8iIiIiIlIQNPkREREREZGCoMmPiIiIiIgUBE1+RERERESkIPz/gS9xfvFbl0cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1000x800 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# represents each one of our diff stacked frames, display them\n",
    "# note: num of frames is impacted based on our steps in previous code box\n",
    "# For [5] jump action, keep looping and running code box with env.step until see mario jump\n",
    "# also our RL agent will have some kind of memory, it will be able to see when playing as mario\n",
    "plt.figure(figsize=(10,8))\n",
    "for idx in range(state.shape[3]):\n",
    "    plt.subplot(1,4,idx+1)\n",
    "    plt.imshow(state[0][:,:,idx])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53d90f32-e18c-41d7-ba8a-64a1d66e3763",
   "metadata": {},
   "source": [
    "## Train the RL Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "8a527c45-e00f-437e-b51a-58cba1b5b43d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import os for file path management\n",
    "import os\n",
    "# Import PPO for algos\n",
    "from stable_baselines3 import PPO\n",
    "# Import Base Callback for saving models\n",
    "from stable_baselines3.common.callbacks import BaseCallback"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "587e135a-e36a-407a-9dc9-d4396554140f",
   "metadata": {},
   "outputs": [],
   "source": [
    "class TrainAndLoggingCallback(BaseCallback):\n",
    "    def __init__(self, check_freq, save_path, verbose=1):\n",
    "        super(TrainAndLoggingCallback, self).__init__(verbose)\n",
    "        self.check_freq = check_freq\n",
    "        self.save_path = save_path\n",
    "        \n",
    "    def _init_callback(self):\n",
    "        if self.save_path is not None:\n",
    "            os.makedirs(self.save_path, exist_ok=True)\n",
    "            \n",
    "    def _on_step(self):\n",
    "        if self.n_calls % self.check_freq == 0:\n",
    "            model_path = os.path.join(self.save_path, \"best_model_{}\".format(self.n_calls))\n",
    "            self.model.save(model_path)\n",
    "        return True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "3c359931-f9d7-4059-b12c-c82a960c7715",
   "metadata": {},
   "outputs": [],
   "source": [
    "CHECKPOINT_DIR = \"./train/\"\n",
    "LOG_DIR = \"./logs/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "1e1b617d-9441-4565-8799-112f99fc77e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup model saving callback\n",
    "callback = TrainAndLoggingCallback(check_freq=10000, save_path=CHECKPOINT_DIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "c2e88b7b-5406-4a05-863d-0b0928f2f886",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using cuda device\n",
      "Wrapping the env in a VecTransposeImage.\n"
     ]
    }
   ],
   "source": [
    "# This is the AI model started\n",
    "model = PPO(\"CnnPolicy\", env, verbose=1, tensorboard_log=LOG_DIR, learning_rate=0.000001,\n",
    "            n_steps=512)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1438db7a-64e3-4b92-ac93-e688a199d829",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logging to ./logs/PPO_2\n",
      "----------------------------\n",
      "| time/              |     |\n",
      "|    fps             | 210 |\n",
      "|    iterations      | 1   |\n",
      "|    time_elapsed    | 2   |\n",
      "|    total_timesteps | 512 |\n",
      "----------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 131           |\n",
      "|    iterations           | 2             |\n",
      "|    time_elapsed         | 7             |\n",
      "|    total_timesteps      | 1024          |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 1.7860672e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.95         |\n",
      "|    explained_variance   | -0.00535      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 165           |\n",
      "|    n_updates            | 10            |\n",
      "|    policy_gradient_loss | -5.02e-05     |\n",
      "|    value_loss           | 401           |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 116           |\n",
      "|    iterations           | 3             |\n",
      "|    time_elapsed         | 13            |\n",
      "|    total_timesteps      | 1536          |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 8.2009006e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.95         |\n",
      "|    explained_variance   | 0.0276        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0806        |\n",
      "|    n_updates            | 20            |\n",
      "|    policy_gradient_loss | -0.000679     |\n",
      "|    value_loss           | 2.02          |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 109           |\n",
      "|    iterations           | 4             |\n",
      "|    time_elapsed         | 18            |\n",
      "|    total_timesteps      | 2048          |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 2.2537773e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.95         |\n",
      "|    explained_variance   | -0.0567       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.148         |\n",
      "|    n_updates            | 30            |\n",
      "|    policy_gradient_loss | -0.000245     |\n",
      "|    value_loss           | 0.878         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 107          |\n",
      "|    iterations           | 5            |\n",
      "|    time_elapsed         | 23           |\n",
      "|    total_timesteps      | 2560         |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 4.873355e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.95        |\n",
      "|    explained_variance   | 0.00303      |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 42.2         |\n",
      "|    n_updates            | 40           |\n",
      "|    policy_gradient_loss | -0.00022     |\n",
      "|    value_loss           | 101          |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 105          |\n",
      "|    iterations           | 6            |\n",
      "|    time_elapsed         | 29           |\n",
      "|    total_timesteps      | 3072         |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 8.020387e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.95        |\n",
      "|    explained_variance   | 0.00443      |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 127          |\n",
      "|    n_updates            | 50           |\n",
      "|    policy_gradient_loss | 0.000224     |\n",
      "|    value_loss           | 290          |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 104           |\n",
      "|    iterations           | 7             |\n",
      "|    time_elapsed         | 34            |\n",
      "|    total_timesteps      | 3584          |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 3.0604308e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.95         |\n",
      "|    explained_variance   | 0.00591       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.127         |\n",
      "|    n_updates            | 60            |\n",
      "|    policy_gradient_loss | 3.06e-05      |\n",
      "|    value_loss           | 5.65          |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 103           |\n",
      "|    iterations           | 8             |\n",
      "|    time_elapsed         | 39            |\n",
      "|    total_timesteps      | 4096          |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 5.5781566e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.95         |\n",
      "|    explained_variance   | -0.0604       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.118         |\n",
      "|    n_updates            | 70            |\n",
      "|    policy_gradient_loss | -0.000164     |\n",
      "|    value_loss           | 1.93          |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 102           |\n",
      "|    iterations           | 9             |\n",
      "|    time_elapsed         | 44            |\n",
      "|    total_timesteps      | 4608          |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 2.6324764e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.95         |\n",
      "|    explained_variance   | 0.0215        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.166         |\n",
      "|    n_updates            | 80            |\n",
      "|    policy_gradient_loss | -0.000202     |\n",
      "|    value_loss           | 1.61          |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 102          |\n",
      "|    iterations           | 10           |\n",
      "|    time_elapsed         | 49           |\n",
      "|    total_timesteps      | 5120         |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 9.190058e-06 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.95        |\n",
      "|    explained_variance   | -0.0318      |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.215        |\n",
      "|    n_updates            | 90           |\n",
      "|    policy_gradient_loss | 0.000124     |\n",
      "|    value_loss           | 1.25         |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 102           |\n",
      "|    iterations           | 11            |\n",
      "|    time_elapsed         | 55            |\n",
      "|    total_timesteps      | 5632          |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 1.8518302e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.95         |\n",
      "|    explained_variance   | 0.025         |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.111         |\n",
      "|    n_updates            | 100           |\n",
      "|    policy_gradient_loss | -0.000137     |\n",
      "|    value_loss           | 0.784         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 101           |\n",
      "|    iterations           | 12            |\n",
      "|    time_elapsed         | 60            |\n",
      "|    total_timesteps      | 6144          |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 4.9914233e-06 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.95         |\n",
      "|    explained_variance   | 0.0218        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.101         |\n",
      "|    n_updates            | 110           |\n",
      "|    policy_gradient_loss | 5.25e-05      |\n",
      "|    value_loss           | 0.526         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 101          |\n",
      "|    iterations           | 13           |\n",
      "|    time_elapsed         | 65           |\n",
      "|    total_timesteps      | 6656         |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 7.814029e-06 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.95        |\n",
      "|    explained_variance   | -0.00591     |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.127        |\n",
      "|    n_updates            | 120          |\n",
      "|    policy_gradient_loss | -7.25e-05    |\n",
      "|    value_loss           | 0.535        |\n",
      "------------------------------------------\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 101         |\n",
      "|    iterations           | 14          |\n",
      "|    time_elapsed         | 70          |\n",
      "|    total_timesteps      | 7168        |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 2.98901e-05 |\n",
      "|    clip_fraction        | 0           |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.95       |\n",
      "|    explained_variance   | 0.00228     |\n",
      "|    learning_rate        | 1e-06       |\n",
      "|    loss                 | 0.145       |\n",
      "|    n_updates            | 130         |\n",
      "|    policy_gradient_loss | -0.000375   |\n",
      "|    value_loss           | 0.383       |\n",
      "-----------------------------------------\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 101         |\n",
      "|    iterations           | 15          |\n",
      "|    time_elapsed         | 76          |\n",
      "|    total_timesteps      | 7680        |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 7.24697e-06 |\n",
      "|    clip_fraction        | 0           |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.95       |\n",
      "|    explained_variance   | -0.00125    |\n",
      "|    learning_rate        | 1e-06       |\n",
      "|    loss                 | 0.118       |\n",
      "|    n_updates            | 140         |\n",
      "|    policy_gradient_loss | -7.1e-05    |\n",
      "|    value_loss           | 0.303       |\n",
      "-----------------------------------------\n",
      "--------------------------------------------\n",
      "| time/                   |                |\n",
      "|    fps                  | 100            |\n",
      "|    iterations           | 16             |\n",
      "|    time_elapsed         | 81             |\n",
      "|    total_timesteps      | 8192           |\n",
      "| train/                  |                |\n",
      "|    approx_kl            | 1.25691295e-05 |\n",
      "|    clip_fraction        | 0              |\n",
      "|    clip_range           | 0.2            |\n",
      "|    entropy_loss         | -1.95          |\n",
      "|    explained_variance   | -0.0019        |\n",
      "|    learning_rate        | 1e-06          |\n",
      "|    loss                 | 0.135          |\n",
      "|    n_updates            | 150            |\n",
      "|    policy_gradient_loss | -0.000141      |\n",
      "|    value_loss           | 0.329          |\n",
      "--------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 100           |\n",
      "|    iterations           | 17            |\n",
      "|    time_elapsed         | 86            |\n",
      "|    total_timesteps      | 8704          |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 7.3928386e-06 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.95         |\n",
      "|    explained_variance   | -0.00212      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.101         |\n",
      "|    n_updates            | 160           |\n",
      "|    policy_gradient_loss | -9.51e-05     |\n",
      "|    value_loss           | 0.224         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 100           |\n",
      "|    iterations           | 18            |\n",
      "|    time_elapsed         | 91            |\n",
      "|    total_timesteps      | 9216          |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 1.3097771e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.95         |\n",
      "|    explained_variance   | -0.00188      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.136         |\n",
      "|    n_updates            | 170           |\n",
      "|    policy_gradient_loss | -0.000173     |\n",
      "|    value_loss           | 0.374         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 100           |\n",
      "|    iterations           | 19            |\n",
      "|    time_elapsed         | 97            |\n",
      "|    total_timesteps      | 9728          |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 3.5805628e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.94         |\n",
      "|    explained_variance   | -0.000507     |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.162         |\n",
      "|    n_updates            | 180           |\n",
      "|    policy_gradient_loss | -0.000506     |\n",
      "|    value_loss           | 0.363         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 99           |\n",
      "|    iterations           | 20           |\n",
      "|    time_elapsed         | 103          |\n",
      "|    total_timesteps      | 10240        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 5.588855e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.94        |\n",
      "|    explained_variance   | 0.000571     |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.184        |\n",
      "|    n_updates            | 190          |\n",
      "|    policy_gradient_loss | -0.000845    |\n",
      "|    value_loss           | 0.334        |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 99           |\n",
      "|    iterations           | 21           |\n",
      "|    time_elapsed         | 108          |\n",
      "|    total_timesteps      | 10752        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 9.610527e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.94        |\n",
      "|    explained_variance   | 0.0222       |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 196          |\n",
      "|    n_updates            | 200          |\n",
      "|    policy_gradient_loss | -0.00067     |\n",
      "|    value_loss           | 438          |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 98           |\n",
      "|    iterations           | 22           |\n",
      "|    time_elapsed         | 113          |\n",
      "|    total_timesteps      | 11264        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0002982769 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.94        |\n",
      "|    explained_variance   | 0.00601      |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 257          |\n",
      "|    n_updates            | 210          |\n",
      "|    policy_gradient_loss | -0.000425    |\n",
      "|    value_loss           | 705          |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 98           |\n",
      "|    iterations           | 23           |\n",
      "|    time_elapsed         | 119          |\n",
      "|    total_timesteps      | 11776        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 7.900607e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.94        |\n",
      "|    explained_variance   | 0.066        |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.116        |\n",
      "|    n_updates            | 220          |\n",
      "|    policy_gradient_loss | -0.000167    |\n",
      "|    value_loss           | 3.93         |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 98            |\n",
      "|    iterations           | 24            |\n",
      "|    time_elapsed         | 124           |\n",
      "|    total_timesteps      | 12288         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 4.0439423e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.94         |\n",
      "|    explained_variance   | -0.00885      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.207         |\n",
      "|    n_updates            | 230           |\n",
      "|    policy_gradient_loss | -0.000167     |\n",
      "|    value_loss           | 1.64          |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 98            |\n",
      "|    iterations           | 25            |\n",
      "|    time_elapsed         | 129           |\n",
      "|    total_timesteps      | 12800         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 1.6484875e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.94         |\n",
      "|    explained_variance   | -0.0525       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0739        |\n",
      "|    n_updates            | 240           |\n",
      "|    policy_gradient_loss | 0.000134      |\n",
      "|    value_loss           | 1.04          |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 98            |\n",
      "|    iterations           | 26            |\n",
      "|    time_elapsed         | 135           |\n",
      "|    total_timesteps      | 13312         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 4.2920234e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.94         |\n",
      "|    explained_variance   | 0.000691      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0715        |\n",
      "|    n_updates            | 250           |\n",
      "|    policy_gradient_loss | -0.000252     |\n",
      "|    value_loss           | 0.77          |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 98            |\n",
      "|    iterations           | 27            |\n",
      "|    time_elapsed         | 140           |\n",
      "|    total_timesteps      | 13824         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 1.6410253e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.94         |\n",
      "|    explained_variance   | 0.0637        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.134         |\n",
      "|    n_updates            | 260           |\n",
      "|    policy_gradient_loss | 3.92e-05      |\n",
      "|    value_loss           | 0.621         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 98            |\n",
      "|    iterations           | 28            |\n",
      "|    time_elapsed         | 145           |\n",
      "|    total_timesteps      | 14336         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 1.7949264e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.94         |\n",
      "|    explained_variance   | -0.00989      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0942        |\n",
      "|    n_updates            | 270           |\n",
      "|    policy_gradient_loss | 0.000115      |\n",
      "|    value_loss           | 0.392         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 98            |\n",
      "|    iterations           | 29            |\n",
      "|    time_elapsed         | 151           |\n",
      "|    total_timesteps      | 14848         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 1.7311075e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.94         |\n",
      "|    explained_variance   | -0.00952      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.105         |\n",
      "|    n_updates            | 280           |\n",
      "|    policy_gradient_loss | -8.83e-05     |\n",
      "|    value_loss           | 0.366         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 98            |\n",
      "|    iterations           | 30            |\n",
      "|    time_elapsed         | 156           |\n",
      "|    total_timesteps      | 15360         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 7.4040145e-06 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.94         |\n",
      "|    explained_variance   | 0.0145        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0706        |\n",
      "|    n_updates            | 290           |\n",
      "|    policy_gradient_loss | 1.03e-06      |\n",
      "|    value_loss           | 0.27          |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 98           |\n",
      "|    iterations           | 31           |\n",
      "|    time_elapsed         | 161          |\n",
      "|    total_timesteps      | 15872        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 8.024159e-06 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.94        |\n",
      "|    explained_variance   | 0.00607      |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.107        |\n",
      "|    n_updates            | 300          |\n",
      "|    policy_gradient_loss | -3.48e-05    |\n",
      "|    value_loss           | 0.258        |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 98            |\n",
      "|    iterations           | 32            |\n",
      "|    time_elapsed         | 166           |\n",
      "|    total_timesteps      | 16384         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 1.6261358e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.94         |\n",
      "|    explained_variance   | -0.00266      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.103         |\n",
      "|    n_updates            | 310           |\n",
      "|    policy_gradient_loss | -0.00016      |\n",
      "|    value_loss           | 0.235         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 98           |\n",
      "|    iterations           | 33           |\n",
      "|    time_elapsed         | 172          |\n",
      "|    total_timesteps      | 16896        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 2.338586e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.94        |\n",
      "|    explained_variance   | 0.00328      |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0775       |\n",
      "|    n_updates            | 320          |\n",
      "|    policy_gradient_loss | -0.000273    |\n",
      "|    value_loss           | 0.211        |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 97           |\n",
      "|    iterations           | 34           |\n",
      "|    time_elapsed         | 177          |\n",
      "|    total_timesteps      | 17408        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 3.166229e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.94        |\n",
      "|    explained_variance   | -0.000283    |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0824       |\n",
      "|    n_updates            | 330          |\n",
      "|    policy_gradient_loss | -0.000395    |\n",
      "|    value_loss           | 0.174        |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 97            |\n",
      "|    iterations           | 35            |\n",
      "|    time_elapsed         | 183           |\n",
      "|    total_timesteps      | 17920         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 9.3943556e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.94         |\n",
      "|    explained_variance   | 0.041         |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 37.4          |\n",
      "|    n_updates            | 340           |\n",
      "|    policy_gradient_loss | 0.000321      |\n",
      "|    value_loss           | 77.5          |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 97            |\n",
      "|    iterations           | 36            |\n",
      "|    time_elapsed         | 188           |\n",
      "|    total_timesteps      | 18432         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00015635614 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.94         |\n",
      "|    explained_variance   | -0.00132      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 50.1          |\n",
      "|    n_updates            | 350           |\n",
      "|    policy_gradient_loss | -0.000584     |\n",
      "|    value_loss           | 106           |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 97           |\n",
      "|    iterations           | 37           |\n",
      "|    time_elapsed         | 194          |\n",
      "|    total_timesteps      | 18944        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 7.156085e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.94        |\n",
      "|    explained_variance   | 0.0244       |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0809       |\n",
      "|    n_updates            | 360          |\n",
      "|    policy_gradient_loss | 1.4e-05      |\n",
      "|    value_loss           | 0.638        |\n",
      "------------------------------------------\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 97          |\n",
      "|    iterations           | 38          |\n",
      "|    time_elapsed         | 199         |\n",
      "|    total_timesteps      | 19456       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.001103882 |\n",
      "|    clip_fraction        | 0           |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.93       |\n",
      "|    explained_variance   | 0.032       |\n",
      "|    learning_rate        | 1e-06       |\n",
      "|    loss                 | 279         |\n",
      "|    n_updates            | 370         |\n",
      "|    policy_gradient_loss | -0.000286   |\n",
      "|    value_loss           | 604         |\n",
      "-----------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 97            |\n",
      "|    iterations           | 39            |\n",
      "|    time_elapsed         | 204           |\n",
      "|    total_timesteps      | 19968         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00015634077 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.93         |\n",
      "|    explained_variance   | -0.0418       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.169         |\n",
      "|    n_updates            | 380           |\n",
      "|    policy_gradient_loss | -0.000729     |\n",
      "|    value_loss           | 2.94          |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 96            |\n",
      "|    iterations           | 40            |\n",
      "|    time_elapsed         | 211           |\n",
      "|    total_timesteps      | 20480         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00016508391 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.93         |\n",
      "|    explained_variance   | 0.141         |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0914        |\n",
      "|    n_updates            | 390           |\n",
      "|    policy_gradient_loss | -0.000667     |\n",
      "|    value_loss           | 1.19          |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 96           |\n",
      "|    iterations           | 41           |\n",
      "|    time_elapsed         | 216          |\n",
      "|    total_timesteps      | 20992        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 9.172596e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.94        |\n",
      "|    explained_variance   | -0.115       |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.15         |\n",
      "|    n_updates            | 400          |\n",
      "|    policy_gradient_loss | -7.36e-05    |\n",
      "|    value_loss           | 1.03         |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 96            |\n",
      "|    iterations           | 42            |\n",
      "|    time_elapsed         | 222           |\n",
      "|    total_timesteps      | 21504         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 1.6871723e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.94         |\n",
      "|    explained_variance   | -0.0105       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0508        |\n",
      "|    n_updates            | 410           |\n",
      "|    policy_gradient_loss | 0.000298      |\n",
      "|    value_loss           | 0.571         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 96            |\n",
      "|    iterations           | 43            |\n",
      "|    time_elapsed         | 227           |\n",
      "|    total_timesteps      | 22016         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 1.7316663e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.94         |\n",
      "|    explained_variance   | -0.0243       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0676        |\n",
      "|    n_updates            | 420           |\n",
      "|    policy_gradient_loss | 2.82e-05      |\n",
      "|    value_loss           | 0.463         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 96           |\n",
      "|    iterations           | 44           |\n",
      "|    time_elapsed         | 233          |\n",
      "|    total_timesteps      | 22528        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 1.473981e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.94        |\n",
      "|    explained_variance   | 0.0268       |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0878       |\n",
      "|    n_updates            | 430          |\n",
      "|    policy_gradient_loss | 7.81e-05     |\n",
      "|    value_loss           | 0.381        |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 96           |\n",
      "|    iterations           | 45           |\n",
      "|    time_elapsed         | 238          |\n",
      "|    total_timesteps      | 23040        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 2.543407e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.94        |\n",
      "|    explained_variance   | 0.0362       |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0402       |\n",
      "|    n_updates            | 440          |\n",
      "|    policy_gradient_loss | -0.00013     |\n",
      "|    value_loss           | 0.231        |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 96            |\n",
      "|    iterations           | 46            |\n",
      "|    time_elapsed         | 244           |\n",
      "|    total_timesteps      | 23552         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 2.4154782e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.94         |\n",
      "|    explained_variance   | 0.0402        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.077         |\n",
      "|    n_updates            | 450           |\n",
      "|    policy_gradient_loss | -0.000105     |\n",
      "|    value_loss           | 0.306         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 96           |\n",
      "|    iterations           | 47           |\n",
      "|    time_elapsed         | 249          |\n",
      "|    total_timesteps      | 24064        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 4.986755e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.93        |\n",
      "|    explained_variance   | 0.0501       |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.118        |\n",
      "|    n_updates            | 460          |\n",
      "|    policy_gradient_loss | -0.000377    |\n",
      "|    value_loss           | 0.312        |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 96           |\n",
      "|    iterations           | 48           |\n",
      "|    time_elapsed         | 254          |\n",
      "|    total_timesteps      | 24576        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 8.864701e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.93        |\n",
      "|    explained_variance   | -0.0128      |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.11         |\n",
      "|    n_updates            | 470          |\n",
      "|    policy_gradient_loss | -0.00062     |\n",
      "|    value_loss           | 0.244        |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 96            |\n",
      "|    iterations           | 49            |\n",
      "|    time_elapsed         | 260           |\n",
      "|    total_timesteps      | 25088         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 1.9248808e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.93         |\n",
      "|    explained_variance   | -0.00777      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0946        |\n",
      "|    n_updates            | 480           |\n",
      "|    policy_gradient_loss | -0.000209     |\n",
      "|    value_loss           | 0.179         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 96            |\n",
      "|    iterations           | 50            |\n",
      "|    time_elapsed         | 265           |\n",
      "|    total_timesteps      | 25600         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 4.2053754e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.93         |\n",
      "|    explained_variance   | -0.0119       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0486        |\n",
      "|    n_updates            | 490           |\n",
      "|    policy_gradient_loss | -0.000359     |\n",
      "|    value_loss           | 0.123         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 96            |\n",
      "|    iterations           | 51            |\n",
      "|    time_elapsed         | 271           |\n",
      "|    total_timesteps      | 26112         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 5.9330952e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.93         |\n",
      "|    explained_variance   | -0.00448      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0778        |\n",
      "|    n_updates            | 500           |\n",
      "|    policy_gradient_loss | -0.000624     |\n",
      "|    value_loss           | 0.219         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 96            |\n",
      "|    iterations           | 52            |\n",
      "|    time_elapsed         | 276           |\n",
      "|    total_timesteps      | 26624         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00011908007 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.93         |\n",
      "|    explained_variance   | -0.00116      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0343        |\n",
      "|    n_updates            | 510           |\n",
      "|    policy_gradient_loss | -0.00106      |\n",
      "|    value_loss           | 0.0997        |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 96            |\n",
      "|    iterations           | 53            |\n",
      "|    time_elapsed         | 281           |\n",
      "|    total_timesteps      | 27136         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00066433207 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.93         |\n",
      "|    explained_variance   | 0.0822        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 104           |\n",
      "|    n_updates            | 520           |\n",
      "|    policy_gradient_loss | -0.00022      |\n",
      "|    value_loss           | 199           |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 96           |\n",
      "|    iterations           | 54           |\n",
      "|    time_elapsed         | 287          |\n",
      "|    total_timesteps      | 27648        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0009904607 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.92        |\n",
      "|    explained_variance   | 0.072        |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 81.8         |\n",
      "|    n_updates            | 530          |\n",
      "|    policy_gradient_loss | 0.000693     |\n",
      "|    value_loss           | 167          |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 96            |\n",
      "|    iterations           | 55            |\n",
      "|    time_elapsed         | 292           |\n",
      "|    total_timesteps      | 28160         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00013382139 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.92         |\n",
      "|    explained_variance   | 0.00449       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.156         |\n",
      "|    n_updates            | 540           |\n",
      "|    policy_gradient_loss | 0.00022       |\n",
      "|    value_loss           | 1.53          |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 96           |\n",
      "|    iterations           | 56           |\n",
      "|    time_elapsed         | 298          |\n",
      "|    total_timesteps      | 28672        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 9.766093e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.92        |\n",
      "|    explained_variance   | -0.0911      |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.116        |\n",
      "|    n_updates            | 550          |\n",
      "|    policy_gradient_loss | 2.74e-05     |\n",
      "|    value_loss           | 1.11         |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 96            |\n",
      "|    iterations           | 57            |\n",
      "|    time_elapsed         | 303           |\n",
      "|    total_timesteps      | 29184         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 1.5745754e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.92         |\n",
      "|    explained_variance   | -0.000197     |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.125         |\n",
      "|    n_updates            | 560           |\n",
      "|    policy_gradient_loss | 0.000605      |\n",
      "|    value_loss           | 0.838         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 95           |\n",
      "|    iterations           | 58           |\n",
      "|    time_elapsed         | 309          |\n",
      "|    total_timesteps      | 29696        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 2.544152e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.92        |\n",
      "|    explained_variance   | -0.0909      |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.212        |\n",
      "|    n_updates            | 570          |\n",
      "|    policy_gradient_loss | 0.000186     |\n",
      "|    value_loss           | 0.7          |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 95            |\n",
      "|    iterations           | 59            |\n",
      "|    time_elapsed         | 315           |\n",
      "|    total_timesteps      | 30208         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 5.9786602e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.92         |\n",
      "|    explained_variance   | -0.0179       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.202         |\n",
      "|    n_updates            | 580           |\n",
      "|    policy_gradient_loss | -0.000144     |\n",
      "|    value_loss           | 0.514         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 95            |\n",
      "|    iterations           | 60            |\n",
      "|    time_elapsed         | 321           |\n",
      "|    total_timesteps      | 30720         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 4.0216604e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.92         |\n",
      "|    explained_variance   | 0.0311        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.131         |\n",
      "|    n_updates            | 590           |\n",
      "|    policy_gradient_loss | -4.58e-05     |\n",
      "|    value_loss           | 0.462         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 95           |\n",
      "|    iterations           | 61           |\n",
      "|    time_elapsed         | 326          |\n",
      "|    total_timesteps      | 31232        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 7.031043e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.92        |\n",
      "|    explained_variance   | -0.00716     |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.107        |\n",
      "|    n_updates            | 600          |\n",
      "|    policy_gradient_loss | -0.00047     |\n",
      "|    value_loss           | 0.353        |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 95            |\n",
      "|    iterations           | 62            |\n",
      "|    time_elapsed         | 332           |\n",
      "|    total_timesteps      | 31744         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 1.2258766e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.92         |\n",
      "|    explained_variance   | 0.0613        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0953        |\n",
      "|    n_updates            | 610           |\n",
      "|    policy_gradient_loss | 5.19e-05      |\n",
      "|    value_loss           | 0.331         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 95           |\n",
      "|    iterations           | 63           |\n",
      "|    time_elapsed         | 337          |\n",
      "|    total_timesteps      | 32256        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 3.248139e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.92        |\n",
      "|    explained_variance   | 0.0417       |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.11         |\n",
      "|    n_updates            | 620          |\n",
      "|    policy_gradient_loss | -0.000169    |\n",
      "|    value_loss           | 0.277        |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 95            |\n",
      "|    iterations           | 64            |\n",
      "|    time_elapsed         | 343           |\n",
      "|    total_timesteps      | 32768         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 7.0286915e-06 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.93         |\n",
      "|    explained_variance   | -0.011        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.065         |\n",
      "|    n_updates            | 630           |\n",
      "|    policy_gradient_loss | -2.63e-06     |\n",
      "|    value_loss           | 0.173         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 95            |\n",
      "|    iterations           | 65            |\n",
      "|    time_elapsed         | 348           |\n",
      "|    total_timesteps      | 33280         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 6.2738196e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.92         |\n",
      "|    explained_variance   | -0.00405      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0837        |\n",
      "|    n_updates            | 640           |\n",
      "|    policy_gradient_loss | -0.000359     |\n",
      "|    value_loss           | 0.21          |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 95            |\n",
      "|    iterations           | 66            |\n",
      "|    time_elapsed         | 354           |\n",
      "|    total_timesteps      | 33792         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 5.1776064e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.92         |\n",
      "|    explained_variance   | 0.0448        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0503        |\n",
      "|    n_updates            | 650           |\n",
      "|    policy_gradient_loss | -0.000475     |\n",
      "|    value_loss           | 0.172         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 95            |\n",
      "|    iterations           | 67            |\n",
      "|    time_elapsed         | 359           |\n",
      "|    total_timesteps      | 34304         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 2.5550718e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.92         |\n",
      "|    explained_variance   | 0.375         |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0731        |\n",
      "|    n_updates            | 660           |\n",
      "|    policy_gradient_loss | -7.56e-05     |\n",
      "|    value_loss           | 0.143         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 95            |\n",
      "|    iterations           | 68            |\n",
      "|    time_elapsed         | 365           |\n",
      "|    total_timesteps      | 34816         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 4.4237473e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.92         |\n",
      "|    explained_variance   | 0.00219       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0481        |\n",
      "|    n_updates            | 670           |\n",
      "|    policy_gradient_loss | -0.000295     |\n",
      "|    value_loss           | 0.141         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 95           |\n",
      "|    iterations           | 69           |\n",
      "|    time_elapsed         | 370          |\n",
      "|    total_timesteps      | 35328        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0016930823 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.91        |\n",
      "|    explained_variance   | 0.0878       |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 178          |\n",
      "|    n_updates            | 680          |\n",
      "|    policy_gradient_loss | -0.000931    |\n",
      "|    value_loss           | 497          |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 95           |\n",
      "|    iterations           | 70           |\n",
      "|    time_elapsed         | 376          |\n",
      "|    total_timesteps      | 35840        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0005324406 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.9         |\n",
      "|    explained_variance   | 0.117        |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 32.6         |\n",
      "|    n_updates            | 690          |\n",
      "|    policy_gradient_loss | -0.000726    |\n",
      "|    value_loss           | 114          |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 95            |\n",
      "|    iterations           | 71            |\n",
      "|    time_elapsed         | 382           |\n",
      "|    total_timesteps      | 36352         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00015909481 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.9          |\n",
      "|    explained_variance   | -0.00179      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0875        |\n",
      "|    n_updates            | 700           |\n",
      "|    policy_gradient_loss | -0.000388     |\n",
      "|    value_loss           | 0.806         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 95            |\n",
      "|    iterations           | 72            |\n",
      "|    time_elapsed         | 387           |\n",
      "|    total_timesteps      | 36864         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00031201076 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.9          |\n",
      "|    explained_variance   | 0.00174       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.063         |\n",
      "|    n_updates            | 710           |\n",
      "|    policy_gradient_loss | -0.00103      |\n",
      "|    value_loss           | 0.624         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 95            |\n",
      "|    iterations           | 73            |\n",
      "|    time_elapsed         | 393           |\n",
      "|    total_timesteps      | 37376         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00017916993 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.9          |\n",
      "|    explained_variance   | 0.0639        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 36.9          |\n",
      "|    n_updates            | 720           |\n",
      "|    policy_gradient_loss | -0.00017      |\n",
      "|    value_loss           | 82.5          |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 95            |\n",
      "|    iterations           | 74            |\n",
      "|    time_elapsed         | 398           |\n",
      "|    total_timesteps      | 37888         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00030159962 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.9          |\n",
      "|    explained_variance   | 0.0352        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.108         |\n",
      "|    n_updates            | 730           |\n",
      "|    policy_gradient_loss | -0.00107      |\n",
      "|    value_loss           | 1.03          |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 94            |\n",
      "|    iterations           | 75            |\n",
      "|    time_elapsed         | 404           |\n",
      "|    total_timesteps      | 38400         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00012440432 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.91         |\n",
      "|    explained_variance   | 0.21          |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0999        |\n",
      "|    n_updates            | 740           |\n",
      "|    policy_gradient_loss | 0.000269      |\n",
      "|    value_loss           | 0.814         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 94            |\n",
      "|    iterations           | 76            |\n",
      "|    time_elapsed         | 409           |\n",
      "|    total_timesteps      | 38912         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 7.6999655e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.91         |\n",
      "|    explained_variance   | -0.0219       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0465        |\n",
      "|    n_updates            | 750           |\n",
      "|    policy_gradient_loss | 1.02e-05      |\n",
      "|    value_loss           | 0.595         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 94            |\n",
      "|    iterations           | 77            |\n",
      "|    time_elapsed         | 415           |\n",
      "|    total_timesteps      | 39424         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 8.6539425e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.91         |\n",
      "|    explained_variance   | 0.0329        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0739        |\n",
      "|    n_updates            | 760           |\n",
      "|    policy_gradient_loss | -0.000457     |\n",
      "|    value_loss           | 0.474         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 94           |\n",
      "|    iterations           | 78           |\n",
      "|    time_elapsed         | 421          |\n",
      "|    total_timesteps      | 39936        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 5.492824e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.91        |\n",
      "|    explained_variance   | 0.0278       |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0654       |\n",
      "|    n_updates            | 770          |\n",
      "|    policy_gradient_loss | -1.7e-05     |\n",
      "|    value_loss           | 0.29         |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 94            |\n",
      "|    iterations           | 79            |\n",
      "|    time_elapsed         | 427           |\n",
      "|    total_timesteps      | 40448         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00011380494 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.91         |\n",
      "|    explained_variance   | 0.249         |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0965        |\n",
      "|    n_updates            | 780           |\n",
      "|    policy_gradient_loss | -0.00079      |\n",
      "|    value_loss           | 0.263         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 94            |\n",
      "|    iterations           | 80            |\n",
      "|    time_elapsed         | 433           |\n",
      "|    total_timesteps      | 40960         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 9.6012256e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.91         |\n",
      "|    explained_variance   | -0.0329       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0581        |\n",
      "|    n_updates            | 790           |\n",
      "|    policy_gradient_loss | -0.000626     |\n",
      "|    value_loss           | 0.209         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 94           |\n",
      "|    iterations           | 81           |\n",
      "|    time_elapsed         | 438          |\n",
      "|    total_timesteps      | 41472        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 8.802314e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.91        |\n",
      "|    explained_variance   | -0.0557      |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0753       |\n",
      "|    n_updates            | 800          |\n",
      "|    policy_gradient_loss | -0.000692    |\n",
      "|    value_loss           | 0.179        |\n",
      "------------------------------------------\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 94          |\n",
      "|    iterations           | 82          |\n",
      "|    time_elapsed         | 444         |\n",
      "|    total_timesteps      | 41984       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 2.78214e-05 |\n",
      "|    clip_fraction        | 0           |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.92       |\n",
      "|    explained_variance   | 0.0932      |\n",
      "|    learning_rate        | 1e-06       |\n",
      "|    loss                 | 0.07        |\n",
      "|    n_updates            | 810         |\n",
      "|    policy_gradient_loss | -0.000202   |\n",
      "|    value_loss           | 0.183       |\n",
      "-----------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 94            |\n",
      "|    iterations           | 83            |\n",
      "|    time_elapsed         | 450           |\n",
      "|    total_timesteps      | 42496         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00014723686 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.92         |\n",
      "|    explained_variance   | 0.0365        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.035         |\n",
      "|    n_updates            | 820           |\n",
      "|    policy_gradient_loss | -0.00101      |\n",
      "|    value_loss           | 0.133         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 94            |\n",
      "|    iterations           | 84            |\n",
      "|    time_elapsed         | 455           |\n",
      "|    total_timesteps      | 43008         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 6.5984554e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.92         |\n",
      "|    explained_variance   | 0.0877        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0476        |\n",
      "|    n_updates            | 830           |\n",
      "|    policy_gradient_loss | -0.000487     |\n",
      "|    value_loss           | 0.132         |\n",
      "-------------------------------------------\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 94          |\n",
      "|    iterations           | 85          |\n",
      "|    time_elapsed         | 461         |\n",
      "|    total_timesteps      | 43520       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.001376153 |\n",
      "|    clip_fraction        | 0           |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.91       |\n",
      "|    explained_variance   | 0.0329      |\n",
      "|    learning_rate        | 1e-06       |\n",
      "|    loss                 | 136         |\n",
      "|    n_updates            | 840         |\n",
      "|    policy_gradient_loss | 0.000727    |\n",
      "|    value_loss           | 389         |\n",
      "-----------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 94            |\n",
      "|    iterations           | 86            |\n",
      "|    time_elapsed         | 467           |\n",
      "|    total_timesteps      | 44032         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00010371697 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.91         |\n",
      "|    explained_variance   | 0.0099        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.243         |\n",
      "|    n_updates            | 850           |\n",
      "|    policy_gradient_loss | -0.000336     |\n",
      "|    value_loss           | 1.06          |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 94            |\n",
      "|    iterations           | 87            |\n",
      "|    time_elapsed         | 472           |\n",
      "|    total_timesteps      | 44544         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 1.2172852e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.91         |\n",
      "|    explained_variance   | 0.133         |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.121         |\n",
      "|    n_updates            | 860           |\n",
      "|    policy_gradient_loss | 0.000269      |\n",
      "|    value_loss           | 0.511         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 94            |\n",
      "|    iterations           | 88            |\n",
      "|    time_elapsed         | 478           |\n",
      "|    total_timesteps      | 45056         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 3.4441357e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.91         |\n",
      "|    explained_variance   | 2.27e-05      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.125         |\n",
      "|    n_updates            | 870           |\n",
      "|    policy_gradient_loss | -0.000125     |\n",
      "|    value_loss           | 0.407         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 94           |\n",
      "|    iterations           | 89           |\n",
      "|    time_elapsed         | 483          |\n",
      "|    total_timesteps      | 45568        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 9.591458e-06 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.91        |\n",
      "|    explained_variance   | 0.06         |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.109        |\n",
      "|    n_updates            | 880          |\n",
      "|    policy_gradient_loss | 0.000114     |\n",
      "|    value_loss           | 0.34         |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 94           |\n",
      "|    iterations           | 90           |\n",
      "|    time_elapsed         | 489          |\n",
      "|    total_timesteps      | 46080        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 2.851698e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.91        |\n",
      "|    explained_variance   | 0.0374       |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.106        |\n",
      "|    n_updates            | 890          |\n",
      "|    policy_gradient_loss | -0.000134    |\n",
      "|    value_loss           | 0.311        |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 94            |\n",
      "|    iterations           | 91            |\n",
      "|    time_elapsed         | 494           |\n",
      "|    total_timesteps      | 46592         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 1.8614577e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.91         |\n",
      "|    explained_variance   | -0.000379     |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.131         |\n",
      "|    n_updates            | 900           |\n",
      "|    policy_gradient_loss | -0.000127     |\n",
      "|    value_loss           | 0.272         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 94            |\n",
      "|    iterations           | 92            |\n",
      "|    time_elapsed         | 500           |\n",
      "|    total_timesteps      | 47104         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 4.6236557e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.91         |\n",
      "|    explained_variance   | 0.00745       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.11          |\n",
      "|    n_updates            | 910           |\n",
      "|    policy_gradient_loss | -0.000303     |\n",
      "|    value_loss           | 0.279         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 94            |\n",
      "|    iterations           | 93            |\n",
      "|    time_elapsed         | 506           |\n",
      "|    total_timesteps      | 47616         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 8.8869594e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.91         |\n",
      "|    explained_variance   | 0.000389      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.142         |\n",
      "|    n_updates            | 920           |\n",
      "|    policy_gradient_loss | -0.000806     |\n",
      "|    value_loss           | 0.262         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 94            |\n",
      "|    iterations           | 94            |\n",
      "|    time_elapsed         | 511           |\n",
      "|    total_timesteps      | 48128         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 7.7286386e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.91         |\n",
      "|    explained_variance   | 0.0124        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.134         |\n",
      "|    n_updates            | 930           |\n",
      "|    policy_gradient_loss | -0.00037      |\n",
      "|    value_loss           | 0.349         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 94            |\n",
      "|    iterations           | 95            |\n",
      "|    time_elapsed         | 517           |\n",
      "|    total_timesteps      | 48640         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00040098105 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.9          |\n",
      "|    explained_variance   | 0.0612        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 28.5          |\n",
      "|    n_updates            | 940           |\n",
      "|    policy_gradient_loss | 0.000616      |\n",
      "|    value_loss           | 70.9          |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 93            |\n",
      "|    iterations           | 96            |\n",
      "|    time_elapsed         | 523           |\n",
      "|    total_timesteps      | 49152         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 5.4128235e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.9          |\n",
      "|    explained_variance   | 0.463         |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0678        |\n",
      "|    n_updates            | 950           |\n",
      "|    policy_gradient_loss | -0.000168     |\n",
      "|    value_loss           | 0.272         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 93            |\n",
      "|    iterations           | 97            |\n",
      "|    time_elapsed         | 529           |\n",
      "|    total_timesteps      | 49664         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 2.0504347e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.91         |\n",
      "|    explained_variance   | -0.0471       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0594        |\n",
      "|    n_updates            | 960           |\n",
      "|    policy_gradient_loss | 6.72e-05      |\n",
      "|    value_loss           | 0.233         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 92            |\n",
      "|    iterations           | 98            |\n",
      "|    time_elapsed         | 542           |\n",
      "|    total_timesteps      | 50176         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 3.5853125e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.91         |\n",
      "|    explained_variance   | 0.0359        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0451        |\n",
      "|    n_updates            | 970           |\n",
      "|    policy_gradient_loss | -0.000142     |\n",
      "|    value_loss           | 0.17          |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 92            |\n",
      "|    iterations           | 99            |\n",
      "|    time_elapsed         | 548           |\n",
      "|    total_timesteps      | 50688         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 4.3772976e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.91         |\n",
      "|    explained_variance   | 0.0255        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0555        |\n",
      "|    n_updates            | 980           |\n",
      "|    policy_gradient_loss | -0.000254     |\n",
      "|    value_loss           | 0.159         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 92            |\n",
      "|    iterations           | 100           |\n",
      "|    time_elapsed         | 553           |\n",
      "|    total_timesteps      | 51200         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 1.7809682e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.91         |\n",
      "|    explained_variance   | 0.07          |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 1.15          |\n",
      "|    n_updates            | 990           |\n",
      "|    policy_gradient_loss | -3.68e-07     |\n",
      "|    value_loss           | 2.22          |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 92           |\n",
      "|    iterations           | 101          |\n",
      "|    time_elapsed         | 559          |\n",
      "|    total_timesteps      | 51712        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0018251564 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.9         |\n",
      "|    explained_variance   | 0.0353       |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 215          |\n",
      "|    n_updates            | 1000         |\n",
      "|    policy_gradient_loss | 0.000824     |\n",
      "|    value_loss           | 487          |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 92            |\n",
      "|    iterations           | 102           |\n",
      "|    time_elapsed         | 564           |\n",
      "|    total_timesteps      | 52224         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 3.9993436e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.89         |\n",
      "|    explained_variance   | 0.373         |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.109         |\n",
      "|    n_updates            | 1010          |\n",
      "|    policy_gradient_loss | 0.000197      |\n",
      "|    value_loss           | 2.05          |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 92           |\n",
      "|    iterations           | 103          |\n",
      "|    time_elapsed         | 570          |\n",
      "|    total_timesteps      | 52736        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0001300131 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.9         |\n",
      "|    explained_variance   | -0.00834     |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.073        |\n",
      "|    n_updates            | 1020         |\n",
      "|    policy_gradient_loss | -0.000459    |\n",
      "|    value_loss           | 0.389        |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 92            |\n",
      "|    iterations           | 104           |\n",
      "|    time_elapsed         | 575           |\n",
      "|    total_timesteps      | 53248         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 7.6748314e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.9          |\n",
      "|    explained_variance   | 0.339         |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.144         |\n",
      "|    n_updates            | 1030          |\n",
      "|    policy_gradient_loss | -0.00055      |\n",
      "|    value_loss           | 0.492         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 92            |\n",
      "|    iterations           | 105           |\n",
      "|    time_elapsed         | 581           |\n",
      "|    total_timesteps      | 53760         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 2.0007952e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.9          |\n",
      "|    explained_variance   | -0.0146       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0619        |\n",
      "|    n_updates            | 1040          |\n",
      "|    policy_gradient_loss | 8.29e-05      |\n",
      "|    value_loss           | 0.278         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 92           |\n",
      "|    iterations           | 106          |\n",
      "|    time_elapsed         | 587          |\n",
      "|    total_timesteps      | 54272        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 4.873448e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.9         |\n",
      "|    explained_variance   | -0.00413     |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0803       |\n",
      "|    n_updates            | 1050         |\n",
      "|    policy_gradient_loss | -0.000268    |\n",
      "|    value_loss           | 0.258        |\n",
      "------------------------------------------\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 92          |\n",
      "|    iterations           | 107         |\n",
      "|    time_elapsed         | 592         |\n",
      "|    total_timesteps      | 54784       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 7.32505e-05 |\n",
      "|    clip_fraction        | 0           |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.9        |\n",
      "|    explained_variance   | 0.227       |\n",
      "|    learning_rate        | 1e-06       |\n",
      "|    loss                 | 0.0908      |\n",
      "|    n_updates            | 1060        |\n",
      "|    policy_gradient_loss | -0.000319   |\n",
      "|    value_loss           | 0.249       |\n",
      "-----------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 92           |\n",
      "|    iterations           | 108          |\n",
      "|    time_elapsed         | 598          |\n",
      "|    total_timesteps      | 55296        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 5.936832e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.9         |\n",
      "|    explained_variance   | 0.045        |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0755       |\n",
      "|    n_updates            | 1070         |\n",
      "|    policy_gradient_loss | -0.000336    |\n",
      "|    value_loss           | 0.201        |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 92           |\n",
      "|    iterations           | 109          |\n",
      "|    time_elapsed         | 604          |\n",
      "|    total_timesteps      | 55808        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 9.342737e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.9         |\n",
      "|    explained_variance   | 0.0592       |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0883       |\n",
      "|    n_updates            | 1080         |\n",
      "|    policy_gradient_loss | -0.000679    |\n",
      "|    value_loss           | 0.269        |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 92           |\n",
      "|    iterations           | 110          |\n",
      "|    time_elapsed         | 609          |\n",
      "|    total_timesteps      | 56320        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 3.462739e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.9         |\n",
      "|    explained_variance   | 0.0358       |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0625       |\n",
      "|    n_updates            | 1090         |\n",
      "|    policy_gradient_loss | -0.000288    |\n",
      "|    value_loss           | 0.166        |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 92            |\n",
      "|    iterations           | 111           |\n",
      "|    time_elapsed         | 615           |\n",
      "|    total_timesteps      | 56832         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 7.4696494e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.9          |\n",
      "|    explained_variance   | -0.0124       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0503        |\n",
      "|    n_updates            | 1100          |\n",
      "|    policy_gradient_loss | -0.000477     |\n",
      "|    value_loss           | 0.123         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 92            |\n",
      "|    iterations           | 112           |\n",
      "|    time_elapsed         | 620           |\n",
      "|    total_timesteps      | 57344         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00012448325 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.9          |\n",
      "|    explained_variance   | 0.298         |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0376        |\n",
      "|    n_updates            | 1110          |\n",
      "|    policy_gradient_loss | -0.000693     |\n",
      "|    value_loss           | 0.114         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 92            |\n",
      "|    iterations           | 113           |\n",
      "|    time_elapsed         | 626           |\n",
      "|    total_timesteps      | 57856         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 1.2162258e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.9          |\n",
      "|    explained_variance   | -0.00251      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0478        |\n",
      "|    n_updates            | 1120          |\n",
      "|    policy_gradient_loss | -7.48e-05     |\n",
      "|    value_loss           | 0.101         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 92           |\n",
      "|    iterations           | 114          |\n",
      "|    time_elapsed         | 631          |\n",
      "|    total_timesteps      | 58368        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 2.306595e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.9         |\n",
      "|    explained_variance   | -0.00314     |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0769       |\n",
      "|    n_updates            | 1130         |\n",
      "|    policy_gradient_loss | -0.00013     |\n",
      "|    value_loss           | 0.159        |\n",
      "------------------------------------------\n",
      "--------------------------------------------\n",
      "| time/                   |                |\n",
      "|    fps                  | 92             |\n",
      "|    iterations           | 115            |\n",
      "|    time_elapsed         | 637            |\n",
      "|    total_timesteps      | 58880          |\n",
      "| train/                  |                |\n",
      "|    approx_kl            | 0.000106780324 |\n",
      "|    clip_fraction        | 0              |\n",
      "|    clip_range           | 0.2            |\n",
      "|    entropy_loss         | -1.9           |\n",
      "|    explained_variance   | 0.00969        |\n",
      "|    learning_rate        | 1e-06          |\n",
      "|    loss                 | 0.0544         |\n",
      "|    n_updates            | 1140           |\n",
      "|    policy_gradient_loss | -0.000698      |\n",
      "|    value_loss           | 0.134          |\n",
      "--------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 92           |\n",
      "|    iterations           | 116          |\n",
      "|    time_elapsed         | 642          |\n",
      "|    total_timesteps      | 59392        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0018202348 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.9         |\n",
      "|    explained_variance   | 0.0794       |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 163          |\n",
      "|    n_updates            | 1150         |\n",
      "|    policy_gradient_loss | 0.000584     |\n",
      "|    value_loss           | 384          |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 92            |\n",
      "|    iterations           | 117           |\n",
      "|    time_elapsed         | 648           |\n",
      "|    total_timesteps      | 59904         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00056586415 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.88         |\n",
      "|    explained_variance   | 0.23          |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 42.3          |\n",
      "|    n_updates            | 1160          |\n",
      "|    policy_gradient_loss | -0.00218      |\n",
      "|    value_loss           | 89.9          |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 92           |\n",
      "|    iterations           | 118          |\n",
      "|    time_elapsed         | 655          |\n",
      "|    total_timesteps      | 60416        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 7.913774e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.88        |\n",
      "|    explained_variance   | 0.0279       |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0652       |\n",
      "|    n_updates            | 1170         |\n",
      "|    policy_gradient_loss | 0.000437     |\n",
      "|    value_loss           | 0.887        |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 92            |\n",
      "|    iterations           | 119           |\n",
      "|    time_elapsed         | 660           |\n",
      "|    total_timesteps      | 60928         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00013500568 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.88         |\n",
      "|    explained_variance   | -0.0742       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.181         |\n",
      "|    n_updates            | 1180          |\n",
      "|    policy_gradient_loss | -0.000177     |\n",
      "|    value_loss           | 0.64          |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 92           |\n",
      "|    iterations           | 120          |\n",
      "|    time_elapsed         | 666          |\n",
      "|    total_timesteps      | 61440        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 4.973577e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.88        |\n",
      "|    explained_variance   | -0.0137      |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0594       |\n",
      "|    n_updates            | 1190         |\n",
      "|    policy_gradient_loss | 0.000222     |\n",
      "|    value_loss           | 0.397        |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 92            |\n",
      "|    iterations           | 121           |\n",
      "|    time_elapsed         | 672           |\n",
      "|    total_timesteps      | 61952         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 3.3812714e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.88         |\n",
      "|    explained_variance   | -0.0403       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.102         |\n",
      "|    n_updates            | 1200          |\n",
      "|    policy_gradient_loss | 1.18e-05      |\n",
      "|    value_loss           | 0.329         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 92            |\n",
      "|    iterations           | 122           |\n",
      "|    time_elapsed         | 678           |\n",
      "|    total_timesteps      | 62464         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 3.6838464e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.88         |\n",
      "|    explained_variance   | -0.027        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0525        |\n",
      "|    n_updates            | 1210          |\n",
      "|    policy_gradient_loss | 0.000164      |\n",
      "|    value_loss           | 0.271         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 92            |\n",
      "|    iterations           | 123           |\n",
      "|    time_elapsed         | 684           |\n",
      "|    total_timesteps      | 62976         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00014744955 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.88         |\n",
      "|    explained_variance   | -0.00851      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0433        |\n",
      "|    n_updates            | 1220          |\n",
      "|    policy_gradient_loss | -0.000714     |\n",
      "|    value_loss           | 0.159         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 92            |\n",
      "|    iterations           | 124           |\n",
      "|    time_elapsed         | 689           |\n",
      "|    total_timesteps      | 63488         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 2.6370632e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.89         |\n",
      "|    explained_variance   | 0.309         |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.135         |\n",
      "|    n_updates            | 1230          |\n",
      "|    policy_gradient_loss | -2.11e-05     |\n",
      "|    value_loss           | 0.28          |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 92            |\n",
      "|    iterations           | 125           |\n",
      "|    time_elapsed         | 695           |\n",
      "|    total_timesteps      | 64000         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 2.4867011e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.89         |\n",
      "|    explained_variance   | 0.0259        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0985        |\n",
      "|    n_updates            | 1240          |\n",
      "|    policy_gradient_loss | 3.09e-06      |\n",
      "|    value_loss           | 0.17          |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 92            |\n",
      "|    iterations           | 126           |\n",
      "|    time_elapsed         | 701           |\n",
      "|    total_timesteps      | 64512         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00015180057 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.89         |\n",
      "|    explained_variance   | -0.0107       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0362        |\n",
      "|    n_updates            | 1250          |\n",
      "|    policy_gradient_loss | -0.00104      |\n",
      "|    value_loss           | 0.14          |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 127           |\n",
      "|    time_elapsed         | 706           |\n",
      "|    total_timesteps      | 65024         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00011345686 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.89         |\n",
      "|    explained_variance   | -0.0103       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.042         |\n",
      "|    n_updates            | 1260          |\n",
      "|    policy_gradient_loss | -0.000507     |\n",
      "|    value_loss           | 0.116         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 91           |\n",
      "|    iterations           | 128          |\n",
      "|    time_elapsed         | 712          |\n",
      "|    total_timesteps      | 65536        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0001540085 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.89        |\n",
      "|    explained_variance   | -0.0086      |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0451       |\n",
      "|    n_updates            | 1270         |\n",
      "|    policy_gradient_loss | -0.000996    |\n",
      "|    value_loss           | 0.112        |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 129           |\n",
      "|    time_elapsed         | 718           |\n",
      "|    total_timesteps      | 66048         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 7.2922325e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.89         |\n",
      "|    explained_variance   | 0.00518       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0663        |\n",
      "|    n_updates            | 1280          |\n",
      "|    policy_gradient_loss | -0.000488     |\n",
      "|    value_loss           | 0.115         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 130           |\n",
      "|    time_elapsed         | 724           |\n",
      "|    total_timesteps      | 66560         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 3.2772892e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.89         |\n",
      "|    explained_variance   | -0.0118       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0457        |\n",
      "|    n_updates            | 1290          |\n",
      "|    policy_gradient_loss | -0.000184     |\n",
      "|    value_loss           | 0.118         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 91           |\n",
      "|    iterations           | 131          |\n",
      "|    time_elapsed         | 729          |\n",
      "|    total_timesteps      | 67072        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 9.204331e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.88        |\n",
      "|    explained_variance   | 0.0129       |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0614       |\n",
      "|    n_updates            | 1300         |\n",
      "|    policy_gradient_loss | -0.000635    |\n",
      "|    value_loss           | 0.118        |\n",
      "------------------------------------------\n",
      "----------------------------------------\n",
      "| time/                   |            |\n",
      "|    fps                  | 91         |\n",
      "|    iterations           | 132        |\n",
      "|    time_elapsed         | 735        |\n",
      "|    total_timesteps      | 67584      |\n",
      "| train/                  |            |\n",
      "|    approx_kl            | 0.00345013 |\n",
      "|    clip_fraction        | 0.000586   |\n",
      "|    clip_range           | 0.2        |\n",
      "|    entropy_loss         | -1.87      |\n",
      "|    explained_variance   | 0.214      |\n",
      "|    learning_rate        | 1e-06      |\n",
      "|    loss                 | 100        |\n",
      "|    n_updates            | 1310       |\n",
      "|    policy_gradient_loss | -0.00245   |\n",
      "|    value_loss           | 332        |\n",
      "----------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 133           |\n",
      "|    time_elapsed         | 741           |\n",
      "|    total_timesteps      | 68096         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00013640325 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.86         |\n",
      "|    explained_variance   | 0.0196        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.116         |\n",
      "|    n_updates            | 1320          |\n",
      "|    policy_gradient_loss | -6.3e-06      |\n",
      "|    value_loss           | 1.31          |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 134           |\n",
      "|    time_elapsed         | 747           |\n",
      "|    total_timesteps      | 68608         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 7.3739444e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.87         |\n",
      "|    explained_variance   | 0.0186        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.115         |\n",
      "|    n_updates            | 1330          |\n",
      "|    policy_gradient_loss | 0.000112      |\n",
      "|    value_loss           | 0.635         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 135           |\n",
      "|    time_elapsed         | 753           |\n",
      "|    total_timesteps      | 69120         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 3.1510834e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.87         |\n",
      "|    explained_variance   | 0.258         |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.109         |\n",
      "|    n_updates            | 1340          |\n",
      "|    policy_gradient_loss | 0.000157      |\n",
      "|    value_loss           | 0.467         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 136           |\n",
      "|    time_elapsed         | 758           |\n",
      "|    total_timesteps      | 69632         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00048172544 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.87         |\n",
      "|    explained_variance   | -0.00964      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.117         |\n",
      "|    n_updates            | 1350          |\n",
      "|    policy_gradient_loss | -0.00241      |\n",
      "|    value_loss           | 0.445         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 137           |\n",
      "|    time_elapsed         | 765           |\n",
      "|    total_timesteps      | 70144         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00013240939 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.88         |\n",
      "|    explained_variance   | 0.226         |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.16          |\n",
      "|    n_updates            | 1360          |\n",
      "|    policy_gradient_loss | -0.000514     |\n",
      "|    value_loss           | 0.49          |\n",
      "-------------------------------------------\n",
      "--------------------------------------------\n",
      "| time/                   |                |\n",
      "|    fps                  | 91             |\n",
      "|    iterations           | 138            |\n",
      "|    time_elapsed         | 770            |\n",
      "|    total_timesteps      | 70656          |\n",
      "| train/                  |                |\n",
      "|    approx_kl            | 0.000109718996 |\n",
      "|    clip_fraction        | 0              |\n",
      "|    clip_range           | 0.2            |\n",
      "|    entropy_loss         | -1.87          |\n",
      "|    explained_variance   | -0.00452       |\n",
      "|    learning_rate        | 1e-06          |\n",
      "|    loss                 | 0.124          |\n",
      "|    n_updates            | 1370           |\n",
      "|    policy_gradient_loss | -0.000784      |\n",
      "|    value_loss           | 0.316          |\n",
      "--------------------------------------------\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 91          |\n",
      "|    iterations           | 139         |\n",
      "|    time_elapsed         | 776         |\n",
      "|    total_timesteps      | 71168       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 9.80437e-05 |\n",
      "|    clip_fraction        | 0           |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.87       |\n",
      "|    explained_variance   | -0.0078     |\n",
      "|    learning_rate        | 1e-06       |\n",
      "|    loss                 | 0.0964      |\n",
      "|    n_updates            | 1380        |\n",
      "|    policy_gradient_loss | -0.000483   |\n",
      "|    value_loss           | 0.339       |\n",
      "-----------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 140           |\n",
      "|    time_elapsed         | 782           |\n",
      "|    total_timesteps      | 71680         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00016409159 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.87         |\n",
      "|    explained_variance   | -0.00416      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0845        |\n",
      "|    n_updates            | 1390          |\n",
      "|    policy_gradient_loss | -0.00129      |\n",
      "|    value_loss           | 0.264         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 141           |\n",
      "|    time_elapsed         | 787           |\n",
      "|    total_timesteps      | 72192         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 2.6173308e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.87         |\n",
      "|    explained_variance   | -0.00201      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.107         |\n",
      "|    n_updates            | 1400          |\n",
      "|    policy_gradient_loss | -0.000113     |\n",
      "|    value_loss           | 0.271         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 91           |\n",
      "|    iterations           | 142          |\n",
      "|    time_elapsed         | 793          |\n",
      "|    total_timesteps      | 72704        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0001414587 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.87        |\n",
      "|    explained_variance   | -0.0188      |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0841       |\n",
      "|    n_updates            | 1410         |\n",
      "|    policy_gradient_loss | -0.000981    |\n",
      "|    value_loss           | 0.217        |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 91           |\n",
      "|    iterations           | 143          |\n",
      "|    time_elapsed         | 799          |\n",
      "|    total_timesteps      | 73216        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 9.213097e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.88        |\n",
      "|    explained_variance   | -0.0103      |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0862       |\n",
      "|    n_updates            | 1420         |\n",
      "|    policy_gradient_loss | -0.000535    |\n",
      "|    value_loss           | 0.124        |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 91           |\n",
      "|    iterations           | 144          |\n",
      "|    time_elapsed         | 804          |\n",
      "|    total_timesteps      | 73728        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 8.809799e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.87        |\n",
      "|    explained_variance   | -0.0367      |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0431       |\n",
      "|    n_updates            | 1430         |\n",
      "|    policy_gradient_loss | -0.000573    |\n",
      "|    value_loss           | 0.143        |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 145           |\n",
      "|    time_elapsed         | 810           |\n",
      "|    total_timesteps      | 74240         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00012414635 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.87         |\n",
      "|    explained_variance   | 0.0246        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0619        |\n",
      "|    n_updates            | 1440          |\n",
      "|    policy_gradient_loss | -0.000711     |\n",
      "|    value_loss           | 0.103         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 146           |\n",
      "|    time_elapsed         | 816           |\n",
      "|    total_timesteps      | 74752         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00030328846 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.87         |\n",
      "|    explained_variance   | -0.0201       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0435        |\n",
      "|    n_updates            | 1450          |\n",
      "|    policy_gradient_loss | -0.00162      |\n",
      "|    value_loss           | 0.1           |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 147           |\n",
      "|    time_elapsed         | 821           |\n",
      "|    total_timesteps      | 75264         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 3.1010713e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.86         |\n",
      "|    explained_variance   | 0.277         |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 1.07          |\n",
      "|    n_updates            | 1460          |\n",
      "|    policy_gradient_loss | -0.00028      |\n",
      "|    value_loss           | 1.47          |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 91           |\n",
      "|    iterations           | 148          |\n",
      "|    time_elapsed         | 827          |\n",
      "|    total_timesteps      | 75776        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0044267755 |\n",
      "|    clip_fraction        | 0.00898      |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.84        |\n",
      "|    explained_variance   | 0.227        |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 177          |\n",
      "|    n_updates            | 1470         |\n",
      "|    policy_gradient_loss | -0.000468    |\n",
      "|    value_loss           | 456          |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 149           |\n",
      "|    time_elapsed         | 833           |\n",
      "|    total_timesteps      | 76288         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00013348192 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.83         |\n",
      "|    explained_variance   | -0.0599       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0626        |\n",
      "|    n_updates            | 1480          |\n",
      "|    policy_gradient_loss | -0.000216     |\n",
      "|    value_loss           | 0.998         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 150           |\n",
      "|    time_elapsed         | 839           |\n",
      "|    total_timesteps      | 76800         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00011509168 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.84         |\n",
      "|    explained_variance   | -0.0671       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0587        |\n",
      "|    n_updates            | 1490          |\n",
      "|    policy_gradient_loss | -0.000355     |\n",
      "|    value_loss           | 0.328         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 151           |\n",
      "|    time_elapsed         | 845           |\n",
      "|    total_timesteps      | 77312         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 4.6347617e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.84         |\n",
      "|    explained_variance   | -0.0518       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0601        |\n",
      "|    n_updates            | 1500          |\n",
      "|    policy_gradient_loss | 4.88e-05      |\n",
      "|    value_loss           | 0.242         |\n",
      "-------------------------------------------\n",
      "--------------------------------------------\n",
      "| time/                   |                |\n",
      "|    fps                  | 91             |\n",
      "|    iterations           | 152            |\n",
      "|    time_elapsed         | 850            |\n",
      "|    total_timesteps      | 77824          |\n",
      "| train/                  |                |\n",
      "|    approx_kl            | 0.000102310674 |\n",
      "|    clip_fraction        | 0              |\n",
      "|    clip_range           | 0.2            |\n",
      "|    entropy_loss         | -1.85          |\n",
      "|    explained_variance   | -0.0349        |\n",
      "|    learning_rate        | 1e-06          |\n",
      "|    loss                 | 0.0626         |\n",
      "|    n_updates            | 1510           |\n",
      "|    policy_gradient_loss | -0.000219      |\n",
      "|    value_loss           | 0.184          |\n",
      "--------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 91           |\n",
      "|    iterations           | 153          |\n",
      "|    time_elapsed         | 856          |\n",
      "|    total_timesteps      | 78336        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 3.217382e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.85        |\n",
      "|    explained_variance   | -0.00822     |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0514       |\n",
      "|    n_updates            | 1520         |\n",
      "|    policy_gradient_loss | 0.000122     |\n",
      "|    value_loss           | 0.158        |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 154           |\n",
      "|    time_elapsed         | 862           |\n",
      "|    total_timesteps      | 78848         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 2.9682764e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.85         |\n",
      "|    explained_variance   | -0.0424       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.044         |\n",
      "|    n_updates            | 1530          |\n",
      "|    policy_gradient_loss | -0.000194     |\n",
      "|    value_loss           | 0.14          |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 155           |\n",
      "|    time_elapsed         | 868           |\n",
      "|    total_timesteps      | 79360         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00011462637 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.85         |\n",
      "|    explained_variance   | -0.0117       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0531        |\n",
      "|    n_updates            | 1540          |\n",
      "|    policy_gradient_loss | -0.000811     |\n",
      "|    value_loss           | 0.133         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 156           |\n",
      "|    time_elapsed         | 874           |\n",
      "|    total_timesteps      | 79872         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00012103154 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.85         |\n",
      "|    explained_variance   | -0.134        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.056         |\n",
      "|    n_updates            | 1550          |\n",
      "|    policy_gradient_loss | -0.000717     |\n",
      "|    value_loss           | 0.143         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 157           |\n",
      "|    time_elapsed         | 881           |\n",
      "|    total_timesteps      | 80384         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 5.5892626e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.85         |\n",
      "|    explained_variance   | -0.0274       |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0416        |\n",
      "|    n_updates            | 1560          |\n",
      "|    policy_gradient_loss | -0.000177     |\n",
      "|    value_loss           | 0.121         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 158           |\n",
      "|    time_elapsed         | 886           |\n",
      "|    total_timesteps      | 80896         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00013780547 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.85         |\n",
      "|    explained_variance   | 0.0373        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0441        |\n",
      "|    n_updates            | 1570          |\n",
      "|    policy_gradient_loss | -0.000818     |\n",
      "|    value_loss           | 0.139         |\n",
      "-------------------------------------------\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 91          |\n",
      "|    iterations           | 159         |\n",
      "|    time_elapsed         | 892         |\n",
      "|    total_timesteps      | 81408       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 7.57915e-05 |\n",
      "|    clip_fraction        | 0           |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.85       |\n",
      "|    explained_variance   | -0.0227     |\n",
      "|    learning_rate        | 1e-06       |\n",
      "|    loss                 | 0.0445      |\n",
      "|    n_updates            | 1580        |\n",
      "|    policy_gradient_loss | -0.000475   |\n",
      "|    value_loss           | 0.113       |\n",
      "-----------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 160           |\n",
      "|    time_elapsed         | 898           |\n",
      "|    total_timesteps      | 81920         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 3.9753388e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.85         |\n",
      "|    explained_variance   | -0.00459      |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0504        |\n",
      "|    n_updates            | 1590          |\n",
      "|    policy_gradient_loss | -0.0002       |\n",
      "|    value_loss           | 0.112         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 161           |\n",
      "|    time_elapsed         | 903           |\n",
      "|    total_timesteps      | 82432         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 3.3606193e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.85         |\n",
      "|    explained_variance   | 0.0296        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0463        |\n",
      "|    n_updates            | 1600          |\n",
      "|    policy_gradient_loss | -0.000204     |\n",
      "|    value_loss           | 0.116         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 162           |\n",
      "|    time_elapsed         | 909           |\n",
      "|    total_timesteps      | 82944         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00010504958 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.84         |\n",
      "|    explained_variance   | 0.0214        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0458        |\n",
      "|    n_updates            | 1610          |\n",
      "|    policy_gradient_loss | -0.000398     |\n",
      "|    value_loss           | 0.107         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 91           |\n",
      "|    iterations           | 163          |\n",
      "|    time_elapsed         | 915          |\n",
      "|    total_timesteps      | 83456        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0046063038 |\n",
      "|    clip_fraction        | 0.0102       |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.81        |\n",
      "|    explained_variance   | 0.215        |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 173          |\n",
      "|    n_updates            | 1620         |\n",
      "|    policy_gradient_loss | -0.000112    |\n",
      "|    value_loss           | 408          |\n",
      "------------------------------------------\n",
      "-----------------------------------------\n",
      "| time/                   |             |\n",
      "|    fps                  | 91          |\n",
      "|    iterations           | 164         |\n",
      "|    time_elapsed         | 920         |\n",
      "|    total_timesteps      | 83968       |\n",
      "| train/                  |             |\n",
      "|    approx_kl            | 0.000514352 |\n",
      "|    clip_fraction        | 0           |\n",
      "|    clip_range           | 0.2         |\n",
      "|    entropy_loss         | -1.77       |\n",
      "|    explained_variance   | 0.386       |\n",
      "|    learning_rate        | 1e-06       |\n",
      "|    loss                 | 50          |\n",
      "|    n_updates            | 1630        |\n",
      "|    policy_gradient_loss | 0.00171     |\n",
      "|    value_loss           | 80          |\n",
      "-----------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 165           |\n",
      "|    time_elapsed         | 926           |\n",
      "|    total_timesteps      | 84480         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00012674357 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.78         |\n",
      "|    explained_variance   | 0.0417        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.105         |\n",
      "|    n_updates            | 1640          |\n",
      "|    policy_gradient_loss | 0.000535      |\n",
      "|    value_loss           | 0.896         |\n",
      "-------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 166           |\n",
      "|    time_elapsed         | 932           |\n",
      "|    total_timesteps      | 84992         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 9.1804424e-05 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.79         |\n",
      "|    explained_variance   | 0.0366        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0737        |\n",
      "|    n_updates            | 1650          |\n",
      "|    policy_gradient_loss | 0.000644      |\n",
      "|    value_loss           | 0.622         |\n",
      "-------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 91           |\n",
      "|    iterations           | 167          |\n",
      "|    time_elapsed         | 938          |\n",
      "|    total_timesteps      | 85504        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 3.862346e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.78        |\n",
      "|    explained_variance   | 0.0114       |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.118        |\n",
      "|    n_updates            | 1660         |\n",
      "|    policy_gradient_loss | 8.61e-05     |\n",
      "|    value_loss           | 0.421        |\n",
      "------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 91           |\n",
      "|    iterations           | 168          |\n",
      "|    time_elapsed         | 944          |\n",
      "|    total_timesteps      | 86016        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 0.0003198809 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.79        |\n",
      "|    explained_variance   | 0.112        |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.0489       |\n",
      "|    n_updates            | 1670         |\n",
      "|    policy_gradient_loss | -0.000571    |\n",
      "|    value_loss           | 0.291        |\n",
      "------------------------------------------\n",
      "-------------------------------------------\n",
      "| time/                   |               |\n",
      "|    fps                  | 91            |\n",
      "|    iterations           | 169           |\n",
      "|    time_elapsed         | 950           |\n",
      "|    total_timesteps      | 86528         |\n",
      "| train/                  |               |\n",
      "|    approx_kl            | 0.00013973413 |\n",
      "|    clip_fraction        | 0             |\n",
      "|    clip_range           | 0.2           |\n",
      "|    entropy_loss         | -1.8          |\n",
      "|    explained_variance   | 0.0444        |\n",
      "|    learning_rate        | 1e-06         |\n",
      "|    loss                 | 0.0635        |\n",
      "|    n_updates            | 1680          |\n",
      "|    policy_gradient_loss | -0.000389     |\n",
      "|    value_loss           | 0.244         |\n",
      "-------------------------------------------\n",
      "--------------------------------------------\n",
      "| time/                   |                |\n",
      "|    fps                  | 91             |\n",
      "|    iterations           | 170            |\n",
      "|    time_elapsed         | 955            |\n",
      "|    total_timesteps      | 87040          |\n",
      "| train/                  |                |\n",
      "|    approx_kl            | 0.000110175926 |\n",
      "|    clip_fraction        | 0              |\n",
      "|    clip_range           | 0.2            |\n",
      "|    entropy_loss         | -1.8           |\n",
      "|    explained_variance   | 0.0372         |\n",
      "|    learning_rate        | 1e-06          |\n",
      "|    loss                 | 0.0517         |\n",
      "|    n_updates            | 1690           |\n",
      "|    policy_gradient_loss | -0.000167      |\n",
      "|    value_loss           | 0.167          |\n",
      "--------------------------------------------\n",
      "------------------------------------------\n",
      "| time/                   |              |\n",
      "|    fps                  | 91           |\n",
      "|    iterations           | 171          |\n",
      "|    time_elapsed         | 961          |\n",
      "|    total_timesteps      | 87552        |\n",
      "| train/                  |              |\n",
      "|    approx_kl            | 4.323444e-05 |\n",
      "|    clip_fraction        | 0            |\n",
      "|    clip_range           | 0.2          |\n",
      "|    entropy_loss         | -1.8         |\n",
      "|    explained_variance   | 0.106        |\n",
      "|    learning_rate        | 1e-06        |\n",
      "|    loss                 | 0.115        |\n",
      "|    n_updates            | 1700         |\n",
      "|    policy_gradient_loss | -5.78e-05    |\n",
      "|    value_loss           | 0.238        |\n",
      "------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Train the AI model, this is where the AI model starts to learn\n",
    "# timesteps is how many frames our model is going to be able to see\n",
    "# imagine every single move is a frame\n",
    "# NOTE: In terminal, tensorboard --logdir=runs\n",
    "model.learn(total_timesteps=4000000, callback=callback)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1e73e68-5511-4e1c-86df-2196a67bf111",
   "metadata": {},
   "source": [
    "## Test it Out\n",
    "\n",
    "Trained the DRL agent's DL model over steps\n",
    "\n",
    "- 500K is where mario started doing better\n",
    "- 1M is where mario fell into holes\n",
    "- 2M & 3M similar as above, but went longer too\n",
    "- 4M is where mario was able to finish the level"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "1aae2c08-3eaa-4c16-afa9-49fe6467cd98",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = PPO.load('./train/best_model_110000.zip')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "b443b237-4d60-4f2c-95a6-ce3224392c0b",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/james/miniconda3/envs/cmpe260_dev/lib/python3.10/site-packages/gym_super_mario_bros/smb_env.py:148: RuntimeWarning: overflow encountered in ubyte_scalars\n",
      "  return (self.ram[0x86] - self.ram[0x071c]) % 256\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[19], line 5\u001b[0m\n\u001b[1;32m      3\u001b[0m action, state \u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mpredict(state)\n\u001b[1;32m      4\u001b[0m state, reward, done, info \u001b[38;5;241m=\u001b[39m env\u001b[38;5;241m.\u001b[39mstep(action)\n\u001b[0;32m----> 5\u001b[0m \u001b[43menv\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrender\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/envs/cmpe260_dev/lib/python3.10/site-packages/stable_baselines3/common/vec_env/base_vec_env.py:281\u001b[0m, in \u001b[0;36mVecEnvWrapper.render\u001b[0;34m(self, mode)\u001b[0m\n\u001b[1;32m    280\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mrender\u001b[39m(\u001b[38;5;28mself\u001b[39m, mode: \u001b[38;5;28mstr\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mhuman\u001b[39m\u001b[38;5;124m\"\u001b[39m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Optional[np\u001b[38;5;241m.\u001b[39mndarray]:\n\u001b[0;32m--> 281\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mvenv\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrender\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmode\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmode\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/envs/cmpe260_dev/lib/python3.10/site-packages/stable_baselines3/common/vec_env/dummy_vec_env.py:85\u001b[0m, in \u001b[0;36mDummyVecEnv.render\u001b[0;34m(self, mode)\u001b[0m\n\u001b[1;32m     73\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m     74\u001b[0m \u001b[38;5;124;03mGym environment rendering. If there are multiple environments then\u001b[39;00m\n\u001b[1;32m     75\u001b[0m \u001b[38;5;124;03mthey are tiled together in one image via ``BaseVecEnv.render()``.\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     82\u001b[0m \u001b[38;5;124;03m:param mode: The rendering type.\u001b[39;00m\n\u001b[1;32m     83\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m     84\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mnum_envs \u001b[38;5;241m==\u001b[39m \u001b[38;5;241m1\u001b[39m:\n\u001b[0;32m---> 85\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43menvs\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m]\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrender\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmode\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmode\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     86\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m     87\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28msuper\u001b[39m()\u001b[38;5;241m.\u001b[39mrender(mode\u001b[38;5;241m=\u001b[39mmode)\n",
      "File \u001b[0;32m~/miniconda3/envs/cmpe260_dev/lib/python3.10/site-packages/gym/core.py:295\u001b[0m, in \u001b[0;36mWrapper.render\u001b[0;34m(self, mode, **kwargs)\u001b[0m\n\u001b[1;32m    294\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mrender\u001b[39m(\u001b[38;5;28mself\u001b[39m, mode\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mhuman\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[0;32m--> 295\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43menv\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrender\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmode\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/envs/cmpe260_dev/lib/python3.10/site-packages/gym/core.py:295\u001b[0m, in \u001b[0;36mWrapper.render\u001b[0;34m(self, mode, **kwargs)\u001b[0m\n\u001b[1;32m    294\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mrender\u001b[39m(\u001b[38;5;28mself\u001b[39m, mode\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mhuman\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[0;32m--> 295\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43menv\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrender\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmode\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/envs/cmpe260_dev/lib/python3.10/site-packages/gym/core.py:295\u001b[0m, in \u001b[0;36mWrapper.render\u001b[0;34m(self, mode, **kwargs)\u001b[0m\n\u001b[1;32m    294\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mrender\u001b[39m(\u001b[38;5;28mself\u001b[39m, mode\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mhuman\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs):\n\u001b[0;32m--> 295\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43menv\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrender\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmode\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/envs/cmpe260_dev/lib/python3.10/site-packages/nes_py/nes_env.py:386\u001b[0m, in \u001b[0;36mNESEnv.render\u001b[0;34m(self, mode)\u001b[0m\n\u001b[1;32m    380\u001b[0m         \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mviewer \u001b[38;5;241m=\u001b[39m ImageViewer(\n\u001b[1;32m    381\u001b[0m             caption\u001b[38;5;241m=\u001b[39mcaption,\n\u001b[1;32m    382\u001b[0m             height\u001b[38;5;241m=\u001b[39mSCREEN_HEIGHT,\n\u001b[1;32m    383\u001b[0m             width\u001b[38;5;241m=\u001b[39mSCREEN_WIDTH,\n\u001b[1;32m    384\u001b[0m         )\n\u001b[1;32m    385\u001b[0m     \u001b[38;5;66;03m# show the screen on the image viewer\u001b[39;00m\n\u001b[0;32m--> 386\u001b[0m     \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mviewer\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mshow\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mscreen\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    387\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m mode \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mrgb_array\u001b[39m\u001b[38;5;124m'\u001b[39m:\n\u001b[1;32m    388\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mscreen\n",
      "File \u001b[0;32m~/miniconda3/envs/cmpe260_dev/lib/python3.10/site-packages/nes_py/_image_viewer.py:149\u001b[0m, in \u001b[0;36mImageViewer.show\u001b[0;34m(self, frame)\u001b[0m\n\u001b[1;32m    147\u001b[0m \u001b[38;5;66;03m# send the image to the window\u001b[39;00m\n\u001b[1;32m    148\u001b[0m image\u001b[38;5;241m.\u001b[39mblit(\u001b[38;5;241m0\u001b[39m, \u001b[38;5;241m0\u001b[39m, width\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_window\u001b[38;5;241m.\u001b[39mwidth, height\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_window\u001b[38;5;241m.\u001b[39mheight)\n\u001b[0;32m--> 149\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_window\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mflip\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/envs/cmpe260_dev/lib/python3.10/site-packages/pyglet/window/xlib/__init__.py:536\u001b[0m, in \u001b[0;36mXlibWindow.flip\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    534\u001b[0m \u001b[38;5;66;03m# TODO canvas.flip?\u001b[39;00m\n\u001b[1;32m    535\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcontext:\n\u001b[0;32m--> 536\u001b[0m     \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcontext\u001b[49m\u001b[38;5;241m.\u001b[39mflip()\n\u001b[1;32m    538\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_sync_resize()\n",
      "File \u001b[0;32m~/miniconda3/envs/cmpe260_dev/lib/python3.10/site-packages/pyglet/window/__init__.py:974\u001b[0m, in \u001b[0;36mBaseWindow.context\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    968\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"A GL config describing the context of this window.  Read-only.\u001b[39;00m\n\u001b[1;32m    969\u001b[0m \n\u001b[1;32m    970\u001b[0m \u001b[38;5;124;03m    :type: :py:class:`pyglet.gl.Config`\u001b[39;00m\n\u001b[1;32m    971\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[1;32m    972\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_config\n\u001b[0;32m--> 974\u001b[0m \u001b[38;5;129m@property\u001b[39m\n\u001b[1;32m    975\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mcontext\u001b[39m(\u001b[38;5;28mself\u001b[39m):\n\u001b[1;32m    976\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"The OpenGL context attached to this window.  Read-only.\u001b[39;00m\n\u001b[1;32m    977\u001b[0m \n\u001b[1;32m    978\u001b[0m \u001b[38;5;124;03m    :type: :py:class:`pyglet.gl.Context`\u001b[39;00m\n\u001b[1;32m    979\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[1;32m    980\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_context\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "state = env.reset()\n",
    "while True:\n",
    "    action, state = model.predict(state)\n",
    "    state, reward, done, info = env.step(action)\n",
    "    env.render()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "38528ed4-0ade-403a-a8c7-5946b6715a78",
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bbb0219-f4bc-4e76-90c6-dc41f439d635",
   "metadata": {},
   "source": [
    "## Ways to Improve Performance\n",
    "\n",
    "Ways to improve our DRL Agent is to train it longer with a slower learning rate. This means it will take us longer to teach our model how to actually perform in a game, but ideally it should end up in a better outcome.\n",
    "\n",
    "We could also try multi-processing, this means that our AI model plays multiple games, which should effectively mean that it plays faster.\n",
    "\n",
    "Last thing that we can try is a different algorithm.\n",
    "\n",
    "What game is next?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ce63e36-b563-4e0c-9f4e-ef208b72c384",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}