09_Stable_Baselines_BC.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"09_Stable_Baselines_BC.ipynb","provenance":[{"file_id":"1NzMruWKS1LSK3OEKH6NyameU8esi9vT6","timestamp":1590356208060}],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"KoWCFFLxt0pI","colab_type":"text"},"source":["<a href=\"https://colab.research.google.com/drive/1HWqta3KrI_x_Gr5ugy6B7cfMTrvDPRQC?usp=sharing\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n","\n","By [Ibrahim Sobh](https://www.linkedin.com/in/ibrahim-sobh-phd-8681757/)\n"]},{"cell_type":"markdown","metadata":{"id":"uzSsqB7Qymsj","colab_type":"text"},"source":["# Behavior Cloning\n","\n","Using expert demonstrations (observations-actions pairs), as a supervised learning problem. The policy network is trained to reproduce the expert behavior. Expert trajectories can be:\n","- Human demonstrations\n","- Trajectories from another controller (e.g. a PID controller)\n","- Trajectories from a trained RL agent\n","\n","--- \n","\n","Steps:  \n","- Generate and save trajectories (ex: using a trained DQN agent)\n","- Load expert trajectories\n","- Initialize RL agent \n","   - Evaluate the untainted agent \n","- Pretrain the RL model (ex: PPO2) in a supervised way\n","   - Evaluate the pre-trained PPO2 model  \n","- Train the RL model for self improvement \n","   - Evaluate the final RL model \n"]},{"cell_type":"markdown","metadata":{"id":"X8MpsMtnyrk4","colab_type":"text"},"source":["## Install"]},{"cell_type":"code","metadata":{"id":"AK88-CxtIzwm","colab_type":"code","outputId":"9477e510-0f60-44bd-f695-272288b9c9e1","executionInfo":{"status":"ok","timestamp":1582019783168,"user_tz":-120,"elapsed":21097,"user":{"displayName":"Ibrahim SOBH","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAYeQM_1LvthZBu6TRjB85IH8s-a5ZMla2NYwtX=s64","userId":"05222794542711757573"}},"colab":{"base_uri":"https://localhost:8080/","height":1000}},"source":["!pip install pyvirtualdisplay \n","!apt-get install -y xvfb python-opengl ffmpeg\n","!pip install gym\n","!pip install box2d-py\n","# !pip install pyglet==1.3.2\n","!pip install pyglet\n","!pip install stable-baselines\n","!pip install stable-baselines --upgrade"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Requirement already satisfied: pyvirtualdisplay in /usr/local/lib/python3.6/dist-packages (0.2.5)\n","Requirement already satisfied: EasyProcess in /usr/local/lib/python3.6/dist-packages (from pyvirtualdisplay) (0.2.10)\n","Reading package lists... Done\n","Building dependency tree       \n","Reading state information... Done\n","python-opengl is already the newest version (3.1.0+dfsg-1).\n","ffmpeg is already the newest version (7:3.4.6-0ubuntu0.18.04.1).\n","xvfb is already the newest version (2:1.19.6-1ubuntu4.3).\n","The following package was automatically installed and is no longer required:\n","  libnvidia-common-430\n","Use 'apt autoremove' to remove it.\n","0 upgraded, 0 newly installed, 0 to remove and 25 not upgraded.\n","Requirement already satisfied: gym in /usr/local/lib/python3.6/dist-packages (0.15.6)\n","Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.6/dist-packages (from gym) (1.4.10)\n","Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from gym) (1.12.0)\n","Requirement already satisfied: cloudpickle~=1.2.0 in /usr/local/lib/python3.6/dist-packages (from gym) (1.2.2)\n","Requirement already satisfied: numpy>=1.10.4 in /usr/local/lib/python3.6/dist-packages (from gym) (1.17.5)\n","Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from gym) (1.4.1)\n","Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from pyglet<=1.5.0,>=1.4.0->gym) (0.16.0)\n","Requirement already satisfied: box2d-py in /usr/local/lib/python3.6/dist-packages (2.3.8)\n","Requirement already satisfied: pyglet in /usr/local/lib/python3.6/dist-packages (1.4.10)\n","Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from pyglet) (0.16.0)\n","Requirement already satisfied: stable-baselines in /usr/local/lib/python3.6/dist-packages (2.9.0)\n","Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from stable-baselines) (1.4.1)\n","Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from stable-baselines) (0.25.3)\n","Requirement already satisfied: cloudpickle>=0.5.5 in /usr/local/lib/python3.6/dist-packages (from stable-baselines) (1.2.2)\n","Requirement already satisfied: gym[atari,classic_control]>=0.10.9 in /usr/local/lib/python3.6/dist-packages (from stable-baselines) (0.15.6)\n","Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from stable-baselines) (1.17.5)\n","Requirement already satisfied: opencv-python in /usr/local/lib/python3.6/dist-packages (from stable-baselines) (4.1.2.30)\n","Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from stable-baselines) (3.1.3)\n","Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from stable-baselines) (0.14.1)\n","Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas->stable-baselines) (2.6.1)\n","Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->stable-baselines) (2018.9)\n","Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from gym[atari,classic_control]>=0.10.9->stable-baselines) (1.12.0)\n","Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.6/dist-packages (from gym[atari,classic_control]>=0.10.9->stable-baselines) (1.4.10)\n","Requirement already satisfied: atari-py~=0.2.0; extra == \"atari\" in /usr/local/lib/python3.6/dist-packages (from gym[atari,classic_control]>=0.10.9->stable-baselines) (0.2.6)\n","Requirement already satisfied: Pillow; extra == \"atari\" in /usr/local/lib/python3.6/dist-packages (from gym[atari,classic_control]>=0.10.9->stable-baselines) (6.2.2)\n","Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->stable-baselines) (2.4.6)\n","Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->stable-baselines) (0.10.0)\n","Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->stable-baselines) (1.1.0)\n","Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from pyglet<=1.5.0,>=1.4.0->gym[atari,classic_control]>=0.10.9->stable-baselines) (0.16.0)\n","Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from kiwisolver>=1.0.1->matplotlib->stable-baselines) (45.1.0)\n","Requirement already up-to-date: stable-baselines in /usr/local/lib/python3.6/dist-packages (2.9.0)\n","Requirement already satisfied, skipping upgrade: cloudpickle>=0.5.5 in /usr/local/lib/python3.6/dist-packages (from stable-baselines) (1.2.2)\n","Requirement already satisfied, skipping upgrade: matplotlib in /usr/local/lib/python3.6/dist-packages (from stable-baselines) (3.1.3)\n","Requirement already satisfied, skipping upgrade: scipy in /usr/local/lib/python3.6/dist-packages (from stable-baselines) (1.4.1)\n","Requirement already satisfied, skipping upgrade: joblib in /usr/local/lib/python3.6/dist-packages (from stable-baselines) (0.14.1)\n","Requirement already satisfied, skipping upgrade: gym[atari,classic_control]>=0.10.9 in /usr/local/lib/python3.6/dist-packages (from stable-baselines) (0.15.6)\n","Requirement already satisfied, skipping upgrade: numpy in /usr/local/lib/python3.6/dist-packages (from stable-baselines) (1.17.5)\n","Requirement already satisfied, skipping upgrade: opencv-python in /usr/local/lib/python3.6/dist-packages (from stable-baselines) (4.1.2.30)\n","Requirement already satisfied, skipping upgrade: pandas in /usr/local/lib/python3.6/dist-packages (from stable-baselines) (0.25.3)\n","Requirement already satisfied, skipping upgrade: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->stable-baselines) (2.4.6)\n","Requirement already satisfied, skipping upgrade: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->stable-baselines) (1.1.0)\n","Requirement already satisfied, skipping upgrade: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->stable-baselines) (2.6.1)\n","Requirement already satisfied, skipping upgrade: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->stable-baselines) (0.10.0)\n","Requirement already satisfied, skipping upgrade: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.6/dist-packages (from gym[atari,classic_control]>=0.10.9->stable-baselines) (1.4.10)\n","Requirement already satisfied, skipping upgrade: six in /usr/local/lib/python3.6/dist-packages (from gym[atari,classic_control]>=0.10.9->stable-baselines) (1.12.0)\n","Requirement already satisfied, skipping upgrade: Pillow; extra == \"atari\" in /usr/local/lib/python3.6/dist-packages (from gym[atari,classic_control]>=0.10.9->stable-baselines) (6.2.2)\n","Requirement already satisfied, skipping upgrade: atari-py~=0.2.0; extra == \"atari\" in /usr/local/lib/python3.6/dist-packages (from gym[atari,classic_control]>=0.10.9->stable-baselines) (0.2.6)\n","Requirement already satisfied, skipping upgrade: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->stable-baselines) (2018.9)\n","Requirement already satisfied, skipping upgrade: setuptools in /usr/local/lib/python3.6/dist-packages (from kiwisolver>=1.0.1->matplotlib->stable-baselines) (45.1.0)\n","Requirement already satisfied, skipping upgrade: future in /usr/local/lib/python3.6/dist-packages (from pyglet<=1.5.0,>=1.4.0->gym[atari,classic_control]>=0.10.9->stable-baselines) (0.16.0)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"6NlznQ0_y2jv","colab_type":"text"},"source":["## Generate and save trajectories"]},{"cell_type":"code","metadata":{"id":"u6bC_jjPyfoC","colab_type":"code","outputId":"2854b9a6-9f7d-4ee4-b661-faee72555802","executionInfo":{"status":"ok","timestamp":1582019837086,"user_tz":-120,"elapsed":74999,"user":{"displayName":"Ibrahim SOBH","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAYeQM_1LvthZBu6TRjB85IH8s-a5ZMla2NYwtX=s64","userId":"05222794542711757573"}},"colab":{"base_uri":"https://localhost:8080/","height":272}},"source":["# Train a DQN agent and generate and save trajectories\n","from stable_baselines import DQN\n","from stable_baselines.gail import generate_expert_traj\n","\n","model = DQN('MlpPolicy', 'CartPole-v0', verbose=1)\n","generate_expert_traj(model, 'expert_cartpole', n_timesteps=int(1e4), n_episodes=10)\n","\n","results_mean_list = [] \n","results_std_list = [] "],"execution_count":0,"outputs":[{"output_type":"stream","text":["Creating environment from the given name, wrapped in a DummyVecEnv.\n"],"name":"stdout"},{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/gym/logger.py:30: UserWarning: \u001b[33mWARN: Box bound precision lowered by casting to float32\u001b[0m\n","  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))\n"],"name":"stderr"},{"output_type":"stream","text":["--------------------------------------\n","| % time spent exploring  | 2        |\n","| episodes                | 100      |\n","| mean 100 episode reward | 19.4     |\n","| steps                   | 1917     |\n","--------------------------------------\n","actions (1136, 1)\n","obs (1136, 4)\n","rewards (1136,)\n","episode_returns (10,)\n","episode_starts (1136,)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"4_cL6yu5y7l2","colab_type":"text"},"source":["## Load expert trajectories"]},{"cell_type":"code","metadata":{"id":"67BI65Q1sWoS","colab_type":"code","outputId":"ebef4d04-b0ed-4d6f-cd04-87546f8ed5cd","executionInfo":{"status":"ok","timestamp":1582019837093,"user_tz":-120,"elapsed":74992,"user":{"displayName":"Ibrahim SOBH","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAYeQM_1LvthZBu6TRjB85IH8s-a5ZMla2NYwtX=s64","userId":"05222794542711757573"}},"colab":{"base_uri":"https://localhost:8080/","height":181}},"source":["from stable_baselines import PPO2\n","from stable_baselines.gail import ExpertDataset\n","# traj_limitation=-1  -> whole dataset\n","dataset = ExpertDataset(expert_path='expert_cartpole.npz',\n","                        traj_limitation=1, batch_size=128)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["actions (1136, 1)\n","obs (1136, 4)\n","rewards (1136,)\n","episode_returns (10,)\n","episode_starts (1136,)\n","Total trajectories: 1\n","Total transitions: 212\n","Average returns: 113.6\n","Std for returns: 7.977468270071653\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"L4MwnYHWxQsf","colab_type":"text"},"source":["## Evalaute the random model"]},{"cell_type":"code","metadata":{"id":"pyX4XcTwxKfm","colab_type":"code","outputId":"e3ae7e0b-6e99-46a1-b9b0-b21f25eea4d1","executionInfo":{"status":"ok","timestamp":1582019837469,"user_tz":-120,"elapsed":75347,"user":{"displayName":"Ibrahim SOBH","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAYeQM_1LvthZBu6TRjB85IH8s-a5ZMla2NYwtX=s64","userId":"05222794542711757573"}},"colab":{"base_uri":"https://localhost:8080/","height":72}},"source":["model = PPO2('MlpPolicy', 'CartPole-v0', verbose=1)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Creating environment from the given name, wrapped in a DummyVecEnv.\n"],"name":"stdout"},{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/gym/logger.py:30: UserWarning: \u001b[33mWARN: Box bound precision lowered by casting to float32\u001b[0m\n","  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))\n"],"name":"stderr"}]},{"cell_type":"code","metadata":{"id":"-XhbIB_wxKiO","colab_type":"code","outputId":"1373cfa2-af81-4b80-8b0c-5a2110f60ae3","executionInfo":{"status":"ok","timestamp":1582019851814,"user_tz":-120,"elapsed":89677,"user":{"displayName":"Ibrahim SOBH","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAYeQM_1LvthZBu6TRjB85IH8s-a5ZMla2NYwtX=s64","userId":"05222794542711757573"}},"colab":{"base_uri":"https://localhost:8080/","height":72}},"source":["import numpy as np\n","# Test The random model\n","\n","env = model.get_env()\n","obs = env.reset()\n","r_list = []\n","\n","for i in range(10):\n","  print(\"\\riteration: {}\".format(i), end=\"\")\n","  reward_sum = 0.0\n","  for _ in range(1000):\n","          action, _ = model.predict(obs)\n","          obs, reward, done, _ = env.step(action)\n","          reward_sum += reward\n","          if done:\n","                  r_list.append(reward_sum)\n","                  reward_sum = 0.0\n","                  obs = env.reset()\n","\n","print('\\nmean, std')\n","print(np.mean(r_list), np.std(r_list))\n","results_mean_list.append(np.mean(r_list))\n","results_std_list.append(np.std(r_list)) \n","env.close()"],"execution_count":0,"outputs":[{"output_type":"stream","text":["iteration: 9\n","mean, std\n","22.29054 11.437996\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"q6BGXrD4zPKw","colab_type":"text"},"source":["## Pretrain the PPO2 model"]},{"cell_type":"code","metadata":{"id":"0XpjGCbpzQQN","colab_type":"code","outputId":"62582f4d-631c-4546-b7da-e5c9fd92d4c2","executionInfo":{"status":"ok","timestamp":1582019861801,"user_tz":-120,"elapsed":99646,"user":{"displayName":"Ibrahim SOBH","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAYeQM_1LvthZBu6TRjB85IH8s-a5ZMla2NYwtX=s64","userId":"05222794542711757573"}},"colab":{"base_uri":"https://localhost:8080/","height":799}},"source":["model.pretrain(dataset, n_epochs=2000)"],"execution_count":0,"outputs":[{"output_type":"stream","text":["Pretraining with Behavior Cloning...\n","==== Training progress 10.00% ====\n","Epoch 200\n","Training loss: 0.628646, Validation loss: 0.604446\n","\n","==== Training progress 20.00% ====\n","Epoch 400\n","Training loss: 0.554622, Validation loss: 0.533716\n","\n","==== Training progress 30.00% ====\n","Epoch 600\n","Training loss: 0.505957, Validation loss: 0.511930\n","\n","==== Training progress 40.00% ====\n","Epoch 800\n","Training loss: 0.501154, Validation loss: 0.492628\n","\n","==== Training progress 50.00% ====\n","Epoch 1000\n","Training loss: 0.524606, Validation loss: 0.476109\n","\n","==== Training progress 60.00% ====\n","Epoch 1200\n","Training loss: 0.549335, Validation loss: 0.439680\n","\n","==== Training progress 70.00% ====\n","Epoch 1400\n","Training loss: 0.410501, Validation loss: 0.419609\n","\n","==== Training progress 80.00% ====\n","Epoch 1600\n","Training loss: 0.371578, Validation loss: 0.403743\n","\n","==== Training progress 90.00% ====\n","Epoch 1800\n","Training loss: 0.389727, Validation loss: 0.400651\n","\n","==== Training progress 100.00% ====\n","Epoch 2000\n","Training loss: 0.338221, Validation loss: 0.369560\n","\n","Pretraining done.\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["<stable_baselines.ppo2.ppo2.PPO2 at 0x7feb7c7f0550>"]},"metadata":{"tags":[]},"execution_count":16}]},{"cell_type":"code","metadata":{"id":"A_mOcVyszQSl","colab_type":"code","outputId":"aaa29747-90ff-4673-ecc2-a800563a96ac","executionInfo":{"status":"ok","timestamp":1582019874968,"user_tz":-120,"elapsed":112796,"user":{"displayName":"Ibrahim SOBH","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAYeQM_1LvthZBu6TRjB85IH8s-a5ZMla2NYwtX=s64","userId":"05222794542711757573"}},"colab":{"base_uri":"https://localhost:8080/","height":72}},"source":["# test the pretrained PPO model\n","env = model.get_env()\n","obs = env.reset()\n","r_list = []\n","\n","for i in range(10):\n","  print(\"\\riteration: {}\".format(i), end=\"\")\n","  reward_sum = 0.0\n","  for _ in range(1000):\n","          action, _ = model.predict(obs)\n","          obs, reward, done, _ = env.step(action)\n","          reward_sum += reward\n","          if done:\n","                  r_list.append(reward_sum)\n","                  reward_sum = 0.0\n","                  obs = env.reset()\n","\n","print('\\nmean, std')\n","print(np.mean(r_list), np.std(r_list))\n","results_mean_list.append(np.mean(r_list))\n","results_std_list.append(np.std(r_list)) \n","env.close()"],"execution_count":0,"outputs":[{"output_type":"stream","text":["iteration: 9\n","mean, std\n","67.86331 42.964966\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"emJBepyHzPSH","colab_type":"text"},"source":["## Train the RL agent"]},{"cell_type":"code","metadata":{"id":"6lv-jK4NzQ0w","colab_type":"code","outputId":"c93f348d-8cfd-4dc8-f70a-5bbe3199f362","executionInfo":{"status":"ok","timestamp":1582020075646,"user_tz":-120,"elapsed":313460,"user":{"displayName":"Ibrahim SOBH","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAYeQM_1LvthZBu6TRjB85IH8s-a5ZMla2NYwtX=s64","userId":"05222794542711757573"}},"colab":{"base_uri":"https://localhost:8080/","height":1000}},"source":["model.learn(int(1e5))"],"execution_count":0,"outputs":[{"output_type":"stream","text":["\u001b[1;30;43mStreaming output truncated to the last 5000 lines.\u001b[0m\n","| policy_entropy     | 0.4632867    |\n","| policy_loss        | -0.011270072 |\n","| serial_timesteps   | 50816        |\n","| time_elapsed       | 102          |\n","| total_timesteps    | 50816        |\n","| value_loss         | 0.36167085   |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0010593967  |\n","| clipfrac           | 0.00390625    |\n","| explained_variance | 0.994         |\n","| fps                | 491           |\n","| n_updates          | 398           |\n","| policy_entropy     | 0.4422183     |\n","| policy_loss        | -0.0013926309 |\n","| serial_timesteps   | 50944         |\n","| time_elapsed       | 102           |\n","| total_timesteps    | 50944         |\n","| value_loss         | 1.8285588     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0012952393  |\n","| clipfrac           | 0.015625      |\n","| explained_variance | 0.993         |\n","| fps                | 516           |\n","| n_updates          | 399           |\n","| policy_entropy     | 0.43209112    |\n","| policy_loss        | -0.0048839655 |\n","| serial_timesteps   | 51072         |\n","| time_elapsed       | 103           |\n","| total_timesteps    | 51072         |\n","| value_loss         | 2.2660174     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0054992186  |\n","| clipfrac           | 0.0703125     |\n","| explained_variance | 0.996         |\n","| fps                | 526           |\n","| n_updates          | 400           |\n","| policy_entropy     | 0.46304616    |\n","| policy_loss        | -0.0036709555 |\n","| serial_timesteps   | 51200         |\n","| time_elapsed       | 103           |\n","| total_timesteps    | 51200         |\n","| value_loss         | 0.7448086     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00082828186 |\n","| clipfrac           | 0.00390625    |\n","| explained_variance | 0.944         |\n","| fps                | 441           |\n","| n_updates          | 401           |\n","| policy_entropy     | 0.4706065     |\n","| policy_loss        | 0.0039258455  |\n","| serial_timesteps   | 51328         |\n","| time_elapsed       | 103           |\n","| total_timesteps    | 51328         |\n","| value_loss         | 15.061535     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 6.6983426e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.912         |\n","| fps                | 402           |\n","| n_updates          | 402           |\n","| policy_entropy     | 0.44181252    |\n","| policy_loss        | 0.00065535924 |\n","| serial_timesteps   | 51456         |\n","| time_elapsed       | 104           |\n","| total_timesteps    | 51456         |\n","| value_loss         | 21.175472     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.001569961  |\n","| clipfrac           | 0.013671875  |\n","| explained_variance | 0.992        |\n","| fps                | 381          |\n","| n_updates          | 403          |\n","| policy_entropy     | 0.44903454   |\n","| policy_loss        | 0.0033258786 |\n","| serial_timesteps   | 51584        |\n","| time_elapsed       | 104          |\n","| total_timesteps    | 51584        |\n","| value_loss         | 5.5930133    |\n","-------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0006131056 |\n","| clipfrac           | 0.00390625   |\n","| explained_variance | 0.51         |\n","| fps                | 510          |\n","| n_updates          | 404          |\n","| policy_entropy     | 0.45671606   |\n","| policy_loss        | 0.0020137816 |\n","| serial_timesteps   | 51712        |\n","| time_elapsed       | 104          |\n","| total_timesteps    | 51712        |\n","| value_loss         | 153.6051     |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00040827526 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.881         |\n","| fps                | 527           |\n","| n_updates          | 405           |\n","| policy_entropy     | 0.46814692    |\n","| policy_loss        | -0.0015238449 |\n","| serial_timesteps   | 51840         |\n","| time_elapsed       | 104           |\n","| total_timesteps    | 51840         |\n","| value_loss         | 29.546276     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.002727983   |\n","| clipfrac           | 0.02734375    |\n","| explained_variance | 0.989         |\n","| fps                | 428           |\n","| n_updates          | 406           |\n","| policy_entropy     | 0.46707597    |\n","| policy_loss        | -0.0058432575 |\n","| serial_timesteps   | 51968         |\n","| time_elapsed       | 105           |\n","| total_timesteps    | 51968         |\n","| value_loss         | 3.050957      |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 0.00031993954  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.994          |\n","| fps                | 507            |\n","| n_updates          | 407            |\n","| policy_entropy     | 0.42414102     |\n","| policy_loss        | -0.00056949886 |\n","| serial_timesteps   | 52096          |\n","| time_elapsed       | 105            |\n","| total_timesteps    | 52096          |\n","| value_loss         | 4.7083044      |\n","---------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0021210338 |\n","| clipfrac           | 0.03515625   |\n","| explained_variance | 0.975        |\n","| fps                | 504          |\n","| n_updates          | 408          |\n","| policy_entropy     | 0.47147143   |\n","| policy_loss        | 0.01054852   |\n","| serial_timesteps   | 52224        |\n","| time_elapsed       | 105          |\n","| total_timesteps    | 52224        |\n","| value_loss         | 6.4391627    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.002838049   |\n","| clipfrac           | 0.044921875   |\n","| explained_variance | 0.997         |\n","| fps                | 448           |\n","| n_updates          | 409           |\n","| policy_entropy     | 0.49514496    |\n","| policy_loss        | -0.0033166166 |\n","| serial_timesteps   | 52352         |\n","| time_elapsed       | 105           |\n","| total_timesteps    | 52352         |\n","| value_loss         | 0.69435924    |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00072223565 |\n","| clipfrac           | 0.00390625    |\n","| explained_variance | 0.997         |\n","| fps                | 533           |\n","| n_updates          | 410           |\n","| policy_entropy     | 0.50538296    |\n","| policy_loss        | -0.0024401287 |\n","| serial_timesteps   | 52480         |\n","| time_elapsed       | 106           |\n","| total_timesteps    | 52480         |\n","| value_loss         | 0.86508095    |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0018889329  |\n","| clipfrac           | 0.0234375     |\n","| explained_variance | 0.971         |\n","| fps                | 525           |\n","| n_updates          | 411           |\n","| policy_entropy     | 0.48436683    |\n","| policy_loss        | -0.0014053787 |\n","| serial_timesteps   | 52608         |\n","| time_elapsed       | 106           |\n","| total_timesteps    | 52608         |\n","| value_loss         | 6.4957767     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0005229876  |\n","| clipfrac           | 0.001953125   |\n","| explained_variance | 0.944         |\n","| fps                | 525           |\n","| n_updates          | 412           |\n","| policy_entropy     | 0.45728323    |\n","| policy_loss        | -0.0004370208 |\n","| serial_timesteps   | 52736         |\n","| time_elapsed       | 106           |\n","| total_timesteps    | 52736         |\n","| value_loss         | 23.34379      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00046230236 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.984         |\n","| fps                | 543           |\n","| n_updates          | 413           |\n","| policy_entropy     | 0.51321       |\n","| policy_loss        | -0.0014906902 |\n","| serial_timesteps   | 52864         |\n","| time_elapsed       | 107           |\n","| total_timesteps    | 52864         |\n","| value_loss         | 3.5198524     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 0.00068299455  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.997          |\n","| fps                | 481            |\n","| n_updates          | 414            |\n","| policy_entropy     | 0.47822672     |\n","| policy_loss        | -0.00063532847 |\n","| serial_timesteps   | 52992          |\n","| time_elapsed       | 107            |\n","| total_timesteps    | 52992          |\n","| value_loss         | 1.3642079      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00017645163 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.996         |\n","| fps                | 508           |\n","| n_updates          | 415           |\n","| policy_entropy     | 0.452845      |\n","| policy_loss        | 0.0010183115  |\n","| serial_timesteps   | 53120         |\n","| time_elapsed       | 107           |\n","| total_timesteps    | 53120         |\n","| value_loss         | 1.5287734     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0009987766 |\n","| clipfrac           | 0.009765625  |\n","| explained_variance | 0.975        |\n","| fps                | 531          |\n","| n_updates          | 416          |\n","| policy_entropy     | 0.47509155   |\n","| policy_loss        | -0.004752716 |\n","| serial_timesteps   | 53248        |\n","| time_elapsed       | 107          |\n","| total_timesteps    | 53248        |\n","| value_loss         | 6.606703     |\n","-------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0014183442 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.977        |\n","| fps                | 531          |\n","| n_updates          | 417          |\n","| policy_entropy     | 0.513254     |\n","| policy_loss        | 0.0038244054 |\n","| serial_timesteps   | 53376        |\n","| time_elapsed       | 108          |\n","| total_timesteps    | 53376        |\n","| value_loss         | 6.0990977    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00054156285 |\n","| clipfrac           | 0.001953125   |\n","| explained_variance | 0.986         |\n","| fps                | 510           |\n","| n_updates          | 418           |\n","| policy_entropy     | 0.5301173     |\n","| policy_loss        | -0.0029715763 |\n","| serial_timesteps   | 53504         |\n","| time_elapsed       | 108           |\n","| total_timesteps    | 53504         |\n","| value_loss         | 4.8193545     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0007656447 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.501        |\n","| fps                | 489          |\n","| n_updates          | 419          |\n","| policy_entropy     | 0.49856797   |\n","| policy_loss        | 0.0032310376 |\n","| serial_timesteps   | 53632        |\n","| time_elapsed       | 108          |\n","| total_timesteps    | 53632        |\n","| value_loss         | 108.2001     |\n","-------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0002390443 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.997        |\n","| fps                | 500          |\n","| n_updates          | 420          |\n","| policy_entropy     | 0.4827454    |\n","| policy_loss        | 0.0014320047 |\n","| serial_timesteps   | 53760        |\n","| time_elapsed       | 108          |\n","| total_timesteps    | 53760        |\n","| value_loss         | 1.3759704    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00065956276 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.998         |\n","| fps                | 534           |\n","| n_updates          | 421           |\n","| policy_entropy     | 0.5025038     |\n","| policy_loss        | -0.0012462029 |\n","| serial_timesteps   | 53888         |\n","| time_elapsed       | 109           |\n","| total_timesteps    | 53888         |\n","| value_loss         | 0.32350537    |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00073776994 |\n","| clipfrac           | 0.005859375   |\n","| explained_variance | 0.991         |\n","| fps                | 486           |\n","| n_updates          | 422           |\n","| policy_entropy     | 0.50871944    |\n","| policy_loss        | 0.004888483   |\n","| serial_timesteps   | 54016         |\n","| time_elapsed       | 109           |\n","| total_timesteps    | 54016         |\n","| value_loss         | 1.322075      |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0007062766 |\n","| clipfrac           | 0.00390625   |\n","| explained_variance | 0.523        |\n","| fps                | 533          |\n","| n_updates          | 423          |\n","| policy_entropy     | 0.47286114   |\n","| policy_loss        | -0.004939893 |\n","| serial_timesteps   | 54144        |\n","| time_elapsed       | 109          |\n","| total_timesteps    | 54144        |\n","| value_loss         | 99.562935    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0015832264  |\n","| clipfrac           | 0.0078125     |\n","| explained_variance | 0.993         |\n","| fps                | 560           |\n","| n_updates          | 424           |\n","| policy_entropy     | 0.48700365    |\n","| policy_loss        | -0.0017577286 |\n","| serial_timesteps   | 54272         |\n","| time_elapsed       | 109           |\n","| total_timesteps    | 54272         |\n","| value_loss         | 1.8297344     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 0.00043588126  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.987          |\n","| fps                | 521            |\n","| n_updates          | 425            |\n","| policy_entropy     | 0.4941703      |\n","| policy_loss        | -0.00055509456 |\n","| serial_timesteps   | 54400          |\n","| time_elapsed       | 109            |\n","| total_timesteps    | 54400          |\n","| value_loss         | 3.2936785      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0010433178  |\n","| clipfrac           | 0.00390625    |\n","| explained_variance | 0.997         |\n","| fps                | 508           |\n","| n_updates          | 426           |\n","| policy_entropy     | 0.50557667    |\n","| policy_loss        | -0.0030410467 |\n","| serial_timesteps   | 54528         |\n","| time_elapsed       | 110           |\n","| total_timesteps    | 54528         |\n","| value_loss         | 3.0035331     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0006269591 |\n","| clipfrac           | 0.005859375  |\n","| explained_variance | 0.617        |\n","| fps                | 485          |\n","| n_updates          | 427          |\n","| policy_entropy     | 0.4670422    |\n","| policy_loss        | -0.005169064 |\n","| serial_timesteps   | 54656        |\n","| time_elapsed       | 110          |\n","| total_timesteps    | 54656        |\n","| value_loss         | 59.374653    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00032742566 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.989         |\n","| fps                | 497           |\n","| n_updates          | 428           |\n","| policy_entropy     | 0.46835396    |\n","| policy_loss        | 0.0003976568  |\n","| serial_timesteps   | 54784         |\n","| time_elapsed       | 110           |\n","| total_timesteps    | 54784         |\n","| value_loss         | 3.1845436     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0001767599 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.991        |\n","| fps                | 498          |\n","| n_updates          | 429          |\n","| policy_entropy     | 0.4828452    |\n","| policy_loss        | 0.0002099846 |\n","| serial_timesteps   | 54912        |\n","| time_elapsed       | 111          |\n","| total_timesteps    | 54912        |\n","| value_loss         | 2.6263523    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0008355838  |\n","| clipfrac           | 0.0078125     |\n","| explained_variance | 0.998         |\n","| fps                | 520           |\n","| n_updates          | 430           |\n","| policy_entropy     | 0.4373789     |\n","| policy_loss        | -0.0033848158 |\n","| serial_timesteps   | 55040         |\n","| time_elapsed       | 111           |\n","| total_timesteps    | 55040         |\n","| value_loss         | 0.41191742    |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0009557386  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.99          |\n","| fps                | 496           |\n","| n_updates          | 431           |\n","| policy_entropy     | 0.4366007     |\n","| policy_loss        | -0.0005273686 |\n","| serial_timesteps   | 55168         |\n","| time_elapsed       | 111           |\n","| total_timesteps    | 55168         |\n","| value_loss         | 1.7001721     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0010363696  |\n","| clipfrac           | 0.005859375   |\n","| explained_variance | 0.999         |\n","| fps                | 539           |\n","| n_updates          | 432           |\n","| policy_entropy     | 0.4290565     |\n","| policy_loss        | -0.0023239523 |\n","| serial_timesteps   | 55296         |\n","| time_elapsed       | 111           |\n","| total_timesteps    | 55296         |\n","| value_loss         | 0.2702717     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00052795967 |\n","| clipfrac           | 0.001953125   |\n","| explained_variance | 0.996         |\n","| fps                | 535           |\n","| n_updates          | 433           |\n","| policy_entropy     | 0.38899985    |\n","| policy_loss        | -0.0042837313 |\n","| serial_timesteps   | 55424         |\n","| time_elapsed       | 112           |\n","| total_timesteps    | 55424         |\n","| value_loss         | 0.9858881     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0015580794  |\n","| clipfrac           | 0.005859375   |\n","| explained_variance | 0.615         |\n","| fps                | 527           |\n","| n_updates          | 434           |\n","| policy_entropy     | 0.41947746    |\n","| policy_loss        | -0.0055121435 |\n","| serial_timesteps   | 55552         |\n","| time_elapsed       | 112           |\n","| total_timesteps    | 55552         |\n","| value_loss         | 98.16152      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00055700925 |\n","| clipfrac           | 0.00390625    |\n","| explained_variance | 0.996         |\n","| fps                | 504           |\n","| n_updates          | 435           |\n","| policy_entropy     | 0.426365      |\n","| policy_loss        | 0.00022897939 |\n","| serial_timesteps   | 55680         |\n","| time_elapsed       | 112           |\n","| total_timesteps    | 55680         |\n","| value_loss         | 2.6731832     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0014225871 |\n","| clipfrac           | 0.01171875   |\n","| explained_variance | 0.997        |\n","| fps                | 554          |\n","| n_updates          | 436          |\n","| policy_entropy     | 0.42705604   |\n","| policy_loss        | 0.0031415503 |\n","| serial_timesteps   | 55808        |\n","| time_elapsed       | 112          |\n","| total_timesteps    | 55808        |\n","| value_loss         | 2.7732685    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0017783053  |\n","| clipfrac           | 0.021484375   |\n","| explained_variance | 0.998         |\n","| fps                | 485           |\n","| n_updates          | 437           |\n","| policy_entropy     | 0.42678612    |\n","| policy_loss        | 0.00015175692 |\n","| serial_timesteps   | 55936         |\n","| time_elapsed       | 112           |\n","| total_timesteps    | 55936         |\n","| value_loss         | 2.1384447     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0011162898 |\n","| clipfrac           | 0.0234375    |\n","| explained_variance | 0.997        |\n","| fps                | 532          |\n","| n_updates          | 438          |\n","| policy_entropy     | 0.45041245   |\n","| policy_loss        | 0.0010421103 |\n","| serial_timesteps   | 56064        |\n","| time_elapsed       | 113          |\n","| total_timesteps    | 56064        |\n","| value_loss         | 0.6320317    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.001714113   |\n","| clipfrac           | 0.017578125   |\n","| explained_variance | 0.999         |\n","| fps                | 504           |\n","| n_updates          | 439           |\n","| policy_entropy     | 0.4558518     |\n","| policy_loss        | -0.0063758474 |\n","| serial_timesteps   | 56192         |\n","| time_elapsed       | 113           |\n","| total_timesteps    | 56192         |\n","| value_loss         | 0.6014276     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00202538    |\n","| clipfrac           | 0.021484375   |\n","| explained_variance | 0.999         |\n","| fps                | 558           |\n","| n_updates          | 440           |\n","| policy_entropy     | 0.43631324    |\n","| policy_loss        | -0.0051521617 |\n","| serial_timesteps   | 56320         |\n","| time_elapsed       | 113           |\n","| total_timesteps    | 56320         |\n","| value_loss         | 0.41420344    |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0018523319  |\n","| clipfrac           | 0.01953125    |\n","| explained_variance | 0.995         |\n","| fps                | 523           |\n","| n_updates          | 441           |\n","| policy_entropy     | 0.42603618    |\n","| policy_loss        | -0.0061233332 |\n","| serial_timesteps   | 56448         |\n","| time_elapsed       | 113           |\n","| total_timesteps    | 56448         |\n","| value_loss         | 1.0246506     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.002102573  |\n","| clipfrac           | 0.021484375  |\n","| explained_variance | 0.961        |\n","| fps                | 547          |\n","| n_updates          | 442          |\n","| policy_entropy     | 0.4530898    |\n","| policy_loss        | 0.0021186168 |\n","| serial_timesteps   | 56576        |\n","| time_elapsed       | 114          |\n","| total_timesteps    | 56576        |\n","| value_loss         | 6.9928865    |\n","-------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0031809493 |\n","| clipfrac           | 0.05078125   |\n","| explained_variance | 0.987        |\n","| fps                | 507          |\n","| n_updates          | 443          |\n","| policy_entropy     | 0.40251508   |\n","| policy_loss        | -0.00789546  |\n","| serial_timesteps   | 56704        |\n","| time_elapsed       | 114          |\n","| total_timesteps    | 56704        |\n","| value_loss         | 2.2951508    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0001099911  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.958         |\n","| fps                | 525           |\n","| n_updates          | 444           |\n","| policy_entropy     | 0.41152412    |\n","| policy_loss        | 0.00013969839 |\n","| serial_timesteps   | 56832         |\n","| time_elapsed       | 114           |\n","| total_timesteps    | 56832         |\n","| value_loss         | 12.142738     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0002957184 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.895        |\n","| fps                | 522          |\n","| n_updates          | 445          |\n","| policy_entropy     | 0.45843497   |\n","| policy_loss        | 0.006859142  |\n","| serial_timesteps   | 56960        |\n","| time_elapsed       | 114          |\n","| total_timesteps    | 56960        |\n","| value_loss         | 19.902542    |\n","-------------------------------------\n","-------------------------------------\n","| approxkl           | 0.009550556  |\n","| clipfrac           | 0.109375     |\n","| explained_variance | 0.925        |\n","| fps                | 505          |\n","| n_updates          | 446          |\n","| policy_entropy     | 0.41855383   |\n","| policy_loss        | -0.012519559 |\n","| serial_timesteps   | 57088        |\n","| time_elapsed       | 115          |\n","| total_timesteps    | 57088        |\n","| value_loss         | 12.27976     |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0015115539  |\n","| clipfrac           | 0.017578125   |\n","| explained_variance | 0.998         |\n","| fps                | 506           |\n","| n_updates          | 447           |\n","| policy_entropy     | 0.40621984    |\n","| policy_loss        | -0.0054601794 |\n","| serial_timesteps   | 57216         |\n","| time_elapsed       | 115           |\n","| total_timesteps    | 57216         |\n","| value_loss         | 0.22636187    |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.002387834  |\n","| clipfrac           | 0.0234375    |\n","| explained_variance | 0.997        |\n","| fps                | 515          |\n","| n_updates          | 448          |\n","| policy_entropy     | 0.41971284   |\n","| policy_loss        | -0.004804652 |\n","| serial_timesteps   | 57344        |\n","| time_elapsed       | 115          |\n","| total_timesteps    | 57344        |\n","| value_loss         | 0.7150649    |\n","-------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0008269743 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.997        |\n","| fps                | 521          |\n","| n_updates          | 449          |\n","| policy_entropy     | 0.40491298   |\n","| policy_loss        | -0.003416195 |\n","| serial_timesteps   | 57472        |\n","| time_elapsed       | 115          |\n","| total_timesteps    | 57472        |\n","| value_loss         | 0.5702586    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0008511377  |\n","| clipfrac           | 0.00390625    |\n","| explained_variance | 0.999         |\n","| fps                | 506           |\n","| n_updates          | 450           |\n","| policy_entropy     | 0.41351935    |\n","| policy_loss        | -0.0025726932 |\n","| serial_timesteps   | 57600         |\n","| time_elapsed       | 116           |\n","| total_timesteps    | 57600         |\n","| value_loss         | 0.42177218    |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0017312212 |\n","| clipfrac           | 0.025390625  |\n","| explained_variance | 0.997        |\n","| fps                | 488          |\n","| n_updates          | 451          |\n","| policy_entropy     | 0.45582905   |\n","| policy_loss        | -0.005209113 |\n","| serial_timesteps   | 57728        |\n","| time_elapsed       | 116          |\n","| total_timesteps    | 57728        |\n","| value_loss         | 2.1391733    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00040830905 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.998         |\n","| fps                | 502           |\n","| n_updates          | 452           |\n","| policy_entropy     | 0.44492167    |\n","| policy_loss        | -0.0029267545 |\n","| serial_timesteps   | 57856         |\n","| time_elapsed       | 116           |\n","| total_timesteps    | 57856         |\n","| value_loss         | 1.140193      |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0017410228 |\n","| clipfrac           | 0.01171875   |\n","| explained_variance | 0.998        |\n","| fps                | 521          |\n","| n_updates          | 453          |\n","| policy_entropy     | 0.39369628   |\n","| policy_loss        | -0.004708397 |\n","| serial_timesteps   | 57984        |\n","| time_elapsed       | 116          |\n","| total_timesteps    | 57984        |\n","| value_loss         | 1.3684089    |\n","-------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0011640765 |\n","| clipfrac           | 0.01171875   |\n","| explained_variance | 0.995        |\n","| fps                | 510          |\n","| n_updates          | 454          |\n","| policy_entropy     | 0.39813796   |\n","| policy_loss        | -0.006025682 |\n","| serial_timesteps   | 58112        |\n","| time_elapsed       | 117          |\n","| total_timesteps    | 58112        |\n","| value_loss         | 3.367509     |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00034707884 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.999         |\n","| fps                | 555           |\n","| n_updates          | 455           |\n","| policy_entropy     | 0.40539214    |\n","| policy_loss        | 0.0023549057  |\n","| serial_timesteps   | 58240         |\n","| time_elapsed       | 117           |\n","| total_timesteps    | 58240         |\n","| value_loss         | 4.282798      |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0009569841 |\n","| clipfrac           | 0.015625     |\n","| explained_variance | 0.995        |\n","| fps                | 464          |\n","| n_updates          | 456          |\n","| policy_entropy     | 0.44021446   |\n","| policy_loss        | 0.0003829361 |\n","| serial_timesteps   | 58368        |\n","| time_elapsed       | 117          |\n","| total_timesteps    | 58368        |\n","| value_loss         | 2.7871175    |\n","-------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0010546818 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.998        |\n","| fps                | 533          |\n","| n_updates          | 457          |\n","| policy_entropy     | 0.44031256   |\n","| policy_loss        | -0.001914208 |\n","| serial_timesteps   | 58496        |\n","| time_elapsed       | 117          |\n","| total_timesteps    | 58496        |\n","| value_loss         | 1.4263352    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 7.524489e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.997         |\n","| fps                | 503           |\n","| n_updates          | 458           |\n","| policy_entropy     | 0.4293304     |\n","| policy_loss        | 0.00048331707 |\n","| serial_timesteps   | 58624         |\n","| time_elapsed       | 118           |\n","| total_timesteps    | 58624         |\n","| value_loss         | 4.0974007     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 8.4986794e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.997         |\n","| fps                | 533           |\n","| n_updates          | 459           |\n","| policy_entropy     | 0.44533163    |\n","| policy_loss        | -0.0009755279 |\n","| serial_timesteps   | 58752         |\n","| time_elapsed       | 118           |\n","| total_timesteps    | 58752         |\n","| value_loss         | 2.0702648     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00048440724 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.991         |\n","| fps                | 474           |\n","| n_updates          | 460           |\n","| policy_entropy     | 0.4407562     |\n","| policy_loss        | -0.0026798625 |\n","| serial_timesteps   | 58880         |\n","| time_elapsed       | 118           |\n","| total_timesteps    | 58880         |\n","| value_loss         | 2.7128854     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0007754301 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.999        |\n","| fps                | 478          |\n","| n_updates          | 461          |\n","| policy_entropy     | 0.435182     |\n","| policy_loss        | 0.0006332325 |\n","| serial_timesteps   | 59008        |\n","| time_elapsed       | 119          |\n","| total_timesteps    | 59008        |\n","| value_loss         | 3.0172668    |\n","-------------------------------------\n","---------------------------------------\n","| approxkl           | 0.0016366926   |\n","| clipfrac           | 0.01953125     |\n","| explained_variance | 0.998          |\n","| fps                | 528            |\n","| n_updates          | 462            |\n","| policy_entropy     | 0.43417725     |\n","| policy_loss        | -0.00015513325 |\n","| serial_timesteps   | 59136          |\n","| time_elapsed       | 119            |\n","| total_timesteps    | 59136          |\n","| value_loss         | 1.7456081      |\n","---------------------------------------\n","---------------------------------------\n","| approxkl           | 0.00010113318  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.981          |\n","| fps                | 482            |\n","| n_updates          | 463            |\n","| policy_entropy     | 0.46582824     |\n","| policy_loss        | -0.00033665728 |\n","| serial_timesteps   | 59264          |\n","| time_elapsed       | 119            |\n","| total_timesteps    | 59264          |\n","| value_loss         | 4.3463836      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 0.001812654   |\n","| clipfrac           | 0.01953125    |\n","| explained_variance | 0.986         |\n","| fps                | 533           |\n","| n_updates          | 464           |\n","| policy_entropy     | 0.4329974     |\n","| policy_loss        | -0.0094275065 |\n","| serial_timesteps   | 59392         |\n","| time_elapsed       | 119           |\n","| total_timesteps    | 59392         |\n","| value_loss         | 3.6081734     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0012055773  |\n","| clipfrac           | 0.009765625   |\n","| explained_variance | 0.998         |\n","| fps                | 480           |\n","| n_updates          | 465           |\n","| policy_entropy     | 0.39645374    |\n","| policy_loss        | -0.0041885423 |\n","| serial_timesteps   | 59520         |\n","| time_elapsed       | 120           |\n","| total_timesteps    | 59520         |\n","| value_loss         | 1.2216974     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.000814224   |\n","| clipfrac           | 0.001953125   |\n","| explained_variance | 0.996         |\n","| fps                | 502           |\n","| n_updates          | 466           |\n","| policy_entropy     | 0.4931999     |\n","| policy_loss        | -0.0053224782 |\n","| serial_timesteps   | 59648         |\n","| time_elapsed       | 120           |\n","| total_timesteps    | 59648         |\n","| value_loss         | 6.7675633     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00061293395 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0887        |\n","| fps                | 502           |\n","| n_updates          | 467           |\n","| policy_entropy     | 0.47603548    |\n","| policy_loss        | 0.0010953308  |\n","| serial_timesteps   | 59776         |\n","| time_elapsed       | 120           |\n","| total_timesteps    | 59776         |\n","| value_loss         | 145.9955      |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.001037285  |\n","| clipfrac           | 0.00390625   |\n","| explained_variance | 0.979        |\n","| fps                | 469          |\n","| n_updates          | 468          |\n","| policy_entropy     | 0.4828645    |\n","| policy_loss        | -0.005733667 |\n","| serial_timesteps   | 59904        |\n","| time_elapsed       | 120          |\n","| total_timesteps    | 59904        |\n","| value_loss         | 4.091013     |\n","-------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0011221772 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.845        |\n","| fps                | 509          |\n","| n_updates          | 469          |\n","| policy_entropy     | 0.4744142    |\n","| policy_loss        | 0.000945588  |\n","| serial_timesteps   | 60032        |\n","| time_elapsed       | 121          |\n","| total_timesteps    | 60032        |\n","| value_loss         | 37.469105    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00010897225 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0131       |\n","| fps                | 548           |\n","| n_updates          | 470           |\n","| policy_entropy     | 0.49182162    |\n","| policy_loss        | 0.00097209285 |\n","| serial_timesteps   | 60160         |\n","| time_elapsed       | 121           |\n","| total_timesteps    | 60160         |\n","| value_loss         | 164.05853     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 2.7014503e-06  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.748          |\n","| fps                | 458            |\n","| n_updates          | 471            |\n","| policy_entropy     | 0.4840106      |\n","| policy_loss        | -2.8021634e-05 |\n","| serial_timesteps   | 60288          |\n","| time_elapsed       | 121            |\n","| total_timesteps    | 60288          |\n","| value_loss         | 8.176581       |\n","---------------------------------------\n","---------------------------------------\n","| approxkl           | 4.387626e-05   |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.557          |\n","| fps                | 529            |\n","| n_updates          | 472            |\n","| policy_entropy     | 0.43232277     |\n","| policy_loss        | -0.00016258517 |\n","| serial_timesteps   | 60416          |\n","| time_elapsed       | 121            |\n","| total_timesteps    | 60416          |\n","| value_loss         | 92.71429       |\n","---------------------------------------\n","-------------------------------------\n","| approxkl           | 8.441989e-05 |\n","| clipfrac           | 0.0          |\n","| explained_variance | -0.0078      |\n","| fps                | 519          |\n","| n_updates          | 473          |\n","| policy_entropy     | 0.41386136   |\n","| policy_loss        | -0.002733517 |\n","| serial_timesteps   | 60544        |\n","| time_elapsed       | 122          |\n","| total_timesteps    | 60544        |\n","| value_loss         | 170.41898    |\n","-------------------------------------\n","-------------------------------------\n","| approxkl           | 9.968374e-06 |\n","| clipfrac           | 0.0          |\n","| explained_variance | -0.752       |\n","| fps                | 529          |\n","| n_updates          | 474          |\n","| policy_entropy     | 0.46721467   |\n","| policy_loss        | 0.0005282337 |\n","| serial_timesteps   | 60672        |\n","| time_elapsed       | 122          |\n","| total_timesteps    | 60672        |\n","| value_loss         | 8.456409     |\n","-------------------------------------\n","---------------------------------------\n","| approxkl           | 4.365406e-06   |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.0287        |\n","| fps                | 530            |\n","| n_updates          | 475            |\n","| policy_entropy     | 0.450575       |\n","| policy_loss        | -0.00048273843 |\n","| serial_timesteps   | 60800          |\n","| time_elapsed       | 122            |\n","| total_timesteps    | 60800          |\n","| value_loss         | 169.86162      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 4.0329676e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.227        |\n","| fps                | 497           |\n","| n_updates          | 476           |\n","| policy_entropy     | 0.4593774     |\n","| policy_loss        | 0.00015156996 |\n","| serial_timesteps   | 60928         |\n","| time_elapsed       | 122           |\n","| total_timesteps    | 60928         |\n","| value_loss         | 7.82007       |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 1.4281688e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.0392        |\n","| fps                | 518            |\n","| n_updates          | 477            |\n","| policy_entropy     | 0.5084726      |\n","| policy_loss        | -0.00050004805 |\n","| serial_timesteps   | 61056          |\n","| time_elapsed       | 123            |\n","| total_timesteps    | 61056          |\n","| value_loss         | 111.23511      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00010304463 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0107       |\n","| fps                | 510           |\n","| n_updates          | 478           |\n","| policy_entropy     | 0.47495046    |\n","| policy_loss        | -0.0021142126 |\n","| serial_timesteps   | 61184         |\n","| time_elapsed       | 123           |\n","| total_timesteps    | 61184         |\n","| value_loss         | 168.90915     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 3.329567e-05   |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.107         |\n","| fps                | 484            |\n","| n_updates          | 479            |\n","| policy_entropy     | 0.46751994     |\n","| policy_loss        | -0.00052063284 |\n","| serial_timesteps   | 61312          |\n","| time_elapsed       | 123            |\n","| total_timesteps    | 61312          |\n","| value_loss         | 7.426972       |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0004957923  |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00544      |\n","| fps                | 520           |\n","| n_updates          | 480           |\n","| policy_entropy     | 0.44483587    |\n","| policy_loss        | 7.8334706e-05 |\n","| serial_timesteps   | 61440         |\n","| time_elapsed       | 123           |\n","| total_timesteps    | 61440         |\n","| value_loss         | 167.71022     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 1.3113843e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00903      |\n","| fps                | 467           |\n","| n_updates          | 481           |\n","| policy_entropy     | 0.41295886    |\n","| policy_loss        | 0.0001933364  |\n","| serial_timesteps   | 61568         |\n","| time_elapsed       | 124           |\n","| total_timesteps    | 61568         |\n","| value_loss         | 168.317       |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 2.213733e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.193        |\n","| fps                | 508           |\n","| n_updates          | 482           |\n","| policy_entropy     | 0.47833154    |\n","| policy_loss        | -0.0006700122 |\n","| serial_timesteps   | 61696         |\n","| time_elapsed       | 124           |\n","| total_timesteps    | 61696         |\n","| value_loss         | 7.513542      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00077365036 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0454       |\n","| fps                | 489           |\n","| n_updates          | 483           |\n","| policy_entropy     | 0.4751861     |\n","| policy_loss        | -0.0056653945 |\n","| serial_timesteps   | 61824         |\n","| time_elapsed       | 124           |\n","| total_timesteps    | 61824         |\n","| value_loss         | 170.79677     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 1.1354917e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0516        |\n","| fps                | 541           |\n","| n_updates          | 484           |\n","| policy_entropy     | 0.50314105    |\n","| policy_loss        | 0.0001265147  |\n","| serial_timesteps   | 61952         |\n","| time_elapsed       | 124           |\n","| total_timesteps    | 61952         |\n","| value_loss         | 153.9875      |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 8.575818e-06   |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.523          |\n","| fps                | 489            |\n","| n_updates          | 485            |\n","| policy_entropy     | 0.47503725     |\n","| policy_loss        | -1.9070925e-05 |\n","| serial_timesteps   | 62080          |\n","| time_elapsed       | 125            |\n","| total_timesteps    | 62080          |\n","| value_loss         | 7.30438        |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00010359618 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0687        |\n","| fps                | 512           |\n","| n_updates          | 486           |\n","| policy_entropy     | 0.47006023    |\n","| policy_loss        | -0.0003198345 |\n","| serial_timesteps   | 62208         |\n","| time_elapsed       | 125           |\n","| total_timesteps    | 62208         |\n","| value_loss         | 158.52806     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 7.0192027e-06 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.000869     |\n","| fps                | 520           |\n","| n_updates          | 487           |\n","| policy_entropy     | 0.47764277    |\n","| policy_loss        | 2.5912537e-05 |\n","| serial_timesteps   | 62336         |\n","| time_elapsed       | 125           |\n","| total_timesteps    | 62336         |\n","| value_loss         | 169.21074     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 7.5841776e-06  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.799          |\n","| fps                | 496            |\n","| n_updates          | 488            |\n","| policy_entropy     | 0.46969992     |\n","| policy_loss        | -0.00033266563 |\n","| serial_timesteps   | 62464          |\n","| time_elapsed       | 125            |\n","| total_timesteps    | 62464          |\n","| value_loss         | 6.825888       |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0008985071  |\n","| clipfrac           | 0.001953125   |\n","| explained_variance | 0.831         |\n","| fps                | 498           |\n","| n_updates          | 489           |\n","| policy_entropy     | 0.43839583    |\n","| policy_loss        | -0.0017081652 |\n","| serial_timesteps   | 62592         |\n","| time_elapsed       | 126           |\n","| total_timesteps    | 62592         |\n","| value_loss         | 38.901176     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00017006497 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.207        |\n","| fps                | 510           |\n","| n_updates          | 490           |\n","| policy_entropy     | 0.49386144    |\n","| policy_loss        | -0.0003512943 |\n","| serial_timesteps   | 62720         |\n","| time_elapsed       | 126           |\n","| total_timesteps    | 62720         |\n","| value_loss         | 7.5861216     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00070272037 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.00895       |\n","| fps                | 547           |\n","| n_updates          | 491           |\n","| policy_entropy     | 0.5072777     |\n","| policy_loss        | -0.0020975405 |\n","| serial_timesteps   | 62848         |\n","| time_elapsed       | 126           |\n","| total_timesteps    | 62848         |\n","| value_loss         | 152.25098     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 4.1202715e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0132       |\n","| fps                | 449           |\n","| n_updates          | 492           |\n","| policy_entropy     | 0.44628805    |\n","| policy_loss        | -0.0009782438 |\n","| serial_timesteps   | 62976         |\n","| time_elapsed       | 126           |\n","| total_timesteps    | 62976         |\n","| value_loss         | 168.74606     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 2.2858403e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.134         |\n","| fps                | 499            |\n","| n_updates          | 493            |\n","| policy_entropy     | 0.48906478     |\n","| policy_loss        | -0.00040943385 |\n","| serial_timesteps   | 63104          |\n","| time_elapsed       | 127            |\n","| total_timesteps    | 63104          |\n","| value_loss         | 7.548629       |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 7.799911e-06  |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0362       |\n","| fps                | 520           |\n","| n_updates          | 494           |\n","| policy_entropy     | 0.5280433     |\n","| policy_loss        | -6.659096e-05 |\n","| serial_timesteps   | 63232         |\n","| time_elapsed       | 127           |\n","| total_timesteps    | 63232         |\n","| value_loss         | 171.13521     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 2.402698e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0184       |\n","| fps                | 551           |\n","| n_updates          | 495           |\n","| policy_entropy     | 0.45162815    |\n","| policy_loss        | -0.0012145592 |\n","| serial_timesteps   | 63360         |\n","| time_elapsed       | 127           |\n","| total_timesteps    | 63360         |\n","| value_loss         | 167.98573     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00012471023 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.154        |\n","| fps                | 462           |\n","| n_updates          | 496           |\n","| policy_entropy     | 0.514646      |\n","| policy_loss        | -0.0025285045 |\n","| serial_timesteps   | 63488         |\n","| time_elapsed       | 127           |\n","| total_timesteps    | 63488         |\n","| value_loss         | 7.574251      |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 0.00054830546  |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.0201        |\n","| fps                | 481            |\n","| n_updates          | 497            |\n","| policy_entropy     | 0.530297       |\n","| policy_loss        | -0.00081236206 |\n","| serial_timesteps   | 63616          |\n","| time_elapsed       | 128            |\n","| total_timesteps    | 63616          |\n","| value_loss         | 170.20738      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 6.170559e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00815      |\n","| fps                | 517           |\n","| n_updates          | 498           |\n","| policy_entropy     | 0.50511754    |\n","| policy_loss        | -0.0014820013 |\n","| serial_timesteps   | 63744         |\n","| time_elapsed       | 128           |\n","| total_timesteps    | 63744         |\n","| value_loss         | 167.98135     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 1.4741768e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.208        |\n","| fps                | 528           |\n","| n_updates          | 499           |\n","| policy_entropy     | 0.48079368    |\n","| policy_loss        | 0.00041169755 |\n","| serial_timesteps   | 63872         |\n","| time_elapsed       | 128           |\n","| total_timesteps    | 63872         |\n","| value_loss         | 7.683088      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 4.172198e-06  |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00836      |\n","| fps                | 483           |\n","| n_updates          | 500           |\n","| policy_entropy     | 0.48140302    |\n","| policy_loss        | -1.590792e-05 |\n","| serial_timesteps   | 64000         |\n","| time_elapsed       | 128           |\n","| total_timesteps    | 64000         |\n","| value_loss         | 168.58557     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 1.0907338e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0396       |\n","| fps                | 479           |\n","| n_updates          | 501           |\n","| policy_entropy     | 0.48787814    |\n","| policy_loss        | -0.0004382208 |\n","| serial_timesteps   | 64128         |\n","| time_elapsed       | 129           |\n","| total_timesteps    | 64128         |\n","| value_loss         | 7.6551213     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 6.605148e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00884      |\n","| fps                | 509           |\n","| n_updates          | 502           |\n","| policy_entropy     | 0.54825187    |\n","| policy_loss        | -0.0016000886 |\n","| serial_timesteps   | 64256         |\n","| time_elapsed       | 129           |\n","| total_timesteps    | 64256         |\n","| value_loss         | 110.068275    |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 9.288882e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00822      |\n","| fps                | 586           |\n","| n_updates          | 503           |\n","| policy_entropy     | 0.5364939     |\n","| policy_loss        | -0.0008122034 |\n","| serial_timesteps   | 64384         |\n","| time_elapsed       | 129           |\n","| total_timesteps    | 64384         |\n","| value_loss         | 168.44206     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 1.694959e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0522       |\n","| fps                | 484           |\n","| n_updates          | 504           |\n","| policy_entropy     | 0.49999037    |\n","| policy_loss        | -0.0006775686 |\n","| serial_timesteps   | 64512         |\n","| time_elapsed       | 129           |\n","| total_timesteps    | 64512         |\n","| value_loss         | 7.707925      |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 1.5427788e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.0207        |\n","| fps                | 513            |\n","| n_updates          | 505            |\n","| policy_entropy     | 0.5320678      |\n","| policy_loss        | -4.8561487e-06 |\n","| serial_timesteps   | 64640          |\n","| time_elapsed       | 130            |\n","| total_timesteps    | 64640          |\n","| value_loss         | 166.56581      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 1.2232002e-06 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00936      |\n","| fps                | 533           |\n","| n_updates          | 506           |\n","| policy_entropy     | 0.53862864    |\n","| policy_loss        | 5.105266e-05  |\n","| serial_timesteps   | 64768         |\n","| time_elapsed       | 130           |\n","| total_timesteps    | 64768         |\n","| value_loss         | 166.8541      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 5.6282306e-06 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0607       |\n","| fps                | 551           |\n","| n_updates          | 507           |\n","| policy_entropy     | 0.5341677     |\n","| policy_loss        | 1.3832236e-05 |\n","| serial_timesteps   | 64896         |\n","| time_elapsed       | 130           |\n","| total_timesteps    | 64896         |\n","| value_loss         | 7.6493864     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 8.063489e-06  |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0231       |\n","| fps                | 484           |\n","| n_updates          | 508           |\n","| policy_entropy     | 0.511738      |\n","| policy_loss        | 2.2085733e-05 |\n","| serial_timesteps   | 65024         |\n","| time_elapsed       | 130           |\n","| total_timesteps    | 65024         |\n","| value_loss         | 169.36931     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 3.3219465e-06 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00705      |\n","| fps                | 500           |\n","| n_updates          | 509           |\n","| policy_entropy     | 0.52096426    |\n","| policy_loss        | 0.00023426418 |\n","| serial_timesteps   | 65152         |\n","| time_elapsed       | 131           |\n","| total_timesteps    | 65152         |\n","| value_loss         | 165.3987      |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 5.670125e-06   |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.112         |\n","| fps                | 519            |\n","| n_updates          | 510            |\n","| policy_entropy     | 0.49593756     |\n","| policy_loss        | -0.00017235347 |\n","| serial_timesteps   | 65280          |\n","| time_elapsed       | 131            |\n","| total_timesteps    | 65280          |\n","| value_loss         | 7.9586797      |\n","---------------------------------------\n","-------------------------------------\n","| approxkl           | 5.092981e-06 |\n","| clipfrac           | 0.0          |\n","| explained_variance | -0.00399     |\n","| fps                | 523          |\n","| n_updates          | 511          |\n","| policy_entropy     | 0.5119873    |\n","| policy_loss        | 0.0001078723 |\n","| serial_timesteps   | 65408        |\n","| time_elapsed       | 131          |\n","| total_timesteps    | 65408        |\n","| value_loss         | 167.48837    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 3.5086527e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00109      |\n","| fps                | 512           |\n","| n_updates          | 512           |\n","| policy_entropy     | 0.52919877    |\n","| policy_loss        | -0.000644726  |\n","| serial_timesteps   | 65536         |\n","| time_elapsed       | 131           |\n","| total_timesteps    | 65536         |\n","| value_loss         | 166.95111     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 1.6616272e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0646       |\n","| fps                | 523           |\n","| n_updates          | 513           |\n","| policy_entropy     | 0.4978245     |\n","| policy_loss        | 8.920848e-05  |\n","| serial_timesteps   | 65664         |\n","| time_elapsed       | 132           |\n","| total_timesteps    | 65664         |\n","| value_loss         | 7.7964993     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 1.661186e-05   |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.00277       |\n","| fps                | 516            |\n","| n_updates          | 514            |\n","| policy_entropy     | 0.5045871      |\n","| policy_loss        | -0.00052905525 |\n","| serial_timesteps   | 65792          |\n","| time_elapsed       | 132            |\n","| total_timesteps    | 65792          |\n","| value_loss         | 166.13763      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 1.9325648e-06 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0285       |\n","| fps                | 532           |\n","| n_updates          | 515           |\n","| policy_entropy     | 0.53105676    |\n","| policy_loss        | 6.960728e-05  |\n","| serial_timesteps   | 65920         |\n","| time_elapsed       | 132           |\n","| total_timesteps    | 65920         |\n","| value_loss         | 7.885806      |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 9.192934e-06   |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.00652       |\n","| fps                | 509            |\n","| n_updates          | 516            |\n","| policy_entropy     | 0.5435243      |\n","| policy_loss        | -0.00042170263 |\n","| serial_timesteps   | 66048          |\n","| time_elapsed       | 132            |\n","| total_timesteps    | 66048          |\n","| value_loss         | 151.46669      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 5.9923103e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00351      |\n","| fps                | 508           |\n","| n_updates          | 517           |\n","| policy_entropy     | 0.5398108     |\n","| policy_loss        | 0.00031198608 |\n","| serial_timesteps   | 66176         |\n","| time_elapsed       | 133           |\n","| total_timesteps    | 66176         |\n","| value_loss         | 165.44064     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 3.969263e-06 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.024        |\n","| fps                | 451          |\n","| n_updates          | 518          |\n","| policy_entropy     | 0.46237594   |\n","| policy_loss        | 2.488424e-05 |\n","| serial_timesteps   | 66304        |\n","| time_elapsed       | 133          |\n","| total_timesteps    | 66304        |\n","| value_loss         | 7.9345293    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00012327779 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00349      |\n","| fps                | 534           |\n","| n_updates          | 519           |\n","| policy_entropy     | 0.50080967    |\n","| policy_loss        | -0.0035413625 |\n","| serial_timesteps   | 66432         |\n","| time_elapsed       | 133           |\n","| total_timesteps    | 66432         |\n","| value_loss         | 166.00688     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00017724885 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00697      |\n","| fps                | 482           |\n","| n_updates          | 520           |\n","| policy_entropy     | 0.5023079     |\n","| policy_loss        | -0.0018122626 |\n","| serial_timesteps   | 66560         |\n","| time_elapsed       | 133           |\n","| total_timesteps    | 66560         |\n","| value_loss         | 164.69275     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 1.0925851e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.206        |\n","| fps                | 514           |\n","| n_updates          | 521           |\n","| policy_entropy     | 0.54406977    |\n","| policy_loss        | 6.932224e-05  |\n","| serial_timesteps   | 66688         |\n","| time_elapsed       | 134           |\n","| total_timesteps    | 66688         |\n","| value_loss         | 8.38305       |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 8.73981e-06    |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.000524      |\n","| fps                | 504            |\n","| n_updates          | 522            |\n","| policy_entropy     | 0.48939747     |\n","| policy_loss        | -0.00032586092 |\n","| serial_timesteps   | 66816          |\n","| time_elapsed       | 134            |\n","| total_timesteps    | 66816          |\n","| value_loss         | 165.1955       |\n","---------------------------------------\n","---------------------------------------\n","| approxkl           | 3.4799265e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.00787        |\n","| fps                | 532            |\n","| n_updates          | 523            |\n","| policy_entropy     | 0.50252414     |\n","| policy_loss        | -0.00024324073 |\n","| serial_timesteps   | 66944          |\n","| time_elapsed       | 134            |\n","| total_timesteps    | 66944          |\n","| value_loss         | 162.9553       |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 1.7159815e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0382       |\n","| fps                | 475           |\n","| n_updates          | 524           |\n","| policy_entropy     | 0.51873815    |\n","| policy_loss        | 0.00073683134 |\n","| serial_timesteps   | 67072         |\n","| time_elapsed       | 134           |\n","| total_timesteps    | 67072         |\n","| value_loss         | 8.009176      |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 1.3844145e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.00133       |\n","| fps                | 507            |\n","| n_updates          | 525            |\n","| policy_entropy     | 0.50621593     |\n","| policy_loss        | -0.00054623047 |\n","| serial_timesteps   | 67200          |\n","| time_elapsed       | 135            |\n","| total_timesteps    | 67200          |\n","| value_loss         | 164.09093      |\n","---------------------------------------\n","-------------------------------------\n","| approxkl           | 1.810005e-05 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.451        |\n","| fps                | 460          |\n","| n_updates          | 526          |\n","| policy_entropy     | 0.53162885   |\n","| policy_loss        | 0.000586823  |\n","| serial_timesteps   | 67328        |\n","| time_elapsed       | 135          |\n","| total_timesteps    | 67328        |\n","| value_loss         | 7.3881655    |\n","-------------------------------------\n","---------------------------------------\n","| approxkl           | 1.3124929e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.0486         |\n","| fps                | 461            |\n","| n_updates          | 527            |\n","| policy_entropy     | 0.5136873      |\n","| policy_loss        | -0.00026449596 |\n","| serial_timesteps   | 67456          |\n","| time_elapsed       | 135            |\n","| total_timesteps    | 67456          |\n","| value_loss         | 102.59524      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 4.6045297e-06 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0229        |\n","| fps                | 472           |\n","| n_updates          | 528           |\n","| policy_entropy     | 0.5293799     |\n","| policy_loss        | 0.0001531227  |\n","| serial_timesteps   | 67584         |\n","| time_elapsed       | 136           |\n","| total_timesteps    | 67584         |\n","| value_loss         | 160.37758     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 2.9903837e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0676       |\n","| fps                | 519           |\n","| n_updates          | 529           |\n","| policy_entropy     | 0.51358396    |\n","| policy_loss        | -0.0008814824 |\n","| serial_timesteps   | 67712         |\n","| time_elapsed       | 136           |\n","| total_timesteps    | 67712         |\n","| value_loss         | 7.616719      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00071067625 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0216       |\n","| fps                | 516           |\n","| n_updates          | 530           |\n","| policy_entropy     | 0.5126453     |\n","| policy_loss        | -0.0028192399 |\n","| serial_timesteps   | 67840         |\n","| time_elapsed       | 136           |\n","| total_timesteps    | 67840         |\n","| value_loss         | 165.02824     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 3.2770185e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.000715     |\n","| fps                | 544           |\n","| n_updates          | 531           |\n","| policy_entropy     | 0.5416438     |\n","| policy_loss        | 0.00024334353 |\n","| serial_timesteps   | 67968         |\n","| time_elapsed       | 136           |\n","| total_timesteps    | 67968         |\n","| value_loss         | 164.94641     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 1.7325847e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.297          |\n","| fps                | 512            |\n","| n_updates          | 532            |\n","| policy_entropy     | 0.52787465     |\n","| policy_loss        | -0.00033780304 |\n","| serial_timesteps   | 68096          |\n","| time_elapsed       | 137            |\n","| total_timesteps    | 68096          |\n","| value_loss         | 7.5360117      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 3.422721e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.128         |\n","| fps                | 490           |\n","| n_updates          | 533           |\n","| policy_entropy     | 0.505143      |\n","| policy_loss        | -0.0005570706 |\n","| serial_timesteps   | 68224         |\n","| time_elapsed       | 137           |\n","| total_timesteps    | 68224         |\n","| value_loss         | 150.39693     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 5.65429e-05   |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.114         |\n","| fps                | 528           |\n","| n_updates          | 534           |\n","| policy_entropy     | 0.5148934     |\n","| policy_loss        | -0.0005632442 |\n","| serial_timesteps   | 68352         |\n","| time_elapsed       | 137           |\n","| total_timesteps    | 68352         |\n","| value_loss         | 149.71811     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0003182487  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.738         |\n","| fps                | 531           |\n","| n_updates          | 535           |\n","| policy_entropy     | 0.5061594     |\n","| policy_loss        | -0.0024351543 |\n","| serial_timesteps   | 68480         |\n","| time_elapsed       | 137           |\n","| total_timesteps    | 68480         |\n","| value_loss         | 5.5551276     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0027706244 |\n","| clipfrac           | 0.04296875   |\n","| explained_variance | 0.0989       |\n","| fps                | 527          |\n","| n_updates          | 536          |\n","| policy_entropy     | 0.52178484   |\n","| policy_loss        | 0.0055811526 |\n","| serial_timesteps   | 68608        |\n","| time_elapsed       | 138          |\n","| total_timesteps    | 68608        |\n","| value_loss         | 149.76703    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 9.96622e-05   |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.00517       |\n","| fps                | 468           |\n","| n_updates          | 537           |\n","| policy_entropy     | 0.5234275     |\n","| policy_loss        | -0.0022806155 |\n","| serial_timesteps   | 68736         |\n","| time_elapsed       | 138           |\n","| total_timesteps    | 68736         |\n","| value_loss         | 169.18886     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 3.959582e-05   |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.105          |\n","| fps                | 419            |\n","| n_updates          | 538            |\n","| policy_entropy     | 0.53697264     |\n","| policy_loss        | -0.00024876965 |\n","| serial_timesteps   | 68864          |\n","| time_elapsed       | 138            |\n","| total_timesteps    | 68864          |\n","| value_loss         | 7.3492837      |\n","---------------------------------------\n","---------------------------------------\n","| approxkl           | 4.106868e-06   |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.0211         |\n","| fps                | 403            |\n","| n_updates          | 539            |\n","| policy_entropy     | 0.58714664     |\n","| policy_loss        | -0.00012179336 |\n","| serial_timesteps   | 68992          |\n","| time_elapsed       | 138            |\n","| total_timesteps    | 68992          |\n","| value_loss         | 166.32355      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 5.6057113e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.016         |\n","| fps                | 417           |\n","| n_updates          | 540           |\n","| policy_entropy     | 0.54932976    |\n","| policy_loss        | -0.0017968162 |\n","| serial_timesteps   | 69120         |\n","| time_elapsed       | 139           |\n","| total_timesteps    | 69120         |\n","| value_loss         | 7.209921      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 8.646867e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00388      |\n","| fps                | 517           |\n","| n_updates          | 541           |\n","| policy_entropy     | 0.5651746     |\n","| policy_loss        | 0.00028164673 |\n","| serial_timesteps   | 69248         |\n","| time_elapsed       | 139           |\n","| total_timesteps    | 69248         |\n","| value_loss         | 154.86781     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 9.494184e-05 |\n","| clipfrac           | 0.0          |\n","| explained_variance | -0.000726    |\n","| fps                | 484          |\n","| n_updates          | 542          |\n","| policy_entropy     | 0.5627281    |\n","| policy_loss        | -0.00076282  |\n","| serial_timesteps   | 69376        |\n","| time_elapsed       | 139          |\n","| total_timesteps    | 69376        |\n","| value_loss         | 169.98143    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 5.3234668e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0292        |\n","| fps                | 495           |\n","| n_updates          | 543           |\n","| policy_entropy     | 0.5658001     |\n","| policy_loss        | -0.0011574113 |\n","| serial_timesteps   | 69504         |\n","| time_elapsed       | 140           |\n","| total_timesteps    | 69504         |\n","| value_loss         | 7.3730254     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 6.3580585e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.00651       |\n","| fps                | 532            |\n","| n_updates          | 544            |\n","| policy_entropy     | 0.55127245     |\n","| policy_loss        | -0.00084881287 |\n","| serial_timesteps   | 69632          |\n","| time_elapsed       | 140            |\n","| total_timesteps    | 69632          |\n","| value_loss         | 170.67784      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 9.40707e-05   |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.229         |\n","| fps                | 444           |\n","| n_updates          | 545           |\n","| policy_entropy     | 0.5245735     |\n","| policy_loss        | -0.0002870774 |\n","| serial_timesteps   | 69760         |\n","| time_elapsed       | 140           |\n","| total_timesteps    | 69760         |\n","| value_loss         | 135.26743     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 3.0323677e-06  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.00135        |\n","| fps                | 513            |\n","| n_updates          | 546            |\n","| policy_entropy     | 0.5783808      |\n","| policy_loss        | -2.2080727e-05 |\n","| serial_timesteps   | 69888          |\n","| time_elapsed       | 140            |\n","| total_timesteps    | 69888          |\n","| value_loss         | 7.2710056      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 2.1565773e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0401        |\n","| fps                | 476           |\n","| n_updates          | 547           |\n","| policy_entropy     | 0.57076406    |\n","| policy_loss        | -0.0005216601 |\n","| serial_timesteps   | 70016         |\n","| time_elapsed       | 141           |\n","| total_timesteps    | 70016         |\n","| value_loss         | 166.18689     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 2.9890612e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.00263       |\n","| fps                | 414           |\n","| n_updates          | 548           |\n","| policy_entropy     | 0.5416448     |\n","| policy_loss        | -0.0009085337 |\n","| serial_timesteps   | 70144         |\n","| time_elapsed       | 141           |\n","| total_timesteps    | 70144         |\n","| value_loss         | 170.12935     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 1.4025378e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.0562        |\n","| fps                | 404            |\n","| n_updates          | 549            |\n","| policy_entropy     | 0.5384567      |\n","| policy_loss        | -0.00012884219 |\n","| serial_timesteps   | 70272          |\n","| time_elapsed       | 141            |\n","| total_timesteps    | 70272          |\n","| value_loss         | 7.4186335      |\n","---------------------------------------\n","---------------------------------------\n","| approxkl           | 3.5005578e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.000304      |\n","| fps                | 407            |\n","| n_updates          | 550            |\n","| policy_entropy     | 0.5509939      |\n","| policy_loss        | -0.00067607523 |\n","| serial_timesteps   | 70400          |\n","| time_elapsed       | 141            |\n","| total_timesteps    | 70400          |\n","| value_loss         | 169.76558      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 3.5541361e-06 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.742         |\n","| fps                | 461           |\n","| n_updates          | 551           |\n","| policy_entropy     | 0.55732995    |\n","| policy_loss        | 0.00010206306 |\n","| serial_timesteps   | 70528         |\n","| time_elapsed       | 142           |\n","| total_timesteps    | 70528         |\n","| value_loss         | 6.34381       |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 8.355257e-06 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.0675       |\n","| fps                | 538          |\n","| n_updates          | 552          |\n","| policy_entropy     | 0.5538104    |\n","| policy_loss        | -0.000584814 |\n","| serial_timesteps   | 70656        |\n","| time_elapsed       | 142          |\n","| total_timesteps    | 70656        |\n","| value_loss         | 103.26543    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 2.9732406e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0223        |\n","| fps                | 532           |\n","| n_updates          | 553           |\n","| policy_entropy     | 0.5548884     |\n","| policy_loss        | -0.0001277068 |\n","| serial_timesteps   | 70784         |\n","| time_elapsed       | 142           |\n","| total_timesteps    | 70784         |\n","| value_loss         | 166.54733     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00030517546 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0819       |\n","| fps                | 465           |\n","| n_updates          | 554           |\n","| policy_entropy     | 0.5689265     |\n","| policy_loss        | -0.002332066  |\n","| serial_timesteps   | 70912         |\n","| time_elapsed       | 143           |\n","| total_timesteps    | 70912         |\n","| value_loss         | 7.376527      |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 0.00014533244  |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.0175        |\n","| fps                | 508            |\n","| n_updates          | 555            |\n","| policy_entropy     | 0.57605547     |\n","| policy_loss        | -0.00084618013 |\n","| serial_timesteps   | 71040          |\n","| time_elapsed       | 143            |\n","| total_timesteps    | 71040          |\n","| value_loss         | 167.96265      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 2.5867497e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.259         |\n","| fps                | 567           |\n","| n_updates          | 556           |\n","| policy_entropy     | 0.50531936    |\n","| policy_loss        | 0.0001172591  |\n","| serial_timesteps   | 71168         |\n","| time_elapsed       | 143           |\n","| total_timesteps    | 71168         |\n","| value_loss         | 120.91948     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00027775814 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.18          |\n","| fps                | 502           |\n","| n_updates          | 557           |\n","| policy_entropy     | 0.5302377     |\n","| policy_loss        | -0.0023466535 |\n","| serial_timesteps   | 71296         |\n","| time_elapsed       | 143           |\n","| total_timesteps    | 71296         |\n","| value_loss         | 6.893901      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 3.2389886e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0308        |\n","| fps                | 519           |\n","| n_updates          | 558           |\n","| policy_entropy     | 0.58710307    |\n","| policy_loss        | 0.00021838502 |\n","| serial_timesteps   | 71424         |\n","| time_elapsed       | 144           |\n","| total_timesteps    | 71424         |\n","| value_loss         | 168.77069     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 7.893894e-06   |\n","| clipfrac           | 0.0            |\n","| explained_variance | -5.48e-06      |\n","| fps                | 463            |\n","| n_updates          | 559            |\n","| policy_entropy     | 0.5778704      |\n","| policy_loss        | -0.00034612778 |\n","| serial_timesteps   | 71552          |\n","| time_elapsed       | 144            |\n","| total_timesteps    | 71552          |\n","| value_loss         | 170.66277      |\n","---------------------------------------\n","----------------------------------------\n","| approxkl           | 1.43003535e-05  |\n","| clipfrac           | 0.0             |\n","| explained_variance | 0.175           |\n","| fps                | 497             |\n","| n_updates          | 560             |\n","| policy_entropy     | 0.57008284      |\n","| policy_loss        | -0.000117719406 |\n","| serial_timesteps   | 71680           |\n","| time_elapsed       | 144             |\n","| total_timesteps    | 71680           |\n","| value_loss         | 7.1893744       |\n","----------------------------------------\n","--------------------------------------\n","| approxkl           | 7.7685545e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0609        |\n","| fps                | 539           |\n","| n_updates          | 561           |\n","| policy_entropy     | 0.50054634    |\n","| policy_loss        | -0.0023445555 |\n","| serial_timesteps   | 71808         |\n","| time_elapsed       | 144           |\n","| total_timesteps    | 71808         |\n","| value_loss         | 160.71541     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00023880275 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.291         |\n","| fps                | 483           |\n","| n_updates          | 562           |\n","| policy_entropy     | 0.5108794     |\n","| policy_loss        | 0.00015999342 |\n","| serial_timesteps   | 71936         |\n","| time_elapsed       | 145           |\n","| total_timesteps    | 71936         |\n","| value_loss         | 128.70819     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 0.00023898273  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.269          |\n","| fps                | 484            |\n","| n_updates          | 563            |\n","| policy_entropy     | 0.59431744     |\n","| policy_loss        | -0.00077539356 |\n","| serial_timesteps   | 72064          |\n","| time_elapsed       | 145            |\n","| total_timesteps    | 72064          |\n","| value_loss         | 6.5039954      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00072385184 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.118         |\n","| fps                | 535           |\n","| n_updates          | 564           |\n","| policy_entropy     | 0.5815262     |\n","| policy_loss        | -0.009715532  |\n","| serial_timesteps   | 72192         |\n","| time_elapsed       | 145           |\n","| total_timesteps    | 72192         |\n","| value_loss         | 141.3897      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00043317408 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.00975       |\n","| fps                | 522           |\n","| n_updates          | 565           |\n","| policy_entropy     | 0.59124225    |\n","| policy_loss        | -0.0007676302 |\n","| serial_timesteps   | 72320         |\n","| time_elapsed       | 145           |\n","| total_timesteps    | 72320         |\n","| value_loss         | 171.51346     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 8.852257e-06  |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0397       |\n","| fps                | 469           |\n","| n_updates          | 566           |\n","| policy_entropy     | 0.601721      |\n","| policy_loss        | 0.00023148465 |\n","| serial_timesteps   | 72448         |\n","| time_elapsed       | 146           |\n","| total_timesteps    | 72448         |\n","| value_loss         | 7.081941      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 2.5899122e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00456      |\n","| fps                | 562           |\n","| n_updates          | 567           |\n","| policy_entropy     | 0.5267386     |\n","| policy_loss        | 0.00013511686 |\n","| serial_timesteps   | 72576         |\n","| time_elapsed       | 146           |\n","| total_timesteps    | 72576         |\n","| value_loss         | 174.07562     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 3.0319989e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0428        |\n","| fps                | 555           |\n","| n_updates          | 568           |\n","| policy_entropy     | 0.54554045    |\n","| policy_loss        | -0.0007593173 |\n","| serial_timesteps   | 72704         |\n","| time_elapsed       | 146           |\n","| total_timesteps    | 72704         |\n","| value_loss         | 165.86096     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00054243574 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.419         |\n","| fps                | 522           |\n","| n_updates          | 569           |\n","| policy_entropy     | 0.5718569     |\n","| policy_loss        | -0.0032537743 |\n","| serial_timesteps   | 72832         |\n","| time_elapsed       | 146           |\n","| total_timesteps    | 72832         |\n","| value_loss         | 6.6195874     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0023380623 |\n","| clipfrac           | 0.033203125  |\n","| explained_variance | 0.218        |\n","| fps                | 503          |\n","| n_updates          | 570          |\n","| policy_entropy     | 0.551489     |\n","| policy_loss        | 0.0016787234 |\n","| serial_timesteps   | 72960        |\n","| time_elapsed       | 147          |\n","| total_timesteps    | 72960        |\n","| value_loss         | 142.47668    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 4.345401e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.046         |\n","| fps                | 500           |\n","| n_updates          | 571           |\n","| policy_entropy     | 0.5370574     |\n","| policy_loss        | 0.00064721436 |\n","| serial_timesteps   | 73088         |\n","| time_elapsed       | 147           |\n","| total_timesteps    | 73088         |\n","| value_loss         | 167.02405     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 1.6282671e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.782         |\n","| fps                | 525           |\n","| n_updates          | 572           |\n","| policy_entropy     | 0.51660424    |\n","| policy_loss        | 0.00011175941 |\n","| serial_timesteps   | 73216         |\n","| time_elapsed       | 147           |\n","| total_timesteps    | 73216         |\n","| value_loss         | 5.348236      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 4.1441617e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.449         |\n","| fps                | 498           |\n","| n_updates          | 573           |\n","| policy_entropy     | 0.56877327    |\n","| policy_loss        | -0.0005695503 |\n","| serial_timesteps   | 73344         |\n","| time_elapsed       | 147           |\n","| total_timesteps    | 73344         |\n","| value_loss         | 109.93705     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0002627537  |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.126        |\n","| fps                | 484           |\n","| n_updates          | 574           |\n","| policy_entropy     | 0.55710626    |\n","| policy_loss        | -0.0018806312 |\n","| serial_timesteps   | 73472         |\n","| time_elapsed       | 148           |\n","| total_timesteps    | 73472         |\n","| value_loss         | 6.691175      |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.008953363  |\n","| clipfrac           | 0.111328125  |\n","| explained_variance | 0.000464     |\n","| fps                | 458          |\n","| n_updates          | 575          |\n","| policy_entropy     | 0.5108651    |\n","| policy_loss        | -0.003448614 |\n","| serial_timesteps   | 73600        |\n","| time_elapsed       | 148          |\n","| total_timesteps    | 73600        |\n","| value_loss         | 113.3567     |\n","-------------------------------------\n","-------------------------------------\n","| approxkl           | 0.000238121  |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.00293      |\n","| fps                | 482          |\n","| n_updates          | 576          |\n","| policy_entropy     | 0.50979877   |\n","| policy_loss        | 0.0008311162 |\n","| serial_timesteps   | 73728        |\n","| time_elapsed       | 148          |\n","| total_timesteps    | 73728        |\n","| value_loss         | 175.84293    |\n","-------------------------------------\n","-------------------------------------\n","| approxkl           | 8.428909e-06 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.764        |\n","| fps                | 439          |\n","| n_updates          | 577          |\n","| policy_entropy     | 0.55208635   |\n","| policy_loss        | 8.925481e-05 |\n","| serial_timesteps   | 73856        |\n","| time_elapsed       | 148          |\n","| total_timesteps    | 73856        |\n","| value_loss         | 5.4857306    |\n","-------------------------------------\n","---------------------------------------\n","| approxkl           | 5.573698e-06   |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.198          |\n","| fps                | 521            |\n","| n_updates          | 578            |\n","| policy_entropy     | 0.54176944     |\n","| policy_loss        | -0.00034687447 |\n","| serial_timesteps   | 73984          |\n","| time_elapsed       | 149            |\n","| total_timesteps    | 73984          |\n","| value_loss         | 143.94531      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 5.5154607e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.431         |\n","| fps                | 499           |\n","| n_updates          | 579           |\n","| policy_entropy     | 0.538041      |\n","| policy_loss        | -0.0010599562 |\n","| serial_timesteps   | 74112         |\n","| time_elapsed       | 149           |\n","| total_timesteps    | 74112         |\n","| value_loss         | 110.89226     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0004143909  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.717         |\n","| fps                | 522           |\n","| n_updates          | 580           |\n","| policy_entropy     | 0.5247124     |\n","| policy_loss        | -0.0046207146 |\n","| serial_timesteps   | 74240         |\n","| time_elapsed       | 149           |\n","| total_timesteps    | 74240         |\n","| value_loss         | 5.053948      |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0020331026 |\n","| clipfrac           | 0.013671875  |\n","| explained_variance | 0.442        |\n","| fps                | 555          |\n","| n_updates          | 581          |\n","| policy_entropy     | 0.5213875    |\n","| policy_loss        | -0.002688645 |\n","| serial_timesteps   | 74368        |\n","| time_elapsed       | 149          |\n","| total_timesteps    | 74368        |\n","| value_loss         | 116.78061    |\n","-------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0001400508 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.00205      |\n","| fps                | 501          |\n","| n_updates          | 582          |\n","| policy_entropy     | 0.5461095    |\n","| policy_loss        | -0.000337251 |\n","| serial_timesteps   | 74496        |\n","| time_elapsed       | 150          |\n","| total_timesteps    | 74496        |\n","| value_loss         | 178.52505    |\n","-------------------------------------\n","---------------------------------------\n","| approxkl           | 1.42251665e-05 |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.261          |\n","| fps                | 519            |\n","| n_updates          | 583            |\n","| policy_entropy     | 0.5475111      |\n","| policy_loss        | -0.00024209288 |\n","| serial_timesteps   | 74624          |\n","| time_elapsed       | 150            |\n","| total_timesteps    | 74624          |\n","| value_loss         | 6.3372326      |\n","---------------------------------------\n","-------------------------------------\n","| approxkl           | 3.118001e-05 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.0114       |\n","| fps                | 533          |\n","| n_updates          | 584          |\n","| policy_entropy     | 0.5620798    |\n","| policy_loss        | -0.001084614 |\n","| serial_timesteps   | 74752        |\n","| time_elapsed       | 150          |\n","| total_timesteps    | 74752        |\n","| value_loss         | 177.02556    |\n","-------------------------------------\n","-------------------------------------\n","| approxkl           | 9.092524e-06 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.000941     |\n","| fps                | 488          |\n","| n_updates          | 585          |\n","| policy_entropy     | 0.5786148    |\n","| policy_loss        | 0.0002821805 |\n","| serial_timesteps   | 74880        |\n","| time_elapsed       | 150          |\n","| total_timesteps    | 74880        |\n","| value_loss         | 177.44896    |\n","-------------------------------------\n","---------------------------------------\n","| approxkl           | 3.8171274e-06  |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.102         |\n","| fps                | 514            |\n","| n_updates          | 586            |\n","| policy_entropy     | 0.5234649      |\n","| policy_loss        | -0.00015439349 |\n","| serial_timesteps   | 75008          |\n","| time_elapsed       | 151            |\n","| total_timesteps    | 75008          |\n","| value_loss         | 6.5315766      |\n","---------------------------------------\n","---------------------------------------\n","| approxkl           | 1.7683715e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.0342         |\n","| fps                | 512            |\n","| n_updates          | 587            |\n","| policy_entropy     | 0.5958336      |\n","| policy_loss        | -0.00020144938 |\n","| serial_timesteps   | 75136          |\n","| time_elapsed       | 151            |\n","| total_timesteps    | 75136          |\n","| value_loss         | 172.17352      |\n","---------------------------------------\n","---------------------------------------\n","| approxkl           | 7.961125e-05   |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.389          |\n","| fps                | 503            |\n","| n_updates          | 588            |\n","| policy_entropy     | 0.59043056     |\n","| policy_loss        | -0.00063462206 |\n","| serial_timesteps   | 75264          |\n","| time_elapsed       | 151            |\n","| total_timesteps    | 75264          |\n","| value_loss         | 6.2015023      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 6.363756e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0705        |\n","| fps                | 533           |\n","| n_updates          | 589           |\n","| policy_entropy     | 0.5882543     |\n","| policy_loss        | -0.0005387992 |\n","| serial_timesteps   | 75392         |\n","| time_elapsed       | 151           |\n","| total_timesteps    | 75392         |\n","| value_loss         | 150.31586     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 1.7060247e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0627        |\n","| fps                | 521           |\n","| n_updates          | 590           |\n","| policy_entropy     | 0.576079      |\n","| policy_loss        | -0.0002838039 |\n","| serial_timesteps   | 75520         |\n","| time_elapsed       | 152           |\n","| total_timesteps    | 75520         |\n","| value_loss         | 167.50734     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00028988873 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0558        |\n","| fps                | 511           |\n","| n_updates          | 591           |\n","| policy_entropy     | 0.53594357    |\n","| policy_loss        | -0.0033399127 |\n","| serial_timesteps   | 75648         |\n","| time_elapsed       | 152           |\n","| total_timesteps    | 75648         |\n","| value_loss         | 6.0081625     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0015068699  |\n","| clipfrac           | 0.005859375   |\n","| explained_variance | -0.00881      |\n","| fps                | 486           |\n","| n_updates          | 592           |\n","| policy_entropy     | 0.5006157     |\n","| policy_loss        | -0.0063481717 |\n","| serial_timesteps   | 75776         |\n","| time_elapsed       | 152           |\n","| total_timesteps    | 75776         |\n","| value_loss         | 181.2906      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 7.48226e-05   |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0139        |\n","| fps                | 505           |\n","| n_updates          | 593           |\n","| policy_entropy     | 0.55690914    |\n","| policy_loss        | -0.0006524937 |\n","| serial_timesteps   | 75904         |\n","| time_elapsed       | 152           |\n","| total_timesteps    | 75904         |\n","| value_loss         | 176.89915     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 1.245381e-05 |\n","| clipfrac           | 0.0          |\n","| explained_variance | -0.00203     |\n","| fps                | 488          |\n","| n_updates          | 594          |\n","| policy_entropy     | 0.5229174    |\n","| policy_loss        | 0.0006431937 |\n","| serial_timesteps   | 76032        |\n","| time_elapsed       | 153          |\n","| total_timesteps    | 76032        |\n","| value_loss         | 6.179116     |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 4.7062964e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.022         |\n","| fps                | 529           |\n","| n_updates          | 595           |\n","| policy_entropy     | 0.51769775    |\n","| policy_loss        | -0.0013975038 |\n","| serial_timesteps   | 76160         |\n","| time_elapsed       | 153           |\n","| total_timesteps    | 76160         |\n","| value_loss         | 176.5705      |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 1.9847894e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.00195       |\n","| fps                | 521            |\n","| n_updates          | 596            |\n","| policy_entropy     | 0.5260419      |\n","| policy_loss        | -0.00017062272 |\n","| serial_timesteps   | 76288          |\n","| time_elapsed       | 153            |\n","| total_timesteps    | 76288          |\n","| value_loss         | 179.19423      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 4.2918677e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.206         |\n","| fps                | 542           |\n","| n_updates          | 597           |\n","| policy_entropy     | 0.5055136     |\n","| policy_loss        | -0.0008998255 |\n","| serial_timesteps   | 76416         |\n","| time_elapsed       | 153           |\n","| total_timesteps    | 76416         |\n","| value_loss         | 6.3447647     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 2.4420428e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.000143      |\n","| fps                | 525           |\n","| n_updates          | 598           |\n","| policy_entropy     | 0.5437489     |\n","| policy_loss        | -0.0003504703 |\n","| serial_timesteps   | 76544         |\n","| time_elapsed       | 154           |\n","| total_timesteps    | 76544         |\n","| value_loss         | 178.5562      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 3.690556e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.65          |\n","| fps                | 510           |\n","| n_updates          | 599           |\n","| policy_entropy     | 0.528602      |\n","| policy_loss        | -0.0008426304 |\n","| serial_timesteps   | 76672         |\n","| time_elapsed       | 154           |\n","| total_timesteps    | 76672         |\n","| value_loss         | 5.910998      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 1.9271218e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0253        |\n","| fps                | 503           |\n","| n_updates          | 600           |\n","| policy_entropy     | 0.53987503    |\n","| policy_loss        | 0.00016906462 |\n","| serial_timesteps   | 76800         |\n","| time_elapsed       | 154           |\n","| total_timesteps    | 76800         |\n","| value_loss         | 109.43935     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 1.043258e-05   |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.515          |\n","| fps                | 501            |\n","| n_updates          | 601            |\n","| policy_entropy     | 0.51301086     |\n","| policy_loss        | -0.00033370638 |\n","| serial_timesteps   | 76928          |\n","| time_elapsed       | 154            |\n","| total_timesteps    | 76928          |\n","| value_loss         | 99.674904      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 5.4065058e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0799        |\n","| fps                | 536           |\n","| n_updates          | 602           |\n","| policy_entropy     | 0.5022814     |\n","| policy_loss        | -0.0014004448 |\n","| serial_timesteps   | 77056         |\n","| time_elapsed       | 155           |\n","| total_timesteps    | 77056         |\n","| value_loss         | 5.860712      |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0009152687 |\n","| clipfrac           | 0.0          |\n","| explained_variance | -1.38e-05    |\n","| fps                | 508          |\n","| n_updates          | 603          |\n","| policy_entropy     | 0.52179927   |\n","| policy_loss        | 0.005865643  |\n","| serial_timesteps   | 77184        |\n","| time_elapsed       | 155          |\n","| total_timesteps    | 77184        |\n","| value_loss         | 176.4568     |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 2.8328757e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0433        |\n","| fps                | 550           |\n","| n_updates          | 604           |\n","| policy_entropy     | 0.5553152     |\n","| policy_loss        | 0.00014914782 |\n","| serial_timesteps   | 77312         |\n","| time_elapsed       | 155           |\n","| total_timesteps    | 77312         |\n","| value_loss         | 170.4964      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 1.3474149e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0444        |\n","| fps                | 538           |\n","| n_updates          | 605           |\n","| policy_entropy     | 0.5724528     |\n","| policy_loss        | -0.0004904757 |\n","| serial_timesteps   | 77440         |\n","| time_elapsed       | 155           |\n","| total_timesteps    | 77440         |\n","| value_loss         | 5.94503       |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 3.6608963e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.00665        |\n","| fps                | 499            |\n","| n_updates          | 606            |\n","| policy_entropy     | 0.5504537      |\n","| policy_loss        | -0.00020638667 |\n","| serial_timesteps   | 77568          |\n","| time_elapsed       | 156            |\n","| total_timesteps    | 77568          |\n","| value_loss         | 179.67671      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 8.404115e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.139         |\n","| fps                | 481           |\n","| n_updates          | 607           |\n","| policy_entropy     | 0.5329479     |\n","| policy_loss        | -0.0015171324 |\n","| serial_timesteps   | 77696         |\n","| time_elapsed       | 156           |\n","| total_timesteps    | 77696         |\n","| value_loss         | 155.69737     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00040156418 |\n","| clipfrac           | 0.00390625    |\n","| explained_variance | 0.154         |\n","| fps                | 501           |\n","| n_updates          | 608           |\n","| policy_entropy     | 0.51327276    |\n","| policy_loss        | -0.0031653454 |\n","| serial_timesteps   | 77824         |\n","| time_elapsed       | 156           |\n","| total_timesteps    | 77824         |\n","| value_loss         | 5.873297      |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0059618456 |\n","| clipfrac           | 0.08203125   |\n","| explained_variance | 0.016        |\n","| fps                | 461          |\n","| n_updates          | 609          |\n","| policy_entropy     | 0.5653492    |\n","| policy_loss        | -0.003757947 |\n","| serial_timesteps   | 77952        |\n","| time_elapsed       | 156          |\n","| total_timesteps    | 77952        |\n","| value_loss         | 180.20937    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 7.119814e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.153         |\n","| fps                | 514           |\n","| n_updates          | 610           |\n","| policy_entropy     | 0.5109788     |\n","| policy_loss        | 4.8266724e-05 |\n","| serial_timesteps   | 78080         |\n","| time_elapsed       | 157           |\n","| total_timesteps    | 78080         |\n","| value_loss         | 159.47969     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00020613709 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.393         |\n","| fps                | 499           |\n","| n_updates          | 611           |\n","| policy_entropy     | 0.5579084     |\n","| policy_loss        | -0.0004585547 |\n","| serial_timesteps   | 78208         |\n","| time_elapsed       | 157           |\n","| total_timesteps    | 78208         |\n","| value_loss         | 5.4125404     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0009421699 |\n","| clipfrac           | 0.0          |\n","| explained_variance | -0.0123      |\n","| fps                | 544          |\n","| n_updates          | 612          |\n","| policy_entropy     | 0.54695183   |\n","| policy_loss        | 0.0045576915 |\n","| serial_timesteps   | 78336        |\n","| time_elapsed       | 157          |\n","| total_timesteps    | 78336        |\n","| value_loss         | 182.68405    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 5.985151e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.779         |\n","| fps                | 504           |\n","| n_updates          | 613           |\n","| policy_entropy     | 0.54370683    |\n","| policy_loss        | -0.0012983094 |\n","| serial_timesteps   | 78464         |\n","| time_elapsed       | 157           |\n","| total_timesteps    | 78464         |\n","| value_loss         | 5.2816105     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00019241619 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.204         |\n","| fps                | 534           |\n","| n_updates          | 614           |\n","| policy_entropy     | 0.49108166    |\n","| policy_loss        | 0.00015265122 |\n","| serial_timesteps   | 78592         |\n","| time_elapsed       | 158           |\n","| total_timesteps    | 78592         |\n","| value_loss         | 135.05887     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 4.791483e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.502         |\n","| fps                | 492           |\n","| n_updates          | 615           |\n","| policy_entropy     | 0.4794188     |\n","| policy_loss        | -0.0008807223 |\n","| serial_timesteps   | 78720         |\n","| time_elapsed       | 158           |\n","| total_timesteps    | 78720         |\n","| value_loss         | 113.64862     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00021279608 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.884         |\n","| fps                | 524           |\n","| n_updates          | 616           |\n","| policy_entropy     | 0.5696026     |\n","| policy_loss        | 0.0016680073  |\n","| serial_timesteps   | 78848         |\n","| time_elapsed       | 158           |\n","| total_timesteps    | 78848         |\n","| value_loss         | 4.5453587     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00045996052 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.463         |\n","| fps                | 494           |\n","| n_updates          | 617           |\n","| policy_entropy     | 0.50553167    |\n","| policy_loss        | 0.0039545065  |\n","| serial_timesteps   | 78976         |\n","| time_elapsed       | 158           |\n","| total_timesteps    | 78976         |\n","| value_loss         | 115.68214     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 9.02682e-05    |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.311          |\n","| fps                | 519            |\n","| n_updates          | 618            |\n","| policy_entropy     | 0.546562       |\n","| policy_loss        | -0.00069010817 |\n","| serial_timesteps   | 79104          |\n","| time_elapsed       | 159            |\n","| total_timesteps    | 79104          |\n","| value_loss         | 142.84695      |\n","---------------------------------------\n","-------------------------------------\n","| approxkl           | 8.786568e-05 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.507        |\n","| fps                | 486          |\n","| n_updates          | 619          |\n","| policy_entropy     | 0.56852263   |\n","| policy_loss        | 0.0002960516 |\n","| serial_timesteps   | 79232        |\n","| time_elapsed       | 159          |\n","| total_timesteps    | 79232        |\n","| value_loss         | 5.4334583    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00014174622 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.121         |\n","| fps                | 519           |\n","| n_updates          | 620           |\n","| policy_entropy     | 0.56076235    |\n","| policy_loss        | 0.00085140555 |\n","| serial_timesteps   | 79360         |\n","| time_elapsed       | 159           |\n","| total_timesteps    | 79360         |\n","| value_loss         | 168.99586     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 5.0016115e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.0631         |\n","| fps                | 500            |\n","| n_updates          | 621            |\n","| policy_entropy     | 0.5780402      |\n","| policy_loss        | -0.00073498674 |\n","| serial_timesteps   | 79488          |\n","| time_elapsed       | 159            |\n","| total_timesteps    | 79488          |\n","| value_loss         | 171.34497      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00012848782 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.139        |\n","| fps                | 560           |\n","| n_updates          | 622           |\n","| policy_entropy     | 0.57352686    |\n","| policy_loss        | -0.001634219  |\n","| serial_timesteps   | 79616         |\n","| time_elapsed       | 160           |\n","| total_timesteps    | 79616         |\n","| value_loss         | 5.348576      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00038742155 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0163        |\n","| fps                | 531           |\n","| n_updates          | 623           |\n","| policy_entropy     | 0.56222165    |\n","| policy_loss        | 0.0010266944  |\n","| serial_timesteps   | 79744         |\n","| time_elapsed       | 160           |\n","| total_timesteps    | 79744         |\n","| value_loss         | 185.53711     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 3.5388395e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.863         |\n","| fps                | 530           |\n","| n_updates          | 624           |\n","| policy_entropy     | 0.51226586    |\n","| policy_loss        | -0.0006105384 |\n","| serial_timesteps   | 79872         |\n","| time_elapsed       | 160           |\n","| total_timesteps    | 79872         |\n","| value_loss         | 4.358933      |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 9.756335e-05   |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.209          |\n","| fps                | 523            |\n","| n_updates          | 625            |\n","| policy_entropy     | 0.5535452      |\n","| policy_loss        | -0.00026553334 |\n","| serial_timesteps   | 80000          |\n","| time_elapsed       | 160            |\n","| total_timesteps    | 80000          |\n","| value_loss         | 95.97776       |\n","---------------------------------------\n","---------------------------------------\n","| approxkl           | 1.26351315e-05 |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.112          |\n","| fps                | 481            |\n","| n_updates          | 626            |\n","| policy_entropy     | 0.5048035      |\n","| policy_loss        | 0.00038059853  |\n","| serial_timesteps   | 80128          |\n","| time_elapsed       | 161            |\n","| total_timesteps    | 80128          |\n","| value_loss         | 166.89366      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00015954606 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.146         |\n","| fps                | 511           |\n","| n_updates          | 627           |\n","| policy_entropy     | 0.5824642     |\n","| policy_loss        | 0.00045061775 |\n","| serial_timesteps   | 80256         |\n","| time_elapsed       | 161           |\n","| total_timesteps    | 80256         |\n","| value_loss         | 5.0218325     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00031917752 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0156        |\n","| fps                | 505           |\n","| n_updates          | 628           |\n","| policy_entropy     | 0.54334486    |\n","| policy_loss        | 0.0012430755  |\n","| serial_timesteps   | 80384         |\n","| time_elapsed       | 161           |\n","| total_timesteps    | 80384         |\n","| value_loss         | 183.8891      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 5.6975023e-06 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.134         |\n","| fps                | 473           |\n","| n_updates          | 629           |\n","| policy_entropy     | 0.51621467    |\n","| policy_loss        | 5.444279e-06  |\n","| serial_timesteps   | 80512         |\n","| time_elapsed       | 161           |\n","| total_timesteps    | 80512         |\n","| value_loss         | 158.20773     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00017875506 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.46          |\n","| fps                | 410           |\n","| n_updates          | 630           |\n","| policy_entropy     | 0.514438      |\n","| policy_loss        | -0.0033432378 |\n","| serial_timesteps   | 80640         |\n","| time_elapsed       | 162           |\n","| total_timesteps    | 80640         |\n","| value_loss         | 4.7136693     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0006747463 |\n","| clipfrac           | 0.0          |\n","| explained_variance | -0.0144      |\n","| fps                | 402          |\n","| n_updates          | 631          |\n","| policy_entropy     | 0.53732246   |\n","| policy_loss        | 0.0008495834 |\n","| serial_timesteps   | 80768        |\n","| time_elapsed       | 162          |\n","| total_timesteps    | 80768        |\n","| value_loss         | 192.69958    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 2.6699992e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.102         |\n","| fps                | 470           |\n","| n_updates          | 632           |\n","| policy_entropy     | 0.54394555    |\n","| policy_loss        | 0.00014772371 |\n","| serial_timesteps   | 80896         |\n","| time_elapsed       | 162           |\n","| total_timesteps    | 80896         |\n","| value_loss         | 174.17561     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00037975184 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.117         |\n","| fps                | 493           |\n","| n_updates          | 633           |\n","| policy_entropy     | 0.58018476    |\n","| policy_loss        | -0.0051410133 |\n","| serial_timesteps   | 81024         |\n","| time_elapsed       | 163           |\n","| total_timesteps    | 81024         |\n","| value_loss         | 4.879184      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0038689692  |\n","| clipfrac           | 0.02734375    |\n","| explained_variance | 0.0168        |\n","| fps                | 527           |\n","| n_updates          | 634           |\n","| policy_entropy     | 0.551474      |\n","| policy_loss        | -0.0031421639 |\n","| serial_timesteps   | 81152         |\n","| time_elapsed       | 163           |\n","| total_timesteps    | 81152         |\n","| value_loss         | 189.62502     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0004059361  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.232         |\n","| fps                | 453           |\n","| n_updates          | 635           |\n","| policy_entropy     | 0.5149197     |\n","| policy_loss        | -0.0028785083 |\n","| serial_timesteps   | 81280         |\n","| time_elapsed       | 163           |\n","| total_timesteps    | 81280         |\n","| value_loss         | 144.3498      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00017732935 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.222        |\n","| fps                | 400           |\n","| n_updates          | 636           |\n","| policy_entropy     | 0.52085847    |\n","| policy_loss        | -0.0008378995 |\n","| serial_timesteps   | 81408         |\n","| time_elapsed       | 163           |\n","| total_timesteps    | 81408         |\n","| value_loss         | 5.0137815     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00020921549 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00884      |\n","| fps                | 434           |\n","| n_updates          | 637           |\n","| policy_entropy     | 0.52055943    |\n","| policy_loss        | 0.0004072599  |\n","| serial_timesteps   | 81536         |\n","| time_elapsed       | 164           |\n","| total_timesteps    | 81536         |\n","| value_loss         | 192.85132     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 3.7751648e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.986          |\n","| fps                | 495            |\n","| n_updates          | 638            |\n","| policy_entropy     | 0.55051935     |\n","| policy_loss        | -0.00040673744 |\n","| serial_timesteps   | 81664          |\n","| time_elapsed       | 164            |\n","| total_timesteps    | 81664          |\n","| value_loss         | 1.7852261      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 6.2212e-05    |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.523         |\n","| fps                | 521           |\n","| n_updates          | 639           |\n","| policy_entropy     | 0.5644817     |\n","| policy_loss        | 0.00013973727 |\n","| serial_timesteps   | 81792         |\n","| time_elapsed       | 164           |\n","| total_timesteps    | 81792         |\n","| value_loss         | 92.206436     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 3.956136e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.438         |\n","| fps                | 489           |\n","| n_updates          | 640           |\n","| policy_entropy     | 0.51865894    |\n","| policy_loss        | -0.0009971808 |\n","| serial_timesteps   | 81920         |\n","| time_elapsed       | 165           |\n","| total_timesteps    | 81920         |\n","| value_loss         | 119.39067     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 6.374376e-05 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.0893       |\n","| fps                | 464          |\n","| n_updates          | 641          |\n","| policy_entropy     | 0.55836326   |\n","| policy_loss        | 0.0014070901 |\n","| serial_timesteps   | 82048        |\n","| time_elapsed       | 165          |\n","| total_timesteps    | 82048        |\n","| value_loss         | 4.456664     |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0011615232  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0393        |\n","| fps                | 523           |\n","| n_updates          | 642           |\n","| policy_entropy     | 0.5188149     |\n","| policy_loss        | -0.0009973795 |\n","| serial_timesteps   | 82176         |\n","| time_elapsed       | 165           |\n","| total_timesteps    | 82176         |\n","| value_loss         | 188.56772     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0006116254  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.545         |\n","| fps                | 500           |\n","| n_updates          | 643           |\n","| policy_entropy     | 0.529798      |\n","| policy_loss        | -0.0014410121 |\n","| serial_timesteps   | 82304         |\n","| time_elapsed       | 165           |\n","| total_timesteps    | 82304         |\n","| value_loss         | 102.91461     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 4.5616238e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0551        |\n","| fps                | 514           |\n","| n_updates          | 644           |\n","| policy_entropy     | 0.5612029     |\n","| policy_loss        | 0.00048521336 |\n","| serial_timesteps   | 82432         |\n","| time_elapsed       | 166           |\n","| total_timesteps    | 82432         |\n","| value_loss         | 4.4224315     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00020118608 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.018         |\n","| fps                | 510           |\n","| n_updates          | 645           |\n","| policy_entropy     | 0.54410857    |\n","| policy_loss        | -0.0015008787 |\n","| serial_timesteps   | 82560         |\n","| time_elapsed       | 166           |\n","| total_timesteps    | 82560         |\n","| value_loss         | 194.79314     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 4.2700245e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.0224         |\n","| fps                | 506            |\n","| n_updates          | 646            |\n","| policy_entropy     | 0.54046106     |\n","| policy_loss        | -0.00019986508 |\n","| serial_timesteps   | 82688          |\n","| time_elapsed       | 166            |\n","| total_timesteps    | 82688          |\n","| value_loss         | 193.97903      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 8.278905e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.053         |\n","| fps                | 479           |\n","| n_updates          | 647           |\n","| policy_entropy     | 0.5815328     |\n","| policy_loss        | -0.0018355119 |\n","| serial_timesteps   | 82816         |\n","| time_elapsed       | 166           |\n","| total_timesteps    | 82816         |\n","| value_loss         | 4.534079      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 4.3929544e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.014         |\n","| fps                | 512           |\n","| n_updates          | 648           |\n","| policy_entropy     | 0.5608266     |\n","| policy_loss        | 0.00026123447 |\n","| serial_timesteps   | 82944         |\n","| time_elapsed       | 167           |\n","| total_timesteps    | 82944         |\n","| value_loss         | 195.05145     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 1.80057e-05    |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.976          |\n","| fps                | 518            |\n","| n_updates          | 649            |\n","| policy_entropy     | 0.56474745     |\n","| policy_loss        | -0.00038247032 |\n","| serial_timesteps   | 83072          |\n","| time_elapsed       | 167            |\n","| total_timesteps    | 83072          |\n","| value_loss         | 2.4631555      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 6.851113e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.391         |\n","| fps                | 514           |\n","| n_updates          | 650           |\n","| policy_entropy     | 0.55997324    |\n","| policy_loss        | 0.00059293327 |\n","| serial_timesteps   | 83200         |\n","| time_elapsed       | 167           |\n","| total_timesteps    | 83200         |\n","| value_loss         | 79.44005      |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 8.412119e-06   |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.00379       |\n","| fps                | 503            |\n","| n_updates          | 651            |\n","| policy_entropy     | 0.53088725     |\n","| policy_loss        | -0.00010726345 |\n","| serial_timesteps   | 83328          |\n","| time_elapsed       | 167            |\n","| total_timesteps    | 83328          |\n","| value_loss         | 197.36111      |\n","---------------------------------------\n","---------------------------------------\n","| approxkl           | 2.4298927e-06  |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.105         |\n","| fps                | 503            |\n","| n_updates          | 652            |\n","| policy_entropy     | 0.5933291      |\n","| policy_loss        | -3.3947406e-05 |\n","| serial_timesteps   | 83456          |\n","| time_elapsed       | 168            |\n","| total_timesteps    | 83456          |\n","| value_loss         | 5.0274286      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 5.0276817e-06 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0377        |\n","| fps                | 509           |\n","| n_updates          | 653           |\n","| policy_entropy     | 0.5211403     |\n","| policy_loss        | -9.012967e-05 |\n","| serial_timesteps   | 83584         |\n","| time_elapsed       | 168           |\n","| total_timesteps    | 83584         |\n","| value_loss         | 186.17145     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 7.455984e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.101         |\n","| fps                | 510           |\n","| n_updates          | 654           |\n","| policy_entropy     | 0.52398944    |\n","| policy_loss        | -0.0011697649 |\n","| serial_timesteps   | 83712         |\n","| time_elapsed       | 168           |\n","| total_timesteps    | 83712         |\n","| value_loss         | 176.50296     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00019103955 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0104        |\n","| fps                | 522           |\n","| n_updates          | 655           |\n","| policy_entropy     | 0.5421078     |\n","| policy_loss        | 0.00058073236 |\n","| serial_timesteps   | 83840         |\n","| time_elapsed       | 168           |\n","| total_timesteps    | 83840         |\n","| value_loss         | 4.3027377     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00073040946 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0431        |\n","| fps                | 484           |\n","| n_updates          | 656           |\n","| policy_entropy     | 0.5358799     |\n","| policy_loss        | 0.0005929696  |\n","| serial_timesteps   | 83968         |\n","| time_elapsed       | 169           |\n","| total_timesteps    | 83968         |\n","| value_loss         | 191.93886     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 8.1907165e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.137         |\n","| fps                | 533           |\n","| n_updates          | 657           |\n","| policy_entropy     | 0.5525628     |\n","| policy_loss        | -0.0010061171 |\n","| serial_timesteps   | 84096         |\n","| time_elapsed       | 169           |\n","| total_timesteps    | 84096         |\n","| value_loss         | 175.45134     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 5.0715254e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.872         |\n","| fps                | 487           |\n","| n_updates          | 658           |\n","| policy_entropy     | 0.51162       |\n","| policy_loss        | 4.3706503e-05 |\n","| serial_timesteps   | 84224         |\n","| time_elapsed       | 169           |\n","| total_timesteps    | 84224         |\n","| value_loss         | 3.5444708     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.001107505  |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.24         |\n","| fps                | 515          |\n","| n_updates          | 659          |\n","| policy_entropy     | 0.5373998    |\n","| policy_loss        | -0.004872311 |\n","| serial_timesteps   | 84352        |\n","| time_elapsed       | 169          |\n","| total_timesteps    | 84352        |\n","| value_loss         | 162.49643    |\n","-------------------------------------\n","---------------------------------------\n","| approxkl           | 0.00021336325  |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.0109        |\n","| fps                | 513            |\n","| n_updates          | 660            |\n","| policy_entropy     | 0.51748896     |\n","| policy_loss        | -0.00062832574 |\n","| serial_timesteps   | 84480          |\n","| time_elapsed       | 170            |\n","| total_timesteps    | 84480          |\n","| value_loss         | 200.69278      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 5.1364263e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.23         |\n","| fps                | 491           |\n","| n_updates          | 661           |\n","| policy_entropy     | 0.5052934     |\n","| policy_loss        | -0.0007276975 |\n","| serial_timesteps   | 84608         |\n","| time_elapsed       | 170           |\n","| total_timesteps    | 84608         |\n","| value_loss         | 4.544472      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 1.101651e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0158        |\n","| fps                | 436           |\n","| n_updates          | 662           |\n","| policy_entropy     | 0.51862395    |\n","| policy_loss        | 0.00053196866 |\n","| serial_timesteps   | 84736         |\n","| time_elapsed       | 170           |\n","| total_timesteps    | 84736         |\n","| value_loss         | 194.66676     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0008625116  |\n","| clipfrac           | 0.001953125   |\n","| explained_variance | 0.605         |\n","| fps                | 509           |\n","| n_updates          | 663           |\n","| policy_entropy     | 0.5063702     |\n","| policy_loss        | -0.0024884525 |\n","| serial_timesteps   | 84864         |\n","| time_elapsed       | 170           |\n","| total_timesteps    | 84864         |\n","| value_loss         | 6.221733      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00077043026 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.179         |\n","| fps                | 523           |\n","| n_updates          | 664           |\n","| policy_entropy     | 0.5144976     |\n","| policy_loss        | -0.0038364828 |\n","| serial_timesteps   | 84992         |\n","| time_elapsed       | 171           |\n","| total_timesteps    | 84992         |\n","| value_loss         | 144.47809     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 0.000115250026 |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.0633         |\n","| fps                | 528            |\n","| n_updates          | 665            |\n","| policy_entropy     | 0.5091774      |\n","| policy_loss        | 0.0003256119   |\n","| serial_timesteps   | 85120          |\n","| time_elapsed       | 171            |\n","| total_timesteps    | 85120          |\n","| value_loss         | 171.84119      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00018305401 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.222         |\n","| fps                | 478           |\n","| n_updates          | 666           |\n","| policy_entropy     | 0.5078171     |\n","| policy_loss        | -0.0004420157 |\n","| serial_timesteps   | 85248         |\n","| time_elapsed       | 171           |\n","| total_timesteps    | 85248         |\n","| value_loss         | 4.1805897     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00041780138 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.028         |\n","| fps                | 537           |\n","| n_updates          | 667           |\n","| policy_entropy     | 0.53571767    |\n","| policy_loss        | 0.0027737804  |\n","| serial_timesteps   | 85376         |\n","| time_elapsed       | 172           |\n","| total_timesteps    | 85376         |\n","| value_loss         | 195.66818     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 5.9681915e-06  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.307          |\n","| fps                | 495            |\n","| n_updates          | 668            |\n","| policy_entropy     | 0.4953619      |\n","| policy_loss        | -0.00022577797 |\n","| serial_timesteps   | 85504          |\n","| time_elapsed       | 172            |\n","| total_timesteps    | 85504          |\n","| value_loss         | 141.90128      |\n","---------------------------------------\n","---------------------------------------\n","| approxkl           | 8.267892e-06   |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.143          |\n","| fps                | 567            |\n","| n_updates          | 669            |\n","| policy_entropy     | 0.5574579      |\n","| policy_loss        | -0.00029033702 |\n","| serial_timesteps   | 85632          |\n","| time_elapsed       | 172            |\n","| total_timesteps    | 85632          |\n","| value_loss         | 4.048898       |\n","---------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0011231189 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.0562       |\n","| fps                | 503          |\n","| n_updates          | 670          |\n","| policy_entropy     | 0.5210525    |\n","| policy_loss        | 0.0022472164 |\n","| serial_timesteps   | 85760        |\n","| time_elapsed       | 172          |\n","| total_timesteps    | 85760        |\n","| value_loss         | 196.07602    |\n","-------------------------------------\n","-------------------------------------\n","| approxkl           | 0.000405098  |\n","| clipfrac           | 0.00390625   |\n","| explained_variance | 0.181        |\n","| fps                | 506          |\n","| n_updates          | 671          |\n","| policy_entropy     | 0.52960366   |\n","| policy_loss        | -0.003674407 |\n","| serial_timesteps   | 85888        |\n","| time_elapsed       | 172          |\n","| total_timesteps    | 85888        |\n","| value_loss         | 173.92648    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00025961053 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.277         |\n","| fps                | 491           |\n","| n_updates          | 672           |\n","| policy_entropy     | 0.5708738     |\n","| policy_loss        | 0.0014051901  |\n","| serial_timesteps   | 86016         |\n","| time_elapsed       | 173           |\n","| total_timesteps    | 86016         |\n","| value_loss         | 3.782098      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 8.499395e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0263        |\n","| fps                | 516           |\n","| n_updates          | 673           |\n","| policy_entropy     | 0.55633926    |\n","| policy_loss        | -0.0013903119 |\n","| serial_timesteps   | 86144         |\n","| time_elapsed       | 173           |\n","| total_timesteps    | 86144         |\n","| value_loss         | 201.19597     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00027783145 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.787         |\n","| fps                | 515           |\n","| n_updates          | 674           |\n","| policy_entropy     | 0.531138      |\n","| policy_loss        | -0.0040093428 |\n","| serial_timesteps   | 86272         |\n","| time_elapsed       | 173           |\n","| total_timesteps    | 86272         |\n","| value_loss         | 2.9968576     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00016500043 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.21          |\n","| fps                | 551           |\n","| n_updates          | 675           |\n","| policy_entropy     | 0.57551473    |\n","| policy_loss        | -4.019169e-05 |\n","| serial_timesteps   | 86400         |\n","| time_elapsed       | 173           |\n","| total_timesteps    | 86400         |\n","| value_loss         | 101.97426     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 3.613301e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0806        |\n","| fps                | 522           |\n","| n_updates          | 676           |\n","| policy_entropy     | 0.545902      |\n","| policy_loss        | -0.0006821798 |\n","| serial_timesteps   | 86528         |\n","| time_elapsed       | 174           |\n","| total_timesteps    | 86528         |\n","| value_loss         | 186.30952     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 0.00018818412  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.79           |\n","| fps                | 527            |\n","| n_updates          | 677            |\n","| policy_entropy     | 0.5567544      |\n","| policy_loss        | -0.00062766066 |\n","| serial_timesteps   | 86656          |\n","| time_elapsed       | 174            |\n","| total_timesteps    | 86656          |\n","| value_loss         | 3.4029427      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0008802394  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0644        |\n","| fps                | 466           |\n","| n_updates          | 678           |\n","| policy_entropy     | 0.5237135     |\n","| policy_loss        | -0.0015148036 |\n","| serial_timesteps   | 86784         |\n","| time_elapsed       | 174           |\n","| total_timesteps    | 86784         |\n","| value_loss         | 190.52579     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00012432123 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0249        |\n","| fps                | 523           |\n","| n_updates          | 679           |\n","| policy_entropy     | 0.51585376    |\n","| policy_loss        | -0.0027297216 |\n","| serial_timesteps   | 86912         |\n","| time_elapsed       | 174           |\n","| total_timesteps    | 86912         |\n","| value_loss         | 200.22353     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00026946043 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.544         |\n","| fps                | 498           |\n","| n_updates          | 680           |\n","| policy_entropy     | 0.5317858     |\n","| policy_loss        | -0.0021437004 |\n","| serial_timesteps   | 87040         |\n","| time_elapsed       | 175           |\n","| total_timesteps    | 87040         |\n","| value_loss         | 3.483165      |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0006925484 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.0423       |\n","| fps                | 493          |\n","| n_updates          | 681          |\n","| policy_entropy     | 0.5332929    |\n","| policy_loss        | 0.003586695  |\n","| serial_timesteps   | 87168        |\n","| time_elapsed       | 175          |\n","| total_timesteps    | 87168        |\n","| value_loss         | 202.24767    |\n","-------------------------------------\n","---------------------------------------\n","| approxkl           | 1.957846e-05   |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.00295        |\n","| fps                | 505            |\n","| n_updates          | 682            |\n","| policy_entropy     | 0.54668033     |\n","| policy_loss        | -0.00054434955 |\n","| serial_timesteps   | 87296          |\n","| time_elapsed       | 175            |\n","| total_timesteps    | 87296          |\n","| value_loss         | 205.9851       |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 2.0256693e-06 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.134        |\n","| fps                | 539           |\n","| n_updates          | 683           |\n","| policy_entropy     | 0.5402531     |\n","| policy_loss        | 0.00010435516 |\n","| serial_timesteps   | 87424         |\n","| time_elapsed       | 176           |\n","| total_timesteps    | 87424         |\n","| value_loss         | 3.858238      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 5.9293834e-06 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00939      |\n","| fps                | 511           |\n","| n_updates          | 684           |\n","| policy_entropy     | 0.4719904     |\n","| policy_loss        | 0.00013774796 |\n","| serial_timesteps   | 87552         |\n","| time_elapsed       | 176           |\n","| total_timesteps    | 87552         |\n","| value_loss         | 208.4629      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00016822234 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0154       |\n","| fps                | 527           |\n","| n_updates          | 685           |\n","| policy_entropy     | 0.49017757    |\n","| policy_loss        | -0.0028744114 |\n","| serial_timesteps   | 87680         |\n","| time_elapsed       | 176           |\n","| total_timesteps    | 87680         |\n","| value_loss         | 198.05898     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00025107857 |\n","| clipfrac           | 0.001953125   |\n","| explained_variance | 0.717         |\n","| fps                | 493           |\n","| n_updates          | 686           |\n","| policy_entropy     | 0.46266738    |\n","| policy_loss        | 0.00044793356 |\n","| serial_timesteps   | 87808         |\n","| time_elapsed       | 176           |\n","| total_timesteps    | 87808         |\n","| value_loss         | 3.5908015     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.002124824   |\n","| clipfrac           | 0.01953125    |\n","| explained_variance | 0.253         |\n","| fps                | 508           |\n","| n_updates          | 687           |\n","| policy_entropy     | 0.50443065    |\n","| policy_loss        | -4.288205e-05 |\n","| serial_timesteps   | 87936         |\n","| time_elapsed       | 177           |\n","| total_timesteps    | 87936         |\n","| value_loss         | 164.75899     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 3.41868e-05   |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.284        |\n","| fps                | 482           |\n","| n_updates          | 688           |\n","| policy_entropy     | 0.47126922    |\n","| policy_loss        | 0.00023621134 |\n","| serial_timesteps   | 88064         |\n","| time_elapsed       | 177           |\n","| total_timesteps    | 88064         |\n","| value_loss         | 3.5569189     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 4.9768336e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.000916      |\n","| fps                | 495            |\n","| n_updates          | 689            |\n","| policy_entropy     | 0.5337395      |\n","| policy_loss        | -0.00033965008 |\n","| serial_timesteps   | 88192          |\n","| time_elapsed       | 177            |\n","| total_timesteps    | 88192          |\n","| value_loss         | 185.04074      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 5.022717e-06  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0867        |\n","| fps                | 485           |\n","| n_updates          | 690           |\n","| policy_entropy     | 0.49997216    |\n","| policy_loss        | 0.00032815104 |\n","| serial_timesteps   | 88320         |\n","| time_elapsed       | 177           |\n","| total_timesteps    | 88320         |\n","| value_loss         | 194.38162     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 3.770205e-05 |\n","| clipfrac           | 0.0          |\n","| explained_variance | -0.0812      |\n","| fps                | 512          |\n","| n_updates          | 691          |\n","| policy_entropy     | 0.51110864   |\n","| policy_loss        | 0.0003114252 |\n","| serial_timesteps   | 88448        |\n","| time_elapsed       | 178          |\n","| total_timesteps    | 88448        |\n","| value_loss         | 3.510622     |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 3.4687906e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0175        |\n","| fps                | 518           |\n","| n_updates          | 692           |\n","| policy_entropy     | 0.52361494    |\n","| policy_loss        | 0.00013008644 |\n","| serial_timesteps   | 88576         |\n","| time_elapsed       | 178           |\n","| total_timesteps    | 88576         |\n","| value_loss         | 206.93996     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 9.4697225e-06  |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.00535       |\n","| fps                | 509            |\n","| n_updates          | 693            |\n","| policy_entropy     | 0.50527084     |\n","| policy_loss        | -0.00034924317 |\n","| serial_timesteps   | 88704          |\n","| time_elapsed       | 178            |\n","| total_timesteps    | 88704          |\n","| value_loss         | 207.43643      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 3.7225596e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0212        |\n","| fps                | 499           |\n","| n_updates          | 694           |\n","| policy_entropy     | 0.49750078    |\n","| policy_loss        | 0.0008765999  |\n","| serial_timesteps   | 88832         |\n","| time_elapsed       | 178           |\n","| total_timesteps    | 88832         |\n","| value_loss         | 3.6120217     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 2.046595e-05   |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.0101         |\n","| fps                | 519            |\n","| n_updates          | 695            |\n","| policy_entropy     | 0.56018347     |\n","| policy_loss        | -1.8954976e-05 |\n","| serial_timesteps   | 88960          |\n","| time_elapsed       | 179            |\n","| total_timesteps    | 88960          |\n","| value_loss         | 207.26195      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 2.2137156e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0212        |\n","| fps                | 529           |\n","| n_updates          | 696           |\n","| policy_entropy     | 0.54947495    |\n","| policy_loss        | -0.0005003887 |\n","| serial_timesteps   | 89088         |\n","| time_elapsed       | 179           |\n","| total_timesteps    | 89088         |\n","| value_loss         | 202.1731      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 9.489472e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.163         |\n","| fps                | 511           |\n","| n_updates          | 697           |\n","| policy_entropy     | 0.54721105    |\n","| policy_loss        | -0.0002255456 |\n","| serial_timesteps   | 89216         |\n","| time_elapsed       | 179           |\n","| total_timesteps    | 89216         |\n","| value_loss         | 3.475583      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00014383139 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0155        |\n","| fps                | 508           |\n","| n_updates          | 698           |\n","| policy_entropy     | 0.4683242     |\n","| policy_loss        | 0.0002282901  |\n","| serial_timesteps   | 89344         |\n","| time_elapsed       | 179           |\n","| total_timesteps    | 89344         |\n","| value_loss         | 201.18967     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 2.6563857e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.342         |\n","| fps                | 466           |\n","| n_updates          | 699           |\n","| policy_entropy     | 0.5077145     |\n","| policy_loss        | 0.00045658264 |\n","| serial_timesteps   | 89472         |\n","| time_elapsed       | 180           |\n","| total_timesteps    | 89472         |\n","| value_loss         | 4.1041927     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 4.2345615e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.0196         |\n","| fps                | 508            |\n","| n_updates          | 700            |\n","| policy_entropy     | 0.49361303     |\n","| policy_loss        | -0.00037823408 |\n","| serial_timesteps   | 89600          |\n","| time_elapsed       | 180            |\n","| total_timesteps    | 89600          |\n","| value_loss         | 125.74471      |\n","---------------------------------------\n","---------------------------------------\n","| approxkl           | 2.931936e-05   |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.0166        |\n","| fps                | 527            |\n","| n_updates          | 701            |\n","| policy_entropy     | 0.55179864     |\n","| policy_loss        | -0.00037023087 |\n","| serial_timesteps   | 89728          |\n","| time_elapsed       | 180            |\n","| total_timesteps    | 89728          |\n","| value_loss         | 204.94392      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0005596251  |\n","| clipfrac           | 0.00390625    |\n","| explained_variance | 0.903         |\n","| fps                | 498           |\n","| n_updates          | 702           |\n","| policy_entropy     | 0.52299625    |\n","| policy_loss        | -0.0042857546 |\n","| serial_timesteps   | 89856         |\n","| time_elapsed       | 180           |\n","| total_timesteps    | 89856         |\n","| value_loss         | 3.2020154     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0064579626 |\n","| clipfrac           | 0.107421875  |\n","| explained_variance | 0.263        |\n","| fps                | 462          |\n","| n_updates          | 703          |\n","| policy_entropy     | 0.5354812    |\n","| policy_loss        | 0.0006044572 |\n","| serial_timesteps   | 89984        |\n","| time_elapsed       | 181          |\n","| total_timesteps    | 89984        |\n","| value_loss         | 159.12701    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00012846498 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0711        |\n","| fps                | 561           |\n","| n_updates          | 704           |\n","| policy_entropy     | 0.51973534    |\n","| policy_loss        | 0.00018479687 |\n","| serial_timesteps   | 90112         |\n","| time_elapsed       | 181           |\n","| total_timesteps    | 90112         |\n","| value_loss         | 193.26196     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 2.2608592e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.196         |\n","| fps                | 476           |\n","| n_updates          | 705           |\n","| policy_entropy     | 0.57145494    |\n","| policy_loss        | 0.0007984892  |\n","| serial_timesteps   | 90240         |\n","| time_elapsed       | 181           |\n","| total_timesteps    | 90240         |\n","| value_loss         | 3.4072492     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.000681438  |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.00605      |\n","| fps                | 513          |\n","| n_updates          | 706          |\n","| policy_entropy     | 0.5582265    |\n","| policy_loss        | -0.004381207 |\n","| serial_timesteps   | 90368        |\n","| time_elapsed       | 181          |\n","| total_timesteps    | 90368        |\n","| value_loss         | 211.14877    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00018355096 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.125         |\n","| fps                | 518           |\n","| n_updates          | 707           |\n","| policy_entropy     | 0.49267638    |\n","| policy_loss        | -0.0019857434 |\n","| serial_timesteps   | 90496         |\n","| time_elapsed       | 182           |\n","| total_timesteps    | 90496         |\n","| value_loss         | 189.53094     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0002104457  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.72          |\n","| fps                | 538           |\n","| n_updates          | 708           |\n","| policy_entropy     | 0.563541      |\n","| policy_loss        | -0.0017628653 |\n","| serial_timesteps   | 90624         |\n","| time_elapsed       | 182           |\n","| total_timesteps    | 90624         |\n","| value_loss         | 3.0344546     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0008814044 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.0213       |\n","| fps                | 531          |\n","| n_updates          | 709          |\n","| policy_entropy     | 0.52651155   |\n","| policy_loss        | 0.0007223892 |\n","| serial_timesteps   | 90752        |\n","| time_elapsed       | 182          |\n","| total_timesteps    | 90752        |\n","| value_loss         | 205.36305    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 1.2932668e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0488        |\n","| fps                | 497           |\n","| n_updates          | 710           |\n","| policy_entropy     | 0.53475654    |\n","| policy_loss        | 0.00039582117 |\n","| serial_timesteps   | 90880         |\n","| time_elapsed       | 182           |\n","| total_timesteps    | 90880         |\n","| value_loss         | 190.47993     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 1.835866e-05 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.646        |\n","| fps                | 482          |\n","| n_updates          | 711          |\n","| policy_entropy     | 0.51416755   |\n","| policy_loss        | -0.000245512 |\n","| serial_timesteps   | 91008        |\n","| time_elapsed       | 183          |\n","| total_timesteps    | 91008        |\n","| value_loss         | 3.1245618    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00031716767 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0182        |\n","| fps                | 501           |\n","| n_updates          | 712           |\n","| policy_entropy     | 0.52701485    |\n","| policy_loss        | -0.0004877064 |\n","| serial_timesteps   | 91136         |\n","| time_elapsed       | 183           |\n","| total_timesteps    | 91136         |\n","| value_loss         | 201.81148     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00013688683 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.376        |\n","| fps                | 523           |\n","| n_updates          | 713           |\n","| policy_entropy     | 0.49361217    |\n","| policy_loss        | -0.0026929504 |\n","| serial_timesteps   | 91264         |\n","| time_elapsed       | 183           |\n","| total_timesteps    | 91264         |\n","| value_loss         | 3.603941      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00012924832 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.000749     |\n","| fps                | 509           |\n","| n_updates          | 714           |\n","| policy_entropy     | 0.52377754    |\n","| policy_loss        | 0.00114807    |\n","| serial_timesteps   | 91392         |\n","| time_elapsed       | 183           |\n","| total_timesteps    | 91392         |\n","| value_loss         | 188.21576     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 4.089107e-05   |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.0136         |\n","| fps                | 512            |\n","| n_updates          | 715            |\n","| policy_entropy     | 0.46732053     |\n","| policy_loss        | -0.00041358138 |\n","| serial_timesteps   | 91520          |\n","| time_elapsed       | 184            |\n","| total_timesteps    | 91520          |\n","| value_loss         | 210.29552      |\n","---------------------------------------\n","---------------------------------------\n","| approxkl           | 8.88327e-05    |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.555         |\n","| fps                | 515            |\n","| n_updates          | 716            |\n","| policy_entropy     | 0.5429762      |\n","| policy_loss        | -0.00050813274 |\n","| serial_timesteps   | 91648          |\n","| time_elapsed       | 184            |\n","| total_timesteps    | 91648          |\n","| value_loss         | 3.9472258      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00019568883 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0314        |\n","| fps                | 559           |\n","| n_updates          | 717           |\n","| policy_entropy     | 0.49291253    |\n","| policy_loss        | -0.0018624873 |\n","| serial_timesteps   | 91776         |\n","| time_elapsed       | 184           |\n","| total_timesteps    | 91776         |\n","| value_loss         | 207.32721     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00012093438 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.144         |\n","| fps                | 505           |\n","| n_updates          | 718           |\n","| policy_entropy     | 0.45380312    |\n","| policy_loss        | -0.0011890022 |\n","| serial_timesteps   | 91904         |\n","| time_elapsed       | 184           |\n","| total_timesteps    | 91904         |\n","| value_loss         | 189.75998     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0002733056 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.497        |\n","| fps                | 511          |\n","| n_updates          | 719          |\n","| policy_entropy     | 0.5423829    |\n","| policy_loss        | 0.0014346134 |\n","| serial_timesteps   | 92032        |\n","| time_elapsed       | 185          |\n","| total_timesteps    | 92032        |\n","| value_loss         | 2.9636345    |\n","-------------------------------------\n","---------------------------------------\n","| approxkl           | 9.810931e-07   |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.0199        |\n","| fps                | 482            |\n","| n_updates          | 720            |\n","| policy_entropy     | 0.5059478      |\n","| policy_loss        | -5.6003337e-05 |\n","| serial_timesteps   | 92160          |\n","| time_elapsed       | 185            |\n","| total_timesteps    | 92160          |\n","| value_loss         | 211.077        |\n","---------------------------------------\n","---------------------------------------\n","| approxkl           | 2.0726144e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.0894         |\n","| fps                | 504            |\n","| n_updates          | 721            |\n","| policy_entropy     | 0.5365191      |\n","| policy_loss        | -0.00048969465 |\n","| serial_timesteps   | 92288          |\n","| time_elapsed       | 185            |\n","| total_timesteps    | 92288          |\n","| value_loss         | 189.00468      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 2.910293e-06  |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.475        |\n","| fps                | 491           |\n","| n_updates          | 722           |\n","| policy_entropy     | 0.4812764     |\n","| policy_loss        | 0.00012961624 |\n","| serial_timesteps   | 92416         |\n","| time_elapsed       | 185           |\n","| total_timesteps    | 92416         |\n","| value_loss         | 3.3546627     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00018405233 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0144        |\n","| fps                | 515           |\n","| n_updates          | 723           |\n","| policy_entropy     | 0.53860873    |\n","| policy_loss        | -0.0018533525 |\n","| serial_timesteps   | 92544         |\n","| time_elapsed       | 186           |\n","| total_timesteps    | 92544         |\n","| value_loss         | 212.45552     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 5.6594345e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.337          |\n","| fps                | 498            |\n","| n_updates          | 724            |\n","| policy_entropy     | 0.48761815     |\n","| policy_loss        | -0.00046469894 |\n","| serial_timesteps   | 92672          |\n","| time_elapsed       | 186            |\n","| total_timesteps    | 92672          |\n","| value_loss         | 3.1531756      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 4.852395e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0358        |\n","| fps                | 554           |\n","| n_updates          | 725           |\n","| policy_entropy     | 0.51623213    |\n","| policy_loss        | -0.0009526266 |\n","| serial_timesteps   | 92800         |\n","| time_elapsed       | 186           |\n","| total_timesteps    | 92800         |\n","| value_loss         | 128.04405     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 1.07142305e-05 |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.00403       |\n","| fps                | 493            |\n","| n_updates          | 726            |\n","| policy_entropy     | 0.57470393     |\n","| policy_loss        | -7.459987e-05  |\n","| serial_timesteps   | 92928          |\n","| time_elapsed       | 186            |\n","| total_timesteps    | 92928          |\n","| value_loss         | 215.21045      |\n","---------------------------------------\n","---------------------------------------\n","| approxkl           | 6.571526e-05   |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.591          |\n","| fps                | 475            |\n","| n_updates          | 727            |\n","| policy_entropy     | 0.5299695      |\n","| policy_loss        | -0.00039677788 |\n","| serial_timesteps   | 93056          |\n","| time_elapsed       | 187            |\n","| total_timesteps    | 93056          |\n","| value_loss         | 3.0745595      |\n","---------------------------------------\n","-------------------------------------\n","| approxkl           | 6.049315e-06 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.0123       |\n","| fps                | 530          |\n","| n_updates          | 728          |\n","| policy_entropy     | 0.51936245   |\n","| policy_loss        | 6.21893e-05  |\n","| serial_timesteps   | 93184        |\n","| time_elapsed       | 187          |\n","| total_timesteps    | 93184        |\n","| value_loss         | 205.27563    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 7.618542e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00765      |\n","| fps                | 532           |\n","| n_updates          | 729           |\n","| policy_entropy     | 0.50712204    |\n","| policy_loss        | -0.0010160911 |\n","| serial_timesteps   | 93312         |\n","| time_elapsed       | 187           |\n","| total_timesteps    | 93312         |\n","| value_loss         | 211.86476     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0017567452  |\n","| clipfrac           | 0.013671875   |\n","| explained_variance | -0.376        |\n","| fps                | 498           |\n","| n_updates          | 730           |\n","| policy_entropy     | 0.52967894    |\n","| policy_loss        | -0.0007421882 |\n","| serial_timesteps   | 93440         |\n","| time_elapsed       | 187           |\n","| total_timesteps    | 93440         |\n","| value_loss         | 4.8420496     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0007786796  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.22          |\n","| fps                | 450           |\n","| n_updates          | 731           |\n","| policy_entropy     | 0.47125584    |\n","| policy_loss        | -0.0015682448 |\n","| serial_timesteps   | 93568         |\n","| time_elapsed       | 188           |\n","| total_timesteps    | 93568         |\n","| value_loss         | 176.78659     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 9.920519e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00438      |\n","| fps                | 512           |\n","| n_updates          | 732           |\n","| policy_entropy     | 0.48688453    |\n","| policy_loss        | -0.0021500597 |\n","| serial_timesteps   | 93696         |\n","| time_elapsed       | 188           |\n","| total_timesteps    | 93696         |\n","| value_loss         | 213.46158     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 1.5464311e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.183         |\n","| fps                | 512            |\n","| n_updates          | 733            |\n","| policy_entropy     | 0.4924769      |\n","| policy_loss        | -1.9519706e-05 |\n","| serial_timesteps   | 93824          |\n","| time_elapsed       | 188            |\n","| total_timesteps    | 93824          |\n","| value_loss         | 3.6057353      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 1.5080825e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.00944       |\n","| fps                | 535           |\n","| n_updates          | 734           |\n","| policy_entropy     | 0.54267853    |\n","| policy_loss        | -0.0005850226 |\n","| serial_timesteps   | 93952         |\n","| time_elapsed       | 188           |\n","| total_timesteps    | 93952         |\n","| value_loss         | 212.06442     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 5.785552e-06   |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.00547        |\n","| fps                | 526            |\n","| n_updates          | 735            |\n","| policy_entropy     | 0.5330114      |\n","| policy_loss        | -0.00010932388 |\n","| serial_timesteps   | 94080          |\n","| time_elapsed       | 189            |\n","| total_timesteps    | 94080          |\n","| value_loss         | 211.61334      |\n","---------------------------------------\n","---------------------------------------\n","| approxkl           | 1.0945098e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.409         |\n","| fps                | 532            |\n","| n_updates          | 736            |\n","| policy_entropy     | 0.50260496     |\n","| policy_loss        | -0.00011466665 |\n","| serial_timesteps   | 94208          |\n","| time_elapsed       | 189            |\n","| total_timesteps    | 94208          |\n","| value_loss         | 3.5374677      |\n","---------------------------------------\n","---------------------------------------\n","| approxkl           | 0.00013742723  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.0191         |\n","| fps                | 517            |\n","| n_updates          | 737            |\n","| policy_entropy     | 0.51020664     |\n","| policy_loss        | -0.00064065633 |\n","| serial_timesteps   | 94336          |\n","| time_elapsed       | 189            |\n","| total_timesteps    | 94336          |\n","| value_loss         | 209.08339      |\n","---------------------------------------\n","-------------------------------------\n","| approxkl           | 7.497586e-06 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.103        |\n","| fps                | 524          |\n","| n_updates          | 738          |\n","| policy_entropy     | 0.520554     |\n","| policy_loss        | 8.105021e-05 |\n","| serial_timesteps   | 94464        |\n","| time_elapsed       | 189          |\n","| total_timesteps    | 94464        |\n","| value_loss         | 3.546874     |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 1.7504986e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0117       |\n","| fps                | 505           |\n","| n_updates          | 739           |\n","| policy_entropy     | 0.50470495    |\n","| policy_loss        | 4.380243e-06  |\n","| serial_timesteps   | 94592         |\n","| time_elapsed       | 190           |\n","| total_timesteps    | 94592         |\n","| value_loss         | 187.87013     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 2.3690118e-06 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0369       |\n","| fps                | 510           |\n","| n_updates          | 740           |\n","| policy_entropy     | 0.47800255    |\n","| policy_loss        | 6.6278735e-05 |\n","| serial_timesteps   | 94720         |\n","| time_elapsed       | 190           |\n","| total_timesteps    | 94720         |\n","| value_loss         | 210.5033      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 8.1041944e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.284         |\n","| fps                | 474           |\n","| n_updates          | 741           |\n","| policy_entropy     | 0.47128934    |\n","| policy_loss        | -4.582014e-05 |\n","| serial_timesteps   | 94848         |\n","| time_elapsed       | 190           |\n","| total_timesteps    | 94848         |\n","| value_loss         | 3.5450583     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 6.3669204e-06 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0203       |\n","| fps                | 506           |\n","| n_updates          | 742           |\n","| policy_entropy     | 0.4771552     |\n","| policy_loss        | 4.23342e-05   |\n","| serial_timesteps   | 94976         |\n","| time_elapsed       | 190           |\n","| total_timesteps    | 94976         |\n","| value_loss         | 211.78188     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 1.8022918e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.0125        |\n","| fps                | 491            |\n","| n_updates          | 743            |\n","| policy_entropy     | 0.5121823      |\n","| policy_loss        | -0.00073862216 |\n","| serial_timesteps   | 95104          |\n","| time_elapsed       | 191            |\n","| total_timesteps    | 95104          |\n","| value_loss         | 208.62991      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00024571965 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.391        |\n","| fps                | 552           |\n","| n_updates          | 744           |\n","| policy_entropy     | 0.5303547     |\n","| policy_loss        | -0.0036717919 |\n","| serial_timesteps   | 95232         |\n","| time_elapsed       | 191           |\n","| total_timesteps    | 95232         |\n","| value_loss         | 3.7104979     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00011591353 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00272      |\n","| fps                | 518           |\n","| n_updates          | 745           |\n","| policy_entropy     | 0.55731577    |\n","| policy_loss        | 0.0007842437  |\n","| serial_timesteps   | 95360         |\n","| time_elapsed       | 191           |\n","| total_timesteps    | 95360         |\n","| value_loss         | 212.8501      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 5.3962476e-06 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0111        |\n","| fps                | 519           |\n","| n_updates          | 746           |\n","| policy_entropy     | 0.5217484     |\n","| policy_loss        | 1.723331e-05  |\n","| serial_timesteps   | 95488         |\n","| time_elapsed       | 191           |\n","| total_timesteps    | 95488         |\n","| value_loss         | 209.17332     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 2.397369e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.272         |\n","| fps                | 479           |\n","| n_updates          | 747           |\n","| policy_entropy     | 0.5352935     |\n","| policy_loss        | 0.00040481414 |\n","| serial_timesteps   | 95616         |\n","| time_elapsed       | 192           |\n","| total_timesteps    | 95616         |\n","| value_loss         | 3.2836778     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00019516642 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00692      |\n","| fps                | 479           |\n","| n_updates          | 748           |\n","| policy_entropy     | 0.53124225    |\n","| policy_loss        | -0.0027546785 |\n","| serial_timesteps   | 95744         |\n","| time_elapsed       | 192           |\n","| total_timesteps    | 95744         |\n","| value_loss         | 210.13382     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 5.3272735e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.44          |\n","| fps                | 474           |\n","| n_updates          | 749           |\n","| policy_entropy     | 0.5272085     |\n","| policy_loss        | -0.0010938194 |\n","| serial_timesteps   | 95872         |\n","| time_elapsed       | 192           |\n","| total_timesteps    | 95872         |\n","| value_loss         | 3.9994524     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 0.00015555437  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.0267         |\n","| fps                | 501            |\n","| n_updates          | 750            |\n","| policy_entropy     | 0.49862373     |\n","| policy_loss        | -0.00056899723 |\n","| serial_timesteps   | 96000          |\n","| time_elapsed       | 192            |\n","| total_timesteps    | 96000          |\n","| value_loss         | 126.906136     |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 2.5202196e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0653        |\n","| fps                | 502           |\n","| n_updates          | 751           |\n","| policy_entropy     | 0.50486666    |\n","| policy_loss        | -0.0004173664 |\n","| serial_timesteps   | 96128         |\n","| time_elapsed       | 193           |\n","| total_timesteps    | 96128         |\n","| value_loss         | 198.70508     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00018149454 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0344        |\n","| fps                | 477           |\n","| n_updates          | 752           |\n","| policy_entropy     | 0.5422839     |\n","| policy_loss        | -0.0025409814 |\n","| serial_timesteps   | 96256         |\n","| time_elapsed       | 193           |\n","| total_timesteps    | 96256         |\n","| value_loss         | 3.381795      |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0010730968 |\n","| clipfrac           | 0.005859375  |\n","| explained_variance | -0.0171      |\n","| fps                | 508          |\n","| n_updates          | 753          |\n","| policy_entropy     | 0.52579117   |\n","| policy_loss        | 0.0009775913 |\n","| serial_timesteps   | 96384        |\n","| time_elapsed       | 193          |\n","| total_timesteps    | 96384        |\n","| value_loss         | 204.88336    |\n","-------------------------------------\n","---------------------------------------\n","| approxkl           | 3.9808972e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.022          |\n","| fps                | 503            |\n","| n_updates          | 754            |\n","| policy_entropy     | 0.55997247     |\n","| policy_loss        | -0.00044353178 |\n","| serial_timesteps   | 96512          |\n","| time_elapsed       | 194            |\n","| total_timesteps    | 96512          |\n","| value_loss         | 204.6165       |\n","---------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0006263433 |\n","| clipfrac           | 0.0          |\n","| explained_variance | -0.0027      |\n","| fps                | 498          |\n","| n_updates          | 755          |\n","| policy_entropy     | 0.5274507    |\n","| policy_loss        | -0.005296217 |\n","| serial_timesteps   | 96640        |\n","| time_elapsed       | 194          |\n","| total_timesteps    | 96640        |\n","| value_loss         | 3.3388557    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00069543923 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0142       |\n","| fps                | 483           |\n","| n_updates          | 756           |\n","| policy_entropy     | 0.52011037    |\n","| policy_loss        | -0.0003327811 |\n","| serial_timesteps   | 96768         |\n","| time_elapsed       | 194           |\n","| total_timesteps    | 96768         |\n","| value_loss         | 212.09816     |\n","--------------------------------------\n","----------------------------------------\n","| approxkl           | 3.093775e-05    |\n","| clipfrac           | 0.0             |\n","| explained_variance | -0.00454        |\n","| fps                | 451             |\n","| n_updates          | 757             |\n","| policy_entropy     | 0.47621047      |\n","| policy_loss        | -0.000115017174 |\n","| serial_timesteps   | 96896           |\n","| time_elapsed       | 194             |\n","| total_timesteps    | 96896           |\n","| value_loss         | 209.47318       |\n","----------------------------------------\n","---------------------------------------\n","| approxkl           | 5.6936744e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | -0.175         |\n","| fps                | 452            |\n","| n_updates          | 758            |\n","| policy_entropy     | 0.5508561      |\n","| policy_loss        | -0.00028860394 |\n","| serial_timesteps   | 97024          |\n","| time_elapsed       | 195            |\n","| total_timesteps    | 97024          |\n","| value_loss         | 3.656048       |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00053873024 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0131        |\n","| fps                | 496           |\n","| n_updates          | 759           |\n","| policy_entropy     | 0.5133348     |\n","| policy_loss        | 0.00057163707 |\n","| serial_timesteps   | 97152         |\n","| time_elapsed       | 195           |\n","| total_timesteps    | 97152         |\n","| value_loss         | 208.59915     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 2.3435325e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0347       |\n","| fps                | 542           |\n","| n_updates          | 760           |\n","| policy_entropy     | 0.50567573    |\n","| policy_loss        | -0.0006111318 |\n","| serial_timesteps   | 97280         |\n","| time_elapsed       | 195           |\n","| total_timesteps    | 97280         |\n","| value_loss         | 212.42476     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 8.281406e-06 |\n","| clipfrac           | 0.0          |\n","| explained_variance | -0.141       |\n","| fps                | 483          |\n","| n_updates          | 761          |\n","| policy_entropy     | 0.5716155    |\n","| policy_loss        | 3.687374e-05 |\n","| serial_timesteps   | 97408        |\n","| time_elapsed       | 195          |\n","| total_timesteps    | 97408        |\n","| value_loss         | 3.5919662    |\n","-------------------------------------\n","---------------------------------------\n","| approxkl           | 1.9565836e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.0241         |\n","| fps                | 515            |\n","| n_updates          | 762            |\n","| policy_entropy     | 0.54037046     |\n","| policy_loss        | -0.00065813377 |\n","| serial_timesteps   | 97536          |\n","| time_elapsed       | 196            |\n","| total_timesteps    | 97536          |\n","| value_loss         | 206.9066       |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00014096286 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.136        |\n","| fps                | 498           |\n","| n_updates          | 763           |\n","| policy_entropy     | 0.5117744     |\n","| policy_loss        | -0.002776838  |\n","| serial_timesteps   | 97664         |\n","| time_elapsed       | 196           |\n","| total_timesteps    | 97664         |\n","| value_loss         | 3.7671328     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 7.6796285e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0505        |\n","| fps                | 514           |\n","| n_updates          | 764           |\n","| policy_entropy     | 0.5209128     |\n","| policy_loss        | 0.00068787043 |\n","| serial_timesteps   | 97792         |\n","| time_elapsed       | 196           |\n","| total_timesteps    | 97792         |\n","| value_loss         | 177.96039     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 2.9228033e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.046          |\n","| fps                | 519            |\n","| n_updates          | 765            |\n","| policy_entropy     | 0.48954254     |\n","| policy_loss        | -2.5972258e-06 |\n","| serial_timesteps   | 97920          |\n","| time_elapsed       | 196            |\n","| total_timesteps    | 97920          |\n","| value_loss         | 201.71979      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 1.3201582e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.683        |\n","| fps                | 470           |\n","| n_updates          | 766           |\n","| policy_entropy     | 0.5618502     |\n","| policy_loss        | 0.00030561356 |\n","| serial_timesteps   | 98048         |\n","| time_elapsed       | 197           |\n","| total_timesteps    | 98048         |\n","| value_loss         | 3.6443043     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 4.732761e-06  |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.0023       |\n","| fps                | 514           |\n","| n_updates          | 767           |\n","| policy_entropy     | 0.5178949     |\n","| policy_loss        | 0.00038412795 |\n","| serial_timesteps   | 98176         |\n","| time_elapsed       | 197           |\n","| total_timesteps    | 98176         |\n","| value_loss         | 211.26833     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 1.23161735e-05 |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.018          |\n","| fps                | 534            |\n","| n_updates          | 768            |\n","| policy_entropy     | 0.509404       |\n","| policy_loss        | -9.00788e-05   |\n","| serial_timesteps   | 98304          |\n","| time_elapsed       | 197            |\n","| total_timesteps    | 98304          |\n","| value_loss         | 205.5871       |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 2.340769e-05  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.29          |\n","| fps                | 515           |\n","| n_updates          | 769           |\n","| policy_entropy     | 0.5590999     |\n","| policy_loss        | 0.00015969959 |\n","| serial_timesteps   | 98432         |\n","| time_elapsed       | 197           |\n","| total_timesteps    | 98432         |\n","| value_loss         | 3.3324213     |\n","--------------------------------------\n","---------------------------------------\n","| approxkl           | 0.00030716613  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.0329         |\n","| fps                | 483            |\n","| n_updates          | 770            |\n","| policy_entropy     | 0.5376779      |\n","| policy_loss        | -0.00068457663 |\n","| serial_timesteps   | 98560          |\n","| time_elapsed       | 198            |\n","| total_timesteps    | 98560          |\n","| value_loss         | 207.50261      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 8.217128e-06  |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.00954      |\n","| fps                | 390           |\n","| n_updates          | 771           |\n","| policy_entropy     | 0.5701371     |\n","| policy_loss        | 8.7757595e-05 |\n","| serial_timesteps   | 98688         |\n","| time_elapsed       | 198           |\n","| total_timesteps    | 98688         |\n","| value_loss         | 210.76587     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 1.3964448e-05 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -1            |\n","| fps                | 393           |\n","| n_updates          | 772           |\n","| policy_entropy     | 0.5167682     |\n","| policy_loss        | 0.00038810936 |\n","| serial_timesteps   | 98816         |\n","| time_elapsed       | 198           |\n","| total_timesteps    | 98816         |\n","| value_loss         | 4.9224906     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 0.0002060164 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.0133       |\n","| fps                | 474          |\n","| n_updates          | 773          |\n","| policy_entropy     | 0.5748448    |\n","| policy_loss        | 0.0012356546 |\n","| serial_timesteps   | 98944        |\n","| time_elapsed       | 199          |\n","| total_timesteps    | 98944        |\n","| value_loss         | 206.13751    |\n","-------------------------------------\n","---------------------------------------\n","| approxkl           | 1.0267735e-05  |\n","| clipfrac           | 0.0            |\n","| explained_variance | 0.287          |\n","| fps                | 508            |\n","| n_updates          | 774            |\n","| policy_entropy     | 0.5381023      |\n","| policy_loss        | -0.00033289718 |\n","| serial_timesteps   | 99072          |\n","| time_elapsed       | 199            |\n","| total_timesteps    | 99072          |\n","| value_loss         | 3.4983778      |\n","---------------------------------------\n","--------------------------------------\n","| approxkl           | 5.94376e-05   |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0332        |\n","| fps                | 528           |\n","| n_updates          | 775           |\n","| policy_entropy     | 0.5457572     |\n","| policy_loss        | -0.0014215021 |\n","| serial_timesteps   | 99200         |\n","| time_elapsed       | 199           |\n","| total_timesteps    | 99200         |\n","| value_loss         | 126.00416     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 9.623035e-06 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.00438      |\n","| fps                | 519          |\n","| n_updates          | 776          |\n","| policy_entropy     | 0.540999     |\n","| policy_loss        | 0.0006248497 |\n","| serial_timesteps   | 99328        |\n","| time_elapsed       | 199          |\n","| total_timesteps    | 99328        |\n","| value_loss         | 209.85329    |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00010880188 |\n","| clipfrac           | 0.0           |\n","| explained_variance | -0.161        |\n","| fps                | 516           |\n","| n_updates          | 777           |\n","| policy_entropy     | 0.484756      |\n","| policy_loss        | -0.000934641  |\n","| serial_timesteps   | 99456         |\n","| time_elapsed       | 200           |\n","| total_timesteps    | 99456         |\n","| value_loss         | 3.7249854     |\n","--------------------------------------\n","-------------------------------------\n","| approxkl           | 3.408008e-05 |\n","| clipfrac           | 0.0          |\n","| explained_variance | 0.000433     |\n","| fps                | 497          |\n","| n_updates          | 778          |\n","| policy_entropy     | 0.58305      |\n","| policy_loss        | 0.001084558  |\n","| serial_timesteps   | 99584        |\n","| time_elapsed       | 200          |\n","| total_timesteps    | 99584        |\n","| value_loss         | 204.557      |\n","-------------------------------------\n","--------------------------------------\n","| approxkl           | 1.2691956e-06 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.0223        |\n","| fps                | 448           |\n","| n_updates          | 779           |\n","| policy_entropy     | 0.4896642     |\n","| policy_loss        | -7.329823e-05 |\n","| serial_timesteps   | 99712         |\n","| time_elapsed       | 200           |\n","| total_timesteps    | 99712         |\n","| value_loss         | 205.85114     |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.00044381298 |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.642         |\n","| fps                | 518           |\n","| n_updates          | 780           |\n","| policy_entropy     | 0.52085036    |\n","| policy_loss        | -0.003929645  |\n","| serial_timesteps   | 99840         |\n","| time_elapsed       | 200           |\n","| total_timesteps    | 99840         |\n","| value_loss         | 5.316996      |\n","--------------------------------------\n","--------------------------------------\n","| approxkl           | 0.0007949546  |\n","| clipfrac           | 0.0           |\n","| explained_variance | 0.517         |\n","| fps                | 434           |\n","| n_updates          | 781           |\n","| policy_entropy     | 0.514961      |\n","| policy_loss        | 0.00016275374 |\n","| serial_timesteps   | 99968         |\n","| time_elapsed       | 201           |\n","| total_timesteps    | 99968         |\n","| value_loss         | 119.33448     |\n","--------------------------------------\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["<stable_baselines.ppo2.ppo2.PPO2 at 0x7feb7c7f0550>"]},"metadata":{"tags":[]},"execution_count":18}]},{"cell_type":"markdown","metadata":{"id":"UuywPEOuzAfQ","colab_type":"text"},"source":["## Final Evalaution"]},{"cell_type":"code","metadata":{"id":"UPnmcObb0Jt8","colab_type":"code","outputId":"fdcadf4c-167e-442c-d0ed-93879f2f3151","executionInfo":{"status":"ok","timestamp":1582020089354,"user_tz":-120,"elapsed":327153,"user":{"displayName":"Ibrahim SOBH","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAYeQM_1LvthZBu6TRjB85IH8s-a5ZMla2NYwtX=s64","userId":"05222794542711757573"}},"colab":{"base_uri":"https://localhost:8080/","height":72}},"source":["env = model.get_env()\n","obs = env.reset()\n","r_list = []\n","\n","for i in range(10):\n","  print(\"\\riteration: {}\".format(i), end=\"\")\n","  reward_sum = 0.0\n","  for _ in range(1000):\n","          action, _ = model.predict(obs)\n","          obs, reward, done, _ = env.step(action)\n","          reward_sum += reward\n","          if done:\n","                  r_list.append(reward_sum)\n","                  reward_sum = 0.0\n","                  obs = env.reset()\n","\n","print('\\nmean, std')\n","print(np.mean(r_list), np.std(r_list))\n","results_mean_list.append(np.mean(r_list))\n","results_std_list.append(np.std(r_list)) \n","env.close()"],"execution_count":0,"outputs":[{"output_type":"stream","text":["iteration: 9\n","mean, std\n","188.66 24.20381\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"uhthMtlq0QJg","colab_type":"code","outputId":"9fca6176-ea45-4a4b-fa79-d99b9571b4a9","executionInfo":{"status":"ok","timestamp":1582020089355,"user_tz":-120,"elapsed":327129,"user":{"displayName":"Ibrahim SOBH","photoUrl":"https://lh3.googleusercontent.com/a-/AAuE7mAYeQM_1LvthZBu6TRjB85IH8s-a5ZMla2NYwtX=s64","userId":"05222794542711757573"}},"colab":{"base_uri":"https://localhost:8080/","height":283}},"source":["import matplotlib.pyplot as plt\n","fig, ax = plt.subplots()\n","plt.xticks(np.arange(len(results_mean_list)), ('random', 'expert', 'RL'))\n","x = np.arange(len(results_mean_list))\n","y = np.array(results_mean_list) \n","e = np.array(results_std_list)\n","plt.errorbar(x, y, e)"],"execution_count":0,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<ErrorbarContainer object of 3 artists>"]},"metadata":{"tags":[]},"execution_count":20},{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXhU9dn/8ffNvq/JsIewRzYDhEVR\nC8G2amtdqqBVaq0tImi19emi7WNrl19Xa7WutFq1IEJFrdo+rVTABXEhYV/CvgpkYQsEst6/P+aA\nIwWJzCSTTD6v68o1Z74zZ84dmHxycuY+52vujoiIJJZ68S5ARERiT+EuIpKAFO4iIglI4S4ikoAU\n7iIiCahBvAsASEpK8tTU1HiXISJSq2RlZeW7e/LJHqsR4Z6amsrixYvjXYaISK1iZltP9ZgOy4iI\nJCCFu4hIAlK4i4gkIIW7iEgCUriLiCQghbuISAJSuIuIJCCFu4hIAlK4i4jEyYTHFzHh8UVV8toK\ndxGRBKRwFxFJQAp3EZEEpHAXEUlACncRkQSkcBcRSUAKdxGRBKRwFxFJQAp3EZEEdNpwN7NuZjbf\nzFab2Sozuz0Yb2dmc81sfXDbNhg3M3vQzDaY2XIzG1rV34SIiHxcZfbcy4A73b0/MAqYamb9gR8A\nr7t7H+D14D7AxUCf4GsS8GjMqxYRkU902nB3913unh0sFwJrgC7AZcDTwdOeBi4Pli8DnvGwd4E2\nZtYp5pWLiMgpfapj7maWCgwB3gM6uPuu4KHdQIdguQuwPWK1HcHYia81ycwWm9nivLy8T1m2iIh8\nkkqHu5m1AOYAd7j7wcjH3N0B/zQbdvdp7p7h7hnJycmfZlURETmNSoW7mTUkHOwz3P2FYHjPscMt\nwW1uML4T6BaxetdgTEREqkllumUMeAJY4+6/j3joZeCGYPkG4O8R418NumZGAQciDt+IiEg1aFCJ\n54wGJgIrzGxpMHY38CtgtpndBGwFxgeP/RO4BNgAFAE3xrRiERE5rdOGu7u/DdgpHh53kuc7MDXK\nukREJAo6Q1VEJAEp3EVEEpDCXUQkAVXmA1UREakC4Y8oq4bCXUQkDg4UlbJuzyGaNapfJa+vwzIi\nItVs5c4DfPGhtzhwpJRGDaomhhXuIiLVaPbi7Vz56DuUlTtndWpFh1ZNqmQ7CncRkWpwtLScu15Y\nzveeX87w1La8ett5tGxSdUfGdcxdRKSKbd9bxJQZ2azYeYCpY3vxnc/2o369U50bGhsKdxGRKrQg\nJ5c7Zi2lvML501cz+Gz/DqdfKQYU7iIiVaCiwnlw3noeeH09/Tq05LHrh5Ga1Lzatq9wFxGJsf1F\nJdz+3FLeWJfHlUO78IvLB9G0iloeT0XhLiISQyt2HGDy9CzyCov5+eUDuW5kCuErp1cvhbuISIw8\n9/427nl5FUnNGzF78jmkd2sTt1oU7iIiUTpaWs49f1/J7MU7OL9PEg9cM4R2zRvFtabThruZPQl8\nEch194HB2CygX/CUNsB+d08PJtBeA+QEj73r7pNjXbSISE2xraCIW2ZkserDg9yW2Zs7Luxb5W2O\nlVGZPfengIeAZ44NuPuEY8tmdh9wIOL5G909PVYFiojUVPPX5nL7c0sAeOKGDMadVT1tjpVRmZmY\n3gz2yP9LML/qeCAztmWJiNRc5RXOA/9Zx4PzNtC/Uyseu34YKe2bxbusj4n2mPv5wB53Xx8x1sPM\nlgAHgR+5+1snW9HMJgGTAFJSUqIsQ0Skeuw9XMLtzy3hrfX5XD2sKz+7fCBNGlZvm2NlRBvu1wIz\nI+7vAlLcvcDMhgEvmdkAdz944oruPg2YBpCRkVF1FzUWEYmRZdv3M2VGNnmFxfzyykFcM7xbXNoc\nK+OMw93MGgBXAsOOjbl7MVAcLGeZ2UagL7A4yjpFROLG3Zn5/nZ+8vIqkls25vlbzmFw1/i1OVZG\nNHvuFwJr3X3HsQEzSwb2unu5mfUE+gCboqxRRCRujpaW88MXVzInewcX9E3mgQnptI1zm2NlVKYV\nciYwBkgysx3Aj939CeAaPn5IBuAC4KdmVgpUAJPdfW9sSxYRqR5bCw4zeXo2a3Yd5Fvj+nD7uD41\nos2xMirTLXPtKca/dpKxOcCc6MsSEYmv/6zew7dnL6WeGX/52nDGpoXiXdKnojNURUQilFc4989d\nx0PzNzCgc7jNsVu7mtXmWBkKdxGRQMGhYm5/bilvb8hnQkY37r1sQI1sc6wMhbuICLB0+36mTM8i\n/3AJv/7yICYMr93n3yjcRaROc3emv7eNn76yig6tmjBn8rkM6to63mVFTeEuInXWkZJyfvjiCl5Y\nspMx/ZL5w4R02jSr+W2OlaFwF5E6aUv+YSZPzyJnTyHfvrAvt2X2pl41tznOuvmcKntthbuI1Dmv\nrdrNnbOXUb9+uM1xTL/a1eZYGQp3EakzysoruG/uOh5dsJFBXVrzyHVDa2WbY2Uo3EWkTsg/VMy3\nZi7hnY0FXDsihR9f2r/WtjlWhsJdRBJe9rZ9TJmezb6iEn5z1WDGZ3SLd0lVTuEuIgnL3fnru1v5\n2aur6di6CXNuOZeBXWp/m2NlKNxFJCEVlZRx9wsreGnph2Smhbh/fDqtmzWMd1nVRuEuIglnU94h\nbpmezbrcQu78bF+mjq3+Nsd4U7iLSEL518rdfPdvy2hQ33j6xhFc0Dc53iXFhcJdRBJCWXkFv30t\nh8ff2MTZXVvzyPXD6NKmabzLihuFu4jUenmF4TbHRZsKuG5kCvdc2p/GDRK3zbEy6p3uCWb2pJnl\nmtnKiLGfmNlOM1safF0S8dhdZrbBzHLM7PNVVbiICEDW1r188Y9vkb1tH/ddfTa/uGJQnQ92qNye\n+1PAQ8AzJ4zf7+6/ixwws/6Ep98bAHQG/mNmfd29PAa1iogc5+489c4WfvGPNXRp25QXp4ygf+dW\n8S6rxqjMNHtvmllqJV/vMuA5dy8GNpvZBmAEsOiMKxQROcHh4jLuemEFLy/7kAvPCnHf+HRaN607\nbY6VcdrDMp/gVjNbHhy2aRuMdQG2RzxnRzD2X8xskpktNrPFeXl5UZQhInXJxrxDXP7wQl5d/iHf\n/Xw/pk3MULCfxJmG+6NALyAd2AXc92lfwN2nuXuGu2ckJ9fNViUR+XT+b8UuLntoIQWHS3jm6yPr\nZP96ZZ1Rt4y77zm2bGZ/Al4N7u4EIi/a0DUYExE5Y2XlFfz6X2v501ubSe/WhkeuG0rnOtzmWBln\nFO5m1snddwV3rwCOddK8DDxrZr8n/IFqH+D9qKsUkTort/Aotz67hPc372XiqO786ItnqRumEk4b\n7mY2ExgDJJnZDuDHwBgzSwcc2ALcDODuq8xsNrAaKAOmqlNGRM7UB1v2MnVGNgePlnL/hLO5YkjX\neJdUa5i7x7sGMjIyfPHixfEuQ0RqCHfnyYVb+OU/19C1bVMemziMtI5qczyRmWW5e8bJHtMZqiJS\noxwqLuP7c5bzj+W7+Fz/Dvxu/Nm0aqJumE9L4S4iNcaG3EImT89mU94hfnBxGjdf0BMzdcOcCYW7\niNQI/1i+i+89v4wmDesz/aaRnNs7Kd4l1WoKdxGJq9LyCn71f2t54u3NDE1pwyPXDaNj6ybxLqvW\nU7iLSNzkHjzK1Gez+WDLPr52bip3X3IWjRpEc+K8HKNwF5G4eG9TAbfOXMKho2U8cE06l6Wf9Eol\ncoYU7iJSrdydP7+1mV/9ay3d2zVj+k0j6dexZbzLSjgKdxGpNoeKy/je88v454rdXDSgI7+9ejAt\n1eZYJRTuIlIt1u8p5ObpWWwtKOLuS9L45vlqc6xKCncRqXKvLPuQ789ZTrNG9ZnxjZGM6tk+3iUl\nPIW7iFSZkrIKfvl/a/jLwi1kdG/Lw9cNpUMrtTlWB4W7iFSJPQePMmVGNllb93Hj6HCbY8P6anOs\nLgp3EYm5RRsLuG1mNkUl5Tx47RC+dHbneJdU5yjcRSRm3J1pb27iN//OIbV9M2Z+cxR9OqjNMR4U\n7iISE4VHS/nu35bzr1W7uWRQR35z1dm0aKyIiZfKTNbxJPBFINfdBwZjvwUuBUqAjcCN7r7fzFKB\nNUBOsPq77j65CuoWkRokZ3chk6dnsW1vET/6wlncdF4PtTnGWWU+3XgKuOiEsbnAQHcfDKwD7op4\nbKO7pwdfCnaRBPf3pTu5/OGFHCou49lvjOQb6l+vEU675+7ubwZ75JFjr0XcfRe4KrZliUhNV1JW\nwS/+sZqnF21leGpbHv7KUEJqc6wxYnFA7OvArIj7PcxsCXAQ+JG7v3WylcxsEjAJICUlJQZliEh1\n2XXgCFNnZJO9bT/fOK8H3784TW2ONUxU4W5mPyQ8EfaMYGgXkOLuBWY2DHjJzAa4+8ET13X3acA0\nCM+hGk0dIlJ93tmQz20zl3C0tJyHvzKULwzuFO+S5CTOONzN7GuEP2gd58Es2+5eDBQHy1lmthHo\nC2j2a5Fazt157I1N/Pbfa+mZ3ILHrh9K75DaHGuqMwp3M7sI+B7wGXcvihhPBva6e7mZ9QT6AJti\nUqmIxM3Bo6XcOXsZc1fv4QuDO/GbLw+mudoca7TKtELOBMYASWa2A/gx4e6YxsDc4FPxYy2PFwA/\nNbNSoAKY7O57q6h2EakGa3Yd5JbpWezYd4T//WJ/vj46Vd0wtUBlumWuPcnwE6d47hxgTrRFiUjN\n8OKSHdz1wgpaNWnIzEmjGJ7aLt4lSSXp7yoR+S/FZeX8/NU1/PXdrYzs0Y4/fmUIoZZqc6xNFO4i\n8jEf7j/ClBnZLN2+n0kX9OR7n+9HA7U51joKdxE57u31+XzruSWUlFXw6HVDuXiQ2hxrK4W7iFBR\n4Tz6xkbuey2HXskteGziMHolt4h3WRIFhbtIHXfgSCl3zl7Kf9bk8qWzO/PLKwepzTEB6H9QpA5b\n/eFBbpmRxc59R/jJpf254Vy1OSYKhbtIHfV81g5++OIK2jRryKybRzGsu9ocE4nCXaSOKS4r595X\nVvPse9sY1bMdf7x2KMktG8e7LIkxhbvIJ5jw+CIAZt18TpwriY2d+48wZXoWy3YcYPJnevE/n+ur\nNscEpXAXqSPeXJfH7c8toazceXziMD4/oGO8S5IqpHAXSXAVFc7D8zfw+/+so2+oJY9NHEaPpObx\nLkuqmMJdJIEdKCrl27OXMm9tLpend+b/XTmIZo30Y18X6H9ZJEGt3HmAW2ZksfvAUX562QAmjuqu\nNsc6ROEukoBmL97O/760knbNGzHr5nMYmtI23iVJNVO4iySQo6Xl3PvKKma+v53Rvdvz4DVDaN9C\nbY51kcJdJEFs31vElBnZrNh5gCljenHn5/pRv54Ow9RVlWpwNbMnzSzXzFZGjLUzs7lmtj64bRuM\nm5k9aGYbzGy5mQ2tquJFJGxBTi6XPvQ2WwoOM23iML53UZqCvY6r7NkLTwEXnTD2A+B1d+8DvB7c\nB7iY8NypfYBJwKPRlykiJ1NR4fzhP+u48akP6NiqCa/ceh6fU/+6UMnDMu7+ppmlnjB8GeG5VQGe\nBhYA3w/Gn3F3B941szZm1sndd8WiYBEJ219Uwh2zlrIgJ48rh3ThF1cMommj+vEuS2qIaI65d4gI\n7N1Ah2C5C7A94nk7grGPhbuZTSK8Z09KSkoUZYjUPSt3HmDy9Cz2HDzKzy8fyHUjU9TmKB8Tk4tK\nBHvp/inXmebuGe6ekZycHIsyROqEWR9s48pH36Giwvnb5HO5Xv3rchLR7LnvOXa4xcw6AbnB+E6g\nW8TzugZjIhKFo6Xl3PP3lcxevIPz+yTxwDVDaNe8UbzLkhoqmj33l4EbguUbgL9HjH816JoZBRzQ\n8XaR6GzfW8RVj73D7MU7uC2zN0/dOELBLp+oUnvuZjaT8IenSWa2A/gx8CtgtpndBGwFxgdP/ydw\nCbABKAJujHHNInXK/LW53DFrKe7OEzdkMO6sDqdfSeq8ynbLXHuKh8ad5LkOTI2mKBGB8grngdfX\n8+Dr6zmrUysev34YKe2bxbssqSV0hqpIDbTvcAm3z1rKm+vyuGpYV35++UCaNFSbo1Sewl2khlm+\nYz+3TM8mr7CYX145iGuGd1M3jHxqCneRGsLdmfn+dn7y8iqSWzbm+VvOYXDXNvEuS2ophbtIDXC0\ntJwfvbSS57N2cEHfZB6YkE5bdcNIFBTuInG2raCIydOzWL3rIN8a14fbx/XRRb8kagp3kTh6fc0e\nvj1rKWbGX742nLFpoXiXJAlC4S4SB+UVzv1z1/HQ/A0M6NyKx64fRrd2anOU2FG4i1SzvYdLuP25\nJby1Pp8JGd2497IBanOUmFO4i1Sjpdv3M2V6FvmHS/j1lwcxYbiuiCpVQ+EuUg3cnRnvbePeV1bR\noVUT5kw+l0FdW8e7LElgCneRKnakpJwfvrSCF7J3MqZfMn+YkE6bZmpzlKqlcBepQlvyDzN5ehY5\newr59oV9uS2zN/XU5ijVQOEuUkXmrt7Dd2YvpX69cJvjmH5qc5Tqo3AXibGy8gp+P3cdjyzYyKAu\nrXnkuqFqc5Rqp3AXiaH8Q8V8a+YS3tlYwLUjuvHjS9XmKPFxxuFuZv2AWRFDPYF7gDbAN4G8YPxu\nd//nGVcoUktkb9vH1BnZ7D1cwm+uGsz4jG6nX0mkipxxuLt7DpAOYGb1Cc+T+iLhmZfud/ffxaRC\nkRrO3fnru1v52aur6di6CXNuOZeBXdTmKPEVq8My44CN7r5V152WuqSopIwfvriSF5fsJDMtxP3j\n02ndrGG8yxKJWbhfA8yMuH+rmX0VWAzc6e77TlzBzCYBkwBSUnSWntQ+m/MPM/mvWazLLeTOz/Zl\n6li1OUrNUS/aFzCzRsCXgL8FQ48CvQgfstkF3Hey9dx9mrtnuHtGcnJytGWIVKt/r9rNl/74NrmF\nR3n6xhHcNq6Pgl1qlFjsuV8MZLv7HoBjtwBm9ifg1RhsQ6RGKCuv4Lev5fD4G5s4u2trHr5uKF3b\nqs1Rap5YhPu1RBySMbNO7r4ruHsFsDIG2xCJu7zCcJvjok0FXDcyhXsu7U/jBmpzlJopqnA3s+bA\nZ4GbI4Z/Y2bpgANbTnhMpFbK2rqXKTOy2V9Uyn1Xn82Xh3WNd0kinyiqcHf3w0D7E8YmRlWRSA3i\n7uw5WMyEx9+lS9umvDhlBP07t4p3WSKnpTNURU7iSEk572zMZ33uIfYVlXLhWSHuG59O66Zqc5Ta\nQeEuEti+t4j5ObnMW5vLoo0FFJdVUM+ga9umTJuYoW4YqVUU7lJnlZZXkLV1H/PXhgN9fe4hALq3\nb8a1I1LITAvxx3nrqWemYJdaR+EudUrBoWIW5OQxLyeXN9flUXi0jAb1jBE92jFheDfGpoXomdSc\nY2daPzx/Q5wrFjkzCndJaO7Oqg8PMi/YO1+2Yz/ukNSiMRcN6EhmWojz+iTRsomOpUtiUbhLwjlU\nXMbb6/OZvzaX+Tm55BYWA3B219bcPq4PmWkhBnZurUMtktAU7pIQNucfZt7aXOavzeW9zQWUljst\nGzfg/L5JjO0XYky/EMktG8e7TJFqo3CXWqmkrIL3N+8NB3pOLpvzDwPQK7k5Xzs3lbFpIYantqNh\n/agvnyRSKyncpdbIPXj0eKvi2+vzOVxSTqMG9RjVsz03nNOdzLQOpLTXdV5EQOEuNVhFhbNsx/5w\nq2JOLit3HgSgY6smfCm9C5lpIUb3bk+zRnobi5xIPxVSoxw4Uspb6/OYtzaXN3LyKDhcQj2DISlt\n+e7n+zG2X4izOrVEk8KIfDKFu8SVu7Mx7xCvrwkfbsnauo+yCqd104Z8pm8ymWkhPtM3mbbNG8W7\nVJFaReEu1e5oaTnvbio4frhl+94jAKR1bMk3L+hJZlqIId3a0EAfhoqcMYW7VIsP9x9h3tpcFuTk\nsnBDAUdKy2nSsB6jeyVx8wW9GJsWokubpvEuUyRhKNylSpRXOEu27Tt+Zuja3YVA+CJcV2d0ZWxa\niHN6tqdJQ012IVIVog53M9sCFALlQJm7Z5hZO2AWkEp4wo7xJ5skWxLL/qIS3lgXfBi6Lo/9RaXU\nr2dkdG/LXRenkZkWoneohT4MFakGsdpzH+vu+RH3fwC87u6/MrMfBPe/H6NtSQ3h7qzdXXj8zNDs\nbfuocGjfvBGZaSEy00Kc3ydZ10AXiYOqOixzGTAmWH4aWIDCPSEUlZTxzoYC5uWEA33XgaMADOjc\nilvH9mZsWojBXdtQX9dtEYmrWIS7A6+ZmQOPu/s0oEPEJNm7gQ4x2I7Eyfa9RcePnS/aVEBJWQXN\nGtXnvN5J3D6uD2PTQnRo1STeZYpIhFiE+3nuvtPMQsBcM1sb+aC7exD8H2Nmk4BJACkpKTEoQ2Kl\ntLyCxVv2HT/Vf0MwiUVq+2ZcNzI8icWIHu1o3EAfhorUVFGHu7vvDG5zzexFYASwx8w6ufsuM+sE\n5J5kvWnANICMjIz/Cn+pXvnBJBbz1+by5vrwJBYN64cnsbhmeDcy00L0TG4R7zJFpJKiCnczaw7U\nc/fCYPlzwE+Bl4EbgF8Ft3+PtlCJrYqKiEkscnJZHkxikdyyMRcPPDaJRTItGqtbVqQ2ivYntwPw\nYtDa1gB41t3/ZWYfALPN7CZgKzA+yu1IDIQnscgLLpObR15hMWYwuGsb7hjXl8y0EAM6t9IkFiIJ\nIKpwd/dNwNknGS8AxkXz2hIbm/IOHb/m+fub9x6fxOKCvsmMTQsxpl8ySS00iYVIotHf3AmmuKz8\no0ks1uaypaAIgN6hFtw4ugdj+4XISG2rSSxEEpzCPQHsOXg0fBGutbks3PDRJBbn9GzPjaN7kJkW\nols7TWIhUpco3Guh8shJLNbmsurD8CQWnVo34bIhXcjsF+JcTWIhUqfpp7+WOHCklDfXhVsVF6zL\nY28wicXQYBKLzLQQaR01iYWIhCncayh3Z33uoeNnhmZt3Ud5hdOm2UeTWFzQR5NYiMjJKdxrkKOl\n5SzaWHC8u2XHvo8msbg5mMQiXZNYiEglKNzjbOf+I8ePnb+zMZ+jpRU0bVif0b3bc8uYXoztF6Kz\nJrEQkU9J4V7NysorWLJ9//FWxWOTWHRr15QJGd0YmxZilCaxEJEoKdyrwb7DH5/E4sCRUhrUMzJS\n23L3JeFJLHolaxILEYkdhXsVcHfW7Co8flXFJRGTWFx4VofwJBZ9k2jVRJNYiEjVULjHSFFJGQs3\nFByfBPrYJBaDurTm1sw+ZKaFGNylta7bIiLVotaH+4THFwEw6+Zzqn3b2wqKmLd2D/Ny8ng3mMSi\neaP6nN8nmW9fGL5uS0iTWIhIHNT6cK9OpeUVfLBl7/Hulo15hwHokdSciaO6k5kWYnhqOxo1UKui\niMSXwv008gqLWZAT7jt/a10+hcXhSSxG9mjPV0aGA71HUvN4lyki8jEK9xNUVDgrPzxwvFVx2Y4D\nAIRaNuaSQZ0YmxbivD5JmsRCRGo0JRRQeLSUt9fnhz8MXffRJBZnd23Ddz770SQWalUUkdrijMPd\nzLoBzxCejcmBae7+gJn9BPgmkBc89W53/2e0hcaSu7Mp//DxY+cfbAkmsWgSnsQis1/4w9D2msRC\nRGqpaPbcy4A73T3bzFoCWWY2N3jsfnf/XfTlxU5xWTnvbdp7/LotW4NJLPqEWvD10T0YmxZiWHdN\nYiEiieGMw93ddwG7guVCM1sDdIlVYbGw+8DR4ycSLdyQT1FJOY0b1OOcXu256bzwrESaxEJEElFM\njrmbWSowBHgPGA3camZfBRYT3rvfd5J1JgGTAFJSUmJRBuUVztLtH01isXpXeBKLzq2bcMWQLmSm\nhTi3VxJNG+m6LVI58Th/QiQWzN2jewGzFsAbwC/c/QUz6wDkEz4O/zOgk7t//ZNeIyMjwxcvXnxG\n2//yIws5cKSUQV3b8EbEJBbDurdlbFqIzLQQ/TpoEgsRSTxmluXuGSd7LKo9dzNrCMwBZrj7CwDu\nvifi8T8Br0azjU/y/ua9ZG3bD0D+4RLG9E1mbFqIz/RNpk0zTWIhInVXNN0yBjwBrHH330eMdwqO\nxwNcAayMrsRTG9ilFZ1bN6FNs4a8ctv51Nd1W0REgOj23EcDE4EVZrY0GLsbuNbM0gkfltkC3BxV\nhZ+gWaMGxz8QVbCLiHwkmm6Zt4GTJWqN6mkXEamL1NQtIpKAFO4iIglI4S4ikoAU7iIiCUjhLiKS\ngBTuIiIJSOEuIpKAFO4iIgmo1s/EpKv2iYj8N+25i4gkIIW7iEgCUriLiCQghbuISAJSuIuIJCCF\nu4hIAlK4i4gkIIW7iEgCUriLiCQgc/d414CZ5QFbo3iJJCA/RuWInEjvL6lK0by/urt78skeqBHh\nHi0zW+zuGfGuQxKT3l9Slarq/aXDMiIiCUjhLiKSgBIl3KfFuwBJaHp/SVWqkvdXQhxzFxGRj0uU\nPXcREYmgcBcRSUAJE+5mtsXMkuJdh4iZXW5m/eNdh9Q8ZlZuZkvNbKWZvWJmbYLxVDNbGctt1Yhw\nt7AaUYtINMysAXA5oHCXkzni7unuPhDYC0ytqg3FLVCD31Q5ZvYMsBJ4wswWm9kqM7s34nlbzOxe\nM8s2sxVmlhaMtzez14Ln/xmwiHW+E/xmXGlmd0Rsb62ZPWVm68xshpldaGYLzWy9mY2o5n8CqUZm\ndr2ZvR/sNT1uZt2D//ckM6tnZm+Z2eci3iczzGyNmT1vZs2C1xhmZm+YWZaZ/dvMOgXjC8zsD2a2\nGPg+8CXgt8G2esXx25aabRHQpcpe3d3j8gWkAhXAqOB+u+C2PrAAGBzc3wLcFixPAf4cLD8I3BMs\nfwFwwqfxDgNWAM2BFsAqYEiwvTJgEOFfalnAk4R/KVwGvBSvfwt9Vfl77SzgFaBhcP8R4KvAN4C/\nAd8FHo94XzowOrj/JPA/QEPgHSA5GJ8APBksLwAeidjeU8BV8f6+9VXzvoBDwW394L13UXA/FVgZ\ny201+FS/CWJvq7u/GyyPN7NJQAOgE+E/a5cHj70Q3GYBVwbLFxxbdvd/mNm+YPw84EV3PwxgZi8A\n5wMvA5vdfUUwvgp43d3dzFYQ/seVxDSO8C/9D8wMoCmQ6+4/MbOrgclAesTzt7v7wmB5OvAt4F/A\nQGBu8Br1gV0R68yq0u9AEgNummkAAAGQSURBVEVTM1tKeI99DTC3qjYU73A/FsA9CO8dDXf3fWb2\nFNAk4nnFwW050dVcHLFcEXG/IsrXlZrNgKfd/a6PDYYPt3QN7rYACoPlE0/+8OA1Vrn7OafYxuEY\n1SqJ7Yi7pwfvvX8TPub+YFVsqKZ8iNmK8A/HATPrAFxciXXeBL4CYGYXA22D8beAy82smZk1B64I\nxqTueh24ysxCAGbWzsy6A78GZgD3AH+KeH6KmR0L8a8AbwM5QPKxcTNraGYDTrG9QqBl7L8NSRTu\nXkT4L8I7gw/hY65GhLu7LwOWAGuBZ4GFn7wGAPcCFwSHV64EtgWvlU34mOf7wHuEj9EvqYKypZZw\n99XAj4DXzGw54T+FU4HhwK/dfQZQYmY3BqvkAFPNbA3hnYZH3b0EuAr4tZktA5YC555ik88B3zWz\nJfpAVU4lyKXlwLXBUD8z2xHxdXU0r6/LD4hEMLNU4FUPt6qJ1Fo1Ys9dRERiS3vuIiIJSHvuIiIJ\nSOEuIpKAFO4iIglI4S4ikoAU7iIiCej/A4Rls1Nvo5V+AAAAAElFTkSuQmCC\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"tags":[]}}]}]}