Bipedal walker using Soft Actor-Critic

[nishantkr18]

Maybe the recent changes in mlpack repo need to be updated for this to work?

[review-notebook-app]

Check out this pull request on 

Review Jupyter notebook visual diffs & provide feedback on notebooks.

---

Powered by ReviewNB

[zoq]

Maybe the recent changes in mlpack repo need to be updated for this to work?

Yes, let me rebuild the image.

[zoq]

Okay, new image build.

[zoq]

Also, we still have to disable the go kernel, but https://lab.mlpack.org/v2/gh/zoq/examples/go-disable?urlpath=lab%2Ftree%2Fforest_covertype_prediction_with_random_forests%2Fcovertype-rf-cpp.ipynb should work fine.

[zoq]

Pretty sure this will take some time to train, so what about we provide a pre-trained model?

[nishantkr18]

Pretty sure this will take some time to train, so what about we provide a pre-trained model?

I was just about to say that! There is an appreciable amount of difference in training speed between my local machine and the binder noteboook.

Btw, when I run the a binder notebook instance, you are able to notice right? I'm a bit curious, where does the computation exactly take place? I'm not sure, but I remember you saying that you've setup a local server for binder, because the ram and computation capacity provided for free was not enough for mlpack. is that so?

[zoq]

Btw, when I run the a binder notebook instance, you are able to notice right? I'm a bit curious, where does the computation exactly take place? I'm not sure, but I remember you saying that you've setup a local server for binder, because the ram and computation capacity provided for free was not enough for mlpack. is that so?

Cling doesn't optimize the code, so the slow down is expected, most of the time I have a tmux session open that also shows the CPU usage. Binder users are guaranteed at least 1GB of RAM, with a maximum of 2GB and a shared CPU. Which works in most cases, some examples exceed the memory constrains, but since the CPU is shared the runtime is somewhat slow.

Right now the mlpack binder runs on a machine with an Intel Core i7-7700 CPU with 64GB RAM, we can use the machine at full so if the computation takes hours/days/weeks no problem.

At some point the Jupyter session will shutdown, should be after 6 hours just to make sure we don't end up with some process that got forgotten and just idles around.

[nishantkr18]

should a simple data::Load("./50qNetwork.xml", "episode", qNetwork); work for loading a model in Jupyter?

[zoq]

should a simple data::Load("./50qNetwork.xml", "episode", qNetwork); work for loading a model in Jupyter?

Yes, that should work, I can upload the 50qNetwork.xml file.

[zoq]

Output from the test env: https://gym.kurg.org/0a440bd6a0cb4/output.webm

Average return in last 1 consecutive episodes: -118.891 steps: 156 Episode return: -118.891
Average return in last 2 consecutive episodes: -114.692 steps: 75 Episode return: -110.494
Average return in last 3 consecutive episodes: -114.921 steps: 74 Episode return: -115.378
Average return in last 4 consecutive episodes: -115.929 steps: 118 Episode return: -118.955
Average return in last 5 consecutive episodes: -111.347 steps: 95 Episode return: -93.0199
Average return in last 6 consecutive episodes: -108.278 steps: 101 Episode return: -92.9317
Average return in last 7 consecutive episodes: -108.714 steps: 71 Episode return: -111.327
Average return in last 8 consecutive episodes: -99.9415 steps: 1600 Episode return: -38.5357
Average return in last 9 consecutive episodes: -103.57 steps: 350 Episode return: -132.596
Average return in last 10 consecutive episodes: -110.352 steps: 1600 Episode return: -171.389
Average return in last 11 consecutive episodes: -112.117 steps: 46 Episode return: -129.771
Average return in last 12 consecutive episodes: -114.33 steps: 1600 Episode return: -138.668
Average return in last 13 consecutive episodes: -123.894 steps: 1217 Episode return: -238.664
Average return in last 14 consecutive episodes: -132.247 steps: 1317 Episode return: -240.833
Average return in last 15 consecutive episodes: -132.872 steps: 128 Episode return: -141.633
Average return in last 16 consecutive episodes: -131.678 steps: 38 Episode return: -113.754
Average return in last 17 consecutive episodes: -131.038 steps: 42 Episode return: -120.805
Average return in last 18 consecutive episodes: -130.617 steps: 44 Episode return: -123.457
Average return in last 19 consecutive episodes: -130.034 steps: 40 Episode return: -119.548
Average return in last 20 consecutive episodes: -129.704 steps: 44 Episode return: -123.422
Average return in last 21 consecutive episodes: -129.402 steps: 44 Episode return: -123.373
Average return in last 22 consecutive episodes: -129.012 steps: 42 Episode return: -120.809
Average return in last 23 consecutive episodes: -128.767 steps: 44 Episode return: -123.382
Average return in last 24 consecutive episodes: -128.52 steps: 44 Episode return: -122.835
Average return in last 25 consecutive episodes: -128.123 steps: 39 Episode return: -118.601
Average return in last 26 consecutive episodes: -127.929 steps: 44 Episode return: -123.077
Average return in last 27 consecutive episodes: -127.759 steps: 44 Episode return: -123.338
Average return in last 28 consecutive episodes: -127.603 steps: 44 Episode return: -123.383
Average return in last 29 consecutive episodes: -127.46 steps: 44 Episode return: -123.465
Average return in last 30 consecutive episodes: -127.321 steps: 44 Episode return: -123.302
Average return in last 31 consecutive episodes: -127.165 steps: 44 Episode return: -122.469
Average return in last 32 consecutive episodes: -127.59 steps: 362 Episode return: -140.778
Average return in last 33 consecutive episodes: -128.174 steps: 318 Episode return: -146.843
Average return in last 34 consecutive episodes: -127.531 steps: 48 Episode return: -106.339
Average return in last 35 consecutive episodes: -127.973 steps: 445 Episode return: -142.979
Average return in last 36 consecutive episodes: -127.275 steps: 51 Episode return: -102.868
Average return in last 37 consecutive episodes: -126.581 steps: 51 Episode return: -101.575
Average return in last 38 consecutive episodes: -126.892 steps: 119 Episode return: -138.409
Average return in last 39 consecutive episodes: -126.279 steps: 47 Episode return: -102.995
Average return in last 40 consecutive episodes: -125.754 steps: 58 Episode return: -105.27
Average return in last 41 consecutive episodes: -125.462 steps: 67 Episode return: -113.79
Average return in last 42 consecutive episodes: -125.419 steps: 68 Episode return: -123.654
Average return in last 43 consecutive episodes: -124.852 steps: 53 Episode return: -101.035
Average return in last 44 consecutive episodes: -124.421 steps: 73 Episode return: -105.865
Average return in last 45 consecutive episodes: -123.922 steps: 66 Episode return: -101.998
Average return in last 46 consecutive episodes: -123.441 steps: 66 Episode return: -101.776
Average return in last 47 consecutive episodes: -122.976 steps: 59 Episode return: -101.583
Average return in last 48 consecutive episodes: -122.503 steps: 59 Episode return: -100.278
Average return in last 49 consecutive episodes: -122.689 steps: 106 Episode return: -131.621
Average return in last 50 consecutive episodes: -122.329 steps: 67 Episode return: -104.709
Average return in last 50 consecutive episodes: -122.057 steps: 69 Episode return: -105.277
Average return in last 50 consecutive episodes: -122.023 steps: 57 Episode return: -108.771
Average return in last 50 consecutive episodes: -121.75 steps: 83 Episode return: -101.763
Average return in last 50 consecutive episodes: -121.38 steps: 74 Episode return: -100.452
Average return in last 50 consecutive episodes: -121.543 steps: 74 Episode return: -101.131
Average return in last 50 consecutive episodes: -121.752 steps: 100 Episode return: -103.414
Average return in last 50 consecutive episodes: -122.018 steps: 64 Episode return: -124.626
Average return in last 50 consecutive episodes: -123.213 steps: 99 Episode return: -98.2776
Average return in last 50 consecutive episodes: -123.009 steps: 58 Episode return: -122.383
Average return in last 50 consecutive episodes: -121.983 steps: 200 Episode return: -120.098
Average return in last 50 consecutive episodes: -121.91 steps: 106 Episode return: -126.121
Average return in last 50 consecutive episodes: -121.228 steps: 121 Episode return: -104.592
Average return in last 50 consecutive episodes: -118.577 steps: 99 Episode return: -106.08
Average return in last 50 consecutive episodes: -116.224 steps: 67 Episode return: -123.188
Average return in last 50 consecutive episodes: -116.132 steps: 286 Episode return: -137.065
Average return in last 50 consecutive episodes: -116.173 steps: 72 Episode return: -115.798
Average return in last 50 consecutive episodes: -116.044 steps: 73 Episode return: -114.356
Average return in last 50 consecutive episodes: -116.235 steps: 112 Episode return: -133.005
Average return in last 50 consecutive episodes: -116.446 steps: 101 Episode return: -130.069
Average return in last 50 consecutive episodes: -116.248 steps: 64 Episode return: -113.556
Average return in last 50 consecutive episodes: -116.011 steps: 49 Episode return: -111.482
Average return in last 50 consecutive episodes: -115.804 steps: 54 Episode return: -110.462
Average return in last 50 consecutive episodes: -115.63 steps: 49 Episode return: -114.68
Average return in last 50 consecutive episodes: -115.42 steps: 56 Episode return: -112.364
Average return in last 50 consecutive episodes: -115.289 steps: 64 Episode return: -112.058
Average return in last 50 consecutive episodes: -115.034 steps: 150 Episode return: -110.31
Average return in last 50 consecutive episodes: -114.829 steps: 60 Episode return: -113.073
Average return in last 50 consecutive episodes: -114.41 steps: 101 Episode return: -102.466
Average return in last 50 consecutive episodes: -114.101 steps: 47 Episode return: -108.016
Average return in last 50 consecutive episodes: -113.794 steps: 46 Episode return: -107.932
Average return in last 50 consecutive episodes: -113.491 steps: 95 Episode return: -107.297
Average return in last 50 consecutive episodes: -112.866 steps: 83 Episode return: -109.571
Average return in last 50 consecutive episodes: -112.061 steps: 94 Episode return: -106.579
Average return in last 50 consecutive episodes: -112.204 steps: 67 Episode return: -113.468
Average return in last 50 consecutive episodes: -111.803 steps: 88 Episode return: -122.957
Average return in last 50 consecutive episodes: -112.436 steps: 168 Episode return: -134.5
Average return in last 50 consecutive episodes: -112.925 steps: 1600 Episode return: -126.034
Average return in last 50 consecutive episodes: -113.476 steps: 740 Episode return: -165.943
Average return in last 50 consecutive episodes: -113.726 steps: 59 Episode return: -115.5
Average return in last 50 consecutive episodes: -113.979 steps: 57 Episode return: -117.913
Average return in last 50 consecutive episodes: -113.925 steps: 104 Episode return: -111.114
Average return in last 50 consecutive episodes: -113.653 steps: 108 Episode return: -110.049
Average return in last 50 consecutive episodes: -113.854 steps: 143 Episode return: -111.069
Average return in last 50 consecutive episodes: -114.03 steps: 197 Episode return: -114.673
Average return in last 50 consecutive episodes: -114.502 steps: 97 Episode return: -125.579
Average return in last 50 consecutive episodes: -114.717 steps: 93 Episode return: -112.536
Average return in last 50 consecutive episodes: -114.934 steps: 73 Episode return: -112.429
Average return in last 50 consecutive episodes: -115.18 steps: 218 Episode return: -112.595
Average return in last 50 consecutive episodes: -114.811 steps: 80 Episode return: -113.187
Average return in last 50 consecutive episodes: -115.025 steps: 76 Episode return: -115.397
Average return in last 50 consecutive episodes: -115.208 steps: 84 Episode return: -114.412
Average return in last 50 consecutive episodes: -115.278 steps: 157 Episode return: -112.273
Average return in last 50 consecutive episodes: -115.58 steps: 84 Episode return: -116.877
Average return in last 50 consecutive episodes: -115.928 steps: 43 Episode return: -117.84
Average return in last 50 consecutive episodes: -116.308 steps: 57 Episode return: -120.133
Average return in last 50 consecutive episodes: -116.646 steps: 53 Episode return: -120.336
Average return in last 50 consecutive episodes: -116.58 steps: 74 Episode return: -121.325
Average return in last 50 consecutive episodes: -117.02 steps: 54 Episode return: -120.278
Average return in last 50 consecutive episodes: -116.992 steps: 65 Episode return: -120.965
Average return in last 50 consecutive episodes: -117.088 steps: 209 Episode return: -124.912
Average return in last 50 consecutive episodes: -116.835 steps: 91 Episode return: -113.443
Average return in last 50 consecutive episodes: -117.004 steps: 80 Episode return: -113.058
Average return in last 50 consecutive episodes: -117.275 steps: 127 Episode return: -119.615
Average return in last 50 consecutive episodes: -117.195 steps: 131 Episode return: -119.207
Average return in last 50 consecutive episodes: -116.886 steps: 190 Episode return: -121.626
Average return in last 50 consecutive episodes: -116.879 steps: 71 Episode return: -115.428
Average return in last 50 consecutive episodes: -116.914 steps: 77 Episode return: -116.127
Average return in last 50 consecutive episodes: -116.592 steps: 79 Episode return: -116.861
Average return in last 50 consecutive episodes: -116.389 steps: 91 Episode return: -119.96
Average return in last 50 consecutive episodes: -116.475 steps: 92 Episode return: -117.863
Average return in last 50 consecutive episodes: -116.57 steps: 79 Episode return: -116.222
Average return in last 50 consecutive episodes: -116.729 steps: 114 Episode return: -118.387
Average return in last 50 consecutive episodes: -116.621 steps: 64 Episode return: -109.308
Average return in last 50 consecutive episodes: -116.588 steps: 71 Episode return: -110.703
Average return in last 50 consecutive episodes: -116.513 steps: 68 Episode return: -108.283
Average return in last 50 consecutive episodes: -119.048 steps: 1538 Episode return: -237.061
Average return in last 50 consecutive episodes: -119.379 steps: 157 Episode return: -129.619
Average return in last 50 consecutive episodes: -120.207 steps: 141 Episode return: -143.899
Average return in last 50 consecutive episodes: -120.961 steps: 166 Episode return: -145.719
Average return in last 50 consecutive episodes: -121.979 steps: 519 Episode return: -158.797
Average return in last 50 consecutive episodes: -124.283 steps: 1158 Episode return: -222.508
Average return in last 50 consecutive episodes: -124.72 steps: 93 Episode return: -131.435
Average return in last 50 consecutive episodes: -124.931 steps: 109 Episode return: -117.105
Average return in last 50 consecutive episodes: -125.487 steps: 335 Episode return: -141.269
Average return in last 50 consecutive episodes: -125.614 steps: 101 Episode return: -129.304
Average return in last 50 consecutive episodes: -125.499 steps: 1600 Episode return: -128.758
Average return in last 50 consecutive episodes: -125.265 steps: 92 Episode return: -114.326
Average return in last 50 consecutive episodes: -123.96 steps: 130 Episode return: -100.695
Average return in last 50 consecutive episodes: -126.011 steps: 1359 Episode return: -218.085
Average return in last 50 consecutive episodes: -125.911 steps: 71 Episode return: -112.898
Average return in last 50 consecutive episodes: -126.045 steps: 103 Episode return: -117.837
Average return in last 50 consecutive episodes: -126.164 steps: 81 Episode return: -115.96
Average return in last 50 consecutive episodes: -126.242 steps: 102 Episode return: -114.989
Average return in last 50 consecutive episodes: -126.135 steps: 74 Episode return: -109.311
Average return in last 50 consecutive episodes: -126.183 steps: 248 Episode return: -128.007
Average return in last 50 consecutive episodes: -126.242 steps: 95 Episode return: -115.441
Average return in last 50 consecutive episodes: -126.827 steps: 547 Episode return: -141.692
Average return in last 50 consecutive episodes: -126.771 steps: 97 Episode return: -109.8
Average return in last 50 consecutive episodes: -126.689 steps: 61 Episode return: -109.092
Average return in last 50 consecutive episodes: -126.498 steps: 73 Episode return: -105.845
Average return in last 50 consecutive episodes: -126.841 steps: 179 Episode return: -131.563
Average return in last 50 consecutive episodes: -126.753 steps: 83 Episode return: -107.888
Average return in last 50 consecutive episodes: -126.634 steps: 56 Episode return: -110.891
Average return in last 50 consecutive episodes: -126.482 steps: 76 Episode return: -110.244
Average return in last 50 consecutive episodes: -127.141 steps: 1600 Episode return: -153.093
Average return in last 50 consecutive episodes: -127.054 steps: 60 Episode return: -115.978
Average return in last 50 consecutive episodes: -126.851 steps: 1600 Episode return: -111.19
Average return in last 50 consecutive episodes: -126.981 steps: 71 Episode return: -126.787
Average return in last 50 consecutive episodes: -126.805 steps: 109 Episode return: -112.159
Average return in last 50 consecutive episodes: -126.797 steps: 63 Episode return: -124.488
Average return in last 50 consecutive episodes: -126.899 steps: 74 Episode return: -118.559
Average return in last 50 consecutive episodes: -126.743 steps: 1600 Episode return: -105.254
Average return in last 50 consecutive episodes: -126.428 steps: 1600 Episode return: -103.854
Average return in last 50 consecutive episodes: -126.246 steps: 1600 Episode return: -110.117
Average return in last 50 consecutive episodes: -126.37 steps: 93 Episode return: -127.845
Average return in last 50 consecutive episodes: -126.614 steps: 67 Episode return: -127.642
Average return in last 50 consecutive episodes: -126.498 steps: 35 Episode return: -110.282
Average return in last 50 consecutive episodes: -126.649 steps: 58 Episode return: -124.453
Average return in last 50 consecutive episodes: -126.423 steps: 34 Episode return: -108.637
Average return in last 50 consecutive episodes: -126.292 steps: 41 Episode return: -111.334
Average return in last 50 consecutive episodes: -126.118 steps: 33 Episode return: -107.488
Average return in last 50 consecutive episodes: -125.98 steps: 1600 Episode return: -111.489
Average return in last 50 consecutive episodes: -126.306 steps: 65 Episode return: -125.634
Average return in last 50 consecutive episodes: -126.43 steps: 1600 Episode return: -116.902
Average return in last 50 consecutive episodes: -126.396 steps: 78 Episode return: -106.556
Average return in last 50 consecutive episodes: -126.31 steps: 1411 Episode return: -232.767
Average return in last 50 consecutive episodes: -126.276 steps: 1600 Episode return: -127.917
Average return in last 50 consecutive episodes: -125.706 steps: 1600 Episode return: -115.432
Average return in last 50 consecutive episodes: -125.285 steps: 1600 Episode return: -124.662
Average return in last 50 consecutive episodes: -124.59 steps: 1600 Episode return: -124.057
Average return in last 50 consecutive episodes: -123.054 steps: 1600 Episode return: -145.71
Average return in last 50 consecutive episodes: -124.648 steps: 1216 Episode return: -211.096
Average return in last 50 consecutive episodes: -124.363 steps: 72 Episode return: -102.857
Average return in last 50 consecutive episodes: -124.223 steps: 114 Episode return: -134.296
Average return in last 50 consecutive episodes: -124.003 steps: 1600 Episode return: -118.285
Average return in last 50 consecutive episodes: -123.475 steps: 443 Episode return: -102.371
Average return in last 50 consecutive episodes: -123.574 steps: 171 Episode return: -119.282
Average return in last 50 consecutive episodes: -124.707 steps: 1467 Episode return: -157.341
Average return in last 50 consecutive episodes: -122.921 steps: 116 Episode return: -128.766
Average return in last 50 consecutive episodes: -122.894 steps: 80 Episode return: -111.562
Average return in last 50 consecutive episodes: -122.783 steps: 116 Episode return: -112.293
Average return in last 50 consecutive episodes: -122.754 steps: 101 Episode return: -114.5
Average return in last 50 consecutive episodes: -122.601 steps: 48 Episode return: -107.347
Average return in last 50 consecutive episodes: -122.858 steps: 1600 Episode return: -122.14
Average return in last 50 consecutive episodes: -122.792 steps: 404 Episode return: -124.707
Average return in last 50 consecutive episodes: -123.892 steps: 572 Episode return: -170.47
Average return in last 50 consecutive episodes: -123.752 steps: 267 Episode return: -134.695
Average return in last 50 consecutive episodes: -125.342 steps: 765 Episode return: -189.274
Average return in last 50 consecutive episodes: -126.487 steps: 985 Episode return: -166.362
Average return in last 50 consecutive episodes: -127.01 steps: 126 Episode return: -131.99
Average return in last 50 consecutive episodes: -127.841 steps: 718 Episode return: -173.124
Average return in last 50 consecutive episodes: -127.868 steps: 84 Episode return: -109.232
Average return in last 50 consecutive episodes: -129.721 steps: 974 Episode return: -203.542
Average return in last 50 consecutive episodes: -129.607 steps: 74 Episode return: -104.546
Average return in last 50 consecutive episodes: -128.579 steps: 80 Episode return: -101.657
Average return in last 50 consecutive episodes: -128.371 steps: 64 Episode return: -105.622
Average return in last 50 consecutive episodes: -128.148 steps: 87 Episode return: -100.02
Average return in last 50 consecutive episodes: -127.677 steps: 72 Episode return: -103.242
Average return in last 50 consecutive episodes: -127.557 steps: 110 Episode return: -106.153
Average return in last 50 consecutive episodes: -129.176 steps: 967 Episode return: -205.455
Average return in last 50 consecutive episodes: -129.716 steps: 253 Episode return: -145.56
Average return in last 50 consecutive episodes: -129.744 steps: 186 Episode return: -106.609
Average return in last 50 consecutive episodes: -130.983 steps: 1600 Episode return: -165.851
Average return in last 50 consecutive episodes: -130.775 steps: 80 Episode return: -99.7106
Average return in last 50 consecutive episodes: -131.085 steps: 1600 Episode return: -143.344
Average return in last 50 consecutive episodes: -132.66 steps: 865 Episode return: -206.383
Average return in last 50 consecutive episodes: -134.561 steps: 1074 Episode return: -205.339
Average return in last 50 consecutive episodes: -134.671 steps: 188 Episode return: -129.959
Average return in last 50 consecutive episodes: -134.907 steps: 257 Episode return: -120.424
Average return in last 50 consecutive episodes: -135.39 steps: 432 Episode return: -135.475
Average return in last 50 consecutive episodes: -135.581 steps: 185 Episode return: -117.02
Average return in last 50 consecutive episodes: -137.605 steps: 1300 Episode return: -212.713
Average return in last 50 consecutive episodes: -138.572 steps: 1600 Episode return: -173.968
Average return in last 50 consecutive episodes: -138.383 steps: 1600 Episode return: -107.459
Average return in last 50 consecutive episodes: -138.542 steps: 1600 Episode return: -114.503
Average return in last 50 consecutive episodes: -137.846 steps: 917 Episode return: -197.964
Average return in last 50 consecutive episodes: -137.967 steps: 164 Episode return: -133.961
Average return in last 50 consecutive episodes: -138.571 steps: 1600 Episode return: -145.636
Average return in last 50 consecutive episodes: -138.488 steps: 1600 Episode return: -120.535
Average return in last 50 consecutive episodes: -138.075 steps: 151 Episode return: -103.414
Average return in last 50 consecutive episodes: -137.21 steps: 75 Episode return: -102.45
Average return in last 50 consecutive episodes: -135.074 steps: 41 Episode return: -104.269
Average return in last 50 consecutive episodes: -135.078 steps: 65 Episode return: -103.086
Average return in last 50 consecutive episodes: -135.599 steps: 1600 Episode return: -160.331
Average return in last 50 consecutive episodes: -136.338 steps: 1600 Episode return: -155.26
Average return in last 50 consecutive episodes: -137.22 steps: 1600 Episode return: -146.471
Average return in last 50 consecutive episodes: -137.409 steps: 119 Episode return: -128.715
Average return in last 50 consecutive episodes: -137.526 steps: 434 Episode return: -163.211
Average return in last 50 consecutive episodes: -138.666 steps: 817 Episode return: -185.745
Average return in last 50 consecutive episodes: -138.881 steps: 1600 Episode return: -122.304
Average return in last 50 consecutive episodes: -139.229 steps: 1600 Episode return: -129.722
Average return in last 50 consecutive episodes: -139.404 steps: 1600 Episode return: -123.203
Average return in last 50 consecutive episodes: -140.176 steps: 922 Episode return: -145.958
Average return in last 50 consecutive episodes: -140.435 steps: 1600 Episode return: -135.101
Average return in last 50 consecutive episodes: -141.08 steps: 349 Episode return: -156.958
Average return in last 50 consecutive episodes: -141.225 steps: 1600 Episode return: -177.74
Average return in last 50 consecutive episodes: -140.819 steps: 132 Episode return: -114.369
Average return in last 50 consecutive episodes: -139.388 steps: 95 Episode return: -117.722
Average return in last 50 consecutive episodes: -138.393 steps: 1600 Episode return: -116.632
Average return in last 50 consecutive episodes: -139.311 steps: 750 Episode return: -177.891
Average return in last 50 consecutive episodes: -138.806 steps: 653 Episode return: -147.861
Average return in last 50 consecutive episodes: -139.245 steps: 1600 Episode return: -131.2
Average return in last 50 consecutive episodes: -138.063 steps: 1600 Episode return: -144.448
Average return in last 50 consecutive episodes: -140.827 steps: 1205 Episode return: -242.724
Average return in last 50 consecutive episodes: -141.83 steps: 1600 Episode return: -151.82
Average return in last 50 consecutive episodes: -144.274 steps: 1453 Episode return: -227.801
Average return in last 50 consecutive episodes: -144.55 steps: 1600 Episode return: -113.81
Average return in last 50 consecutive episodes: -144.963 steps: 1600 Episode return: -123.89
Average return in last 50 consecutive episodes: -147.332 steps: 1412 Episode return: -224.623
Average return in last 50 consecutive episodes: -145.818 steps: 426 Episode return: -129.774
Average return in last 50 consecutive episodes: -145.324 steps: 173 Episode return: -120.848
Average return in last 50 consecutive episodes: -145.386 steps: 171 Episode return: -109.702
Average return in last 50 consecutive episodes: -144.208 steps: 115 Episode return: -106.941
Average return in last 50 consecutive episodes: -144.268 steps: 83 Episode return: -102.746
Average return in last 50 consecutive episodes: -143.527 steps: 115 Episode return: -106.287
Average return in last 50 consecutive episodes: -141.499 steps: 42 Episode return: -104.987
Average return in last 50 consecutive episodes: -139.235 steps: 110 Episode return: -92.1151
Average return in last 50 consecutive episodes: -139.965 steps: 1028 Episode return: -166.441
Average return in last 50 consecutive episodes: -140.339 steps: 256 Episode return: -139.144
Average return in last 50 consecutive episodes: -139.956 steps: 119 Episode return: -116.336
Average return in last 50 consecutive episodes: -139.918 steps: 82 Episode return: -115.131
Average return in last 50 consecutive episodes: -138.005 steps: 148 Episode return: -117.061
Average return in last 50 consecutive episodes: -137.875 steps: 412 Episode return: -167.453
Average return in last 50 consecutive episodes: -138.438 steps: 338 Episode return: -135.608
Average return in last 50 consecutive episodes: -138.267 steps: 60 Episode return: -105.94
Average return in last 50 consecutive episodes: -136.598 steps: 118 Episode return: -114.526
Average return in last 50 consecutive episodes: -136.142 steps: 93 Episode return: -111.14
Average return in last 50 consecutive episodes: -135.508 steps: 63 Episode return: -113.959
Average return in last 50 consecutive episodes: -135.349 steps: 75 Episode return: -112.596
Average return in last 50 consecutive episodes: -135.473 steps: 57 Episode return: -109.613
Average return in last 50 consecutive episodes: -135.667 steps: 82 Episode return: -112.114
Average return in last 50 consecutive episodes: -135.918 steps: 87 Episode return: -116.833
Average return in last 50 consecutive episodes: -136.172 steps: 90 Episode return: -115.808
Average return in last 50 consecutive episodes: -135.313 steps: 101 Episode return: -117.369
Average return in last 50 consecutive episodes: -134.467 steps: 81 Episode return: -112.97
Average return in last 50 consecutive episodes: -134.518 steps: 170 Episode return: -148.994
Average return in last 50 consecutive episodes: -133.546 steps: 1600 Episode return: -80.1327
Average return in last 50 consecutive episodes: -132.616 steps: 102 Episode return: -116.727
Average return in last 50 consecutive episodes: -132.672 steps: 1153 Episode return: -188.506
Average return in last 50 consecutive episodes: -132.7 steps: 1600 Episode return: -123.75
Average return in last 50 consecutive episodes: -132.446 steps: 129 Episode return: -116.993
Average return in last 50 consecutive episodes: -132.763 steps: 1600 Episode return: -139.042
Average return in last 50 consecutive episodes: -132.542 steps: 326 Episode return: -134.904
Average return in last 50 consecutive episodes: -132.16 steps: 49 Episode return: -116.03
Average return in last 50 consecutive episodes: -131.507 steps: 395 Episode return: -124.302
Average return in last 50 consecutive episodes: -130.394 steps: 109 Episode return: -122.092
Average return in last 50 consecutive episodes: -130.726 steps: 109 Episode return: -130.955
Average return in last 50 consecutive episodes: -130.421 steps: 1600 Episode return: -102.463
Average return in last 50 consecutive episodes: -129.813 steps: 1600 Episode return: -86.2441
Average return in last 50 consecutive episodes: -129.139 steps: 326 Episode return: -144.213
Average return in last 50 consecutive episodes: -128.541 steps: 42 Episode return: -117.931
Average return in last 50 consecutive episodes: -129.993 steps: 1365 Episode return: -203.793
Average return in last 50 consecutive episodes: -130.151 steps: 309 Episode return: -152.377
Average return in last 50 consecutive episodes: -128.131 steps: 180 Episode return: -141.73
Average return in last 50 consecutive episodes: -127.628 steps: 148 Episode return: -126.638
Average return in last 50 consecutive episodes: -125.91 steps: 397 Episode return: -141.911
Average return in last 50 consecutive episodes: -125.963 steps: 42 Episode return: -116.487
Average return in last 50 consecutive episodes: -125.822 steps: 41 Episode return: -116.821
Average return in last 50 consecutive episodes: -123.642 steps: 72 Episode return: -115.618
Average return in last 50 consecutive episodes: -124.031 steps: 372 Episode return: -149.212
Average return in last 50 consecutive episodes: -124.028 steps: 1600 Episode return: -120.712
Average return in last 50 consecutive episodes: -125.43 steps: 1528 Episode return: -179.803
Average return in last 50 consecutive episodes: -126.323 steps: 1190 Episode return: -151.571
Average return in last 50 consecutive episodes: -126.835 steps: 124 Episode return: -128.377
Average return in last 50 consecutive episodes: -127.321 steps: 773 Episode return: -130.576
Average return in last 50 consecutive episodes: -128.489 steps: 483 Episode return: -163.376
Average return in last 50 consecutive episodes: -128.949 steps: 1600 Episode return: -115.135
Average return in last 50 consecutive episodes: -126.912 steps: 1600 Episode return: -64.5735
Average return in last 50 consecutive episodes: -126.778 steps: 276 Episode return: -132.432
Average return in last 50 consecutive episodes: -127.042 steps: 84 Episode return: -129.554
Average return in last 50 consecutive episodes: -126.459 steps: 1600 Episode return: -85.9694
Average return in last 50 consecutive episodes: -124.475 steps: 1600 Episode return: -17.8702
Average return in last 50 consecutive episodes: -122.63 steps: 1600 Episode return: -75.2238
Average return in last 50 consecutive episodes: -120.096 steps: 1600 Episode return: -8.90409
Average return in last 50 consecutive episodes: -120.231 steps: 1600 Episode return: -112.664
Average return in last 50 consecutive episodes: -120.333 steps: 1600 Episode return: -119.645
Average return in last 50 consecutive episodes: -120.688 steps: 49 Episode return: -128.87
Average return in last 50 consecutive episodes: -121.552 steps: 1600 Episode return: -157.15
Average return in last 50 consecutive episodes: -121.14 steps: 1541 Episode return: -92.0127
Average return in last 50 consecutive episodes: -118.867 steps: 1600 Episode return: 4.01659
Average return in last 50 consecutive episodes: -118.971 steps: 44 Episode return: -117.312
Average return in last 50 consecutive episodes: -119.132 steps: 47 Episode return: -124.849
Average return in last 50 consecutive episodes: -118.888 steps: 1600 Episode return: -103.635
Average return in last 50 consecutive episodes: -118.793 steps: 106 Episode return: -112.632
Average return in last 50 consecutive episodes: -115.809 steps: 1600 Episode return: 36.2491
Average return in last 50 consecutive episodes: -114.401 steps: 1166 Episode return: -78.5781
Average return in last 50 consecutive episodes: -114.574 steps: 698 Episode return: -88.8064
Average return in last 50 consecutive episodes: -114.489 steps: 40 Episode return: -112.456
Average return in last 50 consecutive episodes: -110.119 steps: 1600 Episode return: 29.9862
Average return in last 50 consecutive episodes: -107.816 steps: 1600 Episode return: -8.59773
Average return in last 50 consecutive episodes: -107.769 steps: 40 Episode return: -114.643
Average return in last 50 consecutive episodes: -105.411 steps: 1600 Episode return: -21.132
Average return in last 50 consecutive episodes: -105.435 steps: 441 Episode return: -136.108
Average return in last 50 consecutive episodes: -105.667 steps: 127 Episode return: -127.637
Average return in last 50 consecutive episodes: -105.795 steps: 157 Episode return: -130.695
Average return in last 50 consecutive episodes: -106.13 steps: 186 Episode return: -138.876
Average return in last 50 consecutive episodes: -104.778 steps: 555 Episode return: -63.337
Average return in last 50 consecutive episodes: -105.581 steps: 300 Episode return: -142.637
Average return in last 50 consecutive episodes: -106.277 steps: 82 Episode return: -121.022
Average return in last 50 consecutive episodes: -103.158 steps: 1532 Episode return: 11.7225
Average return in last 50 consecutive episodes: -102.034 steps: 1000 Episode return: -61.7086
Average return in last 50 consecutive episodes: -100.317 steps: 74 Episode return: -117.943
Average return in last 50 consecutive episodes: -98.9306 steps: 287 Episode return: -83.0655
Average return in last 50 consecutive episodes: -98.5229 steps: 140 Episode return: -121.347
Average return in last 50 consecutive episodes: -98.0835 steps: 239 Episode return: -104.665
Average return in last 50 consecutive episodes: -96.6388 steps: 963 Episode return: -69.678
Average return in last 50 consecutive episodes: -96.8077 steps: 107 Episode return: -124.931
Average return in last 50 consecutive episodes: -96.8958 steps: 259 Episode return: -121.224
Average return in last 50 consecutive episodes: -97.1689 steps: 155 Episode return: -129.275
Average return in last 50 consecutive episodes: -95.38 steps: 1600 Episode return: -59.7673
Average return in last 50 consecutive episodes: -95.1809 steps: 176 Episode return: -110.754
Average return in last 50 consecutive episodes: -93.5505 steps: 161 Episode return: -98.2828
Average return in last 50 consecutive episodes: -93.0019 steps: 223 Episode return: -124.143
Average return in last 50 consecutive episodes: -92.9689 steps: 415 Episode return: -126.725
Average return in last 50 consecutive episodes: -92.3077 steps: 298 Episode return: -97.5179
Average return in last 50 consecutive episodes: -89.8921 steps: 1600 Episode return: -42.594
Average return in last 50 consecutive episodes: -89.9346 steps: 66 Episode return: -117.262
Average return in last 50 consecutive episodes: -89.012 steps: 1600 Episode return: -18.4452
Average return in last 50 consecutive episodes: -87.4671 steps: 1600 Episode return: -55.1881
Average return in last 50 consecutive episodes: -87.0204 steps: 379 Episode return: -107.216
Average return in last 50 consecutive episodes: -83.6134 steps: 1600 Episode return: 84.3791
Average return in last 50 consecutive episodes: -85.6042 steps: 44 Episode return: -117.41
Average return in last 50 consecutive episodes: -86.536 steps: 267 Episode return: -121.814
Average return in last 50 consecutive episodes: -88.7649 steps: 209 Episode return: -120.35
Average return in last 50 consecutive episodes: -88.267 steps: 340 Episode return: -87.7671
Average return in last 50 consecutive episodes: -88.1726 steps: 139 Episode return: -114.924
Average return in last 50 consecutive episodes: -87.4202 steps: 303 Episode return: -91.2532
Average return in last 50 consecutive episodes: -84.89 steps: 828 Episode return: -30.6359
Average return in last 50 consecutive episodes: -85.3432 steps: 1325 Episode return: -114.674
Average return in last 50 consecutive episodes: -87.7983 steps: 55 Episode return: -118.739
Average return in last 50 consecutive episodes: -88.0464 steps: 121 Episode return: -129.719
Average return in last 50 consecutive episodes: -84.5026 steps: 1600 Episode return: 52.343
Average return in last 50 consecutive episodes: -83.4168 steps: 612 Episode return: -49.3472
Average return in last 50 consecutive episodes: -81.6087 steps: 1042 Episode return: -22.2253
Average return in last 50 consecutive episodes: -80.6922 steps: 1354 Episode return: 82.0745
Average return in last 50 consecutive episodes: -80.8009 steps: 1112 Episode return: -84.0151
Average return in last 50 consecutive episodes: -78.8156 steps: 1600 Episode return: 10.4615
Average return in last 50 consecutive episodes: -77.8613 steps: 1600 Episode return: -64.7391
Average return in last 50 consecutive episodes: -79.0752 steps: 1600 Episode return: -30.713
Average return in last 50 consecutive episodes: -81.2954 steps: 255 Episode return: -119.604
Average return in last 50 consecutive episodes: -81.0283 steps: 1600 Episode return: -101.288
Average return in last 50 consecutive episodes: -80.4815 steps: 1600 Episode return: 6.2066
Average return in last 50 consecutive episodes: -78.8229 steps: 990 Episode return: -53.1779
Average return in last 50 consecutive episodes: -73.7923 steps: 1600 Episode return: 123.894
Average return in last 50 consecutive episodes: -73.036 steps: 805 Episode return: -92.8794
Average return in last 50 consecutive episodes: -72.8413 steps: 474 Episode return: -129.143
Average return in last 50 consecutive episodes: -73.0667 steps: 420 Episode return: -74.6044
Average return in last 50 consecutive episodes: -70.5741 steps: 917 Episode return: -18.0094
Average return in last 50 consecutive episodes: -65.1404 steps: 1600 Episode return: 150.667
Average return in last 50 consecutive episodes: -62.1558 steps: 1600 Episode return: 160.949
Average return in last 50 consecutive episodes: -63.4699 steps: 379 Episode return: -127.413
Average return in last 50 consecutive episodes: -58.6936 steps: 1600 Episode return: 120.873
Average return in last 50 consecutive episodes: -59.5657 steps: 168 Episode return: -126.669
Average return in last 50 consecutive episodes: -59.6402 steps: 161 Episode return: -125.075
Average return in last 50 consecutive episodes: -60.4723 steps: 244 Episode return: -146.271
Average return in last 50 consecutive episodes: -62.1037 steps: 273 Episode return: -151.248
Average return in last 50 consecutive episodes: -62.6129 steps: 268 Episode return: -150.39
Average return in last 50 consecutive episodes: -62.9127 steps: 267 Episode return: -136.212
Average return in last 50 consecutive episodes: -63.1422 steps: 191 Episode return: -140.751
Average return in last 50 consecutive episodes: -64.0947 steps: 184 Episode return: -107.394
Average return in last 50 consecutive episodes: -64.6636 steps: 242 Episode return: -139.198
Average return in last 50 consecutive episodes: -62.3061 steps: 1600 Episode return: 19.5929
Average return in last 50 consecutive episodes: -62.7394 steps: 229 Episode return: -145.808
Average return in last 50 consecutive episodes: -63.6258 steps: 1600 Episode return: -171.047
Average return in last 50 consecutive episodes: -58.3143 steps: 1600 Episode return: 168.058
Average return in last 50 consecutive episodes: -55.0228 steps: 1600 Episode return: 121.983
Average return in last 50 consecutive episodes: -50.1167 steps: 1600 Episode return: 128.044
Average return in last 50 consecutive episodes: -48.936 steps: 1240 Episode return: 40.5875
Average return in last 50 consecutive episodes: -50.0093 steps: 111 Episode return: -108.854
Average return in last 50 consecutive episodes: -49.7659 steps: 156 Episode return: -95.0433
Average return in last 50 consecutive episodes: -52.5058 steps: 369 Episode return: -52.6169
Average return in last 50 consecutive episodes: -53.684 steps: 686 Episode return: -176.319
Average return in last 50 consecutive episodes: -50.9626 steps: 972 Episode return: 14.2543
Average return in last 50 consecutive episodes: -48.5051 steps: 1228 Episode return: 2.52817
Average return in last 50 consecutive episodes: -48.4894 steps: 193 Episode return: -86.9865
Average return in last 50 consecutive episodes: -47.8063 steps: 227 Episode return: -80.7648
Average return in last 50 consecutive episodes: -46.9071 steps: 379 Episode return: -46.2942
Average return in last 50 consecutive episodes: -42.7834 steps: 1600 Episode return: 175.55
Average return in last 50 consecutive episodes: -37.8337 steps: 1600 Episode return: 132.811
Average return in last 50 consecutive episodes: -31.1118 steps: 1540 Episode return: 217.355
Average return in last 50 consecutive episodes: -26.837 steps: 1600 Episode return: 84.0199
Average return in last 50 consecutive episodes: -29.1701 steps: 1268 Episode return: -64.3114
Average return in last 50 consecutive episodes: -30.0141 steps: 608 Episode return: -91.5461
Average return in last 50 consecutive episodes: -29.1111 steps: 1197 Episode return: 22.9253
Average return in last 50 consecutive episodes: -31.1557 steps: 606 Episode return: -20.1603
Average return in last 50 consecutive episodes: -28.1917 steps: 1600 Episode return: 64.1859
Average return in last 50 consecutive episodes: -29.5618 steps: 1123 Episode return: -58.0438
Average return in last 50 consecutive episodes: -29.4677 steps: 241 Episode return: -60.0319
Average return in last 50 consecutive episodes: -30.3789 steps: 228 Episode return: -76.2712
Average return in last 50 consecutive episodes: -30.1265 steps: 55 Episode return: -106.984
Average return in last 50 consecutive episodes: -29.9236 steps: 116 Episode return: -91.1435
Average return in last 50 consecutive episodes: -31.6374 steps: 183 Episode return: -79.4874
Average return in last 50 consecutive episodes: -32.572 steps: 110 Episode return: -99.9078
Average return in last 50 consecutive episodes: -36.4161 steps: 241 Episode return: -68.308
Average return in last 50 consecutive episodes: -35.7481 steps: 537 Episode return: -59.4802
Average return in last 50 consecutive episodes: -35.4803 steps: 73 Episode return: -115.752
Average return in last 50 consecutive episodes: -34.3786 steps: 511 Episode return: -19.521
Average return in last 50 consecutive episodes: -34.677 steps: 515 Episode return: -32.9313
Average return in last 50 consecutive episodes: -39.5456 steps: 320 Episode return: -92.7603
Average return in last 50 consecutive episodes: -44.4839 steps: 136 Episode return: -85.9652
Average return in last 50 consecutive episodes: -43.7191 steps: 200 Episode return: -89.1749
Average return in last 50 consecutive episodes: -47.0233 steps: 556 Episode return: -44.3378
Average return in last 50 consecutive episodes: -45.3846 steps: 326 Episode return: -44.7351
Average return in last 50 consecutive episodes: -44.6242 steps: 158 Episode return: -87.0556
Average return in last 50 consecutive episodes: -42.5745 steps: 432 Episode return: -43.7845
Average return in last 50 consecutive episodes: -41.4787 steps: 228 Episode return: -96.4565
Average return in last 50 consecutive episodes: -40.2073 steps: 310 Episode return: -86.822
Average return in last 50 consecutive episodes: -37.2443 steps: 811 Episode return: 11.9388
Average return in last 50 consecutive episodes: -34.2963 steps: 696 Episode return: 6.64793
Average return in last 50 consecutive episodes: -33.0681 steps: 347 Episode return: -45.9815
Average return in last 50 consecutive episodes: -31.2739 steps: 409 Episode return: -49.4867
Average return in last 50 consecutive episodes: -33.5656 steps: 228 Episode return: -94.9924
Average return in last 50 consecutive episodes: -32.3634 steps: 267 Episode return: -85.6988
Average return in last 50 consecutive episodes: -31.0088 steps: 135 Episode return: -103.317
Average return in last 50 consecutive episodes: -34.2352 steps: 734 Episode return: 6.73725
Average return in last 50 consecutive episodes: -38.4204 steps: 219 Episode return: -87.2793
Average return in last 50 consecutive episodes: -42.8759 steps: 208 Episode return: -94.7285
Average return in last 50 consecutive episodes: -45.6712 steps: 78 Episode return: -99.1775
Average return in last 50 consecutive episodes: -44.7993 steps: 450 Episode return: -65.2599
Average return in last 50 consecutive episodes: -44.6509 steps: 164 Episode return: -87.6236
Average return in last 50 consecutive episodes: -45.7377 steps: 126 Episode return: -106.955
Average return in last 50 consecutive episodes: -44.9871 steps: 1600 Episode return: -138.79
Average return in last 50 consecutive episodes: -46.1047 steps: 329 Episode return: -41.6284
Average return in last 50 consecutive episodes: -48.5098 steps: 70 Episode return: -117.723
Average return in last 50 consecutive episodes: -48.8207 steps: 69 Episode return: -102.534
Average return in last 50 consecutive episodes: -47.5248 steps: 842 Episode return: -15.9709
Average return in last 50 consecutive episodes: -45.5471 steps: 1115 Episode return: 52.5912
Average return in last 50 consecutive episodes: -50.992 steps: 397 Episode return: -96.6959
Average return in last 50 consecutive episodes: -55.3877 steps: 280 Episode return: -86.9724
Average return in last 50 consecutive episodes: -61.7377 steps: 129 Episode return: -100.146
Average return in last 50 consecutive episodes: -63.2898 steps: 1006 Episode return: 6.41559
Average return in last 50 consecutive episodes: -62.8447 steps: 418 Episode return: -42.0531
Average return in last 50 consecutive episodes: -61.3269 steps: 764 Episode return: -15.6565
Average return in last 50 consecutive episodes: -61.5386 steps: 1172 Episode return: 12.3367
Average return in last 50 consecutive episodes: -60.7731 steps: 1600 Episode return: 18.1147
Average return in last 50 consecutive episodes: -62.8366 steps: 1600 Episode return: -38.9882
Average return in last 50 consecutive episodes: -63.2021 steps: 1600 Episode return: -76.3156
Average return in last 50 consecutive episodes: -62.4057 steps: 1600 Episode return: -20.2116
Average return in last 50 consecutive episodes: -62.0886 steps: 551 Episode return: -60.4185
Average return in last 50 consecutive episodes: -61.3783 steps: 1600 Episode return: -71.4687
Average return in last 50 consecutive episodes: -61.7228 steps: 139 Episode return: -108.371
Average return in last 50 consecutive episodes: -63.2054 steps: 1600 Episode return: -153.615
Average return in last 50 consecutive episodes: -57.3448 steps: 1600 Episode return: 193.121
Average return in last 50 consecutive episodes: -57.6524 steps: 750 Episode return: -83.6875
Average return in last 50 consecutive episodes: -58.2833 steps: 233 Episode return: -91.0272
Average return in last 50 consecutive episodes: -57.2377 steps: 446 Episode return: -63.4707
Average return in last 50 consecutive episodes: -57.9735 steps: 370 Episode return: -56.3101
Average return in last 50 consecutive episodes: -61.352 steps: 1121 Episode return: -201.858
Average return in last 50 consecutive episodes: -59.0412 steps: 1600 Episode return: 22.7802
Average return in last 50 consecutive episodes: -55.0779 steps: 1600 Episode return: 112.199
Average return in last 50 consecutive episodes: -55.4999 steps: 69 Episode return: -110.274
Average return in last 50 consecutive episodes: -56.8992 steps: 83 Episode return: -114.3
Average return in last 50 consecutive episodes: -58.0932 steps: 102 Episode return: -104.438
Average return in last 50 consecutive episodes: -58.4992 steps: 62 Episode return: -107.356
Average return in last 50 consecutive episodes: -57.7336 steps: 437 Episode return: -5.50598
Average return in last 50 consecutive episodes: -56.4947 steps: 313 Episode return: -34.5095
Average return in last 50 consecutive episodes: -56.9366 steps: 68 Episode return: -108.919
Average return in last 50 consecutive episodes: -59.1374 steps: 130 Episode return: -98.0993
Average return in last 50 consecutive episodes: -60.9453 steps: 166 Episode return: -83.746
Average return in last 50 consecutive episodes: -60.575 steps: 380 Episode return: -27.4685
Average return in last 50 consecutive episodes: -60.3418 steps: 552 Episode return: -37.8243
Average return in last 50 consecutive episodes: -60.298 steps: 441 Episode return: -92.8042
Average return in last 50 consecutive episodes: -60.4821 steps: 122 Episode return: -94.9049
Average return in last 50 consecutive episodes: -60.8028 steps: 134 Episode return: -119.351
Average return in last 50 consecutive episodes: -63.4927 steps: 116 Episode return: -127.756
Average return in last 50 consecutive episodes: -62.5518 steps: 862 Episode return: -40.2358
Average return in last 50 consecutive episodes: -62.0425 steps: 291 Episode return: -69.2622
Average return in last 50 consecutive episodes: -61.793 steps: 244 Episode return: -86.7031
Average return in last 50 consecutive episodes: -61.0264 steps: 495 Episode return: -26.9299
Average return in last 50 consecutive episodes: -61.3661 steps: 130 Episode return: -104.611
Average return in last 50 consecutive episodes: -60.882 steps: 248 Episode return: -82.7467
Average return in last 50 consecutive episodes: -60.5937 steps: 166 Episode return: -124.379
Average return in last 50 consecutive episodes: -61.2449 steps: 209 Episode return: -74.1846
Average return in last 50 consecutive episodes: -59.9798 steps: 331 Episode return: -54.4693
Average return in last 50 consecutive episodes: -59.7024 steps: 157 Episode return: -88.6672
Average return in last 50 consecutive episodes: -60.3326 steps: 353 Episode return: -47.4785
Average return in last 50 consecutive episodes: -63.7025 steps: 79 Episode return: -115.905
Average return in last 50 consecutive episodes: -63.9247 steps: 111 Episode return: -107.804
Average return in last 50 consecutive episodes: -64.3471 steps: 60 Episode return: -108.093
Average return in last 50 consecutive episodes: -64.1154 steps: 119 Episode return: -88.563
Average return in last 50 consecutive episodes: -66.5571 steps: 82 Episode return: -115.668
Average return in last 50 consecutive episodes: -67.79 steps: 144 Episode return: -103.699
Average return in last 50 consecutive episodes: -68.1285 steps: 514 Episode return: -32.5816
Average return in last 50 consecutive episodes: -69.7441 steps: 188 Episode return: -68.4446
Average return in last 50 consecutive episodes: -71.9706 steps: 113 Episode return: -93.2079
Average return in last 50 consecutive episodes: -73.62 steps: 107 Episode return: -121.457
Average return in last 50 consecutive episodes: -74.0327 steps: 111 Episode return: -96.9518
Average return in last 50 consecutive episodes: -75.4709 steps: 157 Episode return: -92.1209
Average return in last 50 consecutive episodes: -75.7887 steps: 251 Episode return: -76.312
Average return in last 50 consecutive episodes: -75.6394 steps: 346 Episode return: -64.0007
Average return in last 50 consecutive episodes: -73.0077 steps: 1454 Episode return: 23.2143
Average return in last 50 consecutive episodes: -71.7958 steps: 261 Episode return: -93.0214
Average return in last 50 consecutive episodes: -77.3361 steps: 1054 Episode return: -83.8921
Average return in last 50 consecutive episodes: -77.2202 steps: 790 Episode return: -77.8955
Average return in last 50 consecutive episodes: -75.1242 steps: 1383 Episode return: 13.7741
Average return in last 50 consecutive episodes: -75.3 steps: 351 Episode return: -72.2583
Average return in last 50 consecutive episodes: -76.2903 steps: 195 Episode return: -105.827
Average return in last 50 consecutive episodes: -74.0555 steps: 143 Episode return: -90.1198
Average return in last 50 consecutive episodes: -75.8893 steps: 238 Episode return: -68.907
Average return in last 50 consecutive episodes: -80.9959 steps: 1600 Episode return: -143.132
Average return in last 50 consecutive episodes: -80.5582 steps: 128 Episode return: -88.3902
Average return in last 50 consecutive episodes: -80.2851 steps: 127 Episode return: -100.643
Average return in last 50 consecutive episodes: -81.3011 steps: 1600 Episode return: -155.238
Average return in last 50 consecutive episodes: -81.1378 steps: 110 Episode return: -99.1887
Average return in last 50 consecutive episodes: -83.2167 steps: 44 Episode return: -109.455
Average return in last 50 consecutive episodes: -84.7164 steps: 36 Episode return: -109.492
Average return in last 50 consecutive episodes: -84.712 steps: 31 Episode return: -108.698
Average return in last 50 consecutive episodes: -84.9631 steps: 37 Episode return: -110.657
Average return in last 50 consecutive episodes: -85.4606 steps: 32 Episode return: -108.618
Average return in last 50 consecutive episodes: -87.1165 steps: 36 Episode return: -110.264
Average return in last 50 consecutive episodes: -88.5558 steps: 32 Episode return: -109.792
Average return in last 50 consecutive episodes: -88.8679 steps: 31 Episode return: -108.408
Average return in last 50 consecutive episodes: -89.1523 steps: 31 Episode return: -109.123
Average return in last 50 consecutive episodes: -88.9649 steps: 32 Episode return: -109.985
Average return in last 50 consecutive episodes: -88.5911 steps: 31 Episode return: -109.064
Average return in last 50 consecutive episodes: -90.1759 steps: 55 Episode return: -119.477
Average return in last 50 consecutive episodes: -90.9716 steps: 33 Episode return: -109.047
Average return in last 50 consecutive episodes: -91.3928 steps: 125 Episode return: -107.765
Average return in last 50 consecutive episodes: -93.2186 steps: 49 Episode return: -118.219
Average return in last 50 consecutive episodes: -93.4624 steps: 49 Episode return: -116.8
Average return in last 50 consecutive episodes: -94.0177 steps: 59 Episode return: -110.513
Average return in last 50 consecutive episodes: -93.0835 steps: 213 Episode return: -77.6684
Average return in last 50 consecutive episodes: -94.1545 steps: 46 Episode return: -127.731
Average return in last 50 consecutive episodes: -95.6001 steps: 50 Episode return: -126.751
Average return in last 50 consecutive episodes: -93.9412 steps: 669 Episode return: -5.72193
Average return in last 50 consecutive episodes: -95.9053 steps: 269 Episode return: -145.685
Average return in last 50 consecutive episodes: -96.1228 steps: 46 Episode return: -126.779
Average return in last 50 consecutive episodes: -96.4543 steps: 47 Episode return: -124.378
Average return in last 50 consecutive episodes: -96.8372 steps: 46 Episode return: -127.241
Average return in last 50 consecutive episodes: -97.5861 steps: 46 Episode return: -126.008
Average return in last 50 consecutive episodes: -97.7834 steps: 46 Episode return: -125.529
Average return in last 50 consecutive episodes: -98.2256 steps: 46 Episode return: -125.813
Average return in last 50 consecutive episodes: -100.09 steps: 46 Episode return: -125.79
Average return in last 50 consecutive episodes: -101.316 steps: 48 Episode return: -129.745
Average return in last 50 consecutive episodes: -102.372 steps: 261 Episode return: -146.009
Average return in last 50 consecutive episodes: -102.853 steps: 245 Episode return: -145.525
Average return in last 50 consecutive episodes: -103.359 steps: 59 Episode return: -122.224
Average return in last 50 consecutive episodes: -103.952 steps: 58 Episode return: -121.787
Average return in last 50 consecutive episodes: -104.811 steps: 57 Episode return: -119.278
Average return in last 50 consecutive episodes: -105.93 steps: 56 Episode return: -119.922
Average return in last 50 consecutive episodes: -109.99 steps: 611 Episode return: -179.812
Average return in last 50 consecutive episodes: -110.32 steps: 65 Episode return: -109.529
Average return in last 50 consecutive episodes: -110.855 steps: 63 Episode return: -110.611
Average return in last 50 consecutive episodes: -111.473 steps: 54 Episode return: -108.806
Average return in last 50 consecutive episodes: -113.932 steps: 52 Episode return: -109.168
Average return in last 50 consecutive episodes: -114.122 steps: 135 Episode return: -81.7646
Average return in last 50 consecutive episodes: -112.841 steps: 1178 Episode return: -41.7842
Average return in last 50 consecutive episodes: -108.912 steps: 1600 Episode return: 106.349
Average return in last 50 consecutive episodes: -102.703 steps: 1234 Episode return: 241.54
Average return in last 50 consecutive episodes: -99.5412 steps: 603 Episode return: 14.951
Average return in last 50 consecutive episodes: -99.2506 steps: 176 Episode return: -73.8602
Average return in last 50 consecutive episodes: -92.9661 steps: 1521 Episode return: 213.578
Average return in last 50 consecutive episodes: -87.8114 steps: 1186 Episode return: 102.501
Average return in last 50 consecutive episodes: -80.981 steps: 1224 Episode return: 242.331
Average return in last 50 consecutive episodes: -76.5607 steps: 1034 Episode return: 111.557
Average return in last 50 consecutive episodes: -72.9853 steps: 1600 Episode return: 69.2803
Average return in last 50 consecutive episodes: -70.9079 steps: 553 Episode return: -4.83062
Average return in last 50 consecutive episodes: -70.5893 steps: 1600 Episode return: -94.7247
Average return in last 50 consecutive episodes: -71.5914 steps: 1600 Episode return: -158.725
Average return in last 50 consecutive episodes: -69.4171 steps: 574 Episode return: -1.54519
Average return in last 50 consecutive episodes: -63.8645 steps: 1600 Episode return: 167.837
Average return in last 50 consecutive episodes: -63.6018 steps: 192 Episode return: -95.2718
Average return in last 50 consecutive episodes: -57.3004 steps: 1600 Episode return: 205.946
Average return in last 50 consecutive episodes: -50.2275 steps: 1255 Episode return: 243.66
Average return in last 50 consecutive episodes: -46.312 steps: 1600 Episode return: 86.7106
Average return in last 50 consecutive episodes: -44.8366 steps: 1600 Episode return: -45.7059
Average return in last 50 consecutive episodes: -45.6727 steps: 581 Episode return: -150.85
Average return in last 50 consecutive episodes: -38.6855 steps: 1266 Episode return: 241.595
Average return in last 50 consecutive episodes: -34.699 steps: 1137 Episode return: 81.1034
Average return in last 50 consecutive episodes: -27.0798 steps: 1040 Episode return: 264.161
Average return in last 50 consecutive episodes: -19.6777 steps: 1043 Episode return: 259.593
Average return in last 50 consecutive episodes: -20.3106 steps: 106 Episode return: -109.317
Average return in last 50 consecutive episodes: -15.4966 steps: 886 Episode return: 112.973
Average return in last 50 consecutive episodes: -15.4712 steps: 123 Episode return: -125.485
Average return in last 50 consecutive episodes: -17.0231 steps: 309 Episode return: -83.3143
Average return in last 50 consecutive episodes: -9.15884 steps: 1207 Episode return: 247.527
current reward: 269.933 iteration: 0
current reward: 269.519 iteration: 1
current reward: 272.248 iteration: 2
current reward: 270.689 iteration: 3
current reward: 269.084 iteration: 4
current reward: 274.329 iteration: 5
current reward: 269.327 iteration: 6
current reward: 269.711 iteration: 7
current reward: 268.206 iteration: 8
current reward: 274.028 iteration: 9
current reward: 272.748 iteration: 10
current reward: 267.385 iteration: 11
current reward: 270.188 iteration: 12
current reward: 270.06 iteration: 13
current reward: 258.547 iteration: 14
current reward: 269.833 iteration: 15
current reward: 268.279 iteration: 16
current reward: 270.715 iteration: 17
current reward: -34.6566 iteration: 18
current reward: 269.889 iteration: 19
current reward: 271.019 iteration: 20
current reward: 268.743 iteration: 21
current reward: 266.891 iteration: 22
current reward: 266.284 iteration: 23
current reward: 269.178 iteration: 24
current reward: 265.153 iteration: 25
current reward: 270.82 iteration: 26
current reward: 265.098 iteration: 27
current reward: 271.255 iteration: 28
current reward: -4.06497 iteration: 29
current reward: 269.047 iteration: 30
current reward: 267.422 iteration: 31
current reward: 267.843 iteration: 32
current reward: 270.291 iteration: 33
current reward: 269.891 iteration: 34
current reward: 269.37 iteration: 35
current reward: 270.05 iteration: 36
current reward: 269.809 iteration: 37
current reward: -114.552 iteration: 38
current reward: 269.222 iteration: 39
current reward: 268.263 iteration: 40
current reward: 267.988 iteration: 41
current reward: 270.948 iteration: 42
current reward: 272.539 iteration: 43
current reward: 267.317 iteration: 44
current reward: 264.931 iteration: 45
current reward: 270.504 iteration: 46
current reward: 265.317 iteration: 47
current reward: 267.617 iteration: 48
current reward: 273.15 iteration: 49

I can also upload the saved models.

[nishantkr18]

Wow! that agent is now much better. Thanks!

[nishantkr18]

@zoq it would be great if you could upload the models. Also, do u think we should also try training with http://gym.openai.com/envs/BipedalWalkerHardcore-v2/?

[zoq]

Model files - http://data.kurg.org/models.tar.gz. I guess since the serialization doesn't work right now with cling (I guess it might work once cereal support is merged), we could merge the notebook without loading the model for now. Let me know what you think.

About the BipedalWalkerHardcore-v2 env, sure let's do it, same architecture, or adding another layer?

[nishantkr18]

we could merge the notebook without loading the model for now.

Do u mean we should keep the notebook ready for run, with the models, but not actually run the notebook, instead wait for cereal support? If that's the case, we can manually set the parameter values for now? Is there an example that does it?

About the BipedalWalkerHardcore-v2 env, sure let's do it, same architecture, or adding another layer?

I think this architecture should do. Lets try it and see..
Also,

        if (averageReturn > -10)
            break;

here, I think we can replace -10 with 100 or something, so as to make the agent get an avg consecutive 50 episodes score of more than 100 for some better performance.

[zoq]

Do u mean we should keep the notebook ready for run, with the models, but not actually run the notebook, instead wait for cereal support? If that's the case, we can manually set the parameter values for now? Is there an example that does it?

That's what I had in mind as well, what we have to do is to assign the weights with Parameters() = loaded paraemter for each model and call ResetParameters for each model as well.

I think this architecture should do. Lets try it and see..

Started.

[zoq]

Still training - https://gist.github.com/zoq/01f67ae240f42f66c6b3534243c222c9

[nishantkr18]

Still training - https://gist.github.com/zoq/01f67ae240f42f66c6b3534243c222c9

Well, it seems we might have to add one more layer, as it is talking too long to converge.

[zoq]

Agreed, what about adding:

.Add(new Linear<>(128, 128));
.Add(new ReLULayer<>());

to both networks?

[nishantkr18]

Agreed, what about adding:

.Add(new Linear<>(128, 128));
.Add(new ReLULayer<>());

to both networks?

Yes, that should do the job.

BTW, if it doesn't slow down much, we can consider using 256 nodes per layer instead of 128?

[zoq]

See the same result with 128 nodes: https://gist.github.com/zoq/60022127679f234a11648dc2691116b0 it develops into the wrong direction. Let me try with 256.

[zoq]

Current status - https://gist.github.com/zoq/9f4285564d74996003644ae53af12990

[nishantkr18]

Current status - https://gist.github.com/zoq/9f4285564d74996003644ae53af12990

I feel that the agent is getting trained, though very slowly, because only in the last view episodes has it finally managed to run for 2000 steps.

We can try for a bit longer and see. But atleast we can get this PR merged.

[zoq]

Yeah, I'll let this train for some more hours and see what the trend is. I'll take a look at the PR later as well.

[zoq]

What do you think, should we reinclude the training part, comment it by default and say something like, uncomment the code above to train the model from scratch?

[zoq]

No improvement so far - https://gist.github.com/zoq/e11b529ac3b46fb9ab3e0616850fb865

[nishantkr18]

What do you think, should we reinclude the training part, comment it by default and say something like, uncomment the code above to train the model from scratch?

Sorry for the delay. Please have a look.

No improvement so far - https://gist.github.com/zoq/e11b529ac3b46fb9ab3e0616850fb865

I guess we need to drop it then. Although changing the init parameters might do something, I'm not so confident about it.

[zoq]

I guess we need to drop it then. Although changing the init parameters might do something, I'm not so confident about it.

Will do some further testing, once we have a model that works reasonably good we can open another PR.

[zoq]

Do you mind to save the weights into a file and load that file instead of pasting the weights into a cell? If I use GitHub to render the notebook the output is a really long cell.

[zoq]

@nishantkr18 Any updates on this one?

[nishantkr18]

Extremely sorry for missing your message. I'll do it right away

[nishantkr18]

Done :)

[zoq]

Hm, what about we save the data as csv and use data::load("sac_q.csv", qNetwork.Parameters()); instead of using ifstream, vector and conv_to?

[nishantkr18]

Well, I tried using csv, but there seems to be some issue with it. Is it ok if I use data::load("sac_q.txt", qNetwork.Parameters()); ?

[zoq]

Yes, that should work, unless we have to transpose the data, depends on the format.

[zoq]

Thanks, this looks great to me.

[mlpack-bot]

Second approval provided automatically after 24 hours. 👍

[zoq]

Thanks for keeping up with the comments.