Add example solving OpenAI Gym MountainCarContinuous env

[jakobj]

This example demonstrates how to solve an Open AI Gym environment with CGP. Fitness function is based on empirical observations to maximize solving speed. Found solution is almost always of the form c \dot x, sometimes c \dot x - x for that last bit of efficiency (try it!). The solution intuitively makes sense, doesn't it? ;) #interpretableRL

tagging our mountaincar expert @weidel-p

[mschmidt87]

This is pretty cool! However, a principle problem that I see is that this has an additional dependency, namely OpenAI Gym. I think there are two possible ways:

• move this example in a new subfolder, that could be called additional_examples or something like this and then explain in the README.md that this has an additional dependency. Perhaps even put gym in a pip requirements file.
• move this example in a new, separate repository with "advanced" examples or miniprojects/demonstrations.

For the example itself:

• It would be really cool to add a visualization of the resulting behavior after the evoluation. Even cooler: Create visualizations with all "evolutionary steps" where the fitness makes a jump towards higher values.
• Add some logging of the found solution.

[weidel-p]

Hey, thanks for tagging me here. It's the first time I look into CGP, it looks very cool and the installation was super easy, it just works. 🥇

Regarding this PR, I agree with @mschmidt87. A bit more information and visualization would be neat. I ran the script, it counted from 0 to 500 and in the end, something reached a fitness for 13.5. Is this a good or bad value? I would also like to see the found solution and if this solution is sufficient to solve the task according to the criteria by the Gym.
Also, as @mschmidt87 said, some visualization would be great. At least the best solution should be rendered.

[jakobj]

Thanks for your comments! ✨

I don't like the idea of creating a separate advanced examples or similar directory. Instead I've added a comment to the docstring of the module that this example needs gym installed. Do you think that's enough?

I've added visualization of the solutions for every increase of the champion's fitness. Please check whether this works for you. 😬

I'm not sure what you mean by "logging the found solution" do you mean just print to the screen? ;)

[jakobj]

"I would also like to see the found solution and if this solution is sufficient to solve the task according to the criteria by the Gym."

Actually I'm not sure what the criteria are for this environment. Do you know where to check this @weidel-p?

[weidel-p]

Yes, it works for me. Looks very good :) 👍

You can find the threshold for solving the task in the leaderboard here:
https://github.com/openai/gym/wiki/Leaderboard
or you can also find it in the python code of the environment: gym/envs/__init__.py

For this task, the threshold is at 90.

[mschmidt87]

Looks good, in principle, but I think the amount of animations is too many at the end. Let's just animate the winning, final solution, and perhaps keep it as a switch to visualize all steps with fitness gain.

With logging the solution, yes, just print it to the screen.

[mschmidt87]

Regarding the additional dependency: What about adding something like:

try:
    import gym
except ImportError:
    raise ImportError("Could not import OpenAI Gym package. Please install it via `pip install gym`.")

[jakobj]

thanks for the reviews @weidel-p @mschmidt87 :)

i've implemeted your suggestions and am happy to report that the found solution passes the solving criteria for this environment. in order to achieve this i needed to adapt the fitness function a bit, putting more emphasis on the reward gathered rather than the number of episodes solved.

please have another look!

[mschmidt87]

Very nice, @jakobj , just a minor suggestion: Why not setting the min_fitness parameter to 90., since that is the official criterium for "passing the task"? 200.0 seems like an arbitrary choice? I think it would be quite impressive for the user to see that 90 is achieved after only ~100 generations.

[jakobj]

yes, i agree this would be the optimal case. however, the requirement is defined as ">90 fitness averaged over 100 consecutive trials". unfortunately we can not use this as an objective during evolution since only high-performance solutions are able to achieve this large number of trials in a reasonable amount of time. hence i opted for this specific combination objective + min fitness.

[mschmidt87]

Okay, understood. As a minor change, I would suggest to use dx/dt instead of \dot x.

[jakobj]

fixed, pls have another look :)

[mschmidt87]

LGTM