Skip to content

cirosantilli/awesome-reinforcement-learning-games

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

54 Commits

README.adoc

README.adoc

 
 

Repository files navigation

Awesome Reinforcement Learning Games

2. Introduction

This is currently just a crazy project idea research, and does not have a major implementation yet.

The goal is to make an open source game / series of games to be played by computers, to serve as interesting problems to develop full AI.

The games are not designed to be fun for human players: they are designed to be fun for human programmers who are coding full AI robots.

We will however add an interactive human player mode so that humans can mess around with the games and better understand them, to develop better AI faster.

The central focus is find or design games that are:

  • simplified versions of the real world, to allow us to simulate it fast enough

  • while still retaining key aspects of the real world like:

    • vision, perspective, Euclidean space and time

    • but still has magic non spacial elements (which in the real world are usually chemical reactions). E.g.:

      • your beef tastes better after you put it in the fire

The goal is to introduce more and more complex mechanics one by one progressively, and try to introduce mechanics such that:

  • humans are able to solve the problem

  • the existing AIs were not able to solve the problem

in order to break the AIs and force AI developperes to make smarter AIs that look more like humans.

AIs should then be tested against mechanics that they have not trained against, much like in: http://www.gvgai.net/

3. GPU

The GPU is the most promising computational breakthrough of our time, and might be the first thing that allows us to reach the critical speed / cost point needed for full AI.

We would like therefore to explore GPU usage for all parts of the simulation / decision making process.

The game / AI loop is:

  1. update the rigid body physics engine world state (e.g. Bullet Physics supports OpenCL, NVIDIA PhysX)

    This step can also generate as output some inputs for the game AI, e.g. if the robot touches a wall, a touch sensation input for that part of the robot.

  2. render scene (with OpenGL GPU, duh)

  3. do the game AI, that is, simulate the brain of the virtual robot / animal

    Inputs: * rendered scene (array of pixels) from the renderer * touch input from the physics engine * other inputs: radio signals / voice from other robots currently held "non-physical" symbolic state, like the Minecraft inventory, or the amount of bullets on an FPS

    + Processing method:

    • try to keep everything inside the GPU to avoid CPU round-trip

      • OpenCL / OpenGL bridges for rendered pixels

      • physics engine GPU / OpenCL bridges

    • use custom OpenCL to make the decisions as much as possible

    • forces generated by the robot’s motors, to be passed as input to the physics engine

    • other side effects like:

      • radio signals / voice emitted to other peer robots for communication

      • magic "non-physical" symbol operation actions, like crafting an item in Minecraft or pulling a trigger on an FPS

  4. to back to step 1

4. Game mechanic

Good game concepts should both:

  • mimic key aspects of simple real robots / animals, which can hopefully be generalized to "real" reality:

    • first person with perspective, possibly 2D to simplify as in http://blog.otoro.net/2015/05/07/creatures-avoiding-planks/

    • goals similar to those of real animals: eat, don’t get eaten, find resources

    • mix a physical simulation engine with purely symbolic information that skips the rigid body simulator.

      • bullets in real life are physical, but in FPS games they are just one integer.

      • Minecraft raw materials are interesting, as they bridge the voxel physics engine and symbolic information: when you mine, you take a voxel from the physics engine, and change a symbol on your inventory

  • simplify the real world

    • we don’t have enough computational power yet for reality

    • we don’t know good AI algorithms yet, start with simpler problems first

  • voxel or vector:

    • the real world is actually voxel, which is tempting

    • however, voxels are expensive to compute, therefore practical games we’ve seen so far:

      • have voxels at least as large as the player. But 50% of our daily thinking is about things smaller than ourselves, e.g. food.

      • have very limited voxel dynamics, e.g. no rotations

Mechanics ideas:

  • extract discrete concepts from continuous data.

    You notice that some pixels always move together. So you interpret them as being one single "object", a discrete concept in your mind.

    Possibly partition tasks as: CPU: symbolic manipulation tasks GPU: continuous data tasks

  • https://en.wikipedia.org/wiki/Object_permanence

    You are a starving predator and you see a single prey turning around a corner.

    The robot should have memory that he is chasing this single prey, and some type of understanding that things disappear from sight around corners, and then turn the corner as well.

  • discrete consequences to continuous action. E.g.: you touch a red wall: then a blue gate opens.

  • topology:

    • two preys go into a closed cave. I enter and eat 2 preys. So then I should go out to find more.

    • I see a prey through a hole in the wall. The hole is too small for me to get in. So I must find another way.

    • I construct a closed barrier around me. Predators cannot eat me! (Minecraft home building)

  • current tool mechanic: the robot has a current tool + a single actuator for it. Like switching weapons on Minecraft / FPS.

  • thinking like someone else: if prey sees me, it runs away. So I find a way to approach without being seen.

  • recognizing places: there is lots of food around the place with two trees. So I hang out around that place more often than others.

Evil ideas:

  • invisible objects. E.g.: Mario hidden blocks. Possibly controlled by agents, like camouflage in nature, or spy in Team Fortress.

  • objects that look like other objects. E.g. a prey that looks like predator. Mimicry in nature: https://en.wikipedia.org/wiki/Mimicry Possibly via metamorphosis.

  • two objects that look the same, but act very differently. Agent then has to identify which one it is.

5. Literature

Existing research in the area.

7. Animal intelligence

Understanding animals could give insights into what our OpenCL intelligence should look like.

13. Spacial hardware

I hereby predict that in the future, if we ever reach full AI, we will have specialized integrated circuits that interpret 3D space somehow.

Here are my current findings:

14. Game engines

Well, we need to code our games at some point.

The major choice is between using a game engine, or going directly to Box2D and SDL.

If we do use a game engine, it has to be open source and C++.

14.1. Urho3D

https://github.com/urho3d/Urho3D just work C++ examples, no bullshit.

Box2D and Bullet backends, SDL based. Feels elegant.

No official platformer example :-( Very few examples overall: urho3d/urho3d#641

2D feels second-rate compared to 3D:

Has a headless mode, and frame limiting removal.

14.2. Cocos2d-x

Cocos2d-x: by far the most popular FOSS engine, but build is annoying on Ubuntu.

Tons of minimal examples inside the source tree.

Surprisingly few larger game examples on GitHub however.

Fix this build failure:

relocation R_X86_64_32 against symbol_ZN15CProfileManager4RootE' can not be used when making a shared object; recompile with -fPIC`

Cocos is downloading pre-built deps from: https://github.com/cocos2d/cocos2d-x-3rd-party-libs-bin which are built with: https://github.com/cocos2d/cocos2d-x-3rd-party-libs-src instead of using the repo builds :-(

TODO: don’t rebuild the entire thing for every new project! make install inexistent???

Relies on fmod for sound which is closed source…​ but devs seem keen to make the dependency optional cocos2d/cocos2d-x#16355 This likely also blocks Debian inclusion. An old request (does not mention fmod): https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=577236

Demo games:

Worked:

Couldn’t make work:

Using:

sudo make install

fails and makes me sad.

14.3. Other game engines

  • https://github.com/ivansafrin/Polycode

    sudo apt-get install libphysfs-dev libopenal-dev libfreetype6-dev

    Bleh, was annoying to compile on Ubuntu, so I quit for a while.

    CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
ASSIMP_INCLUDE_DIR
  used as include directory in directory /home/ciro/git/Polycode/Tools/Contents/polyimport

    Dead since 2015.

    Demos seem promising, actual platformers and pong: https://www.youtube.com/watch?v=Ik3ImYElcD0

    Bullet and box2d frontends.

  • https://github.com/oxygine/oxygine-framework

    Well, the distinctively named "Game" example compiled fine, but I get bad vibes from that project for some reason. Not that many examples either.

    Box2D is not integrated apparently, one of the examples copy pastes Box2D source into the tree to use it.

  • Godot: too much emphasis on scripting BS :-)

  • irrilicht: no rigid body physics, only hard collision, and some bullet3d hacks which I dare not touch ever :-)

  • atomic: Urho3D fork, which was later re-open sourced. More focus on scripting apparently? Fail :-(

About

Ideas for games that serve as interesting vison / spacial / robotics reinforcement learning problems, and how to solve them.

Resources

Readme
Activity

Stars

64 stars

Watchers

9 watching

Forks

16 forks
Report repository

Releases

No releases published

Packages

No packages published