NEAT-spaceinvaders

We use NeuroEvoultion of Augumenting Topologies or NEAT to solve the famous ATARI game SpaceInvaders with the help of OpenAI's gym and NEAT-python

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

Prerequisites

Libraries needed

neat
gym
cv2
pickle
graphviz
time
numpy
matplotlib

Introduction

Often its popular to use reinforcement learning to solve games. Evolutionary algorithm inspite being not so popular in its application, sometimes even outperforms reinforcement learing. Using NEAT, I explore through my project whether the algorithm is robust enough to solve ATARI games like SpaceInvaders. For the game environment I have used OpenAI's gym. The package for NEAT algorithm is neat built by CodeReclaimers

Approach

The bot or agent performs 6 actions namely:

1. NOOP or No operation
2. FIRE
3. RIGHT
4. LEFT
5. RIGHTFIRE
6. LEFTFIRE

Using NEAT we evovle a feed-forward neural network giving one of these 6 outputs. The input provided varies based on these three methods

• Array input bot In this method we feed the network with important information extracted from the image of the current game state. The information fed is an array containing:-

  1. x,y coordinates of the bot
  2. x,y coordinate of all the aliens
  3. x,y coordinate of the nearing bullet in the vicinity space of the bot
  4. Aliens killed We noticed that even though the code ran pretty fast due to less number of inputs hence less time to calculate output but it still didn't perform well. It stagnated rather quickly and dies quickly. It performed only slightly better than a random bot. Overall the bot was not able to beat the game

• Image_v1 input bot In this method we feed the network directly with the image of the current game state. To make the algorithm run fastly, we resized the image and flattened it into a 1331 sized array. The bot was able to beat the game, however it was observed that the bot was consistent in killing decent number of bots but it seldom killed high number of enemies. In this method we observed slow yet steady growth in number of enemies killed with each generation. Overall it took 447 generations to beat the game

• Image_v2 input bot
  In this method we feed the network a cropped version to the bot, as its sufficient to provide the bot with just the aliens in the vicinity of the bot. Also we modified the bullet and elongated it till the first object it strikes. This helps the bot understand the importance of dodging the bullets. We also were able to increase the resolution hence we preserve the speed of the algorithm. We observed that these changes forced the bot to learn a new stratergy to beat the game. In this method we observed stark growth with high stand deviations between different genomes in the generation. Overall it took 117 generations to beat the game

Fitness Function

For any evolution based algorithm, fitness funtion is a very important part of the algorithm. In this we use the following fitness function:-
fitness = 0.8xhigh_score + 0.01xframe + 11xaliens_dead + 0.5xcounter + penalty
Variables:-

• frame: denotes how many frames the bot survived
• aliens_dead: denotes how many aliens it killed
• counter: denotes how much the bot moved from its previous position, the more movement will lead to higher value of counter
• penalty: If the bot is in bullet's trajectory then we add a penalty

Simulation

• Random bot

• Input array bot
  We observe that the bot doesnt perform well and has not learnt anything

• Image_v1 bot
  We observe that the bot is moving constantly and shooting. This is improving its survivabiltiy, in further genrations the bot becomes better in surviving by constant moving. However its not moving based on the bullet trajectory, hence it dies often. Sadly, I wasn't able to have a better footage which is captured using gym's monitor class

• Image_v2 bot
  We observe that the bot has found a really intresting stratergy. It now hides behind a rock and shoots. This exponentially increases its survivabiltiy and hence it was able to beat the game faster.

Visualization

• Image_v1 average fitness graph
  We see that the progress is pretty slow and steady

• Image_v2 average fitness graph
  We see that the progress is fast though with high standard deviation. The spike denotes finishing the game which gives a default fitness value to cross the threshold

• Image_v2 speciation graph

• What the bot sees

• Network The network is too big, so click on the line to see

Running the test

Before running the main code ensure that plots_image,recording_image,saved_models_image directories are created

For running the NEAT algorithm
mkdir plots_image,recording_image,saved_models_image
python AI_bot_image_v2.py
For running the best_bot genome or network
python best_bot_simulation.py
Note: The best_bot may not always beat the game, it may sometimes fail to do so
File description

• 1 - AI_bot_array.py - Main code based on input array method
• 2 - AI_bot_image.py - Main code based on image_v1 input method
• 3 - AI_bot_image_v2.py - Main code based on image_v2 input method
• 4 - config - contains all the important parameters for NEAT algorithm. Please follow the docs for more information
• 5 - inputgenerator.py - code for extracting important information for the main code
• 6 - visualize.py - code for genrating visualization of network
• 7 - neat-checkpoint-116 - checkpoint for image_v2, can start training after gen 116 by loading this file
• 8 - best_bot.pkl - genome or network data for the best bot for image_v2
• 9 - plots_image - directory containg all graphs for image_v2
• 10 - best_bot_simulation.py - code to play the genome contained in the best_bot.pkl

Reference

• NEAT documentation
• NEAT paper
• OpenAI's gym

Further Note

The code can be modified and used for other NEAT based problems by just changing the eval_genomes functions which helps in modification of fitness score of all genomes in the genration