diff --git a/README.md b/README.md index b530adee..859efc60 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,18 @@ [![Build Status](https://travis-ci.org/CodeReclaimers/neat-python.svg)](https://travis-ci.org/CodeReclaimers/neat-python) [![Coverage Status](https://coveralls.io/repos/CodeReclaimers/neat-python/badge.svg?branch=master&service=github)](https://coveralls.io/github/CodeReclaimers/neat-python?branch=master) -## STATUS NOTE ## +## FORK STATUS ## -This project is currently in maintenance-only mode. I will make bug fixes, do cleanup, and possibly improve sample code -as I have time, but I will not be adding any new features. The forks by -[@drallensmith](https://github.com/drallensmith/neat-python) and [@bennr01](https://github.com/bennr01/neat-python) have -been extended beyond this implementation a great deal, so those might be better starting points if you need more -features than what you see here. +This is a fork from [CodeReclaimers](https://github.com/CodeReclaimers/neat-python). +The main focus is Non-dominated Sorting for Multiobjective Fitness. That means having more than one fitness value that should be optimized. +This is done throught the implementation of [NSGA-II](https://ieeexplore.ieee.org/document/996017) as a Reproduction method. More details on the `neat/nsga2/` readme. + +The current repository also presents a hoverboard game/simulation to be used as a problem for testing the NSGA-II feature, as well as examples for training it with and without NSGA-II. +Check the readme on `examples/nsga2` for more details. + +![hoverboard-reference](https://i.imgur.com/CfrdHmr.gif) + +I've tried keeping the minimal amount of change to the core library, so merging to the main fork should be easy. All these changes are backwards-compatible. ## About ## @@ -31,7 +36,7 @@ The documentation, is available on [Read The Docs](http://neat-python.readthedoc ## Citing ## Here is a Bibtex entry you can use to cite this project in a publication. The listed authors are the maintainers of -all iterations of the project up to this point. +all iterations of the project up to this point. ``` @misc{neat-python, @@ -39,4 +44,4 @@ all iterations of the project up to this point. Author = {Alan McIntyre and Matt Kallada and Cesar G. Miguel and Carolina Feher da Silva}, howpublished = {\url{https://github.com/CodeReclaimers/neat-python}} } -``` \ No newline at end of file +``` diff --git a/examples/hoverboard/README.md b/examples/hoverboard/README.md new file mode 100644 index 00000000..12947ba7 --- /dev/null +++ b/examples/hoverboard/README.md @@ -0,0 +1,92 @@ +## NSGA-II examples ## + +The scripts in this directory show examples of using NEAT to control a hoverboard on a game. +It uses Recurrent Networks to control the intensity of both left and right thrusters of the hoverboard, based on it's velocity, angular velocity and normal vector. All those informations could be retreived from real world sensors. + +There are two examples: +- __time__: Evolves network with single-fitness _DefaultReproduction_ method, optimizing flight time. +- __timedist__: Evolves network with _NSGA2Reproduction_ method, optimizing two fitness values: flight time and mean squared distance from center. + +![hoverboard-reference](https://i.imgur.com/CfrdHmr.gif) + +## hoverboard.py + +This file implements the game using [pygame](http://pygame.org/). + +You can manually play it! However, it's designed to be near impossible without AI (or some USB flight controllers, I guess). + +```python +pip install pygame +python hoverboard.py +``` + +- Q/A : +/- left thruster +- P/L : +/- right thruster + +## evolve-time.py + +A reference example using a Recurrent Network with the _DefaultReproduction_ method, optimizing a single value of fitness: flightime. +The input values are: velocity (X/Y), angular velocity and normal vector (X/Y). + +![hoverboard-reference](https://i.imgur.com/UpJ2HA7.gif) + +The evolution converges fast on simple behaviours such as overcoming gravity by boosting both thrusters simultaneously, however a more refined fitness method should include the total variation of velocities and normal vector to help it converge faster to a stable controller. + +``` +> python evolve-time.py + +> python evolve-time.py --help +``` + +## evolve-timedist.py + +A working example using a Recurrent Network with _NSGA2Reproduction_ method, optimizing two fitness values: flight time and mean squared distance from center. +The input values are: velocity (X/Y), angular velocity, normal vector (X/Y) and distance to center (X/Y). + +For each genome, instead of a single cycle this method runs 10 game cycles, starting from 5 preset points (including center) with the starting angle A and -A. The fitness results are accumulated and then divided by 10. + +![hoverboard-reference](https://i.imgur.com/CfrdHmr.gif) + +This method converges a lot faster to results way beyond the convergence point of the default method. More about this at the _Results_ section of this document. + +``` +> python evolve-timedist.py + +> python evolve-timedist.py --help +``` + +## visualize.py + +This is a small tool for viewing the generation data stored at checkpoints. +It allows you to watch the best genome of each generation, as well as plotting fitness and species data over generations. +The plots on the _Results_ section of this document were made with this tool. + +``` +> python visualize.py + +> python visualize.py --help +``` + +## gui.py + +This is an utilitary lib for drawing neat-python networks using pygame. + +# Results + +Here's a quick comparison of results found for this particular hoverboard game with and without the use of NSGA-II. These experiments must be improved in order to better outline the benefits and downsides of this approach. Please feel free to develop them further. + +The fitness value plotted is Flight Time on both cases. As described above, the NSGA-II example takes the average of 10 runs starting from preset points, to avoid developing behaviours biased on starting at the center. +The observed increase in mean convergence does not seem to rely on these 10 runs evaluation, it is actually harder to evolve in those conditions. + +![results_fitness](https://s1.imghub.io/05eik.png) + +The distribution of species over generations is heavily affected by NSGA-II. More research is due to evaluate it's cost and benefits. Overall, the species tend to stabilize, having more time to evolve it's features. +A plot of species on the solution space is due to evaluate their distribution, that should be grouped and moving towards the pareto-front. + +![results_species](https://s1.imghub.io/05H6H.png) + +This plot is messy and needs to be improved. It's a scatter plot of every genome on every generation, color coded. +In order to visualize the overall movement of the population in the solution space, each generation set of points is filled with a Delaunay Triangulation. You can see the generation shapes moving towards the pareto front. +A black line represents the best solution of each generation, so you can see the optimization path and convergence. + +![results_pareto](https://s1.imghub.io/05GrJ.png) diff --git a/examples/hoverboard/config-default b/examples/hoverboard/config-default new file mode 100644 index 00000000..b4e3c12a --- /dev/null +++ b/examples/hoverboard/config-default @@ -0,0 +1,81 @@ +#--- parameters for the XOR-2 experiment ---# + +[NEAT] +fitness_criterion = max +fitness_threshold = 1000 +pop_size = 50 +reset_on_extinction = False + +[DefaultGenome] +# node activation options +activation_default = random +activation_mutate_rate = 0.5 +activation_options = relu gauss + +# node aggregation options +aggregation_default = random +aggregation_mutate_rate = 0.3 +aggregation_options = max sum + +# node bias options +bias_init_mean = 0.0 +bias_init_stdev = 1.0 +bias_max_value = 2.0 +bias_min_value = -2.0 +bias_mutate_power = 0.5 +bias_mutate_rate = 0.7 +bias_replace_rate = 0.1 + +# genome compatibility options +compatibility_disjoint_coefficient = 1.0 +compatibility_weight_coefficient = 0.5 + +# connection add/remove rates +conn_add_prob = 0.5 +conn_delete_prob = 0.5 + +# connection enable options +enabled_default = True +enabled_mutate_rate = 0.01 + +feed_forward = False +initial_connection = partial_nodirect 0.6 + +# node add/remove rates +node_add_prob = 0.4 +node_delete_prob = 0.4 + +# network parameters +num_hidden = 0 +num_inputs = 5 +num_outputs = 2 + +# node response options +response_init_mean = 1.0 +response_init_stdev = 0.0 +response_max_value = 30.0 +response_min_value = -30.0 +response_mutate_power = 0.0 +response_mutate_rate = 0.0 +response_replace_rate = 0.0 + +# connection weight options +weight_init_mean = 0.0 +weight_init_stdev = 60.0 +weight_max_value = 100 +weight_min_value = -100 +weight_mutate_power = 3 +weight_mutate_rate = 0.8 +weight_replace_rate = 0.3 + +[DefaultSpeciesSet] +compatibility_threshold = 20.0 + +[DefaultStagnation] +species_fitness_func = max +max_stagnation = 20 +species_elitism = 2 + +[DefaultReproduction] +elitism = 1 +survival_threshold = 0.2 diff --git a/examples/hoverboard/config-nsga2 b/examples/hoverboard/config-nsga2 new file mode 100644 index 00000000..66dce92d --- /dev/null +++ b/examples/hoverboard/config-nsga2 @@ -0,0 +1,80 @@ +#--- parameters for the XOR-2 experiment ---# + +[NEAT] +fitness_criterion = max +fitness_threshold = 1000 +pop_size = 50 +reset_on_extinction = False + +[DefaultGenome] +# node activation options +activation_default = random +activation_mutate_rate = 0.5 +activation_options = relu gauss + +# node aggregation options +aggregation_default = random +aggregation_mutate_rate = 0.3 +aggregation_options = max sum + +# node bias options +bias_init_mean = 0.0 +bias_init_stdev = 1.0 +bias_max_value = 2.0 +bias_min_value = -2.0 +bias_mutate_power = 0.5 +bias_mutate_rate = 0.7 +bias_replace_rate = 0.1 + +# genome compatibility options +compatibility_disjoint_coefficient = 1.0 +compatibility_weight_coefficient = 0.5 + +# connection add/remove rates +conn_add_prob = 0.5 +conn_delete_prob = 0.5 + +# connection enable options +enabled_default = True +enabled_mutate_rate = 0.01 + +feed_forward = False +initial_connection = partial_nodirect 0.6 + +# node add/remove rates +node_add_prob = 0.4 +node_delete_prob = 0.4 + +# network parameters +num_hidden = 0 +num_inputs = 7 +num_outputs = 2 + +# node response options +response_init_mean = 1.0 +response_init_stdev = 0.0 +response_max_value = 30.0 +response_min_value = -30.0 +response_mutate_power = 0.0 +response_mutate_rate = 0.0 +response_replace_rate = 0.0 + +# connection weight options +weight_init_mean = 0.0 +weight_init_stdev = 60.0 +weight_max_value = 100 +weight_min_value = -100 +weight_mutate_power = 3 +weight_mutate_rate = 0.8 +weight_replace_rate = 0.3 + +[DefaultSpeciesSet] +compatibility_threshold = 20.0 + +[DefaultStagnation] +species_fitness_func = max +max_stagnation = 20 +species_elitism = 2 + +[NSGA2Reproduction] +survival_threshold = 0.2 diff --git a/examples/hoverboard/evolve-time.py b/examples/hoverboard/evolve-time.py new file mode 100644 index 00000000..bb6bf0f5 --- /dev/null +++ b/examples/hoverboard/evolve-time.py @@ -0,0 +1,135 @@ +""" + + Hoverboard: evolve Flight Time (1 fitness) + + Small example tool to control the hoverboard game using NEAT. + It uses the DefaultReproduction method, with a single fitness value: flight time. + + Each genome is evaluated starting from the center, with a given starting angle. + + # USAGE: + > python evolve-time.py + > python evolve-time.py --help + + @author: Hugo Aboud (@hugoaboud) + +""" + +from __future__ import print_function + +## DEBUG +## Uses local version of neat-python +import sys +sys.path.append('../../') +## DEBUG +import neat + +import os +import math +import argparse + +from hoverboard import Game +from visualize import GameReporter, watch + +# General Parameters + +GAME_TIME_STEP = 0.001 +CHECKPOINT_FOLDER = 'checkpoint-time' +CONFIG_FILE = 'config-default' + +# CLI Parameters + +GAME_START_ANGLE = 0 +SILENT = False +FAST_FORWARD = False + +## +# Evaluation +## + +# Evaluate genome +def eval(genome, config): + # Create network + net = neat.nn.RecurrentNetwork.create(genome, config) + # Create game + game = Game(GAME_START_ANGLE,False) + # Run the game and calculate fitness + genome.fitness = 0 + while(True): + # Activate Neural Network + output = net.activate([game.hoverboard.velocity[0], game.hoverboard.velocity[1], game.hoverboard.ang_velocity, game.hoverboard.normal[0], game.hoverboard.normal[1]]) + + # Update game state from output and then update physics + game.hoverboard.set_thrust(output[0], output[1]) + game.update(GAME_TIME_STEP) + + # Fitness (best option): flight time + genome.fitness += GAME_TIME_STEP + + # Fitness (alternatives) + #genome.fitness -= math.sqrt((game.hoverboard.x-0.5)**2+(game.hoverboard.y-0.5)**2)*GAME_TIME_STEP + #genome.fitness -= (game.hoverboard.normal[0]**2) + #genome.fitness -= math.sqrt(game.hoverboard.velocity[0]**2+game.hoverboard.velocity[1]**2) + #genome.fitness -= game.hoverboard.ang_velocity**2 + + # End of game + if (game.reset_flag): break + +# Evaluate generation +def eval_genomes(genomes, config): + # Evaluate each genome looking for the best + for genome_id, genome in genomes: + eval(genome, config) + +## +# Main +## + +def main(): + + # Parse CLI arguments + parser = argparse.ArgumentParser(description='Example of evolving a Neural Network using neat-python to play a 2D hoverboard game.') + parser.add_argument('angle', help="Starting angle of the platform") + parser.add_argument('-c', '--checkpoint', help="Number of a checkpoint on the 'checkpoint-reference' folder to start from") + parser.add_argument('-s', '--silent', help="Don't watch the game", nargs='?', const=True, type=bool) + parser.add_argument('-f', '--fastfwd', help="Fast forward the game preview (10x)", nargs='?', const=True, type=bool) + args = parser.parse_args() + + # Store global parameters + global GAME_START_ANGLE + global SILENT + global FAST_FORWARD + GAME_START_ANGLE = float(args.angle) + SILENT = bool(args.silent) + FAST_FORWARD = bool(args.fastfwd) + + # Load neat configuration. + config = neat.Config(neat.DefaultGenome, neat.DefaultReproduction, + neat.DefaultSpeciesSet, neat.DefaultStagnation, + CONFIG_FILE) + + # Create the population or load from checkpoint + if (args.checkpoint != None): + population = neat.Checkpointer.restore_checkpoint(os.path.join(CHECKPOINT_FOLDER,'gen-'+str(args.checkpoint)), config) + else: + population = neat.Population(config) + + # Add a stdout reporter to show progress in the terminal. + population.add_reporter(neat.StdOutReporter(False)) + + # Add a game reporter to watch the game post evaluation + if (not SILENT): + population.add_reporter(GameReporter(population, GAME_TIME_STEP*(10 if FAST_FORWARD else 1), GAME_START_ANGLE)) + + # Add a checkpointer to save population pickles + if not os.path.exists(CHECKPOINT_FOLDER): + os.makedirs(CHECKPOINT_FOLDER) + population.add_reporter(neat.Checkpointer(1,filename_prefix=os.path.join(CHECKPOINT_FOLDER,'gen-'))) + + # Run until a solution is found. + winner = population.run(eval_genomes) + + # Display the winning genome. + print('\nBest genome:\n{!s}'.format(winner)) + +main() diff --git a/examples/hoverboard/evolve-timedist.py b/examples/hoverboard/evolve-timedist.py new file mode 100644 index 00000000..3fdcad73 --- /dev/null +++ b/examples/hoverboard/evolve-timedist.py @@ -0,0 +1,150 @@ +""" + + Hoverboard: evolve Flight Time + Distance from Center (2 fitnesses), using NSGA-II + + Small example tool to control the hoverboard game using NEAT. + It uses the NSGA2Reproduction method, with two fitness values: flight time and average squared distance from the center. + + Each genome is evaluated starting from 5 preset points (including center), + with a given starting angle and it's inverse (ex. 5° and -5°). + The total fitness is the average for the 10 runs. + + # USAGE: + > python evolve-timedist.py + > python evolve-timedist.py --help + + @author: Hugo Aboud (@hugoaboud) + +""" + +from __future__ import print_function + +## DEBUG +## Uses local version of neat-python +import sys +sys.path.append('../../') +## DEBUG +import neat + +import os +import math +import argparse + +from hoverboard import Game +from visualize import GameReporter, watch + +# General Parameters + +GAME_TIME_STEP = 0.001 +CHECKPOINT_FOLDER = 'checkpoint-timedist' +CONFIG_FILE = 'config-nsga2' + +# CLI Parameters + +GAME_START_ANGLE = 0 +SILENT = False +FAST_FORWARD = False + +## +# Evaluation +## + +# Evaluate genome +def eval(genome, config): + # Create network + net = neat.nn.RecurrentNetwork.create(genome, config) + # Genome fitness to be accumulated + genome.fitness = neat.nsga2.NSGA2Fitness(0,0) + # Eval starting from 5 points + for start in [(0.5,0.5),(0.25,0.25),(0.25,0.75),(0.75,0.25),(0.75,0.75)]: + # Eval with angle and inverse angle + for angle in [GAME_START_ANGLE,-GAME_START_ANGLE]: + # Create game + game = Game(GAME_START_ANGLE,False,start=start) + # Run the game and calculate fitness (list) + fitness = [0,0] + while(True): + # Activate Neural Network + dir = [0.5-game.hoverboard.x, 0.5-game.hoverboard.y] + output = net.activate([game.hoverboard.velocity[0], game.hoverboard.velocity[1], game.hoverboard.ang_velocity, game.hoverboard.normal[0], game.hoverboard.normal[1], dir[0], dir[1]]) + + # Update game state from output and then update physics + game.hoverboard.set_thrust(output[0], output[1]) + game.update(GAME_TIME_STEP) + + # Fitness 0: flight time + fitness[0] += GAME_TIME_STEP + # Fitness 1: distance from center (target) + dist = dir[0]**2+dir[1]**2 + fitness[1] -= dist + + # End of game + if (game.reset_flag): break + # Add fitness to genome fitness + genome.fitness.add(fitness[0],fitness[1]) + + # Take average of runs + genome.fitness.values[0] /= 10 + genome.fitness.values[1] /= 10 + genome.fitness.values[1] /= genome.fitness.values[0] + +# Evaluate generation +def eval_genomes(genomes, config): + # Evaluate each genome + for genome_id, genome in genomes: + eval(genome, config) + +## +# Main +## + +def main(): + + # Parse CLI arguments + parser = argparse.ArgumentParser(description='Example of evolving a Neural Network using neat-python to play a 2D hoverboard game.') + parser.add_argument('angle', help="Starting angle of the platform") + parser.add_argument('-c', '--checkpoint', help="Number of a checkpoint on the 'checkpoint-reference' folder to start from") + parser.add_argument('-s', '--silent', help="Don't watch the game", nargs='?', const=True, type=bool) + parser.add_argument('-f', '--fastfwd', help="Fast forward the game preview (10x)", nargs='?', const=True, type=bool) + args = parser.parse_args() + + # Store global parameters + global GAME_START_ANGLE + global SILENT + global FAST_FORWARD + GAME_START_ANGLE = float(args.angle) + SILENT = bool(args.silent) + FAST_FORWARD = bool(args.fastfwd) + + # Load neat configuration. + # Here's where we load the NSGA-II reproduction module + config = neat.Config(neat.DefaultGenome, neat.nsga2.NSGA2Reproduction, + neat.DefaultSpeciesSet, neat.DefaultStagnation, + CONFIG_FILE) + + # Create the population or load from checkpoint + global population + if (args.checkpoint != None): + population = neat.Checkpointer.restore_checkpoint(os.path.join(CHECKPOINT_FOLDER,'gen-'+str(args.checkpoint)), config) + else: + population = neat.Population(config) + + # Add a stdout reporter to show progress in the terminal. + population.add_reporter(neat.StdOutReporter(False)) + + # Add a game reporter to watch the game post evaluation + if (not SILENT): + population.add_reporter(GameReporter(population, GAME_TIME_STEP*(10 if FAST_FORWARD else 1), GAME_START_ANGLE, True)) + + # Add a checkpointer to save population pickles + if not os.path.exists(CHECKPOINT_FOLDER): + os.makedirs(CHECKPOINT_FOLDER) + population.add_reporter(neat.Checkpointer(1,filename_prefix=os.path.join(CHECKPOINT_FOLDER,'gen-'))) + + # Run until a solution is found. + winner = population.run(eval_genomes) + + # Display the winning genome. + print('\nBest genome:\n{!s}'.format(winner)) + +main() diff --git a/examples/hoverboard/gui.py b/examples/hoverboard/gui.py new file mode 100644 index 00000000..68820c7c --- /dev/null +++ b/examples/hoverboard/gui.py @@ -0,0 +1,192 @@ +""" + + neat-python Neural Network GUI + + Small lib for drawing a neat-python network using pygame + + @author: Hugo Aboud (@hugoaboud) + +""" + +import random +import pygame +import math + +# Dimensions and Positions +DISPLAY = [512,512] +NN_PADDING_TOP = 32 +NN_INPUT_X = 128 +NN_OUTPUT_X = 384 +NN_Y_STEP = 25 +NN_NODE_RADIUS = 7 +NN_CONN_THICK = 1 +NN_INPUT_NAMES = [' velocity Y', + ' velocity Y', + 'angular velocity', + ' normal X', + ' normal Y', + ' dir X', + ' dir Y'] +NN_OUTPUT_COLORS = [(200,0,127), + (0,127,200)] + +# Colors +COLOR_TEXT = (200,200,200) +COLOR_NN_NODE_INPUT = ((200, 150, 0), (0, 150, 200)) +COLOR_NN_NODE = {'relu' : ((200, 100, 100), (100, 100, 200)), + 'tanh' : ((100, 200, 100), (200, 100, 200)), + 'gauss' : ((100, 100, 200), (200, 100, 100)), + 'softplus' : ((100, 100, 200), (200, 100, 100)), + 'sigmoid' : ((100, 100, 200), (200, 100, 100))} +COLOR_NN_CONNECTION = ((200, 150, 0), (0, 150, 200)) +NN_WEIGHT_SCALE = 10 + +## +# Utility +## + +# Interpolate colors using t=[-1,1] +# t = -1 -> colors[0] +# t = 0 -> (0,0,0) +# t = 1 -> colors[1] + +def mixcolor(colors, t): + # clamp t + if (t < -1): t = -1 + elif (t > 1): t = 1 + # interpolate color + color = [0,0,0] + if (t > 0): color = [t*colors[1][0],t*colors[1][1],t*colors[1][2]] + elif (t < 0): color = [-t*colors[0][0],-t*colors[0][1],-t*colors[0][2]] + return color + +## +# GUI +## + +class NeuralNetworkGUI: + + class Node: + def __init__(self, x, y, color, inputs): + self.last_value = 0 + self.value = 0 + self.x = x + self.y = y + self.color = color + self.inputs = inputs + + def render(self, screen): + pygame.draw.circle(screen, mixcolor(self.color,(self.last_value+self.value)/2.0), (self.x, self.y), NN_NODE_RADIUS) + + def set(self, value): + self.last_value = self.value + self.value = value + if (self.value < -1): self.value = -1 + elif (self.value > 1): self.value = 1 + + def __init__(self, generation, genome, species, net, info=True): + self.generation = generation + self.genome = genome + self.species = species + self.net = net + self.info = info + + self.nodes = {} + self.hidden = [] + self.height = NN_PADDING_TOP+(len(self.net.input_nodes)-1)*NN_Y_STEP + + # input nodes + inputs = self.net.input_nodes + for i in range(len(inputs)): + self.nodes[inputs[i]] = self.Node(NN_INPUT_X, NN_PADDING_TOP+i*NN_Y_STEP, COLOR_NN_NODE_INPUT, {}) + + # hidden + output nodes + outputs = self.net.output_nodes + for id, node in self.genome.nodes.items(): + # create node on a random position + x = random.randrange(NN_INPUT_X,NN_OUTPUT_X) + y = random.randrange(NN_PADDING_TOP,NN_PADDING_TOP+max((len(inputs),len(outputs)))) + self.nodes[id] = self.Node(x, y, COLOR_NN_NODE[node.activation], {ids[0]:conn.weight for ids,conn in self.genome.connections.items() if ids[1] == id}) + # output nodes (specific position and color) + if (id == outputs[0]): + self.nodes[id].color = ((0,0,0), NN_OUTPUT_COLORS[0]) + self.nodes[id].x = NN_OUTPUT_X + self.nodes[id].y = NN_PADDING_TOP+1.5*NN_Y_STEP + self.nodes[id].value = 0.5 + elif (id == outputs[1]): + self.nodes[id].color = ((0,0,0), NN_OUTPUT_COLORS[1]) + self.nodes[id].x = NN_OUTPUT_X + self.nodes[id].y = NN_PADDING_TOP+2.5*NN_Y_STEP + self.nodes[id].value = 0.5 + # hidden nodes (add to list) + else: + self.hidden.append(id) + + # Initialize pygame modules + pygame.init() + + # Font for UI + self.font = pygame.font.SysFont(None, 12) + + # moves the hidden nodes around to make the graph more readable + def relax(self, factor=0.01): + for id in self.hidden: + hnode = self.nodes[id] + # get average of distances + avg = 0 + dists = {} + for nid, node in self.nodes.items(): + if (id == nid): continue + dists[nid] = math.sqrt((hnode.x-node.x)**2+(hnode.y-node.y)**2) + avg += dists[nid] + avg /= len(self.nodes)-1 + # move trying to keep all distances on the average + for nid, node in self.nodes.items(): + if (id == nid): continue + dist = dists[nid] + if (dist != 0): + dir = ((node.x-hnode.x)/dist,(node.y-hnode.y)/dist) + hnode.x += dir[0]*(dist-avg)*factor + hnode.y += dir[1]*(dist-avg)*factor + # clamp Y coordinates + if (hnode.y < NN_PADDING_TOP): hnode.y = NN_PADDING_TOP + elif (hnode.y > self.height): hnode.y = self.height + + # render the network + def render_net(self, screen): + # normalize net values + values = self.net.values[self.net.active].copy() + values[-3] /= 360 + + # render node inputs + for id, node in self.nodes.items(): + for id, weight in node.inputs.items(): + b = self.nodes[id] + pygame.draw.line(screen, mixcolor(COLOR_NN_CONNECTION,b.value*weight/NN_WEIGHT_SCALE), (node.x,node.y),(b.x,b.y), NN_CONN_THICK) + + # render nodes + for id, node in self.nodes.items(): + if (id in values): + node.set(values[id]) + node.render(screen) + + # render input names + for i in range(len(self.net.input_nodes)): + img = self.font.render(NN_INPUT_NAMES[i], True, COLOR_TEXT) + screen.blit(img, (NN_INPUT_X-70,NN_PADDING_TOP+i*NN_Y_STEP-6)) + + # relax and render + def render(self, screen): + # relax and render network graph + self.relax() + self.render_net(screen) + # if info enabled, render info texts + if (self.info): + img = self.font.render(str('GENERATION: {0}'.format(self.generation)), True, COLOR_TEXT) + screen.blit(img, (DISPLAY[0]-70,DISPLAY[1]-45)) + img = self.font.render(str('SPECIES: {0}'.format(self.species)), True, COLOR_TEXT) + screen.blit(img, (DISPLAY[0]-70,DISPLAY[1]-30)) + img = self.font.render(str('ID: {0}'.format(self.genome.key)), True, COLOR_TEXT) + screen.blit(img, (DISPLAY[0]-70,DISPLAY[1]-15)) + img = self.font.render(str('FITNESS: {0}'.format(self.genome.fitness)), True, COLOR_TEXT) + screen.blit(img, (10,DISPLAY[1]-15)) diff --git a/examples/hoverboard/hoverboard.py b/examples/hoverboard/hoverboard.py new file mode 100644 index 00000000..14d600a3 --- /dev/null +++ b/examples/hoverboard/hoverboard.py @@ -0,0 +1,260 @@ +""" + + [Hoverboard] + A 2D game consisting of a hoverboard with two thrusters. You can control their thrust individually. + The challenge is to keep the platform floating in the middle of the screen. + If it touches any of the walls, it's destroyed. + + + > pip install pygame + > python hoverboard.py + + + Q : ++ left thruster + A : -- left thruster + P : ++ right thruster + L : -- right thruster + + @author: Hugo Aboud (@hugoaboud) + +""" + +import math +import sys +import pygame +from pygame.locals import QUIT + +""" + Settings +""" + +# Dimensions +DISPLAY = (512,512) # w, h +THRUSTER = (20,300,10) # w, h, padding + +# Colors +COLOR_BACKGROUND = (30,30,30) +COLOR_PLAYER = (180,200,255) +COLOR_THRUSTER_OFF = (50,50,50) +COLOR_THRUSTER_LEFT = (200,0,127) +COLOR_THRUSTER_RIGHT = (0,127,200) + +# Physics +MASS = 2 +GRAVITY = 1.5 +FORCE = 3 +INTERTIA_MOMENTUM = 0.05 + +# Input +INPUT_STEP = 0.1 + +## +# Hoverboard +# Most of the Physics and Rendering is happening here +## + +class Hoverboard: + def __init__(self, x = 0.5, y = 0.5, w = 100, h = 10, angle = 0, velocity = None, ang_velocity = 0): + self.x = x + self.y = y + self.w = w + self.h = h + self.angle = angle + self.velocity = velocity if (velocity != None) else [0,0] + self.ang_velocity = ang_velocity + + # calculate normal + rad = math.pi*self.angle/180 + self.normal = (-math.sin(rad), -math.cos(rad)) + + # default thrust (last is used for smoother display) + self.last_thrust = [0.5,0.5] + self.thrust = [0.5,0.5] + + # create surface to draw the hoverboard into + self.surface = pygame.Surface((w,h*5), pygame.SRCALPHA) + + def render(self, screen): + # clear surface + self.surface.fill((0,0,0,0)) + # draw platform (rect) + pygame.draw.rect(self.surface, COLOR_PLAYER, (0, self.h*2, self.w, self.h)) + # draw thruster left (pixel) + l = (self.thrust[0]+self.last_thrust[0])/2.0 + th = 2*self.h*l + for y in range(int(th)): + for x in range(self.h): + self.surface.set_at((x,self.h*3+y), + (COLOR_THRUSTER_LEFT[0], + COLOR_THRUSTER_LEFT[1], + COLOR_THRUSTER_LEFT[2], + int(255*(1-y/th)))) + # draw thruster right (pixel) + r = (self.thrust[1]+self.last_thrust[1])/2.0 + th = 2*self.h*r + for y in range(int(th)): + for x in range(self.w-self.h,self.w): + self.surface.set_at((x,self.h*3+y), + (COLOR_THRUSTER_RIGHT[0], + COLOR_THRUSTER_RIGHT[1], + COLOR_THRUSTER_RIGHT[2], + int(255*(1-y/th)))) + # rotate surface around center + rotated = pygame.transform.rotate(self.surface, self.angle) + rect = rotated.get_rect() + rect[0] -= rect[2]/2 + rect[1] -= rect[3]/2 + # position and blit + rect[0] += self.x*DISPLAY[0] + rect[1] += self.y*DISPLAY[1] + screen.blit(rotated, rect) + + # update physics + def update(self, delta_t): + # gravity + # > increases Y velocity + self.velocity[1] += GRAVITY*delta_t + + # thrust torque + # > torque to angular acceleration (no radius used, ajusted by inerta momentum) + # > angular acceleration +> angular velocity + ang_accel_l = (FORCE/INTERTIA_MOMENTUM)*self.thrust[0] + ang_accel_r = (FORCE/INTERTIA_MOMENTUM)*self.thrust[1] + self.ang_velocity += (ang_accel_r-ang_accel_l)*delta_t + + # (TODO: drag / air resistance ) + + # thrust force + # > force to linear acceleration + # > linear acceleration +> linear velocity + accel = (FORCE/MASS)*(self.thrust[0]+self.thrust[1]) + self.velocity[0] += self.normal[0]*accel*delta_t + self.velocity[1] += self.normal[1]*accel*delta_t + + # update position and angle + self.x += self.velocity[0]*delta_t + self.y += self.velocity[1]*delta_t + self.angle += self.ang_velocity*delta_t + # update normal + rad = math.pi*self.angle/180 + self.normal = (-math.sin(rad), -math.cos(rad)) + + # update thrust values within [0,1] bounds + # also keep last value for smoother display + def set_thrust(self, left, right): + self.last_thrust = list(self.thrust) + self.thrust[0] = left + if (self.thrust[0] < 0): self.thrust[0] = 0 + elif (self.thrust[0] > 1): self.thrust[0] = 1 + self.thrust[1] = right + if (self.thrust[1] < 0): self.thrust[1] = 0 + elif (self.thrust[1] > 1): self.thrust[1] = 1 + +## +# Game +# Here's the game loop, main rendering method, input handling and +# construction/destruction methods. +## + +class Game: + def __init__(self, start_angle = 0, frontend = True, network_gui = None, start=(0.5,0.5)): + self.start_angle = start_angle + self.frontend = frontend + + # (optional) + # use a NeuralNetworkGUI to display real time genome topology and info + self.network_gui = network_gui + + # reset flag is set when the hoverboard collides with the borders + # it is true after the update when the collision happens, and before the next update + self.reset_flag = False + + # create hoverboard + self.hoverboard = Hoverboard(start[0], start[1], angle = start_angle) + + # initialize pygame modules + pygame.init() + + # if Front-End enabled, open pygame window + if (self.frontend): + self.screen = pygame.display.set_mode(DISPLAY) + + # main render + def render(self): + # clear screen + self.screen.fill(COLOR_BACKGROUND) + + # render hoverboard + self.hoverboard.render(self.screen) + + # render thrusters UI (left and right bars) + l = (self.hoverboard.thrust[0]+self.hoverboard.last_thrust[0])/2.0 # smooth display + r = (self.hoverboard.thrust[1]+self.hoverboard.last_thrust[1])/2.0 # smooth display + pygame.draw.rect(self.screen, COLOR_THRUSTER_OFF, (THRUSTER[2], (DISPLAY[1]-THRUSTER[1])/2,THRUSTER[0],THRUSTER[1]*(1-l))) + pygame.draw.rect(self.screen, COLOR_THRUSTER_LEFT, (THRUSTER[2], (DISPLAY[1]-THRUSTER[1])/2+THRUSTER[1]*(1-l),THRUSTER[0],THRUSTER[1]*l)) + pygame.draw.rect(self.screen, COLOR_THRUSTER_OFF, (DISPLAY[0]-THRUSTER[0]-THRUSTER[2], (DISPLAY[1]-THRUSTER[1])/2,THRUSTER[0],THRUSTER[1]*(1-r))) + pygame.draw.rect(self.screen, COLOR_THRUSTER_RIGHT, (DISPLAY[0]-THRUSTER[0]-THRUSTER[2], (DISPLAY[1]-THRUSTER[1])/2+THRUSTER[1]*(1-r),THRUSTER[0],THRUSTER[1]*r)) + + # (optional) render NeuralNetworkGUI + if (self.network_gui != None): + self.network_gui.render(self.screen) + + # game loop + def loop(self): + last_t = pygame.time.get_ticks() + while True: + t = pygame.time.get_ticks() + # Events + for event in pygame.event.get(): + # Input Handling + if event.type == pygame.KEYDOWN: + if event.key == pygame.K_q: + self.hoverboard.thrust[0] += INPUT_STEP + if (self.hoverboard.thrust[0] > 1): self.hoverboard.thrust[0] = 1 + if event.key == pygame.K_a: + self.hoverboard.thrust[0] -= INPUT_STEP + if (self.hoverboard.thrust[0] < 0): self.hoverboard.thrust[0] = 0 + if event.key == pygame.K_p: + self.hoverboard.thrust[1] += INPUT_STEP + if (self.hoverboard.thrust[1] > 1): self.hoverboard.thrust[1] = 1 + if event.key == pygame.K_l: + self.hoverboard.thrust[1] -= INPUT_STEP + if (self.hoverboard.thrust[1] < 0): self.hoverboard.thrust[1] = 0 + # Quit + if event.type == QUIT: + pygame.quit() + sys.exit() + + # Update + self.update((t-last_t)/1000) + last_t = t + + # update physics, collisions and game state + # this is separated from the game loop so it can be called from an outside loop + # this way, it's possible to control the game loop externally to evaluate the genomes + def update(self, delta_t): + # Clear reset flag + if (self.reset_flag): self.reset_flag = False + # Update hoverboard physics + self.hoverboard.update(delta_t) + # Collision (end game) + if (self.hoverboard.x <= 0 or self.hoverboard.x >= 1 or + self.hoverboard.y <= 0 or self.hoverboard.y >= 1): + self.reset() + # Front-End (Render) + if (self.frontend): + self.render() + pygame.display.update() + + # reset the hoverboard to the initial state + def reset(self): + self.hoverboard = Hoverboard(angle = self.start_angle) + # Set reset flag + self.reset_flag = True + +## +# If you run this file with `python hoverboard.py`, it creates a Game instance +# and runs the `loop` method. This way, the game can also be played without NEAT. +## +if __name__ == "__main__": + Game().loop() diff --git a/examples/hoverboard/visualize.py b/examples/hoverboard/visualize.py new file mode 100644 index 00000000..51e9f456 --- /dev/null +++ b/examples/hoverboard/visualize.py @@ -0,0 +1,288 @@ +""" + + neat-python hoverboard visualize tool + + Small tool for watching neat-python genomes play the hoverboard game, + and also make some nice little plots. + It takes the genomes from checkpoints generated by the evolve-<>.py files. + + # USAGE: + > python visualize.py + > python visualize.py --help + + @author: Hugo Aboud (@hugoaboud) + +""" + +import os +import argparse +from scipy.spatial import Delaunay + +## DEBUG +## Uses local version of neat-python +import sys +sys.path.append('../../') +## DEBUG +import neat +from neat.math_util import mean + +import random +import math + +from matplotlib import pyplot as plt +from matplotlib import gridspec as gridspec +from hoverboard import Game +from gui import NeuralNetworkGUI + +# General Parameters + +GAME_TIME_STEP = 0.001 + +# CLI Parameters + +GAME_START_ANGLE = 0 +FAST_FORWARD = False +JUST_PLOT = False +WATCH_LAST = False +EXPERIMENT = 'time' + +## +# Reporter +# Used to watch the game after each evaluation +## + +class GameReporter(neat.reporting.BaseReporter): + def __init__(self, population, step, angle, dir_input = False): + self.population = population + self.step = step + self.angle = angle + self.dir_input = dir_input + self.best = None + self.gen = 0 + def post_evaluate(self, config, population, species, best_genome): + # If best genome has changed, watch it + if (not self.best or best_genome != self.best): + self.best = best_genome + species = self.population.species.get_species_id(self.best.key) + watch(config, self.step, self.gen, species, self.best, self.angle, self.dir_input) + self.gen += 1 + +## +# Data +# parse populations from checkpoints +## + +def load_checkpoints(folder): + print("Loading checkpoints from {0}...".format(folder)) + # load generations from file + checkpoints = [] + files = os.listdir(folder) + # progress bar vars + step = len(files)/46 + t = 0 + print('[', end='', flush=True) + for filename in files: + # load checkpoint and append to list + checkpoint = neat.Checkpointer.restore_checkpoint(os.path.join(folder,filename)) + checkpoints.append(checkpoint) + # update progress bar + t += 1 + if (t > step): + t -= step + print('.', end='', flush=True) + print(']') + # Sort checkpoints by generation id + checkpoints.sort(key = lambda g: g.generation) + return checkpoints + +## +# Plot Fitness +# Best and Average +## + +def plot_fitness(checkpoints, name): + gens = [c.generation for c in checkpoints] + bests = [c.best_genome.fitness for c in checkpoints] + avgs = [mean([f.fitness for _, f in c.population.items()]) for c in checkpoints] + + fig, ax = plt.subplots(figsize = (10,5)) + ax.set_title(name+" - Fitness over Generations") + ax.plot(gens, bests, color='blue', linewidth=1, label="Best") + ax.plot(gens, avgs, color='black', linewidth=1, label="Average") + ax.legend() + + ax.set_xlabel('Generation') + ax.set_ylabel('Fitness (Flight Time)') + + plt.tight_layout() + fig.savefig(name+'.png', format='png', dpi=300) + plt.show() + plt.close() + +## +# Plot Species +# Stackplot of species member count over generations +## + +def plot_species(checkpoints, name): + gens = [c.generation for c in checkpoints] + species = [c.species.species for c in checkpoints] + + max_species = max(max(id for id,_ in sps.items()) for sps in species)+1 + species = [[(len(sps[s].members) if (s in sps) else 0) for sps in species] for s in range(1,max_species)] + + fig, ax = plt.subplots(figsize = (10,5)) + ax.set_title(name+" - Species over Generations") + ax.stackplot(gens, species) + + ax.set_xlabel('Generation') + ax.set_ylabel('Number of Genomes') + + plt.tight_layout() + fig.savefig(name+'_species.png', format='png', dpi=300) + plt.show() + plt.close() + +## +# Plot Pareto 2D +# Scatter plot of 2 dimensional fitness, to evaluate pareto optimization +## + +def plot_pareto_2d(checkpoints, label0, label1, max0, max1, name, min=0, max=-1, invert=True): + fitnesses = [[f.fitness for _, f in c.population.items()] for c in checkpoints[min:max]] + bests = [c.best_genome.fitness for c in checkpoints[min:max]] + + fig, ax = plt.subplots(figsize = (10,5)) + ax.set_title(name+" - Solution Space") + + if (invert): + ax.set_xlabel(label1) + ax.set_ylabel(label0) + else: + ax.set_xlabel(label0) + ax.set_ylabel(label1) + + # scatter + for gen in fitnesses: + x = [f.values[0] if (f.values[0] < max0) else max0 for f in gen] + y = [f.values[1] if (f.values[1] < max1) else max1 for f in gen] + r = lambda: random.randint(0,255) + color = '#%02X%02X%02X' % (r(),r(),r()) + if (invert): + ax.scatter(y, x, s=3, c=color) + else: + ax.scatter(x, y, s=3, c=color) + # triangulation + tri = Delaunay(list(zip(y,x))) + for t in tri.simplices: + x = [gen[i].values[0] if (gen[i].values[0] < max0) else max0 for i in t] + y = [gen[i].values[1] if (gen[i].values[1] < max1) else max1 for i in t] + ax.fill(y,x,linewidth=0.2,c=color,alpha=0.05) + + x = [f.values[0] if (f.values[0] < max0) else max0 for f in bests] + y = [f.values[1] if (f.values[1] < max1) else max1 for f in bests] + ax.plot(y,x,linewidth=1,c='#000000',label="best genome") + + ax.legend() + + plt.tight_layout() + fig.savefig(name+'_pareto.png', format='png', dpi=300) + plt.show() + plt.close() + +## +# Watch +# watch a genome play the game +## +def watch(config, time_step, generation, species, genome, start_angle, dir_input = False): + # create a recurrent network + net = neat.nn.RecurrentNetwork.create(genome, config) + # create a network GUI to render the topology and info + ui = NeuralNetworkGUI(generation, genome, species, net) + # create a Game with frontend enabled, and the GUI above + game = Game(start_angle,True,ui) + # run the game until reset + while(True): + if (dir_input): + dir = [0.5-game.hoverboard.x, 0.5-game.hoverboard.y] + norm = math.sqrt(dir[0]**2+dir[1]**2) + # activate network + output = net.activate([game.hoverboard.velocity[0], game.hoverboard.velocity[1], game.hoverboard.ang_velocity, game.hoverboard.normal[0], game.hoverboard.normal[1], dir[0], dir[1]]) + else: + output = net.activate([game.hoverboard.velocity[0], game.hoverboard.velocity[1], game.hoverboard.ang_velocity, game.hoverboard.normal[0], game.hoverboard.normal[1]]) + # output to hoverboard thrust + game.hoverboard.set_thrust(output[0], output[1]) + # update game manually from time step + game.update(time_step) + # if game reseted, break + if (game.reset_flag): break + +## +# Main +## + +def main(): + # Parse CLI arguments + parser = argparse.ArgumentParser(description='Tool for visualizing the neat-python checkpoints playing the hoverboard game.') + parser.add_argument('angle', help="Starting angle of the platform") + parser.add_argument('experiment', help="Experiment prefix: (time,rundvnc), default: time", const='time', nargs='?') + parser.add_argument('-f', '--fastfwd', help="Fast forward the game preview (2x)", nargs='?', const=True, type=bool) + parser.add_argument('-p', '--just_plot', help="Don't watch the game, just plot", nargs='?', const=True, type=bool) + parser.add_argument('-l', '--watch_last', help="Watch the last game", nargs='?', const=True, type=bool) + args = parser.parse_args() + + # Store global parameters + global GAME_START_ANGLE + global FAST_FORWARD + global JUST_PLOT + global WATCH_LAST + GAME_START_ANGLE = float(args.angle) + FAST_FORWARD = bool(args.fastfwd) + JUST_PLOT = bool(args.just_plot) + WATCH_LAST = bool(args.watch_last) + + # Check experiment argument + global EXPERIMENT + if (args.experiment is not None): + EXPERIMENT = str(args.experiment) + if (EXPERIMENT not in ('time','timedist')): + print("ERROR: Invalid experiment '" + EXPERIMENT + "'") + return + + # load data + checkpoints = load_checkpoints('checkpoint-'+EXPERIMENT) + + # create neat config from file + cfg_file = {'time':'config-default', + 'timedist':'config-nsga2'}[EXPERIMENT] + repro = {'time':neat.DefaultReproduction, + 'timedist':neat.nsga2.NSGA2Reproduction}[EXPERIMENT] + config = neat.Config(neat.DefaultGenome, repro, neat.DefaultSpeciesSet, neat.DefaultStagnation, cfg_file) + + # run game for the best genome of each checkpoint + # if it's not the same as the last one + last_genome = None + for checkpoint in checkpoints: + # skip repeated genomes + if (checkpoint.best_genome.key != last_genome): + last_genome = checkpoint.best_genome.key + else: + continue + # get species id + species = checkpoint.species.get_species_id(checkpoint.best_genome.key) + # watch the genome play + if (not JUST_PLOT and not WATCH_LAST): + watch(config, GAME_TIME_STEP*(2 if FAST_FORWARD else 1), checkpoint.generation, species, checkpoint.best_genome, GAME_START_ANGLE, EXPERIMENT in ['timedist']) + + # watch the last game + if (WATCH_LAST): + watch(config, GAME_TIME_STEP*(2 if FAST_FORWARD else 1), checkpoint.generation, species, checkpoint.best_genome, GAME_START_ANGLE, EXPERIMENT in ['timedist']) + + # scientific plot + plot_fitness(checkpoints, EXPERIMENT) + plot_species(checkpoints, EXPERIMENT) + if (EXPERIMENT in ['timedist']): + plot_pareto_2d(checkpoints, 'Flight Time', 'Average Squared Distance from Center', 100, 0, EXPERIMENT) + +if __name__ == "__main__": + main() diff --git a/neat/__init__.py b/neat/__init__.py index 8f80630a..7954dec9 100644 --- a/neat/__init__.py +++ b/neat/__init__.py @@ -2,6 +2,7 @@ import neat.nn as nn import neat.ctrnn as ctrnn import neat.iznn as iznn +import neat.nsga2 as nsga2 import neat.distributed as distributed from neat.config import Config diff --git a/neat/checkpoint.py b/neat/checkpoint.py index 9a6e6e31..2464f3b7 100644 --- a/neat/checkpoint.py +++ b/neat/checkpoint.py @@ -43,7 +43,7 @@ def __init__(self, generation_interval=100, time_interval_seconds=300, def start_generation(self, generation): self.current_generation = generation - def end_generation(self, config, population, species_set): + def post_evaluate(self, config, population, species_set, best_genome): checkpoint_due = False if self.time_interval_seconds is not None: @@ -57,23 +57,23 @@ def end_generation(self, config, population, species_set): checkpoint_due = True if checkpoint_due: - self.save_checkpoint(config, population, species_set, self.current_generation) + self.save_checkpoint(config, population, species_set, best_genome, self.current_generation) self.last_generation_checkpoint = self.current_generation self.last_time_checkpoint = time.time() - def save_checkpoint(self, config, population, species_set, generation): + def save_checkpoint(self, config, population, species_set, best_genome, generation): """ Save the current simulation state. """ filename = '{0}{1}'.format(self.filename_prefix, generation) print("Saving checkpoint to {0}".format(filename)) with gzip.open(filename, 'w', compresslevel=5) as f: - data = (generation, config, population, species_set, random.getstate()) + data = (generation, config, population, species_set, best_genome, random.getstate()) pickle.dump(data, f, protocol=pickle.HIGHEST_PROTOCOL) @staticmethod - def restore_checkpoint(filename): + def restore_checkpoint(filename, config = None): """Resumes the simulation from a previous saved point.""" with gzip.open(filename) as f: - generation, config, population, species_set, rndstate = pickle.load(f) + generation, old_config, population, species_set, best_genome, rndstate = pickle.load(f) random.setstate(rndstate) - return Population(config, (population, species_set, generation)) + return Population(config if (config != None) else old_config, (population, species_set, best_genome, generation)) diff --git a/neat/nsga2/README.md b/neat/nsga2/README.md new file mode 100644 index 00000000..46031ebd --- /dev/null +++ b/neat/nsga2/README.md @@ -0,0 +1,47 @@ +# NSGA-II + NEAT + +[NSGA-II](https://www.iitk.ac.in/kangal/Deb_NSGA-II.pdf) (Non-Dominated Sorting Genetic Algorithm) is en Elitist Multiobjective Genetic Algorithm, designed to +efficiently sort and select individuals on a population based on multiple fitness values. + +## Overview + +This is an implementation of NSGA-II as a reproduction method for NEAT. +The `DefaultReproduction` implements a simple single-fitness species-wise tournament with a fixed, user-specified, elitism and survival ratio. +`NSGA2Reproduction` has a few additional steps for sorting the population based on multiple fitness values. The last population is always stored and compared with the new one, to ensure parameterless elitism. + +- Speciation is a novel concept to _NSGA-II_, so the design decision was to sort the merged population with no constraints to species, then pick the overall best genomes. This allows species to grow and shrink based on their elites. The tournament then offsprings the current size of each species. +This seems to keep species more stable over the generations, while also improving their mean convergence. However more studies are required to validate these results. + +- Stagnation is also a novel concept to _NSGA-II_, and the current design decision is to stagnate species of the child population before merging it with the parent one. This allows species that are doing fine to slowly stagnate, instead of going extinct in a single generation. It is still not clear how this impacts the overall behaviour of species. + +Below is an schematic of the reproduction method. +It's described following the [original article](https://www.iitk.ac.in/kangal/Deb_NSGA-II.pdf) Main Loop, on pages 185 and 186. + +![nsga-ii concept](https://raw.githubusercontent.com/hugoaboud/neat-python/a4c90ad777439482831a7c5c6067899e64805a92/neat/nsga2/nsga2neat.svg) + +- The current implementation does not differentiate the first generation, it runs the second procedure with an empty parent population, so the results are the same as the proposed first step; +- Please note that the sorting results on the picture might not be accurate; +- A small rectangle with the text "best_genome" indicates where the best genome is evaluated. Note that it must happen before tournament, but after merging populations, to ensure elitism; +- Stagnation is not illustrated on the schematic, but it's the first thing on the `sort()` method, it removes genomes from child population Q(t); + +You can find a working documented example in [/examples/hoverboard/](https://github.com/hugoaboud/neat-python/tree/master/examples/hoverboard). + +## Implementation Notes + +- In order to avoid unecessary changes to the neat-python library, a class named NSGA2Fitness was created. It overloads the operators used by the lib; +- In order to comply with the single fitness progress/threshold, the first fitness value is used for thresholding and when the fitness object is converted to a float (like in math_util.mean); +- In order to use the multiobjective crowded-comparison operator, fitness functions config should always be set to 'max'; +- Front ranks are negative, so picking the best becomes a maximization problem, the default behaviour of neat-python; + +# Implementation + +1. A _NSGA2Fitness_ class is used to store multiple fitness values for each genome during evaluation (eval_genomes); +2. After all child genomes _Q(t)_ are evaluated, _NSGA2Reproduction.sort()_ method is run by _Population_; +3. _sort()_ starts by removing stagnated genomes and species from the child population _Q(t)_; +4. _sort()_ then merges the child _Q(t)_ and parent _P(t)_ population (saved from the last generation), and sorts it in parento-fronts; inside each pareto-front, genomes are sorted in decreasing order with the crowding-distance operator; if two values from the same front share the same crowding-distance, the decision relies on the first fitness value; +5. The new parent population _P(t+1)_ is selected from the best fronts, and it's species are saved for later merging (in case they go extinct); +6. The best genome is picked up from the new parent population _P(t+1)_; +7. The first fitness value is used to check for fitness threshold and average fitness calculation; +8. _reproduce()_ is then called by _Population_, and the genome tournament (selection, crossover, mutation) creates the new child population _Q(t+1)_; +9. Tournament is done species-wise, for the reasons outlined by the NEAT algorithm (avoid genetic aberrations, preserve innovation); +10. The resulting population _Q(t+1)_ will be evaluated on the next iteration; diff --git a/neat/nsga2/__init__.py b/neat/nsga2/__init__.py new file mode 100644 index 00000000..2ca6a3a9 --- /dev/null +++ b/neat/nsga2/__init__.py @@ -0,0 +1,303 @@ +""" + + Implementation of NSGA-II as a reproduction method for NEAT. + More details on the README.md file. + + @autor: Hugo Aboud (@hugoaboud) + +""" +from __future__ import division + +import math +import random +from itertools import count +from operator import add + +from neat.config import ConfigParameter, DefaultClassConfig +from neat.math_util import mean +from neat.species import Species + +## +# NSGA-II Fitness +# Stores multiple fitness values +# Overloads operators allowing integration to unmodified neat-python +## + +class NSGA2Fitness: + def __init__(self, *values): + self.values = values + self.rank = 0 + self.dist = 0.0 + #self.score = 0.0 + def set(self, *values): + self.values = values + def add(self, *values): + self.values = list(map(add, self.values, values)) + + def dominates(self, other): + d = False + for a, b in zip(self.values, other.values): + if (a < b): return False + elif (a > b): d = True + return d + + # > + def __gt__(self, other): + # comparison of fitnesses on tournament, use crowded-comparison operator + # this is also used by max/min + if (isinstance(other,NSGA2Fitness)): + if (self.rank > other.rank): return True + elif (self.rank == other.rank and self.dist > other.dist): return True + return False + # stagnation.py initializes fitness as -sys.float_info.max + # it's the only place where the next line should be called + return self.rank > other + # >= + def __ge__(self, other): + # population.run() compares fitness to the fitness threshold for termination + # it's the only place where the next line should be called + # it's also the only place where score participates of evolution + # besides that, score is a value for reporting the general evolution + return self.values[0] >= other + # - + def __sub__(self, other): + # used only by reporting->neat.math_util to calculate fitness (score) variance + #return self.score - other + return self.values[0] - other + # float() + def __float__(self): + # used only by reporting->neat.math_util to calculate mean fitness (score) + #return self.score + return float(self.values[0]) + # str() + def __str__(self): + #return "rank:{0},score:{1},values:{2}".format(self.rank, self.score, self.values) + return "rank:{0},dist:{1},values:{2}".format(self.rank, self.dist, self.values) + +## +# NSGA-II Reproduction +# Implements "Non-Dominated Sorting" and "Crowding Distance Sorting" to reproduce the population +## + +class NSGA2Reproduction(DefaultClassConfig): + @classmethod + def parse_config(cls, param_dict): + + return DefaultClassConfig(param_dict, + [ConfigParameter('elitism', int, 0), + ConfigParameter('survival_threshold', float, 0.2), + ConfigParameter('min_species_size', int, 2)]) + + def __init__(self, config, reporters, stagnation): + # pylint: disable=super-init-not-called + self.reproduction_config = config + self.reporters = reporters + self.genome_indexer = count(1) + self.stagnation = stagnation + + # Parent population and species + # This population is mixed with the evaluated population in order to achieve elitism + self.parent_pop = [] + self.parent_species = {} + + # Parento-fronts of genomes (including population and parent population) + # These are created by the sort() method at the end of the fitness evaluation process + self.fronts = [] + + # new population, called by the population constructor + def create_new(self, genome_type, genome_config, num_genomes): + new_genomes = {} + for i in range(num_genomes): + key = next(self.genome_indexer) + g = genome_type(key) + g.configure_new(genome_config) + new_genomes[key] = g + return new_genomes + + # NSGA-II step 1: fast non-dominated sorting + # This >must< be called by the fitness function (aka eval_genomes) + # after a NSGA2Fitness was assigned to each genome + def sort(self, population, species, pop_size, generation): + + ## Stagnation happens only on the child Q(t) population, before merging, + # so the species have a chance to avoid stagnation if they're doing + # generally fine + # Filter out stagnated species genomes, collect the set of non-stagnated + remaining_species = {} # remaining species + for stag_sid, stag_s, stagnant in self.stagnation.update(species, generation): + # stagnant species: remove genomes from child population + if stagnant: + self.reporters.species_stagnant(stag_sid, stag_s) + population = {id:g for id,g in population.items() if g not in stag_s.members} + # non stagnant species: append species to parent species dictionary + else: + remaining_species[stag_sid] = stag_s + + # No genomes left. + if not remaining_species: + species.species = {} + return {} + + ## NSGA-II : step 1 : merge and sort + # Merge populations P(t)+Q(t) and sort by non-dominated fronts + child_pop = [g for _, g in population.items()] + self.parent_pop + + # Merge parent P(t) species and child (Qt) species, + # so all non-stagnated genomes are covered by species.species + species.species = remaining_species + for id, sp in self.parent_species.items(): + if (id in species.species): + species.species[id].members.update(sp.members) + else: + species.species[id] = sp + + ## Non-Dominated Sorting (of P(t)+Q(t)) + # algorithm data + S = {} # genomes dominated by key genome + n = {} # counter of genomes dominating key genome + F = [] # current dominance front + self.fronts = [] # clear dominance fronts + # calculate dominance of every genome to every other genome - O(MN²) + for p in range(len(child_pop)): + S[p] = [] + n[p] = 0 + for q in range(len(child_pop)): + if (p == q): continue + # p dominates q + if (child_pop[p].fitness.dominates(child_pop[q].fitness)): + S[p].append(q) + # q dominates p + elif (child_pop[q].fitness.dominates(child_pop[p].fitness)): + n[p] += 1 + # if genome is non-dominated, set rank and add to front + if (n[p] == 0): + child_pop[p].fitness.rank = 0 + F.append(p) + + # assemble dominance fronts - O(N²) + i = 0 # dominance front iterator + while (len(F) > 0): + # store front + self.fronts.append([child_pop[f] for f in F]) + # new dominance front + Q = [] + # for each genome in current front + for p in F: + # for each genome q dominated by p + for q in S[p]: + # decrease dominate counter of q + n[q] -= 1 + # if q reached new front + if n[q] == 0: + child_pop[q].fitness.rank = -(i+1) + Q.append(q) + # iterate front + i += 1 + F = Q + + ## NSGA-II : step 2 : pareto selection + # Create new parent population P(t+1) from the best fronts + # Sort each front by Crowding Distance, to be used on Tournament + self.parent_pop = [] + for front in self.fronts: + ## Calculate crowd-distance of fitnesses + # First set distance to zero + for genome in front: + genome.dist = 0 + # List of fitnesses to be used for distance calculation + fitnesses = [f.fitness for f in front] + # Iterate each fitness parameter (values) + for m in range(len(fitnesses[0].values)): + # Sort fitnesses by parameter + fitnesses.sort(key=lambda f: f.values[m]) + # Get scale for normalizing values + scale = (fitnesses[-1].values[m]-fitnesses[0].values[m]) + # Set edges distance to infinite, to ensure are picked by the next step + # This helps keeping the population diverse + fitnesses[0].dist = float('inf') + fitnesses[-1].dist = float('inf') + # Increment distance values for each fitness + if (scale > 0): + for i in range(1,len(fitnesses)-1): + fitnesses[i].dist += abs(fitnesses[i+1].values[0]-fitnesses[i-1].values[0])/scale + + ## Sort front by crowd distance + # In case distances are equal (mostly on 'inf' values), use the first value to sort + front.sort(key=lambda g: (g.fitness.dist,g.fitness.values[0]), reverse = True) + + ## Assemble new parent population P(t+1) + # front fits entirely on the parent population, just append it + if (len(self.parent_pop) + len(front) <= pop_size): + self.parent_pop += front + if (len(self.parent_pop) == pop_size): break + # front exceeds parent population, append only what's necessary to reach pop_size + else: + self.parent_pop += front[:pop_size-len(self.parent_pop)] + break + + + ## NSGA-II : post step 2 : Clean Species + # Remove the genomes that haven't passed the crowding-distance step + # (The ones stagnated are already not on this dict) + # Also rebuild SpeciesSet.genome_to_species + species.genome_to_species = {} + for _, sp in species.species.items(): + sp.members = {id:g for id,g in sp.members.items() if g in self.parent_pop} + # map genome to species + for id, g in sp.members.items(): + species.genome_to_species[id] = sp.key + # Remove empty species + species.species = {id:sp for id,sp in species.species.items() if len(sp.members) > 0} + + # self.parent_species should be a deepcopy of the species dictionary, + # in order to avoid being modified by the species.speciate() method + # the species in here are used to keep track of parent_genomes on next sort + self.parent_species = {} + for id, sp in species.species.items(): + self.parent_species[id] = Species(id, sp.created) + self.parent_species[id].members = dict(sp.members) + self.parent_species[id].representative = sp.representative + + ## NSGA-II : end : return parent population P(t+1) to be assigned to child population container Q(t+1) + # this container will be used on the Tournament at NSGA2Reproduction.reproduce() + # to create the real Q(t+1) population + return {g.key:g for g in self.parent_pop} + + # NSGA-II step 2: crowding distance sorting + # this is where NSGA-2 reproduces the population by the fitness rank + # calculated on step 1 + def reproduce(self, config, species, pop_size, generation): + + ## NSGA-II : step 3 : Tournament + # Disclaimer: this method uses no absolute fitness values + # The fitnesses are compared through the crowded-comparison operator + # fitness.values[0] is used for fitness threshold and reporting, but not in here + + ## Tournament + # Each species remains the same size (they grow and shrink based on pareto-fronts, on sort()) + # Only the best are used for mating + # Mating can be sexual or asexual + new_population = {} + for _, sp in species.species.items(): + # Sort species members by crowd distance + members = list(sp.members.values()) + members.sort(key=lambda g: g.fitness, reverse=True) + # Survival threshold: how many members should be used as parents + repro_cutoff = int(math.ceil(self.reproduction_config.survival_threshold * len(members))) + # Use at least two parents no matter what the threshold fraction result is. + members = members[:max(repro_cutoff, 2)] + # spawn the number of members on the species + spawn = len(sp.members) + for i in range(spawn): + # pick two random parents + parent_a = random.choice(members) + parent_b = random.choice(members) + # sexual reproduction + # if a == b, it's asexual reproduction + gid = next(self.genome_indexer) + child = config.genome_type(gid) + child.configure_crossover(parent_a, parent_b, config.genome_config) + child.mutate(config.genome_config) + new_population[gid] = child + + return new_population diff --git a/neat/nsga2/nsga2neat.svg b/neat/nsga2/nsga2neat.svg new file mode 100644 index 00000000..6f229ff5 --- /dev/null +++ b/neat/nsga2/nsga2neat.svg @@ -0,0 +1,14146 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + SPECIATION + CHILDPOPULATION(Q0) + + + + + + + + + + + genome 16 + genome 17 + genome 18 + genome 19 + genome 20 + genome 21 + genome 22 + genome 23 + genome 24 + + genome 25 + + genome 26 + + genome 27 + + genome 28 + + genome 29 + + genome 30 + + + + + genome 15 + + genome 13 + + genome 14 + + + + tournament + + SURIVAL: 50% + (or at least 2 genomes) + + + species 1 + species 2 + species 3 + + + genome 9 + + genome 12 + + genome 11 + + genome 10 + + genome 8 + + + + + genome 7 + + genome 3 + + genome 5 + + genome 1 + + genome 6 + + genome 2 + + genome 4 + + + + + + + + ARRANGEBY SPECIES + DEATH + REPRODUCTION + + + + + + CHILDPOPULATION + + + + + + + + + + + genome 16 + genome 17 + genome 18 + genome 19 + genome 20 + genome 21 + genome 22 + genome 23 + genome 24 + + genome 25 + + genome 26 + + genome 27 + + genome 28 + + genome 29 + + genome 30 + + MUTATION + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + genome 1 + genome 2 + genome 3 + genome 4 + genome 5 + genome 6 + genome 7 + genome 8 + genome 9 + + genome 10 + + genome 11 + + genome 12 + + genome 13 + + genome 14 + + genome 15 + + RANDOMPARENTPOPULATION (P0) + + EVALUATION + + + + + + + + + + + (1)(0.3,0.1) + (2)(0.2,0.2) + (3)(0.6,0.2) + (4)(0.3,0.1) + (5)(0.4,0.3) + (6)(0.3,0.3) + (7)(0.3,0.2) + (8)(0.2,0.4) + (9)(0.4,0.5) + + (10)(0.2,0.8) + + (11)(0.3,0.4) + + (12)(0.4,0.4) + + (13)(0.7,0.1) + + (14)(0.4,0.3) + + (15)(0.2,0.5) + + FITNESSEVALUATION + + + + + + + + + + + + + + + + + + species 1 + species 2 + species 3 + NON-DOMINATEDSORTING + + + + + + + + + + + (1)(0.3,0.1) + (2)(0.2,0.2) + (3)(0.6,0.2) + (4)(0.3,0.1) + (5)(0.4,0.3) + (6)(0.3,0.3) + (7)(0.3,0.2) + (8)(0.2,0.4) + (9)(0.4,0.5) + + (10)(0.2,0.8) + + (11)(0.3,0.4) + + (12)(0.4,0.4) + + (13)(0.7,0.1) + + (14)(0.4,0.3) + + (15)(0.2,0.5) + + + F1 + F2 + F3 + F4 + + NSGA-II + SORT + + genome 15 + + genome 14 + + + + + SORT BYFRONT + + + + + + + + + + + (4) F-4 + (2) F -3 + (3) F -3 + (1) F -4 + (5) F -2 + (6) F -2 + (7) F -3 + (8) F -3 + (9) F -1 + + (10) F -3 + + (11) F -2 + + (12) F -1 + + (13) F -4 + + (14) F -2 + + (15) F -3 + + + + + + + + + + + + + + + + genome 16 + genome 17 + genome 18 + genome 19 + genome 20 + genome 21 + genome 22 + genome 23 + genome 24 + + genome 25 + + genome 26 + + genome 27 + + genome 28 + + genome 29 + + genome 30 + + + + + + + + + genome 2 + + genome 7 + + genome 3 + + + + + + + + + + + + + + genome 9 + + genome 12 + + genome 11 + + genome 10 + + + + + + CHILDPOPULATIONQ(t) + EVALUATION + FITNESSEVALUATION + + + + + + + + + + + + + + + + + + species 1 + species 2 + species 3 + + + + + + + + + + + + + + + + + + + MERGEP(t) + Q(t) + + + PARENTPOPULATIONP(t) + + + + + + + + + + + (16)(0.3,0.2) + (17)(0.4,0.2) + (18)(0.6,0.3) + (19)(0.3,0.1) + (20)(0.4,0.2) + (21)(0.3,0.2) + (22)(0.4,0.2) + (23)(0.2,0.5) + (24)(0.3,0.5) + + (25)(0.2,0.9) + + (26)(0.4,0.4) + + (27)(0.4,0.3) + + (28)(0.8,0.1) + + (29)(0.4,0.5) + + (30)(0.1,0.6) + + NON-DOMINATEDSORTING + + + + + + + + + + + (1)(0.3,0.1) + (2)(0.2,0.2) + (3)(0.6,0.2) + (4)(0.3,0.1) + (5)(0.4,0.3) + (6)(0.3,0.3) + (7)(0.3,0.2) + (8)(0.2,0.4) + (9)(0.4,0.5) + + (10)(0.2,0.8) + + (11)(0.3,0.4) + + (12)(0.4,0.4) + + (13)(0.7,0.1) + + (14)(0.4,0.3) + + (15)(0.2,0.5) + + + + + + + + + + + (1)(0.3,0.1) + (2)(0.2,0.2) + (3)(0.6,0.2) + (4)(0.3,0.1) + (5)(0.4,0.3) + (6)(0.3,0.3) + (7)(0.3,0.2) + (8)(0.2,0.4) + (9)(0.4,0.5) + + (10)(0.2,0.8) + + (11)(0.3,0.4) + + (12)(0.4,0.4) + + (13)(0.7,0.1) + + (14)(0.4,0.3) + + (15)(0.2,0.5) + + + + + + + + + + (16)(0.3,0.2) + (17)(0.4,0.2) + (18)(0.6,0.3) + (19)(0.3,0.1) + (20)(0.4,0.2) + (21)(0.3,0.2) + (22)(0.4,0.2) + (23)(0.2,0.5) + (24)(0.3,0.5) + + (25)(0.2,0.9) + + (26)(0.4,0.4) + + (27)(0.4,0.3) + + (28)(0.8,0.1) + + (29)(0.4,0.5) + + (30)(0.1,0.6) + + + + + *SORTING RESULTS MIGHT NOT BE ACCURATE + F1 + F2 + F3 + F4 + + NSGA-II + MERGE AND SORT + PARETOFRONTS + + + + + + + + + + (1)(0.3,0.1) + (2)(0.2,0.2) + (3)(0.6,0.2) + (4)(0.3,0.1) + (5)(0.4,0.3) + (6)(0.3,0.3) + (7)(0.3,0.2) + (8)(0.2,0.4) + (9)(0.4,0.5) + + (10)(0.2,0.8) + + (11)(0.3,0.4) + + (12)(0.4,0.4) + + (13)(0.7,0.1) + + (14)(0.4,0.3) + + (15)(0.2,0.5) + + + + + + + + + + (16)(0.3,0.2) + (17)(0.4,0.2) + (18)(0.6,0.3) + (19)(0.3,0.1) + (20)(0.4,0.2) + (21)(0.3,0.2) + (22)(0.4,0.2) + (23)(0.2,0.5) + (24)(0.3,0.5) + + (25)(0.2,0.9) + + (26)(0.4,0.4) + + (27)(0.4,0.3) + + (28)(0.8,0.1) + + (29)(0.4,0.5) + + (30)(0.1,0.6) + + (25)(0.2,0.9) + + + + F1 + F2 + F3 + F4 + + NSGA-II + PARETO SELECTION + CROWDINGDISTANCESORTING + + + genome 9 + + genome 29 + + genome 12 + + genome 26 + + genome 18 + + genome 24 + + genome 5 + + genome 14 + + genome 27 + + genome 11 + + genome 6 + + genome 25 + + genome 10 + + genome 20 + + genome 2 + + + + + (2)(0.2,0.2) + + (10)(0.2,0.8) + + (20)(0.4,0.2) + + + + + + + + + + + + + + + + + + PARENTPOPULATIONP(t+1) + + GENERATION = 0 + GENERATION > 0 + + + + + + + + + + + + (16)(0.3,0.2) + (17)(0.4,0.2) + (18)(0.6,0.3) + (19)(0.3,0.1) + (20)(0.4,0.2) + (21)(0.3,0.2) + (22)(0.4,0.2) + (23)(0.2,0.5) + (24)(0.3,0.5) + + (25)(0.2,0.9) + + (26)(0.4,0.4) + + (27)(0.4,0.3) + + (28)(0.8,0.1) + + (29)(0.4,0.5) + + (30)(0.1,0.6) + + + + + + + + + + + + genome 16 + genome 17 + genome 18 + genome 19 + genome 20 + genome 21 + genome 22 + genome 23 + genome 24 + + genome 25 + + genome 26 + + genome 27 + + genome 28 + + genome 29 + + genome 30 + + *SORTING RESULTS MIGHT NOT BE ACCURATE + + + + + + + + + + + + + + + + + + + + genome 9 + + genome 29 + + genome 12 + + genome 26 + + genome 18 + + genome 24 + + genome 5 + + genome 14 + + genome 27 + + genome 11 + + genome 6 + + genome 25 + + genome 10 + + genome 20 + + genome 2 + + + + + + + + + + + + + + + + + + + + + genome 29 + + genome 14 + + genome 2 + tournament + + SURIVAL: 50% + (or at least 2 genomes) + + species 1 + species 2 + species 3 + + genome 10 + + genome 18 + + genome 5 + + genome 6 + + genome 20 + + + genome 24 + + genome 27 + + genome 9 + + genome 11 + + genome 12 + + genome 26 + + genome 25 + + + + + + + ARRANGEBY SPECIES + DEATH + REPRODUCTION + + + CHILDPOPULATIONQ(t+1) + + + + + + + + + + + genome 31 + genome 32 + genome 33 + genome 34 + genome 35 + genome 36 + genome 37 + genome 38 + genome 39 + + genome 40 + + genome 41 + + genome 42 + + genome 43 + + genome 44 + + genome 45 + + MUTATION + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + MERGE + + + + + + genome 14 + + genome 29 + + + CROWD DISTANCERANK + + + + + + + genome 18 + + genome 5 + + genome 6 + + + + + + + + + + + + + + genome 9 + + genome 12 + + genome 26 + + genome 24 + + + + + + + genome 9 + + genome 29 + + genome 12 + + genome 26 + + genome 18 + + genome 24 + + genome 5 + + genome 14 + + genome 27 + + genome 11 + + genome 6 + + genome 25 + + genome 10 + + genome 20 + + genome 2 + + + + + + + + + + + + + + + genome 31 + genome 32 + genome 33 + genome 34 + genome 35 + genome 36 + genome 37 + genome 38 + genome 39 + + genome 40 + + genome 41 + + genome 42 + + genome 43 + + genome 44 + + genome 45 + + + SPECIATION + + + + + + + + + + + genome 31 + genome 32 + genome 33 + genome 34 + genome 35 + genome 36 + genome 37 + genome 38 + genome 39 + + genome 40 + + genome 41 + + genome 42 + + genome 43 + + genome 44 + + genome 45 + + CHILDPOPULATIONQ(t+1) + + + eval_genomes() + NSGA2Reproduction.sort() + NSGA2Reproduction.reproduce() + + DefaultSpeciation.speciate() + + eval_genomes() + NSGA2Reproduction.sort() + NSGA2Reproduction.reproduce() + DefaultSpeciation.speciate() + + + best_genome + + + + best_genome + + + + genome 15 + + genome 13 + + genome 14 + + + + genome 9 + + genome 12 + + genome 11 + + genome 10 + + genome 8 + + + + genome 7 + + genome 3 + + genome 5 + + genome 1 + + genome 6 + + genome 2 + + genome 4 + + + diff --git a/neat/population.py b/neat/population.py index 1269433a..8ab81aaa 100644 --- a/neat/population.py +++ b/neat/population.py @@ -44,10 +44,10 @@ def __init__(self, config, initial_state=None): self.species = config.species_set_type(config.species_set_config, self.reporters) self.generation = 0 self.species.speciate(config, self.population, self.generation) + self.best_genome = None else: - self.population, self.species, self.generation = initial_state + self.population, self.species, self.best_genome, self.generation = initial_state - self.best_genome = None def add_reporter(self, reporter): self.reporters.add(reporter) @@ -87,6 +87,12 @@ def run(self, fitness_function, n=None): # Evaluate all genomes using the user-provided function. fitness_function(list(self.population.items()), self.config) + # Call sorting method of NSGA2Reproduction + # This is the only modification made to the main code, so the best + # genome is evaluated before tournament, to ensure elitism + if (callable(getattr(self.reproduction,'sort',None))): + self.population = self.reproduction.sort(self.population, self.species, self.config.pop_size, self.generation) + # Gather and report statistics. best = None for g in self.population.values(): diff --git a/neat/reporting.py b/neat/reporting.py index 58a581b9..c9338744 100644 --- a/neat/reporting.py +++ b/neat/reporting.py @@ -134,7 +134,7 @@ def post_evaluate(self, config, population, species, best_genome): best_species_id = species.get_species_id(best_genome.key) print('Population\'s average fitness: {0:3.5f} stdev: {1:3.5f}'.format(fit_mean, fit_std)) print( - 'Best fitness: {0:3.5f} - size: {1!r} - species {2} - id {3}'.format(best_genome.fitness, + 'Best fitness: {0:3.5f} - size: {1!r} - species {2} - id {3}'.format(float(best_genome.fitness), best_genome.size(), best_species_id, best_genome.key)) diff --git a/setup.py b/setup.py index c9870225..c91037a2 100644 --- a/setup.py +++ b/setup.py @@ -12,7 +12,7 @@ description='A NEAT (NeuroEvolution of Augmenting Topologies) implementation', long_description='Python implementation of NEAT (NeuroEvolution of Augmenting Topologies), a method ' + 'developed by Kenneth O. Stanley for evolving arbitrary neural networks.', - packages=['neat', 'neat/iznn', 'neat/nn', 'neat/ctrnn'], + packages=['neat', 'neat/iznn', 'neat/nn', 'neat/ctrnn', 'neat/nsga2'], classifiers=[ 'Development Status :: 4 - Beta', 'Intended Audience :: Developers',