Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time


Q-Learning with a Flappy Bird simulator.

FlappyAI Demo


FlappyAI is an AI that uses simple Q-Learning trained on a custom Flappy Bird simulator. It is mainly created for learning purposes. If you are looking for Q-Learning libraries that are more efficient and does not mind the steep learning curve, try TensorFlow instead.


FlappyAI contains these main parts:

  • FlappyGame - FlappyBird simulator
  • Trainer - Q-Learning agent
  • ModelInterface - Interface for the model used by the agent
  • - Part where everything connects together


FlappyAI is written in Python. The required libraries include:

  • pygame - Required for the graphics
  • Standard Python2 libraries

Extending FlappyAI to Other Games

Simply create a new interface and inherits from ModelInterface for your simulator.

from qlearning import Trainer, ModelInterface

class MyInterface(ModelInterface):
    # interacts with your simulator

agent = Trainer(MyInterface())  # creates a new agent with your interface
agent.train()                   # starts the training

How to Run


python -h

for help. FlappyAI has three modes:

  • interactive - No AI involved; Only human input
  • train - Start the training process.
  • test - Let AI plays the game

FlappyAI automatically looks for qtable.p in the current directory. It will create a new one if it cannot find one. This file is used to store the trained Q-table (KNAWLEDGE!). The Q-table file included in the repository is already trained, but if you want to retrain the agent, simply remove or rename the file.

During the training mode, press Ctrl+C to stop and store the Q-table to the file.

Technical Details

State Space

Assuming the game size is 640x480. Parameters considered by the AI are the following:

  • Horizontal distance between bird and nearest pipe divided by 10 (i.e. 0 to 300/10 and 300+)
  • Vertical distance between bird and nearest pipe (-480/10 to 480/10)
  • Bird velocity (-10/5 to 20/5)

Resulting in a total of 43456 states.

The 2D space is discretized into 10x10 tiles, therefore the framerate of the AI must be reduced by a factor of 10 (i.e. game is running in 60fps and the agent will be running in 6fps.)

Q-Learning Background

Q-Learning is a simple reinforcement learning algorithm that has three parameters:

  • α - Learning rate
  • γ - Discount factor
  • ε - Exploration probability (only in ε-greddy Q-Learning)


  • Let S be the set of all states and A be the set of all actions.
  • We also define Q: S x A -> R, where R is the set of all real numbers.
  • Then the algorithm is as follow:
    • Q(s_t, a_t) <- Q(s_t, a_t) + α (r_t + γ maxOverAllActions(Q(s_(t+1), a)) - Q(s_t, a_t))
    • Continues until it converges (or converges within epsilon.)


Simple Q-Learning with a Flappy Bird simulator






No releases published


No packages published