Skip to content

Home

Jump to bottom
JD edited this page Oct 2, 2023 · 4 revisions

Project Wiki - Snake Game with DQN Agent

Table of Contents

  1. Introduction
  2. Getting Started
  3. How to Play
  4. Project Structure
  5. Training the DQN Agent
  6. Visualization and Logging
  7. Contributing
  8. License
  9. Code Explanation

1. Introduction

This project implements the classic Snake game with a Deep Q-Network (DQN) reinforcement learning agent. The DQN agent learns to play the game by interacting with the environment, making decisions, and updating its Q-values based on rewards. In this wiki, you'll find information on how to get started, play the game, understand the project structure, train the DQN agent, and contribute to the project.

2. Getting Started

Prerequisites

Before you get started, make sure you have the following prerequisites installed on your system:

  • Python 3.x
  • pip (Python package manager)

Installation

To install the required Python libraries, run the following command:

pip install numpy tensorflow tqdm colorama matplotlib psutil

3. How to Play

The Snake game can be played as follows:

  • The snake starts with a length of 5 segments.
  • The goal is to collect fruits (green cells) to increase the snake's length.
  • Avoid running into walls and the snake's own body.
  • The game ends if the snake collides with a wall or itself.

4. Project Structure

The project consists of the following key components:

  • DQNAgent.py: The DQN agent implementation for training and making decisions.
  • snakeGame.py: The Snake game simulator.
  • archiver.py: A class for archiving game statistics and generating plots.
  • logger.py: A logger class for recording game and agent information.
  • snakeDQN.py: The main script for training the DQN agent and running game episodes.
  • plots/: A directory for storing generated plots.
  • LICENSE: The project's open-source license file.

5. Training the DQN Agent

Training Parameters The DQN agent uses reinforcement learning to improve its performance over time. Key training parameters and settings can be customized in snakeDQN.py. These include:

  • EPISODES: The number of training episodes.
  • REPLAY_MEMORY_SIZE: The size of the replay memory.
  • UPDATE_TARGET_EVERY: The target network update frequency.
  • epsilon: Epsilon-greedy exploration starting value.
  • EPSILON_DECAY: The rate at which epsilon decays.
  • MIN_EPSILON: The minimum value where epsilon decay stops. Other training and game parameters.

Modifying Training

Feel free to modify these parameters to experiment with training dynamics and agent performance. Training episodes allow the agent to learn and improve its performance.

6. Visualization and Logging

The project includes utilities for visualizing and logging the game and agent performance:

The archiver.py class generates plots to visualize statistics, including average rewards, steps, and fruit counts. The logger.py class logs game and agent information, making it easy to track the agent's behavior during training. Plots are saved in the plots/ directory, and log information is displayed in the console.

7. Contributing

Contributions to this project are welcome! You can contribute by reporting issues, proposing enhancements, or submitting pull requests. Please review the project's contribution guidelines for details.

Reporting Issues

If you encounter bugs, issues, or unexpected behavior, please report them on the project's GitHub issues page. Provide detailed information about the problem and any relevant error messages.

Proposing Enhancements

If you have ideas for enhancements or new features, create enhancement proposals in the project's issues section. Discuss your proposals with the community.

Submitting Pull Requests

If you want to contribute code to the project, submit pull requests for review. Ensure that your code follows the project's coding guidelines and style.

8. License

This project is open-source and available under the MIT License. You are free to use, modify, and distribute this code as allowed by the license.

Have fun playing Snake and training your DQN agent!

Code Base Explanation

This section provides a detailed explanation of the code base for the Snake Game with DQN Agent project. It covers classes, functions, parameters, variables, and arguments used in the code.

DQNAgent.py

DQNAgent Class

  • Parameters:

    • REPLAY_MEMORY_SIZE (int): Size of the replay memory.
    • MIN_REPLAY_MEMORY_SIZE (int): Minimum number of steps in the memory to start training.
    • MINIBATCH_SIZE (int): Number of steps (samples) used for training.
    • UPDATE_TARGET_EVERY (int): Target network update frequency.
    • DISCOUNT (float): Discount factor/alpha.
    • WIDTH (int): Width of the game field.
    • HEIGHT (int): Height of the game field.
    • ACTION_SPACE_SIZE (int): Number of possible actions.
    • TF_VERBOSE (bool): TensorFlow verbose mode.

  • Functions:

    • get_qs(state): Get Q-values for the current state.

      • state (np.ndarray): The current state represented as a NumPy array.

    • update_replay_memory(step): Update the replay memory.

      • step (tuple): A tuple containing information for updating the replay memory.
        • current_state (np.ndarray): The current state represented as a NumPy array.
        • action (int): The action taken.
        • reward (float): The reward received for the action.
        • new_state (np.ndarray): The new state after the action.
        • done (bool): A boolean indicating if the episode is done.

    • train(done, step_count): Train the neural network.

      • done (bool): A boolean indicating if the episode is done.
      • step_count (int): The current step count.

snakeGame.py

snakeGame Class

  • Parameters:

    • WIDTH (int): Width of the game field.
    • HEIGHT (int): Height of the game field.
    • LENGTH (int): Initial length of the snake.
    • NUM_FRUIT (int): Number of fruits in the game.
    • CAN_PORT (bool): Flag to allow the snake to teleport.

  • Functions:

    • initGame(): Initialize the game and create the field.

    • update_field(): Update the game field.

    • update_snake(): Update the snake's position and check for collisions.

    • eat(): Check if the snake eats a fruit.

    • control(action): Control the snake's direction.

      • action (int): The action that controls the snake's direction.

archiver.py

Archiver Class

  • Parameters:

    • AGGREGATE_STATS_EVERY (int): Episodes used for averaging for plotting.
    • EXPERIMENT_NAME (str): Name of the experiment for archiving.

  • Functions:

    • appendLists(episode_reward, epsilon, step_count, fruit_counter): Append episode statistics.

      • episode_reward (float): The total reward obtained in an episode of the game.
      • epsilon (float): Epsilon value used in reinforcement learning algorithms.
      • step_count (int): The number of steps taken in the episode.
      • fruit_counter (int): The number of fruits eaten in each episode.

    • averageLists(): Calculate and store average statistics.

    • saveFig(): Generate and save plots.

logger.py

Logger Class

  • Functions:
    • log(episode, step, reward, reward_overall, action, direction, head, dead, epsilon, run_into_self, cause): Log game and agent information.
      • episode (int): The episode number of the log entry.
      • step (int): The step number or iteration in the training process.
      • reward (float): The reward received at a particular step in the episode.
      • reward_overall (float): The overall reward accumulated in the episode.
      • action (int): The action taken by the agent.
      • direction (list): The current direction of the snake's head.
      • head (list): The current position of the snake's head in the game.
      • dead (bool): A boolean indicating if the agent is dead.
      • epsilon (float): The exploration rate of the agent.
      • run_into_self (bool): A boolean indicating if the snake ran into itself.
      • cause (str): The cause of the snake's death.

snakeDQN.py

  • Variables:

    • ACTION_SPACE_SIZE (int): Number of possible actions.
    • WIDTH (int): Width of the game field.
    • HEIGHT (int): Height of the game field.
    • START_LENGTH (int): Starting length for the snake.
    • NUM_FRUIT (int): Number of apples spawned.
    • CAN_PORT (bool): Flag to allow the snake to come back from the opposite side.
    • EPISODES (int): Number of training episodes.
    • DISCOUNT (float): Discount factor/alpha.
    • REPLAY_MEMORY_SIZE (int): Size of the replay memory.
    • MIN_REPLAY_MEMORY_SIZE (int): Minimum number of steps in memory to start training.
    • MINIBATCH_SIZE (int): Number of steps/samples for training.
    • UPDATE_TARGET_EVERY (int): Terminal states (end of episodes).
    • AGGREGATE_STATS_EVERY (int): Episodes used for averaging for plotting.
    • LOG_EVERY_STEP (bool): Log into the console every step.
    • TF_VERBOSE (bool): TensorFlow verbose mode.
    • EXPERIMENT_NAME (str): Name of the experiment for archiving.

  • Functions:

    • main(): Main function to run game episodes and train the DQN agent.

Clone this wiki locally