Q-Learning and Deep Reinforcement Learning

A comprehensive implementation of reinforcement learning algorithms, progressing from basic Q-learning with lookup tables to advanced Deep Q-Networks (DQN). This project demonstrates three different approaches to training an agent to navigate grid-based environments.

Table of Contents

• Overview
• Features
• Project Structure
• Installation
• Usage
  • Q-Learning with Gymnasium (MountainCar)
  • Q-Learning with Custom Grid Environment
  • Deep Q-Network (DQN)
  • Interactive Jupyter Notebook
• Environment Details
• Hyperparameters
• Algorithm Progression
• Results
• Requirements
• License

Overview

This repository contains three progressive implementations of reinforcement learning:

1. Basic Q-Learning - Uses Gymnasium's MountainCar environment with discretized state space
2. Custom Environment Q-Learning - Implements a grid-world with Blob entities (player, food, enemy)
3. Deep Q-Network - Neural network-based approach using convolutional layers for image-based observations

The main goal is to train an agent (player blob) to navigate a grid environment, catch food (+25 reward), avoid enemies (-300 penalty), and learn optimal movement policies through reinforcement learning.

Features

• Multiple Algorithms: Q-table, dictionary-based Q-learning, and Deep Q-Network
• Custom Environment: Grid-based Blob environment with configurable rewards and penalties
• Visualization: Real-time rendering using OpenCV and training metrics with matplotlib
• Epsilon-Greedy Exploration: Exploration-exploitation trade-off with epsilon decay
• Experience Replay: Replay memory mechanism for DQN training
• TensorBoard Integration: Custom logging for monitoring training progress
• Model Persistence: Save and load trained Q-tables and neural network models

Project Structure

q-learning/
├── q_learning1.py          # Q-Learning with Gymnasium MountainCar
├── q_learning2.py          # Q-Learning with custom Blob environment
├── q_learning3.py          # Deep Q-Network (DQN) implementation
├── helpers.py              # Blob class and BlobEnv environment
├── q_notebook.ipynb        # Interactive Jupyter notebook
├── requirements.txt        # Python dependencies
└── .gitignore             # Git ignore configuration

File Descriptions

File	Algorithm	State Representation	Input Type	Lines
q_learning1.py	Q-Table	Discrete (20×20)	Position/Velocity	71
q_learning2.py	Dictionary Q-Table	Relative Positions	Tuple	182
q_learning3.py	Deep Q-Network	CNN Features	10×10×3 RGB	202
helpers.py	Environment	-	-	178

Installation

Prerequisites

• Python 3.8 or higher
• pip package manager

Setup

1. Clone the repository:

git clone <repository-url>
cd q-learning

2. Install required dependencies:

pip install -r requirements.txt

3. Install additional dependencies (not in requirements.txt):

pip install gymnasium pillow opencv-python matplotlib tqdm

Usage

1. Q-Learning with Gymnasium (MountainCar)

Trains an agent to solve the MountainCar-v0 environment where the goal is to reach position 0.5 by building momentum.

python q_learning1.py

Features:

• 25,000 training episodes
• Discrete state space (20×20 bins)
• 3 actions: push left, no push, push right
• Epsilon decay: 0.9998 per episode
• Learning rate: 0.1
• Discount factor: 0.95

2. Q-Learning with Custom Grid Environment

Implements Q-learning with a custom Blob environment on a 20×20 grid.

python q_learning2.py

Features:

• Dictionary-based Q-table: {(dx_food, dy_food, dx_enemy, dy_enemy): [Q-values]}
• 4 diagonal movement actions
• Visual rendering every 3000 episodes
• Saves trained Q-table as pickle file
• Plots moving average of rewards

State Representation:

state = ((player.x - food.x, player.y - food.y),
         (player.x - enemy.x, player.y - enemy.y))

Rewards:

• Catch food: +25
• Hit enemy: -300
• Each move: -1

3. Deep Q-Network (DQN)

Advanced implementation using convolutional neural networks to process image observations.

python q_learning3.py

Features:

• CNN Architecture:
  • Conv2D (256 filters, 3×3)
  • Conv2D (256 filters, 3×3)
  • MaxPooling2D
  • Dropout (0.2)
  • Dense (64)
  • Output (9 actions)
• Experience Replay buffer (50,000 transitions)
• Target network updated every 5 episodes
• Batch size: 64
• TensorBoard logging
• Saves best models automatically

Network Input: 10×10×3 RGB images where:

• Player: RGB(255, 175, 0) - Orange
• Food: RGB(0, 255, 0) - Green
• Enemy: RGB(0, 0, 255) - Blue

4. Interactive Jupyter Notebook

Explore Q-learning interactively with visualizations.

jupyter notebook q_notebook.ipynb

Environment Details

BlobEnv (Custom Environment)

The custom Blob environment simulates a grid world with three entities:

Entities:

• Player (Orange): Agent controlled by the RL algorithm
• Food (Green): Target to reach for positive reward
• Enemy (Red): Obstacle to avoid for negative penalty

Action Space:

• 9 discrete actions (4 diagonal, 4 cardinal, 1 stay)
• Actions: {0: NE, 1: SW, 2: NW, 3: SE, 4: E, 5: W, 6: N, 7: S, 8: STAY}

Observation Space:

• Image mode: 10×10×3 RGB numpy array
• Vector mode: Tuple of relative positions (dx_food, dy_food, dx_enemy, dy_enemy)

Episode Termination:

• Agent catches food (success)
• Agent hits enemy (failure)
• 200 steps reached (timeout)

Hyperparameters

Q-Learning (q_learning2.py)

Parameter	Value	Description
LEARNING_RATE	0.1	Step size for Q-value updates
DISCOUNT	0.95	Future reward discount factor (γ)
epsilon	0.9 → 0.0	Exploration rate with decay
EPS_DECAY	0.9998	Epsilon multiplier per episode
HM_EPISODES	25,000	Total training episodes
FOOD_REWARD	25	Reward for catching food
ENEMY_PENALTY	300	Penalty for hitting enemy
MOVE_PENALTY	1	Cost per movement

Deep Q-Network (q_learning3.py)

Parameter	Value	Description
LEARNING_RATE	0.001	Adam optimizer learning rate
DISCOUNT	0.99	Future reward discount factor (γ)
REPLAY_MEMORY_SIZE	50,000	Experience buffer capacity
MIN_REPLAY_MEMORY_SIZE	1,000	Min experiences before training
MINIBATCH_SIZE	64	Batch size for training
UPDATE_TARGET_EVERY	5	Episodes between target network updates
EPSILON_DECAY	0.9975	Epsilon decay rate
MIN_EPSILON	0.001	Minimum exploration rate

Algorithm Progression

This repository demonstrates the evolution of reinforcement learning approaches:

1. Tabular Q-Learning (q_learning1.py)

• Pros: Simple, interpretable, guaranteed convergence for small state spaces
• Cons: Doesn't scale to large/continuous state spaces

2. Dictionary-based Q-Learning (q_learning2.py)

• Pros: Efficient for sparse state spaces, easy to implement
• Cons: Still limited to discrete states, no generalization

3. Deep Q-Network (q_learning3.py)

• Pros: Handles high-dimensional inputs, generalizes to unseen states
• Cons: More complex, requires careful tuning, less interpretable

Results

Training Progress

Q-Learning (q_learning2.py):

• Initial performance: Random exploration (~-300 to 0 reward)
• Mid-training: Agent learns to avoid enemy (~0 to +15 reward)
• Final performance: Consistently reaches food (+20 to +25 reward)

Deep Q-Network (q_learning3.py):

• Displays real-time metrics via TensorBoard
• Saves models with naming: {episode}-{max_reward}max-{avg_reward}avg-{min_reward}min-{timestamp}.model
• Track epsilon decay, average Q-values, and episode rewards

Visualization

Both implementations provide visual feedback:

• q_learning2.py: OpenCV window showing agent movement every 3000 episodes
• q_learning3.py: Periodic rendering of the environment
• Matplotlib plots: Moving average of episode rewards

Requirements

tensorflow >= 2.15.0
types-tensorflow >= 2.15.0.20240106
numpy >= 1.24.0
gymnasium
pillow
opencv-python
matplotlib
tqdm

Install all dependencies:

pip install tensorflow numpy gymnasium pillow opencv-python matplotlib tqdm

License

This project is provided as-is for educational purposes. Feel free to use and modify for learning reinforcement learning concepts.

---

Quick Start

# Install dependencies
pip install -r requirements.txt
pip install gymnasium pillow opencv-python matplotlib tqdm

# Run basic Q-learning
python q_learning2.py

# Run Deep Q-Network (requires more compute)
python q_learning3.py

# Explore interactively
jupyter notebook q_notebook.ipynb

Contributing

Contributions are welcome! Areas for improvement:

• Add more environments (Atari, custom mazes)
• Implement other algorithms (A3C, PPO, SAC)
• Improve network architectures
• Add unit tests
• Enhance visualization and logging

References

• Reinforcement Learning: An Introduction by Sutton & Barto
• Playing Atari with Deep Reinforcement Learning - Original DQN paper
• Gymnasium Documentation
• TensorFlow Documentation