AI Cube Soccer 3D

This is a 3D soccer simulation where two AI agents, built with PyTorch, learn to play against each other using Reinforcement Learning. The entire environment is created from scratch using the Ursina game engine.

Features

• 3D Soccer Environment: A fully implemented 3D soccer field with physics, built using the Ursina engine.
• Competing AI Agents: Two independent Deep Q-Learning (DQN) agents control the players, learning and adapting their strategies in real-time.
• Advanced Reinforcement Learning:
  • Deep Q-Network (DQN): The core learning algorithm for both agents.
  • Parameter Space Noise: Encourages exploration by adding noise directly to the neural network's weights, leading to more consistent and state-dependent exploration strategies.
  • Experience Replay & Target Networks: Standard techniques to stabilize DQN training.
  • Macro-Actions: Agents select from a predefined set of action sequences (e.g., "turn left for 15 frames") instead of primitive actions, simplifying the learning task.
• Curriculum Learning: The training starts with easier scenarios (e.g., a player starting with the ball) and gradually transitions to a standard, more difficult kickoff, helping the agents learn fundamental skills first.
• Real-time Visualization: If rendering is enabled, you can watch the agents play. Training progress (rewards, loss, noise decay) is plotted and saved automatically using Matplotlib.
• Headless Mode: Option to disable rendering for significantly faster training on servers or in the background.
• Highly Configurable: Almost every aspect of the game, physics, and AI can be tweaked via the config.py file.

How It Works

The Environment

The game takes place on a walled field with two goals. Each agent controls a cube-shaped "player". The simulation includes basic physics for player movement, friction, and ball interactions.

The AI Agents

Each agent is an independent DQN model that learns to maximize its expected future rewards.

State Representation

To make decisions, each agent receives a "state" vector containing crucial, player-centric information. All positional vectors are rotated relative to the agent's own orientation, meaning the agent always "sees" the world from its own perspective. The state includes:

• Normalized and rotated vectors to the ball, opponent, opponent's goal, and its own goal.
• Normalized and rotated velocity vectors for the player, the opponent, and the ball.
• Distance to the nearest wall in the direction the agent is facing.
• Normalized remaining game time.

Actions

Instead of simple, single-frame actions (like turn left), the agents choose from a set of macro-actions defined in config.py. A macro-action might be "accelerate for 5 frames" or "turn right for 30 frames". This helps the agent learn more meaningful behaviors and simplifies the decision-making process.

Reward Function

The agents are guided by a reward function that encourages skillful play. At each step, an agent receives:

• Positive rewards for:
  • Scoring a goal.
  • Kicking the ball.
  • Getting closer to the ball.
  • Facing the ball.
• Penalties for:
  • Conceding a goal.
  • Scoring an own goal (harsher penalty).
  • Being stationary for too long (to prevent inaction).

Learning Algorithm

The project uses a DQN with several modern enhancements:

1. Experience Replay: The agent stores its experiences (state, action, reward, next_state) in a replay buffer and samples random batches from it for training. This decorrelates experiences and stabilizes learning.
2. Target Network: A separate, slow-updating copy of the main network is used to calculate target Q-values, preventing the "moving target" problem.
3. Parameter Space Noise: Instead of epsilon-greedy exploration (random actions), noise is added directly to the weights of a "perturbed" network for action selection. This allows for more sophisticated, state-dependent exploration.
4. Macro-Action Discounting: The future rewards for a chosen macro-action are correctly discounted over the duration of that action.

Project Structure

soccer_ai/
├── models/             # Saved AI model checkpoints will be stored here
├── screenshots/         # Saved training plots will be stored here
├── ai.py               # Core DQN Agent, Replay Buffer, and PyTorch network
├── config.py           # Central configuration for game, physics, and AI
├── game.py             # Game entities (Player, Ball) and core logic (physics, rewards)
├── main.py             # Main application entry point, game loop management
├── managers.py         # Manages entities and AI agents (state creation, action selection)
├── ui.py               # Manages all UI elements (score, timer, etc.)
├── README.md           # You are here
└── requirements.txt    # Project dependencies

Installation

1. Clone the repository:

   git clone <repository-url>
   cd soccer_ai

2. Create and activate a virtual environment (recommended):

   python3 -m venv venv
   source venv/bin/activate
   # On Windows, use: venv\Scripts\activate

3. Install the dependencies:

   pip install -r requirements.txt

   Note: PyTorch installation can vary depending on your system (CPU/GPU). The requirements.txt file includes the CPU-only version. For other versions, please see the official PyTorch installation instructions.

How to Run

Standard Mode (with Rendering)

To run the simulation and watch the agents play, simply execute:

python main.py

A window will appear, and the simulation will start after you click the "Start Game" button. Training plots will be saved to the screenshots/ directory after each game.

Headless Mode (for faster training)

For pure training without the overhead of rendering, you can run the simulation in headless mode.

1. Open config.py.
2. Set SHOULD_RENDER: False.
3. Run the script:

   python main.py

The simulation will start immediately, printing progress to the console.

Configuration

The config.py file is the central hub for tuning the simulation. You can adjust:

• GAME_CONFIG: Game rules like timer duration, field size, and rendering options.
• PHYSICS_CONFIG: Gameplay physics like player speed, friction, and kick strength.
• ACTIONS & MACRO_ACTIONS: Define the primitive and macro-actions available to the agents.
• DQN_CONFIG: Hyperparameters for the AI, including network size, learning rate, noise settings, and reward/penalty values.
• CURRICULUM_CONFIG: Settings for the curriculum learning schedule.