Gridworld Explorer

An interactive visualization of Iterative Policy Evaluation based on Example 4.1 from Sutton & Barto's Reinforcement Learning: An Introduction.

🚀 View Live Demo

Overview

This application demonstrates how state-values () are updated in a 4×4 gridworld under an equiprobable random policy. Users can interactively step through the Bellman updates to see how the values converge over time.

Key Features

• Interactive Bellman Updates: Click any cell to see the specific calculation for that state's next value.

• Policy Visualization: Toggle "Show Policy Arrows" to see the greedy policy emerging as the value function improves.

• Real-time Convergence: Track the max error () and iteration count as the agent learns the distance to terminal states.

---

RL Concepts Explained

The Bellman Equation

The app uses the Bellman equation for iterative policy evaluation:

$$V_{k+1}(s) = \sum_{a} \pi(a|s) \sum_{s', r} p(s', r|s, a) [r + \gamma V_k(s')]$$

• Reward (): Every move that does not reach a terminal state incurs a reward of -1.
• Policy (): The agent has an equal 25% chance to move in any of the four cardinal directions.
• Terminal States: The top-left and bottom-right cells are goals; once reached, no further rewards are accumulated.

What the Values Mean

The negative numbers in each cell represent the expected number of steps to reach a terminal state. As the simulation iterates, you'll notice values further from the corners become more negative.

---

Built With

• Framework: React with TypeScript

• Build Tool: Vite

• Styling: Tailwind CSS

• Deployment: GitHub Pages via GitHub Actions

---

To help users run your Gridworld Explorer locally, you can add a dedicated section to your README.md. Since your project is built with Vite and TypeScript, the standard process involves installing Node.js dependencies and running the Vite development server.

Add the following block to your README.md:

---

Run Locally

Follow these steps to get the project running on your local machine:

1. Prerequisites

Ensure you have Node.js installed (version 18 or higher is recommended).

2. Clone the Repository

git clone https://github.com/InMatrix/rl_example_gridworld.git
cd rl_example_gridworld

3. Install Dependencies

This project uses npm to manage packages. Run the following command in your terminal:

npm install

4. Start the Development Server

Launch the Vite development server:

npm run dev

Once the server starts, you can view the app by navigating to http://localhost:5173 (or the port specified in your terminal) in your browser.

---