Welcome to reinforcement learning! If you're familiar with supervised learning and neural network training, you're about to discover a fundamentally different approach to machine learning.

{pause}

{.definition title="Reinforcement Learning"}

Instead of learning from labeled examples, an **agent** learns by **acting** in an **environment** and receiving **rewards**.

{pause up=rl-definition}

{pause}

Think of it like learning to play a game:

- You don't know the rules initially

- You try actions and see what happens

- Good moves get rewarded, bad moves get punished

- You gradually learn a strategy

{pause center=rl-definition}

{.block title="Supervised Learning"}

- Fixed dataset with input-output pairs

- Learn to minimize prediction error

- Single training phase

{pause}

{.block title="Reinforcement Learning"}

- Dynamic interaction with environment

- Learn to maximize cumulative reward

- Continuous learning from experience

{pause}

**No labeled data** - the agent must discover what actions are good through trial and error.

{pause down=differences}

{.definition title="Policy π(a|s)"}

The probability of taking action **a** in state **s**.

This is what your neural network learns to represent!

{pause up=policy-intro}

From Sutton & Barto:

{pause}

**Smooth changes** = **stable learning**

{pause}

{.example title="Policy Examples"}

- **Discrete actions**: Softmax over action preferences

- **Continuous actions**: Mean and variance of Gaussian distribution

{pause center=policy-intro}

{.definition title="Episode"}

A complete sequence of interactions from start to terminal state.

{.definition title="Return G_t"}

The total reward from time step t until the end of the episode:

{pause up=episodes}

**Maximize expected return** by learning a better policy.

{pause}

But how do we improve a policy that's represented by a neural network?

{pause down=episodes}

{.theorem title="The REINFORCE Algorithm"}

A **policy gradient** method that directly optimizes the policy parameters to maximize expected return.

{pause up=reinforce-intro}

We want to:

1. **Increase** the probability of actions that led to high returns

2. **Decrease** the probability of actions that led to low returns

{pause}

From Sutton & Barto:

{pause center=reinforce-intro}

The gradient of expected return with respect to policy parameters θ:

{pause}

This looks complicated, but REINFORCE gives us a simple way to estimate it!

{pause}

{.theorem title="REINFORCE Gradient Estimate"}

{pause up=gradient-theorem}

From Sutton & Barto:

{pause down=gradient-theorem}

{.block title="REINFORCE Algorithm"}

1. **Initialize** policy parameters θ randomly

2. **For each episode**:

- Generate episode following π(·|·,θ)

- For each step t in episode:

- Calculate return: $G_t = \sum_{k=t+1}^T R_k$

- Update: $\theta \leftarrow \theta + \alpha G_t \nabla_\theta \ln \pi(A_t|S_t,\theta)$

{pause up=algorithm}

From Sutton & Barto:

{pause}

This makes it an **unbiased** but **high variance** estimator.

{pause center=algorithm}

If your policy network outputs action probabilities, the gradient update becomes:

{pause up=implementation}

You'll typically:

1. Use **automatic differentiation** to compute ∇ ln π

2. **Collect episodes** in batches for stability

3. Apply **baseline subtraction** to reduce variance

{pause down=implementation}

REINFORCE can be **very noisy**. We can subtract a baseline b(s) from returns:

{pause up=baselines}

From Sutton & Barto:

{pause}

{pause}

{.example title="Common Baselines"}

- **Constant**: Average return over recent episodes

- **State-dependent**: Value function V(s) learned separately

{pause center=baselines}

{.block title="REINFORCE with Baseline Algorithm"}

1. **Initialize** policy parameters θ and baseline parameters w

2. **For each episode**:

- Generate episode following π(·|·,θ)

- For each step t:

- $G_t = \sum_{k=t+1}^T R_k$

- $\delta = G_t - b(S_t,w)$

- $\theta \leftarrow \theta + \alpha_\theta \delta \nabla_\theta \ln \pi(A_t|S_t,\theta)$

- $w \leftarrow w + \alpha_w \delta \nabla_w b(S_t,w)$

{pause up=reinforce-baseline}

The baseline is learned to predict expected returns, reducing variance without introducing bias.

{pause down=reinforce-baseline}

From Sutton & Barto:

{pause up=practical}

**Policy updates are more sensitive** - start with smaller learning rates for θ.

{pause}

From Sutton & Barto:

{pause}

REINFORCE with baseline is a simple actor-critic method:

- **Actor**: The policy π(a|s,θ)

- **Critic**: The baseline b(s,w)

{pause center=practical}

{.block title="Key Takeaways"}

✓ **RL learns from interaction**, not labeled data

✓ **REINFORCE optimizes policies directly** using policy gradients

✓ **Returns weight gradient updates** - high returns → strengthen action probabilities

✓ **Baselines reduce variance** without introducing bias

✓ **Actor-critic architectures** combine policy and value learning

{pause up=summary}

- Implement REINFORCE on a simple environment

- Experiment with different baseline functions

- Explore more advanced policy gradient methods (PPO, A3C)

- Consider trust region methods for more stable updates

{pause}

**You now have the foundation to start learning policies through interaction!**

{pause center=summary}

Sutton, R. S., & Barto, A. G. (2018). *Reinforcement learning: An introduction* (2nd ed.). MIT Press.