simple_lqr.md

Simple Regulator

Problem, State and Action Space

The simple regulator problem is a simple linear quadratic regulator problem with a single state. It is an MDP with a single real-valued state and a single real-valued action.

Transitions and Rewards

Transitions are linear Gaussian such that a successor state s' is drawn from the Gaussian distribution \mathcal{N}(s + a, 0.12). Rewards are quadratic, R(s, a) = −s^2, and do not depend on the action. The examples in this text use the initial state distribution \mathcal{N}(0.3, 0.12). �

Optimal Policies

Optimal finite-horizon policies cannot be derivied using the methods from section 7.8 of Algorithms for Decision Making. In this case, T_s = [1], T_a = [1], R_s = [−1], R_a = [0] and w is drawn from \mathcal{N}(0, 0.12). Applications of the Riccati equation require that R_a be negative definite, which it is not.

The optimal policy is \pi(s) = −s, resulting in a successor state distribution centered at the origin. In the policy gradient chapters we often learn parameterized policies of the form \pi_{\theta}(s) = \mathcal{N}(\theta_1 s, \theta_2^2). In such cases, the optimal parameterization for the simple regulator problem is \theta_1 = −1 and \theta_2 is asymptotically close to zero.

The optimal value function for the simple regulator problem is also centered about the origin, with reward decreasing quadratically:

$$\begin{aligned} \mathcal{U}(s) &= -s^2 + \frac{\gamma}{1 + \gamma} \mathbb{E}_{s \sim \mathcal{N}(0, 0.1^2)} \left[-s^2 \right] \\\ &\approx -s^2 - 0.010\frac{\gamma}{1 + \gamma} \end{aligned}$$