[KL-divergence formula] Paper

[philhoonoh]

Hi, my name is Phil.
First of all, thx for the paper. I really like the idea of reducing variance using quantization!

It might be my misunderstanding but I got a question regarding KL divergence in the paper described below:
In the paper, the reward formula, is

Here it says
$KL(p_0\Vert p_{\theta })$
Since $p_0$ is the initial policy (Pretrained LM) and $p_{\theta }$ is the current policy (Model we are training right now),

$KL(p_{\theta }\Vert p_0)$ isn't this the right order?

I attacked the reference example below
reference

Fine-Tuning Language Models from Human Preferences