Repository navigation
Potential issue in Section 2.4: p(r(s') | s, a) = p(s' | s, a) is not generally valid #38
thetumbled
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Dear author,
Thank you very much for this excellent book. I noticed a small point in Section 2.4 that may be mathematically non-rigorous.
In the current version, the text states:
I think the equality
is not generally valid.
If the reward is a deterministic function of the next state, i.e.
then for a given reward value$$r_0$$ , the correct probability should be
Therefore, for a particular next state$$s^\prime$$ , we do not generally have
unless the mapping
is injective, meaning that different next states always correspond to different reward values.
For example, if two different next states$$s_1^\prime$$ and $$s_2^\prime$$ produce the same reward,
then
contains probability mass from both$$s_1^\prime$$ and $$s_2^\prime$$ , and thus is not equal to
in general.
That said, the final Bellman expression in the book is still correct:
It can be derived more rigorously from the joint transition-reward form:
When the reward is fully determined by the next state,
the summation over$$r$$ collapses for each fixed $$s^\prime$$ , giving
So the final formula is correct, but the intermediate equality
seems to require an additional injectivity assumption and may be misleading as stated.
Thanks again for the great book!
All reactions