Probably wrong MCTS implementation

[ChenDRAG]

muzero-pytorch/core/mcts.py

Line 105 in 4c008d0

for node in search_path:

As what I understand, we should reverse search_path and calculate value from bottom to up?

[ChenDRAG]

I checked https://github.com/YeWR/EfficientZero 's implementation of MCTS(based on c language), and am almost certain we should reverse search_path.
https://github.com/YeWR/EfficientZero/blob/a0c094818d750237d5aa14263a65a1e1e4f2bbcb/core/ctree/cnode.cpp#L289

[koulanurag]

Hi @ChenDRAG,

Thanks for finding this potential bug. Superficially, it does look that it should be reversed.

I would get back to this in a day or two to investigate its correctness.

Also, I am also happy to accept a PR on it.

[bAmpT]

The authors corrected the mistake of not reverseing the search_path in the v2 of the pseudocode. Another small error is that the target rewards seem to be misaligned, there is a thread on stackoverflow about that: https://stackoverflow.com/questions/60234530/is-the-reward-value-in-muzeros-pseudocode-misaligned