[WIP | Need Fix + Add Websocket] Add reflective-mcts

[SyHeee]

Simplified MCTS --> Reflective MCTS (R-MCTS)

• Selection: Instead of using UCT (Upper Confidence Bound for Trees), it uses GPT-4 to select the most promising child node based on:

Current trajectory
Available children nodes
Node statistics (visits and values)
The overall goal

• Expansion: This phase remains similar to BFS/DFS, expanding the selected node if it's not terminal and hasn't reached max depth.

• Simulation: Uses the existing trajectory scoring system to evaluate the current path, which provides:

Overall score
Efficiency score
Accuracy score
Robustness score
Detailed feedback

• Backpropagation: Instead of traditional value updates, it uses a reflection-based approach:

If the score is below 0.75, it uses GPT-4 to analyze the trajectory
The reflection suggests which step to backtrack to and why
It then backtracks to that step and continues exploration
Node statistics (visits and values) are still updated for future selection

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
visual-tree-search	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 5, 2025 5:10am

[TataKKKL]

@SyHeee thanks for the PR, but actually simple search agent only contains BFS and DFS, you will need to configure another MCTS agent for MCTS.

[SyHeee]

@SyHeee thanks for the PR, but actually simple search agent only contains BFS and DFS, you will need to configure another MCTS agent for MCTS.

Do you suggest create a new MCTS class? For the simplified version, as in my method, i didn't use UCT and explicit rollout for searching and selection, my design was to leverage current Node and state to do simulation and selecting among candidates. Hence I added the logic under simple_search_agent. Lemme know your thoughts.

[TataKKKL]

Hi Shiying, yes, I suggest you create another MCTS agent class, and you can copy paste some code from the simple tree search agent to the new class, later we can do some refactoring to have a base class and both MCTS agent and simple tree search agent inherit that, but we can do that later.
Since simple tree search itself means just the BFS and DFS, thanks!

[SyHeee]

Hi Shiying, yes, I suggest you create another MCTS agent class, and you can copy paste some code from the simple tree search agent to the new class, later we can do some refactoring to have a base class and both MCTS agent and simple tree search agent inherit that, but we can do that later. Since simple tree search itself means just the BFS and DFS, thanks!

Got it, sure I will create a new module and keep the overlapped methods.

[TataKKKL]

merged this PR: #57