[Reproducing Results] on Alfworld

[ai-nikolai]

Dear Authors,

Thank you for the great work on introducing ReAct.

Since, the original model that you used text-davinci-002 is deprecated on openai the closest two alternatives are: gpt-3.5-turbo and davinci-002. The best performance we get on e.g. the first 10 is 0.3, while the reported results on the first 10 envs of Alfworld are 0.7.

Could you share the traces or advice, what your latest scores on this environment is? Or how to reproduce your score of 0.7. @ysymyth @john-b-yang @descrip

Thanks.

[ai-nikolai]

@ysymyth - any update on the above? Thank you.

[ysymyth]

Have you tried to run on more than 10 cases? 10 seems too noisy to tell anything.

[ai-nikolai]

@ysymyth Thanks a lot for coming back to me on this one.

Yes, we have run the results several times on the entire test set (i.e. 135 environments). We both used your code and our own implementation.

Did you do modifications beyond the ones present in your code (in this repo?):

• e.g. Correcting common syntax mistakes of the model (e.g. put object x in place y -> put object x in/on place y)?
• Sampling outputs again with increasing temperature?
• Anything else?