[Reproducing Results] on Alfworld

[ai-nikolai]

Dear Authors,

Thank you for the great work on introducing ReAct.

Since, the original model that you used text-davinci-002 is deprecated on openai the closest two alternatives are: gpt-3.5-turbo and davinci-002. The best performance we get on e.g. the first 10 is 0.3, while the reported results on the first 10 envs of Alfworld are 0.7.

Could you share the traces or advice, what your latest scores on this environment is? Or how to reproduce your score of 0.7. @ysymyth @john-b-yang @descrip

Thanks.

[ai-nikolai]

@ysymyth - any update on the above? Thank you.

[ysymyth]

Have you tried to run on more than 10 cases? 10 seems too noisy to tell anything.

[ai-nikolai]

@ysymyth Thanks a lot for coming back to me on this one.

Yes, we have run the results several times on the entire test set (i.e. 135 environments). We both used your code and our own implementation.

Did you do modifications beyond the ones present in your code (in this repo?):

• e.g. Correcting common syntax mistakes of the model (e.g. put object x in place y -> put object x in/on place y)?
• Sampling outputs again with increasing temperature?
• Anything else?

[okhat]

Yeah I can confirm that newer GPT-3.5 (both latest & instruct) do appear to perform really poorly, around 30%, when running a close clone of the notebook of alfworld.

I would have just said that GPT-3.5 got much worse on this task or needs new prompts, but I see a recent paper that uses gpt-3.5-instruct with decent ReAct results (54%) here: https://arxiv.org/pdf/2405.17402

So @ai-nikolai and I might be missing something.

[ysymyth]

i think reproducibility will get harder and harder ... close for now but feel free to reopen