Paper, table2

[serotoninpm]

I am impressed with your research. Thank you for your good research.

But I have a question and would like to ask.

According to Table 2 of the paper, success and failure modes are divided.

1. what is the definition of success mode and failure mode?
2. if success mode is a successful case, it should not include false positives, because false positives are predicting something wrong as right.
3. ultimately, Hallucinated reasoning traces or facts are present in both success mode and failure mode. I wonder why?

Thanks!

[ysymyth]

Thanks @serotoninpm !

1. Success and failure are simply defined as whether the LLM is able to reach correct answers after the ReAct procedure (measured by the final response).
2. Therefore, the LM can reach correct answers for the wrong ReAct trajectory. We would like to distinguish such cases from those successful cases where the trajectories are also correct, therefore report "false positive" in the table.
3. Hallucination can happen anytime, both in success and failure cases (even the model hallucinates at some point, it may still be able to reach the correct answer for the wrong reason). Therefore in both modes we report the hallucination traces.

[serotoninpm]

Thank you for your detailed and kind explanation!