You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello! I'm running python -u src/gsm/run.py with "gpt-3.5-turbo" and has the following error. This is happening because "def solution():" not in entire_output in feedback.py.
Question 1: Has this error happened with Codex? I'm wondering if this is because ChatGPT is not always following input exemplar's format perfectly.
Question 2: Even with these errors, src/gsm/run.py keeps running. Should I just ignore these errors? I'm hoping to obtain results that is close to or better than gsm results in your paper.
1%|▊ | 8/1319 [03:04<8:42:28, 23.91s/it]
An error occurred: list index out of range. Traceback (most recent call last):
File "/home/ubuntu/code/hideodeo/self-refine/src/utils.py", line 39, in wrapper
return func(*args, **kwargs)
File "/home/ubuntu/code/hideodeo/self-refine/src/gsm/run.py", line 40, in iterative_gsm
fb_and_maybe_soln = task_feedback(solution=solution)
File "/home/ubuntu/code/hideodeo/self-refine/src/gsm/feedback.py", line 42, in __call__
solution = entire_output.split("def solution():")[1]
IndexError: list index out of range
. Left retries: 2.
The text was updated successfully, but these errors were encountered:
Question 3 (related to Question 1): If this error happened with Codex, how often it happened? Says you have 100 test instances, roughly how many instances do you get this error?
Hey @hideodeo, in general, codex was better at follow few-shot prompts than chatgpt, and threw these exceptions less frequently. Our implementation ignores these exceptions, but perhaps one could do better.
Hello! I'm running
python -u src/gsm/run.py
with "gpt-3.5-turbo" and has the following error. This is happening because "def solution():" not in entire_output in feedback.py.The text was updated successfully, but these errors were encountered: