You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Seems to have something to do with task_prompt_formatting
Change the prompt as follows:
instruction="### Instruction: Summarize the following passage in 3 sentences.\n",
query=f"### Instruction: Summarize the following passage in 3 sentences.\n### Passage: {line['article']}\n### Summary: ",
Not sure if it has something to do with the evaluated model itself. When I use another finetuned model, the original code seems to be fine.
Maybe adjusting the task_prompt_formatting ourselves according to requirements is needed.
When running the evaluation of Sheared-LLaMA-1.3B and original LLaMA-7B on helm|summarization:cnn-dm, I get zero scores:
Output:
The text was updated successfully, but these errors were encountered: