You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
withopen("test_with_response.json", "w") asjson_file:
json.dump(test_data, json_file, indent=4)
delllmllm=LLM.load("meta-llama/Meta-Llama-3-8B-Instruct", access_token="...")
defgenerate_model_scores(json_data, json_key):
scores= []
forentryintqdm(json_data, desc="Scoring entries"):
prompt= (
f"Given the input `{format_input(entry)}` "f"and correct output `{entry['output']}`, "f"score the model response `{entry[json_key]}`"f" on a scale from 0 to 100, where 100 is the best score. "f"Respond with the integer number only."
)
score=llm.generate(prompt, max_new_tokens=50)
try:
scores.append(int(score))
exceptValueError:
continuereturnscoresscores=generate_model_scores(json_data, "response")
print(f"\n{model}")
print(f"Number of scores: {len(scores)} of {len(json_data)}")
print(f"Average score: {sum(scores)/len(scores):.2f}\n")
OUT:
Scoring entries: 100%|██████████| 165/165 [00:30<00:00, 5.50it/s]
response_before
Number of scores: 161 of 165
Average score: 84.02
No description provided.
The text was updated successfully, but these errors were encountered: