Some questions about APPS

[virt9]

hello! i have some problems when evaluating on APPS. firstly ,I cant find any one-shot examples on corresponding generation code like follows:
and i wander how should i use the locally download metrics without downloading "apps_metrics"?
thanks a lot if anyone have some ideas !!!

[loubnabnl]

Hi, I'm not sure I understand, can you provide the code you used to run the evaluation? and wdym by

without downloading "apps_metrics"

[athmanar]

In docuemntation this is given for APPS. However this is never used or I cannot see it in the code? How can I evaluate in single shot for APPS for non finetuned models?

2- Few-shot: for non finetuned models, we provide one example in the prompt for each call type (Standard Input and Call-Based format). We add the examples with an instruction before the prompt above:

one_shot_prompt = (
    "Implement answers to the following questions:\nQUESTION:\n"
    + examples["problem_type1"]
    + "\nUse Standard Input format\nANSWER:\n"
    + examples["solution_type1"]
    + "\nQUESTION:\n"
    + examples["problem_type2"]
    + "\nUse Call-Based format\nANSWER:\n"
    + examples["solution_type2"]
    + "\n"
    + prompt
)