mixtral evaluation PR#8989
mixtral evaluation PR#8989ericharper merged 4 commits intoNVIDIA-NeMo:mainfrom Slyne:mixtral-as-judge-eval-pipeline2
Conversation
Co-authored-by: PannuMuthu <pmuthukumar@nvidia.com> Signed-off-by: Slyne Deng <slyned@nvidia.com>
|
@ethanhe42 |
Signed-off-by: Slyne Deng <slyned@nvidia.com>
|
@Slyne could you move doc to |
…-judge-eval-pipeline2
Signed-off-by: Slyne Deng <slyned@nvidia.com>
|
@ethanhe42 Done. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| if line: | ||
| try: | ||
| res = json.loads(line.decode("utf-8").split("data: ")[1]) | ||
| except: |
Check notice
Code scanning / CodeQL
Except block handles 'BaseException'
|
|
||
| def generate_prompt(args, answer_list): | ||
| f_q = open(os.path.expanduser(args.question_file)) | ||
| f_ans1 = open(os.path.expanduser(answer_list[0])) |
Check warning
Code scanning / CodeQL
File is not always closed
| def generate_prompt(args, answer_list): | ||
| f_q = open(os.path.expanduser(args.question_file)) | ||
| f_ans1 = open(os.path.expanduser(answer_list[0])) | ||
| f_ans2 = open(os.path.expanduser(answer_list[1])) |
Check warning
Code scanning / CodeQL
File is not always closed
| f_q = open(os.path.expanduser(args.question_file)) | ||
| f_ans1 = open(os.path.expanduser(answer_list[0])) | ||
| f_ans2 = open(os.path.expanduser(answer_list[1])) | ||
| rule_dict = json.load(open(os.path.expanduser(args.rule_file), 'r')) |
Check warning
Code scanning / CodeQL
File is not always closed
| rule_dict = json.load(open(os.path.expanduser(args.rule_file), 'r')) | ||
|
|
||
| if os.path.isfile(os.path.expanduser(args.output)): | ||
| cur_reviews = [json.loads(line) for line in open(os.path.expanduser(args.output))] |
Check warning
Code scanning / CodeQL
File is not always closed
| else: | ||
| cur_reviews = [] | ||
|
|
||
| review_file = open(f'{args.output}', 'a') |
Check warning
Code scanning / CodeQL
File is not always closed
|
|
||
| review_file = open(f'{args.output}', 'a') | ||
|
|
||
| context_list = [json.loads(line) for line in open(os.path.expanduser(args.context_file))] |
Check warning
Code scanning / CodeQL
File is not always closed
|
|
||
|
|
||
| def preprocess(args, response_file, model_name): | ||
| questions = [json.loads(q) for q in open(os.path.expanduser(args.question_file), "r")] |
Check warning
Code scanning / CodeQL
File is not always closed
|
|
||
| def preprocess(args, response_file, model_name): | ||
| questions = [json.loads(q) for q in open(os.path.expanduser(args.question_file), "r")] | ||
| responses = [json.loads(r) for r in open(os.path.expanduser(response_file), "r")] |
Check warning
Code scanning / CodeQL
File is not always closed
| answer_file = os.path.join(args.answers_dir, f'{base}_answer{ext}') | ||
| answers_file = os.path.expanduser(answer_file) | ||
| os.makedirs(os.path.dirname(answers_file), exist_ok=True) | ||
| ans_file = open(answers_file, "w") |
Check warning
Code scanning / CodeQL
File is not always closed
* add mixtral evaluation Co-authored-by: PannuMuthu <pmuthukumar@nvidia.com> Signed-off-by: Slyne Deng <slyned@nvidia.com> * fix format & remove unused import Signed-off-by: Slyne Deng <slyned@nvidia.com> * move mixtral as judge evaluation document Signed-off-by: Slyne Deng <slyned@nvidia.com> --------- Signed-off-by: Slyne Deng <slyned@nvidia.com> Co-authored-by: Slyne Deng <slyned@nvidia.com>
4 participants
Please refer to the previous PR:
#8988