Skip to content
This repository was archived by the owner on Sep 23, 2025. It is now read-only.

Conversation

@minmingzhu
Copy link
Contributor

@minmingzhu minmingzhu commented Jun 16, 2024

After trying several solutions, I finally used the tokenize_func from neural_chat:

  1. Using the transformer tokenizer with padding and attention_mask, and copying input_ids to labels.
  2. Using the transformer tokenizer with padding and attention_mask, and masking the input in the prompt.
  3. Using the transformer tokenizer with padding and attention_mask, and masking the response in the prompt.
  4. Using neural_chat's tokenize_func for padding, filling attention_mask and labels which masking the input in the prompt.
<style> </style>
Case Average ARC HellaSwag MMLU TruthfulQA Winogrande
Case 1 62.116 55.97 78.39 60.75 45.86 69.61
Case 2 58.99 54.18 72.85 60.23 42.18 65.51
Case 3 59.064 53.24 73.21 60.85 41.56 66.46
Case 4 63.03 53.67 81.47 60.27 43.02 76.72

@minmingzhu minmingzhu changed the title Fix evaluation [Finetune] Fix evaluation Jun 17, 2024
Signed-off-by: minmingzhu <minming.zhu@intel.com>
Signed-off-by: minmingzhu <minming.zhu@intel.com>
Signed-off-by: minmingzhu <minming.zhu@intel.com>
Signed-off-by: minmingzhu <minming.zhu@intel.com>
Signed-off-by: minmingzhu <minming.zhu@intel.com>
Signed-off-by: minmingzhu <minming.zhu@intel.com>
Signed-off-by: minmingzhu <minming.zhu@intel.com>
Signed-off-by: minmingzhu <minming.zhu@intel.com>
Signed-off-by: minmingzhu <minming.zhu@intel.com>
Signed-off-by: minmingzhu <minming.zhu@intel.com>
Signed-off-by: minmingzhu <minming.zhu@intel.com>
2. fix comments

Signed-off-by: minmingzhu <minming.zhu@intel.com>
Signed-off-by: minmingzhu <minming.zhu@intel.com>
Signed-off-by: minmingzhu <minming.zhu@intel.com>
@minmingzhu minmingzhu merged commit 0b44ac4 into intel:main Jun 28, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants