-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add mteb evaluation #8538
add mteb evaluation #8538
Conversation
Thanks for your contribution! |
@@ -0,0 +1,97 @@ | |||
# MTEB基准评估 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
在原evaluation下加了mteb目录
## 模型评估 | ||
使用评估脚本`eval_mteb.py`: | ||
|
||
- `base_model_name_or_path`: 模型名称或路径 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
把这个README.md和主readme进行融合,维护主readme即可
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
已与主README融合,并已测试过
代码融合后,需要检查一下,原来的readme里面的内容是否还能跑通 |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## develop #8538 +/- ##
========================================
Coverage 53.97% 53.97%
========================================
Files 618 618
Lines 96827 96827
========================================
+ Hits 52258 52259 +1
+ Misses 44569 44568 -1 ☔ View full report in Codecov by Sentry. |
是一个大规模文本嵌入评测基准,包含了丰富的向量检索评估任务和数据集。 | ||
本仓库主要面向其中的中英文检索任务(Retrieval),并以SciFact数据集作为主要示例。 | ||
|
||
使用评估脚本`evaluation/mteb/eval_mteb.py`: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
PR types
New features
PR changes
Others
Description
Add the evaluation scripts for MTEB benchmark in the pipeline example "contrastive_training"