Add XLMRoberta in Embedding Train #10074
Conversation
|
Thanks for your contribution! |
| b = np.tril(np.ones([cur_len, cur_len]), 0) | ||
| input_mask_data[0, 0, offset : offset + cur_len, offset : offset + cur_len] = b | ||
| b = np.ones([cur_len]) | ||
| input_mask_data[0, offset : offset + cur_len] = b |
llm/run_embedding.py
Outdated
| @@ -248,6 +253,7 @@ def main(): | |||
| return_tensors="np", | |||
| return_attention_mask=not model_args.flash_mask, | |||
| pad_to_multiple_of=data_args.pad_to_multiple_of, | |||
| return_position_ids=False | |||
| if isinstance(model_config, XLMRobertaConfig): | ||
| model_class = XLMRobertaSentenceEmbedding | ||
| elif isinstance(model_config, Qwen2Config): | ||
| model_class = Qwen2SentenceEmbedding |
There was a problem hiding this comment.
后面可以考虑加一个 AutoModelForSentenceEmbedding
|
单测有问题 |
Codecov ReportAttention: Patch coverage is
❌ Your patch status has failed because the patch coverage (43.54%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## develop #10074 +/- ##
===========================================
- Coverage 52.11% 49.94% -2.18%
===========================================
Files 730 757 +27
Lines 116557 122583 +6026
===========================================
+ Hits 60744 61223 +479
- Misses 55813 61360 +5547 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Before submitting
testsfolder. If there are codecov issues, please add tests cases first.PR types
New features
PR changes
Models
Description
一、在Embedding训练中增加对XLMRoberta模型的支持，可支持bge-m3及系列模型的微调训练：
1.在XLMRoberta的modeling文件中增加相关模型；
2.调整训练脚本中的模型选择与初始化相关代码；
3.调整embedding dataset相关脚本中的数据构造代码；
4.其他参数文件支持等
二、修复了原XLMRoberta模型recompute开启不正常的问题