Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Support Qwen2 models.
Only docs are updated, since the pipeline requirements are the same as Qwen1.5.
Pipeline Tests
2 LoRA
Known Issue
When do lora (or other peft tuning that uses
peft
library) first and saved model at a dir, say, A, and then do another finetuning work that also specifies the sameoutput_dir
A, pipeline will fail to update the model card, sinceQwen2ForCausalLM
doesn't have attribute.create_or_update_model_card()
. But this will not affect the model saving.Bug logi:
PeftTrainer
and save leads to a modelcard withlibrary_name = 'peft'
.output_dir
, sincehttps://github.com/huggingface/transformers/blob/bdf36dcd48106a4a0278ed7f3cc26cd65ab7b066/src/transformers/trainer.py#L4114,
os.path.exists(model_card_filepath)
isTrue
, and thenis_peft_library
isTrue
..create_or_update_model_card()
, whichQwen2ForCausalLM
doesn't have.We strongly recommend to use different
output_dir
for every finetuning work to avoid unexpected issues like the one above.