[WIP] FIX Make Mixtral LoRA loading work#44478
[WIP] FIX Make Mixtral LoRA loading work#44478BenjaminBossan merged 11 commits intohuggingface:mainfrom
Conversation
Required fixes: - some code was using unordered data structures, making weight order random - adjust alpha to offset increased rank from fusion - import functions from PEFT if available
This can be used in PEFT to apply weight conversion there without having to either re-implement the whole weight conversion machinery or having to call transformer_model.load_adapter. We want to avoid the latter because there is a lot of PEFT custom logic to weight loading which would require adjusting for that case.
|
Note: The added test is a |
mark stuff that can be removed with peft 0.19
…san/transformers into peft-weight-conversion-fixes
Cyrilvallez
left a comment
Cyrilvallez
left a comment
There was a problem hiding this comment.
All good on my side! Thanks for fixing! We can bump our peft dependency once you have released on your end to simplify everything!
| MIN_PEFT_VERSION = "0.18.0" | ||
| IS_PEFT_GE_019 = version.parse(importlib.metadata.version("peft")) >= version.parse("0.19.0") |
There was a problem hiding this comment.
@BenjaminBossan you need to protect this, looks like it crashes if peft is not installed
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
Good catch @Cyrilvallez, I fixed this in the latest commit. Failing CI seems to be unrelated. |
3 participants
Required fixes:
See huggingface/peft#3083.