Type Error: '<=' not supported between instances of 'LoraConfig' and 'int'

[qy1026]

There is an Error when I implement unsloth. I also propose this issue in peft github repo.

huggingface/peft#1248

[danielhanchen]

@qy1026 Does this happen if you run the Alpaca example via Google Colab?

[qy1026]

use get_peft_model(model, r, target_modules, ...), DO NOT use get_peft_model(model, config)

[danielhanchen]

@qy1026 I'm sorry - are you saying the calling convention for get_peft_model has changed? I just ran Unsloth via Google Colab and it looks fine.

Am I doing something else wrong which causes ur code to break?

[qy1026]

I want to combine unsloth with a qlora fitune framework named firefly. The code of model and LoraConfig is as follows:

config = LoraConfig(
        r=args.lora_rank,
        lora_alpha=args.lora_alpha,
        target_modules=target_modules,
        lora_dropout=args.lora_dropout,
        bias="none",
        task_type="CAUSAL_LM",
    )
model = get_peft_model(model, config)

while the code in unsloth is :

# Do model patching and add fast LoRA weights
model = FastLlamaModel.get_peft_model(
    model,
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",],
    lora_alpha = 16,
    lora_dropout = 0, # Currently only supports dropout = 0
    bias = "none",    # Currently only supports bias = "none"
    use_gradient_checkpointing = True,
    random_state = 3407,
    max_seq_length = max_seq_length,
)

I wrongly pass the parameter config into the FastLlamaModel.get_peft_model, which caused the TypeError. The correct way is passing the necessary parameter seperately.

[danielhanchen]

@qy1026 Oh ok!!! Tell me how the finetuning goes! Hopefully it makes it superfast!! :) Again thanks for using Unsloth!