You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Using the latest unsloth I get this error when trying to do a lora finetune of Mistral 7B.
Inferencing works just fine btw.
馃Ε Unsloth: Will patch your computer to enable 2x faster free finetuning.
==((====))== Unsloth: Fast Mistral patching release 2024.5
\\ /| GPU: NVIDIA A2. Max memory: 14.542 GB. Platform = Linux.
O^O/ \_/ \ Pytorch: 2.3.0+cu121. CUDA = 8.6. CUDA Toolkit = 12.1.
\ / Bfloat16 = TRUE. Xformers = 0.0.26.post1. FA = True.
"-____-" Free Apache license: http://github.com/unslothai/unsloth
Unsloth 2024.5 patched 32 layers with 32 QKV layers, 32 O layers and 32 MLP layers.
Traceback (most recent call last):
File "/home/sravanth/unsloth/finetune_unsloth.py", line 66, in <module>
from unsloth import is_bfloat16_supported
ImportError: cannot import name 'is_bfloat16_supported' from 'unsloth' (/home/sravanth/unsloth/unsloth/lib/python3.10/site-packages/unsloth/__init__.py)
Here's the relevant code block. This code works fine in colab though. (JFYI my local install is done as per the github repo instructions. I did not simply copy the colab install cells.)
from unsloth import FastLanguageModel
import torch
max_seq_length = 8192 # Choose any! We auto support RoPE Scaling internally!
dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.
# 4bit pre quantized models we support for 4x faster downloading + no OOMs.
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "unsloth/mistral-7b-bnb-4bit", # Choose ANY! eg teknium/OpenHermes-2.5-Mistral-7B
max_seq_length = max_seq_length,
dtype = dtype,
load_in_4bit = load_in_4bit,
# token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf
)
model = FastLanguageModel.get_peft_model(
model,
r = 32, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",],
lora_alpha = 64,
lora_dropout = 0, # Supports any, but = 0 is optimized
bias = "none", # Supports any, but = "none" is optimized
# [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!
use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context
random_state = 3407,
use_rslora = False, # We support rank stabilized LoRA
loftq_config = None, # And LoftQ
)
from trl import SFTTrainer
from transformers import TrainingArguments
from unsloth import is_bfloat16_supported
trainer = SFTTrainer(
model = model,
tokenizer = tokenizer,
train_dataset = train_dataset,
eval_dataset = eval_dataset,
dataset_text_field = "text",
max_seq_length = max_seq_length,
dataset_num_proc = 2,
packing = False, # Can make training 5x faster for short sequences.
args = TrainingArguments(
per_device_train_batch_size = 1,
gradient_accumulation_steps = 8,
warmup_steps = 5,
num_train_epochs=2,
learning_rate = 2e-4,
fp16 = not is_bfloat16_supported(),
bf16 = is_bfloat16_supported(),
logging_steps = 1,
optim = "adamw_8bit",
weight_decay = 0.01,
lr_scheduler_type = "linear",
seed = 3407,
output_dir = "outputs",
fp16_full_eval = True,
per_device_eval_batch_size = 1,
eval_accumulation_steps = 8,
evaluation_strategy = "steps",
eval_steps = 1,
),
)
trainer_stats = trainer.train()
The text was updated successfully, but these errors were encountered:
@danielhanchen
Thanks. Deleted my whole virtual env and did everything from scratch, and it's working now. For some weir reason I also have to install packaging via pip otherwise it says can't find this module.
Using the latest unsloth I get this error when trying to do a lora finetune of Mistral 7B.
Inferencing works just fine btw.
Here's the relevant code block. This code works fine in colab though. (JFYI my local install is done as per the github repo instructions. I did not simply copy the colab install cells.)
The text was updated successfully, but these errors were encountered: