Skip to content

Using torch_qaic gradScaler and making lora_dropout=0.05 #320

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 19, 2025
Merged

Using torch_qaic gradScaler and making lora_dropout=0.05 #320

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion QEfficient/finetune/configs/peft_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ class lora_config:
target_modules: List[str] = field(default_factory=lambda: ["q_proj", "v_proj"])
bias = "none"
task_type: str = "CAUSAL_LM"
lora_dropout: float = 0.0
lora_dropout: float = 0.05
inference_mode: bool = False # should be False for finetuning


Expand Down
7 changes: 5 additions & 2 deletions QEfficient/finetune/utils/train_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
import torch_qaic.debug as qaic_debug # noqa: F401
import torch_qaic.profile as qaic_profile # noqa: F401
import torch_qaic.utils as qaic_utils # noqa: F401
from torch.qaic.amp import GradScaler as QAicGradScaler
except ImportError as e:
print(f"Warning: {e}. Moving ahead without these qaic modules.")

Expand Down Expand Up @@ -60,7 +61,6 @@ def train(

Returns: results dictionary containing average training and validation perplexity and loss
"""

train_prep = []
train_loss = []
val_prep = []
Expand Down Expand Up @@ -92,7 +92,10 @@ def train(
tensorboard_updates = SummaryWriter()

if train_config.grad_scaler:
scaler = GradScaler()
if device.startswith("qaic"):
scaler = QAicGradScaler()
else:
scaler = GradScaler()

loss_0_counter = torch.tensor([0]).to(device)

Expand Down
Loading