Skip to content

Using torch_qaic gradScaler and making lora_dropout=0.05 #320

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Mar 19, 2025
Merged

Using torch_qaic gradScaler and making lora_dropout=0.05 #320

merged 1 commit into from
Mar 19, 2025

Conversation

@quic-swatia
Copy link
Contributor

@quic-swatia quic-swatia commented Mar 18, 2025

  1. In case of finetuning on qaic, torch_qaic gradScaler will be used
  2. Moving back to lora_dropout = 0.05 on ML Framework team's ask.

@quic-swatia quic-swatia requested a review from vbaddi March 18, 2025 11:20
@quic-swatia quic-swatia self-assigned this Mar 18, 2025
@quic-swatia quic-swatia force-pushed the gradScaler branch 2 times, most recently from 49ed776 to ea3acc7 Compare March 18, 2025 20:07
vbaddi
vbaddi requested changes Mar 19, 2025
View reviewed changes
QEfficient/finetune/utils/train_utils.py Outdated
if train_config.grad_scaler:
scaler = GradScaler()
if device.startswith("qaic"):
scaler = qaic_GradScaler()
Copy link
Contributor

@vbaddi vbaddi Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here.

Copy link
Contributor Author

@quic-swatia quic-swatia Mar 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

1. In case of finetuning on qaic, torch_qaic gradScaler will be used.…
376c6a6 
… 2. Moving back to lora_dropout = 0.05 on ML Framework team's ask.

Signed-off-by: Swati Allabadi <quic-swatia@quicinc.com>
@quic-swatia quic-swatia merged commit 2f50ce3 into quic:main Mar 19, 2025
4 checks passed
ochougul pushed a commit that referenced this pull request Mar 20, 2025
@ochougul
Using torch_qaic gradScaler and making lora_dropout=0.05 (#320)
32651d5 
1. In case of finetuning on qaic, torch_qaic gradScaler will be used
2.  Moving back to lora_dropout = 0.05 on ML Framework team's ask.

Signed-off-by: Swati Allabadi <quic-swatia@quicinc.com>
Co-authored-by: Swati Allabadi <quic-swatia@quicinc.com>
quic-swatia pushed a commit that referenced this pull request Mar 20, 2025
Using torch_qaic gradScaler and making lora_dropout=0.05 (#320)
cd9a6b9 
1. In case of finetuning on qaic, torch_qaic gradScaler will be used
2.  Moving back to lora_dropout = 0.05 on ML Framework team's ask.

Signed-off-by: Swati Allabadi <quic-swatia@quicinc.com>
Co-authored-by: Swati Allabadi <quic-swatia@quicinc.com>
qcdipankar pushed a commit to qcdipankar/efficient-transformers that referenced this pull request Apr 1, 2025
@quic-swatia @qcdipankar
Using torch_qaic gradScaler and making lora_dropout=0.05 (quic#320)
ca30d2f 
1. In case of finetuning on qaic, torch_qaic gradScaler will be used
2.  Moving back to lora_dropout = 0.05 on ML Framework team's ask.

Signed-off-by: Swati Allabadi <quic-swatia@quicinc.com>
Co-authored-by: Swati Allabadi <quic-swatia@quicinc.com>
Signed-off-by: Dipankar Sarkar <quic_dipankar@quicinc.com>
quic-meetkuma pushed a commit to vbaddi/efficient-transformers that referenced this pull request Apr 21, 2025
@quic-meetkuma
Using torch_qaic gradScaler and making lora_dropout=0.05 (quic#320)
f41e65d 
1. In case of finetuning on qaic, torch_qaic gradScaler will be used
2.  Moving back to lora_dropout = 0.05 on ML Framework team's ask.

Signed-off-by: Swati Allabadi <quic-swatia@quicinc.com>
Co-authored-by: Swati Allabadi <quic-swatia@quicinc.com>
eplatero97 pushed a commit to eplatero97/efficient-transformers that referenced this pull request Apr 29, 2025
@eplatero97
Using torch_qaic gradScaler and making lora_dropout=0.05 (quic#320)
32be0bd 
1. In case of finetuning on qaic, torch_qaic gradScaler will be used
2.  Moving back to lora_dropout = 0.05 on ML Framework team's ask.

Signed-off-by: Swati Allabadi <quic-swatia@quicinc.com>
Co-authored-by: Swati Allabadi <quic-swatia@quicinc.com>
eplatero97 pushed a commit to eplatero97/efficient-transformers that referenced this pull request Apr 29, 2025
@eplatero97
Using torch_qaic gradScaler and making lora_dropout=0.05 (quic#320)
3895c92 
1. In case of finetuning on qaic, torch_qaic gradScaler will be used
2.  Moving back to lora_dropout = 0.05 on ML Framework team's ask.

Signed-off-by: Swati Allabadi <quic-swatia@quicinc.com>
Co-authored-by: Swati Allabadi <quic-swatia@quicinc.com>
eplatero97 pushed a commit to eplatero97/efficient-transformers that referenced this pull request Apr 29, 2025
Using torch_qaic gradScaler and making lora_dropout=0.05 (quic#320)
16eea88 
1. In case of finetuning on qaic, torch_qaic gradScaler will be used
2.  Moving back to lora_dropout = 0.05 on ML Framework team's ask.

Signed-off-by: Swati Allabadi <quic-swatia@quicinc.com>
Co-authored-by: Swati Allabadi <quic-swatia@quicinc.com>

Signed-off-by: eplatero <quic_eplatero@quicinc.com>
eplatero97 pushed a commit to eplatero97/efficient-transformers that referenced this pull request Apr 29, 2025
@eplatero97
Using torch_qaic gradScaler and making lora_dropout=0.05 (quic#320)
14e5046 
1. In case of finetuning on qaic, torch_qaic gradScaler will be used
2.  Moving back to lora_dropout = 0.05 on ML Framework team's ask.

Signed-off-by: Swati Allabadi <quic-swatia@quicinc.com>
Co-authored-by: Swati Allabadi <quic-swatia@quicinc.com>

Signed-off-by: eplatero <quic_eplatero@quicinc.com>
eplatero97 pushed a commit to eplatero97/efficient-transformers that referenced this pull request Apr 29, 2025
@eplatero97
Using torch_qaic gradScaler and making lora_dropout=0.05 (quic#320)
387a735 
1. In case of finetuning on qaic, torch_qaic gradScaler will be used
2.  Moving back to lora_dropout = 0.05 on ML Framework team's ask.

Signed-off-by: Swati Allabadi <quic-swatia@quicinc.com>
Co-authored-by: Swati Allabadi <quic-swatia@quicinc.com>

Signed-off-by: eplatero <quic_eplatero@quicinc.com>
eplatero97 pushed a commit to eplatero97/efficient-transformers that referenced this pull request Apr 29, 2025
@eplatero97
Using torch_qaic gradScaler and making lora_dropout=0.05 (quic#320)
916ce53 
1. In case of finetuning on qaic, torch_qaic gradScaler will be used
2.  Moving back to lora_dropout = 0.05 on ML Framework team's ask.

Signed-off-by: Swati Allabadi <quic-swatia@quicinc.com>
Co-authored-by: Swati Allabadi <quic-swatia@quicinc.com>

Signed-off-by: eplatero <quic_eplatero@quicinc.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vbaddi vbaddi vbaddi approved these changes

@quic-rishinr quic-rishinr Awaiting requested review from quic-rishinr quic-rishinr is a code owner

@ochougul ochougul Awaiting requested review from ochougul ochougul is a code owner

Assignees

@quic-swatia quic-swatia

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@quic-swatia @vbaddi