Skip to content

fix: Update lora configs for gemma4#1748

Merged
athitten merged 1 commit intomainfrom
athitten/gemma4_lora_fix
Apr 9, 2026
Merged

fix: Update lora configs for gemma4#1748
athitten merged 1 commit intomainfrom
athitten/gemma4_lora_fix

Conversation

@athitten
Copy link
Copy Markdown
Contributor

@athitten athitten commented Apr 9, 2026

What does this PR do ?

  • Follow-up to feat: Add lora recipes for gemma4 #1731. Increase the LR to 2e-4 and rank (dim) from 8 to 16 across all four Gemma4 PEFT configs (2B, 4B, 31B, 26B-A4B MoE). This is required to converge at a comparable rate and without noise as that of full fine-tuning since LoRa as ~100x fewer trainable parameters.

Loss curves with the final config:

Screenshot 2026-04-09 at 8 01 53 AM

Changelog

  • Add specific line by line info of high level changes in this PR.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?

If you haven't finished some of the above items you can still open "Draft" PR.

Additional Information

  • Related to # (issue)

Signed-off-by: Abhishree Thittenamane <athittenaman@cw-dfw-cs-001-login-02.cm.cluster>
@copy-pr-bot
Copy link
Copy Markdown

copy-pr-bot bot commented Apr 9, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@athitten athitten changed the title Update lora configs for gemma4 fix: Update lora configs for gemma4 Apr 9, 2026
@athitten athitten added docs-only With great power comes great responsibility. r0.4.0 Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge. labels Apr 9, 2026
@athitten
Copy link
Copy Markdown
Contributor Author

athitten commented Apr 9, 2026

/ok to test 710acda

@athitten athitten enabled auto-merge (squash) April 9, 2026 16:24
@athitten athitten merged commit f33d2d5 into main Apr 9, 2026
32 of 34 checks passed
@athitten athitten deleted the athitten/gemma4_lora_fix branch April 9, 2026 16:25
svcnvidia-nemo-ci pushed a commit that referenced this pull request Apr 9, 2026
Update lora configs for gemma4

Signed-off-by: Abhishree Thittenamane <athittenaman@cw-dfw-cs-001-login-02.cm.cluster>
Co-authored-by: Abhishree Thittenamane <athittenaman@cw-dfw-cs-001-login-02.cm.cluster>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
akoumpa pushed a commit that referenced this pull request Apr 10, 2026
fix: Update lora configs for gemma4 (#1748)

Update lora configs for gemma4

Signed-off-by: Abhishree Thittenamane <athittenaman@cw-dfw-cs-001-login-02.cm.cluster>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Abhishree Thittenamane <47577437+athitten@users.noreply.github.com>
Co-authored-by: Abhishree Thittenamane <athittenaman@cw-dfw-cs-001-login-02.cm.cluster>
akoumpa pushed a commit that referenced this pull request Apr 10, 2026
Update lora configs for gemma4

Signed-off-by: Abhishree Thittenamane <athittenaman@cw-dfw-cs-001-login-02.cm.cluster>
Co-authored-by: Abhishree Thittenamane <athittenaman@cw-dfw-cs-001-login-02.cm.cluster>
edjson pushed a commit to edjson/Automodel that referenced this pull request Apr 17, 2026
Update lora configs for gemma4

Signed-off-by: Abhishree Thittenamane <athittenaman@cw-dfw-cs-001-login-02.cm.cluster>
Co-authored-by: Abhishree Thittenamane <athittenaman@cw-dfw-cs-001-login-02.cm.cluster>
edjson pushed a commit to edjson/Automodel that referenced this pull request Apr 18, 2026
Update lora configs for gemma4

Signed-off-by: Abhishree Thittenamane <athittenaman@cw-dfw-cs-001-login-02.cm.cluster>
Co-authored-by: Abhishree Thittenamane <athittenaman@cw-dfw-cs-001-login-02.cm.cluster>
Signed-off-by: Edison <edisonggacc@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-only With great power comes great responsibility. r0.4.0 Auto-cherrypick to release branch. Apply before merge; cherrypick happens after merge.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants