Adding support for Llama2 70B LoRA finetuning

[kartikayk]

Context

As per title

Changelog

• Builder function + config

Test plan

• Trained for one epoch with the following loss

 

• Training Speed

 

• Memory

• generated output using gpt-fast

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/788

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit bda498d with merge base a7b507b ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 50.00000% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 27.53%. Comparing base (a878f3b) to head (0f0df12).
Report is 18 commits behind head on main.

Files	Patch %	Lines
torchtune/models/llama2/_model_builders.py	50.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #788      +/-   ##
==========================================
+ Coverage   27.48%   27.53%   +0.05%     
==========================================
  Files         144      143       -1     
  Lines        5966     5937      -29     
==========================================
- Hits         1640     1635       -5     
+ Misses       4326     4302      -24     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[joecummings]

What is this?

[kartikayk]

oh crap, not sure

[joecummings]

Merge when green