Support llama3.1-8b generation #947
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/947
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 1f309dd with merge base 7b4fa7c ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
build/model.py
Outdated
| high_freq_wavelen = old_context_len / high_freq_factor | ||
| new_freqs = [] | ||
| for freq in freqs: | ||
| wavelen = 2 * math.pi / freq |
There was a problem hiding this comment.
| wavelen = 2 * math.pi / freq | |
| wavelen = 2 * torch.pi / freq |
build/model.py
Outdated
| import math | ||
|
|
There was a problem hiding this comment.
Use pi from torch rather than math
| import math |
| @@ -40,6 +40,12 @@ | |||
| "distribution_path": "meta-llama/Meta-Llama-3-70B-Instruct", | |||
| "transformer_params_key": "Meta-Llama-3-70B" | |||
| }, | |||
| "meta-llama/Meta-Llama-3.1-8B-Instruct": { | |||
There was a problem hiding this comment.
nit: technically you can also add support for the pre-trained model (meta-llama/Meta-Llama-3.1-8B) and the llama guard (meta-llama/Llama-Guard-3-8B). Not a requirement though
There was a problem hiding this comment.
Probably want support for the pre-trained (non-Instruct) version to match with our support for Llama 3.0?
There was a problem hiding this comment.
oh yeah of course. This PR is focused on enabling llama3.1 in torchchat, so didn't cover all possible models. Will have another PR to handle that.
|
|
||
| low_freq_wavelen = old_context_len / low_freq_factor | ||
| high_freq_wavelen = old_context_len / high_freq_factor | ||
| new_freqs = [] |
There was a problem hiding this comment.
Hmm, it feels like one can write this logic in a much more pytorchy-style:
new_freqs = 360 / freq.rad2deg()
new_freqs[new_freqs > high_freq_wavelen ] /= scale_factor
new_freqs[new_freqs < low_freq_wavelen * scale_factor] = apply_smooth_here
|
Awesome! Please update the readme so people can discover it :) |
Llama3.1 8b now is supported in torchchat! 🎉
Local test:
