Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement SuperHOT/interpolated RoPE support #378

Closed
philpax opened this issue Jul 17, 2023 · 1 comment · Fixed by #389
Closed

Implement SuperHOT/interpolated RoPE support #378

philpax opened this issue Jul 17, 2023 · 1 comment · Fixed by #389
Assignees
@LLukas22
Labels
issue:enhancement New feature or request model:llama LLaMA model

Comments

@philpax
Copy link
Collaborator

philpax commented Jul 17, 2023

Another llama.cpp feature that seems to have shrunk the paper-to-implementation pipeline to less than one week!

This allows for a much longer context (assuming you have the (V)RAM for it)

We can probably close out #77 if this is done.
@philpax philpax added issue:enhancement New feature or request model:llama LLaMA model labels Jul 17, 2023
@LLukas22
Copy link
Contributor

To do this we only need a new rope_scaling model parameter. Or am i missing something?

@LLukas22 LLukas22 self-assigned this Jul 26, 2023
@LLukas22 LLukas22 mentioned this issue Jul 26, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@LLukas22 LLukas22

Labels
issue:enhancement New feature or request model:llama LLaMA model
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Custom RoPE Scaling
2 participants
@philpax @LLukas22