Skip to content
This repository has been archived by the owner on Jun 24, 2024. It is now read-only.

Implement SuperHOT/interpolated RoPE support #378

Closed
philpax opened this issue Jul 17, 2023 · 1 comment · Fixed by #389
Closed

Implement SuperHOT/interpolated RoPE support #378

philpax opened this issue Jul 17, 2023 · 1 comment · Fixed by #389
Assignees
Labels
issue:enhancement New feature or request model:llama LLaMA model

Comments

@philpax
Copy link
Collaborator

philpax commented Jul 17, 2023

Another llama.cpp feature that seems to have shrunk the paper-to-implementation pipeline to less than one week!

This allows for a much longer context (assuming you have the (V)RAM for it)

We can probably close out #77 if this is done.

@philpax philpax added issue:enhancement New feature or request model:llama LLaMA model labels Jul 17, 2023
@LLukas22
Copy link
Contributor

To do this we only need a new rope_scaling model parameter. Or am i missing something?

@LLukas22 LLukas22 self-assigned this Jul 26, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
issue:enhancement New feature or request model:llama LLaMA model
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants