-
Notifications
You must be signed in to change notification settings - Fork 238
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Quantized Phi3: Features to add #277
Comments
I'm not too familiar with what you're talking about, but I have seen this as settings in https://github.com/oobabooga/text-generation-webui No clue if that's related. Presumably user config/overrides might be an alternative option for supplying that information (eg: JSON) or if it's usually a field supported in GGUF, those that want that could presumably patch the metadata directly? |
I am not familiar with this interface, but from my experience implementing LongRope, I think they are using a bit of a hack to get this to work because GGUF does not contain the necessary data. We actually load the full long and short factors for non-GGUF phi3 so it is flexible - and guaranteed to be correct unlike theirs - for any sequence length. I have looked at the GGUF metadata, and it does not contain these long/short factors. We could of course implement something like this. It would not be ideal, but it would provide a way for users to put the context length about where they need it. |
I don't know if it's actually used with GGUF models. The interface is simple enough to use, but some parts aren't very clear.
I would suggest deferring until actual feature request is raised 😅 You can just document the caveat with Phi3 quantized? Potentially add a |
The text was updated successfully, but these errors were encountered: