Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gguf: add script for converting falcon 180B huggingface safetensors model to gguf #3049

Closed
wants to merge 1 commit into from

Conversation

logicchains
Copy link
Contributor

It's just a slight modification of convert-falcon-hf-to-gguf.py ; not sure if we want to merge the two into one script somehow to avoid duplication.

@logicchains logicchains mentioned this pull request Sep 6, 2023
15 tasks
@Green-Sky
Copy link
Collaborator

@TheBloke please test and verify :)

@logicchains yea, having a separate file is not ideal. imo in an ideal wold we would have 1 convert.py and a subfolder with model specific code.

@Green-Sky Green-Sky added the 🦅. falcon label Sep 6, 2023
@akawrykow
Copy link
Contributor

The delta between this file and convert-falcon-hf-to-gguf.py is just a few lines. I would really recommend consolidating into convert-falcon-hf-to-gguf (which might be a lot less work than consolidating into convert.py). Some of those changes would most likely make future falcon variations work more easily out of the box.

For example, for these lines
image

you can take a look at e276e4b to see how we've tried to consolidate this before:

image

@ggerganov
Copy link
Owner

Thanks for the updated script - it does work. Here is a sample run on M2 Ultra:

falcon-180b-0.mp4

I agree with @akawrykow's suggestion to merge this into convert-falcon-hf-to-gguf.py instead of having a separate script.

@cebtenzzre cebtenzzre closed this in 48edda3 Oct 5, 2023
yusiwen pushed a commit to yusiwen/llama.cpp that referenced this pull request Oct 7, 2023
Also adds Falcon-180B support.
Closes ggerganov#3049

Co-authored-by: jb <jonathan.t.barnard@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🦅. falcon
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants