Skip to content
Compare
Choose a tag to compare
@simonw simonw released this 09 Dec 05:48

New mechanism for running GGUF files directly, using llm -m gguf. Example:

llm -m gguf \
  -o path una-cybertron-7b-v2-bf16.Q8_0.gguf \
  'Instruction: Five reasons to get a pet walrus
Response:'

This makes it much easier to try out new GGUF files, for example those released by TheBloke on Hugging Face. #26