Skip to content

Conversation

@mfuntowicz
Copy link
Member

@mfuntowicz mfuntowicz commented May 22, 2025

This PR adds the Windows way of installing and running llama.cpp on such OSes.

llama.cpp is now available on WinGet and as easy installable as brew on Windows machines now.

PS C:\Users\momo-> winget install llama.cpp
PS C:\Users\momo-> # Load and run the model:
PS C:\Users\momo-> llama-cli -hf lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M
load_backend: loaded RPC backend from C:\Users\momo-\AppData\Local\Microsoft\WinGet\Packages\ggml.llamacpp_Microsoft.Winget.Source_8wekyb3d8bbwe\ggml-rpc.dll
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 3090 (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
[...]
== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to the AI.
 - To return control without starting a new line, end your input with '/'.
 - If you want to submit another line, end your input with '\'.
 - Not using system message. To change it, set a different value via -sys PROMPT


> Hey how are you?
I'm just a language model, I don't have feelings or emotions like humans do, but I'm functioning properly and ready to help with any questions or tasks you may have! How about you? How's your day going?

Copy link
Member

@julien-c julien-c left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could even make those options dependent on navigator.userAgent.includes("Windows") (when it's run in frontend code)

cc @gary149 on whether we want to do it or not.

In any case this can be in a later PR

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
@mfuntowicz
Copy link
Member Author

Failing tests look unrelated - ok to merge? @julien-c @pcuenca?

@mfuntowicz mfuntowicz merged commit b51d3f4 into main May 22, 2025
5 checks passed
@mfuntowicz mfuntowicz deleted the llamacpp-windows-snippet branch May 22, 2025 11:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants