-
Notifications
You must be signed in to change notification settings - Fork 789
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GPU support may not be enabled #62
Comments
llama.go:333: skipping accelerated runner because num_gpu=0 |
Is that on linux? Can you share more of your OS and machine specs. |
If you are using the Linux profile, you should add this to the
And ensure that you have |
Thank you. That works.
Steph van Schalkwyk
Principal, remcam llc, dba. Remcam Search Engines
Elastic Partner
+1.314.452.2896
***@***.***
Skype: svanschalkwyk
<http://linkedin.com/in/vanschalkwyk>
…On Sat, Oct 21, 2023 at 12:36 PM Matthieu Mérigot-Lombard < ***@***.***> wrote:
If you are using the Linux profile, I believe you should add this to the
llm service:
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
And ensure that you have nvidia-container-toolkit installed
—
Reply to this email directly, view it on GitHub
<#62 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABTCTDNMV4LP7RYORMN5SLYAQB2PAVCNFSM6AAAAAA6J2LBACVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZTHA3TENRYGA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Added it to the readme. |
"routes.go:634: Warning: GPU support may not enabled, check you have installed install GPU drivers: nvidia-smi command failed"
NVIDIA-SMI 535.86.10 Driver Version: 535.86.10 CUDA Version: 12.2
5.15.0-86-generic #96-Ubuntu
docker run --rm --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi:
NVIDIA-SMI 535.86.10 Driver Version: 535.86.10 CUDA Version: 12.2
The text was updated successfully, but these errors were encountered: