-
Notifications
You must be signed in to change notification settings - Fork 6.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for Intel Arc GPUs #1590
Comments
also looking forward ;) PS: the intel IPEX looks not commonly supported - would be nice throug so fallback option would be to use the vulkan api as target 🤔 |
Hi, thanks so much for submitting your issue. At the moment we do not support inference using Intel's GPUs. I'll leave this issue open to track adding Intel support in the future. |
+1 for IPEX support Would it be possible to include oneAPI to support this? OpenCL is currently not working well with Intel GPUs. Vulkan may also be a decent option. |
It looks like llama.cpp now supports SYCL for Intel GPUs. Is Arc support now possible? |
Last Automatic1111 update 1.7.0 included IPEX and initial support for Intel Arc GPUs on Windows, maybe someone could have a look a see what they have done to make it possible. I know this is for Windows only, but is shows that it is possible to integrate it while on Linux it should be easier as Windows support came later. I'm aware that maybe WSL is another different beast, I remember having too much trouble installing Automatic1111 and accessing my Intel Arc GPU due to some limitation with the memory and privileges hardcoded into WSL |
Hey everyone, I made some progress on adding Intel Arc support to ollama: #2458 |
Thank you @felipeagc |
Support for SYCL/Intel GPUs would be quite interesting because:
|
Extremely eager to have support for Arc GPUs. Have an A380 idle in my home server ready to be put to use. As the above commenter said, probably the best price/performance GPU for this work load. I have an ultra layman and loose understanding of all this stuff, but have I correctly surmised that llama.cpp essentially already has Arc support, and it just needs to be implemented/merged into Ollama? And if that’s the case, are we probably in the final stretch? |
I too have an A380 sitting idle in my R520 anxiously waiting for Ollama to recognize it. Thank you all for the progress you have contributed to this. |
Is this now done with the merge of #3278 that has been released in v0.1.140? |
@kozuch not quite. It's close. If you build locally from source, it should work, but we haven't integrated it into our official builds yet. |
Looking forward to it! Let me know once it is available for Windows :) |
You are not branching the releases off main? Why was the #3278 change seen in v0.1.39...v0.1.40 changelist then? |
Is there a release schedule for this? |
How can I build it to enable Intel Arc?
|
Unfortunately users have reported crashing in the Intel GPU management library on some windows systems, so we've had to disable it temporarily until we figure out what's causing the crash. You can re-enable it by setting OLLAMA_INTEL_GPU=1 We don't have docs explaining how to build since it's not reliable yet. You can take a look at the gen_linux.sh and gen_windows.ps1 scripts here for some inspiration on the required tools. |
Quick update - the crash is fixed on main now, but we'll keep it behind the env var I mentioned above until we get #4876 merged and the resulting binaries validated on linux and windows with Arc GPUs. |
Sorry if it isn't appropriate to ask this here, but when do you think this will reach the docker image, and when might there be some documentation for that as well? |
Is there any possibility to manual / auto detect internal gpu size? it seems igpu is detected as a oneapi compute device
But it seems that it is not correctly detecting igpu memory size
I am guessing this is what prevent igpu from working? |
Are you able to do any inference at all on the Arc A380? I am showing it loading the model in GPU memory on my A380 but the processing is still happening on the CPU while the GPU sits idle. Jul 05 18:25:13 cyka-b ollama[578885]: 2024/07/05 18:25:13 routes.go:1064: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_G> NAME ID SIZE PROCESSOR UNTIL |
Is there any way to make ollama find the neo driver's libigdrcl.so library for opencl ? On my setup ollama always returns:
And then a bit later:
I reproduced the error with llama-cpp and it seems like if llama-cpp can only find the level-zero device and not the opencl one it will throw the exception. |
Looking forward :) Intel(R) Iris(R) Xe Graphics
Append some useful links: |
Out of curiosity, what is holding up this PR (#4876) making it to main? It looks like its passing all the relevant PR tests. |
The Windows driver for Intel is crashing with Ollama. |
Some works are being done at #5593 and on llama.cpp side by Intel people. Looks like they are actively working on it! |
Oh then that is very good news! |
Does this include support for the integrated GPU? For example, the Intel Core i9 14900k has an integrated GPU. When I enable the feature on Ubuntu Server 22.04 it crashes. I am curious to know if there are dependencies to have installed for this to work.
|
Any idea how this ticket will affect this effort? Is IPEX or Vulkan the better route to go for Intel GPUs? |
It seems as if IPEX is the only way at this time, and that's only if you use Ubuntu 22.04. It doesn't work at all for me on 24.04. Intel has released their own guidance here: https://www.intel.com/content/www/us/en/content-details/826081/running-ollama-with-open-webui-on-intel-hardware-platform.html Only follow this guide if you can babysit the system it is installed on at every reboot because this guide does not enable any automatic service startups like Ollama and OpenWebUI include/intend as default. Very clunky, if you can even get it to work. I'm almost to the point of discarding my Intel GPU's in favor of amd/nvidia because those simply work so much easier. |
Based on this chart, most all the Intel GPUs/iGPUs should support Vulkan 1.2+: https://www.intel.com/content/www/us/en/support/articles/000005524/graphics.html I am really hoping this is an easier implementation route, because I agree, IPEX seems pretty hard to use in its current form. Even if Vulkan is slower than IPEX, some Intel GPU support will be better than nothing. |
I am also running into the 0 memory available IGPU issue, borrowing configs from @MordragT in NixOS on Intel 185H chips. |
I was seeing exactly the same issue |
I am trying another route, I have build SR-IOV support for my ARC iGPU, and tested it successfully in Kube with plex. Once RAM arrives today, I will attempt to see if I can run OneAPI/IPEX-LLM from kubevirts to give to Ollama. My attempt yesterday showed that it offloaded to CPU, but I had no RAM left. I'll try to update this thread if I have any success. |
Hi, I am on Ubuntu 24.04 LTS and got it working by using their container image The only caveat was that the container was not able to detect my GPU. Digging deeper, I found that my kernel that came with the ubuntu - 6.8.0-40-generic was causing the issue. The workaround until it gets fixed was to set these environmental variables:
After that I got my GPU detected. Also, because it is docker, I don't need to worry about service startup as it is handled by docker. |
No description provided.
The text was updated successfully, but these errors were encountered: