Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CUDA backend #2310

Merged
merged 52 commits into from May 15, 2024
Merged

CUDA backend #2310

merged 52 commits into from May 15, 2024

Conversation

cebtenzzre
Copy link
Member

@cebtenzzre cebtenzzre commented May 6, 2024 •
edited

This PR adds opt-in CUDA support in the GPT4All UI and python bindings using the llama.cpp CUDA backend.

CUDA-enabled devices will appear as e.g. "CUDA: Tesla P40" on supported platforms, alongside their Vulkan counterparts. When one is selected, the CUDA backend will be used instead of the Kompute backend.

When CUDA is not available (e.g. a compatible driver or GPU is not installed), CUDA devices simply do not appear. OOM will cause fallback to CPU, just like with Kompute.

The CUDA runtime libraries are installed to the lib/ directory on Linux and Windows (cudart and cublas). Care is taken to make sure the driver component of CUDA, libnvcuda.dll/libcuda.so, is not installed, as it should come with graphics driver.

Other changes:

  • Scaffolding for the Vulkan and ROCm backends (both have serious limitations atm)
  • llama.cpp updated to commit ggerganov/llama.cpp@83330d8cd from May 8th
  • Device selection is now meaningful on macOS: "Metal" (new option) and "Auto" do the same thing, "CPU" now uses the CPU backend instead of Metal

While I was working on packaging the CUDA libraries, I cleaned up a lot of junk that was installed but not needed:

  • DLL import libraries on Windows
  • Static libraries, including Kompute and fmtlib
  • Various headers and cmake scripts from Kompute, Vulkan itself, and fmtlib
  • An extra copy of the llmodel lib (in bin/ on Linux, in lib/ on Windows)

@cebtenzzre
remove lingering references to SOM license
dc03f81 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
backend: bring some upstream changes into llama.cpp.cmake
c482bae 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
backend: update llama.cpp and list of supported models
1684f7b 
Kompute is no longer working.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
backend: bring more upstream changes into llama.cpp.cmake
dc8cd8c 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
llamamodel: remove dependency on internal llama_token_to_piece
75d05c5 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
backend: initial port to Occam's Vulkan backend
1ebd3cf 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
vulkan: improve the implementation
9f92731 
General:
- Proper implementation of gpuDeviceName()
- Make usingGPUDevice() consistent with Kompute impl
- Disable multi-GPU when selecting a specific device (currently: always)

For the bindings:
- Abort instead of segfaulting if multiple LLMs are loaded
- Implement GPU device selection by name/vendor

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
backend: also build CUDA backend by default
ce2164e 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
backend: make CUDA build useful
011935e 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
backend: add option to build ROCm backend
9e457bf 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
cuda: implement device enumeration
c47900e 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
cmake: don't build CPU variant on Linux/Windows
fef041c 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
llmodel: add a backend field to LLModel::GPUDevice
bbe5cc0 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
llmodel: select a backend, not a build variant
dc334ae 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
llmodel: list GPU devices from all backends
b54151f 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
python: implement selectable GPU backend
d4feaeb 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
chat: implement basic UI backend selection
1a10587 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
cmake: set RUNPATH of llamamodel-mainline-cuda correctly
a9ffe5a 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
backend: fix kompute-avxonly build
d30d5b2 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
cuda: fix dependency bundling on Windows
16170f4 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
ci: install CUDA toolkit
2417105 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
cuda: ignore libcuda.so.* on Linux
9e8f7c3 
This file is part of the graphics driver and should not be bundled with
GPT4All.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
ci: install additional deps to make linuxdeployqt happy
9e54277 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
cmake: do not ship static llama library
335b129 
llama.cpp itself is unconditionally built as a static library.
Installing it with the GUI is pointless.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
llama.cpp: do not install kompute or fmt
b14992c 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
cmake: let linuxdeployqt handle CUDA deps on Linux
2b5d6d3 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
python: fix documentation for device parameter
553fc89 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
cmake: fix chat install if built without Kompute or CUDA
035ea59 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
cmake: clearly indicate how to build without CUDA/Kompute on failure
85b9e2f 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
llmodel: fix compile errors
c9b6732 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
metal: copy default.metallib, not ggml-metal.metal
41ce930 
This makes Metal work again for the python bindings and when using an installed
version of the chat UI.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre cebtenzzre mentioned this pull request May 9, 2024
Open
@cebtenzzre
llamamodel: update model whitelist
14588bc 
- Remove dbrx (never tested), mamba (not supported by CUDA), persimmon
  (CUDA generates garbage), and plamo (fails an assertion)
- Add internlm2 and starcoder2

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre cebtenzzre linked an issue May 10, 2024 that may be closed by this pull request
[Feature] Add support for StarCoder2 model architecture #2100
Closed
@cebtenzzre
build_and_run: mention compiler and CUDA
21bc8c8 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
settings: prefix vk devices with "Vulkan: ", update old names
0cbca27 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
kompute: fix device name leaks
8dbe93d 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre cebtenzzre self-assigned this May 15, 2024
@manyoso
Copy link
Collaborator

manyoso commented May 15, 2024

Can you create an offline installer with this change for linux for testing please?

manyoso
manyoso requested changes May 15, 2024
View reviewed changes
gpt4all-backend/llamamodel.cpp Show resolved Hide resolved
gpt4all-backend/llamamodel.cpp Show resolved Hide resolved
gpt4all-chat/chatllm.cpp Show resolved Hide resolved
@cebtenzzre
chat: bump version to 2.8.0
dbf38b2 
This will be a big release, so increment the minor version.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
Merge branch 'main' into add-cuda-support
1c09b6d
@cebtenzzre
python: update README to reflect CUDA build dependency
c7f8c93 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre
python: bump version for CUDA support
5875d83 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre cebtenzzre merged commit d2a99d9 into main May 15, 2024
6 of 19 checks passed
@cebtenzzre cebtenzzre deleted the add-cuda-support branch May 15, 2024 19:27
cebtenzzre added a commit that referenced this pull request May 15, 2024
@cebtenzzre
cmake: fix Metal build after #2310
7c90f2a 
I don't understand why this is needed, but it works.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
cebtenzzre added a commit that referenced this pull request May 15, 2024
@cebtenzzre
cmake: fix Metal build after #2310 (#2350)
bf9aaab 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
manyoso pushed a commit that referenced this pull request May 15, 2024
@cebtenzzre
cmake: fix Metal build after #2310 (#2350)
a92d266 
I don't understand why this is needed, but it works.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
cebtenzzre added a commit that referenced this pull request May 15, 2024
@cebtenzzre
cmake: fix Metal build after #2310 (#2350)
7fe0639 
Signed-off-by: Jared Van Bortel <jared@nomic.ai>
cebtenzzre added a commit that referenced this pull request May 21, 2024
@cebtenzzre
ci: explicitly use -DCMAKE_BUILD_TYPE=Release
a275447 
This matters now that #2310 removed the default of "Release" in
llama.cpp.cmake.

Signed-off-by: Jared Van Bortel <jared@nomic.ai>
@cebtenzzre cebtenzzre mentioned this pull request May 21, 2024
Open
@iimez iimez mentioned this pull request May 26, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@manyoso manyoso manyoso approved these changes

Assignees

@cebtenzzre cebtenzzre

Labels
None yet
Projects
GPT4All 2024 Roadmap and Active Issues
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature] Add support for StarCoder2 model architecture
2 participants
@cebtenzzre @manyoso