Discussion: Cortex.cpp Engine Dependencies Architecture (e.g. CUDA Toolkit)

[namchuai]

Motivation

1. Do we package the cuda toolkit to the engine?
   Yes? Then will have to do the same for llamacpp, tensorrt-llm and onnx?
   No? Will download separatedly

2. Folder structures (e.g if user have llamacpp, tensorrt at the same time)?

Resources
Llamacpp release
Currently we are downloading toolkit dependency via https://catalog.jan.ai/dist/cuda-dependencies/<version>/<platform>/cuda.tar.gz

cc @vansangpfiev @nguyenhoangthuan99 @dan-homebrew

Update sub-tasks:

• CI to pack CUDA dependencies for cortex.llamacpp #1085
• CI to pack CUDA dependencies for cortex.tensorrt-llm #1086

[vansangpfiev]

On my perspective, we should download CUDA toolkit separately. We support multiple engines: cortex.llamacpp and cortex.tensorrt-llm, both need CUDA toolkit to run. CUDA is backward compatible so we only need the latest CUDA toolkit version that supported by nvidia-driver version.
For example:

• Nvidia driver: "527.41" compatible with CUDA 12.4
• cortex.llamacpp: depends on CUDA 12.2
• cortex.tensorrt-llm: depends on CUDA 12.4
  We only need to download CUDA 12.4 to support both engines .

Edit: I just checked the cuda matrix compatibility and it is incorrect that CUDA is always backward compatible

Related ticket: https://github.com/janhq/cortex/issues/1047

Edit 2: The above image is forward compatibility between cuda and nvidia-version

From CUDA 11 onwards, applications compiled with a CUDA Toolkit release from within a CUDA major release family can run

So yes, CUDA is backward compatible within a CUDA major release
reference: https://docs.nvidia.com/deploy/cuda-compatibility/#minor-version-compatibility

[nguyenhoangthuan99]

I also think we need to Download CUDA toolkit separately, both tensorrt llm and llamacpp require cuda, plus:
inside tensorrt llm package (~ 1GB) doesn't include cuda toolkit lib (cublas, cuparse, ... which is very heavy ~400Mb), if we decided to pack everything in 1 package for both tensorrt-llm and llamacpp, the size will increase.

[namchuai]

I'm referring this table to check for the compatibility between driver and toolkit
https://docs.nvidia.com/deeplearning/cudnn/latest/reference/support-matrix.html#gpu-cuda-toolkit-and-cuda-driver-requirements

[dan-menlo]

Can I verify my understanding of the issue:

Decision
For Nvidia GPU users, the different engines have CUDA dependencies that are large 200-400mb downloads.

1. Per-engine CUDA dependencies (i.e. install separately)
2. Download 1 CUDA Toolkit for all engines

My initial thoughts

• I think per-engine CUDA dependencies are more sustainable architecture
• It is less efficient, but easier to manage long-term
• i.e. llama.cpp packages its cudart files that have been verified
• i.e. TensorRT-LLM packages its own CUDA dependencies (may change in the future)

This will be disk-space inefficient. However, the alternative seems to be dependency hell, which I think is even worse.

Folder Structure

• My ideal outcome for Cortex is where each engine is its own submodule, and manages its own folder structure.
• We have invested in making cortex.llama.cpp a separate module; it should ideally be independent and package with its dependencies

/cortex
    /engines
        /llama.cpp-extension
            /deps                               # CUDA dlls
        /tensorrt-llm-extension
            /deps                               # CUDA dlls

That said, am open to all ideas, especially @vansangpfiev's

[vansangpfiev]

If disk-space inefficient is acceptable, I think we can go with option 1.
Please note that we will have some blockers for this option:

• dynamic library search path: we will have two paths for llamacpp and tensorrt-llm, a potential issue can happen when we mix them together. For ubuntu and MacOS, I think we can solve that issue by compiling with rpath flag. For windows, I have created related issue

• CI changes for cortex.llamacpp and cortex.tensorrt-llm to pack CUDA dependencies

[namchuai]

Thanks @vansangpfiev and @dan-homebrew

I'm confirming that we agree with:
Question 1: Packaging CUDA toolkit dependencies into corresponding engine.
Caveats:

• CI changes for cortex.llamacpp and cortex.tensorrt-llm to pack CUDA dependencies

Question 2: Storing CUDA dependencies under corresponding engines.

/cortex
    /engines
        /cortex.llamacpp
            /deps                               # CUDA dlls
        /cortex.tensorrt-llm
            /deps                               # CUDA dlls

Caveats:

• dynamic library search path on windows

Additional thought
@vansangpfiev , I think when we change the CI for engine, could we associated a file which contains the versions of the engine and info of its dependencies. This will help for engine list command in the future.
wdyt? cc @nguyenhoangthuan99