epic: Improve Cortex Engine Management

[gabrielle-ong]

Goal

• Cortex has a clear Engine Abstraction
• Engines have dependency management
  • Dependency resolution (OS, drivers)
  • Dependency handling (e.g. missing drivers, error messages)
• Engines have state management
  • Engines have metadata stored in cortex.db

Tasklist

• Docs https://cortex.so/docs/engines
  • Cortex Engines Conceptual overview
  • How to add an Engine
• planning: Cortex handling Engine Variants #1453
• planning: Cortex handles Engine Versions #1454
• feat: Engines API endpoints should be consistent #1684

Open Questions

• Do we allow users to run multiple engines in parallel
  • Yes, engines can in parallel
  • Dan: they are running models in parallel, that run engines

Appendix

Improvements to #1072

[dan-menlo]

User Journey

• I download Cortex for the first time
• Cortex Local Installer or Network installer auto-detects my hardware and installs the appropriate variant(s)
  • Decision: should we install all variants?
  • Decision: what happens if user has both an AMD GPU and a Nvidia GPU?
• User downloads model and runs it
  • Tries using default llama.cpp
  • Can switch to other llama.cpp variant and try it (e.g. Vulkan)

[namchuai]

@dan-homebrew, here's my opinion about this task. I think should consolidate both #1453 and #1454 into this ticket.

1. Decision: should we install all variants?

• I don't think we should install all variant by default.

2. Decision: what happens if a user has both an AMD GPU and an Nvidia GPU?

• From my understanding, a user can only have either AMD or Nvidia GPU running at a time. So, I guess the question is more like how to manage engines if user switching back and forth from AMD to Nvidia GPU and vice versa. This use case can expand because the user can unplug the GPU, and we only have CPU left.

So, I think we need a solution at runtime, detect current hardware state and automatically try to choose the best engine possible. We also have to provide a user a way to specify the engine version and variant that user want to run their model.

Approach

I fully agree with option 1 you provided over #1453

File system structure

engines
|--cortex.llamacpp
    |--<variant-version>(e.g. mac-arm64-0.1.34)
        |--libengine.dylib
        |--version.txt

Tasks breakdown

1. Restructuring engine file structure
2. Make small POC to set DLL path on Windows to be sure
3. Engine install command allow download and install with specific version and variant
4. Engine use command cortex engines llama-cpp use <variant-version> allow user to set default version and variant for a particular engine
5. Add -v command cortex engines llama-cpp -v to show current selected (default) engine for llamacpp.
6. Add list command cortex engines llama-cpp [filter] to display list of installed engines for llamacpp and filterable.
7. Update engine uninstall command.
8. Update cortex ps command to display engine variant & version used to load model.
9. Create and update corresponding HTTP APIs.
10. Integrate loading model with this new default engine logic.

Edge cases

1. I'm not sure if multiple variant of the same engine can be loaded both at once.
2. Have to set correct the DLL path for windows dynamically

[dan-menlo]

@namchuai Merging "Cortex handles Engine Variants" into this issue:

Cortex handles Engine Variants

• Cortex has a clean Engines abstraction that allows us to handle llama.cpp versions and variants
  • llama.cpp versions: e.g. b3912, b3909
  • llama.cpp variants: e.g. win-avx-512, win-hipblas, win-llvm
• Jan's "Vulkan" functionality will be replaced by this API (cc @louis-jan)
  • We will need to support more variants, as llama.cpp's support for sycl, llvm grow
• We support multiple hardware types
  • epic: llama.cpp should support LLVM for ARM-based CPUs #1251
  • epic: llama.cpp should support sycl for Intel-based CPUs #1252
• Related: ci: CI packaging of llama.cpp dependencies into the binary file by default for Cortex's integration into Jan #1369

Tasklist

• API Design
• CLI Design

Questions

Scenario

• There are some users who have both Nvidia and AMD GPUs in their computer
  • Jan already supports Vulkan
  • Under the hood, this requires us to switch from llama-cuda-avx2 to llama-vulkan
  • llama.cpp alone has 18 variants at the moment

Cortex needs an elegant way to handle different engine versions + variants, without confusing the user. From my naive perspective, there are two key approaches

Option 1: Every engine is versioned, and maintains a list of variants that it can use

• Engines are versioned, and each version has several variants that can be chosen from
  • CLI: we would support a nvm-like use command
  • API: /engines API endpoint would have a use endpoint

> cortex engines get llama.cpp 
{ 
    version: b3919
    ...
}

> cortex engines llama.cpp variants list
llama-b3912-bin-win-hip-x64-gfx1030
llama-b3912-bin-win-cuda-cu11.7.1-x64

> cortex engines llama.cpp use llama-b3912-bin-win-cuda-cu11.7.1

Option 2: Every engine version/variant is a first-class Engine citizen

• We treat every single engine version/variant as a first-class engine citizen (e.g. llama-b3919-avx-cuda)
  • Users will basically run models using a specific engine variant/version
  • cortex engines list will show a massive long list of engines
• I don't think this is doable, tbh

> cortex engines list

llama.cpp-b3919-cuda
llama.cpp-b3821-vulkan

[dan-menlo]

@namchuai Merging "Cortex handles Engine Versions":

Cortex handles Engine Versions

• llama.cpp is updated frequently, with multiple releases every few days
• Cortex Engines needs to support an updating mechanism
• This needs to be handled in conjunction with planning: Cortex handling Engine Variants #1453

Tasklist

• API Design
• CLI Design
• Cortex Stable defines llama.cpp version

Design

API

CLI

# CLI
> cortex engines update llama.cpp

# API
POST /engines/{engine}/update

Open question: should we allow users to run different versions of llama.cpp?

> cortex engines llama.cpp versions
1. b3919
2. b3909

> cortex engines

Release Management

Cortex Stable and Nightly defines a llama.cpp version that it supports

• cortex update will update llama.cpp to the supported version

Cortex Nightly automatically pulls the latest llama.cpp, and forces us to fix it?