epic: Implement Cortex Hardware API for Nvidia

[vansangpfiev]

Implementation for #1165

• Scope to Nvidia first (AMD, Intel, Qualcomm to subsequent sprints)

Tasklist

(Will fill in details when implement each task)

Hardware API

• Hardware Detection
  • Implement platform-specific detection
  • Implement GPU-specific detection
  • Implement hardware detection function #1591
  • Develop separate modules for Windows, Linux, and macOS to gather hardware information
  • Use appropriate system calls and libraries
• Implement CLI for hardware activation
• Implement RAM check for CPU fallback
• Create API for hardware activation (Nvidia) #1603
• Documentation

/engines

• Modify engine initialization
• Implement hardware passing to engines

/model/start

• Handle ngl settings (in /models/start)
• Implement RAM and VRAM detection
• Implement fallback logic

Jan

• System Monitor
• Model Start
• Engines Settings?

Bugs to Address

• bug: Inference with NVIDIA GPU stop working after resuming from sleep on Linux #1459

Related bugs:

• bug: Models loading onto wrong GPU in multi-GPU setup #1468
• bug: some models failed to load if many GPU are selected #1458
• bug: Cortex-cpp continues to have 1 layer offload to CPU while using GPU #1104
• bug: Failed to Load Model with AMD GPU and Vulkan #1525

Out-of-scope

• AMD
• Intel
• Qualcomm/Snapdragon

[vansangpfiev]

Hardware API Documentation

Get hardware information

GET /v1/hardware

Response:

{
  "cpu": {
    "arch": "string",
    "cores": number,
    "model": "string",
    "instructions": ["string"]
  },
  "os": {
    "version": "string",
    "name": "string"
  },
  "ram": {
    "total": number,
    "available": number,
    "type": "string"
  },
  "storage": {
    "total": number,
    "available": number,
    "type": "string"
  },
  "gpus": [
    {
      "model": "string",
      "vram": "string",
      "driver_version": "string"
    }
  ],
  "power": {
    "battery_life": number,
    "charging_status": "string",
    "is_power_saving": boolean
  },
  "monitors": [
    {
      "resolution": "string",
      "refresh_rate": number,
      "resolution":"string"
    }
  ]
}

Hardware Activation

POST /v1/hardware/activate

{
"gpus": [0, 1]
}

[dan-menlo]

Thanks @vansangpfiev. Will we be implementing deactivate this sprint?

[vansangpfiev]

Thanks @vansangpfiev. Will we be implementing deactivate this sprint?

Since we have /activate endpoint, I think it is redundant to add /deactivate.
By default, we activate all the GPUs. We deactivate all GPUs that are not in request for /activate.

[dan-menlo]

A few notes from our quick call:

Hardware Support

We will need to work with multiple hardware providers, but these can be dealt with in separate sprints:

• For Intel, can we detect iGPUs, NPUs and CPUs? (i.e. Lunar Lake)
• For AMD, can we detect iGPUs
• For Qualcomm/ARM, can we detect Adreno etc

ngl settings

• We detect hardware to recommend ngl setting to users
• /models/start API will infer hardware info from database, and then recommend ngl
• This is not part of Hardware API, but /models/start is using hwinfo

[vansangpfiev]

CLI Documentation:

Get hardware information

cortex hardware list --cpu --os --ram --storage --gpu --power --monitors

If no flag is specified, display all hardware information

Activate hardware

cortex hardware activate --gpus [gpu_list]

gpu_list is required, [] means deactivate all GPUs

Start model

cortex start [model_id] --gpus [gpu_list]

--gpus is optional, if not specified use all activated GPUs

Run

cortex run [model_id] --gpus [gpu_list]

--gpus is optional, if not specified use all activated GPUs

[gabrielle-ong]

Nicely done @vansangpfiev! Testing it out now - 2 quick questions:

1. I cant seem to deactivate the GPU to test without GPU -

cortex-nightly hardware activate --gpus []
Invalid GPU index provided.

2. GPU information has Index=1, ID=0 for the same GPU, which is confusing - can we standardize to using Index like the other fields?
   

[vansangpfiev]

Nicely done @vansangpfiev! Testing it out now - 2 quick questions:

1. I cant seem to deactivate the GPU to test without GPU -

cortex-nightly hardware activate --gpus []
Invalid GPU index provided.

2. GPU information has Index=1, ID=0 for the same GPU, which is confusing - can we standardize to using Index like the other fields?
   

Thanks @gabrielle-ong

1. Let me take a look. Would you mind sharing the cortex.log and cortex-cli.log?
2. Sure, let me fix it. Actually, the ID is the GPU ID that nvidia-smi reports, it can be different from #index.