feat: User Selectable GPUs and GPU-based Model Recommendations

[hiento09]

Changes

• Added a field to settings.json to manage selected GPUs
• Implemented functionality to get list of GPUs from file
• Added GPU monitor to bottom bar
• Implemented model recommendations in hub based on GPU vRam in GPU mode
• Increased interval time to get System Resource from 0.5s to 2s

Reason for changes

• To allow users to select and monitor their GPUs
• To provide model recommendations based on GPU vRam
• To improve system resource retrieval performance

Fixes Issues

• Closes feat: machine with multiple GPUs enables users of both Windows and Linux to select a preferred GPU #1426

Images

• By default, the most powerful GPU will be selected if the user enables GPU mode for the first time. After that, the system will not intervene, allowing the user to choose the GPU they want to use. The next time the app is launched, it will load the GPU according to the user's configuration.

• When GPU mode is enabled, a model must certainly be selected. The user will not be able to deselect all GPUs
  

• When the user chooses to use more than one GPU, a warning will be displayed to remind the user about multi-GPUs mode.

• The bottom bar monitoring GPU resources will also display the corresponding selected GPUs.
  

• The current Hub will also display suggested models depending on whether it's in RAM mode or GPU mode. For GPU mode, it will base its suggestions on the total VRAM of the selected GPUs

• This is a suggestion if I enable all three GPUs. Notice the bottom bar below is enabling three GPUs, so I can load a heavy model up to 18GB as suggested here.
  

• And here, when I try to disable one GPU, you will see that the Hub will recommend differently.

Example settings.json file

{"notify":true,"run_mode":"gpu","nvidia_driver":{"exist":true,"version":"531.18"},"cuda":{"exist":true,"version":"12"},"gpus":[{"id":"0","vram":"12282","name":"NVIDIA GeForce RTX 4070 Ti\r"},{"id":"1","vram":"6144","name":"NVIDIA GeForce GTX 1660 Ti\r"},{"id":"2","vram":"6144","name":"NVIDIA GeForce GTX 1660 Ti"}],"gpu_highest_vram":"0","gpus_in_use":["0","1"],"is_initial":false}

[hiento09]

[Van-QA]

hi @hiento09, from the screenshot, looks like we are missing the separation line between the GPU and Experimental option, can you double check?

On the other hand, I would recommend shortening the warning message if possible 🤔 e.g. Enabling abc, xyz..., may ....

Many thanks

[imtuyethan]

I included the design in this ticket for this implementation:
Related design ticket: #1646
Related FE implementation ticket from @urmauur : #1851

---

Design

Figma link: https://www.figma.com/file/ytn1nRZ17FUmJHTlhmZB9f/Jan-App?type=design&node-id=2596-81009&mode=design&t=vDzqYrfNOU0W8KmX-4

When users have compatible GPU(s)

• GPU Acceleration is turned on by default if users have GPU
• Auto-select the GPU with the highest VRAM
• GPU monitoring shown on the footer

When click on choosing GPU, there are 2 cases:

• If we only detect 1 type of GPU: Show checkbox selection for all the GPU detected.
• If we detect more than 1 type of GPU (Nvidia, AMD,...): Show the type group selection + checkboxes for each GPU. Reason for this is because we only allow users to select multiple GPU within 1 type.

When users select more than 1 GPU

• Show warning If multi-GPU is enabled with different GPU models or without NVLink, it could impact token speed
• Number of GPU shown on footer -> when hover, users will see the detailed specs of each GPU.

When users turn off GPU acceleration although they have compatible GPU

Show warning icon -> when hover, users see tooltip suggestion that users should turn it on: Disabling GPU Acceleration may result in reduced performance. It's recommended to keep this enabled for optimal user experience.

When users don't have any compatible GPU

• GPU acceleration is turned off by default if users don’t have GPU
• Users cannot turn on
• If users try to turn on by clicking on the toggle -> show tooltip Your current device does not have a compatible GPU for monitoring. To enable GPU monitoring, please ensure your device has a supported Nvidia or AMD GPU with updated drivers.