How to verify that whether khoj is using GPU or not

[MuelNova]

I'm using khoj on WSL where both nvidia_smi and torch.cuda.is_available() works.

I installed khoj using CMAKE_ARGS="DLLAMA_CUDA=on" FORCE_CMAKE=1 python -m pip install khoj and no error occurred.

However, when I try to ask something, the CPU Utilization will be 100% and it takes over 100s to generate the answer while GPU Utilization stays low.

Is there anything I can do or is there any option I should enable to use GPU?

[MuelNova]

I have searched Issue and found #418, but it is outdated.

[MuelNova]

few logs

khoj

[08:40:07.344653] INFO     uvicorn.access:              h11_impl.py:431
                           127.0.0.1:44042 - "GET
                           /api/chat/sessions HTTP/1.1"
                           200
[08:40:10.953258] INFO     khoj.configure: 📡          configure.py:392
                           Uploading telemetry to
                           https://khoj.beta.haletic.c
                           om/v1/telemetry...
[08:41:01.762631] INFO     khoj.processor.conversation.util utils.py:48
                           s: Chat streaming took: 79.468
                           seconds
[08:41:01.872344] INFO     khoj.processor.conversation.uti utils.py:124
                           ls: Saved Conversation Turn
                           You (default): "..."
[08:41:01.910534] INFO     khoj.routers.api_chat: Chat  api_chat.py:595
                           response time to first
                           token: 349.621 seconds
[08:41:01.914509] INFO     khoj.routers.api_chat: Chat  api_chat.py:596
                           response total time: 429.234
                           seconds

nvidia_smi

❯ nvidia-smi
Mon Aug 19 16:48:30 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.58.02              Driver Version: 556.12         CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4060 ...    On  |   00000000:01:00.0  On |                  N/A |
| N/A   50C    P5              9W /  140W |    3915MiB /   8188MiB |     44%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A     64699      C   /python3.12                                 N/A      |
+-----------------------------------------------------------------------------------------+

where 64699 is the pid of khoj

[invictus-1]

Did you ever find a solution to this? I'm currently having the same issue where its just cpu being used not gpu.

[MuelNova]

Did you ever find a solution to this? I'm currently having the same issue where its just cpu being used not gpu.

No, I'm now using OpenAI as the backend.