Cannot start tabby on Windows using exe file or Docker

[doanhaivu]

Describe the bug
Hello, I'm starting tabby on Windows in 2 ways and both doesn't work

Using Windows exe file:

⠙  2338.427 s   Starting...2025-02-14T11:05:45.083064Z  WARN llama_cpp_server::supervisor: crates\llama-cpp-server\src\supervisor.rs:98: llama-server <embedding> exited with status code -1073741515, args: `Command { std: "E:\\tabbyml\\tabby_x86_64-windows-msvc-cuda117\\llama-server.exe" "-m" "C:\\Users\\Admin\\.tabby\\models\\TabbyML\\Nomic-Embed-Text\\ggml\\model-00001-of-00001.gguf" "--cont-batching" "--port" "30888" "-np" "1" "--log-disable" "--ctx-size" "4096" "-ngl" "9999" "--embedding" "--ubatch-size" "4096", kill_on_drop: true }`

User Docker, there is only one log line, nothing happens after

2025-02-14 16:52:59 Writing to new file.

Information about your version

.\tabby.exe --version
tabby 0.25.0-rc.2

Information about your GPU

==========
== CUDA ==
==========

CUDA Version 12.2.0

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

Fri Feb 14 09:49:36 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.86.16              Driver Version: 572.16         CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060        On  |   00000000:01:00.0  On |                  N/A |
|  0%   45C    P8             20W /  170W |    1223MiB /  12288MiB |     44%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A              26      G   /Xwayland                             N/A      |
|    0   N/A  N/A              35      G   /Xwayland                             N/A      |
+-----------------------------------------------------------------------------------------+

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Jan_15_19:38:46_Pacific_Standard_Time_2025
Cuda compilation tools, release 12.8, V12.8.61
Build cuda_12.8.r12.8/compiler.35404655_0

nvidia-smi
Fri Feb 14 18:16:45 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 572.16                 Driver Version: 572.16         CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060      WDDM  |   00000000:01:00.0  On |                  N/A |
|  0%   45C    P5             38W /  170W |    1479MiB /  12288MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            2556    C+G   ....0.3065.59\msedgewebview2.exe      N/A      |
|    0   N/A  N/A            4844    C+G   ...al\Programs\Notion\Notion.exe      N/A      |
|    0   N/A  N/A            5556    C+G   ...ps\Microsoft VS Code\Code.exe      N/A      |
|    0   N/A  N/A            5816    C+G   C:\Windows\explorer.exe               N/A      |
|    0   N/A  N/A            6980    C+G   ..._cw5n1h2txyewy\SearchHost.exe      N/A      |
|    0   N/A  N/A            7004    C+G   ...y\StartMenuExperienceHost.exe      N/A      |
|    0   N/A  N/A            7628    C+G   ...xyewy\ShellExperienceHost.exe      N/A      |
|    0   N/A  N/A            8260    C+G   ...App_cw5n1h2txyewy\LockApp.exe      N/A      |
|    0   N/A  N/A           14944    C+G   ...5n1h2txyewy\TextInputHost.exe      N/A      |
|    0   N/A  N/A           15340    C+G   ....0.3065.59\msedgewebview2.exe      N/A      |
|    0   N/A  N/A           16400    C+G   ...IA app\CEF\NVIDIA Overlay.exe      N/A      |
|    0   N/A  N/A           17000    C+G   ...IA app\CEF\NVIDIA Overlay.exe      N/A      |
|    0   N/A  N/A           18156    C+G   ...r\frontend\Docker Desktop.exe      N/A      |
|    0   N/A  N/A           18400    C+G   ...ouryDevice\asus_framework.exe      N/A      |
|    0   N/A  N/A           18812    C+G   ...\slack\app-4.42.117\slack.exe      N/A      |
|    0   N/A  N/A           20516    C+G   ...t\Edge\Application\msedge.exe      N/A      |
|    0   N/A  N/A           20856    C+G   ...\slack\app-4.42.117\slack.exe      N/A      |
|    0   N/A  N/A           21932    C+G   ...Chrome\Application\chrome.exe      N/A      |
|    0   N/A  N/A           23348    C+G   ...2p2nqsd0c76g0\app\ChatGPT.exe      N/A      |
|    0   N/A  N/A           25024    C+G   ...yb3d8bbwe\WindowsTerminal.exe      N/A      |
|    0   N/A  N/A           26020    C+G   ...Chrome\Application\chrome.exe      N/A      |
|    0   N/A  N/A           26176    C+G   ...Telegram Desktop\Telegram.exe      N/A      |
|    0   N/A  N/A           31048    C+G   ...ntrolPanel\SystemSettings.exe      N/A      |
+-----------------------------------------------------------------------------------------+

[rmawatson]

takes a while for the docker to startup. Give it a few minutes.

[doanhaivu]

Thanks, it took 10 minutes for the second log line to appear and 30 minutes to finish stating on Docker.

I'm still curious about the other error though

[zwpaper]

Hi @doanhaivu, The logs displayed during Docker execution, Writing to new file signify that Tabby is currently downloading the model file from the internet. The duration of the download is contingent upon the speed of your network connection and the size of the model.

Regarding the binary failure, you could attempt to execute llama.cpp manually as suggested by the output and then review the error message:

"E:\\tabbyml\\tabby_x86_64-windows-msvc-cuda117\\llama-server.exe" "-m" "C:\\Users\\Admin\\.tabby\\models\\TabbyML\\Nomic-Embed-Text\\ggml\\model-00001-of-00001.gguf" "--cont-batching" "--port" "30888" "-np" "1" "--log-disable" "--ctx-size" "4096" "-ngl" "9999" "--embedding" "--ubatch-size" "4096"

[Lightern8]

Same thing. All was perfect until docker restart/update. I change docker configuration to use local local Embed model (Nomic). All other were remote opeanapi servers, and while tabby do not need local models all was OK. But now llama-server shutting down because of absence of libcudart.so.12. I can't find such library in official docker image. Maybe llamacpp update needed cuda libraries update too?

EDIT2: "CUDA_VERSION=11.7.1" inside container. Latest llamacpp needed 12+ I suppose.

EDIT: registry.tabbyml.com/tabbyml/tabby (as in documentation) - not working
registry.tabbyml.com/tabbyml/tabby:latest - not working
registry.tabbyml.com/tabbyml/tabby:0.25.1-rc.0 - not working
registry.tabbyml.com/tabbyml/tabby:0.24.0 - not working
registry.tabbyml.com/tabbyml/tabby:0.24.0-rc6 - not working

will work only after revert to online services back like:

[model.embedding.http]
kind = "openai/embedding"
model_name = "text-embedding-ada-002"
api_endpoint = "https://api.openai.com/v1"
api_key = "my_token"

but simple delete this section (to use Nomic) - we have eternal timer counting in docker logs.

EDIT3: solved by creating own container with 2 bin files (tabby + llama-server from releases section) and
FROM nvidia/cuda:12.8.0-runtime-ubuntu22.04 + apt-get install -y libgomp1

[aspyk]

I also have the issue with the executable: I manage to print the log of the llama-server.exe command removing the --log-disable option, but it still stops without clear reason. The two last lines are:

common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)

but when I add the --no-warmup, the server seems to start nicely...

main: server is listening on http://127.0.0.1:30888 - starting the main loop
srv  update_slots: all slots are idle

Both cuda and CPU have the same behavior.