You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
unsloth studio load DeepSeek-V4-Flash-0731-GGUF failed:
Failed to load model: llama-server failed to start. Check that the GGUF file is valid and you have enough memory.
ERROR log >>>
INF timestamp=2026-08-04T16:57:06.243613Z event=Detected remote GGUF repo 'unsloth/DeepSeek-V4-Flash-0731-GGUF', variant=UD-IQ3_XXS, vision=False
INF timestamp=2026-08-04T16:57:07.898131Z event=GGUF metadata: context_length=1048576
INF timestamp=2026-08-04T16:57:07.898348Z event=GGUF metadata: chat_template=13698 chars
INF timestamp=2026-08-04T16:57:07.898611Z event=GGUF metadata: model supports reasoning (enable_thinking + reasoning_effort: ['high', 'max'])
INF timestamp=2026-08-04T16:57:07.898820Z event=GGUF metadata: model supports tool calling
INF timestamp=2026-08-04T16:57:08.075729Z event=request_completed method=GET path=/api/inference/monitor status_code=200 process_time_ms=4.59
INF timestamp=2026-08-04T16:57:08.788642Z event=Reusing cached GGUF: /root/.cache/huggingface/hub/models--unsloth--DeepSeek-V4-Flash-0731-GGUF/snapshots/57326b941c4603e24d1a5e71c22520c66e086eb8/UD-IQ3_XXS/DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf
INF timestamp=2026-08-04T16:57:10.941549Z event=GGUF metadata: context_length=1048576
INF timestamp=2026-08-04T16:57:10.941776Z event=GGUF metadata: chat_template=13698 chars
INF timestamp=2026-08-04T16:57:10.942065Z event=GGUF metadata: model supports reasoning (enable_thinking + reasoning_effort: ['high', 'max'])
INF timestamp=2026-08-04T16:57:10.942254Z event=GGUF metadata: model supports tool calling
INF timestamp=2026-08-04T16:57:11.336431Z event=Context auto-reduced: 1048576 -> 4096 (model: 97.1 GB, est. KV cache: 0.1 GB)
INF timestamp=2026-08-04T16:57:11.336993Z event=GGUF size: 97.1 GB, est. KV cache: 0.1 GB, context: 4096, GPUs free: [(0, 22384), (1, 22615)], selected: None, --fit: on
INF timestamp=2026-08-04T16:57:11.338756Z event=Reasoning model: {'enable_thinking': True} by default
INF timestamp=2026-08-04T16:57:11.339002Z event=Starting llama-server: /root/.unsloth/llama.cpp/llama-server -m /root/.cache/huggingface/hub/models--unsloth--DeepSeek-V4-Flash-0731-GGUF/snapshots/57326b941c4603e24d1a5e71c22520c66e086eb8/UD-IQ3_XXS/DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf --port 39059 --parallel 4 --flash-attn on --no-context-shift -c 4096 --alias unsloth/DeepSeek-V4-Flash-0731-GGUF --fit on --metrics --slot-save-path /root/.unsloth/studio/cache/llama-slots --kv-unified --jinja --spec-default --chat-template-kwargs {"enable_thinking": true}
INF timestamp=2026-08-04T16:57:11.341553Z event=llama-server stdout/stderr -> /root/.unsloth/studio/logs/llama-server/llama-1785862631-port-39059-try0.log
ERR timestamp=2026-08-04T16:57:11.477620Z event=llama-server exited with code 1. Output (tail): Full log: /root/.unsloth/studio/logs/llama-server/llama-1785862631-port-39059-try0.log
ERR timestamp=2026-08-04T16:57:11.479034Z event=Error loading model: llama-server failed to start. Check that the GGUF file is valid and you have enough memory. exception=Traceback (most recent call last):
File "/root/.unsloth/studio/unsloth_studio/lib/python3.13/site-packages/studio/backend/routes/inference.py", line 6326, in _load_model_impl
success = await load_with_tensor_fallback(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<5 lines>...
)
^
File "/root/.unsloth/studio/unsloth_studio/lib/python3.13/site-packages/studio/backend/core/inference/tensor_fallback.py", line 52, in load_with_tensor_fallback
success = await attempt_load(requested_tensor, extra_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.unsloth/studio/unsloth_studio/lib/python3.13/site-packages/studio/backend/routes/inference.py", line 6316, in _attempt_gguf_load
return await asyncio.to_thread(
^^^^^^^^^^^^^^^^^^^^^^^^
...<2 lines>...
)
^
File "/root/.local/share/uv/python/cpython-3.13.14-linux-x86_64-gnu/lib/python3.13/asyncio/threads.py", line 26, in to_thread
return await loop.run_in_executor(None, func_call)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.13.14-linux-x86_64-gnu/lib/python3.13/concurrent/futures/thread.py", line 59, in run
result = self.fn(*self.args, **self.kwargs)
File "/root/.unsloth/studio/unsloth_studio/lib/python3.13/site-packages/studio/backend/core/inference/llama_cpp.py", line 1584, in wrapped
return load(self, intent)
File "/root/.unsloth/studio/unsloth_studio/lib/python3.13/site-packages/studio/backend/core/inference/llama_cpp.py", line 10735, in load_model
raise RuntimeError(
...<6 lines>...
)
RuntimeError: llama-server failed to start. Check that the GGUF file is valid and you have enough memory.
But it works fine when run manually.
Log >>>>
root@1573460ad993:/app# /root/.unsloth/llama.cpp/llama-server \
-m /root/.cache/huggingface/hub/models--unsloth--DeepSeek-V4-Flash-0731-GGUF/snapshots/*/UD-IQ3_XXS/DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf \
--port 43117 \
-c 4096 \
--fit on --flash-attn on --no-context-shift --parallel 4 \
--kv-unified --jinja --spec-default \
--chat-template-kwargs '{"enable_thinking": true}'
0.00.139.771 W Setting 'enable_thinking' via --chat-template-kwargs is deprecated. Use --reasoning on / --reasoning off instead.
0.00.141.942 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.406.439 W srv llama_server: -----------------
0.00.406.444 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.406.445 W srv llama_server: this can be a security risk (cross-origin attacks)
0.00.406.445 W srv llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.406.445 W srv llama_server: -----------------
0.00.407.763 I srv load_model: loading model '/root/.cache/huggingface/hub/models--unsloth--DeepSeek-V4-Flash-0731-GGUF/snapshots/57326b941c4603e24d1a5e71c22520c66e086eb8/UD-IQ3_XXS/DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf'
0.07.326.075 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance
0.20.645.386 I srv load_model: initializing, n_slots = 4, n_ctx_slot = 4096, kv_unified = 'true'
0.20.659.501 I srv llama_server: model loaded
0.20.659.511 I srv llama_server: listening on http://127.0.0.1:43117
2.44.656.666 I slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
2.44.656.786 I slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
2.53.135.815 I slot print_timing: id 3 | task 0 | prompt evaltime = 386.92 ms / 5 tokens ( 77.38 ms per token, 12.92 tokens per second)
2.53.135.826 I slot print_timing: id 3 | task 0 |evaltime = 8086.46 ms / 100 tokens ( 80.86 ms per token, 12.37 tokens per second)
2.53.135.827 I slot print_timing: id 3 | task 0 | total time = 8473.38 ms / 105 tokens
2.53.135.840 I slot print_timing: id 3 | task 0 | graphs reused = 99
2.53.135.954 I slot release: id 3 | task 0 | stop processing: n_tokens = 104, truncated = 0
Run unsloth studio log command:
/root/.unsloth/llama.cpp/llama-server -m /root/.cache/huggingface/hub/models--unsloth--DeepSeek-V4-Flash-0731-GGUF/snapshots/57326b941c4603e24d1a5e71c22520c66e086eb8/UD-IQ3_XXS/DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf --port 39059 --parallel 4 --flash-attn on --no-context-shift -c 4096 --alias unsloth/DeepSeek-V4-Flash-0731-GGUF --fit on --metrics --slot-save-path /root/.unsloth/studio/cache/llama-slots --kv-unified --jinja --spec-default --chat-template-kwargs {"enable_thinking": true}
error while handling argument "--chat-template-kwargs": [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing object key - invalid literal; last read: '{e'; expected string literal
usage:
--chat-template-kwargs STRING sets additional params for the json template parser, must be a valid
json object string, e.g. '{"key1":"value1","key2":"value2"}'
(env: LLAMA_ARG_CHAT_TEMPLATE_KWARGS)
to show complete usage, run with -h
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
unsloth studio load DeepSeek-V4-Flash-0731-GGUF failed:
Failed to load model: llama-server failed to start. Check that the GGUF file is valid and you have enough memory.
ERROR log >>>
But it works fine when run manually.
Log >>>>
Run unsloth studio log command:
/root/.unsloth/llama.cpp/llama-server -m /root/.cache/huggingface/hub/models--unsloth--DeepSeek-V4-Flash-0731-GGUF/snapshots/57326b941c4603e24d1a5e71c22520c66e086eb8/UD-IQ3_XXS/DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf --port 39059 --parallel 4 --flash-attn on --no-context-shift -c 4096 --alias unsloth/DeepSeek-V4-Flash-0731-GGUF --fit on --metrics --slot-save-path /root/.unsloth/studio/cache/llama-slots --kv-unified --jinja --spec-default --chat-template-kwargs {"enable_thinking": true} error while handling argument "--chat-template-kwargs": [json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing object key - invalid literal; last read: '{e'; expected string literal usage: --chat-template-kwargs STRING sets additional params for the json template parser, must be a valid json object string, e.g. '{"key1":"value1","key2":"value2"}' (env: LLAMA_ARG_CHAT_TEMPLATE_KWARGS) to show complete usage, run with -hAll reactions