Misc. bug: test-chat-template hangs on Windows (MSYS2 UCRT64 + GCC)

[rillomas]

Name and Version

$ bin/llama-cli.exe --version
load_backend: failed to find ggml_backend_init in C:\Users\devcloud\Documents\repo\llama.cpp\build-ci-debug\bin\ggml-vulkan.dll
load_backend: failed to find ggml_backend_init in C:\Users\devcloud\Documents\repo\llama.cpp\build-ci-debug\bin\ggml-cpu.dll
version: 8829 (6990e2f)
built with GNU 15.2.0 for Windows AMD64

Operating systems

Windows

Which llama.cpp modules do you know to be affected?

Core DLLs?

Command line

bin/test-chat-template.exe`

Problem description & steps to reproduce

I'm observing a complete hang at test-chat-template.exe from 6990e2f when building for CI as LLAMA_FATAL_WARNINGS=OFF GG_BUILD_NINJA=1 GG_BUILD_VULKAN=1 GG_BUILD_LOW_PERF=1 ./ci/run.sh ./results/llama.cpp ./mnt/llama.cpp on MSYS2 UCRT64 + GCC. The test execution seems to finish but just hangs at the end and I can't Ctrl-C. Running from gdb is the same (Can't Ctrl-C). fcc7508 works correctly

I see the same issue when built with cmake -B build_vk -DGGML_VULKAN=ON -DLLAMA_CURL=OFF -DBUILD_SHARED_LIBS=ON but it does not occur when built with cmake -B build_vk -DGGML_VULKAN=ON -DLLAMA_CURL=OFF

First Bad Commit

6990e2f (#21936)

Relevant log output

Logs

# build-ci-debug/bin/test-chat-template.exe
Built-in chat templates:
  bailing
  bailing-think
  bailing2
  chatglm3
  chatglm4
  chatml
  command-r
  deepseek
  deepseek-ocr
  deepseek2
  deepseek3
  exaone-moe
  exaone3
  exaone4
  falcon3
  gemma
  gigachat
  glmedge
  gpt-oss
  granite
  granite-4.0
  grok-2
  hunyuan-dense
  hunyuan-moe
  hunyuan-ocr
  kimi-k2
  llama2
  llama2-sys
  llama2-sys-bos
  llama2-sys-strip
  llama3
  llama4
  megrez
  minicpm
  mistral-v1
  mistral-v3
  mistral-v3-tekken
  mistral-v7
  mistral-v7-tekken
  monarch
  openchat
  orion
  pangu-embedded
  phi3
  phi4
  rwkv-world
  seed_oss
  smolvlm
  solar-open
  vicuna
  vicuna-orca
  yandex
  zephyr


=== teknium/OpenHermes-2.5-Mistral-7B ===



=== mistralai/Mistral-7B-Instruct-v0.2 (NOTE: Old pre-v1 without a system prompt) ===



=== TheBloke/FusionNet_34Bx2_MoE-AWQ ===



=== bofenghuang/vigogne-2-70b-chat ===



=== mlabonne/AlphaMonarch-7B ===



=== google/gemma-7b-it ===



=== OrionStarAI/Orion-14B-Chat ===



=== openchat/openchat-3.5-0106 ===



=== deepseek-ai/deepseek-coder-33b-instruct ===



=== eachadea/vicuna-13b-1.1 ===



=== Orca-Vicuna ===



=== CohereForAI/c4ai-command-r-plus ===



=== Llama-3 ===



=== Phi-3-mini ===



=== Phi-3-small ===



=== Phi-3-medium ===



=== Phi-3-vision ===



=== ChatGLM3 ===



=== ChatGLM4 ===



=== GLMEdge ===



=== MiniCPM-3B-OpenHermes-2.5-v2-GGUF ===



=== DeepSeek-V2 ===



=== ibm-granite/granite-3.0-8b-instruct ===



=== mistralai/Mistral-7B-Instruct-v0.2 (mistralai 'v1' template with a system prompt) ===



=== Mistral-Large-Instruct-2407 (mistralai 'v3' template; modified to have system prompt at start) ===



=== Mistral-Nemo-Instruct-2407 (mistralai 'v3-tekken' template; modified to have system prompt at start) ===



=== mistralai/Mistral-Large-Instruct-2411 (mistralai 'v7' template) ===



=== ai-sage/GigaChat-20B-A3B-instruct ===



=== Infinigence/Megrez-3B-Instruct ===



=== phi-4 ===



=== yandex/YandexGPT-5-Lite-8B-instruct ===



=== inclusionAI/Ling-lite ===



=== ByteDance-Seed/Seed-OSS-36B-Instruct ===



=== ibm-granite/granite-3.x (tool call) ===



=== ibm-granite/granite-4.0 (tool call) ===



=== teknium/OpenHermes-2.5-Mistral-7B (jinja) ===



=== mistralai/Mistral-7B-Instruct-v0.2 (NOTE: Old pre-v1 without a system prompt) (jinja) ===



=== TheBloke/FusionNet_34Bx2_MoE-AWQ (jinja) ===



=== bofenghuang/vigogne-2-70b-chat (jinja) ===



=== mlabonne/AlphaMonarch-7B (jinja) ===



=== google/gemma-7b-it (jinja) ===



=== OrionStarAI/Orion-14B-Chat (jinja) ===



=== openchat/openchat-3.5-0106 (jinja) ===



=== deepseek-ai/deepseek-coder-33b-instruct (jinja) ===



=== eachadea/vicuna-13b-1.1 (jinja) ===



=== Orca-Vicuna (jinja) ===



=== CohereForAI/c4ai-command-r-plus (jinja) ===



=== Llama-3 (jinja) ===



=== Phi-3-mini (jinja) ===



=== Phi-3-small (jinja) ===



=== Phi-3-medium (jinja) ===



=== Phi-3-vision (jinja) ===



=== ChatGLM3 (jinja) ===



=== ChatGLM4 (jinja) ===



=== GLMEdge (jinja) ===



=== MiniCPM-3B-OpenHermes-2.5-v2-GGUF (jinja) ===



=== DeepSeek-V2 (jinja) ===



=== ibm-granite/granite-3.0-8b-instruct (jinja) ===



=== mistralai/Mistral-7B-Instruct-v0.2 (mistralai 'v1' template with a system prompt) (jinja) ===



=== Mistral-Large-Instruct-2407 (mistralai 'v3' template; modified to have system prompt at start) (jinja) ===



=== Mistral-Nemo-Instruct-2407 (mistralai 'v3-tekken' template; modified to have system prompt at start) (jinja) ===



=== mistralai/Mistral-Large-Instruct-2411 (mistralai 'v7' template) (jinja) ===



=== Infinigence/Megrez-3B-Instruct (jinja) ===



=== phi-4 (jinja) ===



=== yandex/YandexGPT-5-Lite-8B-instruct (jinja) ===

render_message_to_json: Neither string content nor typed content is supported by the template. This is unexpected and may lead to issues.


=== inclusionAI/Ling-lite (jinja) ===



=== ByteDance-Seed/Seed-OSS-36B-Instruct (jinja) ===



=== ibm-granite/granite-3.x (tool call) (jinja) ===



=== ibm-granite/granite-4.0 (tool call) (jinja) ===


OK: All tests passed successfully.
<-- HANGS HERE

[rillomas]

@ggerganov The following fixes the hanging. It is intentionally leaking the logger instance so the destructor is never called during DLL teardown. I've confirmed CI passing on Windows MSYS2 UCRT64. I can make a PR if you like this change. A formal fix will be to add a common_log_destroy function and call that from every app, or use a static lib just for the logger part but not sure if you would like to do that.

diff --git a/common/log.cpp b/common/log.cpp
index dec4ef5fc..a3183260c 100644
--- a/common/log.cpp
+++ b/common/log.cpp
@@ -368,14 +368,15 @@ struct common_log * common_log_init() {
 }

 struct common_log * common_log_main() {
-    static struct common_log log;
+    static struct common_log* log;
     static std::once_flag    init_flag;
     std::call_once(init_flag, [&]() {
+        log = new common_log();
         // Set default to auto-detect colors
-        log.set_colors(tty_can_use_colors());
+        log->set_colors(tty_can_use_colors());
     });

-    return &log;
+    return log;
 }

 void common_log_pause(struct common_log * log) {

Based on a debugging session with Copilot GPT5.4 Medium the following is the reason.

1. test-chat-template.exe runs fine.
2. During the run, some template path emits a warning or info log from chat.cpp.
3. That lazily creates the common_log_main() singleton and starts its worker thread inside llama-common.dll.
4. At process exit, because libs : rename libcommon -> libllama-common #21936 moved that singleton into a DLL, its destructor now runs during DLL teardown instead of normal exe teardown.
5. On MSYS2 UCRT64 GCC, that join() during DLL unload is a plausible deadlock point.
6. That gives exactly your symptom: all tests pass, then test-chat-template.exe hangs at the very end.

[ggerganov]

Does #22144 fix the problem?

[rillomas]

I'm still seeing the same issue with 7b9de04. It seems that as long as the common DLL uses the logger singleton the same issue will persist since the DLL's logger instance will get destructed at DLL unload. Using the static lib version of the logger should create a different instance in the DLL vs the app but seems like it doesn't exactly solve the issue.

[ggerganov]

Hm, I'm not sure I understand what is different about test-chat-template compared to the other binaries. Why would they not have the same issue?

Does this patch on master make a difference:

diff --git a/tests/test-chat-template.cpp b/tests/test-chat-template.cpp
index bf45d737c..da1a1e980 100644
--- a/tests/test-chat-template.cpp
+++ b/tests/test-chat-template.cpp
@@ -13,6 +13,7 @@
 #include <cassert>
 
 #include "llama.h"
+#include "log.h"
 #include "common.h"
 #include "chat.h"
 #include "jinja/runtime.h"
@@ -110,6 +111,8 @@ static std::string DEFAULT_JSON_WITH_TOOLS = R"({
 
 
 int main(int argc, char ** argv) {
+    common_log_main();
+
     std::vector<std::string> args(argv, argv + argc);
 
     std::string tmpl_path;

[rillomas]

Issue still occurs with the patch applied on top of 9789512.

I'm not sure I understand what is different about test-chat-template compared to the other binaries. Why would they not have the same issue?

The other binaries do have the same issue. For example llama-cli also hangs the same way with llama-cli -m non-existent-path. Some binaries don't have the issue when I test locally (for example test-thread-safety.exe --model build-ci-debug/tinyllamas/stories15M-q4_0.gguf) and this seems like it is dependent on the state of the logger queue. I see hangs especially when head==tail at the point of destruction. I'm currently checking the differences.