Finding vulkan TTFT regressions reason

[fl0rianr]

Hi, I'm from the downstram Lemonade team and we see an re-occuring TTFT Vulkan regression that might be due to a code change. Once we found the reason we might be able to bring the good timings back.

We had this in a discord discussion very briefly before, but I do not think the current data can be dismissed as generic runner noise. We don't know for sure, but I would appreciate looking together into #22930 if it might be a part of the problem.

We did not change our script for validation and only the first real model request after loading. Let's consider that llama.cpp's 1 or 2 token warmup does not cover all prompt shapes, Vulkan pipelines are compiled lazily, and internal buffers may grow during the first request. These costs are included in our measurement:

Here are the Lemonade results on the same runner label over time/commit history:

llama.cpp	Gemma 4 26B	Qwen3-Coder-Next	GPT-OSS 120B
b9585	0.193 s	0.433 s	0.672 s
b9632	2.171 s	1.458 s	0.996 s
b9747	6.413 s	44.110 s	69.835 s
b10107	6.763 s	44.900 s	104.072 s
b10229	28.422 s	151.038 s	78.753 s
b10241	7.829 s	47.179 s	271.654 s

PR #22930 is the only broad Vulkan change is saw that could have caused this clear regression start. It is affecting internal buffer allocation on all UMA devices, such as the tested Stix halo. The current GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM path cannot test this correctly as it seems.
We also see the situation is changing drastically, so I suspect we have mixed effect.

My propsoal to go after this:
A proper same-session A/B test, compare the historical builds, b9747 with #22930 reverted, and b10229 with the older Vulkan SDK. Each model should receive three identical requests so we can separate cold and warm timings. Additionally record pipeline creation time, selected Vulkan memory types, driver details, prompt length, ubatch, flash attention and graph optimization.

After this test we know, ans don't need to speculate anymore.

On my side I opened a PR lemonade-sdk/lemonade#2915 with the intend to stabilize the runner, review changes are due.

Hope we can tackle this together @0cc4m I also have the hardware for local runs but your Vulkan expertise is vital here

• simplified based on your feedback so readability is improved.

[0cc4m]

The only chance to do anything about this is to reproduce it, I personally haven't seen any regressions from that PR, otherwise I wouldn't have merged it. What's the hardware and software setup of the runner?