Repository navigation
Deployments and workloads — help shape Penny Royal’s optimization priorities #154
Replies: 4 comments 2 replies
|
Hi John - appreciate your work here! In general I am very happy with it since this superseded my previously preferred Qwen 3.5 122B Q5 as a daily driver. Two things I hit: stray single Chinese characters in replies, which the Q5 llama.cpp build never showed (both uncensored versions of the model). I was able to suppress this via system prompt. Apart from this, only the startup time (after cold start/ reboot of my workstation) could be described as quite slow (takes 4 to 5 minutes). Hardware: Disclaimer: Everything below this line was written by the model itself, from my logs and /metrics. My window is 26 h, since recreating the container for 3.1.6 truncated the log. VRAM accounted from /metrics: weights 70.97 + KV pool 13.72 + graphs 0.54 = 85.23 GB, 4.08 GB free at startup. OS / container / version: Ubuntu 26.04.1, kernel 7.0.0-42, driver 580.178.04, Model: OrcaRouter-Qwen3.8-Flash-Next-Uncensored NVFP4 (the uncensored conversion, Workload / clients: one human, three paths on one model. OpenWebUI chat Measured: decode 200 tok/s token-weighted over 800 s. TTFT median 0.5 s, p75 0.75, What I would value:
|
I have. It's rare but I've seen it a few times. Like in the middle of a sentence in English, it uses Chinese characters. When I ask the model about it, it explains that it's just slip, and the Chinese makes sense in context. Like e..g., the one I saved said 搜索结果 = "search results" in the middle of a sentence talking about search results. I'm currently on 3.0.1, and I haven't seen them in a while. I'm serving RadixArk-Qwen3.8-Flash-Next-NVFP4. I've just assumed it's a model/training slip rather than inference engine. Chinese is a big part of its training set, so I wasn't surprised. |
FWIW, I'll use different languages in different sessions, so I have this instead: **Language matching:** Always respond in the same language the user uses. If they write in
English, reply in English. If they write in Portuguese, reply in Portuguese. Do not default to
English — mirror the user's language. The trigger is the language of the
user's *messages*, never the language of task material (skill content, code, documents,
or product strings in another language) — work in Portuguese all day, still reply in English
to an English message.It mostly works, but it's not perfect.
Thanks, I'll try adding this here. |
|
Thanks for this fork - it's been running super fast for me. Hardware:
It has been rock-solid, so not many requests at the moment. Findings:
LMK if you need a TP2 tester. |
Uh oh!
There was an error while loading. Please reload this page.
If you're running Penny Royal, share your deployment and workload here. Knowing what hardware, models, context sizes, concurrency, and workloads people actually use helps me decide where future performance, memory, serving, and compatibility work would make the most difference.
Coding, agents, chat, RAG, research, serving, or something else — a few lines about your actual use are useful.
Copy this if helpful, or describe your setup in your own words. Estimates and partial answers are welcome.
For context size, the amount of input or conversation history you usually use is especially helpful. Approximate ranges are fine. Feel free to mention what already works well for your workload, too.
All reactions