Repository navigation
Cookbook: are consumer SM89 (RTX 40-series, 16 GB) notes welcome on the Qwen3 page? #46804
Replies: 3 comments
|
I can't speak for the maintainers, but the page already sets a precedent for this. The Qwen3 intro says it covers NVIDIA, AMD, Intel Arc and Xeon CPUs ( ). Section 3.2 already has hardware-specific bullets, such as the H200 note for Qwen3-4B/8B and the Xeon CPU note (same file, lines 43 to 55). A small, measured SM89 addition fits that pattern.Where to put it:
On the max_tokens tip, check the wording before you publish it. The scheduler does reserve KV for each admitted request based on sglang/python/sglang/srt/managers/schedule_policy.py Lines 85 to 90 in 8a455f1 On validation, there is no SM89-specific rule I could find. The cookbook PR checklist (
Your plan of sgl-eval GSM8K plus bench_serving, with the settings written out, matches that. Also say which cookbook flags you changed to fit 16 GB. For the install note, link the three issues as open problems and state your own environment (driver, CUDA, glibc, SGLang version). I did not check those issues' current status here. (I looked this up in the code with help from an AI assistant; the links are to the lines I checked.) |
|
@stanleys12 Thanks, this is very helpful. I'll start with a 3.2 config-tip bullet and leave the generator alone for now. Good catch on the clip: I'll word the tip as applying to caps below |
|
@stanleys12 I measured on current main. Here are the results and the bullet I plan to open as a PR. Feedback is welcome before I do. Environment: RTX 4070 Ti SUPER (SM89, 16 GB), driver 610.88, WSL2 (kernel 6.18), glibc 2.43, sglang 1. KV pool on 16 GB. With default flags, the pool holds 16,219 tokens while 2. bench_serving. The commands are the page's 5.1.1 commands plus
At c16, the BF16 pool can't hold 16 × (input + 1000) tokens, so requests queue. FP8 KV removes most of that queueing at the cost of about 8% higher TPOT. 3. GSM8K (sgl-eval). Command:
FP8 KV scores about 1.7 points lower. The spread across repeats overlaps, and all repeats use the same 200 questions, so treat this as a small possible cost rather than a firm one. 4. max_tokens and admission (the clip you pointed out). I sent 128 short-answer chat requests of the same shape at once (distinct prompts, about 547 input tokens each), at temperature 0 and non-thinking, getting about 47 output tokens in every arm. Only the client
The 4096 and 8192 rows match, which confirms the clip. Setting the server-side env var to 1024 gives the same result as a client cap of 1024. The script is below. admit.py"""max_tokens vs admission on a small KV pool: N identical short-answer chat requests
sent at once, cache flushed first; only the client max_tokens changes between arms.
usage: admit.py <arm> <max_tokens> <n> <out.jsonl>"""
import json, random, statistics, sys, threading, time, urllib.request
arm, max_tokens, n, out = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]), sys.argv[4]
URL = "http://127.0.0.1:30000"
def prompt(i):
r = random.Random(i)
rows = "\n".join(f"Station {r.randint(100, 999)}: reading {r.randint(1, 500)} units at hour {r.randint(0, 23)}." for _ in range(30))
return f"{rows}\n\nWhich three stations reported the highest readings? Reply in one short sentence."
def post(path, body=None):
req = urllib.request.Request(URL + path, json.dumps(body).encode() if body is not None else b"", {"Content-Type": "application/json"})
return urllib.request.urlopen(req, timeout=900)
samples, stop = [], threading.Event()
def poll():
while not stop.is_set():
try:
txt = urllib.request.urlopen(URL + "/metrics", timeout=5).read().decode()
g = {k: sum(float(l.rsplit(" ", 1)[1]) for l in txt.splitlines() if l.startswith(f"sglang:{k}{{")) for k in ("num_running_reqs", "num_queue_reqs")}
samples.append(g)
except Exception:
pass
time.sleep(0.2)
res = [None] * n
def one(i):
body = {"model": "Qwen/Qwen3-8B-FP8", "messages": [{"role": "user", "content": prompt(i)}], "max_tokens": max_tokens,
"temperature": 0, "stream": True, "stream_options": {"include_usage": True},
"chat_template_kwargs": {"enable_thinking": False}}
t0 = time.time(); ttft = None; toks = None
with post("/v1/chat/completions", body) as resp:
for line in resp:
line = line.decode().strip()
if not line.startswith("data: ") or line == "data: [DONE]":
continue
d = json.loads(line[6:])
if ttft is None and d.get("choices") and d["choices"][0]["delta"].get("content"):
ttft = time.time() - t0
if d.get("usage"):
toks = d["usage"]["completion_tokens"]; ptoks = d["usage"]["prompt_tokens"]
res[i] = {"ttft": ttft, "e2e": time.time() - t0, "out": toks, "in": ptoks}
post("/flush_cache").read(); time.sleep(1)
th = threading.Thread(target=poll); th.start()
t0 = time.time()
ws = [threading.Thread(target=one, args=(i,)) for i in range(n)]
[w.start() for w in ws]; [w.join() for w in ws]
span = time.time() - t0; stop.set(); th.join()
q = lambda xs, p: sorted(xs)[min(len(xs) - 1, int(p * len(xs)))]
tt = [r["ttft"] for r in res]
row = {"arm": arm, "max_tokens": max_tokens, "n": n, "makespan_s": round(span, 2),
"ttft_p50_s": round(statistics.median(tt), 3), "ttft_p99_s": round(q(tt, 0.99), 3),
"e2e_p50_s": round(statistics.median(r["e2e"] for r in res), 3),
"mean_in_tokens": round(statistics.mean(r["in"] for r in res), 1), "mean_out_tokens": round(statistics.mean(r["out"] for r in res), 1),
"max_out_tokens": max(r["out"] for r in res),
"peak_running": max((s["num_running_reqs"] for s in samples), default=None),
"peak_queue": max((s["num_queue_reqs"] for s in samples), default=None)}
print(json.dumps(row)); open(out, "a").write(json.dumps(row) + "\n")Proposed 3.2 bullet (I left the command generator alone, as you suggested):
|
Uh oh!
There was an error while loading. Please reload this page.
I run Qwen3-8B-FP8 with SGLang on an RTX 4070 Ti SUPER (SM89, 16 GB) and would like to contribute what I measured there. Before writing anything, I wanted to check whether this hardware is in scope for the cookbook, since the current pages target datacenter GPUs. The Qwen3 page already includes Intel Arc and Xeon CPU configs, and #43551 added a hardware-specific tip, so I'm guessing a small addition could fit.
What I would add:
A launch command for Qwen3-8B-FP8 on a 16 GB card, with the flags that matter there.
One config tip: on a small KV pool, the request's max_tokens decides how many requests are admitted at once, so capping it raises concurrency.
A short pip-install note for SM89 that links the open issues: #43591 (toolchain pin), #46420 (pip libcudart link), #45066 (glibc 2.42+).
Questions:
Is SM89 / consumer hardware welcome in the cookbook?
If so, would this be best as a config-tip bullet, an entry in the command generator, or something else?
Is there a required validation? I would run sgl-eval (GSM8K) and bench_serving on the card and report the exact settings.
All reactions