Thanks, and a question about the NVFP4 slowdown

[maverde73]

Hi @clsty, I'm the author of mnemonic-proxy (translated with AI). I think you mentioned the proxy in the long thread on bgm.tv about local models on consumer hardware. If that was you: thank you. A report like yours, from someone using it for real work, matters more to a small project than anything I can measure myself.

Two things you wrote caught my attention:

• with IQ3_S GSQ-RCO the same task took almost half the time with the proxy;
• with NVFP4 GPTQ it took almost twice as long.

I'd like to understand the second one. Your fork is at 0.3.3, so you probably ran 0.3.3–0.3.6. Before 0.3.9 there was a bug that fits the symptom: when the client sent a large output limit (max_tokens; agent clients often send 32K or more), the proxy reserved all of it when checking whether the prompt fit, so it could start a new segment (and write about two minutes of handoff notes) on almost every request. Another possibility is that the strata-nvfp4 fork handles saved state or the prompt cache differently, so every segment switch meant re-reading the prompt. I don't know which, if either.

If you still have the data folder from those NVFP4 runs, these counts would tell (no conversation content needed):

grep -c '"event": "request"' data/journal.jsonl
grep -c '"event": "switch"' data/journal.jsonl

plus the first line the proxy printed at startup (engine, saved state, window). If many requests have a switch, it's the bug above, fixed in 0.3.9. The current version is 0.3.12 (pip install -U mnemonic-proxy).

One more question: may I quote your "same task, almost half the time with IQ3_S" in the README's User reports section, with a link to your comment? Only if you're fine with it, and I'd include the NVFP4 result too.

No pressure on any of this. Thanks again.

[clsty]

That was me. Thanks for writing, and for the detailed explanation. The contents below are AI translated.

Those two numbers came from one comparison, not two: a 2x2x2 matrix (proxy on/off x IQ3_S/NVFP4 x
upstream Strata/strata-nvfp4), 8 runs on the same end-to-end coding task. The pattern was clear in
it — proxy noticeably faster with IQ3_S, noticeably slower with NVFP4.

What I can't give you is the evidence: the data folder for mnemonic-proxy has gone, journald has nothing left for
that service, I never noted the build, and I didn't keep the task itself. So both greps and the
startup line are gone. I'd like to rerun it on the latest mnemonic-proxy with the journal kept and the version written down when I'm available. Currently I'm tied up with another project though.

Caveat: my fork keys session identity on the client's session id, not just the message-chain hash,
so segments get cut differently. And my engine logs, per prompt, how many tokens were reused vs
re-read — which might split your two causes: many switches points to your bug, reused dropping
points to the fork re-reading state.

One more question: may I quote your "same task, almost half the time with IQ3_S" in the README's User reports section, with a link to your comment? Only if you're fine with it, and I'd include the NVFP4 result too.

Sure, feel free. And genuinely curious: how did you find that bgm.tv thread? It's almost entirely in Chinese.