Skip to content

TensorFold 0.6.5: Qwen3.6 drafts with its own MTP layer on Macs, Nemotron on CUDA drafts by measured cost, API keys

Choose a tag to compare

@ashhart ashhart released this 03 Oct 17:51
· 1674 commits to main since this release
  • Qwen3.6-35B-A3B drafts with its own MTP layer on Macs, as it already does on CUDA. Chains of up to four drafts
    are verified in the lane rounds, alone or with other streams. Each round's depth comes from the Mac's measured
    costs, and a round stays plain where drafts would not pay. On an M3 Ultra, greedy, against --no-drafts: short
    prose 1.46x (173 against 119 tok/s), short code 2.05x (245 against 119), thinking on 1.65x, and 1.34x at a
    28,400-token prompt with the same time to first token. Against mlx-vlm 0.7.4 with its MTP drafter that is 1.45-1.73x.
    Every reply's tokens matched plain decoding, greedy and sampled. DFlash v1 stays available with
    --drafter z-lab/Qwen3.6-35B-A3B-DFlash.
  • Nemotron on CUDA drafts as deep as its rows pay for. The MTP head drafts up to 15 levels, one CUDA graph each,
    and a chain stops where the next verify row would cost more time than its draft is expected to save. Window and
    level costs are measured at startup on each GPU, so every card drafts as deep as its own rows allow. Against
    0.6.4's default: about 6% faster on one DGX Spark and on two, and 7% on an RTX PRO 6000. --mtp-confidence sets a
    fixed floor instead (closes #136). Thanks to @mrpmorris for the proposal.
  • Nemotron on M5 Macs: a verify window reads each routed expert once from two rows (eight before), so one stream
    decodes 1-3% faster at three to five lanes, with the same tokens.
  • API keys on both servers. --api-key (repeatable), --api-key-file (one key, or label: key, a line; reread
    on SIGHUP) or TENSORFOLD_API_KEY. Clients send Authorization: Bearer <key> or x-api-key. /health stays
    open, and /metrics needs a key unless --metrics-open.
  • Thinking stays the chat template's default, and the server says so: one startup line names --no-thinking, and
    a reply that reaches max_tokens before it leaves its think block gets a warning line.
  • Fixes: the test suite collects on Macs that have PyTorch but not Triton, and a CUDA CLI test now expects
    Nemotron to take --mtp-confidence.
  • Every reply is still exact. Drafted output equals "draft": false on every engine this release touches.