Skip to content

Releases

v0.28.1

Choose a tag to compare

@yairpatch yairpatch released this 05 Nov 08:25
3b5dbd7

What's Changed

  • chore(ci): a change to what ships must bump the version by @yairpatch in #65
  • docs(readme): rewrite around what a user does, checked against the code by @yairpatch in #66
  • bench: flyweight 0.2.0 against llama.cpp on the same files and settings by @yairpatch in #67
  • chore(tests): split two semicolon statements the linter rejects by @yairpatch in #69
  • perf(prefill): dense prefill 510 -> 624 tok/s; three launches fewer per token by @yairpatch in #68
  • bench: DeepSeek-V4 speed probe and the KV-cache retrieval eval by @yairpatch in #71
  • perf(prefill): MMQ profiling switches and the Nsight findings by @yairpatch in #72
  • perf(prefill): dense prefill 624 -> 824 tok/s; the int8 MMA shape was the bound by @yairpatch in #73
  • perf(prefill): dense prefill 781 -> 824 tok/s, and no dense spill at 32K f16 KV by @yairpatch in #74
  • fix(gpu): Turing (sm_75) rejected the kernel corpus at load (#70) by @yairpatch in #75
  • perf(prefill): tensor-core attention at every context, 32-token tail tiles, small-batch routing, a GPU timeline by @yairpatch in #76
  • perf(prefill): the host prompt-cache spill moves off the request's critical path by @yairpatch in #77
  • ci: install numpy for the tests, and stop running the matrix twice per pull request by @yairpatch in #79
  • perf(prefill): the tensor-core attention crossover is 512 visible tokens, not 1024 by @yairpatch in #80
  • bench: pass 9 -- the attention crossover, and the two kernels the last 346 ms sits in by @yairpatch in #81
  • perf(prefill): hoist the DeltaNet scan's per-token scalar work into a prologue by @yairpatch in #83
  • fix(ci): the type gate, and the two Windows contract tests by @yairpatch in #84
  • fix(test): the golden packer bytes were keyed to glibc's sinf by @yairpatch in #85
  • fix(cli,bench): the two POSIX-only calls the Windows type gate reached by @yairpatch in #86
  • ci(release): let a manual run publish to PyPI by @yairpatch in #87
  • feat(images): Qwen-Image-2.1 text-to-image and editing on the diffusion tower by @yairpatch in #88
  • feat(images): Qwen-Image-2.1 editing with reference images by @yairpatch in #89
  • fix(images): invoke tower edit with condition images and avoid shadowing by @yairpatch in #90
  • feat(diffusion): implement prefix KV caching for Qwen-Image-2.1 by @yairpatch in #91
  • feat(diffusion): support GGUF DiT checkpoints for Qwen-Image-2.1 by @yairpatch in #92
  • docs, test: document --image-transformer in README and add GGUF edit test by @yairpatch in #93
  • feat(ui): 1-click 'Edit this' button in Studio, documentation, and GGUF edit test by @yairpatch in #94
  • feat(server): scoped edit resolver and automatic transcript edit repair by @yairpatch in #95
  • feat(native): Sylvester-Hadamard outlier suppression on Q8 KV cache and prefill query invariance by @yairpatch in #96
  • feat(kv): asymmetric K8V4 KV cache with Sylvester-Hadamard outlier suppression by @yairpatch in #97
  • fix(cli): restore missing NativeV2InferenceService import in _serve by @yairpatch in #98
  • feat(kv): physical page table indirection (P=64) for paged KV cache allocation by @yairpatch in #99
  • feat(attention): split-KV decode attention for long-context acceleration by @yairpatch in #100
  • fix(ci): robust flyweight_gpu_available probe and skip guards on non-GPU runners by @yairpatch in #101
  • fix(ci): check AVX-VNNI in q4_gemm contract and fix mtime resolution in hf_loader test by @yairpatch in #102
  • feat(bench): quality harness for DeltaNet/attention kernel variants by @yairpatch in #103
  • feat(cpu): direct-from-packed IQ3_S fold for the int16 rows path by @yairpatch in #104
  • feat(cpu): native host twins for attention gate, sigmoid router, delta rows by @yairpatch in #105
  • feat(cpu): direct-from-packed folds for all IQ2/IQ3 gate/up types by @yairpatch in #106
  • Admit IQ1_M CPU experts and IQ2_S/IQ1_M routed MMQ by @yairpatch in #107
  • Remove the unused Hadamard KV cache paths by @yairpatch in #108
  • feat(prefill): route Q8_0 experts through MMQ by @yairpatch in #109
  • Default the IQ gate/up direct fold on by @yairpatch in #110
  • Scale prefill streaming from PCIe bandwidth by @yairpatch in #111
  • Share the hybrid expert step between decode drivers by @yairpatch in #112
  • Drop the expert-history sidecar by @yairpatch in #113
  • Serve DeepSeek-V4 on the GPU: FP4 experts, cache, and attention by @yairpatch in #114
  • Keep dense IQ4_NL and NVFP4 in place, and run Muse attention in f32 by @yairpatch in #115
  • Support Codex tool flows with Muse Glimmer by @yairpatch in #116
  • Optimize sliding-window and turbo prefill by @yairpatch in #117
  • Keep Flash-Next from decoding each cache row once per query head by @yairpatch in #118
  • Add Q2_0 dense matvec and make dense requant opt-in by @yairpatch in #119
  • Speed Flash-Next hybrid expert decode by @yairpatch in #120
  • Gather IQ1_M embeddings packed, and charge snapshots and KV stage in the spill check by @yairpatch in #121
  • Make MTP verify and K-quant MoE experts faster by @yairpatch in #122
  • Add -ngl / --n-gpu-layers (dense and MoE) by @yairpatch in #123
  • Placement report at startup + flyweight plan by @yairpatch in #124
  • Auto expert paging: staged under plain hybrid decode by @yairpatch in #125
  • Placement report, staged auto paging, and -ngl all on MoE uses the expert cache by @yairpatch in #127
  • Make MTP faster than plain decode, including under serve by @yairpatch in #129
  • Run on AMD RDNA GPUs through ROCm (--backend rocm) by @yairpatch in #130
  • Get CI green again: mypy, stale tests, verbose Windows build by @yairpatch in #131
  • Stage mismatched K/V cache types instead of attending row by row by @yairpatch in #132
  • Keep the prompt cache on disk across restarts (--cache-dir) by @yairpatch in #133
  • Pin the first prefill checkpoint at the end of the system prompt by @yairpatch in #134
  • Tensor-core decode attention for turbo caches and 6-way groups by @yairpatch in #135
  • RDNA3 matrix cores for prompt projections: the _wmma twins by @yairpatch in #136
  • Run LFM2 and LFM2-MoE (LFM2.5-24B-A2B) on the native runtime by @yairpatch in #138
  • Direct expert paging when FLYWEIGHT_GPU_MISS puts uploads on the decode path by @yairpatch in #139
  • Run Nemotron 3.5 Bolt (nemotron_h_moe) on the native runtime by @yairpatch in #140
  • Build the Nemotron prefill experts under MSVC; clear the CI lint by @yairpatch in #141
  • Accelerate Nemotron H MoE across prefill and decode with strict parity by @yairpatch in #142
  • Stop PR #142's Nemotron changes from breaking the shared expert paths by @yairpatch in #143
  • Settle Nemotron's int8 decode, test the released formats, close the audit's loose ends by @yairpatch in #144
  • Add Raon-1: correct output, batched prefill, device decode with an expert cache by @yairpatch in https://gi...
Read more

flyweight 0.2.0

Choose a tag to compare

@github-actions github-actions released this 18 Sep 07:11
1064d26

What's Changed

  • V2 native runtime by @yairpatch in #1
  • feat(runtime): add K2-Horizon architecture support by @yairpatch in #2
  • feat(server,web): support K2-Horizon thinking tags in reasoning strea… by @yairpatch in #3
  • chore(repo): untrack the profiler dumps and captured stderr by @yairpatch in #4
  • test(native): parity cases for the three K2-Horizon host kernels by @yairpatch in #7
  • docs(repo): contribution scaffolding for outside contributors by @yairpatch in #5
  • feat(server): support K2-Horizon Q8_0 quant and <|ifm|im_end|> EOS te… by @yairpatch in #6
  • feat(runtime): K2-Horizon MoVA 36B-A4B -- MoE blocks, softplus gate, … by @yairpatch in #8
  • feat(runtime): value-expert prefetch and direct paging for K2-Horizon… by @yairpatch in #9
  • perf(runtime): K2-Horizon attention takes the tensor-core and fused decode paths by @yairpatch in #10
  • perf(attention): tensor-core GQA decode kernel for 128-dim heads in 4-way groups by @yairpatch in #11
  • Fix/k2 kernel parity cases by @yairpatch in #12
  • fix(prefill): bound the tensor-core score block by heads, not by the GQA group by @yairpatch in #13
  • ci: release wheel workflow for PyPI publishing by @yairpatch in #14
  • ci: move release workflow into place by @yairpatch in #15
  • Revert "ci: move release workflow into place" by @yairpatch in #16
  • feat(server): K2-Horizon native tool calls by @yairpatch in #17
  • ci: move the release workflow into .github/workflows by @yairpatch in #18
  • ci(release): create the GitHub release page with the wheels attached by @yairpatch in #19
  • build: publish the distribution as flyweight-llm by @yairpatch in #20
  • fix(runtime): honor a --context-window the caller typed instead of shrinking it by @yairpatch in #21
  • fix(runtime): rewind attention-only KV to a shared prefix on every such architecture by @yairpatch in #22
  • fix(server): close a forced thinking block with the tag that opened it by @yairpatch in #23
  • test: skip the four slow parity tests by default, and run the suite in parallel by @yairpatch in #24
  • feat(server): a request log with structure, colour and an off switch by @yairpatch in #25
  • fix(server): keep the column grid when a line is not an ordinary request by @yairpatch in #26
  • Feat/server log by @yairpatch in #27
  • fix(types): clean mypy under the version CI installs by @yairpatch in #28
  • fix(server): honor a request's max_tokens and scope the thinking cap to /v1/messages by @yairpatch in #29
  • Fix/ada kernel load by @yairpatch in #30
  • fix(server): a client that goes away on Windows is not a 500 by @yairpatch in #31
  • fix(sampler): cap temperature inside a tool call, for every client by @yairpatch in #32
  • fix(tokenizer): Qwen checkpoints get the pre-tokenizer their GGUF asks for by @yairpatch in #33
  • fix(tokenizer): K2-Horizon and Bailing GGUFs get their own splitters too by @yairpatch in #34
  • perf(cpu): whole-block VNNI dot for IQ1_S experts by @yairpatch in #35
  • fix(dense-spill): one OpenMP team per spilled block, Q3_K default recorded, Gemma 4 excluded by @yairpatch in #36
  • perf(qwen4exp): capture the routed delta layer front as one graph; bl… by @yairpatch in #37
  • perf(cpu): IQ3_S experts on the whole-block VNNI dot by @yairpatch in #38
  • perf(prefill): open the expert stream on qwen4exp -- pinned mirror, shared staging, int8 IQ1_S rows, routed defaults; fix block_reduce_sum race by @yairpatch in #39
  • perf(prefill): step the auto stream arena down before disabling it by @yairpatch in #40
  • fix: the runtime loads and compiles on machines CI never sees (no-AVX-512 CPUs, older drivers, older toolkits) by @yairpatch in #41
  • fix(gpu): compile kernels to a device cubin, not PTX, so a driver older than the toolkit still loads them by @yairpatch in #42
  • fix(gpu): target the newest architecture NVRTC knows when it predates the GPU by @yairpatch in #43
  • perf(decode): prompt-lookup self-speculation for qwen4exp; fix MTP replay rollback discarding sampler choices by @yairpatch in #44
  • fix(leaks): close four small release gaps found by the memory audit by @yairpatch in #45
  • Fix/load time avx512 by @yairpatch in #47
  • chore(repo): move bench and check scripts out of the repository root by @yairpatch in #48
  • Show prefill progress in the chat UI instead of a blinking cursor by @yairpatch in #49
  • fix(vision): reserve the tower's workspace at load, not on the first image by @yairpatch in #50
  • Reserve the vision workspace at load, and document Flash-Next vision by @yairpatch in #51
  • fix(vision): the tower check called a property by @yairpatch in #52
  • fix(server): an effort level the checkpoint never named failed the render by @yairpatch in #53
  • fix(web): the effort picker offered levels the checkpoint does not have by @yairpatch in #54
  • fix(server): preserve_thinking false took the reasoning the model sti… by @yairpatch in #55
  • fix(server): the off switch was spelled in a field nobody read by @yairpatch in #56
  • perf(qwen): IQ3_S dense projections decoded a weight at a time by @yairpatch in #57
  • feat(qwen): Qwen3.8-Flash-Next packed as Q2_K had no kernels to run on by @yairpatch in #58
  • feat(diffusion): Z-Image-Turbo images and MiniMax-H3 video on the native engine by @yairpatch in #59
  • fix(qwen4exp): QSA selection on turbo KV caches by @yairpatch in #60
  • feat(qwen4exp): run the GSQ-RCO checkpoints; exact int16 prefill GEMM by @yairpatch in #61
  • perf(cpu): hybrid Intel parts ran IQ1_S experts on the float path and decode on E-cores by @yairpatch in #62
  • docs(plans): status line under every plan title, real names for three by @yairpatch in #63
  • fix(ci): workspace contract asserts the old delta floor; MSVC has no sincosf by @yairpatch in #64

New Contributors

Full Changelog: https://github.com/yairpatch/flyweight/commits/v0.2.0