Community GGUF dialect compat for ds4f-mxfp4: dense BF16/F32/Q6_K load-time conversion - #664
Open
nexus-cw wants to merge 8 commits into
Open
Community GGUF dialect compat for ds4f-mxfp4: dense BF16/F32/Q6_K load-time conversion#664nexus-cw wants to merge 8 commits into
nexus-cw wants to merge 8 commits into
Conversation
Community llama.cpp-dialect GGUFs of DeepSeek V4 Flash omit eight deepseek4.* keys the loader hard-requires. Derive the hc, lora and hash-layer values from keys the community dialect does carry, keeping native GGUFs on the exact required path.
Community GGUF exports use llama.cpp tensor naming that differs from ds4 canonical names in about twenty places, including missing suffixes and renamed attention and expert tensors. Add an alias table consulted only when the canonical lookup misses, so native GGUFs are unaffected.
Some community GGUFs omit deepseek4.vocab_size. When the key is missing, derive vocab size from the length of tokenizer.ggml.tokens instead of failing the load.
Community GGUFs carry dense deepseek4 tensors as BF16 or Q6_K, types the runtime kernels do not accept for the dense path. Add a load-time conversion to F16, growing the mmap allocation as needed, and add the BF16 tensor type to the type table.
…ed tensors Extend the load-time dense conversion to F32 inputs and fix the cuda_model_range bookkeeping so SSD streaming addresses converted tensors at their post-conversion size and offset.
dequantize_row_q6_K wrote every block of a multi-block row to the first 256 output slots, so only the last block survived. The original routed-expert caller always passed a single block and never saw this, but the load-time dense conversion dequantizes whole tensors and lost all but the final block, zeroing most of the attention LoRA weights. Apply the per-block output offset. The decode-graph type guard this fix originally shipped with is already present on this branch, so only the dequantizer fix is carried here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
|
Added two README lines to the Community GGUF dialects subsection covering this PR: dense BF16/F32/Q6_K tensors are accepted and converted to F16 at load time. Docs only, no code change (the subsection itself flows in from #662, on which this PR is stacked). 🤖 Generated with Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Addresses #663 (part 2 of 2). Depends on #662 and is stacked on its
branch; the first three commits here are #662. Adds the dense tensor type
side of community GGUF dialect compat: with both PRs, llama.cpp-dialect
GGUFs like
bullerwins/DeepSeek-V4-Flash-0731-MXFP4_MOE-Q8_0load and runon
ds4f-mxfp4.Changes
Three commits on top of #662,
ds4.cplus a smallds4_cuda.cuchange:DS4_TENSOR_BF16 = 30to the type table and converts dense BF16 and Q6_K tensors to F16 at
load, growing the mmap allocation as needed. Native GGUFs never enter
the conversion path.
conversion to F32 dense tensors and fixes cuda_model_range bookkeeping
so SSD streaming addresses converted tensors at their post-conversion
size and offset.
dequantize_row_q6_Kwrote every block of a multi-block row to the first 256 output slots;
only the last block survived. The single-block expert caller never hits
this, but whole-tensor conversion does, and it zeroed most of the
attention LoRA weights. The decode-graph qkv type guard this fix
originally shipped with is already present on the current branch tip,
so only the dequantizer fix is carried.
Testing
make cpuat the branch tip: clean build, no new warnings beyond the twopre-existing
rax.cones.In production this stack serves the bullerwins GGUF on a GB10 CUDA node
with
--ssd-streaming, several weeks of runtime, 12/12 on our evalbattery.