Skip to content

b11103

Pre-release
Pre-release

Choose a tag to compare

View all tags
@github-actions github-actions released this 22 Sep 14:52
b11103
828fdf2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

spec : support DFlash for HunyuanOCR (#28890)

  • model : add DFlash layer-input taps for HunyuanVL

DFlash speculative decoding needs the target graph to expose the residual
stream entering each layer (res->t_layer_inp[il]) - the draft model reads
those tensors to build its cross-context. Qwen3 and the other DFlash-capable
targets register them, but the Hunyuan graphs do not, so serving a DFlash
draft against a HunyuanOCR target aborts during the first graph build:

GGML_ASSERT(t_layer_inp[il] != nullptr && "layer input tensor is null")

Register the tensor at the top of the layer loop, mirroring qwen3. The
layer input is the residual stream entering layer il, i.e. the output of
layer il-1, which is what the draft's target_layers metadata refers to
(the converter writes target_layer_ids+1). hunyuan-dense.cpp reuses this
graph, so it is covered as well; hunyuan-moe has a separate graph and is
untouched.

The vector is only read when a speculative implementation enables those
layer ids, so there is no behaviour change without a draft model.

Tested with tencent/HunyuanOCR 1.5 and its DFlash draft: image requests now
run, draft acceptance is ~0.5 and the OCR output is byte-identical to the
non-speculative run.

Co-authored-by: wendadawen wendadawen@qq.com

  • convert : fix DFlash draft conversion against HunYuan targets

Converting a DFlash draft with a HunYuan target failed in two ways.

  1. DFlashModel.set_vocab() reuses the target class' vocab handling by
    calling it unbound with the draft instance, but HunYuanModel.set_vocab()
    called self._fix_special_tokens(), a method that only exists on
    HunYuanModel, so the conversion always aborted with

    AttributeError: 'DFlashModel' object has no attribute '_fix_special_tokens'

    Make the vocab helpers module-level functions taking the model
    explicitly, so they do not depend on the instance being a HunYuanModel.
    They have no other callers, so the two id lookups are folded into
    _fix_special_tokens().

  2. The delegated call runs with self.dir_model pointed at the target but
    keeps the draft's self.hparams, so config lookups inside the target's
    vocab code (the pad_token_id < 0 guard, eod_token_id) read the draft's
    config instead of the target's. That aborts on targets with
    pad_token_id = -1 (e.g. the HunyuanOCR v1.0 checkpoint) and otherwise
    writes special token ids that disagree with the target.

    Add _vocab_hparams(): it returns the target's config (with text_config
    merged to the root, as TextModel does) when the model is a draft
    converted with --target-model-dir, and the model's own hparams
    otherwise, so a normal conversion is unaffected.

Tested: converting tencent/HunyuanOCR/dflash succeeds with both the 1.5 and
the v1.0 target; converting the base model without --target-model-dir
produces a byte-identical GGUF to before.

Co-authored-by: wendadawen wendadawen@qq.com

  • convert : fix DFlash draft vocab against HunYuan targets

Switch hparams to the target config for the duration of the borrowed
set_vocab(), matching the existing dir_model swap, instead of teaching
HunYuanModel::set_vocab about draft models.

  • convert : fix HunYuan special token ids for DFlash drafts

  • convert : use load_hparams for HunYuan special token ids

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

Assets 37
Loading