Carry ggml-org#25731 (TML Inkling) merged onto b10369 - #99
Carry ggml-org#25731 (TML Inkling) merged onto b10369#99danielhanchen wants to merge 14 commits into
Conversation
Hybrid attention model: 55 sliding-window plus 11 global layers, banded content-dependent relative position bias instead of RoPE, per-layer short convolution state, fine-grained MoE (256 experts top-6 plus 2 shared), attention log-scaling past 128K, 1M context. Includes the GGML_OP_FLASH_ATTN_EXT_BANDED operator (CPU and CUDA, fused into the MMA flash attention kernel with an fp16 accumulator overflow guard), HF to GGUF conversion, chat template with typed content block parsing (interleaved thinking, narration and tool calls), mmproj vision and audio support, and backend op tests at production shapes.
# Conflicts: # ggml/src/ggml-cuda/mmq.cuh # src/llama-model-saver.cpp # src/llama-vocab.h
Upstream 910196f renamed common_chat_params::thinking_end_tag to thinking_end_tags and changed it from std::string to a vector, so data.thinking_end_tag = END_MESSAGE no longer compiles.
# Conflicts: # src/llama-arch.cpp
ggml-org#25731 stopped merging when b10369 landed pocket-tts (ggml-org#26871), which touched the same two mtmd files. Both conflicts are unions: clip-model.h: empty merge base, each side only appends structs. Kept the inkling hmlp layer and the pocket-tts seanet/flow-net blocks. clip.cpp: has_standard_layers is an and-chain of "proj_type is not X". The PR adds INKLING, upstream adds POCKETTTS_GEN. Kept both terms, so each side still excludes what it meant to exclude. additive_merge.py refuses both, correctly: each side also touched the shared closing line, so it cannot prove they are pure additions.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e3420dd68
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| mtmd_inkling_image_preproc_out mtmd_image_preprocess_inkling( | ||
| const clip_image_u8 & img, | ||
| resize_algo algo) { |
There was a problem hiding this comment.
A repo-wide search finds mtmd_image_preprocess_inkling only in this definition and its header declaration; the runtime instead constructs mtmd_image_preprocessor_inkling in mtmd.cpp and calls the separate implementation near line 1001. Keeping this second approximately 80-line preprocessing path and result type creates two subtly different implementations that can drift without affecting tests or production, so remove it and retain the existing preprocessor class.
AGENTS.md reference: AGENTS.md:L80-L82
Useful? React with 👍 / 👎.
| eos_id = int(self.hparams.get("eos_token_id", 200006)) | ||
| if eos_id < 199998: | ||
| # HF-port configs re-save generic bos/eos defaults; the real EOS lives at 199998+ | ||
| eos_id = 200006 |
There was a problem hiding this comment.
When a re-saved config supplies a high-valued but incorrect eos_token_id, such as 200010 for <|end_message|>, this condition accepts it unchanged even though the code immediately below documents 200006 as the sole end-of-generation token. The vocabulary loader also recognizes <|content_model_end_sampling|> as EOG, so such a conversion makes both markers stop generation; a response can then terminate at the first block separator, for example after its thinking block and before its visible answer. Validate or force the configured ID to the actual end-sampling token rather than accepting every ID above 199997.
Useful? React with 👍 / 👎.
| GGML_OP_FILL, | ||
|
|
||
| GGML_OP_FLASH_ATTN_EXT, | ||
| GGML_OP_FLASH_ATTN_EXT_BANDED, |
There was a problem hiding this comment.
Inserting this value in the middle of ggml_op renumbers FLASH_ATTN_BACK, SSM_CONV, and every later operation serialized through rpc_tensor.op. The RPC handshake accepts peers with the same major/minor regardless of patch version, so a new client connected to a 5.0.0 server, or an old client connected to this server, completes negotiation and then has these operations decoded as the wrong enum values. Append the new operation without shifting existing wire values, or make the protocol negotiation reject this incompatible enum layout.
Useful? React with 👍 / 👎.
|
Not needed. I resolved the conflicts on ggml-org#25731 itself, so #100 repins straight to the upstream commit instead of carrying a copy here. |
2 participants
scripts/unsloth/pr-set.jsonpinsggml-org/llama.cpp#25731(TML Inkling) at0a9841fa63. That pin stopped merging when the aged base moved tob10369, which landedmtmd: support pocket-tts (#26871). Both PRs touch the same two mtmd files, so the nightly prebuild aborts on the merge and every arch in the mix is lost, not just Inkling.Preflight run: https://github.com/unslothai/llama.cpp/actions/runs/31593981533
This branch is
0a9841fa63withb10369merged into it and the two conflicts resolved by hand, so the pin set merges again. It carries no changes of my own beyond the resolution.The conflicts
tools/mtmd/clip.cpp, inhas_standard_layers. This is an and-chain of "the projector is not X". Inkling addsPROJECTOR_TYPE_INKLING, pocket-tts addsPROJECTOR_TYPE_POCKETTTS_GEN. Resolved to the union of all four terms, so each side still excludes exactly what it meant to exclude:tools/mtmd/clip-model.h. The merge base for this hunk is empty: both sides only append new struct definitions at the same point. Inkling addsinkling_hmlp_layer, pocket-tts addsclip_seanetandclip_flow_net. Kept all three.additive_merge.pyrefuses this one and is right to, since each side also touched the shared closing brace, so it cannot prove the hunk is a pure addition.Checks
inkling_hmlp_layer,clip_seanetandclip_flow_netare each defined exactly once.PROJECTOR_TYPE_INKLINGandPROJECTOR_TYPE_POCKETTTS_GENare both still declared and still mapped inclip-impl.h.llama-server,llama-mtmd-cliandtest-samplingat-DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON -DLLAMA_BUILD_TESTS=ON, with no warnings from either resolved file.b10369inpr-set.jsonorder with this commit substituted for the old pin:#24423clean, this branch clean,#70additive,#91clean,#95clean.The base branch is
b10369itself, so the diff here is only the Inkling work, matching the arrangement used by #70 and #91.Once
ggml-org#25731is rebased upstream, or a base tag carries it, the pin should move back to the upstream commit and this branch can be deleted.