feat(models): complete DeepSeek-VL2 VLM port#660
Merged
Conversation
…ncoder) Partial: DeepSeekV2Model.forward_from_embeds/embed_tokens_forward/num_layers (additive) + src/vision/encoders/deepseek_vl2.rs SigLIP tower. NOT compiled/validated. Remaining: downsample_mlp_gelu projector + tile-mosaic assembly wrapper, candidate-resolution processor, loader, runtime wiring, real-checkpoint validation. Scoping in memory project_deepseek_vl2_port_533.
Add DeepSeek-VL2 (deepseek_vl_v2): a SigLIP-style ViT tower (fused-qkv full attention, learned position embedding, GELU-tanh MLP, trailing LayerNorm), the downsample_mlp_gelu projector (2x2 space-to-depth then a two-layer GELU MLP), a candidate-resolution tiling processor (LLaVA-style best-fit selection, mean-color padding, global thumbnail plus local tiles), and the 2D tile mosaic (per-row image_newline columns and a view_separator, ordered [global, view_separator, local]) that scatters into the <image> placeholder positions. The text backbone reuses deepseek_v2.rs unchanged: config.json omits the MLA head dims, so the ModelArgs serde defaults match DeepSeek-V2 exactly, and the checkpoint ships pre-fused switch_mlp experts. The <image> id is 100003 and each placeholder expands to the per-image mosaic token count. Validated on mlx-community/deepseek-vl2-small-4bit: coherent text generation and correct image description on both the single-tile (421 tokens) and 2x2 grid (1023 tokens) paths, with 9 unit tests and 4 real-model parity tests passing. Also clears two pre-existing clippy findings surfaced by a newer local toolchain: a needless return in build.rs (the CUDA recipe below it is cfg(not(macos)), so nothing runs after on macOS) and a runtime assertion on constants in switch_layers.rs (moved into a const block).
inureyes
added a commit
that referenced
this pull request
Jul 6, 2026
…s) (#673) Full text (169 models) and VLM re-benchmark on mlxcel 0.3.3 after the five-family VLM port batch (#660-#664), the slice_update/causal-mask/downloader fix batch (#666-#668), and the qwen3.5-MoE sanitize fix (#671). Adds metal_m1ultra_2026-07-06.csv, metal_m1ultra_vlm_2026-07-06.csv, the two post-#671 single-model re-bench CSVs (qwen3.5/3.6-35b-a3b back at 82.25/78.84 tok/s), and per-model mlx-lm/mlx-vlm baselines for 21 newly added models (13 measured, 8 python-side FAIL:warmup), plus the previously uncommitted June baseline CSVs (2026-06-13/14 singles and the 2026-06-16 repfresh pair). Versus 2026-06-15 there are no decode regressions; the qwen VLM family and small text models are 10-16% faster from the intervening fused-MoE and related work. The sweep caught issue #670 (qwen3.5/3.6-MoE silently failing to load since #588), fixed the same day in #671. New-model baseline highlights: ernie-4.5-vl 160% of mlx-vlm decode, deepseek-vl2 112%, idefics3 119%, qwen3-next-80b 118% of mlx-lm; mistral-small-4-119b runs only under mlxcel. Documentation: model_tests_m1ultra.md gets 110 refreshed rows, 14 new VLM rows, 9 new text rows, and updated environment/run-over-run notes; model_tests.md index registers the new CSVs. Also updates the ernie4_5_moe_vl and hunyuan_vl parity tests to the lowercase local model-directory names so the suites keep finding their checkpoints.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Completes the DeepSeek-VL2 (
deepseek_vl_v2) port, building on the WIP SigLIP tower and thedeepseek_v2embeds-forward helpers.src/vision/encoders/deepseek_vl2.rs): SigLIP-style no-CLS ViT (patch conv folded to a linear, learnedpos_embed, fused-qkv full attention, GELU-tanh MLP, trailing LayerNorm). Pixel activations are cast to the tower weight dtype so a bf16 pixel batch meets the f16 tower.src/vision/connectors/deepseek_vl2.rs):downsample_mlp_gelu(2x2 space-to-depth with channel-outermost feature order, then Linear, GELU, Linear).src/vision/processors/deepseek_vl2.rs): candidate-resolution best-fit tiling (max effective area, min-wasted tie-break), mean-color padding, a global thumbnail plus local tiles.src/vision/deepseek_vl2.rs): the 2D tile mosaic (image_newlinecolumns,view_separator, order[global, view_separator, local]) scattered into the<image>positions, over thedeepseek_v2MoE decoder reused unchanged.vlm_deepseek_vl2.rs, detection, runtime dispatch, prompt expansion, model registration, and docs.The text backbone reuses
deepseek_v2.rswith no changes:config.jsonomits the MLA head dims, so theModelArgsserde defaults match DeepSeek-V2, and the checkpoint ships pre-fusedswitch_mlpexperts. The<image>id is 100003 (the tokenizer's<image>; the config carries noimage_token_index).Validation
Real checkpoint
mlx-community/deepseek-vl2-small-4bit:9 co-located unit tests and 4 real-model parity tests (
tests/deepseek_vl_v2_parity.rs) pass. Release build (metal,accelerate),cargo fmt, andcargo clippy --all-targets -- -D warningsare clean.Drive-by
Clears two pre-existing clippy findings surfaced by a newer local toolchain: a needless
returninbuild.rs(the CUDA recipe after it iscfg(not(macos)), so nothing runs after on macOS) and a runtime assertion on constants inswitch_layers.rs(moved into aconstblock).Closes #533