Skip to content

feat(models): complete DeepSeek-VL2 VLM port#660

Merged
inureyes merged 2 commits into
mainfrom
feat/issue-533-deepseek-vl2
Jul 5, 2026
Merged

feat(models): complete DeepSeek-VL2 VLM port#660
inureyes merged 2 commits into
mainfrom
feat/issue-533-deepseek-vl2

Conversation

@inureyes

@inureyes inureyes commented Jul 5, 2026

Copy link
Copy Markdown
Member

Summary

Completes the DeepSeek-VL2 (deepseek_vl_v2) port, building on the WIP SigLIP tower and the deepseek_v2 embeds-forward helpers.

  • Vision tower (src/vision/encoders/deepseek_vl2.rs): SigLIP-style no-CLS ViT (patch conv folded to a linear, learned pos_embed, fused-qkv full attention, GELU-tanh MLP, trailing LayerNorm). Pixel activations are cast to the tower weight dtype so a bf16 pixel batch meets the f16 tower.
  • Projector (src/vision/connectors/deepseek_vl2.rs): downsample_mlp_gelu (2x2 space-to-depth with channel-outermost feature order, then Linear, GELU, Linear).
  • Processor (src/vision/processors/deepseek_vl2.rs): candidate-resolution best-fit tiling (max effective area, min-wasted tie-break), mean-color padding, a global thumbnail plus local tiles.
  • Wrapper (src/vision/deepseek_vl2.rs): the 2D tile mosaic (image_newline columns, view_separator, order [global, view_separator, local]) scattered into the <image> positions, over the deepseek_v2 MoE decoder reused unchanged.
  • Loader / wiring: vlm_deepseek_vl2.rs, detection, runtime dispatch, prompt expansion, model registration, and docs.

The text backbone reuses deepseek_v2.rs with no changes: config.json omits the MLA head dims, so the ModelArgs serde defaults match DeepSeek-V2, and the checkpoint ships pre-fused switch_mlp experts. The <image> id is 100003 (the tokenizer's <image>; the config carries no image_token_index).

Validation

Real checkpoint mlx-community/deepseek-vl2-small-4bit:

  • Text-only: "What is the capital of France?" produced "The capital of France is Paris."
  • Solid image: "What is the main color?" produced "The main color of this image is orange."
  • Shapes image (448px, 2x2 grid, 1023 image tokens): produced "three geometric shapes: a red circle, a blue square, and a green triangle."

9 co-located unit tests and 4 real-model parity tests (tests/deepseek_vl_v2_parity.rs) pass. Release build (metal,accelerate), cargo fmt, and cargo clippy --all-targets -- -D warnings are clean.

Drive-by

Clears two pre-existing clippy findings surfaced by a newer local toolchain: a needless return in build.rs (the CUDA recipe after it is cfg(not(macos)), so nothing runs after on macOS) and a runtime assertion on constants in switch_layers.rs (moved into a const block).

Closes #533

inureyes added 2 commits July 5, 2026 00:27
…ncoder)

Partial: DeepSeekV2Model.forward_from_embeds/embed_tokens_forward/num_layers (additive) + src/vision/encoders/deepseek_vl2.rs SigLIP tower. NOT compiled/validated. Remaining: downsample_mlp_gelu projector + tile-mosaic assembly wrapper, candidate-resolution processor, loader, runtime wiring, real-checkpoint validation. Scoping in memory project_deepseek_vl2_port_533.
Add DeepSeek-VL2 (deepseek_vl_v2): a SigLIP-style ViT tower (fused-qkv full attention, learned position embedding, GELU-tanh MLP, trailing LayerNorm), the downsample_mlp_gelu projector (2x2 space-to-depth then a two-layer GELU MLP), a candidate-resolution tiling processor (LLaVA-style best-fit selection, mean-color padding, global thumbnail plus local tiles), and the 2D tile mosaic (per-row image_newline columns and a view_separator, ordered [global, view_separator, local]) that scatters into the <image> placeholder positions. The text backbone reuses deepseek_v2.rs unchanged: config.json omits the MLA head dims, so the ModelArgs serde defaults match DeepSeek-V2 exactly, and the checkpoint ships pre-fused switch_mlp experts. The <image> id is 100003 and each placeholder expands to the per-image mosaic token count. Validated on mlx-community/deepseek-vl2-small-4bit: coherent text generation and correct image description on both the single-tile (421 tokens) and 2x2 grid (1023 tokens) paths, with 9 unit tests and 4 real-model parity tests passing. Also clears two pre-existing clippy findings surfaced by a newer local toolchain: a needless return in build.rs (the CUDA recipe below it is cfg(not(macos)), so nothing runs after on macOS) and a runtime assertion on constants in switch_layers.rs (moved into a const block).
@inureyes inureyes added type:enhancement New features, capabilities, or significant additions area:models Model architectures, weights, loading, metadata priority:medium Medium priority labels Jul 5, 2026
@inureyes inureyes merged commit c257af7 into main Jul 5, 2026
5 checks passed
@inureyes inureyes deleted the feat/issue-533-deepseek-vl2 branch July 5, 2026 06:15
inureyes added a commit that referenced this pull request Jul 6, 2026
…s) (#673)

Full text (169 models) and VLM re-benchmark on mlxcel 0.3.3 after the five-family VLM port batch (#660-#664), the slice_update/causal-mask/downloader fix batch (#666-#668), and the qwen3.5-MoE sanitize fix (#671). Adds metal_m1ultra_2026-07-06.csv, metal_m1ultra_vlm_2026-07-06.csv, the two post-#671 single-model re-bench CSVs (qwen3.5/3.6-35b-a3b back at 82.25/78.84 tok/s), and per-model mlx-lm/mlx-vlm baselines for 21 newly added models (13 measured, 8 python-side FAIL:warmup), plus the previously uncommitted June baseline CSVs (2026-06-13/14 singles and the 2026-06-16 repfresh pair). Versus 2026-06-15 there are no decode regressions; the qwen VLM family and small text models are 10-16% faster from the intervening fused-MoE and related work. The sweep caught issue #670 (qwen3.5/3.6-MoE silently failing to load since #588), fixed the same day in #671. New-model baseline highlights: ernie-4.5-vl 160% of mlx-vlm decode, deepseek-vl2 112%, idefics3 119%, qwen3-next-80b 118% of mlx-lm; mistral-small-4-119b runs only under mlxcel. Documentation: model_tests_m1ultra.md gets 110 refreshed rows, 14 new VLM rows, 9 new text rows, and updated environment/run-over-run notes; model_tests.md index registers the new CSVs. Also updates the ernie4_5_moe_vl and hunyuan_vl parity tests to the lowercase local model-directory names so the suites keep finding their checkpoints.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:models Model architectures, weights, loading, metadata priority:medium Medium priority type:enhancement New features, capabilities, or significant additions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(models): port DeepSeek-VL2 (MoE text + SigLIP/SAM vision)

1 participant