chore(ci): bump the pinned Rust toolchain to 1.97.1 - #1066
Merged
Conversation
The pin sat at 1.93.1 (2026-02-11), four releases behind stable 1.97.1 (2026-07-14). Bump it and move the `dtolnay/rust-toolchain` tag in `ci.yml` with it, as the comment on that step requires. The `@stable` steps in `release.yml`, `nightly-verify.yml`, and `pipeline-parallel-ci.yml` were never building at a different version: the action runs `rustup default` and never exports `RUSTUP_TOOLCHAIN`, which `rust-toolchain.toml` overrides per directory, so every cargo invocation in the tree already resolved to the pin. `cargo fmt` produces no diff at 1.97.1, so the bump reformats nothing. Six new clippy lints fire under `-D warnings` and are fixed here, all mechanical and behavior-preserving: question_mark memory_estimate.rs, sanitize.rs collapsible_match chat_request.rs for_kv_map request_router_tests.rs unnecessary_cast dbrx_tests.rs unneeded_wildcard_pattern pipeline_remote_real_models.rs
inureyes
added
type:chore
inureyes
added a commit
that referenced
this pull request
Aug 7, 2026
…alidation (#1071) ## Summary Capstone of the Florence-2 epic #850: wires the engine (#852), DaViT tower (#853), fusion (#854), and processor (#855) into model detection, registration, loading, and the CLI generate path, then validates caption, OCR, and detection on the real `models/Florence-2-base-ft-bf16` checkpoint. No new model math; this makes the family reachable. Florence-2 is an encoder-decoder (seq2seq) VLM: the BART decoder generates by cross-attending to cached encoder output over the fused image+prompt sequence, with a dual self/cross-attention cache (`Florence2SeqCache`) that does not fit the decoder-only `KVCache` contract every other `LoadedModel` family runs on. The integration therefore follows the DiffusionGemma phase-1 precedent: `LoadedModel::Florence2VLM` exists as a first-class variant, the CLI routes it to a dedicated task pipeline before the autoregressive loop, and `mlxcel-server` refuses the checkpoint at startup with a named error until a seq2seq worker path exists. ## What changed - `src/models/florence2/runtime.rs` (new): `Florence2VlmModel` (fused model + processor as one loadable unit), `run_task` (prompt expand, 768x768 preprocess, greedy seq2seq decode, coordinate parse against the original image size, returns the generated-token count for stats), `parse_task_prompt` for the CLI `-p` syntax, and an honest-minimal `LanguageModel` impl (teacher-forced BART forward with the shift-right built on-device, `make_caches` empty, `supports_batching`/`supports_padded_prefill` false) for trait completeness only. - `src/models/florence2/florence2_runtime_tests.rs` (new): prompt-parse matrix (marker/bare/case-insensitive forms, `<loc_*>` region input without separating space, unknown/unclosed/empty errors listing valid markers, input-validation deferral to `Florence2Task::expand`). - `src/models/detection.rs`: `"florence2"` arm returning `ModelType::Florence2VLM`; `src/models/detection_tests.rs` covers it with the real config shape (empty `vision_config.model_type`). - `src/models/mod.rs`: `ModelType::Florence2VLM` variant, `ALL_MODEL_TYPES` entry, `metadata()` arm ("Florence-2 (DaViT + BART seq2seq, task prompts, bf16/f16 only)", family "Other VLM"), exhaustiveness-test list entry, and re-exports for `Florence2VlmModel` / `Florence2RunOutput`. - `src/model_metadata.rs`: `Florence2VLM => { kind: Vlm, directory: Vlm, weight: None, adapter: Some(...) }` registration; adapter loading is refused with a named message. - `src/loading/vlm_florence2.rs` (new) plus the `try_load_vlm_model_from_dir` arm in `src/loading/mod.rs` and module registration in `src/loading/vlm.rs`. - `src/loaded_model.rs`: `Florence2VLM` variant and `delegate_language_model!` arm. - `src/commands/generate_florence2.rs` (new): CLI driver. Requires exactly one `--image`, rejects `--audio`/`--video`, parses the task prompt, decodes the image through `mlxcel::decode_image_payloads_with_limits` with the shared `ImageInputLimits` (decompression-bomb defense handed over by #855), runs the task, renders parsed boxes / quad boxes / polygons one instance per line in original-image pixels (a raw-text fallback distinguishes "model found nothing" from "parser rejected the answer"; `--profile` also prints the raw answer), and prints the standard throughput line. Renderer unit tests sit beside it. - `src/commands/generate.rs`: early-exit branch for `LoadedModel::Florence2VLM` in `run_generate_once`, after the DiffusionGemma / LLaDA-2 exits and before the VLM-embedding path, so the decoder-only path of every other model is untouched. - `src/server/startup.rs`: `start_server` bails for Florence-2 before any worker spawns, pointing at the CLI. Without this the decoder-only worker would load the model and serve garbage through the trait-completeness forward. - `src/distributed/tensor_parallel/inference.rs`: placeholder `fallback_architecture` arm (`"florence2"`) keeps the exhaustive dispatch table total; TP planning rejects the string as unsupported, same as Whisper/Kokoro. - `docs/supported-models.md`: Florence-2 entry documenting the architecture, the fifteen task markers, the CLI form, the server limitation, and the bf16/f16-only constraint. ## Scoping decisions and corrections to the issue text - The issue names `mlx-community/Florence-2-base-ft-bf16` "or `-4bit`" as the validation target. The `-4bit` (and 8/6/3-bit) conversions cannot work: neither the BART stack nor the DaViT tower has a quantized code path, and a packed uint32 weight reaching MLX aborts the process, so `Florence2Model::load` rejects quantized checkpoints with a named error (a deliberate #854 decision). Validation ran on the bf16 export, and `docs/supported-models.md` states the bf16/f16-only limitation plainly. Quantized support is follow-up work on the epic. - The issue says "`mlxcel list` reports florence2", but `mlxcel list` is the downloaded-model store listing (name/size/modified, no architecture column); the architecture catalog is `mlxcel arch` (alias `supported`). Both outputs are recorded below: `arch` now lists Florence-2 under Other VLM, and `list --models-dir models` shows the checkpoint. - The issue's implementation plan says "wire the seq2seq VLM decode path into `generate` / `generate_vlm`". The decode path deliberately does not run through `compute_vlm_embeddings` (`generate_vlm`): that surface produces merged input embeddings for a decoder-only prefill and cannot express an encoder pass plus cross-attention decode. The wiring point that preserves the decoder-only path for every other model is the pre-loop early exit in `run_generate_once`, the same integration shape DiffusionGemma and LLaDA-2 use. For the same reason there is no `VlmRuntimeRef::Florence2`: `is_vlm()` stays false so no image ever routes into the embedding-merge path. - The issue lists "`_tests.rs` unit tests (DaViT shapes, cross-attention, task-prompt parse)". DaViT-shape and cross-attention tests landed with #853/#852 (`florence2_davit_tests.rs`, `florence2_tests.rs`, plus the real-checkpoint parity suites under `tests/`); this PR adds the task-prompt-parse and runtime-surface tests that belong to the integration layer. - TP / distributed arch-string: added as a total-table placeholder only; a 0.23B seq2seq model has no TP plan. - Server integration is a startup-time refusal rather than an endpoint: the batched/legacy workers assume autoregressive `LanguageModel` decode, and a Florence-2 seq2seq worker loop is follow-up work on the epic (mirrors DiffusionGemma, whose server support arrived in a later phase). ## Validation Real-checkpoint CLI runs (`models/Florence-2-base-ft-bf16`, Apple Silicon Metal). All four answers agree with the byte-identical mlx-vlm parity references recorded by #855 in `tests/florence2_processor_parity.rs` (`REF_ANSWERS` / `REF_OD_BOX` / `REF_OCR_QUAD`, fixture `tests/fixtures/test_image.png`). ``` $ ./target/release/mlxcel generate -m models/Florence-2-base-ft-bf16 --image tests/fixtures/test_image.png -p "<CAPTION>" -n 100 Generating... <CAPTION> unanswerable [Generated 4 tokens in 0.20s = 20.10 tok/s] $ ./target/release/mlxcel generate -m models/Florence-2-base-ft-bf16 --image tests/fixtures/test_image.png -p "<OCR>" -n 100 Generating... <OCR> 0 [Generated 2 tokens in 0.13s = 14.82 tok/s] $ ./target/release/mlxcel generate -m models/Florence-2-base-ft-bf16 --image tests/fixtures/test_image.png -p "<OD>" -n 100 --profile Generating... <OD> poster: [0.1, 0.1, 223.7, 223.7] [Generated 7 tokens in 0.14s = 49.08 tok/s] [Raw answer] <s>poster<loc_0><loc_0><loc_998><loc_998> $ ./target/release/mlxcel generate -m models/Florence-2-base-ft-bf16 --image tests/fixtures/test_image.png -p "<OCR_WITH_REGION>" -n 100 Generating... <OCR_WITH_REGION> 0:00 PM: [0.1, 223.9, 17.1, 223.9, 17.1, 223.9, 0.1, 223.9] [Generated 13 tokens in 0.16s = 80.48 tok/s] ``` Architecture catalog and store listing: ``` $ ./target/release/mlxcel arch | grep -i florence - Florence-2 (DaViT + BART seq2seq, task prompts, bf16/f16 only) $ ./target/release/mlxcel list --models-dir models 1 model · 522.4 MiB · models NAME SIZE MODIFIED Florence-2-base-ft-bf16 522.4 MiB 1 hour ago ``` Guard rails exercised: ``` $ ./target/release/mlxcel-server -m models/Florence-2-base-ft-bf16 --port 18923 Error: Florence-2 is an encoder-decoder (seq2seq) VLM that mlxcel-server cannot serve yet. Run it through the CLI instead: mlxcel generate -m <model> --image <image> -p '<CAPTION>' (or another task marker such as <OCR> or <OD>). $ ./target/release/mlxcel generate -m models/Florence-2-base-ft-bf16 --image ... -p "describe this image" Error: -p/--prompt: prompt "describe this image" does not start with a Florence-2 task; valid markers: <OCR>, <OCR_WITH_REGION>, <CAPTION>, ... <REGION_PROPOSAL> $ ./target/release/mlxcel generate -m models/Florence-2-base-ft-bf16 -p "<CAPTION>" Error: Florence-2 is an image-task model: pass --image <path> together with a task prompt such as -p '<CAPTION>', -p '<OCR>', or -p '<OD>' ``` Tests (test-fast profile, metal,accelerate): - [x] `cargo test --lib --bins florence2` (florence2 unit suites incl. the new runtime prompt-parse and CLI renderer tests, plus `detection_tests::florence2_model_type_is_detected`): all pass - [x] `cargo test --lib every_variant` / `all_model_types` / `family_order` / `model_metadata`: all pass (registry exhaustiveness guards) - [x] `cargo clippy --profile test-fast --features metal,accelerate --lib --tests -- -D warnings`: clean - [x] `cargo fmt --check`: clean Pre-existing failure, not caused by this PR: `loading::vlm::gemma_unified::tests::unified_sanitize_quantized_split_dequant_equivalence` fails identically at current `main` (9550388) on this machine (checked out and re-run both ways); it involves only Gemma 4 Unified quantized-weight sanitize, which this change never touches, and may be related to the Rust 1.97.1 toolchain bump from #1066. Closes #856.
This was referenced Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The pin sat at 1.93.1 (2026-02-11), four releases behind stable 1.97.1 (2026-07-14). This bumps
rust-toolchain.tomland moves thedtolnay/rust-toolchaintag inci.ymlwith it, as the comment on that step requires.The
@stableworkflows were never on a different versionrelease.yml,nightly-verify.ymlandpipeline-parallel-ci.ymlinstalldtolnay/rust-toolchain@stable, which looks like it would build at a newer compiler than the pin. It does not. That action runsrustup default <toolchain>and never exportsRUSTUP_TOOLCHAIN, andrust-toolchain.tomloverridesrustup defaultper directory, so every cargo invocation in the tree already resolved to the pin. Reproduced locally: with the default toolchain set tostable,rustc --versionreports 1.93.1 inside the repository and 1.97.1 outside it.nightly-verify.ymldocuments this deliberately at lines 271-275, so those three workflows are left alone.What changed
rust-toolchain.tomlchannel1.93.1 to 1.97.1.github/workflows/ci.ymldtolnay/rust-toolchaintag, two occurrencesCONTRIBUTING.mdCHANGELOG.md[Unreleased]entrycargo fmtproduces no diff at 1.97.1, so the bump reformats nothing. Six new clippy lints fire under-D warningsand are fixed here, all mechanical and behavior-preserving:question_marksrc/execution/memory_estimate.rs,src/models/sanitize.rscollapsible_matchsrc/server/chat_request.rsfor_kv_mapsrc/distributed/disaggregated/request_router_tests.rsunnecessary_castsrc/models/dbrx_tests.rsunneeded_wildcard_patterntests/pipeline_remote_real_models.rsThe
collapsible_matchfix is the only one that changes control-flow shape: the nestedifinside theMessageContent::Partsarm becomes a match guard. When the guard does not hold the arm now falls to_ => {}, which is what the emptyifdid before, so behavior is identical.Validation
Run on Apple M5 Max, macOS 26.6, under 1.97.1:
make verify-versionsmake verify-kernel-dtype-keysmake verify-fmtmake verify-clippy(--workspace --all-targets --features metal,accelerate -- -D warnings)make verify-test(--workspace --profile test-fast)Clippy covers all five workspace members.
mlxcel-coreandmlxcel-xlacompile rather than check because of their build scripts, but the lint driver runs on them and they are green.The five test failures are pre-existing and unrelated
They are not caused by this bump. Every one of them reproduces on
mainat 1.93.1 with byte-identical values:layers::tests::chunked_causal_attention_matches_fast_causallayers::tests::chunked_query_attention_matches_unchunked_with_maskmla::decode::decode_tests::absorbed_attention_respects_an_additive_causal_maskdrifted by 0.0014691837mla::decode::decode_tests::expand_latent_reproduces_the_up_projection0.30357933044433594 != 0.3037950135765861mllama_parity::sub_max_real_tiles_keep_the_legacy_real_rows_byte_identical5.9604645e-8All five are host dependent: they pass on the M1 Ultra nightly runner (last successful
nightly-verify.ymlrun 30939291504 reports... okfor all five) and fail deterministically on M5 Max.The first four are tracked in #1065: f32 matmul and single-query SDPA lose about fp16 precision on M5. The fifth is a byte-identity assertion whose observed difference is 0.5 ULP of the output scale, the same class as #939, and is fixed separately following the precedent of #953.
The test delta introduced by this PR is zero.