feat(models): Florence-2 end-to-end integration and real-checkpoint validation

[inureyes]

Summary

Capstone of the Florence-2 epic #850: wires the engine (#852), DaViT tower (#853), fusion (#854), and processor (#855) into model detection, registration, loading, and the CLI generate path, then validates caption, OCR, and detection on the real models/Florence-2-base-ft-bf16 checkpoint. No new model math; this makes the family reachable.

Florence-2 is an encoder-decoder (seq2seq) VLM: the BART decoder generates by cross-attending to cached encoder output over the fused image+prompt sequence, with a dual self/cross-attention cache (Florence2SeqCache) that does not fit the decoder-only KVCache contract every other LoadedModel family runs on. The integration therefore follows the DiffusionGemma phase-1 precedent: LoadedModel::Florence2VLM exists as a first-class variant, the CLI routes it to a dedicated task pipeline before the autoregressive loop, and mlxcel-server refuses the checkpoint at startup with a named error until a seq2seq worker path exists.

What changed

• src/models/florence2/runtime.rs (new): Florence2VlmModel (fused model + processor as one loadable unit), run_task (prompt expand, 768x768 preprocess, greedy seq2seq decode, coordinate parse against the original image size, returns the generated-token count for stats), parse_task_prompt for the CLI -p syntax, and an honest-minimal LanguageModel impl (teacher-forced BART forward with the shift-right built on-device, make_caches empty, supports_batching/supports_padded_prefill false) for trait completeness only.
• src/models/florence2/florence2_runtime_tests.rs (new): prompt-parse matrix (marker/bare/case-insensitive forms, <loc_*> region input without separating space, unknown/unclosed/empty errors listing valid markers, input-validation deferral to Florence2Task::expand).
• src/models/detection.rs: "florence2" arm returning ModelType::Florence2VLM; src/models/detection_tests.rs covers it with the real config shape (empty vision_config.model_type).
• src/models/mod.rs: ModelType::Florence2VLM variant, ALL_MODEL_TYPES entry, metadata() arm ("Florence-2 (DaViT + BART seq2seq, task prompts, bf16/f16 only)", family "Other VLM"), exhaustiveness-test list entry, and re-exports for Florence2VlmModel / Florence2RunOutput.
• src/model_metadata.rs: Florence2VLM => { kind: Vlm, directory: Vlm, weight: None, adapter: Some(...) } registration; adapter loading is refused with a named message.
• src/loading/vlm_florence2.rs (new) plus the try_load_vlm_model_from_dir arm in src/loading/mod.rs and module registration in src/loading/vlm.rs.
• src/loaded_model.rs: Florence2VLM variant and delegate_language_model! arm.
• src/commands/generate_florence2.rs (new): CLI driver. Requires exactly one --image, rejects --audio/--video, parses the task prompt, decodes the image through mlxcel::decode_image_payloads_with_limits with the shared ImageInputLimits (decompression-bomb defense handed over by feat(models): Florence-2 processor, task prompts, and location tokens (sub of #850) #855), runs the task, renders parsed boxes / quad boxes / polygons one instance per line in original-image pixels (a raw-text fallback distinguishes "model found nothing" from "parser rejected the answer"; --profile also prints the raw answer), and prints the standard throughput line. Renderer unit tests sit beside it.
• src/commands/generate.rs: early-exit branch for LoadedModel::Florence2VLM in run_generate_once, after the DiffusionGemma / LLaDA-2 exits and before the VLM-embedding path, so the decoder-only path of every other model is untouched.
• src/server/startup.rs: start_server bails for Florence-2 before any worker spawns, pointing at the CLI. Without this the decoder-only worker would load the model and serve garbage through the trait-completeness forward.
• src/distributed/tensor_parallel/inference.rs: placeholder fallback_architecture arm ("florence2") keeps the exhaustive dispatch table total; TP planning rejects the string as unsupported, same as Whisper/Kokoro.
• docs/supported-models.md: Florence-2 entry documenting the architecture, the fifteen task markers, the CLI form, the server limitation, and the bf16/f16-only constraint.

Scoping decisions and corrections to the issue text

• The issue names mlx-community/Florence-2-base-ft-bf16 "or -4bit" as the validation target. The -4bit (and 8/6/3-bit) conversions cannot work: neither the BART stack nor the DaViT tower has a quantized code path, and a packed uint32 weight reaching MLX aborts the process, so Florence2Model::load rejects quantized checkpoints with a named error (a deliberate feat(models): Florence-2 vision-language fusion + full weight loading (sub of #850) #854 decision). Validation ran on the bf16 export, and docs/supported-models.md states the bf16/f16-only limitation plainly. Quantized support is follow-up work on the epic.
• The issue says "mlxcel list reports florence2", but mlxcel list is the downloaded-model store listing (name/size/modified, no architecture column); the architecture catalog is mlxcel arch (alias supported). Both outputs are recorded below: arch now lists Florence-2 under Other VLM, and list --models-dir models shows the checkpoint.
• The issue's implementation plan says "wire the seq2seq VLM decode path into generate / generate_vlm". The decode path deliberately does not run through compute_vlm_embeddings (generate_vlm): that surface produces merged input embeddings for a decoder-only prefill and cannot express an encoder pass plus cross-attention decode. The wiring point that preserves the decoder-only path for every other model is the pre-loop early exit in run_generate_once, the same integration shape DiffusionGemma and LLaDA-2 use. For the same reason there is no VlmRuntimeRef::Florence2: is_vlm() stays false so no image ever routes into the embedding-merge path.
• The issue lists "_tests.rs unit tests (DaViT shapes, cross-attention, task-prompt parse)". DaViT-shape and cross-attention tests landed with feat(models): Florence-2 DaViT vision backbone (sub of #850) #853/feat(models): Florence-2 BART-style seq2seq encoder-decoder engine + text core (sub of #850) #852 (florence2_davit_tests.rs, florence2_tests.rs, plus the real-checkpoint parity suites under tests/); this PR adds the task-prompt-parse and runtime-surface tests that belong to the integration layer.
• TP / distributed arch-string: added as a total-table placeholder only; a 0.23B seq2seq model has no TP plan.
• Server integration is a startup-time refusal rather than an endpoint: the batched/legacy workers assume autoregressive LanguageModel decode, and a Florence-2 seq2seq worker loop is follow-up work on the epic (mirrors DiffusionGemma, whose server support arrived in a later phase).

Validation

Real-checkpoint CLI runs (models/Florence-2-base-ft-bf16, Apple Silicon Metal). All four answers agree with the byte-identical mlx-vlm parity references recorded by #855 in tests/florence2_processor_parity.rs (REF_ANSWERS / REF_OD_BOX / REF_OCR_QUAD, fixture tests/fixtures/test_image.png).

$ ./target/release/mlxcel generate -m models/Florence-2-base-ft-bf16 --image tests/fixtures/test_image.png -p "<CAPTION>" -n 100
Generating...
<CAPTION>
unanswerable

[Generated 4 tokens in 0.20s = 20.10 tok/s]

$ ./target/release/mlxcel generate -m models/Florence-2-base-ft-bf16 --image tests/fixtures/test_image.png -p "<OCR>" -n 100
Generating...
<OCR>
0

[Generated 2 tokens in 0.13s = 14.82 tok/s]

$ ./target/release/mlxcel generate -m models/Florence-2-base-ft-bf16 --image tests/fixtures/test_image.png -p "<OD>" -n 100 --profile
Generating...
<OD>
poster: [0.1, 0.1, 223.7, 223.7]

[Generated 7 tokens in 0.14s = 49.08 tok/s]
[Raw answer] <s>poster<loc_0><loc_0><loc_998><loc_998>

$ ./target/release/mlxcel generate -m models/Florence-2-base-ft-bf16 --image tests/fixtures/test_image.png -p "<OCR_WITH_REGION>" -n 100
Generating...
<OCR_WITH_REGION>
0:00 PM: [0.1, 223.9, 17.1, 223.9, 17.1, 223.9, 0.1, 223.9]

[Generated 13 tokens in 0.16s = 80.48 tok/s]

Architecture catalog and store listing:

$ ./target/release/mlxcel arch | grep -i florence
  - Florence-2 (DaViT + BART seq2seq, task prompts, bf16/f16 only)

$ ./target/release/mlxcel list --models-dir models
1 model · 522.4 MiB · models
  NAME                          SIZE  MODIFIED
  Florence-2-base-ft-bf16  522.4 MiB  1 hour ago

Guard rails exercised:

$ ./target/release/mlxcel-server -m models/Florence-2-base-ft-bf16 --port 18923
Error: Florence-2 is an encoder-decoder (seq2seq) VLM that mlxcel-server cannot serve yet. Run it through the CLI instead: mlxcel generate -m <model> --image <image> -p '<CAPTION>' (or another task marker such as <OCR> or <OD>).

$ ./target/release/mlxcel generate -m models/Florence-2-base-ft-bf16 --image ... -p "describe this image"
Error: -p/--prompt: prompt "describe this image" does not start with a Florence-2 task; valid markers: <OCR>, <OCR_WITH_REGION>, <CAPTION>, ... <REGION_PROPOSAL>

$ ./target/release/mlxcel generate -m models/Florence-2-base-ft-bf16 -p "<CAPTION>"
Error: Florence-2 is an image-task model: pass --image <path> together with a task prompt such as -p '<CAPTION>', -p '<OCR>', or -p '<OD>'

Tests (test-fast profile, metal,accelerate):

• cargo test --lib --bins florence2 (florence2 unit suites incl. the new runtime prompt-parse and CLI renderer tests, plus detection_tests::florence2_model_type_is_detected): all pass
• cargo test --lib every_variant / all_model_types / family_order / model_metadata: all pass (registry exhaustiveness guards)
• cargo clippy --profile test-fast --features metal,accelerate --lib --tests -- -D warnings: clean
• cargo fmt --check: clean

Pre-existing failure, not caused by this PR: loading::vlm::gemma_unified::tests::unified_sanitize_quantized_split_dequant_equivalence fails identically at current main (9550388) on this machine (checked out and re-run both ways); it involves only Gemma 4 Unified quantized-weight sanitize, which this change never touches, and may be related to the Rust 1.97.1 toolchain bump from #1066.

Closes #856.