Summary
The capstone that wires Florence-2 into mlxcel's model detection, registration, and generation paths, then validates the whole model on a real checkpoint across caption + OCR + detection modes. No new model math lives here; it connects the pieces from #852, #853, #854, and #855 into the runtime and proves the integrated deliverable.
Part of epic
#850
Dependencies
#852 (seq2seq engine + text core), #853 (DaViT backbone), #854 (fusion + weight loading), #855 (processor + task prompts + location tokens)
Upstream reference
Architecture notes
Implementation plan
- Add the
"florence2" arm to src/models/detection.rs.
- Register in
src/model_metadata.rs via for_each_model_registration!.
- Wire the seq2seq VLM decode path into
generate / generate_vlm.
- Add the TP / distributed arch-string if applicable.
- Update
docs/supported-models.md.
- Add
_tests.rs unit tests (DaViT shapes, cross-attention, task-prompt parse).
- Validate on a real checkpoint (
mlx-community/Florence-2-base-ft-bf16 or -4bit) across caption + OCR + detection modes via ./target/release/mlxcel generate -m <model> --image <img> -p "<task prompt>", confirming mlxcel list reports florence2.
- Follow
docs/adding-models.md.
Acceptance criteria
Effort
MEDIUM.
Note: this sub-issue is tracked by the checklist in #850.
Summary
The capstone that wires Florence-2 into mlxcel's model detection, registration, and generation paths, then validates the whole model on a real checkpoint across caption + OCR + detection modes. No new model math lives here; it connects the pieces from #852, #853, #854, and #855 into the runtime and proves the integrated deliverable.
Part of epic
#850
Dependencies
#852 (seq2seq engine + text core), #853 (DaViT backbone), #854 (fusion + weight loading), #855 (processor + task prompts + location tokens)
Upstream reference
Architecture notes
Implementation plan
"florence2"arm tosrc/models/detection.rs.src/model_metadata.rsviafor_each_model_registration!.generate/generate_vlm.docs/supported-models.md._tests.rsunit tests (DaViT shapes, cross-attention, task-prompt parse).mlx-community/Florence-2-base-ft-bf16or-4bit) across caption + OCR + detection modes via./target/release/mlxcel generate -m <model> --image <img> -p "<task prompt>", confirmingmlxcel listreports florence2.docs/adding-models.md.Acceptance criteria
"florence2"arm added tosrc/models/detection.rs; registered insrc/model_metadata.rsviafor_each_model_registration!.generate/generate_vlm;mlxcel listreports florence2.docs/supported-models.mdupdated;_tests.rsunit tests added.Effort
MEDIUM.
Note: this sub-issue is tracked by the checklist in #850.