Skip to content

feat(models): Florence-2 end-to-end integration and real-checkpoint validation (sub of #850) #856

Description

@inureyes

Summary

The capstone that wires Florence-2 into mlxcel's model detection, registration, and generation paths, then validates the whole model on a real checkpoint across caption + OCR + detection modes. No new model math lives here; it connects the pieces from #852, #853, #854, and #855 into the runtime and proves the integrated deliverable.

Part of epic

#850

Dependencies

#852 (seq2seq engine + text core), #853 (DaViT backbone), #854 (fusion + weight loading), #855 (processor + task prompts + location tokens)

Upstream reference

Architecture notes

Implementation plan

  • Add the "florence2" arm to src/models/detection.rs.
  • Register in src/model_metadata.rs via for_each_model_registration!.
  • Wire the seq2seq VLM decode path into generate / generate_vlm.
  • Add the TP / distributed arch-string if applicable.
  • Update docs/supported-models.md.
  • Add _tests.rs unit tests (DaViT shapes, cross-attention, task-prompt parse).
  • Validate on a real checkpoint (mlx-community/Florence-2-base-ft-bf16 or -4bit) across caption + OCR + detection modes via ./target/release/mlxcel generate -m <model> --image <img> -p "<task prompt>", confirming mlxcel list reports florence2.
  • Follow docs/adding-models.md.

Acceptance criteria

  • "florence2" arm added to src/models/detection.rs; registered in src/model_metadata.rs via for_each_model_registration!.
  • seq2seq VLM decode path wired into generate / generate_vlm; mlxcel list reports florence2.
  • TP / distributed arch-string added if applicable.
  • docs/supported-models.md updated; _tests.rs unit tests added.
  • Real checkpoint loads and produces correct output for at least caption + OCR + detection modes.
  • Integration: the epic deliverable is a functionally integrated model reachable from the CLI and server, not standalone modules.

Effort

MEDIUM.

Note: this sub-issue is tracked by the checklist in #850.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:modelsModel architectures, weights, loading, metadatapriority:mediumMedium prioritystatus:doneCompletedtype:enhancementNew features, capabilities, or significant additions

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions