Skip to content

feat(models): Florence-2 processor, task prompts, and location tokens (sub of #850) #855

Description

@inureyes

Summary

The Florence-2 processor layer: task-prompt tokens and their expansions, image preprocessing, tokenizer wiring, and location-token handling with post-processing into structured coordinates for the spatial tasks.

Part of epic

#850

Dependencies

#854 (vision-language fusion + full weight loading)

Upstream reference

Architecture notes

  • Task-prompt tokens and their prompt expansions: <CAPTION>, <DETAILED_CAPTION>, <MORE_DETAILED_CAPTION>, <OCR>, <OCR_WITH_REGION>, <OD>, <DENSE_REGION_CAPTION>, <CAPTION_TO_PHRASE_GROUNDING>, <REGION_TO_SEGMENTATION>, and the rest of the task set.
  • Image preprocessing (resize / normalize) per the processor.
  • Tokenizer wiring.
  • Location tokens <loc_0> .. <loc_999>: emitted by the spatial tasks and post-processed into boxes / quad-boxes / regions.

Implementation plan

  • Implement the processor / task-prompt stage in the Florence-2 module under src/models/florence2/, feeding the fused model from feat(models): Florence-2 vision-language fusion + full weight loading (sub of #850) #854.
  • Reference the coordinate / OCR post-processing in the existing OCR VLM modules (dots_ocr and paddleocr under src/models/) as a partial reference for the location-token to box / region conversion.
  • Wire the tokenizer and image preprocessing into the VLM input path so a task prompt drives the correct mode end to end.

Acceptance criteria

  • Task-prompt tokens and their expansions handled for the full task set.
  • Image preprocessing (resize / normalize) matches the processor.
  • Location tokens parse / emit correctly and post-process into boxes / quad-boxes / regions.
  • Task-prompt round trip works; OCR and detection outputs parse into correct structured coordinates on a real image.
  • Integration: the processor is wired into the model's actual input/output path (task prompt in, structured result out through the fused model from feat(models): Florence-2 vision-language fusion + full weight loading (sub of #850) #854), not a standalone parser.

Effort

MEDIUM.

Note: this sub-issue is tracked by the checklist in #850.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:modelsModel architectures, weights, loading, metadatapriority:mediumMedium prioritystatus:doneCompletedtype:enhancementNew features, capabilities, or significant additions

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions