You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Florence-2 processor layer: task-prompt tokens and their expansions, image preprocessing, tokenizer wiring, and location-token handling with post-processing into structured coordinates for the spatial tasks.
Task-prompt tokens and their prompt expansions: <CAPTION>, <DETAILED_CAPTION>, <MORE_DETAILED_CAPTION>, <OCR>, <OCR_WITH_REGION>, <OD>, <DENSE_REGION_CAPTION>, <CAPTION_TO_PHRASE_GROUNDING>, <REGION_TO_SEGMENTATION>, and the rest of the task set.
Image preprocessing (resize / normalize) per the processor.
Tokenizer wiring.
Location tokens <loc_0> .. <loc_999>: emitted by the spatial tasks and post-processed into boxes / quad-boxes / regions.
Reference the coordinate / OCR post-processing in the existing OCR VLM modules (dots_ocr and paddleocr under src/models/) as a partial reference for the location-token to box / region conversion.
Wire the tokenizer and image preprocessing into the VLM input path so a task prompt drives the correct mode end to end.
Acceptance criteria
Task-prompt tokens and their expansions handled for the full task set.
Image preprocessing (resize / normalize) matches the processor.
Location tokens parse / emit correctly and post-process into boxes / quad-boxes / regions.
Task-prompt round trip works; OCR and detection outputs parse into correct structured coordinates on a real image.
Summary
The Florence-2 processor layer: task-prompt tokens and their expansions, image preprocessing, tokenizer wiring, and location-token handling with post-processing into structured coordinates for the spatial tasks.
Part of epic
#850
Dependencies
#854 (vision-language fusion + full weight loading)
Upstream reference
Architecture notes
<CAPTION>,<DETAILED_CAPTION>,<MORE_DETAILED_CAPTION>,<OCR>,<OCR_WITH_REGION>,<OD>,<DENSE_REGION_CAPTION>,<CAPTION_TO_PHRASE_GROUNDING>,<REGION_TO_SEGMENTATION>, and the rest of the task set.<loc_0>..<loc_999>: emitted by the spatial tasks and post-processed into boxes / quad-boxes / regions.Implementation plan
src/models/florence2/, feeding the fused model from feat(models): Florence-2 vision-language fusion + full weight loading (sub of #850) #854.src/models/) as a partial reference for the location-token to box / region conversion.Acceptance criteria
Effort
MEDIUM.
Note: this sub-issue is tracked by the checklist in #850.