feat(server): serve Florence-2 through a seq2seq worker loop

[inureyes]

Problem / Background

Epic #850 landed Florence-2 support, and it works end to end through the CLI:

mlxcel generate -m models/Florence-2-base-ft-bf16 --image <img> -p '<OD>'

mlxcel-server does not serve it. Startup refuses the checkpoint with a named error, added by #856 (PR #1071) in src/server/startup.rs:

Florence-2 is an encoder-decoder (seq2seq) VLM that mlxcel-server cannot serve yet. Run it through the CLI instead...

The reason is architectural, not incidental. mlxcel's generation engine is decoder-only everywhere except Whisper's dedicated ASR pipeline. Florence-2 is BART-style seq2seq: it runs an encoder pass over the fused vision-plus-prompt sequence, then decodes autoregressively with cross-attention against the cached encoder output. #856 wired this into the CLI as a pre-loop early exit in run_generate_once (src/commands/generate.rs dispatching to src/commands/generate_florence2.rs), following the DiffusionGemma and LLaDA-2 pattern, deliberately not routing through compute_vlm_embeddings, which cannot express an encoder pass plus a cross-attention decode. The server's worker loop is decoder-only and would serve garbage from the trait-completeness forward, so the startup refusal is the correct interim behavior rather than a bug.

Proposed Solution

Give mlxcel-server a seq2seq worker path so Florence-2 becomes reachable over HTTP, then remove the startup refusal.

Shape of the work:

• A seq2seq worker variant that runs the encoder pass once per request, caches the encoder output, and drives the cross-attention decode loop, parallel to how the decoder-only worker loop is structured today. Whisper's ASR pipeline is the closest existing precedent for a non-decoder-only serving path.
• Request handling that maps an OpenAI-compatible chat/completions request carrying an image plus a Florence-2 task prompt onto the same parse_task_prompt / Florence2Task path the CLI uses, and returns the post-processed Florence2TaskResult (text, boxes, polygons, OCR regions) in a stable response shape.
• Removal of the Florence2VLM bail in src/server/startup.rs once the path is live.

Security requirements carried forward from #855

Both items were explicitly handed forward and must be honored by any server surface.

1. Decompression-bomb defense is not inside the processor. preprocess_with_sizes (src/vision/processors/florence2.rs) takes an already-decoded DynamicImage, so it cannot bound the decode. Every request path must route decoding through decode_request_image_with_limits / ImageInputLimits (src/server/model_worker.rs, src/server/media.rs). The CLI path added by feat(models): Florence-2 end-to-end integration and real-checkpoint validation (sub of #850) #856 already does this via decode_image_payloads_with_limits (src/lib.rs).
2. Task prompt input text is untrusted on a server surface. Florence2Task::expand (src/models/florence2/tasks.rs) interpolates caller-supplied text into the encoder prompt for 7 of the 15 task modes (CaptionToPhraseGrounding, ReferringExpressionSegmentation, RegionToSegmentation, OpenVocabularyDetection, RegionToCategory, RegionToDescription, RegionToOcr). The CLI sidesteps the question by rejecting anything that is not a recognized task marker; a server accepting free-form input needs explicit validation (length bound, character/format constraints, and for the region tasks a check that the coordinate form is well formed).

Acceptance Criteria

• mlxcel-server accepts a Florence-2 checkpoint at startup and reaches a serving state.
• An OpenAI-compatible request carrying an image and a task prompt returns the same answer the CLI produces for the same model, image, and task marker (verified against a real checkpoint, for example Florence-2-base-ft-bf16, on at least the pure-text <CAPTION> and the box-producing <OD> paths).
• Server-side image decoding for this family is bounded by ImageInputLimits through decode_request_image_with_limits; an oversized or bomb payload is rejected before decode rather than after.
• Task-prompt input text reaching Florence2Task::expand is validated at the request boundary for the 7 input-taking task modes, with a rejection test for malformed and oversized input.
• The Florence2VLM startup refusal in src/server/startup.rs is removed, and no code path can hand a Florence-2 checkpoint to a decoder-only worker loop.
• Tests cover the seq2seq worker loop (encoder cache reuse across decode steps) and the request-to-response mapping.
• docs/supported-models.md updated to state that Florence-2 is servable, not CLI-only.

Technical Considerations

• The encoder output is per-request state, not per-token state. It must live alongside the decode KV cache for the request's lifetime and be dropped with it; a shared or leaked encoder cache across requests would cross-contaminate answers.
• Batching semantics are open. Florence-2's encoder pass has a different cost profile from the decode loop, so a naive drop into the existing continuous-batching admission logic may not be appropriate. Serving requests one at a time on the seq2seq path is an acceptable first landing if the limitation is documented and measured.
• Response shape for the structured task outputs (boxes, polygons, OCR regions) is a design decision this issue must settle, since OpenAI's chat schema has no native place for them. Options include serializing into the message content or an mlxcel-specific response field.

[inureyes]

Response shape decided

The open design question under "Technical Considerations" (where the structured task outputs belong, given that OpenAI's chat schema has no native place for boxes, polygons, or OCR regions) is settled:

Both, side by side. message.content carries the same human-readable text the CLI prints, and the structured coordinates go in a separate mlxcel-specific response field as JSON.

Rationale: a standard OpenAI-compatible client keeps working and sees sensible text, while a client that wants coordinates reads them without re-parsing a formatted string. Serializing JSON into content was rejected because it makes the common case unreadable, and text-only was rejected because it forces every consumer to reimplement the parsing that src/models/florence2/postprocess.rs already does.

Implementer should follow whatever extension-field convention the server already uses for non-OpenAI fields rather than inventing a new one, and document the field in docs/responses-api.md alongside the docs/supported-models.md update this issue already requires.