Questions about Qwen2.5-VL grounding fine-tuning with LLaMA-Factory (dataset format & bbox scaling)

[Rainbow-cocktail]

Hi, thanks for your great work on Qwen-VL!

I am currently using LLaMA-Factory to fine-tune Qwen2.5-VL for grounding tasks.
I have read many related issues (e.g. LLaMA-Factory #7015, Qwen-VL #721 and #900, etc.), but some key points still seem unclear. I have two questions and would really appreciate your clarification:

1️⃣ Dataset construction (special tokens)

When building the grounding dataset, do we need to explicitly include the special tokens such as:

<|box_start|> <|box_end|>
<|object_ref_start|> <|object_ref_end|>
# Example:
{
  "messages": [
    {"role":"system","content":"You are a helpful assistant."},
    {"role":"user","content":"找到图像中的<|object_ref_start|>羊<|object_ref_end|>"},
    {"role":"assistant","content":"<|box_start|>(493,169),(539,297)<|box_end|>"}
  ],
  "images": ["mllm_demo_data/person.jpg"]
} 

as mentioned in Issue #7015?
Or does Qwen / LLaMA-Factory automatically insert these tokens when using the proper data format? Or I need to remove these characters

2️⃣ Coordinate / bbox offset after image resizing

During training or inference, the model will typically resize input images internally.

For grounding tasks:

👉 Do we need to manually adjust the bbox coordinates to match the resized image,
or does LLaMA-Factory (or Qwen’s vision processor) automatically handle coordinate scaling?

If we must handle it manually, what is the recommended method?

Option A: Resize the image ourselves to the model’s expected size (e.g., 448/560/672/896 = 28×patch) and scale bbox accordingly

Option B: Convert all bounding boxes to relative 0–1000 format (as Qwen3-VL suggests)

Option C: Other official approach?

---

Thanks again for your work! Looking forward to your guidance!

[HumanZhong]

@Rainbow-cocktail
Hi, here is the explanation:

1. (do we need to explicitly include the special tokens)
   No, you don't need to include grounding-related special tokens (but if you decide to include some, it should also be ok). You can refer to the 2d_grounding cookbook for qwen2.5-vl and qwen3-vl answer format(which is a json list by default). And you can also choose your own output format.
2. (Coordinate / bbox offset after image resizing)
   It is recommended to use option A for qwen2.5-vl and to use option B for qwen3-vl.
   The image resizing and bbox resizing process is like this:

# this step will be automatically processed during inference if you use the official processor, no need to include this in your inference code, but you may need this to pre-calculate the gt_bbox for training in Qwen2.5-VL
resized_h, resized_w = smart_resize(img_h, img_w, factor=28 or 32, min_pixels=xxx, max_pixels=xxx) # 28 for 2.5-vl and 32 for 3-vl

# this step is only in Qwen2.5-VL and should be executed to get the gt_bbox(resized_bbox)
resized_bbox = np.round(np.array(ori_bbox).reshape(-1, 4) / np.array([img_w, img_h, img_w, img_h]) * np.array([resized_w, resized_h, resized_w, resized_h])).astype(np.int32)

# this step is only in Qwen3-VL and should be executed to get the gt_bbox(resized_bbox)
resized_bbox = np.round(np.array(ori_bbox).reshape(-1, 4) / np.array([img_w, img_h, img_w, img_h]) * np.array([1000, 1000, 1000, 1000])).astype(np.int32)

Hope this helps, if you have further questions, feel free to ask.

[minhduc01168]

@HumanZhong Hi, thanks for your great work on Qwen-VL!
I want to ask about problem 2. (Coordinate / bbox offset after image resizing)

When doing fine-tuning data preparation for Qwen2.5 model, I need to:

• Manually resize the image to the expected size of the model (eg: min_pixels: int = 256 * 28 * 28, max_pixels: int = 1280 * 28 * 28) and scale the bbox accordingly
• After the fine-tune process to perform inference, do I need to convert back from the model's output bbox to the actual bbox on the original image?

[minhduc01168]

@HumanZhong Hi, thanks for your great work on Qwen-VL!
After I did fine-tuning Qwen/Qwen2.5-VL-3B-Instruct on my dataset for Key Information Extraction problem, I got pretty good results but I encountered some problems as follows:

• The results have some correct bboxes but still more than half of the bboxes are incorrect. I still can't determine whether it's due to the problem in my training bbox labeling process or the quality of the fine-tuning model is not good?
• The extracted information is quite accurate and impressive but there are still some extracted information that are wrong or missing?
  Can you give any suggestions from your experience with the model?
  The result I illustrate bbox as shown below: