Checking Awni's video branch improvements are merged into main?

[southkorea2013]

Hi Prince,

Last December I asked Awni help to fix some issues of mlx-vlm video branch for a customer PoC, so I am wondering whether the improvements are already merged into main branch. I use the following steps to build the mlx-vlm env for the PoC.

pip install git+https://github.com/awni/mlx-vlm.git@video
pip install qwen_vl_utils

1 The video branch can significantly reduce the memory footprint for video inference using Qwen-VL.
2 Also fix the some bugs during inference, for example cannot read the timestamp in the video frame.

python -m mlx_vlm.video_generate --model mlx-community/Qwen2-VL-7B-Instruct-bf16 --max-tokens 500 --prompt "Describe this video" --video /Users/dmp/vlm/test.mp4 --max-pixels 1920 1080 --fps 2.0
date
Thu Mar 13 21:08:42 CST 2025
objc[93822]: Class AVFFrameReceiver is implemented in both /Users/dmp/miniconda3/envs/vlm/lib/python3.11/site-packages/av/.dylibs/libavdevice.61.3.100.dylib (0x1238103a8) and /Users/dmp/miniconda3/envs/vlm/lib/python3.11/site-packages/decord/.dylibs/libavdevice.59.7.100.dylib (0x129cdca10). One of the two will be used. Which one is undefined.
objc[93822]: Class AVFAudioReceiver is implemented in both /Users/dmp/miniconda3/envs/vlm/lib/python3.11/site-packages/av/.dylibs/libavdevice.61.3.100.dylib (0x1238103f8) and /Users/dmp/miniconda3/envs/vlm/lib/python3.11/site-packages/decord/.dylibs/libavdevice.59.7.100.dylib (0x129cdca60). One of the two will be used. Which one is undefined.
Loading model: mlx-community/Qwen2-VL-7B-Instruct-bf16
Fetching 14 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:00<00:00, 23850.63it/s]
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
==========
Video: /Users/dmp/vlm/test.mp4 

Prompt: <|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
<|vision_start|><|video_pad|><|vision_end|>Describe this video<|im_end|>
<|im_start|>assistant

qwen-vl-utils using decord to read video.
The given max_pixels[2073600] exceeds limit[602112].
Token indices sequence length is longer than the specified maximum sequence length for this model (43224 > 32768). Running this sequence through the model will result in indexing errors
Generating video description...
The video depicts a virtual reality scene featuring a large, yellow excavator situated in a spacious, well-lit room with high ceilings. The excavator is positioned in the center of the room, surrounded by various objects and furniture. The room appears to be a museum or gallery space, as indicated by the presence of large, abstract paintings on the walls and sculptures displayed throughout the room. The excavator is the focal point of the scene, with its large size and bright color making it stand out against the more subdued background. The overall atmosphere of the video is modern and artistic, with a blend of technology and creativity.
==========
Prompt: 289.731 tokens-per-sec
Generation: 29.654 tokens-per-sec
Peak memory: 152.406 GB
Thu Mar 13 21:11:24 CST 2025

Thanks.

Nan

[southkorea2013]

I evaluate the latest mlx-vlm, I find the issue 2 is not resolved now. The output date time is totally wrong.

(vlm) dmp@MacStudio1 ~ % date
python -m mlx_vlm.video_generate --model mlx-community/Qwen2.5-VL-32B-Instruct-4bit --max-tokens 500 --prompt "Please describe the date time of the upper right corner" --video /Users/dmp/time.mp4 --max-pixels 720 480 --fps 0.5
date
Thu Mar 27 16:52:19 CST 2025
This is a beta version of the video understanding. It may not work as expected.
Loading model: mlx-community/Qwen2.5-VL-32B-Instruct-4bit
Fetching 16 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:00<00:00, 15023.25it/s]
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.50, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
numpy reader: video_path=/Users/dmp/time.mp4, total_frames=681, video_fps=59.94005994005994, time=0.000s
Generating response...
==========
Files: ['/Users/dmp/time.mp4'] 

Prompt: <|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
<|vision_start|><|video_pad|><|vision_end|>Please describe the date time of the upper right corner<|im_end|>
<|im_start|>assistant

mx.metal.get_peak_memory is deprecated and will be removed in a future version. Use mx.get_peak_memory instead.
The date and time in the upper right corner of the image are as follows:

- **Date**: 2023-2-27
- **Time**: 2:13:29

This indicates that the screenshot was taken on February 27, 2023, at 2:13:29 PM.
==========
Prompt: 841 tokens, 206.716 tokens-per-sec
Generation: 75 tokens, 32.043 tokens-per-sec
Peak memory: 20.936 GB
Thu Mar 27 16:52:30 CST 2025

Thanks,
Nan

[Blaizzy]

Was issue two solved in the video branch?

If not, I would recommend trying Qwen2.5-VL it's much better with timestamps.

[southkorea2013]

Was issue two solved in the video branch?

If not, I would recommend trying Qwen2.5-VL it's much better with timestamps.

Yes, this two issues are fixed in Awni's video branch. I check with latest mlx-vlm and Qwen2.5-VL, I find the second issue is not resolved. Thanks.

[Blaizzy]

Got it

Note to self: implement the changes here
awni@85cdc44

[c-m-g-ogawa]

I believe the main fix needed is the following line
https://github.com/Blaizzy/mlx-vlm/blame/48c8a55e8a74642c4a633619bdd25683f42b2325/mlx_vlm/models/qwen2_5_vl/vision.py#L284

from pos_ids.append(mx.repeat(stacked_pos_ids, t, axis=0)) to pos_ids.append(mx.tile(stacked_pos_ids, (t, 1)))

This also fixes video understanding as currently the model does not seem to be correctly processing movement.

With version 0.1.21 and this video, before the fix (incorrect description, frame order seems to be garbled):

▶ python -m mlx_vlm.video_generate --model mlx-community/Qwen2.5-VL-7B-Instruct-bf16 --max-tokens 100 --prompt "Describe this video in 100 words" --video crash.mp4 --max-pixels 360 360 --fps 2.0                      
This is a beta version of the video understanding. It may not work as expected.
Loading model: mlx-community/Qwen2.5-VL-7B-Instruct-bf16
Fetching 14 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:00<00:00, 92765.02it/s]
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
numpy reader: video_path=crash.mp4, total_frames=240, video_fps=23.98, time=0.000s
Generating response...
==========
Files: ['crash.mp4'] 

Prompt: <|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
<|vision_start|><|video_pad|><|vision_end|>Describe this video in 100 words<|im_end|>
<|im_start|>assistant

mx.metal.get_peak_memory is deprecated and will be removed in a future version. Use mx.get_peak_memory instead.
The video shows a sequence of frames from a security camera capturing a parking lot scene. Throughout the video, a black car is seen repeatedly entering and exiting a designated parking space. The car's movement is consistent, with its entry and exit appearing to be smooth and uninterrupted. The parking lot is marked with white lines, indicating the boundaries of the parking spaces. In the background, there are other parked cars and a few scattered objects, possibly trash or debris, on the ground. The camera angle remains constant
==========
Prompt: 1560 tokens, 411.284 tokens-per-sec
Generation: 100 tokens, 46.468 tokens-per-sec
Peak memory: 17.622 GB

After the fix (correct description):

▶ python -m mlx_vlm.video_generate --model mlx-community/Qwen2.5-VL-7B-Instruct-bf16 --max-tokens 100 --prompt "Describe this video in 100 words" --video crash.mp4 --max-pixels 360 360 --fps 2.0
This is a beta version of the video understanding. It may not work as expected.
Loading model: mlx-community/Qwen2.5-VL-7B-Instruct-bf16
Fetching 14 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:00<00:00, 61166.93it/s]
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
numpy reader: video_path=crash.mp4, total_frames=240, video_fps=23.98, time=0.000s
Generating response...
==========
Files: ['crash.mp4'] 

Prompt: <|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
<|vision_start|><|video_pad|><|vision_end|>Describe this video in 100 words<|im_end|>
<|im_start|>assistant

mx.metal.get_peak_memory is deprecated and will be removed in a future version. Use mx.get_peak_memory instead.
The video depicts a parking lot scene where two cars are involved in a minor accident. A silver car is seen backing out of a parking space, and as it moves, it appears to hit the rear of a black station wagon parked in front. The impact causes a slight dent on the bumper of the black car, and smoke is seen coming from the point of contact, suggesting a minor fire or fender-bender. The silver car continues to move forward, and the black car remains stationary. A
==========
Prompt: 1560 tokens, 412.288 tokens-per-sec
Generation: 100 tokens, 46.501 tokens-per-sec
Peak memory: 17.622 GB

[Blaizzy]

Thanks @g-ogawa-ais this is awesome!

I will fix and will be out on the next release with llama 4👌🏽