GLM4.1V Video Inference Bug Inquiry

[aminer-404]

Hi, @hjh0119
Thanks for your pr.
I want to train 4.1v using videos, but there are todos in code. How to solve these bugs? Is there any other ways to train 4.1v in videos?

https://github.com/modelscope/ms-swift/blob/bf0257370abf5eddab7814fef166d87d0e25d557/swift/llm/template/template/glm.py#L133C3-L133C38

ms-swift/swift/llm/template/template/glm.py

Line 171 in bf02573

# TODO: model video infer bug

[hjh0119]

You can follow the way this model's processor handles videos, and insert the logic for video processing.

[aminer-404]

It seems there are some difference between 4.5v and 4.1v, and the processor seems is made for 4.5v. Then, how to use the processor?

[hjh0119]

you can search for 4.1V processor code in transformers

[aminer-404]

it seems there are no <video_00>, <video_01> .. tokens for 4.1v in transformers. but the processor generate these tokens.