Skip to content

Add video-scaling function for SFT padding#4353

Merged
copybara-service[bot] merged 1 commit into
mainfrom
hengtaoguo-sft-video
Jul 6, 2026
Merged

Add video-scaling function for SFT padding#4353
copybara-service[bot] merged 1 commit into
mainfrom
hengtaoguo-sft-video

Conversation

@hengtaoguo

@hengtaoguo hengtaoguo commented Jul 5, 2026

Copy link
Copy Markdown
Collaborator

Description

Scale-to-fit before padding in video preprocessor and fix audio-in-video sequence serialization. This PR serves as the groundwork for varied-size video SFT:

  • Scale-to-fit before padding: Improved smart_resize() preprocessing step in Qwen3-Omni image preprocessor to proportionally downscale videos exceeding max_h/w_pixels (resolving downstream shape crashes during batch padding) while preserving aspect ratio as closely as possible..
  • Unit test: Added test_scale_to_fit_video_before_padding to verify correct aspect-ratio preserving downscaling and successful padding setup.
  • Fixed a typo in add_extra_tokens_for_qwen3_omni where <|audio_pad|> (151675) was appended at the end of the interleaved video-audio sequence instead of <|audio_end|> (151670).

Tests

Example: Video of (220, 896) is resized to (128, 512) by new smart_resize: divisible by factor 32 and didn't exceed max canvas 512, preserving the aspect ratio as closely as possible:

python -m pytest tests/unit/qwen3_omni_layers_test.py::TestQwen3OmniMoeVisionPatchEmbed::test_scale_to_fit_video_before_padding -vv -s

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@codecov

codecov Bot commented Jul 5, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 11 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/maxtext/multimodal/processor_qwen3_omni.py 0.00% 11 Missing ⚠️

📢 Thoughts on this report? Let us know!

@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

🤖 Hi @hengtaoguo, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

## 📋 Review Summary

This PR introduces video rescaling functionality (scale_to_fit_video_grid) to proactively prevent sequence shape crashes during padding by enforcing the video_max_grid_h/w constraints. The test additions are well thought out and the fix for audio_end token appending is correct.

🔍 General Feedback

  • The core implementation of aspect-ratio preserving downscaling is solid and functionally sound.
  • There are a few edge cases related to floating point precision and consistency of the image scaling factor between the intermediate and final steps of the preprocessing pipeline that should be addressed.
  • Great job adding the unit tests to explicitly verify the grid limits and downscaling behavior!

Comment thread src/maxtext/multimodal/processor_qwen3_omni.py Outdated
Comment thread src/maxtext/multimodal/processor_qwen3_omni.py Outdated
Comment thread src/maxtext/multimodal/processor_qwen3_omni.py Outdated
Comment thread src/maxtext/multimodal/processor_qwen3_omni.py Outdated
Comment thread tests/unit/qwen3_omni_layers_test.py Outdated

@aireenmei aireenmei left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work!

@copybara-service
copybara-service Bot merged commit 577b84a into main Jul 6, 2026
118 of 138 checks passed
@copybara-service
copybara-service Bot deleted the hengtaoguo-sft-video branch July 6, 2026 20:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants