Skip to content

[evaluation] Generation mode and videomme benchmark support for Gemma4 - #865

Merged
mhs4670go merged 1 commit into
Samsung:mainfrom
Torrero:gemma_eval_videomme
Aug 6, 2026
Merged

[evaluation] Generation mode and videomme benchmark support for Gemma4#865
mhs4670go merged 1 commit into
Samsung:mainfrom
Torrero:gemma_eval_videomme

Conversation

@Torrero

@Torrero Torrero commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

This PR adds generation support and Video-MME benchmark support to the Gemma4 PTQ wrappers:

  1. Generation supportQuantGemma4ForCausalLM and QuantGemma4ForConditionalGeneration now inherit from GenerationMixin and implement all the methods HF's generate().

  2. Video-MME benchmark support — The Gemma4 adapter's evaluate() method now support videomme benchmark.
    For gemma4 evaluation with max_position_embeddings=2048 we should use max_num_frames = 21 unlike qwen3-vl where max_num_frames=35 is used.
    In Gemma4, the minimum limit of soft_tokens_per_frame=70 (comes from processor.video_processor.max_soft_tokens) when for qwen3-vl we can control the minimum tokens per frame.

**The formula:**

input_budget     = max_position_embeddings - max_new_tokens - safety_margin
visual_budget    = input_budget - text_token_margin
max_num_frames   = visual_budget // soft_tokens_per_frame


max_position_embeddings=2048
max_new_tokens=30
safety_margin=4
text_token_margin=512
soft_tokens_per_frame=70 

**The arithmetic:**

input_budget  = 2048 - 30 - 4      = 2014
visual_budget = 2014 - 512          = 1502
max_num_frames = 1502 / 70        = 21

Co-Authored-By: Cline

TICO-DCO-1.0-Signed-off-by: Evgenii Maltsev e.maltsev@samsung.com

@Torrero
Torrero requested a review from mhs4670go August 5, 2026 14:32
@Torrero
Torrero force-pushed the gemma_eval_videomme branch from 742ab0f to b0978a4 Compare August 5, 2026 15:07
This commit supports generation mode and videomme benchmark for Gemma4

Co-Authored-By: Cline

TICO-DCO-1.0-Signed-off-by:  Evgenii Maltsev <e.maltsev@samsung.com>
@Torrero
Torrero force-pushed the gemma_eval_videomme branch from b0978a4 to 26b6430 Compare August 5, 2026 15:29

@mhs4670go mhs4670go left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mhs4670go
mhs4670go merged commit afb0852 into Samsung:main Aug 6, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants