[evaluation] [DRAFT] Gemma4 evaluation - #851
Draft
Torrero wants to merge 2 commits into
Draft
Conversation
This commit support benchmarks evaluation for Gemma4 Co-authored-by: Cline TICO-DCO-1.0-Signed-off-by: Evgenii Maltsev <e.maltsev@samsung.com>
Contributor
|
From #852 (comment),
Before switching to a different PPL protocol, I compared raw PPL/logits on the unwrapped HF model. Therefore, I ran below script and the ppls are small enough. use_cache=True: internal_nll=2.476937, manual_nll=2.476937, ppl=11.90
use_cache=False: internal_nll=2.476937, manual_nll=2.476937, ppl=11.90import math
import torch
import torch.nn.functional as F
import transformers
from datasets import load_dataset
from transformers import AutoModelForMultimodalLM, AutoProcessor
MODEL_ID = "google/gemma-4-E2B-it"
print("transformers:", transformers.__version__)
processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForMultimodalLM.from_pretrained(
MODEL_ID,
dtype=torch.bfloat16,
device_map="auto",
)
model.eval()
dataset = load_dataset(
"Salesforce/wikitext",
"wikitext-2-raw-v1",
split="test",
)
text = "\n\n".join(
example["text"]
for example in dataset
if example["text"].strip()
)
input_ids = processor.tokenizer(
text,
return_tensors="pt",
).input_ids[:, :256]
input_device = model.get_input_embeddings().weight.device
input_ids = input_ids.to(input_device)
attention_mask = torch.ones_like(input_ids)
def evaluate_one_window(use_cache: bool) -> tuple[float, float]:
with torch.inference_mode():
outputs = model(
input_ids=input_ids,
attention_mask=attention_mask,
labels=input_ids,
use_cache=use_cache,
logits_to_keep=0,
)
logits = outputs.logits[:, :-1].float()
targets = input_ids[:, 1:].to(logits.device)
manual_nll = F.cross_entropy(
logits.reshape(-1, logits.shape[-1]),
targets.reshape(-1),
reduction="mean",
)
internal_nll = float(outputs.loss)
manual_nll_value = float(manual_nll)
print(
f"use_cache={use_cache}: "
f"internal_nll={internal_nll:.6f}, "
f"manual_nll={manual_nll_value:.6f}, "
f"ppl={math.exp(manual_nll_value):.2f}"
)
return internal_nll, manual_nll_value
evaluate_one_window(use_cache=True)
evaluate_one_window(use_cache=False) |
Contributor
Author
|
I ran script from your comment (#851 (comment)) using different devices and environments but I couldn't reproduce the same low numbers for the mentioned instruction-tuned model (
My results for 2 different workstation with different GPU and CPU: Could you please provide information about your environment, maybe I missed something. |
Contributor
|
@Torrero Hmm.. it's weird. Here's my env. Could you share the result of below scirpt? import math
import torch
import torch.nn.functional as F
import transformers
from datasets import load_dataset
from transformers import AutoModelForMultimodalLM, AutoProcessor
import tokenizers
MODEL_ID = "/home/seongwoo.chae/models/Qwen3-VL-4B-Instruct"
print("transformers:", transformers.__version__)
processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForMultimodalLM.from_pretrained(
MODEL_ID,
dtype=torch.bfloat16,
device_map="auto",
)
model.eval()
dataset = load_dataset(
"Salesforce/wikitext",
"wikitext-2-raw-v1",
split="test",
)
text = "\n\n".join(
example["text"]
for example in dataset
if example["text"].strip()
)
input_ids = processor.tokenizer(
text,
return_tensors="pt",
).input_ids[:, :256]
input_device = model.get_input_embeddings().weight.device
input_ids = input_ids.to(input_device)
attention_mask = torch.ones_like(input_ids)
def evaluate_one_window(use_cache: bool) -> tuple[float, float]:
with torch.inference_mode():
outputs = model(
input_ids=input_ids,
attention_mask=attention_mask,
labels=input_ids,
use_cache=use_cache,
logits_to_keep=0,
)
logits = outputs.logits[:, :-1].float()
targets = input_ids[:, 1:].to(logits.device)
manual_nll = F.cross_entropy(
logits.reshape(-1, logits.shape[-1]),
targets.reshape(-1),
reduction="mean",
)
internal_nll = float(outputs.loss)
manual_nll_value = float(manual_nll)
print(
f"use_cache={use_cache}: "
f"internal_nll={internal_nll:.6f}, "
f"manual_nll={manual_nll_value:.6f}, "
f"ppl={math.exp(manual_nll_value):.2f}"
)
return internal_nll, manual_nll_value
evaluate_one_window(use_cache=True)
evaluate_one_window(use_cache=False)
# Test
tokenizer = processor.tokenizer
print("transformers:", transformers.__version__)
print("transformers path:", transformers.__file__)
print("tokenizers:", tokenizers.__version__)
print("tokenizer class:", type(tokenizer))
print("tokenizer path:", tokenizer.name_or_path)
print("tokenizer commit:", tokenizer.init_kwargs.get("_commit_hash"))
print("model commit:", getattr(model.config, "_commit_hash", None))
print("add_bos_token:", getattr(tokenizer, "add_bos_token", None))
print("bos_token:", tokenizer.bos_token)
print("bos_token_id:", tokenizer.bos_token_id)
print("first input ids:", input_ids[0, :10].tolist())
print(
"first tokens:",
tokenizer.convert_ids_to_tokens(input_ids[0, :10].tolist()),
)
print(
"starts with BOS:",
input_ids[0, 0].item() == tokenizer.bos_token_id,
)
input_device = model.get_input_embeddings().weight.device
@torch.inference_mode()
def score_input_ids(input_ids: torch.Tensor) -> tuple[float, float]:
input_ids = input_ids.to(input_device)
attention_mask = torch.ones_like(input_ids)
outputs = model(
input_ids=input_ids,
attention_mask=attention_mask,
labels=input_ids,
use_cache=False,
logits_to_keep=0,
)
logits = outputs.logits[:, :-1].float()
targets = input_ids[:, 1:].to(logits.device)
nll = F.cross_entropy(
logits.reshape(-1, logits.shape[-1]),
targets.reshape(-1),
reduction="mean",
)
nll_value = float(nll)
return nll_value, math.exp(nll_value)
original_add_bos = getattr(tokenizer, "add_bos_token", None)
try:
for add_bos in (False, True):
tokenizer.add_bos_token = add_bos
test_ids = tokenizer(
text,
return_tensors="pt",
).input_ids[:, :256]
nll, ppl = score_input_ids(test_ids)
print(
f"add_bos_token={add_bos}: "
f"first_id={test_ids[0, 0].item()}, "
f"starts_with_bos="
f"{test_ids[0, 0].item() == tokenizer.bos_token_id}, "
f"nll={nll:.6f}, "
f"ppl={ppl:.2f}"
)
finally:
if original_add_bos is not None:
tokenizer.add_bos_token = original_add_bostransformers: 5.9.0
Loading weights: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 713/713 [00:01<00:00, 481.32it/s]
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
[transformers] Token indices sequence length is longer than the specified maximum sequence length for this model (298938 > 262144). Running this sequence through the model will result in indexing errors
use_cache=True: internal_nll=2.476937, manual_nll=2.476937, ppl=11.90
use_cache=False: internal_nll=2.476937, manual_nll=2.476937, ppl=11.90
transformers: 5.9.0
transformers path: /home/seongwoo.chae/TICO/.venv/lib/python3.10/site-packages/transformers/__init__.py
tokenizers: 0.22.2
tokenizer class: <class 'transformers.models.qwen2.tokenization_qwen2.Qwen2Tokenizer'>
tokenizer path: /home/seongwoo.chae/models/Qwen3-VL-4B-Instruct
tokenizer commit: None
model commit: None
add_bos_token: False
bos_token: None
bos_token_id: None
first input ids: [284, 8397, 425, 10965, 465, 284, 14731, 8397, 425, 10965]
first tokens: ['Ġ=', 'ĠRobert', 'ĠB', 'oul', 'ter', 'Ġ=', 'ĠĊĊĊ', 'ĠRobert', 'ĠB', 'oul']
starts with BOS: False
add_bos_token=False: first_id=284, starts_with_bos=False, nll=2.476937, ppl=11.90
add_bos_token=True: first_id=284, starts_with_bos=False, nll=2.476937, ppl=11.90 |
Contributor
Author
|
This is my output: |
Contributor
|
@Torrero Then, the ppl seems small enough now. |
Contributor
Author
|
@mhs4670go But this is result for for |
Contributor
|
@Torrero Sorry for confusion. It's my bad that I used a different model. Thanks for checking this across multiple environments. My previous low-PPL result was caused by a mistake on my side. I reran the
Results: The missing BOS token explains a large part of the extremely high raw PPL, but adding BOS alone is still not enough to obtain a reasonable value. The gemma-4-E2B-it tokenizer has I therefore agree that Gemma4 IT should use a separately labelled chat-formatted PPL mode rather than the current raw mode. One remaining detail is terminology: the current implementation renders a fixed user instruction and scores WikiText as assistant-side text. This is more precisely a chat-prefixed or assistant-response PPL, rather than a true context/target continuation split. It is still suitable for comparing FP32 and quantized Gemma4 models though. Therefore, please proceed what you was going to merge. I'll review them. Thank you again! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This draft PR adds benchmarks evaluation support for Gemma4 with GPTQ quantizer.
Evaluation
google/gemma-4-e2b-it*For the instruction-tuned model (gemma-4-e2b-it) the perplexity evaluation in raw mode produces huge values (>1000), it should be evaluated in the chat-continuation mode, but its behavior requires clarification. (in progress).
**GPTQ mode is under evaluation.
test_Gemma_orig_eval.log
gemma_PTQ_W4A16_ple8bit.log
Run command for PTQ_W4A16_ple8bit
TICO-DCO-1.0-Signed-off-by: Evgenii Maltsev e.maltsev@samsung.com