model.generate temperature parameter is completely ineffective

[AndreaSottana]

System Info

• transformers version: 4.26.1
• Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.31
• Python version: 3.10.4
• Huggingface_hub version: 0.12.1
• PyTorch version (GPU?): 1.13.1+cu117 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: Yes

Who can help?

@gante @sg

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Hello,
I am trying to generate text using different models and different temperature parameters. I have noticed, however, that while changing hyperparameters such as num_beams affects the output text, changing the temperature parameter doesn't seem to do anything, and setting a temperature to 0.0 or 1.0 (very different) always leads to the same output. This has been observed across multiple different language models. I suspect this might be a bug such that the set temperature is not shown to the model. In order to reproduce, run the example below (feel free to try with different text samples to convince yourself it's not a one-off occurrence)

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, AutoConfig
from accelerate import init_empty_weights, infer_auto_device_map
import torch

tokenizer = AutoTokenizer.from_pretrained('bigscience/T0pp')  # feel free to try a different model
config = AutoConfig.from_pretrained('bigscience/T0pp')

max_memory={i: "24GiB" for i in range(torch.cuda.device_count())}
with init_empty_weights():
    model = AutoModelForSeq2SeqLM.from_config(config)
    device_map = infer_auto_device_map(model, no_split_module_classes=['T5Block'])
    print(device_map)
    device_map['lm_head'] = 0

model = AutoModelForSeq2SeqLM.from_pretrained('bigscience/T0pp', device_map=device_map, load_in_8bit=True, max_memory=max_memory)
text = "Complete the following story: Once upon a time there was a "
input_ids = tokenizer.encode(text, return_tensors='pt').to(0)
for temp in [0.0, 1.0]:
    beam_outputs = model.generate(
        input_ids, 
        max_length=512,
        num_beams=5,
        no_repeat_ngram_size=4,
        temperature=temp,
        num_return_sequences=1, 
        early_stopping=True,
    )
    print(tokenizer.decode(beam_outputs[0], skip_special_tokens=True))

Expected behavior

I would expect the two printed outputs to be different. I understand that occasionally they might be the same, but I've tried with over 1,000 different inputs, and the generated outputs with temperature=0 and temperature=1 are ALWAYS the same which means there is something wrong

[gante]

Hey @AndreaSottana 👋

I would recommend reading our blog post on how to generate.

TL;DR -- there are several generation modes, and not all .generate() parameters are active for a given generation mode. In particular, the popular temperature, top_p, and top_k are only active when do_sample=True is also passed. Some tasks benefit from do_sample=True, while others do not. Popular tools like ChatGPT operate with sampling.

We are aware that our .generate() has too many options and too little checks/examples, we are working on it 🤗

[AndreaSottana]

Hi @gante
Thank you very much for clarifying, I wasn't aware that some parameters were not effective when do_sample=False. Closing the issue for now