How to generate different outputs with same model?

[NilsHellwig]

tokenizer.batch_decode(
                model.generate(**inputs, max_new_tokens=max_new_tokens, use_cache=False),
            )[0]

I have a very simple problem: How can I generate a different output with the same model. temperature, top_k etc. don't change the LLM's output. Setting the seed doesn't work, I'm not setting any seeds atm. I'm using gemma-3-1b-it and SFTTrainer.

[void-mckenzie]

https://huggingface.co/docs/transformers/v4.49.0/en/main_classes/text_generation#transformers.[Generation](https://huggingface.co/docs/transformers/v4.49.0/en/main_classes/text_generation#transformers.GenerationMixin.generate)Mixin.generate - Check this out

[NilsHellwig]

Thank you @void-mckenzie ! But which parameter does allow for different generations?

[void-mckenzie]

If you scroll down to the generate method in the previous link, you can see that it takes in a generation_config parameter. You can pass your parameters packed through this class (transformers.GenerationConfig)

[void-mckenzie]

Hello @NilsHellwig , I just tried to replicate your bug and I can confirm that when you just change the parameters you mentioned, there is no change in the output. However, digging through the code, I found out that you need to pass "do_sample=True", for your top_k and temperature to have any effect. Once I did that, it works fine.

So, your code should look like:

tokenizer.batch_decode(
                model.generate(**inputs, max_new_tokens=max_new_tokens, use_cache=False, temperature=1.5, top_p=0.95, top_k = 250, do_sample = True),
)[0]

[NilsHellwig]

@void-mckenzie Thank you very much! I will try that tomorrow! Best regards, Nils.

[void-mckenzie]

Just following up, did it work? @NilsHellwig

[NilsHellwig]

@void-mckenzie Yes, worked out! Sorry for not keeping you updated :O