inference kwargs

[1024er]

Thanks for the great work, I reproduced the training, but at inference time tends to generate shorter text. I am using:

generated = model.generate(batch["input_ids"], max_length=512)

Does the interface on the demo web page adjust other kwargs?
Thanks

[rtaori]

Hi, for the demo we use:

full_completion = model_current.generate(
    inputs=input_ids,
    attention_mask=attention_mask,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
    num_beams=1,
    max_new_tokens=600,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)

[puyuanOT]

Got an error:

TypeError: The current model class (LlamaModel) is not compatible with .generate(), as it doesn't have a language model head. Please use one of the following classes instead: {'LlamaForCausalLM'}

[robotagi]

Got an error:

TypeError: The current model class (LlamaModel) is not compatible with .generate(), as it doesn't have a language model head. Please use one of the following classes instead: {'LlamaForCausalLM'}

Have you solved it?

[zyg18181818]

i think you may use the wrong model type, for example you use AutoModel,but AutoModelForCausalLM is the correct choice