Support min_new_tokens generation config in pytorch engine

[grimoire]

tested on chat and benchmark_pytorch_throughput

[AllentDan]

Is this expected behavior of the model for min_new_tokens?

[grimoire]

@AllentDan Fixed, eos would be ignored before reaching min_new_tokens

[AllentDan]

Got this.

[grimoire]

Got this.

@lvhan028 is this OK?

[lvhan028]

@irexyc what are the behavior of transformers and turbomind if min_new_token is set?

[irexyc]

what are the behavior of transformers and turbomind if min_new_token is set?

set score of eos token to -inf when generated token length < min_new_tokens

https://github.com/huggingface/transformers/blob/main/src/transformers/generation/logits_process.py#L160

[lvhan028]

@AllentDan @irexyc I think PyTorch engine did the right thing. When min_new_tokens is set, eos_id and stop_words tokens should be banned when the generated token number is less than min_new_tokens

[AllentDan]

@AllentDan @irexyc I think PyTorch engine did the right thing. When min_new_tokens is set, eos_id and stop_words tokens should be banned when the generated token number is less than min_new_tokens

Is it possible that stop_words got banned, but other possible new tokens are used when n>1? min_new_tokens to me, is a minimal number of newly generated tokens while not output tokens like <eoa>, <Bot> to users.

[lvhan028]

stop_words should be banned. They cannot be generated.

[lvhan028]

from lmdeploy import pipeline, GenerationConfig, TurbomindEngineConfig, PytorchEngineConfig

# test torch min new tokens
pipe = pipeline('/workspace/models-140/InternLM/internlm2-chat-7b/', 
                backend_config=PytorchEngineConfig())

response = pipe('hi', gen_config=GenerationConfig(
    min_new_tokens=100
))

print(response)

The generated response is:

Response(text='你好！有什么我可以帮助你的吗？', generate_token_len=8, input_token_len=103, session_id=0, finish_reason='stop')

I think generated_token_len shouldn't be less than min_new_tokens

[grimoire]

@lvhan028
https://github.com/grimoire/lmdeploy/blob/637cc512027b41986fcfabba8f2011e682aa37e5/lmdeploy/messages.py#L87

gen_config conversion does not include min_new_tokens