ChatGLM3 output token size is not generated as expected

[JunxiChhen]

Benchmark cmd (-ic is 128):

numactl -C 0-55 -m 0 python benchmark.py     -m /root/.cache/huggingface/hub/chatglm3-6b-ov/pytorch/dldt/FP16     -p "It is done, and submitted..."     -n 2     -bs 1     -d CPU --torch_compile_backend openvino     -ic 128 --num_beams 1 -lc bfloat16_config.json 2>&1 | tee -a ./logs/0.log

BTW, ChatGLM2's output size is right.

[peterchen-intel]

@JunxiChhen The reason is that ChatGLM3 gives "end token" at output size 17 with the input prompt. The WA is to update prompt to let ChatGLM3 generate more tokens (>=128).
-ic means the maximum output token size (or maximum inference number).
There is a ongoing #289 which is trying to "force" generate expected output size even "end token" is given.

[peterchen-intel]

@JunxiChhen PR has been merged. Please verify.

[yangkunx]

When I update the latest commit id, run case with blew error:

[peterchen-intel]

@yangkunx This new issue should be fixed by PR#435

[peterchen-intel]

#289 was just reverted, due to the performance impact. New PR #457 is WIP

[peterchen-intel]

New PR #457 was merged. @yangkunx @JunxiChhen please verify.