Support qwen1.5 in turbomind engine

[lvhan028]

Note: window attention is not supported.

[zhyncs]

Hi @lvhan028 Does this pr support Qwen1.5-0.5B and Qwen1.5-1.8B? Thanks.

[lvhan028]

0.5B no, since its head_dim is 64 while the turbomind engine hardcode the head_dim 128
1.8B Yes.

[zhyncs]

0.5B no, since its head_dim is 64 while the turbomind engine hardcode the head_dim 128 1.8B Yes.

Thanks for your reply. Maybe we could also update the supported table. Thanks.

[lvhan028]

0.5B no, since its head_dim is 64 while the turbomind engine hardcode the head_dim 128 1.8B Yes.

Thanks for your reply. Maybe we could also update the supported table. Thanks.

sure. updated.

[RunningLeon]

opecompass results look OK

模型	mmlu	gsm8k
qwen1.5-7b-chat-HF	61.48	55.65
qwen1.5-7b-chat-TB	61.47	54.74

[RunningLeon]

LGTM

[irexyc]

Have you tried lmdeploy convert command to convert the model?

[lvhan028]

Have you tried lmdeploy convert command to convert the model?

not yet. I'll test it asap

[lvhan028]

lmdeploy convert qwen <qwen1.5-model-path>

[irexyc]

Traceback (most recent call last):
  File "/home/chenxin/miniconda3/envs/38/bin/lmdeploy", line 33, in <module>
    sys.exit(load_entry_point('lmdeploy', 'console_scripts', 'lmdeploy')())
  File "/home/chenxin/ws3/vl/lmdeploy/cli/entrypoint.py", line 26, in run
    args.run(args)
  File "/home/chenxin/ws3/vl/lmdeploy/cli/cli.py", line 151, in convert
    main(**kwargs)
  File "/home/chenxin/ws3/vl/lmdeploy/turbomind/deploy/converter.py", line 265, in main
    tokenizer_path = get_tokenizer_path(model_path, tokenizer_path)
  File "/home/chenxin/ws3/vl/lmdeploy/turbomind/deploy/converter.py", line 45, in get_tokenizer_path
    assert tokenizer_path, 'please supply tokenizer path by --tokenizer-path'
AssertionError: please supply tokenizer path by --tokenizer-path

[lvhan028]

Traceback (most recent call last):
  File "/home/chenxin/miniconda3/envs/38/bin/lmdeploy", line 33, in <module>
    sys.exit(load_entry_point('lmdeploy', 'console_scripts', 'lmdeploy')())
  File "/home/chenxin/ws3/vl/lmdeploy/cli/entrypoint.py", line 26, in run
    args.run(args)
  File "/home/chenxin/ws3/vl/lmdeploy/cli/cli.py", line 151, in convert
    main(**kwargs)
  File "/home/chenxin/ws3/vl/lmdeploy/turbomind/deploy/converter.py", line 265, in main
    tokenizer_path = get_tokenizer_path(model_path, tokenizer_path)
  File "/home/chenxin/ws3/vl/lmdeploy/turbomind/deploy/converter.py", line 45, in get_tokenizer_path
    assert tokenizer_path, 'please supply tokenizer path by --tokenizer-path'
AssertionError: please supply tokenizer path by --tokenizer-path

What's the command?

[irexyc]

What's the command?

lmdeploy convert qwen /mnt/140/Qwen/Qwen1.5-7B-Chat/

[xiaoxiaoyuwen]

@lvhan028 dose this support qwen1.5-14b-chat int8 kv cache ?
I run the calibrate cause error

RuntimeError: Currently, quantification and calibration of Qwen2ForCausalLM are not supported.

[lvhan028]

hi, @xiaoxiaoyuwen
we are going to remove offline kv int8 and adopt online kv int8 in PR #1377
PR #1412 provides the guide.
This feature will be released in the next version. Stay tuned.