support Internvl chat v1.1, v1.2 and v1.2-plus

[irexyc]

Motivation

support

• https://huggingface.co/OpenGVLab/InternVL-Chat-Chinese-V1-2
• https://huggingface.co/OpenGVLab/InternVL-Chat-Chinese-V1-2-Plus
• https://huggingface.co/OpenGVLab/InternVL-Chat-Chinese-V1-1

Modification

• add chat template (internvl_zh)
• add source model (InternVLModel)
• add vision model (InternVLVisionModel)
• add InternVLChatTemplateWrapper

Usage:

The vision model is big, so we need to reduce kv-cache in order to load the vision model on single gpu.

With 4 x A100, the memory usage is about 47G x 1 and 35G x 3 when using the following setting. (InternVL-Chat-Chinese-V1-2-Plus)

from lmdeploy import pipeline, TurbomindEngineConfig
from lmdeploy.vl import load_image

pipe = pipeline('/home/chenxin/InternVL-Chat-Chinese-V1-2-Plus/', log_level='INFO',
    backend_config=TurbomindEngineConfig(cache_max_entry_count=0.3, tp=4))

im = load_image('tiger.jpeg')
out = pipe(('描述这个图片', im))

TODO

The vision model is big compraed with qwen / llava which use about 12G gpu memory. I will make another PR to improve it.

[lvhan028]

supported_models.md，以及 readme中的模型支持列表，请更新

[lvhan028]

test with OpenGVLab/InternVL-Chat-Chinese-V1-2-Plus

• pipeline (tp4)
• gradio (tp4)
• api_server (tp4)

[AllentDan]

test with OpenGVLab/InternVL-Chat-Chinese-V1-1

• pipeline (tp4 && tp8)
• api_server (tp4)