Add deepseek vl

[AllentDan]

#1321

[irexyc]

@zhoujh1113

麻烦修改下面这两个地方，看下还会不会挂掉或者卡住。cache_max_entry_count 可以设小一点。

backend_config=TurbomindEngineConfig(tp=2, session_len=8192, cache_max_entry_count=0.5)

lmdeploy/lmdeploy/vl/model/deepseek.py

Line 24 in c9b61e3

def __init__(self, model_path, device='cuda'):

这个地方改成cuda:0

https://github.com/InternLM/lmdeploy/blob/c9b61e354b473de5e3d7c319aa3f053ef9bd54f3/lmdeploy/vl/engine.py#L101C2-L108C23
这个地方改成

with torch.device('cuda:0'):
    time_start = time.perf_counter()
    outputs = self.model.forward(inputs)
    time_end = time.perf_counter()
    logger.info(f'ImageEncoder forward {len(inputs)} images, '
                f'cost {time_end - time_start:.3f}s')

[irexyc]

may use init_empty_weights to accelerate loading

[AllentDan]

tried. But seemed the output of the model would be wrong.

[lvhan028]

Accelerating loading model is very important. Please investigate

[lvhan028]

ValueError: Could not find the operator torchvision::nms. Please make sure you have already registered the operator and (if registered from C++) loaded it via torch.ops.load_library.

[AllentDan]

what's the version of torch and torchvision are you using?

torch 2.1.2+cu118
torchvision 0.16.2