Mistral AWQ not generating any output

[annthsu]

I have tried all the methods mentioned in #2728, but none of them have worked at all,responses are always empty
I use 2*A6000 for GPU
Here is my Docker run command
Can anyone help me for this issue?Thank you !

docker run --rm --runtime nvidia --gpus all --env NCCL_P2P_DISABLE=1 --env CUDA_VISIBLE_DEVICES=0,1 --shm-size 10g -p 8002:8002 \
    -e MODEL=/app/model \
    -e API=se \
    -v /home/merge_model/:/app/model \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --ipc=host \
    vllm/vllm-openai:latest \
    --port 8002 \
    --model /app/model/mistral/testtest \
    --served-model-name "openchat" \
    --max-num-seqs 2000 \
    --max-num-batched-tokens 327680 \
    --tensor-parallel-size 2 \
    --disable-custom-all-reduce \
    --enforce-eager \
    --quantization 'awq' \
    --gpu-memory-utilization 1 \
    --max-model-len 21440

[whitebill-eng]

Hi! Try to use another chat template ex.: --chat-template ./examples/template_chatml.jinja

[hmellor]

Closing as stale