[BUG] webui加载模型后，回答全用英文

[twwch]

是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this?

• 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答？ | Is there an existing answer for this in FAQ?

• 我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior

webui加载模型后，回答全用英文

期望行为 | Expected Behavior

用中文回答

复现方法 | Steps To Reproduce

generation_config.json

{
  "chat_format":"chatml",
  "do_sample": true,
  "eos_token_id": 151643,
  "max_new_tokens": 512,
  "max_window_size": 6144,
  "pad_token_id": 151643,
  "top_k": 0,
  "top_p": 0.5,
  "transformers_version": "4.31.0"
}

config.json

{
  "_name_or_path": "./",
  "architectures": [
    "QWenLMHeadModel"
  ],
  "attn_dropout_prob": 0.0,
  "auto_map": {
    "AutoConfig": "configuration_qwen.QWenConfig",
    "AutoModelForCausalLM": "modeling_qwen.QWenLMHeadModel"
  },
  "bf16": false,
  "emb_dropout_prob": 0.0,
  "fp16": false,
  "fp32": false,
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 22016,
  "kv_channels": 128,
  "layer_norm_epsilon": 1e-06,
  "max_position_embeddings": 8192,
  "model_type": "qwen",
  "no_bias": true,
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "onnx_safe": null,
  "rotary_emb_base": 10000,
  "rotary_pct": 1.0,
  "scale_attn_weights": true,
  "seq_length": 2048,
  "tie_word_embeddings": false,
  "tokenizer_type": "QWenTokenizer",
  "torch_dtype": "bfloat16",
  "transformers_version": "4.31.0",
  "use_cache": true,
  "use_dynamic_ntk": true,
  "use_flash_attn": false,
  "use_logn_attn": true,
  "visual": {
    "heads": 16,
    "image_size": 448,
    "image_start_id": 151857,
    "layers": 48,
    "mlp_ratio": 4.9231,
    "output_dim": 4096,
    "patch_size": 14,
    "width": 1664
  },
  "vocab_size": 151936
}

运行环境 | Environment

- OS:ubuntu
- Python:3.9
- Transformers:4.31.0
- PyTorch:2.0.1
- CUDA (`python -c 'import torch; print(torch.version.cuda)'`):11.7

备注 | Anything else?

No response

[twwch]

回答效果也没有demo页面的效果好

[ShuaiBai623]

可以尝试更小的top p(如0.3/0.4)看看是否有帮助

[twwch]

可以尝试更小的top p(如0.3/0.4)看看是否有帮助

还是英文回复

[ShuaiBai623]

我这边重新clone代码测试也没有遇到这种情况，你可以尝试下重新拉下代码

[twwch]

英文回答的问题解决了，但是效果没有demo好，例如

demo中

自行搭建的

图片

[twwch]

回答明显是错误的，demo的推理参数能否分享一下