[Bug]: Exception ValueError when running inference example #41
Description
Is there an existing issue ? / 是否已有相关的 issue ?
- I have searched, and there is no existing issue. / 我已经搜索过了，没有相关的 issue。
Describe the bug / 描述这个 bug
Exception when running the example:
python inference.py --model_path <vllmcpm_repo_path> --prompt_path prompts/prompt_demo.txt
ValueError: The checkpoint you are trying to load has model type cpm_dragonfly but Transformers does not recognize this architecture. This could be because of an issue with the chec
kpoint, or because your version of Transformers is out of date.
To Reproduce / 如何复现
Follow quick setup guide to setup on Ubuntu 20.04 and python 3.10
Run example:
cd inference/vllm/examples/infer_cpm
python inference.py --model_path <vllmcpm_repo_path> --prompt_path prompts/prompt_demo.txt
Expected behavior / 期望的结果
Run the example without error
Screenshots / 截图
No response
Environment / 环境
- OS: Ubuntu 20.04
- Pytorch: torch 2.2.0
- CUDA: CUDA 12.0
- Device: RTX3080Additional context / 其他信息
Below patch workaround the problem:
diff --git a/inference/vllm/vllm/transformers_utils/config.py b/inference/vllm/vllm/transformers_utils/config.py
index 15ca432..625952c 100644
--- a/inference/vllm/vllm/transformers_utils/config.py
+++ b/inference/vllm/vllm/transformers_utils/config.py
@@ -38,7 +38,7 @@ def get_config(model: str,
raise RuntimeError(err_msg) from e
else:
raise e
- except KeyError as e:
+ except ValueError as e:
if os.path.exists(model):
config = {}
with open(f"{model}/config.json", 'r') as fin: