Set trust remote code for custom model #435
Conversation
wgzintel
commented
May 10, 2024
wgzintel
commented
- set trust_remote_code=true for 'baichuan', 'qwen', 'chatglm' to permit loading of these models.
- enable Phi2 and starcoder models.
…openvino.genai into guozhong/set_trust_remote_code
llm_bench/python/utils/ov_utils.py
Outdated
| @@ -143,7 +143,12 @@ def create_text_gen_model(model_path, device, **kwargs): | |||
| else: | |||
| remote_code = False | |||
| try: | |||
| model_config = AutoConfig.from_pretrained(model_path) | |||
| remote_models = ['baichuan', 'qwen', 'chatglm'] | |||
There was a problem hiding this comment.
I think we need to differentiate qwen and qwen1.5 models here, because the second one does not required remote code
There was a problem hiding this comment.
@eaidova Since these models throw exception in model_class.from_pretrained() instead of AutoConfig.from_pretrained(), we are thinking to catch the exception of model_class.from_pretrained() and call both AutoConfig.from_pretrained()/model_class.from_pretrained() again with trust_remote_code=True. Any preference from your side?
There was a problem hiding this comment.
from my side no, only possibly it may be time consuming as leads to loading model twice, but as we speak only for loading IR and model already downloaded, it should not be issue
There was a problem hiding this comment.
could you please check with these changes?
#440
explicit setting trust_remote_code=False for config prevent waiting user input and allow capturing exception as expected
peterchen-intel
merged commit 86c2098
into
openvinotoolkit:releases/2024/1
