You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I registered a custom model named test-llama-3-instruct which was already present in my directory using xinference register. After registration when the model is launched using xinference launch. It throws an error at
After diagnosis found that URI is creating the wrong symlink. Instead of /home/ubuntu/projects/llm/LLaMA/Meta-Llama-3-8B-Instruct, it creates a symlink to /home/ubuntu/projects/llm/LLaMa/Meta-Llama-3-8B-Instruct", so the program to register the model creates a wrong cache link and coverts LLaMA in URI to camelcase LLaMa probably its an error in logic. Its a very subtle bug and took me long to figure out,but the problem can be fixed if I update the symlink. Now will try to see if can fix the code and create a PR, but its a trivial change so probably core developer can do it.
{
"version": 1,
"context_length": 8192,
"model_name": "test-llama-3-instruct",
"model_lang": [
"en"
],
"model_ability": [
"chat"
],
"model_description": "Llama 3 is an auto-regressive language model that uses an optimized transformer architecture",
"model_family": "llama-3-instruct",
"model_specs": [
{
"model_format": "pytorch",
"model_size_in_billions": 8,
"quantizations": [
"4-bit",
"8-bit",
"none"
],
"model_id": "test-llama-3-instruct",
"model_uri": "file:///home/ubuntu/projects/llm/LLaMA/Meta-Llama-3-8B-Instruct"
}],
"prompt_style": {
"style_name": "LLAMA3",
"system_prompt": "You are a helpful assistant.",
"roles": [
"user",
"assistant"
],
"intra_message_sep": "\n\n",
"inter_message_sep": "<|eot_id|>",
"stop_token_ids": [
128001,
128009
],
"stop": [
"<|end_of_text|>",
"<|eot_id|>"
]
}
}
To Reproduce
Just use the given custom_model.json file to register the model and try to launch the inference model.
Your Python version.
3.11.3
The version of xinference you use.
Commit: cb9dbb2
Expected behavior
It should not change the directory path to CamelCase should take it as is.
Additional context
None
The text was updated successfully, but these errors were encountered:
Describe the bug
I registered a custom model named test-llama-3-instruct which was already present in my directory using xinference register. After registration when the model is launched using xinference launch. It throws an error at
inference/xinference/model/llm/llm_family.py
Line 383 in cb9dbb2
After diagnosis found that URI is creating the wrong symlink. Instead of /home/ubuntu/projects/llm/LLaMA/Meta-Llama-3-8B-Instruct, it creates a symlink to /home/ubuntu/projects/llm/LLaMa/Meta-Llama-3-8B-Instruct", so the program to register the model creates a wrong cache link and coverts LLaMA in URI to camelcase LLaMa probably its an error in logic. Its a very subtle bug and took me long to figure out,but the problem can be fixed if I update the symlink. Now will try to see if can fix the code and create a PR, but its a trivial change so probably core developer can do it.
To Reproduce
Just use the given custom_model.json file to register the model and try to launch the inference model.
Your Python version.
3.11.3
The version of xinference you use.
Commit: cb9dbb2
Expected behavior
It should not change the directory path to CamelCase should take it as is.
Additional context
None
The text was updated successfully, but these errors were encountered: