Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: model_uri converts symlink in cache to CamelCase case automatically resuling in a wrong path to directory or file #1569

Open
vikrantrathore opened this issue May 31, 2024 · 0 comments
Labels
bug Something isn't working
Milestone

Comments

@vikrantrathore
Copy link

vikrantrathore commented May 31, 2024

Describe the bug

I registered a custom model named test-llama-3-instruct which was already present in my directory using xinference register. After registration when the model is launched using xinference launch. It throws an error at

os.symlink(src_root, cache_dir, target_is_directory=True)

After diagnosis found that URI is creating the wrong symlink. Instead of /home/ubuntu/projects/llm/LLaMA/Meta-Llama-3-8B-Instruct, it creates a symlink to /home/ubuntu/projects/llm/LLaMa/Meta-Llama-3-8B-Instruct", so the program to register the model creates a wrong cache link and coverts LLaMA in URI to camelcase LLaMa probably its an error in logic. Its a very subtle bug and took me long to figure out,but the problem can be fixed if I update the symlink. Now will try to see if can fix the code and create a PR, but its a trivial change so probably core developer can do it.

{
    "version": 1,
    "context_length": 8192,
    "model_name": "test-llama-3-instruct",
    "model_lang": [
	"en"
    ],
    "model_ability": [
	"chat"
    ],
    "model_description": "Llama 3 is an auto-regressive language model that uses an optimized transformer architecture",
    "model_family": "llama-3-instruct",
    "model_specs": [
	{
	    "model_format": "pytorch",
	    "model_size_in_billions": 8,
	    "quantizations": [
		"4-bit",
		"8-bit",
		"none"
	    ],
	    "model_id": "test-llama-3-instruct",
	    "model_uri": "file:///home/ubuntu/projects/llm/LLaMA/Meta-Llama-3-8B-Instruct"
	}],
    "prompt_style": {
	"style_name": "LLAMA3",
	"system_prompt": "You are a helpful assistant.",
	"roles": [
	    "user",
	    "assistant"
	],
	"intra_message_sep": "\n\n",
	"inter_message_sep": "<|eot_id|>",
	"stop_token_ids": [
	    128001,
	    128009
	],
	"stop": [
	    "<|end_of_text|>",
	    "<|eot_id|>"
	]
    }
}

To Reproduce

Just use the given custom_model.json file to register the model and try to launch the inference model.

  1. Your Python version.
    3.11.3

  2. The version of xinference you use.
    Commit: cb9dbb2

Expected behavior

It should not change the directory path to CamelCase should take it as is.

Additional context

None

@XprobeBot XprobeBot added the bug Something isn't working label May 31, 2024
@XprobeBot XprobeBot modified the milestones: v0.11.3, v0.11.4, v0.12.0, v0.12.1 May 31, 2024
@XprobeBot XprobeBot modified the milestones: v0.12.1, v0.12.2 Jun 14, 2024
@XprobeBot XprobeBot modified the milestones: v0.12.2, v0.12.4 Jun 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants