discussion: Simplify Model Structures for Cortex

[gabrielle-ong]

Simplify Model Structures - Models Names (Repo, Source, version, ID, Alias)

Goals:

1. Model list should be compact and not overwhelm user with super long model names
2. User's UX - should be easy to call a specific model with an expected alias

Concepts

• Model Repo: i.e. tinyllama, or bartowski/...
• Model Source: huggingface, cortex
• Model Version: i.e. specific quant, that belongs to a Model Repo
• Model ID: should be :
• Model Alias: user-defined shortname

Questions

1. How are we currently generating the model alias for HF models?
2. How do we align with user expectations? to run the model using the same alias they pulled it with?
3. How does this map between model.yaml and model table in cortex.db, which need to be mapped to Model List

Resources

Output of cortex models list

[nguyenhoangthuan99]

Logic for generating short aliases for models in the nightly version:

1. Start with the unique model ID, e.g.,
   huggingface.co:bartowski:Meta-Llama-3.1-8B-Instruct-GGUF:Meta-Llama-3.1-8B-Instruct-Q2_K

2. Check if the lowercase version of the last part (Meta-Llama-3.1-8B-Instruct-Q2_K ->
   meta-llama-3.1-8b-instruct-q2_k) is unique. If so, use it as the alias. If not, proceed to step 3.

3. Progressively include more parts of the ID from right to left, converting to lowercase,
   until a unique alias is found or the full original ID is reached.

Rationale for this approach:

1. Many quantization levels are typically stored in the same HuggingFace repository folder
   (e.g., https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/tree/main, https://huggingface.co/QuantFactory/Mistral-Nemo-Base-2407-GGUF/tree/main).
   The filename often contains sufficient information about the model architecture and
   quantization level. When users download from HuggingFace, providing the GGUF filename
   as the primary alias helps them identify the specific quantization level they're using or testing.
   

2. Generating a shortened, unique, and human-readable alias from a random string is challenging.
   Using only part of the ID (e.g., bartowski:Meta-Llama-3.1-8B-Instruct-GGUF) as an alias
   could lead to confusion when users want to pull different quantization levels from the same
   repository.

3. To improve user experience, we should clearly display both the correct model ID and the
   model alias after download completion. The current approach needs improvement in this regard.

This is my personal perspective and doesn't necessarily represent the majority of users. If you have any recommendations for improving the algorithm to enhance user experience, I'm open to updating the logic. I believe changing the default model alias to bartowski/Meta-Llama-3.1-8B-Instruct-GGUF may not be ideal. The CLI app is primarily used by developers, who typically require more detailed information about the model they're working with.

[dan-menlo]

@nguyenhoangthuan99 I see your point. I think we need to broaden the scope of the discussion beyond Model Alias, and deal more broadly with how we handle "Model Naming" in Cortex (i.e. Repo, Version, ID, aliases)

Model Nomenclature

I think we should clearly define different concepts:

• Model Repo: i.e. tinyllama, or bartowski/...
• Model Source: huggingface, cortex
• Model Version: i.e. specific quant, that belongs to a Model Repo
• Model ID: should be <Model Source>:<Version>
• Model Alias: user-defined shortname

We should also be very clear on the behavior for cortex pull and cortex run.

• Most users will be used to having the Model Repo as the "name"
• They used that phrase to pull the repo, and would ostensibly take that as the Model ID
• Model Alias is a "shortname" that is user-defined (should not be system defined)

Model List

• Given the complexity of Huggingface Model IDs and names, I think we need a simpler way to show Model Lists
• I've created a separate issue in #1426 to track this

Aliases for HF Models?

• We can consider forcing users to "alias" a Huggingface model as they are downloading it
• As you correctly pointed out, Huggingface Model names are very long
• However, I could see this being very complicated for users, UX-wise

cc @gabrielle-ong @0xSage
cc @vansangpfiev @namchuai

[dan-menlo]

Merging in my issue on model list, which is really an issue on Model Naming across our repo:

Issue

I would like to propose a clearer way to handle model list. I think our current method leads to poor UX, and overcomplicates things:

Goal

• Model list should be compact and not overwhelm user with super long model names
  • Current status quo is unreadable
• Huggingface listings are very long/complicated
  • I don't think there's an easy path to simplify them
  • We need to brainstorm for ways to simplify
• Model List will be part of other commands, e.g. cortex run without any args pulls up numbered list for user to choose from

The big issue with this is the Huggingface Models:

• We need to figure out model id and model alias for Huggingface
• The issue is that there is no standard format for Huggingface

From my POV, the "raw" data structure is as follows. We will need to simplify this to human-readable terms:

Index	model repo	model version	model source	engine
1	tinyllama	q4_km	cortex	llama.cpp
2	tinyllama	q2	cortex	llama.cpp
3	bartowski/Meta-Llama-3.1-8B-Instruct-GGUF	meta-llama-3.1-8b-Instruct-IQ2_m.gguf	huggingface	llama.cpp
4	bartowski/Meta-Llama-3.1-8B-Instruct-GGUF	meta-llama-3.1-8b-instruct-q4k_m.gguf	huggingface	llama.cpp

Resources

Current Model List looks really bad for Huggingface models

[dan-menlo]

@nguyenhoangthuan99 @gabrielle-ong For Model Naming, can I recommend the following:

• We have / in human-readable Model IDs and versions
• This should match the "model source" that users use when cortex pull or cortex run
• We can convert this to : behind the scenes, e.g. for folders etc

Cortex

This is very straightforward, as we are defining the standards:

Field	Value	Example
Model Repo	repo_name	tinyllama
Model Version	branch_name	q4k_m
Model Source	domain	cortex
Model ID	UUID repo:branch	tinyllama:q4k_m
Model Alias	user-defined	string

Huggingface

Field	Value	Example
Model Repo	repo_name	bartowski/Meta-Llama-3.1-8B-Instruct-GGUF tinyllama
Model Version	quant_filename	meta-llama-3.1-8b-Instruct-IQ2_m.gguf
Model Source	domain	huggingface
Model ID	UUID repo:branch	bartowski/Meta-Llama-3.1-8B-Instruct-GGUF tinyllama:meta-llama-3.1-8b-Instruct-IQ2_m.gguf
Model Alias	user-defined	string

[dan-menlo]

@nguyenhoangthuan99 For Huggingface Models, I think the long-term solution is:

1. User is prompted to pick alias when pulling Huggingface repo
2. cortex run <huggingface_repo> shows menu if more than 1 version is downloaded
3. cortex run without args shows list of downloaded models, user can pick by number

The core issue to this is that Model ID should ideally != Huggingface Repo name and filename, which is insanely long. I see a user-defined alias as the only long-term solution.