Cortex.cpp: Model Folder

[dan-menlo]

Overview

• Linked: Discussion: Cortex.cpp Data Structures #1040
• Where and how do we store models?
• How do we structure model folders?
• How do we detect models?
• What is the format of our model.yaml?

Docs

• https://cortex.so/docs/model-yaml
• feat: Jan allows bulk import and subfolder recognition for model management jan#2843
• feat: Jan can update Models jan#3251 (poorly scoped)
• bug: Dropping GGUF metadata when creating model.yaml for Model Downloaded via URL jan#3558

[freelerobot]

Data Folder Questions

1. What is the data structure of ~/.cortexcpp/models?
   Where do the following go

• model yaml
• model binaries, especially of multiple binaries (model_1_of_5.bin)
• versions of the same model, e.g. llama3.1, llama3.2
• presets (if any, can also defer for later discussion)

2. Is our preference on a more flat folder structure? rather than super nested?

3. Previously, we had many bugs resulting from expecting folder & file names to be a certain way.
   e.g. we expected unique_model_id (used by backend) to be the same as the model folder name
   Something to be aware of in this iteration 🙏

Model Downloading

1. What happens when model download fails halfway (e.g. internet disconnected)?
2. How do we detect models? e.g. if users "import models locally" would it still work
3. How do we version models?

• If we update our model.yaml, or remote model binary in the HF branch, will download still work?
• Or will "redownloading/updating" fail due to "model exists"

4. Letting users do cortex models update is currently out of scope right?

Model importing

1. Can users import existing models?
2. Is it hard copy or symlink

Model YAML

1. We are auto populating the model.yaml if user downloads a new GGUF file (not from our HF repo)?
2. What happens when user deletes the YAML accidentally?
3. What happens when YAML is there but Binary is deleted? (should we spec little unit tests like this)
4. Is this up to date? https://cortex.so/docs/model-yaml/
5. If users update YAMLs, when do changes take effect?

[vansangpfiev]

Data Folder Questions

Q 1. What is the data structure of ~/.cortexcpp/models?

~/.cortexcpp/
|___ models
       |__ tinyllama.yaml
       |__ tinyllama
       |     |__ model_01.gguf
       |     |__ model_02.gguf
       |     |__ model.yml
       |__ llama3.1
       |__ llama3.2

After downloading model (from cortexso or other HF repository), cortex generates tinyllama.yaml file which is used for model management.

Q 2. Is our preference on a more flat folder structure? rather than super nested?
Can you give an example of the flat folder structure?
Q 3. Can you elaborate more about the issue? Do we have any ticket to track that issue yet?

Model importing
Q 1. Can users import existing models?

• We don't support it yet. TBD when we will support it.
  Q 2. Is it hard copy or symlink
• It will be a symlink

cc: @0xSage

[nguyenhoangthuan99]

Model Yaml

1. Folder structure:

~/.cortexcpp/
|___ models
       |__ tinyllama.yaml
       |__ tinyllama
       |     |__ model_01.gguf
       |     |__ model_02.gguf
       |     |__ model.yml
       |__ llama3.1
       |__ llama3.2

2. With model not from cortexso:

• When model is downloaded, it will parse and save information to <model_id>/model.yml and <model_id>.yaml, this 2 files is the same.
• When user delete 1 of 2 .yaml file. We can provide a command to recover it like cortex-cpp models recover, to check and resolve yml error.

3. when YAML is there but binary is deleted, will raise No such file or directory error when load models.
4. The doc from https://cortex.so/docs/model-yaml/ is up to date.
5. To apply update user need to stop running models and re run chat to load new configuration

[namchuai]

Model Downloading

1. What happens when model download fails halfway (e.g. internet disconnected)?

• We don't support resume failed/pause download.
• If model download is failed, its <model_id>.yaml (inside <data_folder>/models) won't be created and won't display in our model list.

2. How do we detect models? e.g. if users "import models locally" would it still work

• Scan for <model_id>.yaml file inside <data_folder>/models.

3. How do we version models?

• Currently, we using a field version inside yaml file to store the model's version. Please not that we don't have logic to support model update at the moment. For now, Version is just for display purpose.

3.1. If we update our model.yaml, or remote model binary in the HF branch, will download still work? Or will "redownloading/updating" fail due to "model exists"

• It will display model exists.

4. Letting users do cortex models update is currently out of scope right?

• Yes, it is. We haven't work on this.

[freelerobot]

I think there's some confusion on the parameters.

1. Version refers to model.yaml version , not model versions.
2. Versioning the same model remains unaddressed, e.g. we update a parameter or something, how would users know htey need to update the model. Ok if out of scope for now.
3. I strongly advise against forcing yaml filename to be model id, e.g. <model_id>.yaml.

We have an id field inside of the model.yamls So what happens if the user change that field but not the filename?

[freelerobot]

1. I'm confused. Which of the following yamls do we actually use? You guys included 2 yamls for 1 model.

~/.cortexcpp/
|___ models
       |__ tinyllama.yaml                 # this is called tinyllama.yaml. Is this the one?
       |__ tinyllama
       |     |__ model_01.gguf
       |     |__ model_02.gguf
       |     |__ model.yml                 # this is called model.yaml. Or is this the one?

Is it A or B:
A

/models
  model1.yaml       # file: relative or absolute path?
  model2.yaml
  /model1               # this foldername should be flexible
      model1.bin
  /model2
      model2.bin

@vansangpfiev By flat I mean A. It fully decouples the need for model binaries to live inside of /cortex folder path.
And the only place where we track the location of model binary is inside the model.yaml parameter.
But then this introduces edge cases when we update & delete model binaries.

B

/models
  /model1            
      model1.yaml
      model1.bin
  /model2
       model2.yaml
       model2.bin

This is fine, but is it longer to search for all models when we have to look inside folders?

3. Matching model folder name to model binary name to model id. This is what really messed us up in the past. @louis-jan
   Is there a better way? Can we decouple model_id from file/folder names. Users may change those at their whim.
   Recall: there is a files binary location field inside model.yamls that can help track where the binaries are.

What's the relationship among the following

• Folder name
• Binary name
• model.yaml name
• model name (user will likely edit this)
• model id

I think we still lack clarity and not on the same page for this
@vansangpfiev @namchuai @nguyenhoangthuan99 cc @dan-homebrew