enhancement: improved cortex model hub

[ramonpzg]

The cortex hub currently shows the models that can be pulled using cortex pull <model-name> without any additional information about the model. In the future, we want the hub to provide high- and low-level details per model on different hardware where cortex can be deployed to and on commonly used benchmarks like MMLU or SWE on regular and custom hardware. The information provided won't be the same as the Model Cards provided by HuggingFace but more akin to a Menu of useful information where users can pick whichever recipe suits them best.

For example, each row will have a little drop-down arrow on the right-hand side:

Each arrow will reveal a menu of metrics that the user can tweak for each model.

Each model will have a dedicated page as well with additional information that won't fit in the table above. The table will slightly resemble a model card but will be focused on benchmarks alongside a mini-tutorial for usage with Cortex. We'll call it a "Bench Card."

We would have this be fully automated.

1. Pick a model
2. Trigger script that
   1. Quantizes the Model
   2. Runs Benchmarks on Different Hardware
   3. Capture Results
3. Have an LLM describe
4. Generate a YAML file and populate it
5. Add it to the Hub

For the YAML file, we can take inspiration from the one used in the model cards and have something like:

model-index:
  - name: llama3
    results:
      - task:
          type: hardware-benchmark
        architecture:
          - name: x86
            - time-to-first-token: 0.7
            - ...
          - name: arm
            -

[ramonpzg]

Any thoughts on this for a few sprints down the line @louis-menlo and @dan-menlo ? 🤔

[gau-nernst]

@ramonpzg Your suggestions look good! Personally for me, I think having a "model size" (in terms of GB) is useful. Currently both https://cortex.so/models and cortex pull doesn't show model size -> good to estimate how much to download as well as if model can fit in RAM