Skip to content

feat(model): curate the built-in catalog by hardware target#84

Merged
juhovainio merged 5 commits into
mainfrom
fallback-model-catalog
Jul 8, 2026
Merged

feat(model): curate the built-in catalog by hardware target#84
juhovainio merged 5 commits into
mainfrom
fallback-model-catalog

Conversation

@rominf

@rominf rominf commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

What

rocm model dumped every built-in recipe as a flat list — including tiny smoke/test paths — and didn't say what the list was for. This turns it into a short, curated catalog of current popular open-weight models, grouped by the hardware path each targets, with each entry sized to a quantization that fits a single GPU (multi-GPU serving isn't supported).

Recommended models — run one with `rocm serve <model>`

Strix Halo (Lemonade / llama.cpp)
  unsloth/Qwen3.6-35B-A3B-GGUF:Q4_K_M  Q4_K_M GGUF, ~22 GiB
  unsloth/gemma-4-26B-A4B-it-GGUF:Q4_K_M  Q4_K_M GGUF, ~17 GiB

MI300X (vLLM)
  Qwen/Qwen3.6-27B  BF16, ~54 GiB (1x MI300X)
  google/gemma-4-31B-it  BF16, ~62 GiB (1x MI300X)

These are recommendations — you can serve any compatible Hugging Face model:
  rocm serve <owner/repo>          # vLLM, e.g. Qwen/Qwen3.6-27B
  rocm serve <owner/repo>:<quant>  # Lemonade GGUF, e.g. unsloth/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Use `rocm model --verbose` for details.

How

  • The built-in list is now an embedded JSON file (model_catalog.json) using the same schema as a signed recipe index, so the offline default and hosted indexes share one format and the catalog is easy to curate. A unit test parses and validates it, so a malformed catalog can't ship.
  • Featured models use canonical Hugging Face ids with the quant that fits one GPU. Strix Halo entries use the owner/repo:variant GGUF form that rocm serve needs (via the Lemonade canonical-name serving path); MI300X entries serve at BF16.
  • rocm model shows only the curated featured list for the built-in catalog, grouped by hardware target, plus a note that any compatible Hugging Face model can be served. A configured recipe index is shown in full and named in the header.
  • The default assistant, smoke/test, and superseded recipes stay fully resolvable for rocm serve and are listed under rocm model --verbose (annotated as hidden) — nothing is removed from resolution.

Testing

  • cargo test -p rocm-core (incl. catalog parse/validate + featured/platform helpers) and cargo test -p rocm --bin rocm model pass.
  • cargo clippy / cargo fmt clean.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall the design is clean — embedding a JSON catalog, parsing it once with OnceLock, and reusing the existing recipe-index schema is a good approach. A few findings below, ranging from a medium bug to a doc nit.

Comment thread apps/rocm/src/main.rs Outdated
Comment thread crates/rocm-core/src/model_catalog.json
Comment thread crates/rocm-core/src/lib.rs Outdated
Comment thread crates/rocm-core/src/lib.rs
@juhovainio

Copy link
Copy Markdown
Collaborator

Two follow-up suggestions for this PR or a future iteration:

1. Move platform definitions into model_catalog.json

MODEL_CATALOG_PLATFORMS is currently a hardcoded &[&str] in lib.rs, and model_recipe_target_platform maps engine names to those strings with a match arm. This has the same sync-drift risk as the FEATURED_MODEL_IDS constant. A top-level platforms array in the JSON (alongside recipes) would keep everything in one place:

"platforms": [
  {
    "label": "Strix Halo (Lemonade / llama.cpp)",
    "engines": ["lemonade", "llama.cpp", "llamacpp"],
    "gfx_families": ["gfx120X-all"]
  },
  {
    "label": "MI300X (vLLM)",
    "engines": ["vllm"],
    "gfx_families": ["gfx906", "gfx908", "gfx90a", "gfx90a-all"]
  }
]

model_recipe_target_platform would then look up the recipe's first preferred engine against this list instead of a hardcoded constant. Adding a new hardware target in future only requires a JSON edit.

2. Highlight the current platform in rocm model output

The gfx target detection already exists (detect_host_gfx_target + normalize_therock_family, the same probe rocm examine uses). If render_model_registry_text_with_context_and_host called it and matched the result against each platform's gfx_families, the matching platform header could be annotated — e.g.:

Strix Halo (Lemonade / llama.cpp)  ← your GPU
  unsloth/Qwen3.6-35B-A3B-GGUF:Q4_K_M  ...

MI300X (vLLM)
  Qwen/Qwen3.6-27B  ...

This makes it immediately clear which section is relevant for the user's hardware without them having to know which GPU maps to which platform. The detection is a fast sysfs read on Linux / WMI probe on Windows, same cost as rocm examine.

rominf and others added 5 commits July 8, 2026 17:43
`rocm model` listed every built-in recipe flat, including tiny smoke/test
paths, and did not say what the list was for. Turn it into a short curated
list of current popular open-weight models, each sized to a quant that fits
a single GPU (multi-GPU serving is not supported).

- Author the built-in list as an embedded JSON file using the same schema as
  a signed recipe index, so the offline default and hosted indexes share one
  format and the catalog is easy to curate. A unit test parses and validates
  it so a malformed catalog cannot ship.
- Feature current popular models under canonical Hugging Face ids with the
  quant that fits one GPU: Qwen3.6-35B-A3B and Gemma 4 26B-A4B as Q4_K_M GGUF
  on Strix Halo (served via Lemonade's owner/repo:variant form), and
  Qwen3.6-27B and Gemma 4 31B as BF16 on one MI300X via vLLM.
- `rocm model` shows only the featured list for the built-in catalog, grouped
  by hardware target, with a note that any compatible Hugging Face model can be
  served. A configured recipe index is shown in full and named in the header.
  The default assistant, smoke/test, and superseded recipes stay resolvable for
  `rocm serve` and appear under `--verbose`.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
- Add `featured` boolean field to ModelRecipeRecord; mark the four
  curated models in model_catalog.json. model_recipe_featured() now
  reads the field directly, eliminating the hardcoded FEATURED_MODEL_IDS
  Rust constant that had to be kept in sync by hand.

- Add `platforms` array to ModelRecipeIndexDocument with label, engines,
  and gfx_families per platform. MODEL_CATALOG_PLATFORMS constant and
  the engine-name match arm are removed; model_recipe_target_platform_label()
  looks up the recipe's engine against the catalog's platform list instead.
  Signed indexes that omit platforms fall back to the built-in definitions.

- rocm model highlights the matching platform with "← your GPU" by
  probing detect_host_gfx_target() + normalize_therock_family() and
  matching against each platform's gfx_families list.

- Footer "These are recommendations..." is now only emitted for the
  built-in catalog, not for operator-configured SignedIndex registries.

- Remove dangling gpt-oss-120b manual_alternative entries (no recipe
  in the catalog resolves that id).

- Fix docstring: catalog validation is test-time, not build-time.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
Signed-off-by: Juho Vainio <juho.vainio@amd.com>
Signed-off-by: Juho Vainio <juho.vainio@amd.com>
Add three hardware platform entries to model_catalog.json replacing the
previous two (Strix Halo / MI300X):

- AMD Ryzen AI — Strix Halo (Lemonade / llama.cpp): gfx1151
- AMD Radeon — RX 7000/9000, Radeon PRO (Lemonade / llama.cpp): gfx110X-all, gfx120X-all
- AMD Instinct — MI300X, MI350X, MI355X (vLLM): gfx906, gfx908, gfx90a, gfx94X-dcgpu, gfx950-dcgpu

Lemonade recipes now appear under both Strix Halo and Radeon by
matching per-platform engine lists directly rather than assigning each
recipe to only the first matching platform.

Signed-off-by: Juho Vainio <juho.vainio@amd.com>
@juhovainio
juhovainio force-pushed the fallback-model-catalog branch from 2b9f761 to 3b841e4 Compare July 8, 2026 17:44
@juhovainio
juhovainio enabled auto-merge July 8, 2026 17:50
@juhovainio
juhovainio added this pull request to the merge queue Jul 8, 2026
Merged via the queue into main with commit 21640b6 Jul 8, 2026
15 checks passed
@juhovainio
juhovainio deleted the fallback-model-catalog branch July 8, 2026 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants