refactor: model manager v3 #8607

psychedelicious · 2025-10-09T10:19:54Z

Summary

This PR:

Refactors the model manager's identification (aka probing/classification) system. The new system is an evolution of the classify API introduced in Model classification api #7742. All legacy probe logic has been ported over to the new system, too.
Introduces the concept of "unknown" models (i.e. a model that we couldn't positively identify) so that a failure to identify a model doesn't require the user to re-download the model.
Normalizes model storage into a flat directory structure.
Introduces correctness tests for model identification (aka probing/classification). Previously, we did not test for correctness except for a very small number of cases. There is a detailed guide for adding test cases, which is pretty easy.

It adds 3 migrations:

Update the models table, removing the UNIQUE constraint on the combination of name, base and model type. The only unique thing for models is now the model's path on disk.
Migrate model configs in the DB where necessary.

This PR was previously split into 3 smaller PRs:

TODO

Update the model manager UI to let users fix an incorrectly-identified model. This is partially working.
Add guide for adding support for a new model type (identification, loading, testing).

Related Issues / Discussions

Gajillions of related issues.

QA Instructions

Because this PR does both DB and model file storage migrations, it needs careful testing on backed-up Invoke installs. I will add more detail about how to test it when I do the next alpha release.
The new model identification system works differently from the legacy probe and it is likely some things won't identify correctly. We'll need to test installing a variety of models, ideally adding them to the new model identification test suite.

Merge Plan

This will need an alpha release before RC.

Checklist

The PR has a short but descriptive title, suitable for a changelog
Tests added / updated (if applicable)
❗Changes to a redux slice have a corresponding migration
Documentation added / updated (if applicable)
Updated What's New copy (if doing a release after this PR)

…, simplify model manager models list

I can't remember what this was for and don't see any reference to it. Maybe it's just remnants from a previous implementation?

…lects

…name model"

… list of configs

Store models in a flat directory structure. Each model is in a dir named its unique key (a UUID). Inside that dir is either the model file or the model dir.

now worky

- Add concept of match certainty to new probe - Port CLIP Embed models to new API - Fiddle with stuff

Split the big migration that did all of these things into 3: - Migration 22: Remove unique contraint on base/name/type in models table - Migration 23: Migrate configs to v6.8.0 schemas - Migration 24: Normalize file storage

…ails

Previously a path w/ any of the known weights suffixes would be seen as a weights file, even if it was a directory. We now check to ensure the candidate path is actually a file before adding it to the list of weights.

Overhaul of model identification (probing) tests. Previously we didn't test the correctness of probing except in a few narrow cases - now we do. See tests/model_identification/README.md for a detailed overview of the new test setup. It includes instructions for adding a new test case. In brief: - Download the model you want to add as a test case - Run a script against it to generate the test model files - Fill in the expected model type/format/base/etc in the generated test metadata JSON file Included test cases: - All starter models - A handful of other models that I had installed - Models present in the previous test cases as smoke tests, now also tested for correctness

psychedelicious added 30 commits October 9, 2025 08:03

feat(mm): add UnknownModelConfig

f784731

refactor(ui): move model categorisation-ish logic to central location…

108976b

…, simplify model manager models list

refactor(ui)refactor(ui): more cleanup of model categories

2998f0e

refactor(ui): remove unused excludeSubmodels

fffa7b5

I can't remember what this was for and don't see any reference to it. Maybe it's just remnants from a previous implementation?

feat(nodes): add unknown as model base

106a956

chore(ui): typegen

a11f543

feat(ui): add unknown model base support in ui

fcb68e9

feat(ui): allow changing model type in MM, fix up base and variant se…

159b960

…lects

feat(mm): omit model description instead of making it "base type file…

2b231ec

…name model"

feat(app): add setting to allow unknown models

66c02bd

feat(ui): allow changing model format in MM

0799a81

feat(app): add the installed model config to install complete events

724f6bd

chore(ui): typegen

b85faaa

feat(ui): toast warning when installed model is unidentified

86cc48e

docs: update config docstrings

8bc9526

chore(ui): typegen

2358413

tests(mm): fix test for MM, leave the UnknownModelConfig class in the…

a5de59d

… list of configs

tidy(ui): prefer types from zod schemas for model attrs

ce6864d

chore(ui): lint

3f809a5

fix(ui): wrong translation string

52635db

feat(mm): normalized model storage

fb44048

Store models in a flat directory structure. Each model is in a dir named its unique key (a UUID). Inside that dir is either the model file or the model dir.

feat(mm): add migration to flat model storage

4d3d8b5

fix(mm): normalized multi-file/diffusers model installation no worky

96159da

now worky

refactor: port MM probes to new api

cabc627

- Add concept of match certainty to new probe - Port CLIP Embed models to new API - Fiddle with stuff

feat(mm): port TIs to new API

57218ff

tidy(mm): remove unused probes

3f7e7d9

feat(mm): port spandrel to new API

e6443ab

fix(mm): parsing for spandrel

33a8e3a

fix(mm): loader for clip embed

bca822c

fix(mm): tis use existing weight_files method

ae7f51f

psychedelicious added 14 commits October 9, 2025 08:03

feat(mm): fix clip vision starter model bases, add ref to actual models

af5b5d9

feat(mm): add model config schema migration logic

eb7040e

fix(mm): duplicate import

0eaf300

refactor(mm): split big migration into 3

fc104db

Split the big migration that did all of these things into 3: - Migration 22: Remove unique contraint on base/name/type in models table - Migration 23: Migrate configs to v6.8.0 schemas - Migration 24: Normalize file storage

fix(mm): pop base/type/format when creating unknown model config

6197b50

fix(db): migration 22 insert only real cols

917a847

fix(db): migration 23 fall back to unknown model when config change f…

8eb8fde

…ails

feat(db): run migrations 23 and 24

b6abe34

fix(mm): false negative on flux lora

133f8f6

fix(mm): vae checkpoint probe checking for dir instead of file

4decb91

fix(mm): ModelOnDisk skips dirs when looking for weights

41d7c02

Previously a path w/ any of the known weights suffixes would be seen as a weights file, even if it was a directory. We now check to ensure the candidate path is actually a file before adding it to the list of weights.

feat(mm): add method to get main model defaults from a base

3ed5cfa

feat(mm): do not log when multiple non-unknown model matches

ec8c871

refactor(mm): continued iteration on model identifcation

bdad8fe

github-actions bot added api python PRs that change python files Root invocations PRs that change invocations backend PRs that change backend files services PRs that change app services frontend PRs that change frontend files python-tests PRs that change python tests labels Oct 9, 2025

psychedelicious mentioned this pull request Oct 9, 2025

feat: allow unknown models to be installed #8582

Closed

5 tasks

psychedelicious force-pushed the psyche/refactor/mm-3 branch from e99d3d7 to 7c00cdf Compare October 9, 2025 10:32

psychedelicious added 4 commits October 10, 2025 10:29

fix(mm): omit type/format/base when creating unknown config instance

4e660f6

feat(mm): use ValueError for model id sanity checks

d74eda5

feat(mm): add flag for updating models to allow class changes

c4d8203

tests(mm): fix remaining MM tests

e155f10

psychedelicious force-pushed the psyche/refactor/mm-3 branch from 5044e15 to e155f10 Compare October 10, 2025 01:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor: model manager v3 #8607

refactor: model manager v3 #8607

Uh oh!

psychedelicious commented Oct 9, 2025

Uh oh!

Uh oh!

refactor: model manager v3 #8607

Are you sure you want to change the base?

refactor: model manager v3 #8607

Uh oh!

Conversation

psychedelicious commented Oct 9, 2025

Summary

TODO

Related Issues / Discussions

QA Instructions

Merge Plan

Checklist

Uh oh!

Uh oh!