-
Notifications
You must be signed in to change notification settings - Fork 0
AI and Model Validation
AI is entirely optional and off by default. Inspectarr scores indexers with a deterministic weighted formula on its own, and detection, severity and remediation never consult a model at all. Ollama adds an advisory overlay on top of indexer scoring, plus natural-language notification digests.
Settings → AI opens with an Enable AI features toggle. Everything else on the page depends on it.
While it is off:
- indexer AI scoring is skipped and unavailable on the Indexers page
- AI notification digests and periodic summaries are skipped
- no request is ever made to an Ollama server
- the rest of the AI settings, and the Ollama options on the Notifications page, are greyed out
Turning it off does not erase anything. Your URL, model and system prompt stay on disk, so switching back on restores exactly what you had.
Detection, severity grading, quarantine and remediation are unaffected either way — none of them use AI.
prowlarr:
ollama:
enabled: false # master switch; ships offUpgrading? If AI was already working before this switch existed, it stays on — a configured URL is treated as enabled unless you say otherwise. Only new installs start disabled.
Turn the switch on first; the fields below stay disabled until you do.
| Setting | Purpose |
|---|---|
| Enable AI features | Master switch. Off by default. |
| Ollama URL | Base URL of your Ollama server, e.g. http://192.168.1.10:11434. |
| Timeout | Hard cutoff per scoring run, in seconds. Allow extra on the first run after a model swap — the model has to load. |
| Model | Which model performs scoring. Populated live from Ollama. |
| Cache TTL | How long an AI result is reused for identical input. |
| Model update check | How often to check whether your model has been replaced. 0 disables. |
| System prompt | The scoring prompt. Loads with the built-in default; edit to customise. |
A Test Connection button confirms the URL before you rely on it.
prowlarr:
ollama:
enabled: true
url: "http://192.168.1.10:11434"
model: "qwen2.5:7b-instruct"
timeout: 300
cache_ttl_hours: 24
update_check_hours: 24
system_prompt: "" # empty = built-in defaultNot every model can do this job, and the ways they fail are quiet rather than loud. A model with too small a context window does not error — it loses the scoring instructions and returns fabricated data that looks like a valid score. Others return the right shape with the wrong key names, or echo the input back as "reasoning".
Validation catches all of that before a model can be selected.
1. Discrimination. One unambiguously good indexer and one unambiguously bad one, scored in a single call. The model must rate the good one ≥ 70, the bad one ≤ 40, and separate them by at least 25 points.
The margin matters: a model scoring them 55 and 50 orders them correctly while being useless. This also catches models that return a constant, and models with an inverted scale.
2. Schema compliance. Every returned entry must carry a numeric health_score in range and a non-empty reasoning string that is not simply the input echoed back.
3. Context capacity. Synthetic indexers matching your actual indexer count are sent in one prompt, and every one must come back with no invented IDs.
This is the test that matters most. Because it is sized to your deployment, "passed" means passed here — not in the abstract.
Two buttons sit beside the model selector, and they are not alternatives.
Calibrate does everything Quick Check does, and then one more thing.
| Quick Check | Calibrate | |
|---|---|---|
| Discrimination | yes | yes |
| Schema compliance | yes | yes |
| Context capacity | yes | yes |
| Batch ceiling search | no | yes |
| Typical duration | 1–3 minutes | 10–40 minutes |
| Resulting badge | amber ✓ validated
|
green ✓ calibrated
|
If you only ever press Calibrate, you are not skipping anything. Quick Check exists because it is fast, not because it tests something Calibrate misses.
Selecting a model in the dropdown chooses a candidate; it is not applied until you click Apply. Both runs happen in the background — you can leave the page — and only one runs at a time.
Calibrate searches for the largest number of indexers this model handles reliably in one request, trying 25, then 20, 15, 10 and 5, stopping at the first size that works.
Each candidate size must come back clean twice in a row. That is where the time goes, and it is deliberate.
Silent omission is intermittent. One model here passed a single-pass check at 39 indexers and, the next day on the same host with nothing changed, dropped 15 of 39. One clean run is a single sample of something that fails only some of the time. Accepting it is how a confident, wrong number gets written down and then trusted for months.
A candidate must also finish comfortably inside your timeout, not merely produce the right answer. A batch of 25 once scored 39 of 39 on a CPU-only host while taking 269–295 seconds against a 300 second timeout — a setting that works right up until the machine is slightly busier, and then a whole batch returns nothing.
Failed sizes cost one run each; only the winning size costs two. And if discrimination or schema fail, Calibrate stops without searching at all — hunting for the best batch size of a model that cannot score is minutes spent proving nothing.
- Auditioning models. Six candidates, and you want to know which can score at all. Quick Check each in a couple of minutes, then Calibrate only the one or two worth keeping. Spending 40 minutes calibrating a model you are about to discard is wasted time.
-
Re-checking after an
ollama pullon unchanged hardware, to confirm the new build still behaves. - A temporary host you are still configuring or about to replace.
- Very few indexers — if everything fits in one call, the ceiling question barely arises.
When Quick Check is not enough: any time you are about to rely on a model for real scoring on hardware you have not measured. That is exactly what the amber badge is telling you.
This is the part most easily missed.
The batch ceiling is a property of the model and the machine together, so a result earned on one server says little about another. Measured here, the identical model and batch size took 436 seconds on one host and 237 on another — and a model that scored cleanly at 25 per call on one dropped indexers on the other.
Inspectarr records which Ollama URL each verdict came from, shown in the Host column:
| Host column | Meaning |
|---|---|
this host |
The verdict describes the server you are using now |
other host |
Earned elsewhere — re-run it here |
unknown |
Recorded before Inspectarr tracked hosts |
If you change prowlarr.ollama.url, existing verdicts do not carry over. Selecting a model whose only passing result came from a different server is refused, and the message names both servers.
A model that has not passed cannot be applied by accident. Apply does not start a run; it explains the situation and offers Cancel or Apply anyway, with wording matched to the actual problem — never tested, tested and failed, previously forced, or passed on a different host.
Applying anyway records the model as forced — distinct from failed, because you chose it knowingly. A later successful validation supersedes that.
A model that is validated but not calibrated applies normally, with an inline note: it scores correctly, and only its batch ceiling is unmeasured.
| Colour | State | Meaning |
|---|---|---|
| default text | — untested |
Nothing is known about this model here |
| red |
✗ failed / ⚠ forced / ⚠ other host
|
Do not rely on it |
| amber | ✓ validated |
It scores correctly; the batch ceiling was never measured |
| green | ✓ calibrated |
Validated, and the ceiling is measured on this host |
Colour is never the only signal — every state also carries a symbol and a word, because some browsers (notably Safari on macOS and most mobile pickers) ignore colour inside a dropdown entirely.
Amber is not a warning that something is wrong. It means one specific thing is unmeasured. A model can sit at amber indefinitely and score perfectly well; you are simply relying on the default batch size rather than one measured on your hardware.
The comparison table below the selector lists every model you have tested — per-test results, the calibrated batch size, which host it was measured on, average response time and date. It is the practical way to answer "which of these actually works here", especially when hunting for one model that serves several applications.
- You changed the Ollama host, or its URL
- You changed the model
- The host's hardware changed — a GPU came online, or was removed
-
ollama pullreplaced the build behind a model name - Your indexer count grew substantially
Inspectarr records throughput and GPU offload alongside each verdict, so a result measured on hardware that has since changed can be spotted rather than silently trusted.
If the model behind a name is replaced (a re-pull of the same tag), the stored digest no longer matches and Inspectarr flags it so you can re-validate. It checks the local digest only and never pulls — the check is free and changes nothing on your Ollama host.
When a URL and model are configured, Inspectarr sends all indexer data in one batch and the returned scores replace the deterministic ones. See Prowlarr Indexer Scoring.
Results are cached by a hash of the payload plus the model and system prompt, so changing either always rescores rather than silently reusing the previous model's answers.
There is no longer a hard context floor. The prompt is compact and short-keyed, and Inspectarr splits the work across several calls when it will not fit — see Small context windows below. A 4k model is workable.
What matters instead is the batch ceiling: how many indexers the model will faithfully reason about in one call, which is lower than how many fit and is measured by Calibrate.
Independently of scoring, Ollama can narrate notification digests and periodic summaries — notifications.digest.use_ollama and notifications.summary.use_ollama. Both controls are disabled on the Notifications page until AI is enabled and an Ollama URL is configured. See Notifications.
Asking for prose does not guarantee prose. A model fine-tuned on a structured task will answer in its own schema regardless of the prompt — one security-log model returned a full MITRE ATT&CK object, and early versions sent that entire blob to the phone, braces and all.
Inspectarr now checks the shape of the answer, not just that a reply arrived:
| The model returns | What is sent |
|---|---|
| Prose | The prose |
| JSON containing a readable summary field | That field |
| JSON with nothing readable in it | The built-in plain summary |
| A reply cut off mid-object | The built-in plain summary |
Falling back is not a failure — the plain summary is deterministic and always correct, and it is strictly better than a wall of JSON. The reason is written to the log so you can see it happened.
Length is capped rather than merely requested, since "max 500 characters" in a prompt is a request and truncating mid-word is what happens when a model ignores it.
If your notifications look like JSON, the model is the cause, not the configuration. Any model works for scoring and narration independently, so a model that scores well but narrates badly can simply have narration turned off.
- Prowlarr Indexer Scoring — what the AI score overlays
-
Configuration — every
prowlarr.ollamakey
Inspectarr checks whether a newer build of your model has been published, and can pull it for you.
The check costs one request and never downloads anything to find out — Ollama's local model digest happens to be the hash of the published manifest, so comparing them is enough. The badge tells you which kind of update it found, because they call for opposite responses:
| Badge | Meaning | What to do |
|---|---|---|
| Model changed since validation | The build installed locally is not the one you validated | Re-validate |
| Newer build available upstream | A newer build is published | Pull it, then re-validate |
Pulling runs in the background with a progress bar and can be several gigabytes; scoring keeps using the current build until it finishes. The new build is not validated — a different build of the same name can score differently, which is the entire reason the validation gate exists.
prowlarr:
ollama:
auto_update_check: true # ask the registry; false to never call out
update_check_hours: 24 # how often to checkauto_update_check is the one check that leaves your network, so it is
opt-out. Everything else Inspectarr does with Ollama stays on your LAN.
Cached AI scores cannot go stale across a rebuild: the scoring cache is keyed on the model's digest as well as its name, so a re-pulled model can never serve the previous build's scores.
Scoring used to send every indexer in one prettified prompt. At 37 indexers that is roughly 5,500 tokens, so a model with a 4k window lost the instructions and returned invented data instead of erroring.
The prompt is now compact and uses short field names — about 3,000 tokens for the same 37 — and Inspectarr splits the work across several calls when needed, merging the results. You do not have to do anything; it simply takes a little longer when it splits.
This is worth knowing if you are choosing a model. Measured on
qwen2.5-coder:7b:
| Indexers | Window | Result |
|---|---|---|
| 25 | 4k | All 25 scored correctly |
| 30 | 4k | Echoes the input back instead of scoring — in 3 seconds, with the window barely half full |
| 37 | 8k | Well-formed JSON that silently leaves five out |
The last row is the one to watch for. It looks exactly like success: nothing errors, the response parses, and the missing indexers quietly keep their deterministic score.
So how many indexers a model will actually reason about is a separate limit from how many fit, and it is much lower.
prowlarr:
ollama:
context_window: 4096 # also sent to Ollama as num_ctx
max_indexers_per_call: 25 # the item limit, not the token limitBoth are properties of your model. If yours is larger, raise them and re-run validation — the context test is exactly the check that tells you whether it copes.