Repository navigation
AI Model Selection Routing Privacy and Cost Management
Last reviewed: 2026-11-09 ยท Models, prices, quotas and service terms change; verify before deployment.
A model-routing policy decides which provider or local model should perform a specific class of work based on required capability, privacy, measured quality, latency, resilience and price. Cheapest per token is not necessarily cheapest per verified outcome.
| Workload | Hard Requirements | Typical Preferred Route |
|---|---|---|
| Sensitive client evidence summary | Approved data boundary, access control, grounding | Authorized local/private endpoint |
| Public documentation lookup | Source access, accurate citations | Current retrieval + capable model |
| Coding with repository tools | Tool/schema reliability, code eval pass | Model tested on target repo/tasks |
| High-stakes release review | Independent evidence, approvals | Verified testing + reviewer, not model alone |
| Routine synthetic log categorization | Low cost, fixed schema, fast response | Small model if benchmark passes |
| Long research report | Large source set, context and grounding | RAG/hybrid retrieval + evaluated model |
| Offline lab | No external network and auditable local setup | Pinned local model/environment |
A local model avoids cloud inference calls only if all embedding, tool, telemetry, retrieval and fallback paths remain local. Self-hosting is not automatically a privacy guarantee.
Evaluate each candidate against a standardized dataset and profile:
| Dimension | Measure | Failure Threshold |
|---|---|---|
| Correctness | Acceptance by verified task rubric | Below required pass rate |
| Grounding | Claim-to-source support | Unsupported critical claims |
| Tool behavior | Invalid/unauthorized calls | Any unapproved high-impact side effect |
| Context | Input/output limits and real retrieval fidelity | Truncation or lost source |
| Latency | P50/P95 end-to-end | Exceeds service objective |
| Availability | Error/timeouts over a defined window | Unacceptable outage impact |
| Privacy | Geography, retention, subprocessor/data terms | Outside authorized policy |
| Cost | Input/output/cache/tool/retry costs per accepted task | Exceeds budget |
| Operations | Deployment, rollback, updates, licensing | Unmaintainable |
Record date, model ID, provider, API version, settings, quantization (for local models), prompt revision and tool versions. Avoid claims that a specific named model is categorically best forever.
INPUT -> classify sensitivity, scope and required tools
-> deny unapproved destinations
-> choose tested model/provider meeting capability + data policy
-> enforce time/cost/turn budget
-> validate output and tool choices
-> fallback ONLY if destination has equivalent approval and controls
-> verify end state and record actual cost/performance
Illustrative policy, not a vendor configuration file:
routes:
confidential_client:
approved_destinations: ["local-private"]
require_retrieval_acl: true
fallback: "stop-and-escalate"
public_research:
approved_destinations: ["cloud-provider-a", "local-private"]
require_source_citations: true
fallback: "another-approved-destination"
synthetic_triage:
approved_destinations: ["small-local-model"]
max_tool_calls: 6
require_schema_validation: true
global:
max_retry_attempts: 2
no_unapproved_cross_boundary_fallback: true
require_human_release_approval: trueWhen a private endpoint fails, do not silently send confidential input to an external API as a fallback.
For a usage-priced API, estimate:
estimated_model_cost =
input_tokens / 1_000_000 * input_price_per_million
+ output_tokens / 1_000_000 * output_price_per_million
+ any applicable tool, retrieval, cache or other service charges
For locally hosted models also count amortized hardware, power, storage, monitoring, maintenance and operator time. Include unsuccessful attempts, retries and evaluation runs.
Illustrative calculation only: at assumed rates of $1.00 per million input tokens and $4.00 per million output tokens, a task using 15,000 input and 3,000 output tokens would have $0.015 + $0.012 = $0.027 model usage cost, excluding all other charges. These are hypothetical rates, not current provider prices.
More meaningful metric: total cost per independently accepted task, including failures and rework. Measure both unit economics and quality; a cheap model needing four retries can be worse than a more expensive reliable model.
- Choose 30+ representative, approved test tasks with explicit success criteria and risk categories.
- Include a held-out set, exact identifiers, long-context tasks, tool errors and ambiguous requests.
- Fix prompt, retrieval source version and tool schemas as much as possible.
- Run each candidate with same data and bounded settings; record model ID and revisions.
- Collect real input/output token usage, latency, failed calls and source grounding.
- Independently check accepted results, especially security and client-facing outputs.
- Compare Pareto frontier: acceptable quality versus actual cost/latency.
- Apply privacy/security hard gates before optimizing cost.
- Test fallback during controlled outage and rate-limit conditions.
- Publish routing decisions with date and re-evaluation triggers.
For Ollama or another local stack:
- Pin model artifact and quantization; verify checksums/provenance.
- Check GPU/CPU memory and concurrent-user throughput.
- Restrict listening interfaces and API access; require authentication at the gateway where appropriate.
- Validate local embedding service, telemetry exporter and network egress, not inference endpoint alone.
- Track model load times, queue depth, first-token latency, inference speed, memory pressure and unexpected OOM.
- Test model upgrades on a staging endpoint and keep rollback artifacts.
- Keep client data separated across workspaces and indexes.
Use Offline LLM Deployment for architecture and installation details; this guide focuses on routing decisions and measurement.
| Failure | Why | Mitigation |
|---|---|---|
| Confidential input sent to cloud | Blind automatic fallback | Destination allowlist and fail-closed rule |
| Cheap route hallucinates | No task-specific eval | Quality floor and sample-based audit |
| Huge context becomes expensive | Excessive retrieval dumps | Chunk/rerank, compaction and budgets |
| API model changes behavior | Unpinned alias / provider update | Record model ID, benchmark release |
| Local inference overload | More concurrent jobs than capacity | Queue/concurrency caps and backpressure |
| Repeated tool loops consume budget | No stop condition | Max turns, calls, time and spend |
| Prompt cache hides costs | Incorrect usage attribution | Track actual billed usage and cache metadata |
A routing decision is ready when (a) target data boundary approved, (b) test suite run and reviewed, (c) measurable quality floor achieved, (d) fallback tested and privacy-preserving, (e) tool permissions validated, (f) costs measured rather than guessed, and (g) owner has an upgrade/rollback plan.
Offline LLM ยท Agent Evals ยท Observability ยท RAG ยท Context Engineering
References: OpenAI Model Docs ยท OpenAI Agent Evals ยท Ollama Docs ยท NIST AI RMF.
๐ Source: AI/model-routing-and-cost-management.md | โ๏ธ Suggest an edit | โฌ
๏ธ Back to Artificial Intelligence
- ๐บ๏ธ Browse by Topic
- ๐ All Repository Files
- ๐งญ Learning Paths
- ๐ Master Guide
- ๐ค Glossary
- โ๏ธ Legal & Disclaimer
๐ Foundations & Learning
๐ก๏ธ Defense & Investigations
- Incident Response
- Open Source Intelligence (OSINT)
- Reverse Engineering
- Operations Security (OPSEC)
- Compliance
โ๏ธ Platforms & Infrastructure
๐ก Hardware, Wireless & Field Projects
- PNWC Cyberdeck
- Hardware Testing
- Hardware Hacking
- Software-Defined Radio
- Wardriving
- uConsole
- Space Security
๐งฐ Assessment & Field Reference
๐ฆ Downloads & Project Resources
๐๏ธ Additional Topics