ModelScout

Discover. Test. Find what works.

A provider-neutral Python CLI and tool for discovering models exposed by OpenAI-compatible AI API endpoints, actually testing whether those models work, recording health and latency, and producing a reliable local registry of usable models.

---

The Core Principle

DISCOVERED ≠ WORKING

A model appearing in GET /v1/models does not prove that it can successfully process a real inference request. Catalog metadata only reveals what an endpoint advertises. Often, advertised models are deprecated, misconfigured, cold, unauthenticated, rate-limited, or completely failing upstream.

ModelScout verifies LLM endpoints with deterministic, lightweight probes to build a factual registry of working models:

OpenAI-compatible API
        │
        ▼
    Discovery (/v1/models)
        │
        ▼
  Real Probes (/v1/chat/completions)
        │
        ▼
 Health Classification & Latency
        │
        ▼
Persistence (Local SQLite Registry)
        │
        ▼
Working Model Registry View

---

Why ModelScout?

• Zero Fabricated Assumptions: Eliminates guessing which models behind an aggregator or self-hosted endpoint are currently operational.
• Provider-Neutral: Works universally against any OpenAI-compatible API (e.g. vLLM, Ollama, LocalAI, LM Studio, commercial gateways, or proxies).
• Compound Identity (provider:model): Distinguishes provider_a:llama-3 from provider_b:llama-3. Models on different hosts are tracked with independent latency and reliability history.
• Lightweight Real Probes: Issues minimal token requests (Reply with exactly: OK, max 5 tokens) to test actual execution without burning quota or context windows.
• Derived Working Registry: Querying your registry is instant and read-only from local storage. Query commands never fire unwanted external network probes.
• Zero External Dependencies: Built strictly on the Python standard library (urllib, sqlite3, concurrent.futures, json, argparse). Lightweight, fast, and easy to audit.

---

Architecture

modelscout/
├── cli/          # Command-line interface (discover, check, models, status)
├── config/       # Global configuration and environment settings
├── connections/  # OpenAI-compatible HTTP client and connection manager
├── discovery/    # /v1/models query engine and catalog normalization
├── probes/       # Minimal real-request probe executor with bounded concurrency
├── health/       # Health state definitions and persistence-aware state tracker
├── registry/     # Derived working-model registry view and metrics aggregation
├── ranking/      # Latency and reliability scoring foundation
└── storage/      # Local SQLite database schema and persistence layer

See docs/ARCHITECTURE.md for full design specifications.

---

Installation

From Source

git clone https://github.com/xxxurya/ModelScout.git
cd ModelScout
pip install .

For development:

pip install -e .

Requires Python 3.8+ with zero external dependencies.

---

Quick Start & Usage

ModelScout provides a clean, Unix-friendly command-line interface. All commands support --json for automation and script integration.

1. Discover Models

Query GET /v1/models from any OpenAI-compatible endpoint and store the advertised catalog locally:

modelscout discover --url "https://api.example.com/v1" --api-key "YOUR_API_KEY"

Output:

[✓] Connection 'custom' (custom): Discovered 12 models (142.3ms)
+-------------------------+--------------------+--------+
| Model ID                | Owned By           | Object |
+-------------------------+--------------------+--------+
| gpt-4o-mini             | system             | model  |
| meta-llama/llama-3.3-70b| system             | model  |
| qwen/qwen-2.5-72b       | system             | model  |
+-------------------------+--------------------+--------+

2. Run Real Health Probes

Send minimal real inference probes (POST /v1/chat/completions) across discovered models to measure latency and test operational health:

modelscout check --url "https://api.example.com/v1" --api-key "YOUR_API_KEY"

Filter probes to specific model names or providers:

# Probe only models matching 'llama'
modelscout check --model llama

# Discover first, then probe with 10 concurrent workers
modelscout check --discover-first --concurrency 10

Sample output:

[✓] custom:gpt-4o-mini -> HEALTHY (218.4ms) [OK]
[✓] custom:meta-llama/llama-3.3-70b -> HEALTHY (412.1ms) [OK]
[✗] custom:unsupported-model-x -> UNAVAILABLE (85.2ms) [Model not found or unallocated]

3. View Working Registry

Query the derived local registry. Notice that models reads directly from local SQLite storage without issuing network requests:

# List all verified working models
modelscout models

# List only healthy models
modelscout models --status WORKING

# List non-working, degraded, or temporarily unavailable models
modelscout models --failed

# Include all historical catalog entries (even if inactive/unadvertised)
modelscout models --all

# Rank verified models by fastest latency
modelscout models --fastest

# Rank verified models by consecutive reliability streak
modelscout models --reliable

Sample output:

+---------------------+-------------------+---------+-----------+---------+---------+
| Provider            | Model ID          | Status  | Latency   | Success | Streak  |
+---------------------+-------------------+---------+-----------+---------+---------+
| custom              | gpt-4o-mini       | WORKING | 218.4ms   | 100.0%  | 5       |
| custom              | llama-3.3-70b     | WORKING | 412.1ms   | 100.0%  | 3       |
+---------------------+-------------------+---------+-----------+---------+---------+

4. Overall Health Status

Inspect registry metrics, test coverage, and model availability breakdown:

modelscout status

Output:

========================================
ModelScout Status Summary
========================================
Connections Configured:    1
Registry Summary:
  Total Discovered:        12
  Total Tested:            12
  [✓] Working:             10
  [!] Degraded:            1
  [~] Temp Unavailable:    0
  [✗] Not Working:         1
  [?] Not Tested:          0
========================================

---

Configuration

ModelScout works out-of-the-box with CLI flags or can load configurations from ~/.modelscout/config.json (or via --config path/to/config.json / MODELSCOUT_CONFIG).

Example Configuration (~/.modelscout/config.json)

{
  "concurrency": 5,
  "probe_timeout": 15.0,
  "degraded_latency_threshold_ms": 3000.0,
  "connections": [
    {
      "id": "local-vllm",
      "name": "Local vLLM Server",
      "base_url": "http://127.0.0.1:8000/v1"
    },
    {
      "id": "gateway",
      "name": "Gateway Service",
      "base_url": "https://api.example.com/v1",
      "api_key_env": "GATEWAY_API_KEY"
    }
  ]
}

Environment Variables

Variable	Description	Default
MODELSCOUT_API_KEY	Default fallback API key if not specified per connection	None
MODELSCOUT_BASE_URL	Quick default base URL if no configuration file is used	None
MODELSCOUT_CONFIG	Path to custom configuration JSON	~/.modelscout/config.json
MODELSCOUT_DATA_DIR	Directory for local SQLite database and caches	~/.modelscout
MODELSCOUT_CONCURRENCY	Maximum concurrent probe workers	5
MODELSCOUT_TIMEOUT	Network timeout for probe requests in seconds	15.0

---

Health States & Transitions

ModelScout classifies model responsiveness into deterministic health states:

Health State	HTTP / Condition Mapping	Registry Usability Status
HEALTHY	HTTP 200 within latency threshold	WORKING
DEGRADED	Latency exceeds threshold, or 1-2 transient failures on previously working model	DEGRADED
RATE_LIMITED	HTTP 429	TEMPORARILY_UNAVAILABLE
AUTH_ERROR	HTTP 401 / 403	NOT_WORKING
TIMEOUT	Network or socket timeout	NOT_WORKING
UPSTREAM_ERROR	HTTP 5xx or connection disconnects	NOT_WORKING
UNAVAILABLE	HTTP 404 or unsupported model error, or >=3 consecutive failures	NOT_WORKING
UNKNOWN	Discovered model not yet probed	NOT_TESTED

Transition Dynamics

• Success: Transitions to HEALTHY (or DEGRADED if response duration exceeds the configurable latency threshold), resets failure count, and increments reliability streak.
• First Failure: If previously healthy, transitions to DEGRADED to preserve stability while signaling reduced confidence.
• Repeated Failures: Two consecutive failures classify as DEGRADED. Three or more consecutive failures classify as UNAVAILABLE.
• Immediate Terminal Failures: Authentication errors (401/403) immediately trigger AUTH_ERROR; rate limits (429) immediately trigger RATE_LIMITED.

---

Security & Privacy

• Zero Credential Persistence: ModelScout never writes API keys, tokens, or Authorization headers to SQLite, log files, or terminal traces.
• In-Memory Secrets: Credentials passed via --api-key or environment variables exist strictly in volatile memory for outbound requests.
• Sanitized Outputs: All JSON exports sanitize authentication headers and connection secrets.
• Isolated Local Storage: Runtime databases (~/.modelscout/modelscout.db) are kept locally and are explicitly git-ignored.

---

Development & Testing

ModelScout includes a hermetic automated test suite using in-process HTTP mock servers. No external network access or live API keys are required to run tests.

# Run unit & integration test suite
python3 -m unittest discover tests

---

Contributing

Contributions are welcome! Please read CONTRIBUTING.md for development environment setup, coding guidelines, and pull request conventions.

---

License

MIT License. See LICENSE for details.