InferDoctor v0.6.0 - Closed-Loop AI App Optimization

InferDoctor v0.6.0 - Closed-Loop AI App Optimization

InferDoctor v0.6.0 moves beyond checking whether a local AI stack is reachable. It helps developers diagnose, build, validate, measure, compare, optimize, and verify whether a local or self-hosted AI application actually improved.

Why This Release Matters

A local AI system is not useful merely because it runs. For demos and early customer-facing prototypes, users notice time to first token, streaming behavior, endpoint stability, retrieval delay, cold start, and whether progress is visible. v0.6 adds a closed-loop workflow for improving those practical user-experience signals without turning InferDoctor into a heavyweight benchmark framework.

Guided Application Setup

• inferdoctor quickstart <goal> now gives a concise path from goal to template, endpoint configuration, validation commands, baseline creation, comparison, and optimization planning.
• Supported goals include customer-service, restaurant-ordering, document-qa, rag, local-api, chatbot, and not-sure.
• The command remains advisory. It does not install runtimes, download models, start services, or contact endpoints.

Performance Baselines

• inferdoctor perf baseline create saves sanitized performance smoke-test JSON as a reusable baseline.
• inferdoctor perf baseline show, list, and delete manage local user baselines.
• Baselines redact endpoint credentials and sensitive query parameters.
• Baselines are stored in a user-local data directory by default, not in the repository.

Before-and-After Comparison

• inferdoctor perf compare before.json after.json compares TTFT, total latency, generation duration, TPS, success rate, streaming state, and readiness category.
• Console, JSON, and Markdown output are supported.
• The comparison warns when reports differ by endpoint, model, test type, streaming mode, run count, or metric quality.
• Machine-readable fields remain stable and language-neutral.

Evidence-Based Optimization Plans

• inferdoctor optimize plan turns reports, baselines, comparisons, or supplied runtime facts into prioritized next actions.
• Recommendations include evidence level, observation, action, verification command, expected impact category, and limitation.
• The command does not promise exact performance gains.

Application Experience Profiles

Built-in profiles explain what matters for common application goals:

• interactive-chat
• customer-service
• restaurant-ordering
• document-qa
• rag
• local-api
• batch-processing
• internal-prototype

Profiles explain TTFT, streaming, TPS, total latency, stability, and progress-feedback priorities as transparent heuristics, not universal SLA guarantees.

Safe Local and Private Endpoint Workflows

• Localhost smoke tests remain bounded and timeout-limited.
• LAN or private endpoints require explicit --allow-non-local before sending a live smoke-test prompt.
• InferDoctor uses harmless built-in prompts and does not send private documents.
• Public endpoints are not contacted automatically.
• Endpoint credentials are redacted from console output and saved reports where applicable.

Starter Template Improvements

Principal templates now include performance verification guidance:

• customer-service
• restaurant-ordering
• local-doc-qa

Generated READMEs explain endpoint configuration, streaming, validation, smoke tests, baseline creation, comparison, and optimization planning.

Reference Applications

v0.6 adds small educational reference apps:

• examples/reference_apps/customer_service/
• examples/reference_apps/local_doc_qa/

They demonstrate dry-run, config validation, streaming-first behavior, retrieval progress, and performance verification instructions. They are examples, not production software.

Privacy and Safety Boundaries

InferDoctor v0.6 does not:

• download models
• install AI runtimes
• start or stop services
• modify system settings
• run long load tests
• perform concurrency benchmarks
• evaluate model quality
• upload private documents
• contact public endpoints automatically

Limitations

• Performance commands are smoke tests, not formal benchmarks.
• Recommendations are heuristic and do not promise exact speedups.
• One or two runs are useful for sanity checks, not production capacity planning.
• Model answer quality is not measured.
• Dify integration and the Local/Private RAG Starter Kit are not included in v0.6 and remain future roadmap items.

InferDoctor v0.5.1 - Fix PyPI Documentation Links

InferDoctor v0.5.1 - Fix PyPI Documentation Links

InferDoctor v0.5.1 is a focused documentation hotfix for the PyPI package description.

What changed

• Fixed the Japanese quickstart link when the README is rendered on PyPI.
• Converted repository-relative README links to stable GitHub URLs.
• Updated Japanese installation instructions to use pip install inferdoctor as the primary path.
• Updated Japanese documentation for v0.5 functionality:
  • performance endpoint smoke tests
  • streaming tests
  • TTFT
  • exact versus estimated TPS
  • bounded runs and warmup
  • endpoint optimization
  • RAG optimization
  • streaming-first templates
  • first-step en/zh/ja health-dashboard support
• Added offline README link validation to prevent future PyPI long-description link regressions.

Behavior changes

None.

No runtime behavior, CLI behavior, API behavior, metric schemas, performance smoke-test behavior, or generated template behavior changed.

PyPI

PyPI publishing is intentionally not performed by this release-note step.

InferDoctor v0.5.0 - Performance UX and First-Step i18n

InferDoctor v0.5.0 - Performance UX and First-Step i18n

InferDoctor v0.5.0 expands the project from local AI setup diagnosis into early user-experience validation for local AI apps. It also adds the first step of localized dashboard output for English, Chinese, and Japanese.

InferDoctor remains lightweight and safe by default. It does not download models, install AI runtimes, start services, modify system settings, or run long benchmarks.

Highlights

Performance UX

• inferdoctor perf endpoint for bounded OpenAI-compatible endpoint smoke tests.
• inferdoctor perf streaming for streaming and time-to-first-token smoke checks.
• Clear metric semantics for TTFT, total latency, generation duration, rough TPS, and token estimate quality.
• Bounded --runs and --warmup support for cold and warm observations without becoming a load test.
• Console, JSON, and Markdown performance reports.
• Endpoint credentials and sensitive query parameters are redacted in reports.
• Transparent user-experience readiness categories and heuristic explanations.
• inferdoctor optimize endpoint for endpoint responsiveness advice.
• inferdoctor optimize rag for RAG perceived-latency and progress-feedback advice.
• Streaming-first starter templates with dry-run and config-check paths.

First-Step i18n

• --language auto, --language en, --language zh, and --language ja.
• Localized health dashboard and inferdoctor check console summary.
• Automatic language detection from LANGUAGE, LC_ALL, LC_MESSAGES, and LANG.
• Unsupported language values are rejected instead of silently ignored.
• English fallback remains available.

This is not full CLI localization yet. Non-dashboard commands, generated templates, JSON schemas, Markdown reports, and structured field names may remain English so automation and issue reports stay stable.

Example Commands

pip install inferdoctor
inferdoctor
inferdoctor --language zh
inferdoctor --language ja
inferdoctor perf streaming --endpoint http://127.0.0.1:8000/v1 --model local-model --runs 2 --warmup 1
inferdoctor optimize endpoint --runtime vllm --vram 24 --model-size 14b --streaming
inferdoctor optimize rag --top-k 8 --ttft 2.5 --streaming

Safety Notes

• Performance commands are timeout-bounded smoke tests, not formal benchmarks.
• Model quality is not evaluated.
• No model is downloaded or started.
• No runtime is installed or configured.
• No endpoint is called unless the user explicitly runs a perf smoke-test command with an endpoint and model.
• Template validation and smoke tests remain local, dry-run oriented checks.

Known Limitations

• TPS may be estimated when the endpoint does not return usage fields.
• A small number of runs is useful for smoke testing but not enough for production benchmark claims.
• RAG optimization advice uses user-provided metrics and heuristics; InferDoctor does not profile a vector database or embedding model.
• First-step i18n covers the health dashboard and check summary, not the full CLI.

Upgrade Notes

Existing v0.4.1 workflows remain valid. The main behavior change is that --language is now clearly scoped to the health dashboard and inferdoctor check; unsupported placements or unsupported languages fail visibly.

InferDoctor v0.4.1 - Beginner-Friendly Local AI App Setup

InferDoctor v0.4.1 is the first PyPI release of InferDoctor.

InferDoctor is a local AI stack doctor and setup assistant. It helps developers diagnose local AI environments, understand what their machine can realistically run, and start building practical local AI apps.

Local AI is powerful, but setup is often confusing. Ports, CUDA, drivers, endpoints, runtimes, model size, Docker, templates, and app scaffolding all get in the way. InferDoctor helps you check the stack, understand what is missing, and take the next practical step.

Install

pip install inferdoctor

Quick Start

inferdoctor

Example Workflow

pip install inferdoctor

inferdoctor

inferdoctor stack plan --goal customer-service --vram 24

inferdoctor template create customer-service --output ./customer-service-demo

inferdoctor template validate ./customer-service-demo

inferdoctor template smoke-test ./customer-service-demo

What Is New

• First PyPI release
• Local AI stack health check
• Overall health score and Top Fixes
• Hardware capacity guidance
• Model fit estimates
• Local AI stack planning
• Starter template generation
• Template validation
• Template smoke tests
• Dry-run bootstrap planning
• Beginner-friendly workflow docs
• Support for common local AI tools and runtimes

Supported Checks and Ecosystem

InferDoctor includes lightweight checks and guidance for:

• Ollama
• vLLM
• SGLang
• Xinference
• Dify
• Open WebUI
• LM Studio
• llama.cpp
• Docker
• CUDA
• NVIDIA GPUs
• OpenAI-compatible local endpoints

InferDoctor does not replace these tools. It helps you understand whether your local stack is ready to use them, what is missing, and what to try next.

Setup Assistant Workflow

InferDoctor is no longer just a checker. It now helps users move through a practical local AI setup flow:

1. Diagnose the machine and local AI stack.
2. Understand hardware capacity.
3. Choose a reasonable setup path.
4. Generate a starter app template.
5. Validate the generated project.
6. Run a safe template smoke test.
7. Follow a dry-run bootstrap plan before doing anything risky.

Useful commands:

inferdoctor

inferdoctor capacity

inferdoctor recommend --goal customer-service --vram 24

inferdoctor stack plan --goal customer-service --vram 24

inferdoctor stack bootstrap --goal customer-service --dry-run

inferdoctor template list

inferdoctor template show customer-service

inferdoctor template create customer-service --output ./customer-service-demo

inferdoctor template validate ./customer-service-demo

inferdoctor template smoke-test ./customer-service-demo

inferdoctor model fit --size 14b --quant q4 --vram 24

Starter Templates

InferDoctor includes starter templates for local AI app scenarios such as:

• Customer service chatbot
• Restaurant ordering assistant
• Local document Q&A
• Personal knowledge base
• Meeting notes assistant
• OpenAI-compatible API demo
• Ollama chat starter
• Dify RAG starter
• Open WebUI starter
• vLLM API starter
• SGLang API starter

Templates are designed to be beginner-friendly and local-first. They generate project files, configuration examples, sample data, prompts, and troubleshooting notes.

Why InferDoctor?

Model recommendation tools help you choose a model.

InferDoctor helps you understand whether your local AI stack is ready, what is broken, and what to fix next.

It is designed for people who want to run local AI but do not want to spend hours debugging ports, drivers, endpoints, configs, runtimes, and model size assumptions.

Safety

InferDoctor is lightweight and safe by default.

It does not download models by default.

It does not run model inference by default.

It does not install AI runtimes.

It does not modify system settings.

Template and bootstrap flows are dry-run or file-generation oriented unless explicitly extended in the future.

Capacity and model-fit outputs are heuristic guidance, not benchmark results or hardware guarantees.

Known Limitations

• InferDoctor does not install Ollama, vLLM, SGLang, Dify, Open WebUI, or other runtimes for you.
• It does not download model weights.
• It does not benchmark real inference speed.
• It does not guarantee that a specific model will fit or perform well.
• Some checks depend on local services being reachable on their configured endpoints.
• Capacity and model-fit estimates are intentionally conservative heuristics.

Roadmap

Next planned areas include:

• More starter templates
• Safer bootstrap workflows
• Docker Compose generation improvements
• Dify and Open WebUI starter flows
• Japanese enterprise RAG setup examples
• Community template registry foundation
• More practical local AI workflow guides

Links

GitHub:

https://github.com/anguoyang/inferdoctor

PyPI:

https://pypi.org/project/inferdoctor/

Install:

pip install inferdoctor

InferDoctor v0.3.0 - Local AI Setup Assistant

InferDoctor v0.3.0

InferDoctor v0.2.0 - Local AI Stack Health Check

InferDoctor v0.2.0 turns the project into a practical, screenshot-friendly local AI stack doctor.

Local AI is powerful, but debugging it is painful. InferDoctor helps developers find out why Ollama, vLLM, SGLang, Xinference, Dify, CUDA, NVIDIA, OpenAI-compatible endpoints, or local AI services are not working.

Highlights

• One-command local AI stack health dashboard
• Overall health score
• Top Fixes
• OpenAI-compatible endpoint diagnostics
• SGLang checker
• vLLM, Xinference, Ollama, Dify, CUDA, and NVIDIA checks
• inferdoctor explain
• inferdoctor capacity
• Scenario readiness diagnostics
• Safe diagnostic profile export
• llama.cpp, LM Studio, Open WebUI, and Docker checkers
• Japanese quickstart documentation
• Screenshot-friendly demo outputs
• PyPI packaging preparation

Install

pip install inferdoctor

Quick Start

inferdoctor

Why InferDoctor?

Model recommendation tools help you choose a model.

InferDoctor helps you understand why your local AI stack is broken.

InferDoctor is lightweight and read-only by default.

It does not download models.
It does not run models.
It does not install AI runtimes.
It does not modify system settings.

What's next?

The dev branch is already moving toward v0.3.0 with local AI setup assistance, template generation, guided init, recommendations, and model fit estimation.