InferDoctor v0.6.0 - Closed-Loop AI App Optimization

InferDoctor v0.6.0 - Closed-Loop AI App Optimization

InferDoctor v0.6.0 moves beyond checking whether a local AI stack is reachable. It helps developers diagnose, build, validate, measure, compare, optimize, and verify whether a local or self-hosted AI application actually improved.

Why This Release Matters

A local AI system is not useful merely because it runs. For demos and early customer-facing prototypes, users notice time to first token, streaming behavior, endpoint stability, retrieval delay, cold start, and whether progress is visible. v0.6 adds a closed-loop workflow for improving those practical user-experience signals without turning InferDoctor into a heavyweight benchmark framework.

Guided Application Setup

• inferdoctor quickstart <goal> now gives a concise path from goal to template, endpoint configuration, validation commands, baseline creation, comparison, and optimization planning.
• Supported goals include customer-service, restaurant-ordering, document-qa, rag, local-api, chatbot, and not-sure.
• The command remains advisory. It does not install runtimes, download models, start services, or contact endpoints.

Performance Baselines

• inferdoctor perf baseline create saves sanitized performance smoke-test JSON as a reusable baseline.
• inferdoctor perf baseline show, list, and delete manage local user baselines.
• Baselines redact endpoint credentials and sensitive query parameters.
• Baselines are stored in a user-local data directory by default, not in the repository.

Before-and-After Comparison

• inferdoctor perf compare before.json after.json compares TTFT, total latency, generation duration, TPS, success rate, streaming state, and readiness category.
• Console, JSON, and Markdown output are supported.
• The comparison warns when reports differ by endpoint, model, test type, streaming mode, run count, or metric quality.
• Machine-readable fields remain stable and language-neutral.

Evidence-Based Optimization Plans

• inferdoctor optimize plan turns reports, baselines, comparisons, or supplied runtime facts into prioritized next actions.
• Recommendations include evidence level, observation, action, verification command, expected impact category, and limitation.
• The command does not promise exact performance gains.

Application Experience Profiles

Built-in profiles explain what matters for common application goals:

• interactive-chat
• customer-service
• restaurant-ordering
• document-qa
• rag
• local-api
• batch-processing
• internal-prototype

Profiles explain TTFT, streaming, TPS, total latency, stability, and progress-feedback priorities as transparent heuristics, not universal SLA guarantees.

Safe Local and Private Endpoint Workflows

• Localhost smoke tests remain bounded and timeout-limited.
• LAN or private endpoints require explicit --allow-non-local before sending a live smoke-test prompt.
• InferDoctor uses harmless built-in prompts and does not send private documents.
• Public endpoints are not contacted automatically.
• Endpoint credentials are redacted from console output and saved reports where applicable.

Starter Template Improvements

Principal templates now include performance verification guidance:

• customer-service
• restaurant-ordering
• local-doc-qa

Generated READMEs explain endpoint configuration, streaming, validation, smoke tests, baseline creation, comparison, and optimization planning.

Reference Applications

v0.6 adds small educational reference apps:

• examples/reference_apps/customer_service/
• examples/reference_apps/local_doc_qa/

They demonstrate dry-run, config validation, streaming-first behavior, retrieval progress, and performance verification instructions. They are examples, not production software.

Privacy and Safety Boundaries

InferDoctor v0.6 does not:

• download models
• install AI runtimes
• start or stop services
• modify system settings
• run long load tests
• perform concurrency benchmarks
• evaluate model quality
• upload private documents
• contact public endpoints automatically

Limitations

• Performance commands are smoke tests, not formal benchmarks.
• Recommendations are heuristic and do not promise exact speedups.
• One or two runs are useful for sanity checks, not production capacity planning.
• Model answer quality is not measured.
• Dify integration and the Local/Private RAG Starter Kit are not included in v0.6 and remain future roadmap items.