Skip to content

InferDoctor v0.5.0 - Performance UX and First-Step i18n

Choose a tag to compare

View all tags
@anguoyang anguoyang released this 12 Jul 11:17
v0.5.0
f37ff3d

InferDoctor v0.5.0 - Performance UX and First-Step i18n

InferDoctor v0.5.0 expands the project from local AI setup diagnosis into early user-experience validation for local AI apps. It also adds the first step of localized dashboard output for English, Chinese, and Japanese.

InferDoctor remains lightweight and safe by default. It does not download models, install AI runtimes, start services, modify system settings, or run long benchmarks.

Highlights

Performance UX

  • inferdoctor perf endpoint for bounded OpenAI-compatible endpoint smoke tests.
  • inferdoctor perf streaming for streaming and time-to-first-token smoke checks.
  • Clear metric semantics for TTFT, total latency, generation duration, rough TPS, and token estimate quality.
  • Bounded --runs and --warmup support for cold and warm observations without becoming a load test.
  • Console, JSON, and Markdown performance reports.
  • Endpoint credentials and sensitive query parameters are redacted in reports.
  • Transparent user-experience readiness categories and heuristic explanations.
  • inferdoctor optimize endpoint for endpoint responsiveness advice.
  • inferdoctor optimize rag for RAG perceived-latency and progress-feedback advice.
  • Streaming-first starter templates with dry-run and config-check paths.

First-Step i18n

  • --language auto, --language en, --language zh, and --language ja.
  • Localized health dashboard and inferdoctor check console summary.
  • Automatic language detection from LANGUAGE, LC_ALL, LC_MESSAGES, and LANG.
  • Unsupported language values are rejected instead of silently ignored.
  • English fallback remains available.

This is not full CLI localization yet. Non-dashboard commands, generated templates, JSON schemas, Markdown reports, and structured field names may remain English so automation and issue reports stay stable.

Example Commands

pip install inferdoctor
inferdoctor
inferdoctor --language zh
inferdoctor --language ja
inferdoctor perf streaming --endpoint http://127.0.0.1:8000/v1 --model local-model --runs 2 --warmup 1
inferdoctor optimize endpoint --runtime vllm --vram 24 --model-size 14b --streaming
inferdoctor optimize rag --top-k 8 --ttft 2.5 --streaming

Safety Notes

  • Performance commands are timeout-bounded smoke tests, not formal benchmarks.
  • Model quality is not evaluated.
  • No model is downloaded or started.
  • No runtime is installed or configured.
  • No endpoint is called unless the user explicitly runs a perf smoke-test command with an endpoint and model.
  • Template validation and smoke tests remain local, dry-run oriented checks.

Known Limitations

  • TPS may be estimated when the endpoint does not return usage fields.
  • A small number of runs is useful for smoke testing but not enough for production benchmark claims.
  • RAG optimization advice uses user-provided metrics and heuristics; InferDoctor does not profile a vector database or embedding model.
  • First-step i18n covers the health dashboard and check summary, not the full CLI.

Upgrade Notes

Existing v0.4.1 workflows remain valid. The main behavior change is that --language is now clearly scoped to the health dashboard and inferdoctor check; unsupported placements or unsupported languages fail visibly.

Assets 2
Loading