Skip to content

v1.2.0: Model Registry & Multi-Model Routing

Choose a tag to compare

View all tags
@zjykzj zjykzj released this 02 Sep 13:11
· 11 commits to main since this release
v1.2.0
fa8e8e9

Added

  • Model registry: declarative multi-model config (models/registry.yaml, PyYAML; models/registry.example.yaml shipped) + request-level model routing via the model field — sync /predict, /predict/segment, /predict/classify and async /predict/query, /predict/callback accept it; engines/registry.py is pure metadata (lazy parse + process cache, no predictor instances); without a registry file the service synthesizes a single-model registry from the historical INFERFORGE_[SEG_|CLS_]MODEL_PATH env vars (byte-identical behavior, zero-cost upgrade)
  • Model registry: per-model class names (optional classes: txt per entry; omitted -> built-in COCO-80 / ImageNet-1k); out-of-range class ids degrade to a class_N label + warning instead of failing the request (engines.base.class_label, used by draw_detections and the task-layer JSON mapping)
  • Model registry: business code 10 (model not found / capability mismatch) registered in utils/response.py + docs/status-codes.md; async submit apis reject unknown models synchronously (validate_model) and workers re-check for web/worker registry drift; defaults derivation fails loudly when a capability has multiple models and no declared default
  • Model registry: task layer predictor caches keyed by registered model name (per-capability dict + double-checked locking); /health/ready probes each enabled capability's DEFAULT model; start.sh preflight replaced by scripts/preflight_models.py (enumerates registered models of enabled capabilities, validates YAML at boot; the bash truthy mirror is gone — utils/switches.py is now the single source)
  • Metrics: inferforge_predictor_loaded gains a model label (predict_phase_seconds stays model-unlabeled — engines don't know their registry name; limitation documented in docs/metrics.md)
  • Tooling: scripts gain --model (test_predict*, run_detection/segment/classify, benchmark detect mode); new docs/model-registry.md (format, routing semantics, default derivation, switch relationship, backward compatibility), indexed in docs/README.md
  • Warmup: INFERFORGE_PRELOAD=1 startup preload — web startup event (runs per gunicorn worker, after fork) and celery worker_process_init each load the DEFAULT model of the capabilities they serve (web: detect + switch-gated seg/cls; worker: detect only — seg/cls are sync-only); best-effort per capability (a broken model logs and stays 503 via readiness; readiness stays the source of truth); never preloads non-default registered models

Fixed

  • Metrics: multiprocess file hygiene — utils.metrics.mark_process_dead deletes this process's metric files (*_{pid}.db glob; prometheus_client's own mark_process_dead <=0.26 only covers live gauges) at every graceful exit point: app lifespan shutdown (each worker's runtime files), gunicorn on_exit (the master's import-time files — preload_app imports metrics without ever serving), celery worker_shutdown (main process) + worker_process_shutdown (prefork children), and scripts/preflight_models.py unsets PROMETHEUS_MULTIPROC_DIR before project imports so it creates none; gunicorn's worker_exit hook is deliberately unused (uvicorn workers reset SIGTERM/SIGINT to SIG_DFL — uvicorn #894 — and never reach it); deploy hygiene for crash leftovers documented in docs/metrics.md §3

See CHANGELOG.md for the full change history.

Assets 2
Loading