Mlflow

[BehRoooz]

Title

Add training API, MLflow registry promotion flow, and Docker/Compose integration

Description

This PR introduces an end-to-end retraining workflow with MLflow registry support and exposes training as an asynchronous FastAPI service. It also updates Docker/Compose so training jobs can run via API, log runs to MLflow, and return run/model links for tracking and promotion.

What Changed

• Added a new training-api service:
  • POST /api/train/train to submit training jobs
  • GET /api/train/jobs/{job_id} to poll job state (queued → running → succeeded/failed)
  • Startup/shutdown worker lifecycle, SQLite-backed job store, structured status/progress/error payloads
  • Added services/training-api/README.md with usage docs
• Added retraining and promotion pipeline scripts:
  • scripts/retrain_pipeline.py to orchestrate train → holdout eval → promotion
  • scripts/promote_model.py to assign registry alias (e.g. champion) based on thresholded holdout metric
• Enhanced training/evaluation MLflow integration:
  • scripts/train.py and scripts/evaluate_holdout.py now log richer params/metrics/artifacts and run summaries
• Updated serving path to use registry-loaded models:
  • Changes in services/go-prediction-api/* to improve model loading from MLflow registry and inference schema/behavior alignment
• Docker/Compose updates:
  • docker/docker_training/Dockerfile.training updated for API runtime
  • docker-compose.yml adds/updates trainer-api and MLflow settings for integrated operation

Why

Move from manual CLI-only training to an API-driven, asynchronous training workflow
Standardize model lifecycle management through MLflow registry
Enable reproducible retraining/promotion and easier operational integration with the existing inference stack