Please leave a ⭐ if you like this repo. Also please provide your comments if you would like to see something added

Pre requisites for running on PC

1. Windows 10 PC (could run on Mac but I have not tested it!)
2. Docker Desktop up and running with minimum 6 vcores and 6GB RAM
3. Docker Desktop Kubernetes engine up and running

TLDR;

If you want to quickly get the environment up without getting into the details, do the following:

1. Clone this git repo
2. cd to dockerfiles folder and execute docker_build.bat
3. cd to k8s folder and execute start_platform.bat
4. Various services are started at the the following ports
   • MLFlow UI : http://localhost:30005
   • Minio UI : http://localhost:30009
   • Postgres : http://localhost:32345
   • Jupyter notebook : http://localhost:30008
5. Connect to the MLFlow UI @ http://localhost:30005. You can see that experiments are getting logged. These are also getting registered and promoted to Staging. Finally 2 different versions of Regression models are promoted and served in Canary deployment mode.
6. To check the prediction you can use the client under the client folder by installing the requirements.txt and then executing streamlit run client.py. From this client UI you can also disable or enable the canary deployment

Details

1. Clone this get repo
2. Download Kaggle Credit card fraud dataset from https://www.kaggle.com/mlg-ulb/creditcardfraud and put it under ./dataset
3. Install Docker Desktop and enable Kubernetes from settings
4. Install pgAdmin
5. pip install all the required python libraries including mlflow and miniconda
6. docker pull postgres
7. docker pull minio/minio
8. Build the docker images for mlflow server and logreg serving using docker_build.bat.
   • The mlflow server is for the mlflow UI and logreg serving is to serve the model registered as "Staging"
   • Note: If you are running this in pure Docker environment without using Kubernetes you may have to change the IP addresses of the postgres and minio server in the docker files before building them

Docker environment

1. Set the following environment variables:
   AWS_ACCESS_KEY_ID=minioadmin
AWS_SECRET_ACCESS_KEY=minioadmin
MLFLOW_S3_ENDPOINT_URL=http://localhost:9000
MLFLOW_TRACKING_URI=postgresql+psycopg2://postgres:password@localhost:5432
MLFLOW_ARTIFACT_STORE=s3://mlruns
2. docker run -d --rm -p 5432:5432 -e POSTGRES_PASSWORD=password --name postgres postgres
   • Open pgAdmin and connect to the postgres server running in the container (host=localhost, port=5432)
3. docker run -d --rm -p 9000:9000 --name minio minio/minio server /data
   The above will start the containers with ephemeral storage. To start them with persistent volume, do the following:
   • Create a local folder for it to be mounted as persistent volume in the containers and add this as "file sharing" resource under Docker desktop --> Settings --> Resources --> File Sharing
   • use the -v option with docker run command:
     docker run -d --rm -p 9000:9000 -v [local path]:/data--name minio minio/minio server /data
   • Running the postgres container with persistent volume on windows is a bit more involved and needs a docker compose file. The container path that needs to be mounted is /var/lib/postgresql/data
4. docker run -d --rm -p 5000:5000 --name mlflow_server mlflow_server
   
5. Logon to minio UI (http://localhost:9000) and add a bucket named mlruns
6. Logon to MLFlow UI using http://localhost:5000
7. Run the practice_code/mlflow_sk.pynb for sklearn LogisticRegression Model
8. Use the MLFlow UI to register one of the models as "logreg" and promote it to "Staging"
9. docker run -d --rm -p 1235:1235 --name mlflow_logreg_infer mlflow_logreg_infer to start model serving
   
10. Change the port in practice_code/mlflow_serving.ipynb to 1235 and run it. Check the returned inference values

Kubernetes environment

1. Set the following environment variables:
   AWS_ACCESS_KEY_ID=minioadmin
AWS_SECRET_ACCESS_KEY=minioadmin
MLFLOW_S3_ENDPOINT_URL=http://localhost:30009
MLFLOW_TRACKING_URI=postgresql+psycopg2://postgres:password@localhost:32345
MLFLOW_ARTIFACT_STORE=s3://mlruns
2. Run kubectl apply -f .
3. Logon to minio UI (http://localhost:30009) and add a bucket named mlruns
4. Logon to MLFlow UI using http://localhost:30005
5. Open pgAdmin and connect to the postgres server running in kubernetes (host=localhost, port=32345)
6. Run the practice_code/mlflow_sk.pynb for sklearn LogisticRegression Model
7. Use the MLFlow UI to register one of the models as "logreg" and promote it to "Staging"
8. Change the port in practice_code/mlflow_serving.ipynb to 31235 and run it. Check the returned inference values

Local Environment

1. Install sqlite3 if it does not already exist (Normally this comes by default with Python installation)
2. Create a folder for the sqlite db
3. Set the following environment variables:
   MLFLOW_TRACKING_URI=sqlite:///[Path/to/sqlite db file]
MLFLOW_ARTIFACT_STORE=file:/[Path/to/local/folder]
4. Run mlflow ui --backend-store-uri %MLFLOW_TRACKING_URI% OR mlflow server --backend-store-uri %MLFLOW_TRACKING_URI% --default-artifact-root %MLFLOW_ARTIFACT_STORE% to start the MLFlow server
5. Logon to MLFlow UI using http://localhost:5000
6. Run the practice_code/mlflow_sk.pynb for sklearn LogisticRegression Model
7. Use the MLFlow UI to register one of the models as "logreg" and promote it to "Staging"
8. Run mlflow models serve --model-uri models:/logreg/Staging -p 1235 to start serving the model
9. Change the port in practice_code/mlflow_serving.ipynb to 1235 and run it. Check the returned inference values

Further trials

1. Try to serve the Tensorflow model for MNIST dataset in mlflow_tf.ipynb

What I liked about MLFlow

1. Trivial setup
2. Intuitive server UI
3. Minimal code instrumentation for simple tasks like logging params, metrics, models and data
4. Trivial to serve a new version of the model --> Just promote the new version from the UI or using the API
5. Integration with Git

Issues with MLFlow

Model Serving

1. Community edition does not have any user managment and security feature
2. If the developement environment is seperate from the production environment, how to deploy the model artifacts and the metadata from developement to production?

👉The above issues make MLFlow practically limited to internal developement enviornment only. It cannot be used in any real production usecase.

Model Training

1. It is possible to create replicable training pipeline flows using MLFlow Projects but this requires one to learn the MLflow Project DSL and also a good knowledge of MLFlow APIs to write the code. For eg, refer https://github.com/mlflow/mlflow/tree/master/examples/multistep_workflow