custom_model.md

Bring your own code with the MLOpsPython repository template

This document provides steps to follow when using this repository as a template to train models and deploy the models with real-time inference in Azure ML with your own scripts and data.

1. Follow the MLOpsPython Getting Started guide
2. Bootstrap the project
3. Configure training data
4. [If necessary] Convert your ML experimental code into production ready code
5. Replace the training code
6. [Optional] Update the evaluation code
7. Customize the build agent environment
8. [If appropriate] Replace the score code

Follow the Getting Started guide

Follow the Getting Started guide to set up the infrastructure and pipelines to execute MLOpsPython.

Take a look at the Repo Details document for a description of the structure of this repository.

Bootstrap the project

Bootstrapping will prepare the directory structure to be used for your project name which includes:

• renaming files and folders from the base project name diabetes_regression to your project name
• fixing imports and absolute path based on your project name
• deleting and cleaning up some directories

Note: Since the bootstrap script will rename the diabetes_regression folder to the project name of your choice, we'll refer to your project as [project name] when paths are involved.

To bootstrap from the existing MLOpsPython repository:

1. Ensure Python 3 is installed locally
2. From a local copy of the code, run the bootstrap.py script in the bootstrap folder python bootstrap.py -d [dirpath] -n [projectname]
   • [dirpath] is the absolute path to the root of the directory where MLOpsPython is cloned
   • [projectname] is the name of your ML project

Configure training data

The training ML pipeline uses a sample diabetes dataset as training data.

Important Convert the template to use your own Azure ML Dataset for model training via these steps:

1. Create a Dataset in your Azure ML workspace
2. Update the DATASET_NAME and DATASTORE_NAME variables in .pipelines/[project name]-variables-template.yml

Convert your ML experimental code into production ready code

The MLOpsPython template creates an Azure Machine Learning (ML) pipeline that invokes a set of Azure ML pipeline steps (see ml_service/pipelines/[project name]_build_train_pipeline.py). If your experiment is currently in a Jupyter notebook, it will need to be refactored into scripts that can be run independently and dropped into the template which the existing Azure ML pipeline steps utilize.

1. Refactor your experiment code into scripts
2. [Recommended] Prepare unit tests

Examples of all these scripts are provided in this repository. See the Convert ML experimental code to production code tutorial for a step by step guide and additional details.

Replace training code

The template contains three scripts in the [project name]/training folder. Update these scripts for your experiment code.

• train.py contains the platform-agnostic logic required to do basic data preparation and train the model. This script can be invoked against a static data file for local development.
• train_aml.py is the entry script for the ML pipeline step. It invokes the functions in train.py in an Azure ML context and adds logging. train_aml.py loads parameters for training from [project name]/parameters.json and passes them to the training function in train.py. If your experiment code can be refactored to match the function signatures in train.py, this file shouldn't need many changes.
• test_train.py contains tests that guard against functional regressions in train.py. Remove this file if you have no tests for your own code.

Add any dependencies required by training to [project name]/conda_dependencies.yml]. This file will be used to generate the environment that the pipeline steps will run in.

Update evaluation code

The MLOpsPython template uses the evaluate_model script to compare the performance of the newly trained model and the current production model based on Mean Squared Error. If the performance of the newly trained model is better than the current production model, then the pipelines continue. Otherwise, the pipelines are canceled.

To keep the evaluation step, replace all instances of mse in [project name]/evaluate/evaluate_model.py with the metric that you want.

To disable the evaluation step, either:

• set the DevOps pipeline variable RUN_EVALUATION to false
• uncomment RUN_EVALUATION in .pipelines/[project name]-variables-template.yml and set the value to false

Customize the build agent environment

The DevOps pipeline definitions in the MLOpsPython template run several steps in a Docker container that contains the dependencies required to work through the Getting Started guide. If additional dependencies are required to run your unit tests or generate your Azure ML pipeline, there are a few options:

• Add a pipeline step to install dependencies required by unit tests to .pipelines/code-quality-template.yml. Recommended if you only have a small number of test dependencies.
• Create a new Docker image containing your dependencies. See docs/custom_container.md. Recommended if you have a larger number of dependencies, or if the overhead of installing additional dependencies on each run is too high.
• Remove the container references from the pipeline definition files and run the pipelines on self hosted agents with dependencies pre-installed.

Replace score code

For the model to provide real-time inference capabilities, the score code needs to be replaced. The MLOpsPython template uses the score code to deploy the model to do real-time scoring on ACI, AKS, or Web apps.

If you want to keep scoring:

1. Update or replace [project name]/scoring/score.py
2. Add any dependencies required by scoring to [project name]/conda_dependencies.yml
3. Modify the test cases in the ml_service/util/smoke_test_scoring_service.py script to match the schema of the training features in your data