In this Code Pattern, we'll demonstrate how to monitor a machine learning model using Watson OpenScale. You can using any model deployment serving engine, but we will create one using a Python Flask wrapper with Watson Machine Learning to simulate this.

When the reader has completed this Code Pattern, they will understand how to:

• Create a data mart for a machine learning model with Watson OpenScale.
• Log the payload for the model using Watson OpenScale.
• Configure OpenScale to monitor that deployment for Fairness, Quality, and Drift.

1. User deploys application server using VM or CodeEngine on the IBM Cloud .
2. User creates a Jupyter notebook on Cloud Pak for Data and configures Watson OpenScale.
3. Watson OpenScale is used to monitor a Machine Learning model for payload logging and quality.
4. The application server is used for scoring the deployed model.

• A custom ML framework
• A Machine learning provider backed by a custom ML framework, i.e running in a Code Engine or VM(i.e. classic virtual server) on the IBM Cloud using the provided Flask app.
• An instance of Cloud Pak for Data
• Watson OpenScale provisioned on Cloud Pak for Data.
• IBM Cloud Object Storage (COS)

1. Deploy a model
2. Create custom machine learning provider
3. Create a Watson OpenScale service
4. Create COS bucket and get credentials
5. Create a notebook on Cloud Pak for Data
6. Run the notebook

In this example, we use Watson Machine Learning in IBM public cloud to simulate an existing Custom ML framework. Refer to Train GermanCreditRisk model and promote to a deployment space to deploy the model.

In order to support Custom ML framework for Watson OpenScale, implement two REST APIs to wrap the existing custom ML framework:

• /v1/deployments: Lists all deployments. This optional API can be used to configure the OpenScale insight dashboard to discover models in a deployment space. See Manually configure Machine learning provider.
• /v1/deployments/{deployment_id}/online: Makes an online prediction. The OpenScale uses this API for scoring requests.

Follow one of methods below to setup a custom machine learning provider which serves the above APIs by:

git clone https://github.com/IBM/monitor-custom-ml-engine-with-watson-openscale
cd monitor-custom-ml-engine-with-watson-openscale

python3 -m pip install -r requirements.txt

export APIKEY="<your API key>"

flask --app app.py run --port=5001 --host=0.0.0.0

 * Serving Flask app 'app'
 * Debug mode: on
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on all addresses (0.0.0.0)

curl -XPOST -H "Content-Type: application/json" -d '{"fields":["CheckingStatus","LoanDuration","CreditHistory","LoanPurpose","LoanAmount","ExistingSavings","EmploymentDuration","InstallmentPercent","Sex","OthersOnLoan","CurrentResidenceDuration","OwnsProperty","Age","InstallmentPlans","Housing","ExistingCreditsCount","Job","Dependents","Telephone","ForeignWorker"],"values":[["less_0",11,"prior_payments_delayed","car_used",728,"less_100","4_to_7",2,"male","none",3,"car_other",30,"stores","own",1,"skilled",1,"none","yes"]]}' http://<hostname>:5001/spaces/<space_id>/v1/deployments/<deployment_id>/online

{
  "fields": [
    "prediction",
    "probability"
  ],
  "values": [
    [
      "No Risk",
      [
        0.754556565695645,
        0.24544343430435495
      ]
    ]
  ]
}

or

curl -XPOST -H "Content-Type: application/json" -d '{"fields":["CheckingStatus","LoanDuration","CreditHistory","LoanPurpose","LoanAmount","ExistingSavings","EmploymentDuration","InstallmentPercent","Sex","OthersOnLoan","CurrentResidenceDuration","OwnsProperty","Age","InstallmentPlans","Housing","ExistingCreditsCount","Job","Dependents","Telephone","ForeignWorker"],"values":[["less_0",11,"prior_payments_delayed","car_used",728,"less_100","4_to_7",2,"male","none",3,"car_other",30,"stores","own",1,"skilled",1,"none","yes"]]}' http://application-06.wqfpjpzgx8v.us-east.codeengine.appdomain.cloud/spaces/<space_id>/v1/deployments/<deployment_id>/online

{
  "fields": [
    "prediction",
    "probability"
  ],
  "values": [
    [
      "No Risk",
      [
        0.754556565695645,
        0.24544343430435495
      ]
    ]
  ]
}

• Find or create your IBM Cloud Object Storage service instance and get the Service Credentials for use as COS_API_KEY_ID and COS_RESOURCE_INSTANCE_ID:

  

• In your Cloud Pak for Data, click New Project + under Projects, choose the Create an empty project tile, pick a name for project, select your storage service instance and then Create Project.

• Using the project you've created, click on the New asset + tab. Under Code editors menu option, choose Jupyter notebook editor to create a notebook.

• Select the From URL tab. [1]

• Enter a name for the notebook. [2]

• Optionally, enter a description for the notebook. [3]

• For Runtime select the Runtime 22.2 on Python 3.10 XXS option. [4]

• Under Notebook URL provide the following url: https://raw.githubusercontent.com/IBM/monitor-custom-ml-engine-with-watson-openscale/master/notebooks/WatsonOpenScaleAndCustomMLEngine.ipynb

• Click the Create button. [6]

• Move your cursor to each code cell and run the code in it. Read the comments for each cell to understand what the code is doing. Important when the code in a cell is still running, the label to the left changes to In [*]:. Do not continue to the next cell until the code is finished running.

• Fill the ibmcloud_api_key in cell Provide your IBM Cloud API key

• Fill the CUSTOM_ML_PROVIDER_SCORING_URL in cell REST API to support custom ML provider

• Add the COS credentials in cell Cloud object storage details.

• Either use the internal Database, which requires No Changes or Add your DB_CREDENTIALS and SCHEMA_NAME after reading the instructions in cell Set up Database

This code pattern is licensed under the Apache License, Version 2. Separate third-party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to the Developer Certificate of Origin, Version 1.1 and the Apache License, Version 2.

Apache License FAQ