In this Code Pattern, we'll demonstrate how to monitor a machine learning model using Watson OpenScale. You can using any model deployment serving engine, but we will create one using a Python Flask wrapper with Watson Machine Learning to simulate this.
When the reader has completed this Code Pattern, they will understand how to:
- Create a data mart for a machine learning model with Watson OpenScale.
- Log the payload for the model using Watson OpenScale.
- Configure OpenScale to monitor that deployment for Fairness, Quality, and Drift.
- User deploys application server using VM or CodeEngine on the IBM Cloud .
- User creates a Jupyter notebook on Cloud Pak for Data and configures Watson OpenScale.
- Watson OpenScale is used to monitor a Machine Learning model for payload logging and quality.
- The application server is used for scoring the deployed model.
- A custom ML framework
- A Machine learning provider backed by a custom ML framework, i.e running in a Code Engine or VM(i.e. classic virtual server) on the IBM Cloud using the provided Flask app.
- An instance of Cloud Pak for Data
- Watson OpenScale provisioned on Cloud Pak for Data.
- IBM Cloud Object Storage (COS)
- Deploy a model
- Create custom machine learning provider
- Create a Watson OpenScale service
- Create COS bucket and get credentials
- Create a notebook on Cloud Pak for Data
- Run the notebook
In this example, we use Watson Machine Learning in IBM public cloud to simulate an existing Custom ML framework. Refer to Train GermanCreditRisk model and promote to a deployment space to deploy the model.
Find Space ID, Deployment ID, and Model ID
- In the Cloud Pak for Data instance, click the (☰) menu and click
Deployment. From the table, select the deployment space where the model was deployed.
- Find the
Space ID(GUID)under theManagetab as shown
- Under the
Deploymentstab, select the model which your have deployed earlier
- Find the
Deployment IDandModel IDas shown
In order to support Custom ML framework for Watson OpenScale, implement two REST APIs to wrap the existing custom ML framework:
- /v1/deployments: Lists all deployments. This optional API can be used to configure the OpenScale insight dashboard to discover models in a deployment space. See Manually configure Machine learning provider.
- /v1/deployments/{deployment_id}/online: Makes an online prediction. The OpenScale uses this API for scoring requests.
Follow one of methods below to setup a custom machine learning provider which serves the above APIs by:
Custom Machine Learning Provider Setup for VM
If needed, create a VM on the IBM Cloud. Login the VM in a terminal and clone the monitor-custom-ml-engine-with-watson-openscale repo:
git clone https://github.com/IBM/monitor-custom-ml-engine-with-watson-openscale
cd monitor-custom-ml-engine-with-watson-openscaleThe code in app.py can be used to start a Gunicorn/Flask application that can be hosted in a VM, such that it can be accessible from your OpenScale service instance. This code does the following:
- It wraps a Watson Machine Learning model that is deployed to a space.
- The hosting application URL should contain the SPACE ID and the DEPLOYMENT ID. The app can be used to talk to the target WML model/deployment.
- Having said that, this is only for this tutorial purpose, and you can define your Custom ML provider endpoint in any fashion you want, such that it wraps your own custom ML engine.
- The scoring request and response payload should conform to the schema as described here at: https://aiopenscale-custom-deployement-spec.mybluemix.net/#/
To start the application in the VM using the below code, make sure you install the python packages using:
python3 -m pip install -r requirements.txtBecause the app.py needs to authenticate with the custom ML model deployment, you need to generate an apikey for WML api client to connect. After generating API, run:
export APIKEY="<your API key>"Start the API server:
flask --app app.py run --port=5001 --host=0.0.0.0You should see the app server starting:
* Serving Flask app 'app'
* Debug mode: on
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
* Running on all addresses (0.0.0.0)
Test the app by sending an online scoring request:
curl -XPOST -H "Content-Type: application/json" -d '{"fields":["CheckingStatus","LoanDuration","CreditHistory","LoanPurpose","LoanAmount","ExistingSavings","EmploymentDuration","InstallmentPercent","Sex","OthersOnLoan","CurrentResidenceDuration","OwnsProperty","Age","InstallmentPlans","Housing","ExistingCreditsCount","Job","Dependents","Telephone","ForeignWorker"],"values":[["less_0",11,"prior_payments_delayed","car_used",728,"less_100","4_to_7",2,"male","none",3,"car_other",30,"stores","own",1,"skilled",1,"none","yes"]]}' http://<hostname>:5001/spaces/<space_id>/v1/deployments/<deployment_id>/onlineYou should see similar scoring output:
{
"fields": [
"prediction",
"probability"
],
"values": [
[
"No Risk",
[
0.754556565695645,
0.24544343430435495
]
]
]
}or
Custom Machine Learning Provider Setup for IBM CodeEngine
- In the IBM Cloud, click on the (☰) menu, hover over
Code Engine, click onProjects, and click onCreate +
- Enter a project name and click on
Create:
- In your project overview, select
Secrets and configmapsand click onCreate +
- Select
Secrettile and enter the key and value pair. You can generate an apikey for the APIKEY value.
- Return to your project overview, select
applicationand click onCreate +to create an application
- Select
Source code, enter https://github.com/IBM/monitor-custom-ml-engine-with-watson-openscale in theSource code URL, click on theSpecify build detailand follow the screen instruction.
- In the same page, go down to the
Environment variables (optional), selectReference to key in secret, make sure the createdAPIKEYenv which you have created earlier is present and then click on theAdd +
- Finally click on
Createto create the application.
- Once the application is in
Readystate, copy thepublicURL.
- Test the app by sending an online scoring request:
curl -XPOST -H "Content-Type: application/json" -d '{"fields":["CheckingStatus","LoanDuration","CreditHistory","LoanPurpose","LoanAmount","ExistingSavings","EmploymentDuration","InstallmentPercent","Sex","OthersOnLoan","CurrentResidenceDuration","OwnsProperty","Age","InstallmentPlans","Housing","ExistingCreditsCount","Job","Dependents","Telephone","ForeignWorker"],"values":[["less_0",11,"prior_payments_delayed","car_used",728,"less_100","4_to_7",2,"male","none",3,"car_other",30,"stores","own",1,"skilled",1,"none","yes"]]}' http://application-06.wqfpjpzgx8v.us-east.codeengine.appdomain.cloud/spaces/<space_id>/v1/deployments/<deployment_id>/online- You should see similar scoring output:
{
"fields": [
"prediction",
"probability"
],
"values": [
[
"No Risk",
[
0.754556565695645,
0.24544343430435495
]
]
]
}Setup OpenScale on Cloud Pak for Data if needed
-
In the Cloud Pak for Data instance, go the (☰) menu and under
Servicessection, click on theService instancesmenu option -
If you did not find the
OpenScale-xxxinstance from the instances table, click on theAdd service + -
Clink on the
Watson OpenScale tileto create the service -
Enter a name for the OpenScale service instance and click on the
Create
(Optional) Manually configure Machine learning provider
(Optional) Watson OpenScale can list your deployed models on the Insights dashboard where you can click Add to dashboard, select the deployments that you want to monitor, and click Configure. To utilize this feature, modify your_deployment_id and your_model_id placeholders in the deployments function in the app.py. Find those IDs using the instructions.
You can now manually configure the Machine learning providers through the Insights dashboard using this API endpoint:

-
Find or create your IBM Cloud Object Storage service instance and get the Service Credentials for use as
COS_API_KEY_IDandCOS_RESOURCE_INSTANCE_ID:
-
In your Cloud Pak for Data, click
New Project +under Projects, choose theCreate an empty projecttile, pick a name for project, select yourstorage serviceinstance and thenCreate Project. -
Using the project you've created, click on the
New asset +tab. UnderCode editorsmenu option, chooseJupyter notebook editorto create a notebook.
-
Select the
From URLtab. [1] -
Enter a name for the notebook. [2]
-
Optionally, enter a description for the notebook. [3]
-
For
Runtimeselect theRuntime 22.2 on Python 3.10 XXSoption. [4] -
Under
Notebook URLprovide the following url: https://raw.githubusercontent.com/IBM/monitor-custom-ml-engine-with-watson-openscale/master/notebooks/WatsonOpenScaleAndCustomMLEngine.ipynb -
Click the
Createbutton. [6]
-
Move your cursor to each code cell and run the code in it. Read the comments for each cell to understand what the code is doing. Important when the code in a cell is still running, the label to the left changes to In [*]:. Do not continue to the next cell until the code is finished running.
-
Fill the
ibmcloud_api_keyin cell Provide your IBM Cloud API key -
Fill the
CUSTOM_ML_PROVIDER_SCORING_URLin cell REST API to support custom ML provider -
Add the COS credentials in cell Cloud object storage details.
-
Either use the internal Database, which requires No Changes or Add your
DB_CREDENTIALSandSCHEMA_NAMEafter reading the instructions in cell Set up Database
This code pattern is licensed under the Apache License, Version 2. Separate third-party code objects invoked within this code pattern are licensed by their respective providers pursuant to their own separate licenses. Contributions are subject to the Developer Certificate of Origin, Version 1.1 and the Apache License, Version 2.




















