OpenScale Headless Subscription with Custom Metrics Provider.ipynb

{"cells": [{"metadata": {"id": "130546d4-d72c-46f1-a506-246ace42ad56"}, "cell_type": "markdown", "source": "<img src=\"https://github.com/pmservice/ai-openscale-tutorials/raw/master/notebooks/images/banner.png\" align=\"left\" alt=\"banner\">"}, {"metadata": {"id": "a6d32920-c6f8-4df6-91a7-925640a9920a"}, "cell_type": "markdown", "source": "# Working with a custom metrics provider"}, {"metadata": {"id": "ab7073b7-baf8-48ba-b9fb-e39e497596aa"}, "cell_type": "markdown", "source": "This notebook demonstrates how to configure the custom monitor and custom metrics deployment by using IBM Watson OpenScale. It requires service credentials for the following services:\n  * Watson OpenScale\n  * Watson Machine Learning"}, {"metadata": {"id": "8c3c7fcc-8701-45f3-ab34-af1b093b590c"}, "cell_type": "markdown", "source": "## 1. Set up your environment. <a name=\"setup\"></a>\n\nBefore you use the sample code in this notebook, you must perform the following setup tasks:"}, {"metadata": {"collapsed": true, "id": "8b4212ee-4116-4a47-a586-f0997df1ec53"}, "cell_type": "markdown", "source": "### Install the  `ibm-watson-machine-learning` and `ibm-watson-openscale` packages."}, {"metadata": {"id": "2eb75b61-93b5-4139-96d5-fae98c8c9173", "scrolled": true}, "cell_type": "code", "source": "!pip install --upgrade ibm-watson-machine-learning   | tail -n 1\n!pip install --upgrade ibm-watson-openscale --no-cache | tail -n 1\n!pip install --upgrade wget --no-cache | tail -n 1", "execution_count": null, "outputs": []}, {"metadata": {"id": "cd55a125-14e4-440c-a2ee-c58813d75f9e"}, "cell_type": "markdown", "source": "### Action: restart the kernel!"}, {"metadata": {"id": "2abe81f0-cd47-4fe0-9b5c-505712c17752", "scrolled": true}, "cell_type": "markdown", "source": "### Credentials for IBM Cloud Pak for Data\nTo authenticate, in the following code boxes, replace the sample data with your own credentials. Get the information from your system administrator or through the Cloud Pak for Data dashboard.\n\n\n### Obtaining your Watson OpenScale credentials\n\nYou can retrieve the URL by running the following command: `oc get route -n namespace1 --no-headers | awk '{print $2}'` Replace the `namespace1` variable with your namespace.\n\nYou should have been assigned a username and password when you were added to the Cloud Pak for Data system. You might need to ask either your database administrator or your system administrator for some of the information.\n"}, {"metadata": {"id": "37f4abdea5d34ffd87dc6716b8ff989c"}, "cell_type": "code", "source": "import warnings\nwarnings.filterwarnings('ignore')", "execution_count": null, "outputs": []}, {"metadata": {"id": "b2e0ac27f9bc4f7ba3e7a4c42fecd60b"}, "cell_type": "code", "source": "username = 'admin'\napi_key = 'xxxxxxxxx'\nurl = 'https://cpd-namespace1.apps.xxxxxxxxxxxx.cp.fyre.ibm.com'\npassword = 'password'", "execution_count": null, "outputs": []}, {"metadata": {"id": "7b7a7c6aa06b4a8f89058781d4802afc"}, "cell_type": "markdown", "source": "### Python function details"}, {"metadata": {"id": "cd5ac51a000f49aa936c57db4cc766a0"}, "cell_type": "code", "source": "PYTHON_FUNCTION_NAME = 'Custom Metrics Provider Function'\nDEPLOYMENT_NAME = 'Custom Metrics Provider Deployment'", "execution_count": null, "outputs": []}, {"metadata": {"id": "ab528158384342eb8c3197134c06685c"}, "cell_type": "markdown", "source": "### OpenScale Custom Metrics Provider name"}, {"metadata": {"id": "1756a84762e64b4e8de64142a0e17808"}, "cell_type": "code", "source": "CUSTOM_METRICS_PROVIDER_NAME = \"Custom Metrics Provider\"", "execution_count": null, "outputs": []}, {"metadata": {"id": "5af9a96c2b964af3851128fec40136eb"}, "cell_type": "markdown", "source": "### OpenSale Custom Monitor name"}, {"metadata": {"id": "c510f3021c6540a993b5f54d165497dc"}, "cell_type": "code", "source": "###################################################################\n# UPDATE your custom monitor name in the following field and then run this cell.\n####################################################################\nCUSTOM_MONITOR_NAME = 'Sample Model Performance'", "execution_count": null, "outputs": []}, {"metadata": {"id": "3f499be0-0ed7-46c9-9bf4-753c2529525f"}, "cell_type": "code", "source": "############################################################################################\n# Paste your Watson OpenScale credentials into the following section and then run this cell.\n############################################################################################\nWOS_CREDENTIALS = {\n    \"username\": username,\n    \"apikey\": api_key,\n    \"url\": url\n}", "execution_count": null, "outputs": []}, {"metadata": {"id": "425bf0cc14b943878147dc9676846b0b"}, "cell_type": "code", "source": "WML_CREDENTIALS = WOS_CREDENTIALS.copy()\nWML_CREDENTIALS['instance_id']='openshift'\nWML_CREDENTIALS['version']='4.0'\nWML_CREDENTIALS", "execution_count": null, "outputs": []}, {"metadata": {"id": "61f6830d7dd24c7891ed3f7c66179038"}, "cell_type": "markdown", "source": "# 1. Create the custom metrics provider - Python function.\n\nThe Python function receives the required variables, such as the `datamart_id`, `monitor_id`, `monitor_instance_id`, `monitor_run_id`, `monitor_instance_parameters` and `subscription_id` from the Watson OpenScale service when it is invoked by the custom monitor. \n\nIn the Python function, add your own logic to compute the custom metrics in the `get_metrics` method, publish the metrics to the Watson Openscale service and update the status of the run to the `finished` state in the custom monitor instance run.\n\nUpdate the `WOS_CREDENTIALS` in the Python function. "}, {"metadata": {"id": "be33969270434e84bb4d42c25042878b"}, "cell_type": "code", "source": "#wml_python_function\nparms = {\n        \"url\": WOS_CREDENTIALS[\"url\"],\n        \"username\": WOS_CREDENTIALS[\"username\"],\n        \"apikey\": WOS_CREDENTIALS[\"apikey\"]\n    }\ndef custom_metrics_provider(parms = parms):\n    \n    import json\n    import requests\n    import base64\n    from requests.auth import HTTPBasicAuth\n    import time\n    import uuid\n    import datetime\n    \n    headers = {}\n    headers[\"Content-Type\"] = \"application/json\"\n    headers[\"Accept\"] = \"application/json\"\n    \n    \n    # Get the access token\n    def get_access_token():\n        url = '{}/icp4d-api/v1/authorize'.format(parms['url'])\n        payload = {\n            'username': parms['username'],\n            'api_key': parms['apikey']\n        }\n        response = requests.post(url, headers=headers, json=payload, verify=False)\n        json_data = response.json()\n        access_token = json_data['token']\n        return access_token\n    \n    def get_feedback_dataset_id(access_token, data_mart_id, subscription_id):\n        headers[\"Authorization\"] = \"Bearer {}\".format(access_token)\n        DATASETS_URL =  parms[\"url\"] + \"/openscale/{0}/v2/data_sets?target.target_id={1}&target.target_type=subscription&type=feedback\".format(data_mart_id, subscription_id)\n        response = requests.get(DATASETS_URL, headers=headers, verify=False)\n        json_data = response.json()\n        feedback_dataset_id = None\n        if \"data_sets\" in json_data and len(json_data[\"data_sets\"]) > 0:\n            feedback_dataset_id = json_data[\"data_sets\"][0][\"metadata\"][\"id\"]\n        \n        return feedback_dataset_id\n    \n    def get_feedback_data(access_token, data_mart_id, feedback_dataset_id):\n        json_data = None\n        if feedback_dataset_id is not None:\n            headers[\"Authorization\"] = \"Bearer {}\".format(access_token)\n            DATASETS_STORE_RECORDS_URL = parms[\"url\"] + \"/openscale/{0}/v2/data_sets/{1}/records?limit={2}&format=list\".format(data_mart_id, feedback_dataset_id, 100)\n            response = requests.get(DATASETS_STORE_RECORDS_URL, headers=headers, verify=False)\n            json_data = response.json()\n            return json_data\n    \n    #Update the run status to Finished in the Monitor Run\n    def update_monitor_run_status(base_url, access_token, custom_monitor_instance_id, run_id, status, error_msg = None):\n        monitor_run_url = base_url + '/v2/monitor_instances/' + custom_monitor_instance_id + '/runs/'+run_id\n        completed_timestamp = datetime.datetime.utcnow().strftime(\"%Y-%m-%dT%H:%M:%S.%fZ\")\n        patch_payload  = []\n        base_path = \"/status\"\n        \n        patch_payload.append(get_patch_request_field(base_path, \"state\", status))\n        patch_payload.append(get_patch_request_field(base_path, \"completed_at\", completed_timestamp))\n        if error_msg != None:\n            error_json = get_error_json(error_msg)\n            patch_payload.append(get_patch_request_field(base_path, \"failure\", error_json))\n        \n        headers[\"Authorization\"] = \"Bearer {}\".format(access_token)\n        response = requests.patch(monitor_run_url, headers=headers, json = patch_payload, verify=False)\n        monitor_run_response = response.json()\n        return response.status_code, monitor_run_response\n    \n    \n    def get_error_json(error_message):\n        trace = str(uuid.uuid4())\n        error_json = {\n            'trace': trace,\n            'errors': [{\n                'code': \"custom_metrics_error_code\",\n                'message': str(error_message)\n            }]\n        }\n        return error_json\n    \n    def get_patch_request_field(base_path, field_name, field_value, op_name=\"replace\"):\n        field_json = {\n            \"op\": op_name,\n            \"path\": \"{0}/{1}\".format(base_path, field_name),\n            \"value\": field_value\n        }\n        return field_json\n    \n    #Add your code to compute the custom metrics. \n    def get_metrics(access_token, data_mart_id, subscription_id):\n        #Add the logic here to compute the metrics. Use the below metric names while creating the custom monitor definition\n        feedback_dataset_id = get_feedback_dataset_id(access_token, data_mart_id, subscription_id)\n        json_data = get_feedback_data(access_token, data_mart_id, feedback_dataset_id)\n        gender_less40_fav_prediction_ratio = 0\n        if json_data is not None:\n            fields = json_data['records'][0]['fields']\n            values = json_data['records'][0]['values']\n            import pandas as pd\n            feedback_data = pd.DataFrame(values, columns = fields)\n            female_less40_fav_prediction = len(feedback_data.query('Sex == \\'female\\' & Age <= 40 & Risk == \\'No Risk\\''))\n            male_less40_fav_prediction = len(feedback_data.query('Sex == \\'male\\' & Age <= 40 & Risk == \\'No Risk\\''))\n            gender_less40_fav_prediction_ratio = female_less40_fav_prediction / male_less40_fav_prediction\n        \n        #Remove the tag(\"region\": \"us-south\") in below metrics while publishing the metric values to Openscale Datamart \n        #if the custom monitor definition is not created with tags\n        metrics = {\"specificity\": 1.2, \"sensitivity\": 0.85, \"gender_less40_fav_prediction_ratio\": gender_less40_fav_prediction_ratio, \"region\": \"us-south\"}\n        #metrics = {\"specificity\": 1.2, \"sensitivity\": 0.85, \"gender_less40_fav_prediction_ratio\": gender_less40_fav_prediction_ratio}\n        \n        return metrics\n        \n        \n    # Publishes the Custom Metrics to OpenScale\n    def publish_metrics(base_url, access_token, data_mart_id, subscription_id, custom_monitor_id, custom_monitor_instance_id, custom_monitoring_run_id, timestamp):\n        # Generate an monitoring run id, where the publishing happens against this run id\n        custom_metrics = get_metrics(access_token, data_mart_id, subscription_id)\n        measurements_payload = [\n                  {\n                    \"timestamp\": timestamp,\n                    \"run_id\": custom_monitoring_run_id,\n                    \"metrics\": [custom_metrics]\n                  }\n                ]\n        headers[\"Authorization\"] = \"Bearer {}\".format(access_token)\n        measurements_url = base_url + '/v2/monitor_instances/' + custom_monitor_instance_id + '/measurements'\n        response = requests.post(measurements_url, headers=headers, json = measurements_payload, verify=False)\n        published_measurement = response.json()\n     \n        return response.status_code, published_measurement\n        \n    \n    def publish( input_data ):\n        \n        timestamp = datetime.datetime.utcnow().strftime(\"%Y-%m-%dT%H:%M:%S.%fZ\")\n        \n        payload = input_data.get(\"input_data\")[0].get(\"values\")\n        data_mart_id = payload['data_mart_id']\n        subscription_id = payload['subscription_id']\n        custom_monitor_id = payload['custom_monitor_id']\n        custom_monitor_instance_id = payload['custom_monitor_instance_id']\n        custom_monitor_instance_params  = payload['custom_monitor_instance_params']\n        custom_monitor_run_id = payload['custom_monitor_run_id']\n\n        base_url = parms['url'] + '/openscale' + '/' + data_mart_id\n        access_token = get_access_token()\n        \n        published_measurements = []\n        error_msgs = []\n        run_status = \"finished\"\n        \n        try:\n            last_run_time = custom_monitor_instance_params.get(\"last_run_time\")\n            max_records = custom_monitor_instance_params.get(\"max_records\")\n            min_records = custom_monitor_instance_params.get(\"min_records\")\n            error_msg = None\n            \n            status_code, published_measurement = publish_metrics(base_url, access_token, data_mart_id, subscription_id, custom_monitor_id, custom_monitor_instance_id, custom_monitor_run_id, timestamp)\n            if int(status_code) in [200, 201, 202]:\n                published_measurements.append(published_measurement)\n            else:\n                run_status = \"error\"\n                error_msg = published_measurement\n                error_msgs.append(error_msg)\n                    \n            status_code, response = update_monitor_run_status(base_url, access_token, custom_monitor_instance_id, custom_monitor_run_id, run_status, error_msg)\n            \n            if not int(status_code) in [200, 201, 202]:\n                error_msgs.append(response)\n                \n        except Exception as ex:\n            error_msgs.append(str(ex))\n        if len(error_msgs) == 0:\n            response_payload = {\n                \"predictions\" : [{ \n                    \"values\" : published_measurements\n                }]\n\n            }\n        else:\n            response_payload = {\n                \"predictions\":[],\n                \"errors\": error_msgs\n            }\n        \n        return response_payload\n        \n    return publish\n    ", "execution_count": null, "outputs": []}, {"metadata": {"id": "83073319d9f044ca846bcdc4e3515fbc"}, "cell_type": "markdown", "source": "## Register the custom metrics provider and create a deployment. <a name=\"deployment\"></a>"}, {"metadata": {"id": "7c5fdcdd9d5b437884c4b8c8cbc4f788"}, "cell_type": "code", "source": "import json\nfrom ibm_watson_machine_learning import APIClient\n\nwml_client = APIClient(WML_CREDENTIALS)\nwml_client.version", "execution_count": null, "outputs": []}, {"metadata": {"id": "7aa39e6c48c3439392e18fd55d8b389d"}, "cell_type": "code", "source": "wml_client.spaces.list(limit=10)", "execution_count": null, "outputs": []}, {"metadata": {"id": "90ddd0f1be664d768ab45e69d5e4c3a3"}, "cell_type": "code", "source": "space_id = 'eff8e06f-2572-445a-8d77-ec0dc8e18c8b'\nwml_client.set.default_space(space_id)", "execution_count": null, "outputs": []}, {"metadata": {"id": "14987959013b4cb1908389ee9dc5a723"}, "cell_type": "markdown", "source": "### Remove existing function and deployment."}, {"metadata": {"id": "860810481977469f87da3b5fe2351459"}, "cell_type": "code", "source": "deployments_list = wml_client.deployments.get_details()\nfor deployment in deployments_list[\"resources\"]:\n    model_id = deployment[\"entity\"][\"asset\"][\"id\"]\n    deployment_id = deployment[\"metadata\"][\"id\"]\n    if deployment[\"metadata\"][\"name\"] == DEPLOYMENT_NAME:\n        print(\"Deleting deployment id\", deployment_id)\n        wml_client.deployments.delete(deployment_id)\n        print(\"Deleting model id\", model_id)\n        wml_client.repository.delete(model_id)\n\nwml_client.repository.list_functions()", "execution_count": null, "outputs": []}, {"metadata": {"id": "bedb0b25c5b8409884e7dc8b7a73c6f3"}, "cell_type": "markdown", "source": "### Create the function meta properties.\n"}, {"metadata": {"id": "6b08dee3d2414540a4d53b95ec542914"}, "cell_type": "code", "source": "software_spec_id =  wml_client.software_specifications.get_id_by_name('runtime-22.1-py3.9')\nprint(software_spec_id)\nfunction_meta_props = {\n     wml_client.repository.FunctionMetaNames.NAME: PYTHON_FUNCTION_NAME,\n     wml_client.repository.FunctionMetaNames.SOFTWARE_SPEC_ID: software_spec_id\n     }", "execution_count": null, "outputs": []}, {"metadata": {"id": "453f1df183bc45f7b1ed280c23c7da1e"}, "cell_type": "markdown", "source": "### Store the Python function."}, {"metadata": {"id": "81a5a2a8e8d64de495c9e084d84a5296"}, "cell_type": "code", "source": "function_artifact = wml_client.repository.store_function(meta_props=function_meta_props, function=custom_metrics_provider)\nfunction_uid = wml_client.repository.get_function_id(function_artifact)\nprint(\"Function UID = \" + function_uid)", "execution_count": null, "outputs": []}, {"metadata": {"id": "4468fd43983a4bec90b390b9a888e39b"}, "cell_type": "code", "source": "function_details = wml_client.repository.get_details(function_uid)\nfrom pprint import pprint\npprint(function_details)", "execution_count": null, "outputs": []}, {"metadata": {"id": "94974bda97ed4277b9261eef044deeea"}, "cell_type": "markdown", "source": "### Deploy the Python function.\n"}, {"metadata": {"id": "cf90daf670f7466e81fe291f04e87f87"}, "cell_type": "code", "source": "hardware_spec_id = wml_client.hardware_specifications.get_id_by_name('M')\nhardware_spec_id", "execution_count": null, "outputs": []}, {"metadata": {"id": "20a27821a41c45b5b5e625a677908a59"}, "cell_type": "markdown", "source": "### Create deployment metadata for the Python function."}, {"metadata": {"id": "63f1d36a23d941c9a919da5405bbc94f"}, "cell_type": "code", "source": "deploy_meta = {\n wml_client.deployments.ConfigurationMetaNames.NAME: DEPLOYMENT_NAME,\n wml_client.deployments.ConfigurationMetaNames.ONLINE: {},\n wml_client.deployments.ConfigurationMetaNames.HARDWARE_SPEC: { \"id\": hardware_spec_id}\n}", "execution_count": null, "outputs": []}, {"metadata": {"id": "f3f6519a6aee4d4da077bdd93a0a02f3"}, "cell_type": "markdown", "source": "### Create a deployment."}, {"metadata": {"id": "1719d181882948f286a57b5893049f6c"}, "cell_type": "code", "source": "deployment_details = wml_client.deployments.create(function_uid, meta_props=deploy_meta)", "execution_count": null, "outputs": []}, {"metadata": {"id": "ce9ba098d01a40a08515a49ea936e6a7"}, "cell_type": "markdown", "source": "### Get the python function scoring URL."}, {"metadata": {"id": "4f7fe70dcdfc4ab493f45a94a4571ff7"}, "cell_type": "code", "source": "created_at = deployment_details['metadata']['created_at']\nfind_string_pos = created_at.find(\"T\")\nif find_string_pos != -1:\n    current_date = created_at[0:find_string_pos]\nscoring_url = wml_client.deployments.get_scoring_href(deployment_details)\nscoring_url = scoring_url + \"?version=\"+current_date\nprint(scoring_url)\n", "execution_count": null, "outputs": []}, {"metadata": {"id": "289837a230c54fb89ab93d9474f582ec"}, "cell_type": "markdown", "source": "# 2. Generate training data schema for configuring OpenScale"}, {"metadata": {"id": "77b157faee7746c79c50ed10cfb646a5"}, "cell_type": "markdown", "source": "### Describe the training data"}, {"metadata": {"id": "64255888c34344c08e497edc7706e40a"}, "cell_type": "code", "source": "feature_columns=[\"CheckingStatus\",\"LoanDuration\",\"CreditHistory\",\"LoanPurpose\",\"LoanAmount\",\"ExistingSavings\",\"EmploymentDuration\",\"InstallmentPercent\",\"Sex\",\"OthersOnLoan\",\"CurrentResidenceDuration\",\"OwnsProperty\",\"Age\",\"InstallmentPlans\",\"Housing\",\"ExistingCreditsCount\",\"Job\",\"Dependents\",\"Telephone\",\"ForeignWorker\"]\ncat_features=[\"CheckingStatus\",\"CreditHistory\",\"LoanPurpose\",\"ExistingSavings\",\"EmploymentDuration\",\"Sex\",\"OthersOnLoan\",\"OwnsProperty\",\"InstallmentPlans\",\"Housing\",\"Job\",\"Telephone\",\"ForeignWorker\"]\nclass_label = \"Risk\"", "execution_count": null, "outputs": []}, {"metadata": {"id": "9bf766cfdb32479a8357f28d4f0dfa80"}, "cell_type": "markdown", "source": "### Get the training data"}, {"metadata": {"id": "76cf8a64fab34e55a648bc11798839c5"}, "cell_type": "code", "source": "!rm german_credit_data_biased_training.csv\n!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/german_credit_data_biased_training.csv", "execution_count": null, "outputs": []}, {"metadata": {"id": "5c2087e2c31a42708c04d721c5552634"}, "cell_type": "code", "source": "import pandas as pd\ndata_df=pd.read_csv (\"german_credit_data_biased_training.csv\")\ndata_df.head()", "execution_count": null, "outputs": []}, {"metadata": {"id": "1394bf8c0ef4479e8e117827c29e183a"}, "cell_type": "markdown", "source": "### Generate training data schema"}, {"metadata": {"id": "cba362c0b378436d8856204552150568"}, "cell_type": "code", "source": "from ibm_watson_openscale.utils.training_stats import TrainingStats", "execution_count": null, "outputs": []}, {"metadata": {"id": "9c9f8e9b8fa8439a908d368cbf6d340b"}, "cell_type": "code", "source": "input_parameters = {\n    \"label_column\": class_label,\n    \"feature_columns\": feature_columns,\n    \"categorical_columns\": cat_features,\n    \"fairness_inputs\": None,  \n    \"problem_type\" : \"binary\"\n}", "execution_count": null, "outputs": []}, {"metadata": {"id": "a7111947a2f4400380ff0663eb7ed6ca"}, "cell_type": "code", "source": "training_stats = TrainingStats(data_df,input_parameters, explain=False, fairness=False, drop_na=True)", "execution_count": null, "outputs": []}, {"metadata": {"id": "476b7fc4c21249009ab1f8cfb4f29160"}, "cell_type": "code", "source": "config_json = training_stats.get_training_statistics()", "execution_count": null, "outputs": []}, {"metadata": {"id": "d6b7bd0522fe453daa7421825053bbc5"}, "cell_type": "code", "source": "config_json[\"notebook_version\"] = 5.0", "execution_count": null, "outputs": []}, {"metadata": {"id": "dca91cb83a384bd08e22d2cf8ab2e0b1"}, "cell_type": "code", "source": "config_json", "execution_count": null, "outputs": []}, {"metadata": {"id": "c1e8aca2d07345cf97e6cde4a920a34d"}, "cell_type": "markdown", "source": "# 3. Configure OpenScale \n\nThe notebook will now import the necessary libraries and set up a Python OpenScale client."}, {"metadata": {"id": "62aa6bb5cb294ebf9558fafa98a2250b"}, "cell_type": "code", "source": "from ibm_watson_openscale import APIClient\nfrom ibm_watson_openscale.utils import *\nfrom ibm_watson_openscale.supporting_classes import *\nfrom ibm_watson_openscale.supporting_classes.enums import *\nfrom ibm_watson_openscale.base_classes.watson_open_scale_v2 import *\n\nimport json\nimport requests\nimport base64\nfrom requests.auth import HTTPBasicAuth\nimport time", "execution_count": null, "outputs": []}, {"metadata": {"id": "408e6b721ff340588e1b7704c7cb3afa"}, "cell_type": "code", "source": "WOS_GUID = '00000000-0000-0000-0000-1661432974606959'", "execution_count": null, "outputs": []}, {"metadata": {"id": "20d9037ade2648ca911f791137c302e0"}, "cell_type": "markdown", "source": "## Get a instance of the OpenScale SDK client"}, {"metadata": {"id": "6d4fd050ea3a45d98327f83ac05467db"}, "cell_type": "code", "source": "authenticator = CloudPakForDataAuthenticator(\n        url=WOS_CREDENTIALS['url'],\n        username=WOS_CREDENTIALS['username'],\n        apikey=WOS_CREDENTIALS['apikey'],\n        disable_ssl_verification=True,\n    )\n\nwos_client = APIClient(service_url=WOS_CREDENTIALS['url'],authenticator=authenticator, service_instance_id = WOS_GUID)\nwos_client.version", "execution_count": null, "outputs": []}, {"metadata": {"id": "160469ff1b5a42a4970f8c566d06ef26"}, "cell_type": "markdown", "source": "## Set up datamart"}, {"metadata": {"id": "3115299efc0c48d59134ee6589ab9cc3"}, "cell_type": "code", "source": "wos_client.data_marts.show()", "execution_count": null, "outputs": []}, {"metadata": {"id": "8ae188d7cecf4e8981ee989132c163a7"}, "cell_type": "code", "source": "data_marts = wos_client.data_marts.list().result.data_marts\ndata_mart_id=data_marts[0].metadata.id\nprint('Using existing datamart {}'.format(data_mart_id))", "execution_count": null, "outputs": []}, {"metadata": {"id": "53131d5bd0184a73a1b295d12bffb000"}, "cell_type": "code", "source": "data_mart_details = wos_client.data_marts.list().result.data_marts[0]\ndata_mart_details.to_dict()", "execution_count": null, "outputs": []}, {"metadata": {"id": "61eeafc95b2d4a889c34d85cadec220e"}, "cell_type": "code", "source": "wos_client.service_providers.show()", "execution_count": null, "outputs": []}, {"metadata": {"id": "a32829a411a742a9801fbcfd89680576"}, "cell_type": "markdown", "source": "## Remove existing service provider\n\nMultiple service providers for the same engine instance are avaiable in Watson OpenScale. To avoid multiple service providers of used WML instance in the tutorial notebook the following code deletes existing service provder(s) and then adds new one."}, {"metadata": {"id": "103f1592e95546f3bcd9b2ef4532ecc8"}, "cell_type": "code", "source": "SERVICE_PROVIDER_NAME = \"RC - OpenScale Headless Service Provider\"\nSERVICE_PROVIDER_DESCRIPTION = \"Added by tutorial WOS notebook to showcase Headless Subscription functionality.\"", "execution_count": null, "outputs": []}, {"metadata": {"id": "bdf98e505c67491b8e10eff8aacdc327"}, "cell_type": "code", "source": "service_providers = wos_client.service_providers.list().result.service_providers\nfor service_provider in service_providers:\n    service_instance_name = service_provider.entity.name\n    if service_instance_name == SERVICE_PROVIDER_NAME:\n        service_provider_id = service_provider.metadata.id\n        wos_client.service_providers.delete(service_provider_id)\n        print(\"Deleted existing service_provider for WML instance: {}\".format(service_provider_id))", "execution_count": null, "outputs": []}, {"metadata": {"id": "073e9e48dbca40299c77788358505b43"}, "cell_type": "markdown", "source": "## Add service provider\n\nWatson OpenScale needs to be bound to the Watson Machine Learning instance to capture payload data into and out of the model.\n\nNote: Here the service provider is created with empty credentials, meaning no endpoint. Just to demonstrate the use case were we don't need an actual end point serving requests."}, {"metadata": {"id": "2422ded6d4a4491e8e983116e938016d"}, "cell_type": "code", "source": "MLCredentials = {}\nadded_service_provider_result = wos_client.service_providers.add(\n        name=SERVICE_PROVIDER_NAME,\n        description=SERVICE_PROVIDER_DESCRIPTION,\n        service_type=ServiceTypes.CUSTOM_MACHINE_LEARNING,\n        operational_space_id = \"production\",\n        credentials=MLCredentials,\n        background_mode=False\n    ).result\nservice_provider_id = added_service_provider_result.metadata.id", "execution_count": null, "outputs": []}, {"metadata": {"id": "35624aa5bc01415a82057e40b00c5a8f"}, "cell_type": "code", "source": "print(wos_client.service_providers.get(service_provider_id).result)", "execution_count": null, "outputs": []}, {"metadata": {"id": "44146084ca5d4eab862c2a40f4058ae7"}, "cell_type": "markdown", "source": "## Subscriptions"}, {"metadata": {"id": "9c78bee42b4d46618b5e5426673e58ad"}, "cell_type": "markdown", "source": "Remove existing credit risk subscriptions\n\nThis code removes previous subscriptions to the model to refresh the monitors with the new model and new data."}, {"metadata": {"id": "8fe06931ec7d43de8ad15aedf6e3dbfe"}, "cell_type": "code", "source": "wos_client.subscriptions.show()", "execution_count": null, "outputs": []}, {"metadata": {"id": "05ee3886f68a4f20812fda72900f077e"}, "cell_type": "markdown", "source": "## Remove the existing subscription"}, {"metadata": {"id": "ad6b9b83982a400c8dbebe3431815f0f"}, "cell_type": "code", "source": "SUBSCRIPTION_NAME = \"RC - GCR Headless Subscription\"", "execution_count": null, "outputs": []}, {"metadata": {"id": "359724c3182b4f268a0c72d682d0f713"}, "cell_type": "code", "source": "subscriptions = wos_client.subscriptions.list().result.subscriptions\nfor subscription in subscriptions:\n    if subscription.entity.asset.name == '[asset] ' + SUBSCRIPTION_NAME:\n        sub_model_id = subscription.metadata.id\n        wos_client.subscriptions.delete(subscription.metadata.id)\n        print('Deleted existing subscription for model', sub_model_id)", "execution_count": null, "outputs": []}, {"metadata": {"id": "f5758c74bfbf4c0187082ff1e8f02acb"}, "cell_type": "code", "source": "print(\"Data Mart ID: \" + data_mart_id)\nprint(\"Service Provide ID: \" + service_provider_id)\nimport uuid\nasset_id = str(uuid.uuid4())\nasset_name = '[asset] ' + SUBSCRIPTION_NAME\nurl = None\n\nasset_deployment_id = str(uuid.uuid4())\nasset_deployment_name = asset_name", "execution_count": null, "outputs": []}, {"metadata": {"id": "1ab0c35da1ca49bd8efaaeb911359f6f"}, "cell_type": "code", "source": "prediction_column = \"prediction\"\nprobability_columns = ['probability']\npredicted_target_column = \"prediction\"\nsubscription_details = wos_client.subscriptions.add(data_mart_id,\n    service_provider_id,\n    asset=Asset(\n        asset_id=asset_id,\n        name=asset_name,\n        url=url,\n        asset_type=AssetTypes.MODEL,\n        input_data_type=InputDataType.STRUCTURED,\n        problem_type=ProblemType.BINARY_CLASSIFICATION\n    ),\n    deployment=AssetDeploymentRequest(\n        deployment_id=asset_deployment_id,\n        name=asset_deployment_name,\n        deployment_type= DeploymentTypes.ONLINE\n    ),\n    training_data_stats=config_json,\n    prediction_field = prediction_column,\n    predicted_target_field = predicted_target_column,\n    probability_fields = probability_columns,\n    background_mode = False,\n    deployment_name = asset_name\n    ).result\n\nsubscription_id = subscription_details.metadata.id\nprint(\"Subscription id {}\".format(subscription_id))", "execution_count": null, "outputs": []}, {"metadata": {"id": "315cd0a619ce42ee8ca94f5fa874f3a3"}, "cell_type": "code", "source": "import time\n\ntime.sleep(5)\npayload_data_set_id = None\npayload_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, \n                                                target_target_id=subscription_id, \n                                                target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id\nif payload_data_set_id is None:\n    print(\"Payload data set not found. Please check subscription status.\")\nelse:\n    print(\"Payload data set id:\", payload_data_set_id)", "execution_count": null, "outputs": []}, {"metadata": {"id": "844fdc2077e74bf39f034b80dfec34a9"}, "cell_type": "code", "source": "wos_client.subscriptions.get(subscription_id).result.to_dict()", "execution_count": null, "outputs": []}, {"metadata": {"id": "cbca79f653d247a7859a779c23055cff"}, "cell_type": "markdown", "source": "## Push a payload record to setup the required schemas in the subscription\n\nNote : No scoring is done against the model. The PayloadRecord is constructed with the request and response from the model/deployment."}, {"metadata": {"id": "829954547670499a824ad668e40519a8"}, "cell_type": "markdown", "source": "## Scoring Request Payload"}, {"metadata": {"id": "4f2e4964eacb4d469e840d8e5048b15c"}, "cell_type": "code", "source": "scoring_request =   {\n        \"fields\": [\n            \"CheckingStatus\",\n            \"LoanDuration\",\n            \"CreditHistory\",\n            \"LoanPurpose\",\n            \"LoanAmount\",\n            \"ExistingSavings\",\n            \"EmploymentDuration\",\n            \"InstallmentPercent\",\n            \"Sex\",\n            \"OthersOnLoan\",\n            \"CurrentResidenceDuration\",\n            \"OwnsProperty\",\n            \"Age\",\n            \"InstallmentPlans\",\n            \"Housing\",\n            \"ExistingCreditsCount\",\n            \"Job\",\n            \"Dependents\",\n            \"Telephone\",\n            \"ForeignWorker\",\n            \"Risk\"\n        ],\n        \"values\": [\n            [\n                \"no_checking\",\n                28,\n                \"outstanding_credit\",\n                \"appliances\",\n                5990,\n                \"500_to_1000\",\n                \"greater_7\",\n                5,\n                \"male\",\n                \"co-applicant\",\n                3,\n                \"car_other\",\n                55,\n                \"none\",\n                \"free\",\n                2,\n                \"skilled\",\n                2,\n                \"yes\",\n                \"yes\",\n                \"Risk\"\n            ],\n            [\n                \"greater_200\",\n                22,\n                \"all_credits_paid_back\",\n                \"car_used\",\n                3376,\n                \"less_100\",\n                \"less_1\",\n                3,\n                \"female\",\n                \"none\",\n                2,\n                \"car_other\",\n                32,\n                \"none\",\n                \"own\",\n                1,\n                \"skilled\",\n                1,\n                \"none\",\n                \"yes\",\n                \"No Risk\"\n            ]\n        ],\n        \"meta\": {\n            \"fields\": [\n                \"referrer_gender\"\n            ],\n            \"values\": [\n                [\n                    \"male\"\n                ],\n                [\n                    \"female\"\n                ]\n            ]\n        }\n    }", "execution_count": null, "outputs": []}, {"metadata": {"id": "dc204450969e41a1a1cec7a534fe309d"}, "cell_type": "markdown", "source": "## Scoring Response Payload"}, {"metadata": {"id": "03ee8c7b6ab04a608ed8c04509037a1b"}, "cell_type": "code", "source": "scoring_response = {\n    \"predictions\": [\n        {\n            \"fields\": [\n                \"prediction\",\n                \"probability\"\n            ],\n            \"values\": [\n                [\n                    \"Risk\",\n                    [\n                        0.104642951112211,\n                        0.895357048887789\n                    ]\n                ],\n                [\n                    \"No Risk\",\n                    [\n                        0.892112895920181,\n                        0.10788710407981907\n                    ]\n                ]\n            ]\n        }\n    ]\n}", "execution_count": null, "outputs": []}, {"metadata": {"id": "1ee6a1c323404fbb850f8e44132e8dff"}, "cell_type": "code", "source": "from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord\n\nrecords_list=[]\nfor x in range(10):\n    pl_record = PayloadRecord(request=scoring_request, response=scoring_response)\n    records_list.append(pl_record)\n\nwos_client.data_sets.store_records(data_set_id=payload_data_set_id, request_body=records_list)", "execution_count": null, "outputs": []}, {"metadata": {"id": "6c9a39df6b444a488da8382935393ca8"}, "cell_type": "code", "source": "time.sleep(5)\npl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)\nprint(\"Number of records in the payload logging table: {}\".format(pl_records_count))\nif pl_records_count == 0:\n    raise Exception(\"Payload logging did not happen!\")", "execution_count": null, "outputs": []}, {"metadata": {"id": "1d2a7a6249b043ba899f46346d51d3ce"}, "cell_type": "markdown", "source": "## Fetch the subscription details to confirm schemas are setup"}, {"metadata": {"id": "76912212b98447889a30bfec03092230"}, "cell_type": "code", "source": "wos_client.subscriptions.get(subscription_id).result.to_dict()", "execution_count": null, "outputs": []}, {"metadata": {"id": "1f586af362c84a4a853fa07686f39f5f"}, "cell_type": "markdown", "source": "## Enable feedback logging"}, {"metadata": {"id": "5318bd2b2dd44a578b9c0e2799f7446c"}, "cell_type": "code", "source": "wos_client.subscriptions.create_feedback_table(subscription_id)", "execution_count": null, "outputs": []}, {"metadata": {"id": "b88c0cc07c414ade86357afeedbd31e3"}, "cell_type": "markdown", "source": "## Get feedback logging dataset ID"}, {"metadata": {"id": "0bb5915a69044546abbd2c6fe311629e"}, "cell_type": "code", "source": "feedback_dataset_id = None\nfeedback_dataset = wos_client.data_sets.list(type=DataSetTypes.FEEDBACK, \n                                                target_target_id=subscription_id, \n                                                target_target_type=TargetTypes.SUBSCRIPTION).result\nfeedback_dataset_id = feedback_dataset.data_sets[0].metadata.id\nif feedback_dataset_id is None:\n    print(\"Feedback data set not found. Please check quality monitor status.\")", "execution_count": null, "outputs": []}, {"metadata": {"id": "ae6f3d5f90de477b8047a2ebb1bcffec"}, "cell_type": "code", "source": "feedback_dataset_id", "execution_count": null, "outputs": []}, {"metadata": {"id": "f4d4033892de4b4891d13d1f4e26bee6"}, "cell_type": "code", "source": "feedback_payload = {\n    \"fields\": [\n        \"CheckingStatus\",\n        \"LoanDuration\",\n        \"CreditHistory\",\n        \"LoanPurpose\",\n        \"LoanAmount\",\n        \"ExistingSavings\",\n        \"EmploymentDuration\",\n        \"InstallmentPercent\",\n        \"Sex\",\n        \"OthersOnLoan\",\n        \"CurrentResidenceDuration\",\n        \"OwnsProperty\",\n        \"Age\",\n        \"InstallmentPlans\",\n        \"Housing\",\n        \"ExistingCreditsCount\",\n        \"Job\",\n        \"Dependents\",\n        \"Telephone\",\n        \"ForeignWorker\",\n        \"Risk\",\n        \"_original_probability\",\n        \"_original_prediction\",\n        \"_debiased_probability\",\n        \"_debiased_prediction\"\n    ],\n    \"values\": [\n        [\n            \"less_0\",\n            18,\n            \"credits_paid_to_date\",\n            \"car_new\",\n            462,\n            \"less_100\",\n            \"1_to_4\",\n            2,\n            \"female\",\n            \"none\",\n            2,\n            \"savings_insurance\",\n            37,\n            \"stores\",\n            \"own\",\n            2,\n            \"skilled\",\n            1,\n            \"none\",\n            \"yes\",\n            \"No Risk\",\n            [\n                0.767955712021837,\n                0.23204428797816307\n            ],\n            \"Risk\",\n            [\n                0.767955712021837,\n                0.23204428797816307\n            ],\n            \"Risk\"\n        ],\n        [\n            \"less_0\",\n            15,\n            \"prior_payments_delayed\",\n            \"furniture\",\n            250,\n            \"less_100\",\n            \"1_to_4\",\n            2,\n            \"male\",\n            \"none\",\n            3,\n            \"real_estate\",\n            28,\n            \"none\",\n            \"own\",\n            2,\n            \"skilled\",\n            1,\n            \"yes\",\n            \"no\",\n            \"No Risk\",\n            [\n                0.7419002139563244,\n                0.25809978604367556\n            ],\n            \"Risk\",\n            [\n                0.767955712021837,\n                0.23204428797816307\n            ],\n            \"Risk\"\n        ],\n        [\n            \"0_to_200\",\n            28,\n            \"credits_paid_to_date\",\n            \"retraining\",\n            3693,\n            \"less_100\",\n            \"greater_7\",\n            3,\n            \"male\",\n            \"none\",\n            2,\n            \"savings_insurance\",\n            32,\n            \"none\",\n            \"own\",\n            1,\n            \"skilled\",\n            1,\n            \"none\",\n            \"yes\",\n            \"No Risk\",\n            [\n                0.6935080115729353,\n                0.3064919884270647\n            ],\n            \"Risk\",\n            [\n                0.8,\n                0.2\n            ],\n            \"Risk\"\n        ],\n        [\n            \"no_checking\",\n            28,\n            \"prior_payments_delayed\",\n            \"education\",\n            6235,\n            \"500_to_1000\",\n            \"greater_7\",\n            3,\n            \"male\",\n            \"none\",\n            3,\n            \"unknown\",\n            57,\n            \"none\",\n            \"own\",\n            2,\n            \"skilled\",\n            1,\n            \"none\",\n            \"yes\",\n            \"Risk\",\n            [\n                0.331110352092386,\n                0.668889647907614\n            ],\n            \"Risk\",\n            [\n                0.9,\n                0.1\n            ],\n            \"Risk\"\n        ],\n        [\n            \"no_checking\",\n            32,\n            \"outstanding_credit\",\n            \"vacation\",\n            9604,\n            \"500_to_1000\",\n            \"greater_7\",\n            6,\n            \"male\",\n            \"co-applicant\",\n            5,\n            \"unknown\",\n            57,\n            \"none\",\n            \"free\",\n            2,\n            \"skilled\",\n            2,\n            \"yes\",\n            \"yes\",\n            \"Risk\",\n            [\n                0.11270206970758759,\n                0.8872979302924124\n            ],\n            \"Risk\",\n            [\n                0.1,\n                0.9\n            ],\n            \"Risk\"\n        ],\n        [\n            \"no_checking\",\n            9,\n            \"prior_payments_delayed\",\n            \"car_new\",\n            1032,\n            \"100_to_500\",\n            \"4_to_7\",\n            3,\n            \"male\",\n            \"none\",\n            4,\n            \"savings_insurance\",\n            41,\n            \"none\",\n            \"own\",\n            1,\n            \"management_self-employed\",\n            1,\n            \"none\",\n            \"yes\",\n            \"No Risk\",\n            [\n                0.6704819620865308,\n                0.32951803791346923\n            ],\n            \"Risk\",\n            [\n                0.767955712021837,\n                0.23204428797816307\n            ],\n            \"Risk\"\n        ],\n        [\n            \"less_0\",\n            16,\n            \"credits_paid_to_date\",\n            \"vacation\",\n            3109,\n            \"less_100\",\n            \"4_to_7\",\n            3,\n            \"female\",\n            \"none\",\n            1,\n            \"car_other\",\n            36,\n            \"none\",\n            \"own\",\n            2,\n            \"skilled\",\n            1,\n            \"none\",\n            \"yes\",\n            \"No Risk\",\n            [\n                0.6735810290914039,\n                0.3264189709085961\n            ],\n            \"Risk\",\n            [\n                0.6,\n                0.4\n            ],\n            \"Risk\"\n        ],\n        [\n            \"0_to_200\",\n            11,\n            \"credits_paid_to_date\",\n            \"car_new\",\n            4553,\n            \"less_100\",\n            \"less_1\",\n            3,\n            \"female\",\n            \"none\",\n            3,\n            \"savings_insurance\",\n            22,\n            \"none\",\n            \"own\",\n            1,\n            \"management_self-employed\",\n            1,\n            \"none\",\n            \"yes\",\n            \"No Risk\",\n            [\n                0.637964656269084,\n                0.362035343730916\n            ],\n            \"Risk\",\n            [\n                0.767955712021837,\n                0.23204428797816307\n            ],\n            \"Risk\"\n        ],\n        [\n            \"no_checking\",\n            35,\n            \"outstanding_credit\",\n            \"appliances\",\n            7138,\n            \"500_to_1000\",\n            \"greater_7\",\n            5,\n            \"male\",\n            \"co-applicant\",\n            4,\n            \"unknown\",\n            49,\n            \"none\",\n            \"free\",\n            2,\n            \"skilled\",\n            2,\n            \"yes\",\n            \"yes\",\n            \"Risk\",\n            [\n                0.11270206970758759,\n                0.8872979302924124\n            ],\n            \"Risk\",\n            [\n                0.767955712021837,\n                0.23204428797816307\n            ],\n            \"Risk\"\n        ],\n        [\n            \"less_0\",\n            5,\n            \"all_credits_paid_back\",\n            \"car_new\",\n            1523,\n            \"less_100\",\n            \"unemployed\",\n            2,\n            \"female\",\n            \"none\",\n            2,\n            \"real_estate\",\n            19,\n            \"none\",\n            \"rent\",\n            1,\n            \"management_self-employed\",\n            1,\n            \"none\",\n            \"yes\",\n            \"No Risk\",\n            [\n                0.7304597628653227,\n                0.26954023713467745\n            ],\n            \"Risk\",\n            [\n                0.767955712021837,\n                0.23204428797816307\n            ],\n            \"Risk\"\n        ]\n    ]\n}", "execution_count": null, "outputs": []}, {"metadata": {"id": "9d8499cd3e5f4c8fbe1a895e2d547e81"}, "cell_type": "code", "source": "fields = feedback_payload['fields']\nvalues = feedback_payload['values']", "execution_count": null, "outputs": []}, {"metadata": {"id": "f56eac75cf6940ae83d8b149ed7c1841"}, "cell_type": "code", "source": "import pandas as pd\nscored_feedback_data = pd.DataFrame(values, columns = fields)", "execution_count": null, "outputs": []}, {"metadata": {"id": "f894a1c7975f456a90a0a9228274e9f8"}, "cell_type": "code", "source": "len(scored_feedback_data)", "execution_count": null, "outputs": []}, {"metadata": {"id": "91342309aa6f4a61a05f28811cff5033"}, "cell_type": "code", "source": "scored_feedback_data = scored_feedback_data.loc[scored_feedback_data.index.repeat(2)]", "execution_count": null, "outputs": []}, {"metadata": {"id": "e3108089ee8d44218e3f1810d7873564"}, "cell_type": "code", "source": "len(scored_feedback_data)", "execution_count": null, "outputs": []}, {"metadata": {"id": "72cc3d71a4f2487ba3a70aadcedf2ec6"}, "cell_type": "code", "source": "fields = scored_feedback_data.columns.tolist()\nvalues = scored_feedback_data[fields].values.tolist()\n\nfeedback_payload = {\"fields\": fields, \"values\": values}", "execution_count": null, "outputs": []}, {"metadata": {"id": "321e03bd3c04406f93874d21e8529e3b"}, "cell_type": "markdown", "source": "### Store the feedback payload using the data sets API"}, {"metadata": {"id": "d9cb27abe6f645a7882c8b12381840de"}, "cell_type": "code", "source": "import urllib3, requests, json\ndef generate_access_token():\n    headers={}\n    headers[\"Accept\"] = \"application/json\"\n    auth = HTTPBasicAuth(username, password)\n    \n    ICP_TOKEN_URL= WOS_CREDENTIALS[\"url\"] + \"/v1/preauth/validateAuth\"\n    \n    response = requests.get(ICP_TOKEN_URL, headers=headers, auth=auth, verify=False)\n    json_data = response.json()\n    icp_access_token = json_data['accessToken']\n    return icp_access_token", "execution_count": null, "outputs": []}, {"metadata": {"id": "77db93ac5b87422a89d7d7897c502216"}, "cell_type": "code", "source": "icp_access_token = generate_access_token()", "execution_count": null, "outputs": []}, {"metadata": {"id": "93fd6a9e4fcf421a8d5a93305e751457"}, "cell_type": "code", "source": "header = {\n    'Content-Type': 'application/json', \n    'Authorization': 'Bearer ' + icp_access_token\n}", "execution_count": null, "outputs": []}, {"metadata": {"id": "778c599544c740098105813113584fec"}, "cell_type": "code", "source": "DATASETS_STORE_RECORDS_URL =   WOS_CREDENTIALS[\"url\"] + \"/openscale/{0}/v2/data_sets/{1}/records\".format(data_mart_id, feedback_dataset_id)\nfor x in range(10):\n    response = requests.post(DATASETS_STORE_RECORDS_URL, json=feedback_payload, headers=header, verify=False)\n    json_data = response.json()\n    print(json_data)\n    time.sleep(3)", "execution_count": null, "outputs": []}, {"metadata": {"id": "84a151e157da412793741ba31d62b503"}, "cell_type": "markdown", "source": "### Wait for sometime, and make sure the records have reached to data sets related table."}, {"metadata": {"id": "f0c8924fed8f49a09db728b96f23b395"}, "cell_type": "code", "source": "time.sleep(10)\nDATASETS_STORE_RECORDS_URL =   WOS_CREDENTIALS[\"url\"] + \"/openscale/{0}/v2/data_sets/{1}/records?limit={2}&include_total_count={3}\".format(data_mart_id, feedback_dataset_id, 1, \"true\")\nresponse = requests.get(DATASETS_STORE_RECORDS_URL, headers=header, verify=False)\njson_data = response.json()\nprint(json_data['total_count'])", "execution_count": null, "outputs": []}, {"metadata": {"id": "9cce9ad8f9bf4fb095680d91ad38b2a0"}, "cell_type": "markdown", "source": "# Register OpenScale subscription with the Custom Metrics Provider\n\nYou can configure custom metrics for the subscriptions that have predefined monitors, such as fairness, quality, or drift or without predefined monitors."}, {"metadata": {"id": "73c85183a81f4997837587b2c64753c6"}, "cell_type": "markdown", "source": "## Configure Custom Metrics Provider with Watson OpenScale."}, {"metadata": {"id": "addc28e79d494e68a5cb6a942d4f8bac"}, "cell_type": "code", "source": "from ibm_watson_openscale import APIClient\nfrom ibm_watson_openscale.base_classes.watson_open_scale_v2 import MonitorMeasurementRequest\nfrom ibm_watson_openscale.base_classes.watson_open_scale_v2 import MonitorMetricRequest\nfrom ibm_watson_openscale.base_classes.watson_open_scale_v2 import MetricThreshold\nfrom ibm_watson_openscale.supporting_classes.enums import MetricThresholdTypes\nfrom ibm_watson_openscale.base_classes.watson_open_scale_v2 import MonitorTagRequest\nfrom ibm_watson_openscale.base_classes.watson_open_scale_v2 import Target\nfrom ibm_watson_openscale.supporting_classes.enums import TargetTypes\nfrom ibm_watson_openscale.base_classes.watson_open_scale_v2 import IntegratedSystems\nfrom ibm_watson_openscale.base_classes.watson_open_scale_v2 import MonitorInstanceSchedule, ScheduleStartTime, MonitorRuntime\n\nfrom datetime import datetime, timezone, timedelta\nimport uuid", "execution_count": null, "outputs": []}, {"metadata": {"id": "09b669c1459a4f75ad0d040d433795e4"}, "cell_type": "markdown", "source": "## Create the integrated system for the custom metrics provider. <a name=\"custom\"></a>\n\n\nUpdate the custom metrics deployment URL, which is created during the Python function creation in the integrated system. Watson OpenScale invokes the deployment URL at runtime to compute the custom metrics. \n\nYou must define the authentication type based on the communication with custom metrics deployment. Watson OpenScale supports 2 types of authentication: basic and bearer. If custom metrics deployment accepts the `basic` authentication type, then provide `auth_type=basic` otherwise use `auth_type=bearer`."}, {"metadata": {"id": "f119dff4-b24f-4a12-9843-0065729cc084"}, "cell_type": "code", "source": "CUSTOM_METRICS_PROVIDER_CREDENTIALS = {\n    \"auth_type\":\"bearer\",\n    \"token_info\": {\n        \"url\": \"{}/icp4d-api/v1/authorize\".format(WOS_CREDENTIALS['url']),\n        \"headers\": {\n            \"Content-Type\": \"application/json\",\n            \"Accept\": \"application/json\"\n        },\n        \"payload\": {\n            \"username\": WOS_CREDENTIALS['username'],\n            \"api_key\": WOS_CREDENTIALS['apikey'],\n        },\n        \"method\": \"post\"\n    }\n}", "execution_count": null, "outputs": []}, {"metadata": {"id": "296fc469-b17b-412d-a252-f392b00e05aa"}, "cell_type": "markdown", "source": "### Remove existing integrated system "}, {"metadata": {"id": "198929f5-1ad4-4e9a-ab35-7f180ebeb1ed"}, "cell_type": "code", "source": "# Delete existing custom metrics provider integrated systems if present\nintegrated_systems = IntegratedSystems(wos_client).list().result.integrated_systems\nfor system in integrated_systems:\n    if system.entity.type == 'custom_metrics_provider' and system.entity.name == CUSTOM_METRICS_PROVIDER_NAME:\n        print(\"Deleting integrated system {}\".format(system.entity.name))\n        IntegratedSystems(wos_client).delete(integrated_system_id=system.metadata.id)", "execution_count": null, "outputs": []}, {"metadata": {"id": "31eb1a73-780b-4565-97aa-53a2ad1e6d6c"}, "cell_type": "code", "source": "custom_metrics_integrated_system = IntegratedSystems(wos_client).add(\n    name=CUSTOM_METRICS_PROVIDER_NAME,\n    description=CUSTOM_METRICS_PROVIDER_NAME,\n    type=\"custom_metrics_provider\",\n    credentials= CUSTOM_METRICS_PROVIDER_CREDENTIALS,\n    connection={\n        \"display_name\": CUSTOM_METRICS_PROVIDER_NAME,\n        \"endpoint\": scoring_url\n    }\n).result\n\nintegrated_system_id = custom_metrics_integrated_system.metadata.id\nprint(custom_metrics_integrated_system)", "execution_count": null, "outputs": []}, {"metadata": {"id": "8bb4bd32-b0b5-4ddc-835f-c40cab1c779a"}, "cell_type": "markdown", "source": "## Set up the custom monitor definition and instance. <a name=\"instance\"></a>\n"}, {"metadata": {"id": "59b9827e31ab45edbad79789213703d1"}, "cell_type": "markdown", "source": "### Check for the existence of the custom monitor definition.\n"}, {"metadata": {"id": "14e3f1e2e2f24bfd86f86feee24619e0"}, "cell_type": "code", "source": "def get_custom_monitor_definition():\n    monitor_definitions = wos_client.monitor_definitions.list().result.monitor_definitions\n    for definition in monitor_definitions:\n        if CUSTOM_MONITOR_NAME == definition.entity.name:\n            return definition\n    return None   ", "execution_count": null, "outputs": []}, {"metadata": {"id": "30720ce70ddf485782ebf767a606366b"}, "cell_type": "markdown", "source": "### Create the  custom monitor definition.\n\nUpdate the custom metric names, threshold types (`LOWER_LIMIT`, `UPPER_LIMIT`) and default values as required. You can define the threshold type as lower limit, upper limit, or both."}, {"metadata": {"id": "18f34443-76c8-4b9c-85da-a58b10003e7c"}, "cell_type": "code", "source": "###################################################################\n# Update your custom monitor metrics names in the following field. Use the same metric names for creating the \n# monitor definition and publishing the metrics to Openscale in your python function\n####################################################################\nCUSTOM_MONITOR_METRICS_NAMES = ['sensitivity','specificity', 'gender_less40_fav_prediction_ratio']\n#Update the tag values if you want to fetch the metrics by tags\nTAGS= ['region']\nTAG_DESCRIPTION =['customer geographical region'] ", "execution_count": null, "outputs": []}, {"metadata": {"id": "e56a0fa8-bbb4-4d9c-b23b-c8362c958390"}, "cell_type": "markdown", "source": "### Create  schedule for custom monitor\n\nCreates a schedule for custom monitor with a defined repeat type and repeat interval. Supported repeat types are \"minute\", \"hour\", \"day\", \"week\", \"month\", \"year\". Defaults to once every hour if `repeat_interval` and  `repeat_type` values are not specified\n"}, {"metadata": {"id": "1cc51c9c-d32f-48a1-ac9c-c624fee545da"}, "cell_type": "code", "source": "#Enable the schedule for your custom monitor. Update the value to false if you want to disable the schedule\nENABLE_SCHEDULE = True\n\n#Update the repeat interval and repeat type: Default is 1 hour\nrepeat_interval = 1\nrepeat_type = \"hour\"\n\n#Update the delay unit and interval to trigger the first schedule run after the custom monitor instance is created. Default is 30 minutes\ndelay_unit = \"minute\"\ndelay_time= 30\n\nstart_time= ScheduleStartTime(type = \"relative\", delay_unit= delay_unit, delay = delay_time)\nschedule = MonitorInstanceSchedule(repeat_interval=repeat_interval,repeat_unit=repeat_type, start_time=start_time)    ", "execution_count": null, "outputs": []}, {"metadata": {"id": "df7bfd7b6f274d54897133f94dc14310"}, "cell_type": "code", "source": "#Update the Threshold types and default values of the metrics\n\ndef custom_metric_definitions():\n    \n    metrics = [MonitorMetricRequest(name=CUSTOM_MONITOR_METRICS_NAMES[0],\n                                    thresholds=[MetricThreshold(type=MetricThresholdTypes.LOWER_LIMIT, default=0.8)]),\n              MonitorMetricRequest(name=CUSTOM_MONITOR_METRICS_NAMES[1],\n                                 thresholds=[MetricThreshold(type=MetricThresholdTypes.LOWER_LIMIT, default=0.6),MetricThreshold(type=MetricThresholdTypes.UPPER_LIMIT, default=1)]),\n              MonitorMetricRequest(name=CUSTOM_MONITOR_METRICS_NAMES[2],\n                                 thresholds=[MetricThreshold(type=MetricThresholdTypes.LOWER_LIMIT, default=0.6),MetricThreshold(type=MetricThresholdTypes.UPPER_LIMIT, default=1)])]\n    \n    #Comment the below tags code if there are no tags to be created\n    tags = [MonitorTagRequest(name=TAGS[0], description=TAG_DESCRIPTION[0])]\n    \n    return metrics, tags", "execution_count": null, "outputs": []}, {"metadata": {"id": "79e2f275e13948e5834e77ec4dfadeb6"}, "cell_type": "code", "source": "def create_custom_monitor_definition():\n    # check if the custom monitor definition already exists or not\n    existing_definition = get_custom_monitor_definition()\n\n    # if it does not exists, then create a new one.\n    if existing_definition is None:\n        metrics, tags = custom_metric_definitions()\n        if ENABLE_SCHEDULE:\n            monitor_runtime = MonitorRuntime(type=\"custom_metrics_provider\")\n            custom_monitor_details = wos_client.monitor_definitions.add(name=CUSTOM_MONITOR_NAME, metrics=metrics, tags=tags, schedule = schedule, monitor_runtime = monitor_runtime, background_mode=False).result\n        else:\n            custom_monitor_details = wos_client.monitor_definitions.add(name=CUSTOM_MONITOR_NAME, metrics=metrics, tags=tags, background_mode=False).result\n    else:\n        # otherwise, send the existing definition\n        custom_monitor_details = existing_definition\n    return custom_monitor_details", "execution_count": null, "outputs": []}, {"metadata": {"id": "8d352bcf09e646d0a6d174e54078f2c6"}, "cell_type": "code", "source": "custom_monitor_details = create_custom_monitor_definition()\nprint(custom_monitor_details)\ncustom_monitor_id = custom_monitor_details.metadata.id\ncustom_monitor_id\n", "execution_count": null, "outputs": []}, {"metadata": {"id": "3a0dbf50822c40c680cc9430eb88306c"}, "cell_type": "markdown", "source": "### Check the existence of custom monitor instance.\n"}, {"metadata": {"id": "7176b942f61f4d0381fb63b92101e76f"}, "cell_type": "code", "source": "def get_custom_monitor_instance(custom_monitor_id):\n    monitor_instances = wos_client.monitor_instances.list(data_mart_id = WOS_GUID, monitor_definition_id = custom_monitor_id, target_target_id = subscription_id).result.monitor_instances\n    if len(monitor_instances) == 1:\n        return monitor_instances[0]\n    return None", "execution_count": null, "outputs": []}, {"metadata": {"id": "32c84dce-c23a-42ef-b874-477a71b4ef36"}, "cell_type": "code", "source": "# Openscale MRM service invokes custom metrics deployment url during runtime and wait for the default time of 60 second's to \n# to check the run status ie finished/Failed and fetch the latest measurement. Increase the wait time, if the runtime deployment \n# takes more than 60 seconds to compute and publish the custom metrics \n\n#Update the wait time here.\ncustom_metrics_wait_time = 300 #time in seconds <update the time here>\n\n#Maximum number of records to consider during custom monitor run evaluation. Update max_records value as per the requirement\nmax_records = None\n#Minimum number of records to consider during custom monitor run evaluation. Update min_records value as per the requirement\nmin_records = None", "execution_count": null, "outputs": []}, {"metadata": {"id": "5c49fe8f-7921-410c-9983-6f998522d715"}, "cell_type": "markdown", "source": "### Update the custom monitor instance."}, {"metadata": {"id": "67ed74d7-928a-4f64-a7f7-dc9bfb5d5fbc"}, "cell_type": "code", "source": "def update_custom_monitor_instance(custom_monitor_instance_id):\n    payload = [\n     {\n       \"op\": \"replace\",\n       \"path\": \"/parameters\",\n       \"value\": {\n           \"custom_metrics_provider_id\": integrated_system_id,\n           \"custom_metrics_wait_time\":   custom_metrics_wait_time,\n            \"enable_custom_metric_runs\": True\n       }\n     }\n    ]\n    if max_records is not None:\n        payload[0][\"value\"][\"max_records\"] = max_records    \n    if min_records is not None:\n        payload[0][\"value\"][\"min_records\"] = min_records \n        \n    response = wos_client.monitor_instances.update(custom_monitor_instance_id, payload, update_metadata_only = True)\n    result = response.result\n    return result", "execution_count": null, "outputs": []}, {"metadata": {"id": "678df1de54164d6284cdf9f255affd8a"}, "cell_type": "markdown", "source": "### For the custom monitor definition, create a custom monitor instance.\n"}, {"metadata": {"id": "97624e1c0a8e4648af05162690c96c66"}, "cell_type": "code", "source": "def create_custom_monitor_instance(custom_monitor_id):\n    # Check if an custom monitor instance already exists\n    existing_monitor_instance = get_custom_monitor_instance(custom_monitor_id)\n\n    # If it does not exist, then create one\n    if existing_monitor_instance is None:\n        target = Target(\n                target_type=TargetTypes.SUBSCRIPTION,\n                target_id=subscription_id\n            )\n        parameters = {\n            \"custom_metrics_provider_id\": integrated_system_id,\n            \"custom_metrics_wait_time\":   custom_metrics_wait_time,\n            \"enable_custom_metric_runs\": True\n        }\n        if max_records is not None:\n            parameters[\"max_records\"] =   max_records\n        if min_records  is not None:\n            parameters[\"min_records\"] =   min_records \n        \n        #Update your custom monitor metric ids in the below thresholds to update the default value\n        thresholds = [ MetricThresholdOverride(metric_id=CUSTOM_MONITOR_METRICS_NAMES[0], type = MetricThresholdTypes.LOWER_LIMIT, value=0.85),\n                       MetricThresholdOverride(metric_id=CUSTOM_MONITOR_METRICS_NAMES[1], type = MetricThresholdTypes.LOWER_LIMIT, value=0.7),\n                       MetricThresholdOverride(metric_id=CUSTOM_MONITOR_METRICS_NAMES[2], type = MetricThresholdTypes.UPPER_LIMIT, value=0.95)\n                     ]\n        # create the custom monitor instance id here.\n        custom_monitor_instance_details = wos_client.monitor_instances.create(\n                    data_mart_id=WOS_GUID,\n                    background_mode=False,\n                    monitor_definition_id=custom_monitor_id,\n                    target=target,\n                    parameters=parameters,\n                    thresholds = thresholds\n        ).result\n    else:\n        # otherwise, update the existing one with latest integrated system details.\n        instance_id = existing_monitor_instance.metadata.id\n        custom_monitor_instance_details = update_custom_monitor_instance(instance_id)\n    return custom_monitor_instance_details", "execution_count": null, "outputs": []}, {"metadata": {"id": "ffb159c5c1f54ffe9f1b7a48b727810c"}, "cell_type": "code", "source": "monitor_instance_details = create_custom_monitor_instance(custom_monitor_id)\ncustom_monitor_instance_id = monitor_instance_details.metadata.id\nprint(monitor_instance_details)\n", "execution_count": null, "outputs": []}, {"metadata": {"id": "29eabd43-f976-4ce4-a403-020556d479a8"}, "cell_type": "markdown", "source": "### Run the custom monitor <a name=\"run\"></a>\n"}, {"metadata": {"id": "0a40ab77-1ae1-4109-98bd-6552bb48c6a8"}, "cell_type": "code", "source": "#Execute the custom metrics provider deployment\nmonitor_instance_run_info = wos_client.monitor_instances.run(\n        background_mode=False,\n        monitor_instance_id=custom_monitor_instance_id\n     ).result\n\nmonitor_instance_run_info\ncustom_monitor_run_id = monitor_instance_run_info.metadata.id\n", "execution_count": null, "outputs": []}, {"metadata": {"id": "2b524be7-8a65-410b-8e85-2b26f5b62958"}, "cell_type": "markdown", "source": "### Show Custom Metrics"}, {"metadata": {"id": "2bbbcac1-83de-4428-8147-0a0678677e5c"}, "cell_type": "code", "source": "wos_client.monitor_instances.show_metrics(monitor_instance_id=custom_monitor_instance_id)", "execution_count": null, "outputs": []}, {"metadata": {"id": "1c0fca1109bf4c048fe61d6de5815179"}, "cell_type": "markdown", "source": "## Recap of the steps performed in this notebook\n\n- Create a python function\n- Deploy the python function to WML\n- Create an OpenScale Integrated System pointing to the python function\n- Create a Custom Monitor Definition mentioning various custom metrics\n- Create a Custom Monitor Instance and specify the Integrated System ID in the monitor instance configuration.\n- Run the Custom monitor"}, {"metadata": {"id": "210c3b5835494f67812f0704bad84705"}, "cell_type": "markdown", "source": "## Congratulations\n\nYou have finished configuring Custom Monitor Definition and Monitor instance and executing Custom Monitor Run for your Subscription. You can also run the custom monitor from `Watson OpenScale Dashboard`(https://url-to-your-cp4d-cluster/aiopenscale). Click the tile of your model and select `Evaluate Now` option from `Actions` drop down menu to run the custom monitor."}], "metadata": {"kernelspec": {"name": "python3", "display_name": "Python 3.9", "language": "python"}, "language_info": {"name": "python", "version": "3.9.12", "mimetype": "text/x-python", "codemirror_mode": {"name": "ipython", "version": 3}, "pygments_lexer": "ipython3", "nbconvert_exporter": "python", "file_extension": ".py"}, "varInspector": {"cols": {"lenName": 16, "lenType": 16, "lenVar": 40}, "kernels_config": {"python": {"delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())"}, "r": {"delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) "}}, "types_to_exclude": ["module", "function", "builtin_function_or_method", "instance", "_Feature"], "window_display": false}}, "nbformat": 4, "nbformat_minor": 1}