Assets/notebooks/Factsheet_Custom_Facts_Utilities.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction <a name=\"introduction\"></a>\n",
    "\n",
    "The notebook will **create an empty external model for demonstration purposes**, **setup custom facts definitions** for it and show assets and model usecases utilities related to  **Custom facts** <br/>\n",
    "For more information around helper utilities see [here](https://ibm-aigov-facts-client.mybluemix.net/)\n",
    "\n",
    "It requires following services in the platform:\n",
    "\n",
    "- Watson Knowledge catalog\n",
    "- AI factsheet\n",
    "- Facts client python sdk (>=1.0.42)\n",
    "\n",
    "Sections: \n",
    "\n",
    "- [Setup](#setup)\n",
    "    - [Cloud](#setupcloud)\n",
    "    - [Cloud Pak for data](#setupcpd)\n",
    "- [Initialize client](#init)\n",
    "    - [Cloud](#setupcloudinit)\n",
    "    - [Cloud Pak for data](#setupcpdinit)\n",
    "- [Track custom model development facts](#trackdevfact)\n",
    "- [Create external model properties](#createprop)\n",
    "- [Save external model to catalog](#save)\n",
    "- [Track model and model use case custom facts](#customfacts)\n",
    "    - [Create custom facts definitions](#customfactsdef)\n",
    "    - [Get custom facts definitions for model](#getcustomfactsdef)\n",
    "        - [Set custom facts](#set)\n",
    "        - [Get custom facts](#get)\n",
    "        - [Get all facts](#get_all)\n",
    "        - [Remove custom facts](#remove)\n",
    "        - [Add to model usecase ](#add_mu)\n",
    "    - [Create custom facts definitions for model usecase](#customfactsdefmu)\n",
    "    - [Get custom facts definitions for model usecase](#getcustomfactsdefmu)\n",
    "    - [Get model usecase ](#get_mu)\n",
    "        - [Set custom facts](#set_mu_custom)\n",
    "        - [Get custom facts](#get_mu_custom)\n",
    "        - [Get all facts](#get_all_mu)\n",
    "        - [Remove custom facts](#remove_mu_custom)\n",
    "        - [Get tracked model usecases ](#get_tracked_mu)\n",
    "- [Remove model from model usecase ](#remove_mu)\n",
    "- [Cleanup](#clean)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"setupcloud\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup<a name=\"setup\"></a>\n",
    "This notebook uses external models to illustrate custom facts. For it to work an administrator needs to enable support for external models on the `Manage` tab of the `Model Inventory` page. This will also make sure the platform asset catalog (PAC) is created.\n",
    "\n",
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!pip install wget --quiet\n",
    "!pip install pandas --quiet\n",
    "!pip install -U ibm-aigov-facts-client --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "import json\n",
    "import wget\n",
    "import pandas as pd\n",
    "import shutil\n",
    "import time\n",
    "from IPython.core.display import display, Markdown\n",
    "from ibm_aigov_facts_client import AIGovFactsClient\n",
    "from ibm_aigov_facts_client.supporting_classes.factsheet_utils import DeploymentDetails,TrainingDataReference,ExternalModelSchemas\n",
    "\n",
    "warnings.filterwarnings('ignore')\n",
    "shutil.rmtree('./mlruns', ignore_errors=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- This sample will use IBM Cloud by default. If you prefer to use Cloud pak for data, set `use_cp4d=True`\n",
    "- Flag `run_cleanup_at_end` offers option to delete created assets at the end of the notebook.The notebook will show URL to UI for model and model use case at certain cells. By dafault we set it to `run_cleanup_at_end=False` so you can access UI and see the changes. If you decide to cleanup assets at the end, set `run_cleanup_at_end=True` and remember cells showing links to UI will `NOT` work in that case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "use_cp4d=False\n",
    "run_cleanup_at_end=False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to top](#introduction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## IBM Cloud<a name=\"setupcloud\"></a>\n",
    "\n",
    "Your Cloud API key can be generated by going to the Users section of the Cloud console. From that page, go to **Manage->Access(IAM)->API keys-> Create**. Give your key a name and click Create, then copy the created key and use as API_KEY.\n",
    "\n",
    "NOTE: You can also get OpenScale API_KEY using IBM CLOUD CLI.\n",
    "\n",
    "How to install IBM Cloud (bluemix) console: instruction\n",
    "\n",
    "How to get api key using console:\n",
    "\n",
    "```\n",
    "bx login --sso\n",
    "bx iam api-key-create 'my_key'\n",
    "```\n",
    "\n",
    "User apikey used below should have `EDITOR` access to platform asset catalog."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "API_KEY=\"***\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"setupcpd\"></a>\n",
    "## Cloud Pak for Data\n",
    "\n",
    "\n",
    "- Service url is the Cloud pak for data platform host URL. For skytap environment, it would be the internal nginx URL.\n",
    "- You can either use user password or platform apikey to authenticate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "if use_cp4d:\n",
    "    from ibm_aigov_facts_client import AIGovFactsClient,CloudPakforDataConfig\n",
    "\n",
    "    creds=CloudPakforDataConfig(service_url=\"***\",\n",
    "        username=\"***\",\n",
    "        api_key=\"***\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_identifier=\"PytorchExternalModelId\"\n",
    "model_name=\"PytorchExternalModel\"\n",
    "experiment_name=\"PytorchExperiment\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to top](#introduction)\n",
    "# Client Initialization\n",
    "<a name=\"init\"></a>\n",
    "## IBM Cloud<a name=\"setupcloudinit\"></a>\n",
    "\n",
    "- If running this notebook multiple times with same experiment name or anytime face error saying `Experiment with same name already exists`, use `set_as_current_experiment=True` when initiating client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022/12/12 18:37:24 INFO : Experiment successfully created with ID 1 and name PytorchExperiment\n"
     ]
    }
   ],
   "source": [
    "if not use_cp4d:\n",
    "    facts_client = AIGovFactsClient(api_key=API_KEY,experiment_name=experiment_name,external_model=True,enable_autolog=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cloud Pak for Data<a name=\"setupcpdinit\"></a>\n",
    "\n",
    "- If running this notebook multiple times with same experiment name or anytime face error saying `Experiment with same name already exists`, use `set_as_current_experiment=True` when initiating client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "if use_cp4d:\n",
    "    facts_client = AIGovFactsClient(experiment_name=experiment_name,external_model=True,cloud_pak_for_data_configs=creds,enable_autolog=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Track development facts manually <a name=\"trackdevfact\"></a>\n",
    "\n",
    "You can collect different model training facts (i.e., training metrics, params and tags)using our facts client. For external models, facts can be captured using autolog or you can capture these facts manually if model is built with framework that is not supported for autolog. In that case, only option is to capture different model training facts and publish to factsheet manually. Steps to follow:\n",
    "\n",
    "- Make sure to initiate client with `enable_autolog=False`\n",
    "- Start tracing, ideally on top of the noetbook. It will intiate a run under provided experiment name\n",
    "- Calculate and log metrics,params and tags as applicable\n",
    "- get current run id\n",
    "- Export the corresponding run payload to factsheet\n",
    "- End tracing.\n",
    "\n",
    "In this example, assume the model is built using regular `Pytorch` which is note supported for autolog so we will log some training facts manually and export to factsheet before saving the model stub.\n",
    "For additonal information on manual logging, you can check [here](https://ibm-aigov-facts-client.mybluemix.net/#factsheet-manual-log-elements)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022/12/12 18:37:24 INFO : Manual tracing initiated successfully under run c783a8502fca401390dd909fb4ab8bfd\n",
      "2022/12/12 18:37:24 INFO : logged metrics ['ROC training', 'F1 training'] successfully under run c783a8502fca401390dd909fb4ab8bfd\n",
      "2022/12/12 18:37:24 INFO : logged params ['learning_rate', 'epochs'] successfully under run c783a8502fca401390dd909fb4ab8bfd\n",
      "2022/12/12 18:37:24 INFO : logged tags ['engineering', 'release.version'] successfully under run c783a8502fca401390dd909fb4ab8bfd\n",
      "2022/12/12 18:37:29 INFO : logging results to factsheet for run_id c783a8502fca401390dd909fb4ab8bfd\n",
      "2022/12/12 18:37:31 INFO : Successfully logged results to Factsheet service for run_id c783a8502fca401390dd909fb4ab8bfd under asset_id: 34fc295b-a96c-4281-a51e-aee847932bf8 \n",
      "2022/12/12 18:37:31 INFO : Manual tracing ended successfully for run c783a8502fca401390dd909fb4ab8bfd\n"
     ]
    }
   ],
   "source": [
    "#start tracing \n",
    "facts_client.manual_log.start_trace()\n",
    "\n",
    "#log matrics\n",
    "facts_client.manual_log.log_metrics({\"ROC training\": 0.8, \"F1 training\": 0.9})\n",
    "\n",
    "# log training params\n",
    "p={\"learning_rate\": 0.01, \"epochs\": 10}\n",
    "facts_client.manual_log.log_params(params=p)\n",
    "\n",
    "# log tags related to model\n",
    "facts_client.manual_log.set_tags({\"engineering\": \"SaaS Platform\",\n",
    "        \"release.version\": \"1.1.0\"})\n",
    "\n",
    "time.sleep(5)\n",
    "# get current run id\n",
    "current_run=facts_client.runs.get_current_run_id()\n",
    "\n",
    "#export to factsheet\n",
    "facts_client.export_facts.export_payload_manual(current_run)\n",
    "#end tracing\n",
    "facts_client.manual_log.end_trace()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to top](#introduction)\n",
    "## Create model properties<a name=\"createprop\"></a>\n",
    "\n",
    "To demonstrate custom facts we need to create a placeholder model on which we can show the APIs. \n",
    "The model itself does not matter for custom fact demonstration so we will just create a sample one quickly.\n",
    "In a real end-to-end the model would be created based on real Sagemaker/Azure/Custom information and use real names and ids from these environments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Get all necessary details for the external model:\n",
    "    - Input and Output schema details\n",
    "    - Training data reference informations\n",
    "    - Model deployment details from the platform where model is stored\n",
    "- You can learn about these supporting classes in [here](https://ibm-aigov-facts-client.mybluemix.net/#helper-classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'external_model_facts_payload.json'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!rm external_model_facts_payload.json\n",
    "wget.download(\"https://raw.githubusercontent.com/IBM/ai-governance-factsheet-samples/main/Assets/data/external_model_facts_payload.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"external_model_facts_payload.json\", 'r') as f:\n",
    "    payload = json.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "external_schemas=ExternalModelSchemas(input=payload[\"schemas\"][\"input\"],output=payload[\"schemas\"][\"output\"])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Derive training data reference schema from model training input schema\n",
    "\n",
    "training_data_schema={}\n",
    "\n",
    "input_schema= payload[\"schemas\"][\"input\"][0]\n",
    "training_data_schema[\"schema\"]=input_schema\n",
    "train_data_ref=TrainingDataReference(schema=training_data_schema)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can get deployment details for model asset from platforms where model is built (i.e., AzureML, Sagemaker etc.) and save as metadata in model use case.\n",
    "- ID: Deployment identifier specific to providers\n",
    "- NAME: Name of the deployment\n",
    "- TYPE: Deployment type\n",
    "- SCORING ENDPOINT: Deployment scoring endpoint url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "deployment_details=DeploymentDetails(identifier=\"098f6bcd4621d373cade4e832627b4f6\"\n",
    "                                    ,name=\"german_credit_risk_deployment\"\n",
    "                                    ,deployment_type=\"online\"\n",
    "                                    ,scoring_endpoint=\"test.iscore/score\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to top](#introduction)\n",
    "# Save external model to catalog<a name=\"save\"></a>\n",
    "\n",
    "By default external model stubs are saved in the platform asset catalog which an administrator will have to enable.\n",
    "See [the documentation](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/factsheet-external.html) for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022/12/12 18:37:35 INFO : External model asset saved successfully under asset_id c560f9d4-5ec7-431c-914b-394241f3f39a and catalog 745c21d7-ab6f-4408-9b3b-5d4760ccee1a\n",
      "2022/12/12 18:37:36 INFO : Current model information: {'asset_id': 'c560f9d4-5ec7-431c-914b-394241f3f39a', 'container_type': 'catalog', 'container_id': '745c21d7-ab6f-4408-9b3b-5d4760ccee1a', 'facts_type': 'modelfacts_user'}\n"
     ]
    }
   ],
   "source": [
    "external_model=facts_client.external_model_facts.save_external_model_asset(model_identifier=model_identifier,name=model_name\n",
    "                        ,deployment_details=deployment_details\n",
    "                        ,schemas=external_schemas\n",
    "                        ,training_data_reference=train_data_ref\n",
    "                        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'name': 'PytorchExternalModel',\n",
       " 'asset_type': 'model_stub',\n",
       " 'url': 'https://dataplatform.cloud.ibm.com/data/catalogs/745c21d7-ab6f-4408-9b3b-5d4760ccee1a/asset/c560f9d4-5ec7-431c-914b-394241f3f39a/asset-preview?context=cpdaas',\n",
       " 'asset_id': 'c560f9d4-5ec7-431c-914b-394241f3f39a',\n",
       " 'container_type': 'catalog',\n",
       " 'container_id': '745c21d7-ab6f-4408-9b3b-5d4760ccee1a',\n",
       " 'facts_type': 'modelfacts_user'}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "external_model.get_info(verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "[Click here to see the external model and logged development facts with it in UI](https://dataplatform.cloud.ibm.com/data/catalogs/745c21d7-ab6f-4408-9b3b-5d4760ccee1a/asset/c560f9d4-5ec7-431c-914b-394241f3f39a/asset-preview?context=cpdaas)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ui_url = external_model.get_info(verbose=True)[\"url\"]\n",
    "display(Markdown(\"[Click here to see the external model and logged development facts with it in UI](\" + ui_url + \")\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to top](#introduction)\n",
    "## Log model and model use case custom facts<a name=\"customfacts\"></a> \n",
    "\n",
    "\n",
    "### Create custom facts definitions<a name=\"customfactsdef\"></a> \n",
    "\n",
    "- User should have manage catalog permission\n",
    "- Before you can set/get a custom fact you need to define the \"slots\" or \"schema\" of what custom facts you want to add to a model or model use case. In the defition you can specify the name, id and type for each fact \"slot\".\n",
    "- Technically custom facts definitions are global asset attributes you want to see visible in your account. By default you create definitions under `modelfacts_user` that is associated to the model asset. You can also use `model_entry_user` as facts type to create definitions under model usecase.\n",
    "- It can be of string, integer and date type only\n",
    "- You can set list of strings if the attributes definition property `is_array` is set to True only.\n",
    "- You can check the documentation to understand different options [here](https://ibm-aigov-facts-client.mybluemix.net/#factsheet-asset-elements)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Asset_type_definition.csv'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!rm Asset_type_definition.csv\n",
    "wget.download(\"https://raw.githubusercontent.com/IBM/ai-governance-factsheet-samples/main/Assets/data/Asset_type_definition.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>type</th>\n",
       "      <th>description</th>\n",
       "      <th>placeholder</th>\n",
       "      <th>is_array</th>\n",
       "      <th>required</th>\n",
       "      <th>hidden</th>\n",
       "      <th>readonly</th>\n",
       "      <th>default_value</th>\n",
       "      <th>label</th>\n",
       "      <th>minimum</th>\n",
       "      <th>maximum</th>\n",
       "      <th>min_length</th>\n",
       "      <th>max_length</th>\n",
       "      <th>is_searchable</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>TrainingData_Size</td>\n",
       "      <td>integer</td>\n",
       "      <td>Number of records in the training data</td>\n",
       "      <td>desc of userattrstr</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>0</td>\n",
       "      <td>TrainingData_Size</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>TrainingData_Ratio</td>\n",
       "      <td>string</td>\n",
       "      <td>Percentage of records in the training data</td>\n",
       "      <td>Percentage of records in the training data</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>0</td>\n",
       "      <td>TrainingData_Ratio</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>TestData_Size</td>\n",
       "      <td>integer</td>\n",
       "      <td>desc of userattrint</td>\n",
       "      <td>desc of userattrint</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>0</td>\n",
       "      <td>TestData_Size</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>TestData_Ratio</td>\n",
       "      <td>string</td>\n",
       "      <td>Percentage of records in the test data</td>\n",
       "      <td>Percentage of records in the test data</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>0</td>\n",
       "      <td>TestData_Ratio</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Train_Class_Distributions</td>\n",
       "      <td>string</td>\n",
       "      <td>class distributions</td>\n",
       "      <td>class distributions</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>0</td>\n",
       "      <td>Train_Class_Distributions</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        name     type  \\\n",
       "0          TrainingData_Size  integer   \n",
       "1         TrainingData_Ratio   string   \n",
       "2              TestData_Size  integer   \n",
       "3             TestData_Ratio   string   \n",
       "4  Train_Class_Distributions   string   \n",
       "\n",
       "                                  description  \\\n",
       "0      Number of records in the training data   \n",
       "1  Percentage of records in the training data   \n",
       "2                         desc of userattrint   \n",
       "3      Percentage of records in the test data   \n",
       "4                         class distributions   \n",
       "\n",
       "                                  placeholder  is_array  required  hidden  \\\n",
       "0                         desc of userattrstr     False      True   False   \n",
       "1  Percentage of records in the training data     False      True   False   \n",
       "2                         desc of userattrint     False      True   False   \n",
       "3      Percentage of records in the test data     False      True   False   \n",
       "4                         class distributions      True      True   False   \n",
       "\n",
       "   readonly  default_value                      label  minimum  maximum  \\\n",
       "0     False              0          TrainingData_Size      NaN      NaN   \n",
       "1     False              0         TrainingData_Ratio      NaN      NaN   \n",
       "2     False              0              TestData_Size      NaN      NaN   \n",
       "3     False              0             TestData_Ratio      NaN      NaN   \n",
       "4     False              0  Train_Class_Distributions      NaN      NaN   \n",
       "\n",
       "   min_length  max_length  is_searchable  \n",
       "0         NaN         NaN           True  \n",
       "1         NaN         NaN           True  \n",
       "2         NaN         NaN          False  \n",
       "3         NaN         NaN           True  \n",
       "4         NaN         NaN          False  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('Asset_type_definition.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022/12/12 18:37:38 INFO : Creating definitions for type modelfacts_user\n",
      "2022/12/12 18:37:39 INFO : Custom facts definitions updated Successfully\n"
     ]
    }
   ],
   "source": [
    "facts_client.assets.create_custom_facts_definitions(\"Asset_type_definition.csv\",overwrite=True)\n",
    "time.sleep(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get custom facts definitions<a name=\"getcustomfactsdef\"></a> \n",
    "\n",
    "This will define new custom fact slots for sample custom facts like TrainingData_Size, TrainingData_Ratio, TrainingData_Size, TestData_Size, Train_Class_Distributions\n",
    "You could define any kind of custom fact. But note that they are \"global\" i.e. they will be available as slots on every model in the same way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'description': 'The model fact user asset type to capture user defined attributes.',\n",
       " 'fields': [{'key': 'TrainingData_Size',\n",
       "   'type': 'integer',\n",
       "   'facet': False,\n",
       "   'is_array': False,\n",
       "   'search_path': 'TrainingData_Size',\n",
       "   'is_searchable_across_types': False},\n",
       "  {'key': 'TrainingData_Ratio',\n",
       "   'type': 'string',\n",
       "   'facet': False,\n",
       "   'is_array': False,\n",
       "   'search_path': 'TrainingData_Ratio',\n",
       "   'is_searchable_across_types': False},\n",
       "  {'key': 'TestData_Ratio',\n",
       "   'type': 'string',\n",
       "   'facet': False,\n",
       "   'is_array': False,\n",
       "   'search_path': 'TestData_Ratio',\n",
       "   'is_searchable_across_types': False}],\n",
       " 'relationships': [],\n",
       " 'properties': {'TrainingData_Size': {'type': 'integer',\n",
       "   'description': 'Number of records in the training data',\n",
       "   'placeholder': 'desc of userattrstr',\n",
       "   'is_array': False,\n",
       "   'required': True,\n",
       "   'hidden': False,\n",
       "   'readonly': False,\n",
       "   'default_value': 0.0,\n",
       "   'minimum': '',\n",
       "   'maximum': '',\n",
       "   'label': {'default': 'TrainingData_Size', 'en': 'TrainingData_Size'}},\n",
       "  'TrainingData_Ratio': {'type': 'string',\n",
       "   'description': 'Percentage of records in the training data',\n",
       "   'placeholder': 'Percentage of records in the training data',\n",
       "   'is_array': False,\n",
       "   'required': True,\n",
       "   'hidden': False,\n",
       "   'readonly': False,\n",
       "   'default_value': 0.0,\n",
       "   'minimum': '',\n",
       "   'maximum': '',\n",
       "   'label': {'default': 'TrainingData_Ratio', 'en': 'TrainingData_Ratio'}},\n",
       "  'TestData_Size': {'type': 'integer',\n",
       "   'description': 'desc of userattrint',\n",
       "   'placeholder': 'desc of userattrint',\n",
       "   'is_array': False,\n",
       "   'required': True,\n",
       "   'hidden': False,\n",
       "   'readonly': False,\n",
       "   'default_value': 0.0,\n",
       "   'minimum': '',\n",
       "   'maximum': '',\n",
       "   'label': {'default': 'TestData_Size', 'en': 'TestData_Size'}},\n",
       "  'TestData_Ratio': {'type': 'string',\n",
       "   'description': 'Percentage of records in the test data',\n",
       "   'placeholder': 'Percentage of records in the test data',\n",
       "   'is_array': False,\n",
       "   'required': True,\n",
       "   'hidden': False,\n",
       "   'readonly': False,\n",
       "   'default_value': 0.0,\n",
       "   'minimum': '',\n",
       "   'maximum': '',\n",
       "   'label': {'default': 'TestData_Ratio', 'en': 'TestData_Ratio'}},\n",
       "  'Train_Class_Distributions': {'type': 'string',\n",
       "   'description': 'class distributions',\n",
       "   'placeholder': 'class distributions',\n",
       "   'is_array': True,\n",
       "   'required': True,\n",
       "   'hidden': False,\n",
       "   'readonly': False,\n",
       "   'default_value': 0.0,\n",
       "   'minimum': '',\n",
       "   'maximum': '',\n",
       "   'label': {'default': 'Train_Class_Distributions',\n",
       "    'en': 'Train_Class_Distributions'}}},\n",
       " 'localized_metadata_attributes': {'name': {'default': 'Additional details',\n",
       "   'en': 'Additional details'}},\n",
       " 'global_search_searchable': ['TrainingData_Size',\n",
       "  'TrainingData_Ratio',\n",
       "  'TestData_Ratio'],\n",
       " 'attribute_only': False,\n",
       " 'is_column_custom_attribute': False,\n",
       " 'version': 13.0,\n",
       " 'name': 'modelfacts_user',\n",
       " 'scope': 'ACCOUNT',\n",
       " 'allow_decorators': True}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "facts_client.assets.get_facts_definitions(type_name=\"modelfacts_user\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to top](#introduction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"getmodel\"></a> \n",
    "## Set custom facts<a name=\"set\"></a> \n",
    "This will set the values for 4 of the custom fact \"slots\" define above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022/12/12 18:37:56 INFO : Custom fact TrainingData_Size successfully set to new value 5000\n",
      "2022/12/12 18:37:56 INFO : Custom fact TrainingData_Ratio successfully set to new value 70% of total\n",
      "2022/12/12 18:37:56 INFO : Custom fact TestData_Size successfully set to new value 1000\n",
      "2022/12/12 18:37:56 INFO : Custom fact TestData_Ratio successfully set to new value 30% of total\n"
     ]
    }
   ],
   "source": [
    "time.sleep(10)\n",
    "external_model.set_custom_fact(fact_id=\"TrainingData_Size\",value=5000)\n",
    "external_model.set_custom_fact(fact_id=\"TrainingData_Ratio\",value=\"70% of total\")\n",
    "external_model.set_custom_fact(fact_id=\"TestData_Size\",value=1000)\n",
    "external_model.set_custom_fact(fact_id=\"TestData_Ratio\",value=\"30% of total\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Train_Class_Distributions` attribute have `is_array` set to `True` thus allows to set string or list of strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022/12/12 18:37:57 INFO : Custom fact Train_Class_Distributions successfully set to new value ['Risk 30%', 'No Risk 70%']\n"
     ]
    }
   ],
   "source": [
    "external_model.set_custom_fact(fact_id=\"Train_Class_Distributions\",value=[\"Risk 30%\",\"No Risk 70%\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- You can set/update multiple facts at once and current values will be replaced"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022/12/12 18:37:57 INFO : Custom facts ['TestData_Size', 'TrainingData_Size', 'Train_Class_Distributions'] successfully set to values [1500, 5000, ['Risk 35%', 'No Risk 65%']]\n"
     ]
    }
   ],
   "source": [
    "external_model.set_custom_facts({\"TestData_Size\": 1500, \"TrainingData_Size\": 5000, \"Train_Class_Distributions\":[\"Risk 35%\",\"No Risk 65%\"]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to top](#introduction)\n",
    "## Get custom facts<a name=\"get\"></a> \n",
    "- You can get all facts or fact by given name used in facts definitions. You can also visit the UI to view the customs facts set. You can get URL by running `external_model.get_info(verbose=True)[\"url\"]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "[Click here to see the external model and it's custom fact values in the UI](https://dataplatform.cloud.ibm.com/data/catalogs/745c21d7-ab6f-4408-9b3b-5d4760ccee1a/asset/c560f9d4-5ec7-431c-914b-394241f3f39a/asset-preview?context=cpdaas)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ui_url = external_model.get_info(verbose=True)[\"url\"]\n",
    "display(Markdown(\"[Click here to see the external model and it's custom fact values in the UI](\" + ui_url + \")\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5000.0"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "external_model.get_custom_fact_by_id(fact_id=\"TrainingData_Size\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'TrainingData_Size': 5000.0,\n",
       " 'TrainingData_Ratio': '70% of total',\n",
       " 'TestData_Size': 1500.0,\n",
       " 'TestData_Ratio': '30% of total',\n",
       " 'Train_Class_Distributions': ['Risk 35%', 'No Risk 65%']}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "external_model.get_custom_facts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to top](#introduction)\n",
    "## Get all facts<a name=\"get_all\"></a> \n",
    "- You can get all facts of all types (`modelfacts_user` or `model_entry_user`) or filter by any fact type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'TrainingData_Size': 5000.0,\n",
       " 'TrainingData_Ratio': '70% of total',\n",
       " 'TestData_Size': 1500.0,\n",
       " 'TestData_Ratio': '30% of total',\n",
       " 'Train_Class_Distributions': ['Risk 35%', 'No Risk 65%']}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "external_model.get_facts_by_type()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'metadata': {'usage': {'last_updated_at': '2022-12-12T23:37:57Z',\n",
       "   'last_updater_id': 'IBMid-310002F0G1',\n",
       "   'last_update_time': 1670888277825,\n",
       "   'last_accessed_at': '2022-12-12T23:37:57Z',\n",
       "   'last_access_time': 1670888277825,\n",
       "   'last_accessor_id': 'IBMid-310002F0G1',\n",
       "   'access_count': 0},\n",
       "  'rov': {'mode': 0,\n",
       "   'collaborator_ids': {},\n",
       "   'member_roles': {'IBMid-310002F0G1': {'user_iam_id': 'IBMid-310002F0G1',\n",
       "     'roles': ['OWNER']}}},\n",
       "  'name': 'PytorchExternalModel',\n",
       "  'tags': ['development'],\n",
       "  'asset_type': 'model_stub',\n",
       "  'origin_country': 'us',\n",
       "  'rating': 0.0,\n",
       "  'total_ratings': 0,\n",
       "  'catalog_id': '745c21d7-ab6f-4408-9b3b-5d4760ccee1a',\n",
       "  'created': 1670883399779,\n",
       "  'created_at': '2022-12-12T22:16:39Z',\n",
       "  'owner_id': 'IBMid-310002F0G1',\n",
       "  'size': 0,\n",
       "  'version': 2.0,\n",
       "  'asset_state': 'available',\n",
       "  'asset_attributes': ['model_stub',\n",
       "   'modelfacts_system',\n",
       "   'wos_modeldeployment_information',\n",
       "   'modelfacts_user',\n",
       "   'notebook_experiment'],\n",
       "  'asset_id': 'c560f9d4-5ec7-431c-914b-394241f3f39a',\n",
       "  'asset_category': 'USER',\n",
       "  'creator_id': 'IBMid-310002F0G1'},\n",
       " 'entity': {'model_stub': {'model_id': 'ddefabbde572cbfe759f54c61106718d',\n",
       "   'schemas': {'input': [{'fields': [{'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'CheckingStatus',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'modeling_role': 'feature'},\n",
       "        'nullable': True,\n",
       "        'name': 'LoanDuration',\n",
       "        'type': 'integer'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'CreditHistory',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'LoanPurpose',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'modeling_role': 'feature'},\n",
       "        'nullable': True,\n",
       "        'name': 'LoanAmount',\n",
       "        'type': 'integer'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'ExistingSavings',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'EmploymentDuration',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'modeling_role': 'feature'},\n",
       "        'nullable': True,\n",
       "        'name': 'InstallmentPercent',\n",
       "        'type': 'integer'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'Sex',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'OthersOnLoan',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'modeling_role': 'feature'},\n",
       "        'nullable': True,\n",
       "        'name': 'CurrentResidenceDuration',\n",
       "        'type': 'integer'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'OwnsProperty',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'modeling_role': 'feature'},\n",
       "        'nullable': True,\n",
       "        'name': 'Age',\n",
       "        'type': 'integer'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'InstallmentPlans',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'Housing',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'modeling_role': 'feature'},\n",
       "        'nullable': True,\n",
       "        'name': 'ExistingCreditsCount',\n",
       "        'type': 'integer'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'Job',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'modeling_role': 'feature'},\n",
       "        'nullable': True,\n",
       "        'name': 'Dependents',\n",
       "        'type': 'integer'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'Telephone',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'ForeignWorker',\n",
       "        'type': 'string'}],\n",
       "      'type': 'struct'}],\n",
       "    'output': [{'fields': [{'metadata': {'modeling_role': 'record-id',\n",
       "         'primary_key': True},\n",
       "        'nullable': False,\n",
       "        'name': 'scoring_id',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'modeling_role': 'record-timestamp'},\n",
       "        'nullable': False,\n",
       "        'name': 'scoring_timestamp',\n",
       "        'type': 'timestamp'},\n",
       "       {'metadata': {},\n",
       "        'nullable': False,\n",
       "        'name': 'deployment_id',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {},\n",
       "        'nullable': True,\n",
       "        'name': 'asset_revision',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'CheckingStatus',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'modeling_role': 'feature'},\n",
       "        'nullable': True,\n",
       "        'name': 'LoanDuration',\n",
       "        'type': 'integer'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'CreditHistory',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'LoanPurpose',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'modeling_role': 'feature'},\n",
       "        'nullable': True,\n",
       "        'name': 'LoanAmount',\n",
       "        'type': 'integer'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'ExistingSavings',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'EmploymentDuration',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'modeling_role': 'feature'},\n",
       "        'nullable': True,\n",
       "        'name': 'InstallmentPercent',\n",
       "        'type': 'integer'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'Sex',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'OthersOnLoan',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'modeling_role': 'feature'},\n",
       "        'nullable': True,\n",
       "        'name': 'CurrentResidenceDuration',\n",
       "        'type': 'integer'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'OwnsProperty',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'modeling_role': 'feature'},\n",
       "        'nullable': True,\n",
       "        'name': 'Age',\n",
       "        'type': 'integer'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'InstallmentPlans',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'Housing',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'modeling_role': 'feature'},\n",
       "        'nullable': True,\n",
       "        'name': 'ExistingCreditsCount',\n",
       "        'type': 'integer'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'Job',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'modeling_role': 'feature'},\n",
       "        'nullable': True,\n",
       "        'name': 'Dependents',\n",
       "        'type': 'integer'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'Telephone',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'measure': 'discrete',\n",
       "         'modeling_role': 'feature',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'ForeignWorker',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'modeling_role': 'prediction',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'Scored Labels',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'modeling_role': 'probability'},\n",
       "        'nullable': True,\n",
       "        'name': 'Scored Probabilities',\n",
       "        'type': {'containsNull': True,\n",
       "         'type': 'array',\n",
       "         'elementType': 'double'}},\n",
       "       {'metadata': {'modeling_role': 'prediction-probability'},\n",
       "        'nullable': True,\n",
       "        'name': 'prediction_probability',\n",
       "        'type': 'double'},\n",
       "       {'metadata': {'modeling_role': 'debiased-prediction',\n",
       "         'columnInfo': {'columnLength': 64.0}},\n",
       "        'nullable': True,\n",
       "        'name': 'debiased_prediction',\n",
       "        'type': 'string'},\n",
       "       {'metadata': {'modeling_role': 'debiased-probability'},\n",
       "        'nullable': True,\n",
       "        'name': 'debiased_probability',\n",
       "        'type': {'containsNull': True,\n",
       "         'type': 'array',\n",
       "         'elementType': 'double'}}],\n",
       "      'type': 'struct'}]},\n",
       "   'training_data_references': {'schema': {'fields': []}},\n",
       "   'deployment_details': [{'id': 'fb469d7ef430b0baf0cab6c436e70375',\n",
       "     'name': 'german_credit_risk_deployment',\n",
       "     'type': 'online',\n",
       "     'scoring_endpoint': 'test.iscore/score',\n",
       "     'is_deleted': False}]},\n",
       "  'modelfacts_system': {'space_details': {'space_type': 'development',\n",
       "    'is_spacetype_openscale_value': False,\n",
       "    'space_id': ''},\n",
       "   'training_information': {'features': 0.0},\n",
       "   'model_information': {'model_type': '',\n",
       "    'model_description': '',\n",
       "    'input_type': '',\n",
       "    'algorithm': '',\n",
       "    'prediction_type': '',\n",
       "    'software_spec': '',\n",
       "    'software_spec_id': '',\n",
       "    'asset_type': 'model_stub'},\n",
       "   'deployment_details': [{'id': 'fb469d7ef430b0baf0cab6c436e70375',\n",
       "     'deployment_type': 'Online',\n",
       "     'is_deleted': False,\n",
       "     'name': 'german_credit_risk_deployment'}],\n",
       "   'modelentry_information': {'lmid': ''},\n",
       "   'model_identity_information': {'master_id': ''}},\n",
       "  'wos_modeldeployment_information': {},\n",
       "  'modelfacts_user': {'TrainingData_Size': 5000.0,\n",
       "   'TrainingData_Ratio': '70% of total',\n",
       "   'TestData_Size': 1500.0,\n",
       "   'TestData_Ratio': '30% of total',\n",
       "   'Train_Class_Distributions': ['Risk 35%', 'No Risk 65%']},\n",
       "  'notebook_experiment': {'experiment_id': 'ab1913ca062d4a0fb3fb0d0181bbb459',\n",
       "   'name': 'PytorchExperiment',\n",
       "   'runs': [{'run_id': 'e4d25cb087ad4043ae6d92cfbbb51325',\n",
       "     'created_date': '12/12/2022 17:16:34',\n",
       "     'metrics': [{'key': 'ROC training', 'value': '0.8'},\n",
       "      {'key': 'F1 training', 'value': '0.9'}],\n",
       "     'params': [{'key': 'learning_rate', 'value': '0.01'},\n",
       "      {'key': 'epochs', 'value': '10'}],\n",
       "     'tags': [{'key': 'release.version', 'value': '1.1.0'},\n",
       "      {'key': 'facts.user', 'value': 'rounoksalehin'},\n",
       "      {'key': 'facts.source.type', 'value': 'LOCAL'},\n",
       "      {'key': 'engineering', 'value': 'SaaS Platform'},\n",
       "      {'key': 'facts.source.name',\n",
       "       'value': '/Users/rounoksalehin/opt/anaconda3/lib/python3.9/site-packages/ipykernel_launcher.py'}],\n",
       "     'artifacts': []},\n",
       "    {'run_id': '5592724360b341e09d48cf57df639ae6',\n",
       "     'created_date': '12/12/2022 17:18:52',\n",
       "     'metrics': [{'key': 'ROC training', 'value': '0.8'},\n",
       "      {'key': 'F1 training', 'value': '0.9'}],\n",
       "     'params': [{'key': 'learning_rate', 'value': '0.01'},\n",
       "      {'key': 'epochs', 'value': '10'}],\n",
       "     'tags': [{'key': 'release.version', 'value': '1.1.0'},\n",
       "      {'key': 'facts.user', 'value': 'rounoksalehin'},\n",
       "      {'key': 'facts.source.type', 'value': 'LOCAL'},\n",
       "      {'key': 'engineering', 'value': 'SaaS Platform'},\n",
       "      {'key': 'facts.source.name',\n",
       "       'value': '/Users/rounoksalehin/opt/anaconda3/lib/python3.9/site-packages/ipykernel_launcher.py'}],\n",
       "     'artifacts': []},\n",
       "    {'run_id': '6d7d192d759d4fc18b0f9d10807146d7',\n",
       "     'created_date': '12/12/2022 18:32:55',\n",
       "     'metrics': [{'key': 'ROC training', 'value': '0.8'},\n",
       "      {'key': 'F1 training', 'value': '0.9'}],\n",
       "     'params': [{'key': 'learning_rate', 'value': '0.01'},\n",
       "      {'key': 'epochs', 'value': '10'}],\n",
       "     'tags': [{'key': 'release.version', 'value': '1.1.0'},\n",
       "      {'key': 'facts.user', 'value': 'rounoksalehin'},\n",
       "      {'key': 'facts.source.type', 'value': 'LOCAL'},\n",
       "      {'key': 'engineering', 'value': 'SaaS Platform'},\n",
       "      {'key': 'facts.source.name',\n",
       "       'value': '/Users/rounoksalehin/opt/anaconda3/lib/python3.9/site-packages/ipykernel_launcher.py'}],\n",
       "     'artifacts': []},\n",
       "    {'run_id': '2aaaf82f6aec48a08a875dce4fef146d',\n",
       "     'created_date': '12/12/2022 18:35:22',\n",
       "     'metrics': [{'key': 'ROC training', 'value': '0.8'},\n",
       "      {'key': 'F1 training', 'value': '0.9'}],\n",
       "     'params': [{'key': 'learning_rate', 'value': '0.01'},\n",
       "      {'key': 'epochs', 'value': '10'}],\n",
       "     'tags': [{'key': 'release.version', 'value': '1.1.0'},\n",
       "      {'key': 'facts.user', 'value': 'rounoksalehin'},\n",
       "      {'key': 'facts.source.type', 'value': 'LOCAL'},\n",
       "      {'key': 'engineering', 'value': 'SaaS Platform'},\n",
       "      {'key': 'facts.source.name',\n",
       "       'value': '/Users/rounoksalehin/opt/anaconda3/lib/python3.9/site-packages/ipykernel_launcher.py'}],\n",
       "     'artifacts': []},\n",
       "    {'run_id': 'c2089fa111ca4149aa4f67dabd59c028',\n",
       "     'created_date': '12/12/2022 15:06:21',\n",
       "     'metrics': [{'key': 'ROC training', 'value': '0.8'},\n",
       "      {'key': 'F1 training', 'value': '0.9'}],\n",
       "     'params': [{'key': 'learning_rate', 'value': '0.01'},\n",
       "      {'key': 'epochs', 'value': '10'}],\n",
       "     'tags': [{'key': 'release.version', 'value': '1.1.0'},\n",
       "      {'key': 'facts.user', 'value': 'rounoksalehin'},\n",
       "      {'key': 'facts.source.type', 'value': 'LOCAL'},\n",
       "      {'key': 'engineering', 'value': 'SaaS Platform'},\n",
       "      {'key': 'facts.source.name',\n",
       "       'value': '/Users/rounoksalehin/opt/anaconda3/lib/python3.9/site-packages/ipykernel_launcher.py'}],\n",
       "     'artifacts': []}]}},\n",
       " 'href': 'https://api.dataplatform.cloud.ibm.com/v2/assets/c560f9d4-5ec7-431c-914b-394241f3f39a?catalog_id=745c21d7-ab6f-4408-9b3b-5d4760ccee1a'}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "external_model.get_all_facts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Remove custom facts<a name=\"remove\"></a> \n",
    "\n",
    "You can remove one or multiple facts by asscociated fact ids. This does not delete the attribute as a whole rather removes the associated value(s)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022/12/12 18:38:00 INFO :  Value of Fact id TrainingData_Size removed successfully\n"
     ]
    }
   ],
   "source": [
    "external_model.remove_custom_fact(fact_id=\"TrainingData_Size\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022/12/12 18:38:00 INFO : Values of Fact ids ['TrainingData_Ratio', 'TestData_Size', 'TestData_Ratio', 'Train_Class_Distributions'] removed successfully\n"
     ]
    }
   ],
   "source": [
    "external_model.remove_custom_facts(fact_ids=[\"TrainingData_Ratio\",\"TestData_Size\",\"TestData_Ratio\",\"Train_Class_Distributions\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "[Click here to see the external model and it's custom fact values after removing ids in the UI](https://dataplatform.cloud.ibm.com/data/catalogs/745c21d7-ab6f-4408-9b3b-5d4760ccee1a/asset/c560f9d4-5ec7-431c-914b-394241f3f39a/asset-preview?context=cpdaas)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ui_url = external_model.get_info(verbose=True)[\"url\"]\n",
    "display(Markdown(\"[Click here to see the external model and it's custom fact values after removing ids in the UI](\" + ui_url + \")\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to top](#introduction)\n",
    "## Add model to model usecase<a name=\"add_mu\"></a> \n",
    "\n",
    "- Model use case helps to track model asset lifecycle across different environments like development,pre-production and production.\n",
    "- You can link a model to existing model use case by providing `model_usecase_id` or create a new use case and link to that by providing `model_usecase_name` and `model_usecase_desc`.\n",
    "- You can link to existing model use case by invoking `external_model.add_tracking_model_usecase(model_usecase_id=\"<model usecase id>\")`\n",
    "- If `model_usecase_catalog_id` is not given, it uses the default available platform asset catalog id. You need to have `EDITOR` access to platform asset catalog.\n",
    "- When you open the model use case in model inventory, `model_usecase_id` refers to value after `asset/{ usecase id}` in browser URL.\n",
    "- You can also invoke `get_tracking_model_usecase()` to return currently tracked model use case and get model use case details by calling `get_tracking_model_usecase().get_info()`\n",
    "- If using Cloud pak for data, make sure Openpages integration is disabled in your platform (create catalog permission needed). You can check it from UI `(Model inventory -> Manage)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022/12/12 18:38:01 INFO : Initiate linking model to new model use case......\n",
      "2022/12/12 18:38:14 INFO : Successfully finished linking Model c560f9d4-5ec7-431c-914b-394241f3f39a to model use case\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'model_entry_catalog_id': '745c21d7-ab6f-4408-9b3b-5d4760ccee1a',\n",
       " 'model_entry_id': '46c17518-59e2-4fa7-8ef3-2b7d86158f93',\n",
       " 'model_entry_name': 'ModelUsecaseDraft',\n",
       " 'model_entry_status': 'draft'}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# creating new model usecase and link to that\n",
    "external_model.add_tracking_model_usecase(model_usecase_name=\"ModelUsecaseDraft\",model_usecase_desc=\"Draft model usecase for testing\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to top](#introduction)\n",
    "## Create custom facts definitions for Model usecase facts type\n",
    "<a name=\"customfactsdefmu\"></a> \n",
    "- As you set facts definitions on model asset, you can also set it to model usecase level which is saved in catalog.\n",
    "- You can change the header title (default is `Additional details`) by providing `section_name`.\n",
    "- Facts type for model usecase is `model_entry_user`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Asset_type_definition_me.csv'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!rm Asset_type_definition_me.csv\n",
    "wget.download(\"https://raw.githubusercontent.com/IBM/ai-governance-factsheet-samples/main/Assets/data/Asset_type_definition_me.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022/12/12 18:38:15 INFO : Creating definitions for type model_entry_user\n",
      "2022/12/12 18:38:17 INFO : Custom facts definitions updated Successfully\n"
     ]
    }
   ],
   "source": [
    "facts_client.assets.create_custom_facts_definitions(csvFilePath=\"Asset_type_definition_me.csv\",section_name=\"Custom Model Details Title\", type_name=\"model_entry_user\")\n",
    "time.sleep(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>type</th>\n",
       "      <th>description</th>\n",
       "      <th>placeholder</th>\n",
       "      <th>is_array</th>\n",
       "      <th>required</th>\n",
       "      <th>hidden</th>\n",
       "      <th>readonly</th>\n",
       "      <th>default_value</th>\n",
       "      <th>label</th>\n",
       "      <th>minimum</th>\n",
       "      <th>maximum</th>\n",
       "      <th>min_length</th>\n",
       "      <th>max_length</th>\n",
       "      <th>is_searchable</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>model_scope</td>\n",
       "      <td>string</td>\n",
       "      <td>Model Scope</td>\n",
       "      <td>Model Scope</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Model Scope</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>1000</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>model_purpose</td>\n",
       "      <td>string</td>\n",
       "      <td>Model Purpose</td>\n",
       "      <td>Model Purpose</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Model Purpose</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5</td>\n",
       "      <td>1000</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>model_use</td>\n",
       "      <td>string</td>\n",
       "      <td>Model Use</td>\n",
       "      <td>Model Use</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Model Use</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5</td>\n",
       "      <td>500</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>model_sponsor</td>\n",
       "      <td>string</td>\n",
       "      <td>Model Sponsor</td>\n",
       "      <td>Model Sponsor</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Model Sponsor</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5</td>\n",
       "      <td>200</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            name    type    description    placeholder  is_array  required  \\\n",
       "0    model_scope  string    Model Scope    Model Scope     False      True   \n",
       "1  model_purpose  string  Model Purpose  Model Purpose     False      True   \n",
       "2      model_use  string      Model Use      Model Use      True      True   \n",
       "3  model_sponsor  string  Model Sponsor  Model Sponsor     False      True   \n",
       "\n",
       "   hidden  readonly  default_value          label  minimum  maximum  \\\n",
       "0   False     False            NaN    Model Scope      NaN      NaN   \n",
       "1   False     False            NaN  Model Purpose      NaN      NaN   \n",
       "2   False     False            NaN      Model Use      NaN      NaN   \n",
       "3   False     False            NaN  Model Sponsor      NaN      NaN   \n",
       "\n",
       "   min_length  max_length  is_searchable  \n",
       "0           2        1000           True  \n",
       "1           5        1000           True  \n",
       "2           5         500           True  \n",
       "3           5         200           True  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv('Asset_type_definition_me.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get custom facts definitions<a name=\"getcustomfactsdefmu\"></a> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'description': 'The model fact user asset type to capture user defined attributes.',\n",
       " 'fields': [{'key': 'model_scope',\n",
       "   'type': 'string',\n",
       "   'facet': False,\n",
       "   'is_array': False,\n",
       "   'search_path': 'model_scope',\n",
       "   'is_searchable_across_types': False},\n",
       "  {'key': 'model_purpose',\n",
       "   'type': 'string',\n",
       "   'facet': False,\n",
       "   'is_array': False,\n",
       "   'search_path': 'model_purpose',\n",
       "   'is_searchable_across_types': False},\n",
       "  {'key': 'model_use',\n",
       "   'type': 'string',\n",
       "   'facet': False,\n",
       "   'is_array': True,\n",
       "   'search_path': 'model_use',\n",
       "   'is_searchable_across_types': False},\n",
       "  {'key': 'model_sponsor',\n",
       "   'type': 'string',\n",
       "   'facet': False,\n",
       "   'is_array': False,\n",
       "   'search_path': 'model_sponsor',\n",
       "   'is_searchable_across_types': False}],\n",
       " 'relationships': [],\n",
       " 'properties': {'model_scope': {'type': 'string',\n",
       "   'description': 'Model Scope',\n",
       "   'placeholder': 'Model Scope',\n",
       "   'is_array': False,\n",
       "   'required': True,\n",
       "   'hidden': False,\n",
       "   'readonly': False,\n",
       "   'default_value': '',\n",
       "   'minimum': '',\n",
       "   'maximum': '',\n",
       "   'min_length': 2,\n",
       "   'max_length': 1000,\n",
       "   'label': {'default': 'Model Scope', 'en': 'Model Scope'}},\n",
       "  'model_purpose': {'type': 'string',\n",
       "   'description': 'Model Purpose',\n",
       "   'placeholder': 'Model Purpose',\n",
       "   'is_array': False,\n",
       "   'required': True,\n",
       "   'hidden': False,\n",
       "   'readonly': False,\n",
       "   'default_value': '',\n",
       "   'minimum': '',\n",
       "   'maximum': '',\n",
       "   'min_length': 5,\n",
       "   'max_length': 1000,\n",
       "   'label': {'default': 'Model Purpose', 'en': 'Model Purpose'}},\n",
       "  'model_use': {'type': 'string',\n",
       "   'description': 'Model Use',\n",
       "   'placeholder': 'Model Use',\n",
       "   'is_array': True,\n",
       "   'required': True,\n",
       "   'hidden': False,\n",
       "   'readonly': False,\n",
       "   'default_value': '',\n",
       "   'minimum': '',\n",
       "   'maximum': '',\n",
       "   'min_length': 5,\n",
       "   'max_length': 500,\n",
       "   'label': {'default': 'Model Use', 'en': 'Model Use'}},\n",
       "  'model_sponsor': {'type': 'string',\n",
       "   'description': 'Model Sponsor',\n",
       "   'placeholder': 'Model Sponsor',\n",
       "   'is_array': False,\n",
       "   'required': True,\n",
       "   'hidden': False,\n",
       "   'readonly': False,\n",
       "   'default_value': '',\n",
       "   'minimum': '',\n",
       "   'maximum': '',\n",
       "   'min_length': 5,\n",
       "   'max_length': 200,\n",
       "   'label': {'default': 'Model Sponsor', 'en': 'Model Sponsor'}}},\n",
       " 'localized_metadata_attributes': {'name': {'default': 'Custom Model Details Title',\n",
       "   'en': 'Custom Model Details Title'}},\n",
       " 'decorates': [{'asset_type_name': 'model_entry'}],\n",
       " 'global_search_searchable': ['model_scope',\n",
       "  'model_purpose',\n",
       "  'model_use',\n",
       "  'model_sponsor'],\n",
       " 'attribute_only': False,\n",
       " 'is_column_custom_attribute': False,\n",
       " 'version': 12.0,\n",
       " 'name': 'model_entry_user',\n",
       " 'scope': 'ACCOUNT',\n",
       " 'allow_decorators': True}"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "facts_client.assets.get_facts_definitions(type_name=\"model_entry_user\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to top](#introduction)\n",
    "## Get model usecase<a name=\"get_mu\"></a> \n",
    "- You can get to model usecase [helpers](https://ibm-aigov-facts-client.mybluemix.net/#model-use-case-helpers) in two ways.\n",
    "    - Using `facts_client.assets.get_model_usecase()` utility returns a model usecase object. It requires model_usecase_id and catalaog id (Optional)\n",
    "    <br>\n",
    "    \n",
    "    ```\n",
    "    model_usecase=facts_client.assets.get_model_usecase(model_usecase_id=model_usecase.get_info()[\"model_usecase_id\"])\n",
    "    model_usecase=facts_client.assets.get_model_usecase(model_usecase_id=model_usecase.get_info()[model_usecase_id],catalog_id=model_usecase.get_info()[\"catalog_id\"])\n",
    "    ```\n",
    "    \n",
    "    <br>\n",
    "\n",
    "    - Model usecase id and usecase catalog id can be fetched from UI. When you open the model usecase in model inventory, usecase id refers to value after `asset/{ usecase id}` in browser URL. Catalog id refers refers to value after `catalogs/{catalog id}` in browser URL.\n",
    "    - If getting current model usecase, it returns a model usecase object allows to use all usecase [helpers](https://ibm-aigov-facts-client.mybluemix.net/#ibm_aigov_facts_client.factsheet.asset_utils_model.ModelAssetUtilities.get_tracking_model_usecase).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'model_usecase_id': '46c17518-59e2-4fa7-8ef3-2b7d86158f93',\n",
       " 'container_type': 'catalog',\n",
       " 'catalog_id': '745c21d7-ab6f-4408-9b3b-5d4760ccee1a',\n",
       " 'facts_type': 'model_entry_user'}"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_usecase=external_model.get_tracking_model_usecase()\n",
    "model_usecase.get_info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to top](#introduction)\n",
    "## Set custom facts for model usecase<a name=\"set_mu_custom\"></a> \n",
    "\n",
    "- Custom facts related to model usecase like `Model usecase`, `Model Purpose` etc. can be set here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022/12/12 18:38:24 INFO : Custom fact model_scope successfully set to new value Internal systems\n",
      "2022/12/12 18:38:24 INFO : Custom fact model_purpose successfully set to new value Predicting risk factor when giving loans\n",
      "2022/12/12 18:38:25 INFO : Custom fact model_sponsor successfully set to new value IBM\n"
     ]
    }
   ],
   "source": [
    "model_usecase.set_custom_fact(fact_id=\"model_scope\",value=\"Internal systems\")\n",
    "model_usecase.set_custom_fact(fact_id=\"model_purpose\",value=\"Predicting risk factor when giving loans\")\n",
    "model_usecase.set_custom_fact(fact_id=\"model_sponsor\",value=\"IBM\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022/12/12 18:38:25 INFO : Custom fact model_use successfully set to new value ['Determining risk factors', 'Client debt collections']\n"
     ]
    }
   ],
   "source": [
    "model_usecase.set_custom_fact(fact_id=\"model_use\",value=[\"Determining risk factors\",\"Client debt collections\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to top](#introduction)\n",
    "## Get custom facts for model usecase<a name=\"get_mu_custom\"></a> \n",
    "\n",
    "- You can get all facts or fact by given name used in facts definitions. You can also visit the UI to view the customs facts set. You can get URL by running `model_usecase.get_info(verbose=True)[\"url\"]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "[Click here to see the external model use case and it's custom facts values in the UI](https://dataplatform.cloud.ibm.com/data/catalogs/745c21d7-ab6f-4408-9b3b-5d4760ccee1a/asset/46c17518-59e2-4fa7-8ef3-2b7d86158f93?context=cpdaas)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ui_url = model_usecase.get_info(verbose=True)[\"url\"]\n",
    "display(Markdown(\"[Click here to see the external model use case and it's custom facts values in the UI](\" + ui_url + \")\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Internal systems'"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_usecase.get_custom_fact_by_id(fact_id=\"model_scope\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'model_scope': 'Internal systems',\n",
       " 'model_purpose': 'Predicting risk factor when giving loans',\n",
       " 'model_sponsor': 'IBM',\n",
       " 'model_use': ['Determining risk factors', 'Client debt collections']}"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_usecase.get_custom_facts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to top](#introduction)\n",
    "## Get all facts<a name=\"get_all_mu\"></a> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'metadata': {'usage': {'last_updated_at': '2022-12-12T23:38:25Z',\n",
       "   'last_updater_id': 'IBMid-310002F0G1',\n",
       "   'last_update_time': 1670888305652,\n",
       "   'last_accessed_at': '2022-12-12T23:38:25Z',\n",
       "   'last_access_time': 1670888305652,\n",
       "   'last_accessor_id': 'IBMid-310002F0G1',\n",
       "   'access_count': 0},\n",
       "  'rov': {'mode': 0,\n",
       "   'collaborator_ids': {},\n",
       "   'member_roles': {'IBMid-310002F0G1': {'user_iam_id': 'IBMid-310002F0G1',\n",
       "     'roles': ['OWNER']}}},\n",
       "  'name': 'ModelUsecaseDraft',\n",
       "  'description': 'Draft model usecase for testing',\n",
       "  'asset_type': 'model_entry',\n",
       "  'origin_country': 'us',\n",
       "  'rating': 0.0,\n",
       "  'total_ratings': 0,\n",
       "  'catalog_id': '745c21d7-ab6f-4408-9b3b-5d4760ccee1a',\n",
       "  'created': 1670888282292,\n",
       "  'created_at': '2022-12-12T23:38:02Z',\n",
       "  'owner_id': 'IBMid-310002F0G1',\n",
       "  'size': 0,\n",
       "  'version': 2.0,\n",
       "  'asset_state': 'available',\n",
       "  'asset_attributes': ['model_entry', 'modelfacts_global', 'model_entry_user'],\n",
       "  'asset_id': '46c17518-59e2-4fa7-8ef3-2b7d86158f93',\n",
       "  'asset_category': 'USER',\n",
       "  'creator_id': 'IBMid-310002F0G1'},\n",
       " 'entity': {'model_entry': {'lmid': '745c21d7-ab6f-4408-9b3b-5d4760ccee1a:46c17518-59e2-4fa7-8ef3-2b7d86158f93',\n",
       "   'status': 'draft'},\n",
       "  'modelfacts_global': {'physical_models': [{'id': 'c560f9d4-5ec7-431c-914b-394241f3f39a',\n",
       "     'name': 'PytorchExternalModel',\n",
       "     'type': 'external_model',\n",
       "     'container_id': '745c21d7-ab6f-4408-9b3b-5d4760ccee1a',\n",
       "     'container_name': 'Development space',\n",
       "     'container_type': 'catalog',\n",
       "     'is_deleted': False,\n",
       "     'master_id': 'ddefabbde572cbfe759f54c61106718d',\n",
       "     'source_asset_id': '',\n",
       "     'algorithm': '',\n",
       "     'deployment_space_type': 'development',\n",
       "     'deployments': [{'id': 'fb469d7ef430b0baf0cab6c436e70375',\n",
       "       'status': 'Pending Evaluation',\n",
       "       'name': 'german_credit_risk_deployment',\n",
       "       'model_revision_id': '',\n",
       "       'model_revision_commit_message': '',\n",
       "       'is_deleted': False,\n",
       "       'has_alert': False,\n",
       "       'alerts': []}]}]},\n",
       "  'model_entry_user': {'model_scope': 'Internal systems',\n",
       "   'model_purpose': 'Predicting risk factor when giving loans',\n",
       "   'model_sponsor': 'IBM',\n",
       "   'model_use': ['Determining risk factors', 'Client debt collections']}},\n",
       " 'href': 'https://api.dataplatform.cloud.ibm.com/v2/assets/46c17518-59e2-4fa7-8ef3-2b7d86158f93?catalog_id=745c21d7-ab6f-4408-9b3b-5d4760ccee1a'}"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_usecase.get_all_facts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'model_scope': 'Internal systems',\n",
       " 'model_purpose': 'Predicting risk factor when giving loans',\n",
       " 'model_sponsor': 'IBM',\n",
       " 'model_use': ['Determining risk factors', 'Client debt collections']}"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Returns all model_entry_user facts \n",
    "model_usecase.get_facts_by_type()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to top](#introduction)\n",
    "## Remove custom facts for model usecase<a name=\"remove_mu_custom\"></a> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022/12/12 18:38:27 INFO :  Value of Fact id model_scope removed successfully\n"
     ]
    }
   ],
   "source": [
    "model_usecase.remove_custom_fact(fact_id=\"model_scope\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022/12/12 18:38:28 INFO : Values of Fact ids ['model_purpose', 'model_sponsor', 'model_use'] removed successfully\n"
     ]
    }
   ],
   "source": [
    "model_usecase.remove_custom_facts(fact_ids=[\"model_purpose\",\"model_sponsor\",\"model_use\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to top](#introduction)\n",
    "## Get tracked models for model usecase<a name=\"get_tracked_mu\"></a> \n",
    "- It returns all models tracked under current model usecase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'id': 'c560f9d4-5ec7-431c-914b-394241f3f39a',\n",
       "  'name': 'PytorchExternalModel',\n",
       "  'type': 'external_model',\n",
       "  'container_id': '745c21d7-ab6f-4408-9b3b-5d4760ccee1a',\n",
       "  'container_name': 'Development space',\n",
       "  'container_type': 'catalog',\n",
       "  'is_deleted': False,\n",
       "  'master_id': 'ddefabbde572cbfe759f54c61106718d',\n",
       "  'source_asset_id': '',\n",
       "  'algorithm': '',\n",
       "  'deployment_space_type': 'development',\n",
       "  'deployments': [{'id': 'fb469d7ef430b0baf0cab6c436e70375',\n",
       "    'status': 'Pending Evaluation',\n",
       "    'name': 'german_credit_risk_deployment',\n",
       "    'model_revision_id': '',\n",
       "    'model_revision_commit_message': '',\n",
       "    'is_deleted': False,\n",
       "    'has_alert': False,\n",
       "    'alerts': []}]}]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_usecase.get_tracked_models()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to top](#introduction)\n",
    "## Remove model from model usecase<a name=\"remove_mu\"></a> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022/12/12 18:38:29 INFO : Successfully finished unregistering WKC Model c560f9d4-5ec7-431c-914b-394241f3f39a from model use case.\n"
     ]
    }
   ],
   "source": [
    "external_model.remove_tracking_model_usecase()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to top](#introduction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleanup<a name=\"clean\"></a> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "[Click here to see the created external model details in UI](https://dataplatform.cloud.ibm.com/data/catalogs/745c21d7-ab6f-4408-9b3b-5d4760ccee1a/asset/c560f9d4-5ec7-431c-914b-394241f3f39a/asset-preview?context=cpdaas)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "[Click here to see the created external model use case in UI](https://dataplatform.cloud.ibm.com/data/catalogs/745c21d7-ab6f-4408-9b3b-5d4760ccee1a/asset/46c17518-59e2-4fa7-8ef3-2b7d86158f93?context=cpdaas)"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# remove model usecase and model asset\n",
    "if run_cleanup_at_end:\n",
    "    facts_client.assets.remove_asset(asset_id=model_usecase.get_info()[\"model_usecase_id\"],container_type=model_usecase.get_info()[\"container_type\"],container_id=model_usecase.get_info()[\"catalog_id\"])\n",
    "    facts_client.assets.remove_asset(asset_id=external_model.get_info()[\"asset_id\"],container_type=external_model.get_info()[\"container_type\"],container_id=external_model.get_info()[\"container_id\"])\n",
    "else:\n",
    "    model_ui_url = external_model.get_info(verbose=True)[\"url\"]\n",
    "    display(Markdown(\"[Click here to see the created external model details in UI](\" + model_ui_url + \")\"))\n",
    "    model_usecase_ui_url = model_usecase.get_info(verbose=True)[\"url\"]\n",
    "    display(Markdown(\"[Click here to see the created external model use case in UI](\" + model_usecase_ui_url + \")\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[back to top](#introduction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations. You have successfully finished the end to end sample around custom facts utilities like creating and getting facts definitions for model asset and model usecase, tracking/untracking a model to model usecase and set/get/remove helpers for custom facts for model and model usecase."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2020, 2022 IBM. This notebook and its source code are released under the terms of the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "671fb7aa161d7cd648bbbcad6b003541cd9cf13e8157e186cbe0090b82566204"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}