cpd4.0/notebooks/python_sdk/experiments/autoai/Use AutoAI with Watson Studio project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use AutoAI with Watson Studio project `ibm-watson-machine-learning`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook contains the steps and code to demonstrate support of AutoAI experiments in Watson Machine Learning service inside Watson Studio's projects. It introduces commands for data retrieval, training experiments and scoring.\n",
    "\n",
    "Some familiarity with Python is helpful. This notebook uses Python 3.7.\n",
    "\n",
    "\n",
    "## Learning goals\n",
    "\n",
    "The learning goals of this notebook are:\n",
    "\n",
    "-  Work with Watson Machine Learning experiments to train AutoAI model using Watson Studio project.\n",
    "-  Online, Batch deployment and score the trained model trained.\n",
    "\n",
    "\n",
    "## Contents\n",
    "\n",
    "This notebook contains the following parts:\n",
    "\n",
    "1.\t[Setup](#setup)\n",
    "2.\t[Optimizer definition](#definition)\n",
    "3.\t[Experiment Run](#run)\n",
    "4.\t[Deploy and Score](#scoring)\n",
    "5.  [Clean up](#cleanup)\n",
    "6.\t[Summary and next steps](#summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"setup\"></a>\n",
    "## 1. Set up the environment\n",
    "\n",
    "Before you use the sample code in this notebook, you must perform the following setup tasks:\n",
    "\n",
    "-  Contact with your Cloud Pack for Data administrator and ask him for your account credentials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connection to WML\n",
    "\n",
    "Authenticate the Watson Machine Learning service on IBM Cloud Pack for Data. You need to provide platform `url`, your `username` and `api_key`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "username = 'PASTE YOUR USERNAME HERE'\n",
    "api_key = 'PASTE YOUR API_KEY HERE'\n",
    "url = 'PASTE THE PLATFORM URL HERE'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "wml_credentials = {\n",
    "    \"username\": username,\n",
    "    \"apikey\": api_key,\n",
    "    \"url\": url,\n",
    "    \"instance_id\": 'openshift',\n",
    "    \"version\": '4.0'\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively you can use `username` and `password` to authenticate WML services.\n",
    "\n",
    "```\n",
    "wml_credentials = {\n",
    "    \"username\": ***,\n",
    "    \"password\": ***,\n",
    "    \"url\": ***,\n",
    "    \"instance_id\": 'openshift',\n",
    "    \"version\": '4.0'\n",
    "}\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install and import the `ibm-watson-machine-learning` package\n",
    "**Note:** `ibm-watson-machine-learning` documentation can be found <a href=\"http://ibm-wml-api-pyclient.mybluemix.net/\" target=\"_blank\" rel=\"noopener no referrer\">here</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U ibm-watson-machine-learning | tail -n 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibm_watson_machine_learning import APIClient\n",
    "\n",
    "client = APIClient(wml_credentials)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with spaces\n",
    "\n",
    "First of all, you need to create a space that will be used for your work. If you do not have space already created, you can use `{PLATFORM_URL}/ml-runtime/spaces?context=icp4data` to create one.\n",
    "\n",
    "- Click New Deployment Space\n",
    "- Create an empty space\n",
    "- Go to space `Settings` tab\n",
    "- Copy `space_id` and paste it below\n",
    "\n",
    "**Tip**: You can also use SDK to prepare the space for your work. More information can be found [here](https://github.com/IBM/watson-machine-learning-samples/blob/master/cpd4.0/notebooks/python_sdk/instance-management/Space%20management.ipynb).\n",
    "\n",
    "**Action**: Assign space ID below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "space_id = 'PASTE YOUR SPACE ID HERE'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use the `list` method to print all existing spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.spaces.list(limit=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with projects\n",
    "\n",
    "First of all, you need to create a project that will be used for your work. If you do not have space already created follow bellow steps.\n",
    "\n",
    "- Open IBM Cloud Pak main page\n",
    "- Click all projects\n",
    "- Create an empty project\n",
    "- Copy `project_id` from url and paste it below\n",
    "\n",
    "**Action**: Assign project ID below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_id = 'PASTE YOUR PROJECT ID HERE'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To be able to interact with all resources available in Watson Machine Learning, you need to set the **project** which you will be using."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'SUCCESS'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.set.default_project(project_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"definition\"></a>\n",
    "## 2. Optimizer definition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training data connection\n",
    "\n",
    "Define connection information to training data CSV file. This example uses the German Credit Risk dataset. \n",
    "\n",
    "The dataset can be downloaded from [here](https://github.com/IBM/watson-machine-learning-samples/blob/master/data/credit_risk/german_credit_data_biased_training.csv)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = 'credit_risk_training.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget -q https://raw.githubusercontent.com/IBM/watson-machine-learning-samples/master/cpd4.0/data/credit_risk/german_credit_data_biased_training.csv \\\n",
    "    -O credit_risk_training.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating data asset...\n",
      "SUCCESS\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'metadata': {'project_id': 'c65187e1-d07b-4867-99c1-68d6e5d179c6',\n",
       "  'guid': '048eae91-0379-475c-9684-d544ac063dc5',\n",
       "  'href': '/v2/assets/048eae91-0379-475c-9684-d544ac063dc5?project_id=c65187e1-d07b-4867-99c1-68d6e5d179c6',\n",
       "  'asset_type': 'data_asset',\n",
       "  'created_at': '2021-05-05T14:52:45Z',\n",
       "  'last_updated_at': '2021-05-05T14:52:45Z'},\n",
       " 'entity': {'data_asset': {'mime_type': 'text/csv'}}}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "asset_details = client.data_assets.create('german_credit_data_biased_training', filename)\n",
    "asset_details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'048eae91-0379-475c-9684-d544ac063dc5'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.data_assets.get_id(asset_details)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "from ibm_watson_machine_learning.helpers import DataConnection, AssetLocation\n",
    "\n",
    "\n",
    "credit_risk_conn = DataConnection(data_asset_id=client.data_assets.get_id(asset_details))\n",
    "\n",
    "training_data_reference=[credit_risk_conn]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimizer configuration\n",
    "\n",
    "Provide the input information for AutoAI optimizer:\n",
    "- `name` - experiment name\n",
    "- `prediction_type` - type of the problem\n",
    "- `prediction_column` - target column name\n",
    "- `scoring` - optimization metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "from ibm_watson_machine_learning.experiment import AutoAI\n",
    "\n",
    "experiment = AutoAI(wml_credentials, project_id)\n",
    "\n",
    "pipeline_optimizer = experiment.optimizer(\n",
    "    name='Credit Risk Prediction - AutoAI',\n",
    "    desc='Sample notebook',\n",
    "    prediction_type=AutoAI.PredictionType.BINARY,\n",
    "    prediction_column='Risk',\n",
    "    scoring=AutoAI.Metrics.ROC_AUC_SCORE,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Configuration parameters can be retrieved via `get_params()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'name': 'Credit Risk Prediction - AutoAI',\n",
       " 'desc': 'Sample notebook',\n",
       " 'prediction_type': 'binary',\n",
       " 'prediction_column': 'Risk',\n",
       " 'prediction_columns': None,\n",
       " 'timestamp_column_name': None,\n",
       " 'scoring': 'roc_auc',\n",
       " 'holdout_size': 0.1,\n",
       " 'max_num_daub_ensembles': 2,\n",
       " 't_shirt_size': 'm',\n",
       " 'train_sample_rows_test_size': None,\n",
       " 'include_only_estimators': None,\n",
       " 'backtest_num': None,\n",
       " 'lookback_window': None,\n",
       " 'forecast_window': None,\n",
       " 'backtest_gap_length': None,\n",
       " 'cognito_transform_names': None,\n",
       " 'data_join_graph': False,\n",
       " 'csv_separator': ',',\n",
       " 'excel_sheet': 0,\n",
       " 'encoding': 'utf-8',\n",
       " 'positive_label': None,\n",
       " 'drop_duplicates': True,\n",
       " 'text_processing': None,\n",
       " 'word2vec_feature_number': None,\n",
       " 'run_id': None}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_optimizer.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"run\"></a>\n",
    "## 3. Experiment run\n",
    "\n",
    "Call the `fit()` method to trigger the AutoAI experiment. You can either use interactive mode (synchronous job) or background mode (asychronous job) by specifying `background_model=True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [],
   "source": [
    "run_details = pipeline_optimizer.fit(\n",
    "            training_data_reference=training_data_reference,\n",
    "            background_mode=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use the `get_run_status()` method to monitor AutoAI jobs in background mode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'completed'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_optimizer.get_run_status()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"comparison\"></a>\n",
    "### 3.1 Pipelines comparison\n",
    "\n",
    "You can list trained pipelines and evaluation metrics information in\n",
    "the form of a Pandas DataFrame by calling the `summary()` method. You can\n",
    "use the DataFrame to compare all discovered pipelines and select the one\n",
    "you like for further testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "pycharm": {
     "is_executing": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Enhancements</th>\n",
       "      <th>Estimator</th>\n",
       "      <th>training_roc_auc_(optimized)</th>\n",
       "      <th>holdout_precision</th>\n",
       "      <th>training_average_precision</th>\n",
       "      <th>holdout_average_precision</th>\n",
       "      <th>training_log_loss</th>\n",
       "      <th>holdout_recall</th>\n",
       "      <th>training_precision</th>\n",
       "      <th>holdout_accuracy</th>\n",
       "      <th>holdout_balanced_accuracy</th>\n",
       "      <th>training_recall</th>\n",
       "      <th>holdout_f1</th>\n",
       "      <th>holdout_log_loss</th>\n",
       "      <th>training_accuracy</th>\n",
       "      <th>holdout_roc_auc</th>\n",
       "      <th>training_balanced_accuracy</th>\n",
       "      <th>training_f1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pipeline Name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Pipeline_3</th>\n",
       "      <td>HPO, FE, Text_FE</td>\n",
       "      <td>XGBClassifier</td>\n",
       "      <td>0.850961</td>\n",
       "      <td>0.720000</td>\n",
       "      <td>0.776621</td>\n",
       "      <td>0.758251</td>\n",
       "      <td>0.438131</td>\n",
       "      <td>0.646707</td>\n",
       "      <td>0.727062</td>\n",
       "      <td>0.797595</td>\n",
       "      <td>0.760100</td>\n",
       "      <td>0.609448</td>\n",
       "      <td>0.681388</td>\n",
       "      <td>0.452604</td>\n",
       "      <td>0.791881</td>\n",
       "      <td>0.838450</td>\n",
       "      <td>0.746672</td>\n",
       "      <td>0.662355</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pipeline_4</th>\n",
       "      <td>HPO, FE, Text_FE</td>\n",
       "      <td>XGBClassifier</td>\n",
       "      <td>0.850961</td>\n",
       "      <td>0.720000</td>\n",
       "      <td>0.776621</td>\n",
       "      <td>0.758251</td>\n",
       "      <td>0.438131</td>\n",
       "      <td>0.646707</td>\n",
       "      <td>0.727062</td>\n",
       "      <td>0.797595</td>\n",
       "      <td>0.760100</td>\n",
       "      <td>0.609448</td>\n",
       "      <td>0.681388</td>\n",
       "      <td>0.452604</td>\n",
       "      <td>0.791881</td>\n",
       "      <td>0.838450</td>\n",
       "      <td>0.746672</td>\n",
       "      <td>0.662355</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pipeline_2</th>\n",
       "      <td>HPO, Text_FE</td>\n",
       "      <td>XGBClassifier</td>\n",
       "      <td>0.850237</td>\n",
       "      <td>0.718121</td>\n",
       "      <td>0.776149</td>\n",
       "      <td>0.771116</td>\n",
       "      <td>0.438916</td>\n",
       "      <td>0.640719</td>\n",
       "      <td>0.728629</td>\n",
       "      <td>0.795591</td>\n",
       "      <td>0.757106</td>\n",
       "      <td>0.615436</td>\n",
       "      <td>0.677215</td>\n",
       "      <td>0.448791</td>\n",
       "      <td>0.793890</td>\n",
       "      <td>0.843770</td>\n",
       "      <td>0.749667</td>\n",
       "      <td>0.666840</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pipeline_1</th>\n",
       "      <td>Text_FE</td>\n",
       "      <td>XGBClassifier</td>\n",
       "      <td>0.849105</td>\n",
       "      <td>0.711111</td>\n",
       "      <td>0.778861</td>\n",
       "      <td>0.748337</td>\n",
       "      <td>0.438104</td>\n",
       "      <td>0.574850</td>\n",
       "      <td>0.748306</td>\n",
       "      <td>0.779559</td>\n",
       "      <td>0.728690</td>\n",
       "      <td>0.601464</td>\n",
       "      <td>0.635762</td>\n",
       "      <td>0.467235</td>\n",
       "      <td>0.797903</td>\n",
       "      <td>0.829071</td>\n",
       "      <td>0.749222</td>\n",
       "      <td>0.666275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pipeline_7</th>\n",
       "      <td>HPO, FE, Text_FE</td>\n",
       "      <td>LogisticRegression</td>\n",
       "      <td>0.841439</td>\n",
       "      <td>0.562791</td>\n",
       "      <td>0.753225</td>\n",
       "      <td>0.709092</td>\n",
       "      <td>0.489643</td>\n",
       "      <td>0.724551</td>\n",
       "      <td>0.613763</td>\n",
       "      <td>0.719439</td>\n",
       "      <td>0.720709</td>\n",
       "      <td>0.731870</td>\n",
       "      <td>0.633508</td>\n",
       "      <td>0.519397</td>\n",
       "      <td>0.754630</td>\n",
       "      <td>0.818321</td>\n",
       "      <td>0.748990</td>\n",
       "      <td>0.666909</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pipeline_8</th>\n",
       "      <td>HPO, FE, Text_FE</td>\n",
       "      <td>LogisticRegression</td>\n",
       "      <td>0.841439</td>\n",
       "      <td>0.562791</td>\n",
       "      <td>0.753225</td>\n",
       "      <td>0.709092</td>\n",
       "      <td>0.489643</td>\n",
       "      <td>0.724551</td>\n",
       "      <td>0.613763</td>\n",
       "      <td>0.719439</td>\n",
       "      <td>0.720709</td>\n",
       "      <td>0.731870</td>\n",
       "      <td>0.633508</td>\n",
       "      <td>0.519397</td>\n",
       "      <td>0.754630</td>\n",
       "      <td>0.818321</td>\n",
       "      <td>0.748990</td>\n",
       "      <td>0.666909</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pipeline_5</th>\n",
       "      <td>Text_FE</td>\n",
       "      <td>LogisticRegression</td>\n",
       "      <td>0.835550</td>\n",
       "      <td>0.571429</td>\n",
       "      <td>0.746420</td>\n",
       "      <td>0.707113</td>\n",
       "      <td>0.500038</td>\n",
       "      <td>0.742515</td>\n",
       "      <td>0.596948</td>\n",
       "      <td>0.727455</td>\n",
       "      <td>0.731197</td>\n",
       "      <td>0.746507</td>\n",
       "      <td>0.645833</td>\n",
       "      <td>0.522030</td>\n",
       "      <td>0.745261</td>\n",
       "      <td>0.816427</td>\n",
       "      <td>0.745570</td>\n",
       "      <td>0.662839</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pipeline_6</th>\n",
       "      <td>HPO, Text_FE</td>\n",
       "      <td>LogisticRegression</td>\n",
       "      <td>0.835550</td>\n",
       "      <td>0.571429</td>\n",
       "      <td>0.746420</td>\n",
       "      <td>0.707113</td>\n",
       "      <td>0.500038</td>\n",
       "      <td>0.742515</td>\n",
       "      <td>0.596948</td>\n",
       "      <td>0.727455</td>\n",
       "      <td>0.731197</td>\n",
       "      <td>0.746507</td>\n",
       "      <td>0.645833</td>\n",
       "      <td>0.522030</td>\n",
       "      <td>0.745261</td>\n",
       "      <td>0.816427</td>\n",
       "      <td>0.745570</td>\n",
       "      <td>0.662839</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Enhancements           Estimator  \\\n",
       "Pipeline Name                                         \n",
       "Pipeline_3     HPO, FE, Text_FE       XGBClassifier   \n",
       "Pipeline_4     HPO, FE, Text_FE       XGBClassifier   \n",
       "Pipeline_2         HPO, Text_FE       XGBClassifier   \n",
       "Pipeline_1              Text_FE       XGBClassifier   \n",
       "Pipeline_7     HPO, FE, Text_FE  LogisticRegression   \n",
       "Pipeline_8     HPO, FE, Text_FE  LogisticRegression   \n",
       "Pipeline_5              Text_FE  LogisticRegression   \n",
       "Pipeline_6         HPO, Text_FE  LogisticRegression   \n",
       "\n",
       "               training_roc_auc_(optimized)  holdout_precision  \\\n",
       "Pipeline Name                                                    \n",
       "Pipeline_3                         0.850961           0.720000   \n",
       "Pipeline_4                         0.850961           0.720000   \n",
       "Pipeline_2                         0.850237           0.718121   \n",
       "Pipeline_1                         0.849105           0.711111   \n",
       "Pipeline_7                         0.841439           0.562791   \n",
       "Pipeline_8                         0.841439           0.562791   \n",
       "Pipeline_5                         0.835550           0.571429   \n",
       "Pipeline_6                         0.835550           0.571429   \n",
       "\n",
       "               training_average_precision  holdout_average_precision  \\\n",
       "Pipeline Name                                                          \n",
       "Pipeline_3                       0.776621                   0.758251   \n",
       "Pipeline_4                       0.776621                   0.758251   \n",
       "Pipeline_2                       0.776149                   0.771116   \n",
       "Pipeline_1                       0.778861                   0.748337   \n",
       "Pipeline_7                       0.753225                   0.709092   \n",
       "Pipeline_8                       0.753225                   0.709092   \n",
       "Pipeline_5                       0.746420                   0.707113   \n",
       "Pipeline_6                       0.746420                   0.707113   \n",
       "\n",
       "               training_log_loss  holdout_recall  training_precision  \\\n",
       "Pipeline Name                                                          \n",
       "Pipeline_3              0.438131        0.646707            0.727062   \n",
       "Pipeline_4              0.438131        0.646707            0.727062   \n",
       "Pipeline_2              0.438916        0.640719            0.728629   \n",
       "Pipeline_1              0.438104        0.574850            0.748306   \n",
       "Pipeline_7              0.489643        0.724551            0.613763   \n",
       "Pipeline_8              0.489643        0.724551            0.613763   \n",
       "Pipeline_5              0.500038        0.742515            0.596948   \n",
       "Pipeline_6              0.500038        0.742515            0.596948   \n",
       "\n",
       "               holdout_accuracy  holdout_balanced_accuracy  training_recall  \\\n",
       "Pipeline Name                                                                 \n",
       "Pipeline_3             0.797595                   0.760100         0.609448   \n",
       "Pipeline_4             0.797595                   0.760100         0.609448   \n",
       "Pipeline_2             0.795591                   0.757106         0.615436   \n",
       "Pipeline_1             0.779559                   0.728690         0.601464   \n",
       "Pipeline_7             0.719439                   0.720709         0.731870   \n",
       "Pipeline_8             0.719439                   0.720709         0.731870   \n",
       "Pipeline_5             0.727455                   0.731197         0.746507   \n",
       "Pipeline_6             0.727455                   0.731197         0.746507   \n",
       "\n",
       "               holdout_f1  holdout_log_loss  training_accuracy  \\\n",
       "Pipeline Name                                                    \n",
       "Pipeline_3       0.681388          0.452604           0.791881   \n",
       "Pipeline_4       0.681388          0.452604           0.791881   \n",
       "Pipeline_2       0.677215          0.448791           0.793890   \n",
       "Pipeline_1       0.635762          0.467235           0.797903   \n",
       "Pipeline_7       0.633508          0.519397           0.754630   \n",
       "Pipeline_8       0.633508          0.519397           0.754630   \n",
       "Pipeline_5       0.645833          0.522030           0.745261   \n",
       "Pipeline_6       0.645833          0.522030           0.745261   \n",
       "\n",
       "               holdout_roc_auc  training_balanced_accuracy  training_f1  \n",
       "Pipeline Name                                                            \n",
       "Pipeline_3            0.838450                    0.746672     0.662355  \n",
       "Pipeline_4            0.838450                    0.746672     0.662355  \n",
       "Pipeline_2            0.843770                    0.749667     0.666840  \n",
       "Pipeline_1            0.829071                    0.749222     0.666275  \n",
       "Pipeline_7            0.818321                    0.748990     0.666909  \n",
       "Pipeline_8            0.818321                    0.748990     0.666909  \n",
       "Pipeline_5            0.816427                    0.745570     0.662839  \n",
       "Pipeline_6            0.816427                    0.745570     0.662839  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary = pipeline_optimizer.summary()\n",
    "summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can visualize the scoring metric calculated on a holdout data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "pd.options.plotting.backend = \"plotly\"\n",
    "\n",
    "summary.holdout_roc_auc.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"scoring\"></a>\n",
    "## 4. Deploy and Score\n",
    "\n",
    "In this section you will learn how to deploy and score trained model using project in a specified deployment space as a webservice and batch using WML instance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Webservice deployment creation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preparing an AutoAI Deployment...\n",
      "Published model uid: 4d648b47-e1f6-443c-b061-7cf3a43bf196\n",
      "Deploying model 4d648b47-e1f6-443c-b061-7cf3a43bf196 using V4 client.\n",
      "\n",
      "\n",
      "#######################################################################################\n",
      "\n",
      "Synchronous deployment creation for uid: '4d648b47-e1f6-443c-b061-7cf3a43bf196' started\n",
      "\n",
      "#######################################################################################\n",
      "\n",
      "\n",
      "initializing.\n",
      "ready\n",
      "\n",
      "\n",
      "------------------------------------------------------------------------------------------------\n",
      "Successfully finished deployment creation, deployment_uid='e46839f6-b6b8-406b-8b47-d2aa4974a2b6'\n",
      "------------------------------------------------------------------------------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from ibm_watson_machine_learning.deployment import WebService\n",
    "\n",
    "service = WebService(\n",
    "    source_wml_credentials=wml_credentials,\n",
    "    source_project_id=project_id,\n",
    "    target_wml_credentials=wml_credentials,\n",
    "    target_space_id=space_id\n",
    ")\n",
    "\n",
    "service.create(\n",
    "    experiment_run_id=run_details['metadata']['id'],\n",
    "    model='Pipeline_1', \n",
    "    deployment_name=\"Credit Risk Deployment AutoAI\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deployment object could be printed to show basic information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name: Credit Risk Deployment AutoAI, id: f3649f8d-0076-4bde-a759-08f3e98a3204, scoring_url: https://cpd-wmlautoai-apr27.apps.ocp46wmlautoai2.cp.fyre.ibm.com/ml/v4/deployments/f3649f8d-0076-4bde-a759-08f3e98a3204/predictions, asset_id: a672f3a2-87c1-4413-90c9-3ea75202ab86\n"
     ]
    }
   ],
   "source": [
    "print(service)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To show all available information about the deployment use the `.get_params()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'entity': {'asset': {'id': 'a672f3a2-87c1-4413-90c9-3ea75202ab86'},\n",
       "  'custom': {},\n",
       "  'deployed_asset_type': 'model',\n",
       "  'hardware_spec': {'id': 'c076e82c-b2a7-4d20-9c0f-1f0c2fdf5a24',\n",
       "   'name': 'M',\n",
       "   'num_nodes': 1},\n",
       "  'hybrid_pipeline_hardware_specs': [{'hardware_spec': {'name': 'S',\n",
       "     'num_nodes': 1},\n",
       "    'node_runtime_id': 'auto_ai.kb'}],\n",
       "  'name': 'Credit Risk Deployment AutoAI',\n",
       "  'online': {},\n",
       "  'space_id': 'f91684fb-6a8e-4396-b77c-313bf3adb914',\n",
       "  'status': {'online_url': {'url': 'https://cpd-wmlautoai-apr27.apps.ocp46wmlautoai2.cp.fyre.ibm.com/ml/v4/deployments/f3649f8d-0076-4bde-a759-08f3e98a3204/predictions'},\n",
       "   'state': 'ready'}},\n",
       " 'metadata': {'created_at': '2021-05-05T12:24:54.268Z',\n",
       "  'id': 'f3649f8d-0076-4bde-a759-08f3e98a3204',\n",
       "  'modified_at': '2021-05-05T12:24:54.268Z',\n",
       "  'name': 'Credit Risk Deployment AutoAI',\n",
       "  'owner': '1000330999',\n",
       "  'space_id': 'f91684fb-6a8e-4396-b77c-313bf3adb914'}}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "service.get_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scoring of webservice\n",
    "You can make scoring request by calling `score()` on deployed pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'type': 'data_asset', 'connection': {}, 'location': {'href': '/v2/assets/048eae91-0379-475c-9684-d544ac063dc5?project_id=c65187e1-d07b-4867-99c1-68d6e5d179c6', 'id': '048eae91-0379-475c-9684-d544ac063dc5'}}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline_optimizer.get_data_connections()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = pipeline_optimizer.get_data_connections()[0].read()\n",
    "\n",
    "train_X = train_df.drop(['Risk'], axis=1)\n",
    "train_y = train_df.Risk.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = service.score(payload=train_X.iloc[:10])\n",
    "predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to work with the web service in an external Python application you can retrieve the service object by:\n",
    " - Initialize the service by `service = WebService(wml_credentials)`\n",
    " - Get deployment_id by `service.list()` method\n",
    " - Get webservice object by `service.get('deployment_id')` method\n",
    "\n",
    "After that you can call `service.score()` method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deleting deployment\n",
    "You can delete the existing deployment by calling the `service.delete()` command.\n",
    "To list the existing web services you can use `service.list()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Batch deployment creation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A batch deployment processes input data from a inline data and return predictions in scoring details or processes from data asset and writes the output to a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CheckingStatus</th>\n",
       "      <th>LoanDuration</th>\n",
       "      <th>CreditHistory</th>\n",
       "      <th>LoanPurpose</th>\n",
       "      <th>LoanAmount</th>\n",
       "      <th>ExistingSavings</th>\n",
       "      <th>EmploymentDuration</th>\n",
       "      <th>InstallmentPercent</th>\n",
       "      <th>Sex</th>\n",
       "      <th>OthersOnLoan</th>\n",
       "      <th>CurrentResidenceDuration</th>\n",
       "      <th>OwnsProperty</th>\n",
       "      <th>Age</th>\n",
       "      <th>InstallmentPlans</th>\n",
       "      <th>Housing</th>\n",
       "      <th>ExistingCreditsCount</th>\n",
       "      <th>Job</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>Telephone</th>\n",
       "      <th>ForeignWorker</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0_to_200</td>\n",
       "      <td>31</td>\n",
       "      <td>credits_paid_to_date</td>\n",
       "      <td>other</td>\n",
       "      <td>1889</td>\n",
       "      <td>100_to_500</td>\n",
       "      <td>less_1</td>\n",
       "      <td>3</td>\n",
       "      <td>female</td>\n",
       "      <td>none</td>\n",
       "      <td>3</td>\n",
       "      <td>savings_insurance</td>\n",
       "      <td>32</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>less_0</td>\n",
       "      <td>18</td>\n",
       "      <td>credits_paid_to_date</td>\n",
       "      <td>car_new</td>\n",
       "      <td>462</td>\n",
       "      <td>less_100</td>\n",
       "      <td>1_to_4</td>\n",
       "      <td>2</td>\n",
       "      <td>female</td>\n",
       "      <td>none</td>\n",
       "      <td>2</td>\n",
       "      <td>savings_insurance</td>\n",
       "      <td>37</td>\n",
       "      <td>stores</td>\n",
       "      <td>own</td>\n",
       "      <td>2</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>less_0</td>\n",
       "      <td>15</td>\n",
       "      <td>prior_payments_delayed</td>\n",
       "      <td>furniture</td>\n",
       "      <td>250</td>\n",
       "      <td>less_100</td>\n",
       "      <td>1_to_4</td>\n",
       "      <td>2</td>\n",
       "      <td>male</td>\n",
       "      <td>none</td>\n",
       "      <td>3</td>\n",
       "      <td>real_estate</td>\n",
       "      <td>28</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>2</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0_to_200</td>\n",
       "      <td>28</td>\n",
       "      <td>credits_paid_to_date</td>\n",
       "      <td>retraining</td>\n",
       "      <td>3693</td>\n",
       "      <td>less_100</td>\n",
       "      <td>greater_7</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>none</td>\n",
       "      <td>2</td>\n",
       "      <td>savings_insurance</td>\n",
       "      <td>32</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>no_checking</td>\n",
       "      <td>28</td>\n",
       "      <td>prior_payments_delayed</td>\n",
       "      <td>education</td>\n",
       "      <td>6235</td>\n",
       "      <td>500_to_1000</td>\n",
       "      <td>greater_7</td>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>none</td>\n",
       "      <td>3</td>\n",
       "      <td>unknown</td>\n",
       "      <td>57</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>2</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  CheckingStatus  LoanDuration           CreditHistory LoanPurpose  \\\n",
       "0       0_to_200            31    credits_paid_to_date       other   \n",
       "1         less_0            18    credits_paid_to_date     car_new   \n",
       "2         less_0            15  prior_payments_delayed   furniture   \n",
       "3       0_to_200            28    credits_paid_to_date  retraining   \n",
       "4    no_checking            28  prior_payments_delayed   education   \n",
       "\n",
       "   LoanAmount ExistingSavings EmploymentDuration  InstallmentPercent     Sex  \\\n",
       "0        1889      100_to_500             less_1                   3  female   \n",
       "1         462        less_100             1_to_4                   2  female   \n",
       "2         250        less_100             1_to_4                   2    male   \n",
       "3        3693        less_100          greater_7                   3    male   \n",
       "4        6235     500_to_1000          greater_7                   3    male   \n",
       "\n",
       "  OthersOnLoan  CurrentResidenceDuration       OwnsProperty  Age  \\\n",
       "0         none                         3  savings_insurance   32   \n",
       "1         none                         2  savings_insurance   37   \n",
       "2         none                         3        real_estate   28   \n",
       "3         none                         2  savings_insurance   32   \n",
       "4         none                         3            unknown   57   \n",
       "\n",
       "  InstallmentPlans Housing  ExistingCreditsCount      Job  Dependents  \\\n",
       "0             none     own                     1  skilled           1   \n",
       "1           stores     own                     2  skilled           1   \n",
       "2             none     own                     2  skilled           1   \n",
       "3             none     own                     1  skilled           1   \n",
       "4             none     own                     2  skilled           1   \n",
       "\n",
       "  Telephone ForeignWorker  \n",
       "0      none           yes  \n",
       "1      none           yes  \n",
       "2       yes            no  \n",
       "3      none           yes  \n",
       "4      none           yes  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_payload_df = train_df.drop(['Risk'], axis=1)[:5]\n",
    "batch_payload_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create batch deployment for `Pipeline_2` created in AutoAI experiment with the `run_id`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preparing an AutoAI Deployment...\n",
      "Published model uid: 35a98cac-c3b7-448e-9a02-245400c93a82\n",
      "Deploying model 35a98cac-c3b7-448e-9a02-245400c93a82 using V4 client.\n",
      "\n",
      "\n",
      "#######################################################################################\n",
      "\n",
      "Synchronous deployment creation for uid: '35a98cac-c3b7-448e-9a02-245400c93a82' started\n",
      "\n",
      "#######################################################################################\n",
      "\n",
      "\n",
      "ready.\n",
      "\n",
      "\n",
      "------------------------------------------------------------------------------------------------\n",
      "Successfully finished deployment creation, deployment_uid='756558ad-78f1-4d74-9e6d-555aa9cfff21'\n",
      "------------------------------------------------------------------------------------------------\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from ibm_watson_machine_learning.deployment import Batch\n",
    "\n",
    "service_batch = Batch(\n",
    "    source_wml_credentials=wml_credentials,\n",
    "    source_project_id=project_id,\n",
    "    target_wml_credentials=wml_credentials,\n",
    "    target_space_id=space_id\n",
    ")\n",
    "service_batch.create(\n",
    "            experiment_run_id=run_details['metadata']['id'],\n",
    "            model=\"Pipeline_2\",\n",
    "            deployment_name=\"Credit Risk Batch Deployment AutoAI\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Score batch deployment with inline payload as pandas DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "##########################################################################\n",
      "\n",
      "Synchronous scoring for id: '478b6b5b-a80f-43f5-98f9-63e9da889720' started\n",
      "\n",
      "##########################################################################\n",
      "\n",
      "\n",
      "queued..\n",
      "running\n",
      "completed\n",
      "Scoring job  '478b6b5b-a80f-43f5-98f9-63e9da889720' finished successfully.\n"
     ]
    }
   ],
   "source": [
    "scoring_params = service_batch.run_job(\n",
    "            payload=batch_payload_df,\n",
    "            background_mode=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'fields': ['prediction', 'probability'],\n",
       "  'values': [['No Risk', [0.7743273377418518, 0.225672647356987]],\n",
       "   ['No Risk', [0.8309863209724426, 0.16901369392871857]],\n",
       "   ['No Risk', [0.8892747759819031, 0.11072523891925812]],\n",
       "   ['No Risk', [0.7456686496734619, 0.2543313801288605]],\n",
       "   ['Risk', [0.22297507524490356, 0.7770249247550964]]]}]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scoring_params['entity']['scoring'].get('predictions')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deleting deployment\n",
    "You can delete the existing deployment by calling the `service_batch.delete()` command.\n",
    "To list the existing: \n",
    "- batch services you can use `service_batch.list()`, \n",
    "- scoring jobs you can use `service_batch.list_jobs()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"cleanup\"></a>\n",
    "## 5. Clean up "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to clean up all created assets:\n",
    "- experiments\n",
    "- trainings\n",
    "- pipelines\n",
    "- model definitions\n",
    "- models\n",
    "- functions\n",
    "- deployments\n",
    "\n",
    "please follow up this sample [notebook](https://github.com/IBM/watson-machine-learning-samples/blob/master/cpd4.0/notebooks/python_sdk/instance-management/Machine%20Learning%20artifacts%20management.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"summary\"></a>\n",
    "## 6. Summary and next steps\n",
    "\n",
    " You successfully completed this notebook!.\n",
    " \n",
    " You learned how to use `ibm-watson-machine-learning` to run AutoAI experiments using project. \n",
    " \n",
    " Check out our _[Online Documentation](https://www.ibm.com/cloud/watson-studio/autoai)_ for more samples, tutorials, documentation, how-tos, and blog posts. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authors\n",
    "\n",
    "**Jan Sołtysik**, Intern in Watson Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2020, 2021 IBM. This notebook and its source code are released under the terms of the MIT License."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}