notebooks/classification-credit-card-fraud/auto-ml-classification-credit-card-fraud.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "\n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/classification-credit-card-fraud/auto-ml-classification-credit-card-fraud.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automated Machine Learning\n",
    "_**Classification of credit card fraudulent transactions on remote compute **_\n",
    "\n",
    "## Contents\n",
    "1. [Introduction](#Introduction)\n",
    "1. [Setup](#Setup)\n",
    "1. [Train](#Train)\n",
    "1. [Results](#Results)\n",
    "1. [Test](#Test)\n",
    "1. [Acknowledgements](#Acknowledgements)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "In this example we use the associated credit card dataset to showcase how you can use AutoML for a simple classification problem. The goal is to predict if a credit card transaction is considered a fraudulent charge.\n",
    "\n",
    "This notebook is using remote compute to train the model.\n",
    "\n",
    "If you are using an Azure Machine Learning Compute Instance, you are all set. Otherwise, go through the [configuration](../configuration.ipynb) notebook first if you haven't already to establish your connection to the AzureML Workspace. \n",
    "\n",
    "In this notebook you will learn how to:\n",
    "1. Create an experiment using an existing workspace.\n",
    "2. Configure AutoML using `AutoMLConfig`.\n",
    "3. Train the model using remote compute.\n",
    "4. Explore the results.\n",
    "5. Test the fitted model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "As part of the setup you have already created an Azure ML `Workspace` object. For Automated ML you will need to create an `Experiment` object, which is a named object in a `Workspace` used to run experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "gather": {
     "logged": 1635192346318
    }
   },
   "outputs": [],
   "source": [
    "import logging\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "import azureml.core\n",
    "from azureml.core.experiment import Experiment\n",
    "from azureml.core.workspace import Workspace\n",
    "from azureml.core.dataset import Dataset\n",
    "from azureml.train.automl import AutoMLConfig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This sample notebook may use features that are not available in previous versions of the Azure ML SDK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "gather": {
     "logged": 1635192350225
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This notebook was created using version 1.29 of the Azure ML SDK\n",
      "You are currently using version 1.34.0 of the Azure ML SDK\n"
     ]
    }
   ],
   "source": [
    "print(\"This notebook was created using version 1.29 of the Azure ML SDK\")\n",
    "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "gather": {
     "logged": 1635192431275
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Subscription ID</th>\n",
       "      <td>381b38e9-9840-4719-a5a0-61d9585e1e91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Workspace</th>\n",
       "      <td>cesardl-automl-ncentralus-demo-ws</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Resource Group</th>\n",
       "      <td>cesardl-automl-ncentralus-demo-ws-resgrp</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Location</th>\n",
       "      <td>northcentralus</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Experiment Name</th>\n",
       "      <td>test-set-support-automl-classification-ccard-remote</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                    \n",
       "Subscription ID  381b38e9-9840-4719-a5a0-61d9585e1e91               \n",
       "Workspace        cesardl-automl-ncentralus-demo-ws                  \n",
       "Resource Group   cesardl-automl-ncentralus-demo-ws-resgrp           \n",
       "Location         northcentralus                                     \n",
       "Experiment Name  test-set-support-automl-classification-ccard-remote"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ws = Workspace.from_config()\n",
    "\n",
    "# choose a name for experiment\n",
    "experiment_name = 'test-set-support-automl-classification-ccard-remote'\n",
    "\n",
    "experiment=Experiment(ws, experiment_name)\n",
    "\n",
    "output = {}\n",
    "output['Subscription ID'] = ws.subscription_id\n",
    "output['Workspace'] = ws.name\n",
    "output['Resource Group'] = ws.resource_group\n",
    "output['Location'] = ws.location\n",
    "output['Experiment Name'] = experiment.name\n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "outputDf = pd.DataFrame(data = output, index = [''])\n",
    "outputDf.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Create or Attach existing AmlCompute\n",
    "A compute target is required to execute the Automated ML run. In this tutorial, you create AmlCompute as your training compute resource.\n",
    "#### Creation of AmlCompute takes approximately 5 minutes. \n",
    "If the AmlCompute with that name is already in your workspace this code will skip the creation process.\n",
    "As with other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. Please read [this article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) on the default limits and how to request more quota."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "gather": {
     "logged": 1635192440084
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found existing cluster, use it.\n",
      "Succeeded\n",
      "AmlCompute wait for completion finished\n",
      "\n",
      "Minimum number of nodes requested have been provisioned\n"
     ]
    }
   ],
   "source": [
    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "\n",
    "# Choose a name for your CPU cluster\n",
    "cpu_cluster_name = \"cpu-cluster\"\n",
    "\n",
    "# Verify that cluster does not exist already\n",
    "try:\n",
    "    compute_target = ComputeTarget(workspace=ws, name=cpu_cluster_name)\n",
    "    print('Found existing cluster, use it.')\n",
    "except ComputeTargetException:\n",
    "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_DS12_V2',\n",
    "                                                           max_nodes=6)\n",
    "    compute_target = ComputeTarget.create(ws, cpu_cluster_name, compute_config)\n",
    "\n",
    "compute_target.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Data\n",
    "\n",
    "Load the credit card dataset from a csv file containing both training features and labels. The features are inputs to the model, while the training labels represent the expected output of the model. Next, we'll split the data using random_split and extract the training data for the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "gather": {
     "logged": 1635192456177
    }
   },
   "outputs": [],
   "source": [
    "data = \"https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/creditcard.csv\"\n",
    "dataset = Dataset.Tabular.from_delimited_files(data)\n",
    "training_data, test_data = dataset.random_split(percentage=0.8, seed=223)\n",
    "label_column_name = 'Class'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train\n",
    "\n",
    "Instantiate a AutoMLConfig object. This defines the settings and data used to run the experiment.\n",
    "\n",
    "|Property|Description|\n",
    "|-|-|\n",
    "|**task**|classification or regression|\n",
    "|**primary_metric**|This is the metric that you want to optimize. Classification supports the following primary metrics: <br><i>accuracy</i><br><i>AUC_weighted</i><br><i>average_precision_score_weighted</i><br><i>norm_macro_recall</i><br><i>precision_score_weighted</i>|\n",
    "|**enable_early_stopping**|Stop the run if the metric score is not showing improvement.|\n",
    "|**n_cross_validations**|Number of cross validation splits.|\n",
    "|**training_data**|Input dataset, containing both features and label column.|\n",
    "|**label_column_name**|The name of the label column.|\n",
    "\n",
    "**_You can find more information about primary metrics_** [here](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-configure-auto-train#primary-metric)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_settings = {\n",
    "    \"primary_metric\": 'average_precision_score_weighted',\n",
    "    \"enable_early_stopping\": True,\n",
    "    \"max_concurrent_iterations\": 4, # This is a limit for testing purpose, please increase it as per cluster size\n",
    "    \"iterations\": 10,\n",
    "    \"experiment_timeout_hours\": 0.25, # This is a time limit for testing purposes, remove it for real use cases, this will drastically limit ablity to find the best model possible\n",
    "    \"verbosity\": logging.INFO,\n",
    "}\n",
    "\n",
    "\n",
    "automl_config = AutoMLConfig(task = 'classification',\n",
    "                             compute_target = compute_target,\n",
    "                             training_data = training_data,\n",
    "                             label_column_name = label_column_name,\n",
    "                             \n",
    "                             # Use validation set split\n",
    "                             validation_size=0.2,\n",
    "                             \n",
    "                             # Use test set split (This will be used only for the winer model Test metrics)\n",
    "                             test_size=0.1,\n",
    "                             \n",
    "                             **automl_settings\n",
    "                            )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Call the `submit` method on the experiment object and pass the run configuration. Depending on the data and the number of iterations this can run for a while. Validation errors and current status will be shown when setting `show_output=True` and the execution will be synchronous."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submitting remote run.\n",
      "No run_configuration provided, running on cpu-cluster with default configuration\n",
      "Running on remote compute: cpu-cluster\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table style=\"width:100%\"><tr><th>Experiment</th><th>Id</th><th>Type</th><th>Status</th><th>Details Page</th><th>Docs Page</th></tr><tr><td>test-set-support-automl-classification-ccard-remote</td><td>AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82</td><td>automl</td><td>NotStarted</td><td><a href=\"https://ml.azure.com/runs/AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82?wsid=/subscriptions/381b38e9-9840-4719-a5a0-61d9585e1e91/resourcegroups/cesardl-automl-ncentralus-demo-ws-resgrp/workspaces/cesardl-automl-ncentralus-demo-ws&amp;tid=72f988bf-86f1-41af-91ab-2d7cd011db47\" target=\"_blank\" rel=\"noopener\">Link to Azure Machine Learning studio</a></td><td><a href=\"https://docs.microsoft.com/en-us/python/api/overview/azure/ml/intro?view=azure-ml-py\" target=\"_blank\" rel=\"noopener\">Link to Documentation</a></td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Current status: FeaturesGeneration. Generating features for the dataset.\n",
      "Current status: DatasetBalancing. Performing class balancing sweeping\n",
      "Current status: ModelSelection. Beginning model selection.\n",
      "\n",
      "****************************************************************************************************\n",
      "DATA GUARDRAILS: \n",
      "\n",
      "TYPE:         Class balancing detection\n",
      "STATUS:       ALERTED\n",
      "DESCRIPTION:  To decrease model bias, please cancel the current run and fix balancing problem.\n",
      "              Learn more about imbalanced data: https://aka.ms/AutomatedMLImbalancedData\n",
      "DETAILS:      Imbalanced data can lead to a falsely perceived positive effect of a model's accuracy because the input data has bias towards one class.\n",
      "+---------------------------------+---------------------------------+--------------------------------------+\n",
      "|Size of the smallest class       |Name/Label of the smallest class |Number of samples in the training data|\n",
      "+=================================+=================================+======================================+\n",
      "|275                              |True                             |163951                                |\n",
      "+---------------------------------+---------------------------------+--------------------------------------+\n",
      "\n",
      "****************************************************************************************************\n",
      "\n",
      "TYPE:         Missing feature values imputation\n",
      "STATUS:       PASSED\n",
      "DESCRIPTION:  No feature missing values were detected in the training data.\n",
      "              Learn more about missing value imputation: https://aka.ms/AutomatedMLFeaturization\n",
      "\n",
      "****************************************************************************************************\n",
      "\n",
      "TYPE:         High cardinality feature detection\n",
      "STATUS:       PASSED\n",
      "DESCRIPTION:  Your inputs were analyzed, and no high cardinality features were detected.\n",
      "              Learn more about high cardinality feature handling: https://aka.ms/AutomatedMLFeaturization\n",
      "\n",
      "****************************************************************************************************\n",
      "\n",
      "****************************************************************************************************\n",
      "ITERATION: The iteration being evaluated.\n",
      "PIPELINE: A summary description of the pipeline being evaluated.\n",
      "DURATION: Time taken for the current iteration.\n",
      "METRIC: The result of computing score on the fitted pipeline.\n",
      "BEST: The best observed score thus far.\n",
      "****************************************************************************************************\n",
      "\n",
      " ITERATION   PIPELINE                                       DURATION      METRIC      BEST\n",
      "         2   MaxAbsScaler ExtremeRandomTrees                0:00:35       0.9993    0.9993\n",
      "         3   SparseNormalizer XGBoostClassifier             0:01:13       0.9995    0.9995\n",
      "         0   MaxAbsScaler LightGBM                          0:01:48       0.9994    0.9995\n",
      "         1   MaxAbsScaler XGBoostClassifier                 0:00:55       0.9996    0.9996\n",
      "         6   SparseNormalizer XGBoostClassifier             0:01:29       0.9995    0.9996\n",
      "         7   StandardScalerWrapper LogisticRegression       0:00:33       0.9996    0.9996\n",
      "         4   MaxAbsScaler LightGBM                          0:02:35       0.9991    0.9996\n",
      "         5   MaxAbsScaler LogisticRegression                0:01:54       0.9991    0.9996\n",
      "         8    VotingEnsemble                                0:00:37       0.9997    0.9997\n",
      "         9    StackEnsemble                                 0:01:38       0.9997    0.9997\n"
     ]
    }
   ],
   "source": [
    "remote_run = experiment.submit(automl_config, show_output = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you need to retrieve a run that already started, use the following code\n",
    "#from azureml.train.automl.run import AutoMLRun\n",
    "#remote_run = AutoMLRun(experiment = experiment, run_id = '<replace with your run id>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Widget for Monitoring Runs\n",
    "\n",
    "The widget will first report a \"loading\" status while running the first iteration. After completing the first iteration, an auto-updating graph and table will be shown. The widget will refresh once per minute, so you should see the graph update as child runs complete.\n",
    "\n",
    "**Note:** The widget displays a link at the bottom. Use this link to open a web interface to explore the individual run details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": [
     "widget-rundetails-sample"
    ]
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2e3800b27c524823bc2df933c33aedf7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "_AutoMLWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', 's…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/aml.mini.widget.v1": "{\"status\": \"Completed\", \"workbench_run_details_uri\": \"https://ml.azure.com/runs/AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82?wsid=/subscriptions/381b38e9-9840-4719-a5a0-61d9585e1e91/resourcegroups/cesardl-automl-ncentralus-demo-ws-resgrp/workspaces/cesardl-automl-ncentralus-demo-ws&tid=72f988bf-86f1-41af-91ab-2d7cd011db47\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"run_properties\": {\"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"created_utc\": \"2021-10-25T21:57:21.636387Z\", \"properties\": {\"num_iterations\": \"10\", \"training_type\": \"TrainFull\", \"acquisition_function\": \"EI\", \"primary_metric\": \"average_precision_score_weighted\", \"train_split\": \"0.2\", \"acquisition_parameter\": \"0\", \"num_cross_validation\": null, \"target\": \"cpu-cluster\", \"AMLSettingsJsonString\": \"{\\\"path\\\":null,\\\"name\\\":\\\"test-set-support-automl-classification-ccard-remote\\\",\\\"subscription_id\\\":\\\"381b38e9-9840-4719-a5a0-61d9585e1e91\\\",\\\"resource_group\\\":\\\"cesardl-automl-ncentralus-demo-ws-resgrp\\\",\\\"workspace_name\\\":\\\"cesardl-automl-ncentralus-demo-ws\\\",\\\"region\\\":\\\"northcentralus\\\",\\\"compute_target\\\":\\\"cpu-cluster\\\",\\\"spark_service\\\":null,\\\"azure_service\\\":\\\"remote\\\",\\\"many_models\\\":false,\\\"pipeline_fetch_max_batch_size\\\":1,\\\"enable_batch_run\\\":false,\\\"enable_run_restructure\\\":false,\\\"start_auxiliary_runs_before_parent_complete\\\":true,\\\"enable_code_generation\\\":false,\\\"iterations\\\":10,\\\"primary_metric\\\":\\\"average_precision_score_weighted\\\",\\\"task_type\\\":\\\"classification\\\",\\\"data_script\\\":null,\\\"test_size\\\":0.1,\\\"test_include_predictions_only\\\":false,\\\"validation_size\\\":0.2,\\\"n_cross_validations\\\":null,\\\"y_min\\\":null,\\\"y_max\\\":null,\\\"num_classes\\\":null,\\\"featurization\\\":\\\"auto\\\",\\\"_ignore_package_version_incompatibilities\\\":false,\\\"is_timeseries\\\":false,\\\"max_cores_per_iteration\\\":1,\\\"max_concurrent_iterations\\\":4,\\\"iteration_timeout_minutes\\\":null,\\\"mem_in_mb\\\":null,\\\"enforce_time_on_windows\\\":false,\\\"experiment_timeout_minutes\\\":15,\\\"experiment_exit_score\\\":null,\\\"whitelist_models\\\":null,\\\"blacklist_algos\\\":[\\\"TensorFlowLinearClassifier\\\",\\\"TensorFlowDNN\\\"],\\\"supported_models\\\":[\\\"BernoulliNaiveBayes\\\",\\\"LinearSVM\\\",\\\"RandomForest\\\",\\\"LogisticRegression\\\",\\\"SGD\\\",\\\"XGBoostClassifier\\\",\\\"AveragedPerceptronClassifier\\\",\\\"GradientBoosting\\\",\\\"TensorFlowLinearClassifier\\\",\\\"MultinomialNaiveBayes\\\",\\\"DecisionTree\\\",\\\"LightGBM\\\",\\\"KNN\\\",\\\"SVM\\\",\\\"ExtremeRandomTrees\\\",\\\"TensorFlowDNN\\\"],\\\"private_models\\\":[],\\\"auto_blacklist\\\":true,\\\"blacklist_samples_reached\\\":false,\\\"exclude_nan_labels\\\":true,\\\"verbosity\\\":20,\\\"_debug_log\\\":\\\"azureml_automl.log\\\",\\\"show_warnings\\\":false,\\\"model_explainability\\\":true,\\\"service_url\\\":null,\\\"sdk_url\\\":null,\\\"sdk_packages\\\":null,\\\"enable_onnx_compatible_models\\\":false,\\\"enable_split_onnx_featurizer_estimator_models\\\":false,\\\"vm_type\\\":\\\"STANDARD_D12_V2\\\",\\\"telemetry_verbosity\\\":20,\\\"send_telemetry\\\":true,\\\"enable_dnn\\\":false,\\\"scenario\\\":\\\"SDK-1.13.0\\\",\\\"environment_label\\\":null,\\\"save_mlflow\\\":false,\\\"force_text_dnn\\\":false,\\\"enable_feature_sweeping\\\":true,\\\"enable_early_stopping\\\":true,\\\"early_stopping_n_iters\\\":10,\\\"metrics\\\":null,\\\"enable_metric_confidence\\\":false,\\\"enable_ensembling\\\":true,\\\"enable_stack_ensembling\\\":true,\\\"ensemble_iterations\\\":10,\\\"enable_tf\\\":false,\\\"enable_subsampling\\\":false,\\\"subsample_seed\\\":null,\\\"enable_nimbusml\\\":false,\\\"enable_streaming\\\":false,\\\"force_streaming\\\":false,\\\"track_child_runs\\\":true,\\\"allowed_private_models\\\":[],\\\"label_column_name\\\":\\\"Class\\\",\\\"weight_column_name\\\":null,\\\"cv_split_column_names\\\":null,\\\"positive_label\\\":null,\\\"enable_local_managed\\\":false,\\\"_local_managed_run_id\\\":null,\\\"cost_mode\\\":1,\\\"lag_length\\\":0,\\\"metric_operation\\\":\\\"maximize\\\",\\\"preprocess\\\":true}\", \"DataPrepJsonString\": \"{\\\\\\\"training_data\\\\\\\": {\\\\\\\"datasetId\\\\\\\": \\\\\\\"72710eb4-5208-465d-85dc-6330b6828dcc\\\\\\\"}, \\\\\\\"datasets\\\\\\\": 0}\", \"EnableSubsampling\": \"False\", \"runTemplate\": \"AutoML\", \"azureml.runsource\": \"automl\", \"display_task_type\": \"classification\", \"dependencies_versions\": \"{\\\"azureml-widgets\\\": \\\"1.34.0\\\", \\\"azureml-train\\\": \\\"1.34.0\\\", \\\"azureml-train-restclients-hyperdrive\\\": \\\"1.34.0\\\", \\\"azureml-train-core\\\": \\\"1.34.0\\\", \\\"azureml-train-automl\\\": \\\"1.34.0\\\", \\\"azureml-train-automl-runtime\\\": \\\"1.34.0.post1\\\", \\\"azureml-train-automl-client\\\": \\\"1.34.0\\\", \\\"azureml-tensorboard\\\": \\\"1.34.0\\\", \\\"azureml-telemetry\\\": \\\"1.34.0\\\", \\\"azureml-sdk\\\": \\\"1.34.0\\\", \\\"azureml-samples\\\": \\\"0+unknown\\\", \\\"azureml-responsibleai\\\": \\\"1.34.0\\\", \\\"azureml-pipeline\\\": \\\"1.34.0\\\", \\\"azureml-pipeline-steps\\\": \\\"1.34.0\\\", \\\"azureml-pipeline-core\\\": \\\"1.34.0\\\", \\\"azureml-opendatasets\\\": \\\"1.34.0\\\", \\\"azureml-mlflow\\\": \\\"1.34.0\\\", \\\"azureml-interpret\\\": \\\"1.34.0\\\", \\\"azureml-inference-server-http\\\": \\\"0.3.1\\\", \\\"azureml-explain-model\\\": \\\"1.34.0\\\", \\\"azureml-defaults\\\": \\\"1.34.0\\\", \\\"azureml-dataset-runtime\\\": \\\"1.34.0\\\", \\\"azureml-dataprep\\\": \\\"2.22.2\\\", \\\"azureml-dataprep-rslex\\\": \\\"1.20.2\\\", \\\"azureml-dataprep-native\\\": \\\"38.0.0\\\", \\\"azureml-datadrift\\\": \\\"1.34.0\\\", \\\"azureml-core\\\": \\\"1.34.0\\\", \\\"azureml-contrib-services\\\": \\\"1.34.0\\\", \\\"azureml-contrib-server\\\": \\\"1.34.0\\\", \\\"azureml-contrib-reinforcementlearning\\\": \\\"1.34.0\\\", \\\"azureml-contrib-pipeline-steps\\\": \\\"1.34.0\\\", \\\"azureml-contrib-notebook\\\": \\\"1.34.0\\\", \\\"azureml-contrib-fairness\\\": \\\"1.34.0\\\", \\\"azureml-contrib-dataset\\\": \\\"1.34.0\\\", \\\"azureml-contrib-automl-pipeline-steps\\\": \\\"1.34.0\\\", \\\"azureml-cli-common\\\": \\\"1.34.0\\\", \\\"azureml-automl-runtime\\\": \\\"1.34.0.post1\\\", \\\"azureml-automl-dnn-nlp\\\": \\\"1.34.0\\\", \\\"azureml-automl-core\\\": \\\"1.34.0.post1\\\", \\\"azureml-accel-models\\\": \\\"1.34.0\\\"}\", \"_aml_system_scenario_identification\": \"Remote.Parent\", \"ClientType\": \"SDK\", \"environment_cpu_name\": \"AzureML-AutoML\", \"environment_cpu_label\": \"prod\", \"environment_gpu_name\": \"AzureML-AutoML-GPU\", \"environment_gpu_label\": \"prod\", \"root_attribution\": \"automl\", \"attribution\": \"AutoML\", \"Orchestrator\": \"AutoML\", \"CancelUri\": \"https://northcentralus.api.azureml.ms/jasmine/v1.0/subscriptions/381b38e9-9840-4719-a5a0-61d9585e1e91/resourceGroups/cesardl-automl-ncentralus-demo-ws-resgrp/providers/Microsoft.MachineLearningServices/workspaces/cesardl-automl-ncentralus-demo-ws/experimentids/77f8e477-de26-42cc-a847-09f78a84c07e/cancel/AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"ClientSdkVersion\": \"1.34.1\", \"snapshotId\": \"00000000-0000-0000-0000-000000000000\", \"SetupRunId\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_setup\", \"SetupRunContainerId\": \"dcid.AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_setup\", \"FeaturizationRunJsonPath\": \"featurizer_container.json\", \"FeaturizationRunId\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_featurize\", \"ProblemInfoJsonString\": \"{\\\"dataset_num_categorical\\\": 0, \\\"is_sparse\\\": false, \\\"subsampling\\\": false, \\\"has_extra_col\\\": true, \\\"dataset_classes\\\": 2, \\\"dataset_features\\\": 30, \\\"dataset_samples\\\": 163951, \\\"single_frequency_class_detected\\\": false}\", \"ModelExplainRunId\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_ModelExplain\", \"azureml.git.repository_uri\": \"https://github.com/Azure/automl-testdataset-preview.git\", \"mlflow.source.git.repoURL\": \"https://github.com/Azure/automl-testdataset-preview.git\", \"azureml.git.branch\": \"main\", \"mlflow.source.git.branch\": \"main\", \"azureml.git.commit\": \"d2a2829489edbed961029aa33af5a3bdd7904698\", \"mlflow.source.git.commit\": \"d2a2829489edbed961029aa33af5a3bdd7904698\", \"azureml.git.dirty\": \"True\"}, \"tags\": {\"model_explain_run\": \"best_run\", \"_aml_system_azureml.automlComponent\": \"AutoML\", \"pipeline_id\": \"\", \"score\": \"\", \"predicted_cost\": \"\", \"fit_time\": \"\", \"training_percent\": \"\", \"iteration\": \"\", \"run_preprocessor\": \"\", \"run_algorithm\": \"\", \"automl_best_child_run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_8\", \"model_explain_best_run_child_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_8\"}, \"end_time_utc\": \"2021-10-25T22:13:18.143823Z\", \"status\": \"Completed\", \"log_files\": {}, \"log_groups\": [], \"run_duration\": \"0:15:56\", \"run_number\": \"44\", \"run_queued_details\": {\"status\": \"Completed\", \"details\": null}}, \"child_runs\": [{\"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_setup\", \"run_number\": 45, \"metric\": null, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": null, \"start_time\": \"2021-10-25T22:01:40.527697Z\", \"end_time\": \"2021-10-25T22:05:01.546734Z\", \"created_time\": \"2021-10-25T21:57:34.855458Z\", \"created_time_dt\": \"2021-10-25T21:57:34.855458Z\", \"duration\": \"0:07:26\", \"iteration\": null, \"goal\": null, \"run_name\": \"Completed\", \"run_properties\": null}, {\"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_featurize\", \"run_number\": 46, \"metric\": null, \"status\": \"Completed\", \"run_type\": \"automl.featurization\", \"training_percent\": null, \"start_time\": \"2021-10-25T22:05:02.441623Z\", \"end_time\": \"2021-10-25T22:06:16.728362Z\", \"created_time\": \"2021-10-25T22:05:02.034439Z\", \"created_time_dt\": \"2021-10-25T22:05:02.034439Z\", \"duration\": \"0:01:14\", \"iteration\": null, \"goal\": null, \"run_name\": \"Completed\", \"run_properties\": null}, {\"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_0\", \"run_number\": 48, \"metric\": null, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": \"100\", \"start_time\": \"2021-10-25T22:06:17.765521Z\", \"end_time\": \"2021-10-25T22:08:06.256507Z\", \"created_time\": \"2021-10-25T22:06:17.765521Z\", \"created_time_dt\": \"2021-10-25T22:06:17.765521Z\", \"duration\": \"0:01:48\", \"iteration\": \"0\", \"goal\": \"average_precision_score_weighted_max\", \"run_name\": \"MaxAbsScaler, LightGBM\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.9994057, \"best_metric\": 0.9994057}, {\"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_3\", \"run_number\": 49, \"metric\": null, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": \"100\", \"start_time\": \"2021-10-25T22:06:17.768817Z\", \"end_time\": \"2021-10-25T22:07:30.796006Z\", \"created_time\": \"2021-10-25T22:06:17.768817Z\", \"created_time_dt\": \"2021-10-25T22:06:17.768817Z\", \"duration\": \"0:01:13\", \"iteration\": \"3\", \"goal\": \"average_precision_score_weighted_max\", \"run_name\": \"SparseNormalizer, XGBoostClassifier\", \"run_properties\": \"copy=True, norm='l2'\", \"primary_metric\": 0.99948957, \"best_metric\": 0.99960235}, {\"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_2\", \"run_number\": 50, \"metric\": null, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": \"100\", \"start_time\": \"2021-10-25T22:06:17.764475Z\", \"end_time\": \"2021-10-25T22:06:52.573168Z\", \"created_time\": \"2021-10-25T22:06:17.764475Z\", \"created_time_dt\": \"2021-10-25T22:06:17.764475Z\", \"duration\": \"0:00:34\", \"iteration\": \"2\", \"goal\": \"average_precision_score_weighted_max\", \"run_name\": \"MaxAbsScaler, ExtremeRandomTrees\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.99933822, \"best_metric\": 0.99960235}, {\"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_1\", \"run_number\": 51, \"metric\": null, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": \"100\", \"start_time\": \"2021-10-25T22:08:22.213174Z\", \"end_time\": \"2021-10-25T22:09:17.168996Z\", \"created_time\": \"2021-10-25T22:06:17.794068Z\", \"created_time_dt\": \"2021-10-25T22:06:17.794068Z\", \"duration\": \"0:02:59\", \"iteration\": \"1\", \"goal\": \"average_precision_score_weighted_max\", \"run_name\": \"MaxAbsScaler, XGBoostClassifier\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.99960235, \"best_metric\": 0.99960235}, {\"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_4\", \"run_number\": 52, \"metric\": null, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": \"100\", \"start_time\": \"2021-10-25T22:08:59.662569Z\", \"end_time\": \"2021-10-25T22:11:34.283245Z\", \"created_time\": \"2021-10-25T22:06:55.125911Z\", \"created_time_dt\": \"2021-10-25T22:06:55.125911Z\", \"duration\": \"0:04:39\", \"iteration\": \"4\", \"goal\": \"average_precision_score_weighted_max\", \"run_name\": \"MaxAbsScaler, LightGBM\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.99909592, \"best_metric\": 0.99960235}, {\"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_5\", \"run_number\": 53, \"metric\": null, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": \"100\", \"start_time\": \"2021-10-25T22:09:39.087705Z\", \"end_time\": \"2021-10-25T22:11:33.317543Z\", \"created_time\": \"2021-10-25T22:07:33.689059Z\", \"created_time_dt\": \"2021-10-25T22:07:33.689059Z\", \"duration\": \"0:03:59\", \"iteration\": \"5\", \"goal\": \"average_precision_score_weighted_max\", \"run_name\": \"MaxAbsScaler, LogisticRegression\", \"run_properties\": \"copy=True\", \"primary_metric\": 0.99910239, \"best_metric\": 0.99960235}, {\"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_6\", \"run_number\": 54, \"metric\": null, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": \"100\", \"start_time\": \"2021-10-25T22:08:09.05491Z\", \"end_time\": \"2021-10-25T22:09:37.817732Z\", \"created_time\": \"2021-10-25T22:08:09.05491Z\", \"created_time_dt\": \"2021-10-25T22:08:09.05491Z\", \"duration\": \"0:01:28\", \"iteration\": \"6\", \"goal\": \"average_precision_score_weighted_max\", \"run_name\": \"SparseNormalizer, XGBoostClassifier\", \"run_properties\": \"copy=True, norm='max'\", \"primary_metric\": 0.99954039, \"best_metric\": 0.99960235}, {\"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_7\", \"run_number\": 55, \"metric\": null, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": \"100\", \"start_time\": \"2021-10-25T22:09:20.032362Z\", \"end_time\": \"2021-10-25T22:09:52.919479Z\", \"created_time\": \"2021-10-25T22:09:20.032362Z\", \"created_time_dt\": \"2021-10-25T22:09:20.032362Z\", \"duration\": \"0:00:32\", \"iteration\": \"7\", \"goal\": \"average_precision_score_weighted_max\", \"run_name\": \"StandardScalerWrapper, LogisticRegression\", \"run_properties\": \"\\n    copy=True,\\n    with_mean=True,\\n    with_std=False\\n\", \"primary_metric\": 0.99955242, \"best_metric\": 0.99960235}, {\"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_9\", \"run_number\": 56, \"metric\": null, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": \"100\", \"start_time\": \"2021-10-25T22:11:35.180301Z\", \"end_time\": \"2021-10-25T22:13:12.82868Z\", \"created_time\": \"2021-10-25T22:11:35.180301Z\", \"created_time_dt\": \"2021-10-25T22:11:35.180301Z\", \"duration\": \"0:01:37\", \"iteration\": \"9\", \"goal\": \"average_precision_score_weighted_max\", \"run_name\": \"StackEnsemble\", \"run_properties\": \"\\n    base_learners=[('1', Pipeline(\\n        memory=None,\\n        steps=[('maxabsscaler', MaxAbsScaler(\\n            copy=True\\n        \", \"primary_metric\": 0.99971944, \"best_metric\": 0.99973769}, {\"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_8\", \"run_number\": 57, \"metric\": null, \"status\": \"Completed\", \"run_type\": \"azureml.scriptrun\", \"training_percent\": \"100\", \"start_time\": \"2021-10-25T22:11:35.187448Z\", \"end_time\": \"2021-10-25T22:12:12.40812Z\", \"created_time\": \"2021-10-25T22:11:35.187448Z\", \"created_time_dt\": \"2021-10-25T22:11:35.187448Z\", \"duration\": \"0:00:37\", \"iteration\": \"8\", \"goal\": \"average_precision_score_weighted_max\", \"run_name\": \"VotingEnsemble\", \"run_properties\": \"\\n    estimators=[('1', Pipeline(\\n        memory=None,\\n        steps=[('maxabsscaler', MaxAbsScaler(\\n            copy=True\\n        \", \"primary_metric\": 0.99973769, \"best_metric\": 0.99973769}], \"children_metrics\": {\"categories\": [0], \"series\": {\"f1_score_micro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"f1_score_micro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.99934127061579, 0.9960720210793403, 0.001683419537425588, 0.9974138772323607, 0.9409583292671025, 0.6385039523763053, 0.9959500341563384, 0.9783107250902703, 0.99934127061579, 0.9959744315409388]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"f1_score_micro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579]}], \"weighted_accuracy\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"weighted_accuracy\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9997549148837067, 0.9962849838931771, 2.8434586906508057e-06, 0.9977022906777789, 0.9410053214054287, 0.6379442008195129, 0.9961627916161735, 0.9783961993780483, 0.9995841341627256, 0.9961628328257198]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"weighted_accuracy_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9997549148837067, 0.9997549148837067, 0.9997549148837067, 0.9997549148837067, 0.9997549148837067, 0.9997549148837067, 0.9997549148837067, 0.9997549148837067, 0.9997549148837067, 0.9997549148837067]}], \"recall_score_micro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"recall_score_micro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.99934127061579, 0.9960720210793403, 0.001683419537425588, 0.9974138772323607, 0.9409583292671025, 0.6385039523763053, 0.9959500341563384, 0.9783107250902703, 0.99934127061579, 0.9959744315409388]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"recall_score_micro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579]}], \"recall_score_weighted\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"recall_score_weighted\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.99934127061579, 0.9960720210793403, 0.001683419537425588, 0.9974138772323607, 0.9409583292671025, 0.6385039523763052, 0.9959500341563384, 0.9783107250902703, 0.99934127061579, 0.9959744315409388]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"recall_score_weighted_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579]}], \"log_loss\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"log_loss\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.0038095218926388714, 0.02753681583713592, 3.732291045847202, 0.2121006199518145, 0.33577641940034325, 1.7356585492412264, 0.06008175725985037, 0.1240619706871904, 0.08728756467542635, 0.39651368378925067]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"log_loss_min\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.0038095218926388714, 0.0038095218926388714, 0.0038095218926388714, 0.0038095218926388714, 0.0038095218926388714, 0.0038095218926388714, 0.0038095218926388714, 0.0038095218926388714, 0.0038095218926388714, 0.0038095218926388714]}], \"accuracy\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"accuracy\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.99934127061579, 0.9960720210793403, 0.001683419537425588, 0.9974138772323607, 0.9409583292671025, 0.6385039523763053, 0.9959500341563384, 0.9783107250902703, 0.99934127061579, 0.9959744315409388]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"accuracy_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579]}], \"AUC_macro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"AUC_macro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9541572233018856, 0.967028091198908, 0.9586362028057551, 0.9746499889672456, 0.9808469259346231, 0.9810258761476811, 0.9852726011197095, 0.9831335218287385, 0.9865510193167059, 0.9862319017670471]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"AUC_macro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9541572233018856, 0.967028091198908, 0.967028091198908, 0.9746499889672456, 0.9808469259346231, 0.9810258761476811, 0.9852726011197095, 0.9852726011197095, 0.9865510193167059, 0.9865510193167059]}], \"precision_score_micro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"precision_score_micro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.99934127061579, 0.9960720210793403, 0.001683419537425588, 0.9974138772323607, 0.9409583292671025, 0.6385039523763053, 0.9959500341563384, 0.9783107250902703, 0.99934127061579, 0.9959744315409388]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"precision_score_micro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579, 0.99934127061579]}], \"matthews_correlation\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"matthews_correlation\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.7947031845741636, 0.49483425192047903, 0.0, 0.5574270486399274, 0.14693131891243594, 0.0519126732756993, 0.4890611351664289, 0.24707666154294794, 0.8144168645406298, 0.4961125744127825]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"matthews_correlation_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.7947031845741636, 0.7947031845741636, 0.7947031845741636, 0.7947031845741636, 0.7947031845741636, 0.7947031845741636, 0.7947031845741636, 0.7947031845741636, 0.8144168645406298, 0.8144168645406298]}], \"average_precision_score_weighted\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"average_precision_score_weighted\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9994056966966292, 0.9996023473097911, 0.9993382192439357, 0.9994895666634291, 0.9990959202940611, 0.9991023930694461, 0.999540391745959, 0.9995524167286169, 0.9997376883433837, 0.9997194396194001]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"average_precision_score_weighted_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9994056966966292, 0.9996023473097911, 0.9996023473097911, 0.9996023473097911, 0.9996023473097911, 0.9996023473097911, 0.9996023473097911, 0.9996023473097911, 0.9997376883433837, 0.9997376883433837]}], \"f1_score_weighted\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"f1_score_weighted\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9993236991454996, 0.9970680512324217, 5.658277423209955e-06, 0.9978945395549076, 0.9679841028212142, 0.777637050278511, 0.996994258699823, 0.9875660748218353, 0.9993571428056712, 0.9970157248754824]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"f1_score_weighted_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9993236991454996, 0.9993236991454996, 0.9993236991454996, 0.9993236991454996, 0.9993236991454996, 0.9993236991454996, 0.9993236991454996, 0.9993236991454996, 0.9993571428056712, 0.9993571428056712]}], \"AUC_micro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"AUC_micro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9998449235140208, 0.9995505900329015, 0.0032249743666748895, 0.9997399852367133, 0.9865475578429918, 0.6888886381439441, 0.9996973451702735, 0.9917318848097927, 0.9999454808810456, 0.9995574405623093]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"AUC_micro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9998449235140208, 0.9998449235140208, 0.9998449235140208, 0.9998449235140208, 0.9998449235140208, 0.9998449235140208, 0.9998449235140208, 0.9998449235140208, 0.9999454808810456, 0.9999454808810456]}], \"AUC_weighted\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"AUC_weighted\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9541565173237895, 0.967028002951646, 0.9586362028057552, 0.9746499889672458, 0.980846925934623, 0.9811500400453054, 0.9852726011197094, 0.9831345807958823, 0.9865510193167059, 0.9862319017670471]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"AUC_weighted_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9541565173237895, 0.967028002951646, 0.967028002951646, 0.9746499889672458, 0.980846925934623, 0.9811500400453054, 0.9852726011197094, 0.9852726011197094, 0.9865510193167059, 0.9865510193167059]}], \"balanced_accuracy\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"balanced_accuracy\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.8766894015784454, 0.9329252808039636, 0.5, 0.9118948675910097, 0.9270244395874352, 0.8044788732494135, 0.9328641844917371, 0.9529662879403672, 0.9273285044224875, 0.9401105613033314]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"balanced_accuracy_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.8766894015784454, 0.9329252808039636, 0.9329252808039636, 0.9329252808039636, 0.9329252808039636, 0.9329252808039636, 0.9329252808039636, 0.9529662879403672, 0.9529662879403672, 0.9529662879403672]}], \"f1_score_macro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"f1_score_macro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.8967816189511081, 0.712537759250785, 0.001680590398713983, 0.7584426386275792, 0.5095111585996811, 0.3939479225322676, 0.7087741735556646, 0.5574397533947255, 0.9067315775700886, 0.7115336261096998]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"f1_score_macro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.8967816189511081, 0.8967816189511081, 0.8967816189511081, 0.8967816189511081, 0.8967816189511081, 0.8967816189511081, 0.8967816189511081, 0.8967816189511081, 0.9067315775700886, 0.9067315775700886]}], \"average_precision_score_micro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"average_precision_score_micro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9996405512748467, 0.9993400495965414, 0.30830374053435433, 0.9997304063450214, 0.9867525075924797, 0.6661763494078391, 0.9996884225123226, 0.9868374931765789, 0.9999461757339565, 0.9989796139261261]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"average_precision_score_micro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9996405512748467, 0.9996405512748467, 0.9996405512748467, 0.9997304063450214, 0.9997304063450214, 0.9997304063450214, 0.9997304063450214, 0.9997304063450214, 0.9999461757339565, 0.9999461757339565]}], \"recall_score_macro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"recall_score_macro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.8766894015784454, 0.9329252808039636, 0.5, 0.9118948675910097, 0.9270244395874352, 0.8044788732494135, 0.9328641844917371, 0.9529662879403672, 0.9273285044224875, 0.9401105613033314]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"recall_score_macro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.8766894015784454, 0.9329252808039636, 0.9329252808039636, 0.9329252808039636, 0.9329252808039636, 0.9329252808039636, 0.9329252808039636, 0.9529662879403672, 0.9529662879403672, 0.9529662879403672]}], \"norm_macro_recall\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"norm_macro_recall\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.7533788031568909, 0.8658505616079273, 0.0, 0.8237897351820194, 0.8540488791748704, 0.608957746498827, 0.8657283689834743, 0.9059325758807344, 0.854657008844975, 0.8802211226066627]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"norm_macro_recall_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.7533788031568909, 0.8658505616079273, 0.8658505616079273, 0.8658505616079273, 0.8658505616079273, 0.8658505616079273, 0.8658505616079273, 0.9059325758807344, 0.9059325758807344, 0.9059325758807344]}], \"precision_score_macro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"precision_score_macro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9191471467778981, 0.6413990749275378, 0.000841709768712794, 0.6885947962721121, 0.512639096545742, 0.5022127361562636, 0.6381384753575297, 0.5336928366991658, 0.8880356812053598, 0.6398101455243445]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"precision_score_macro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9191471467778981, 0.9191471467778981, 0.9191471467778981, 0.9191471467778981, 0.9191471467778981, 0.9191471467778981, 0.9191471467778981, 0.9191471467778981, 0.9191471467778981, 0.9191471467778981]}], \"average_precision_score_macro\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"average_precision_score_macro\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9163031733114289, 0.9135082229550493, 0.8373298095827609, 0.8630432958124936, 0.7428186901524049, 0.7462247758680312, 0.8721004855797542, 0.8774371792991751, 0.9300352383788006, 0.924935309589435]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"average_precision_score_macro_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9163031733114289, 0.9163031733114289, 0.9163031733114289, 0.9163031733114289, 0.9163031733114289, 0.9163031733114289, 0.9163031733114289, 0.9163031733114289, 0.9300352383788006, 0.9300352383788006]}], \"precision_score_weighted\": [{\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"markers\", \"name\": \"precision_score_weighted\", \"stepped\": false, \"type\": \"scatter\", \"data\": [0.9993137961215227, 0.9985726734459562, 2.8339013389861806e-06, 0.9986586869771743, 0.998203859177356, 0.9982476775843375, 0.9985616685550278, 0.9983055641312905, 0.9993794300401153, 0.9985917366611043]}, {\"categories\": [\"0\", \"1\", \"2\", \"3\", \"4\", \"5\", \"6\", \"7\", \"8\", \"9\"], \"mode\": \"lines\", \"name\": \"precision_score_weighted_max\", \"stepped\": true, \"type\": \"scatter\", \"data\": [0.9993137961215227, 0.9993137961215227, 0.9993137961215227, 0.9993137961215227, 0.9993137961215227, 0.9993137961215227, 0.9993137961215227, 0.9993137961215227, 0.9993794300401153, 0.9993794300401153]}]}, \"metricName\": null, \"primaryMetricName\": \"average_precision_score_weighted\", \"showLegend\": false}, \"run_metrics\": [{\"name\": \"experiment_status\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], \"series\": [{\"data\": [\"DatasetEvaluation\", \"FeaturesGeneration\", \"DatasetFeaturization\", \"DatasetFeaturizationCompleted\", \"DatasetBalancing\", \"ModelSelection\", \"BestRunExplainModel\", \"ModelExplanationDataSetSetup\", \"PickSurrogateModel\", \"EngineeredFeatureExplanations\", \"EngineeredFeatureExplanations\", \"RawFeaturesExplanations\"]}]}, {\"name\": \"experiment_status_description\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], \"series\": [{\"data\": [\"Gathering dataset statistics.\", \"Generating features for the dataset.\", \"Beginning to fit featurizers and featurize the dataset.\", \"Completed fit featurizers and featurizing the dataset.\", \"Performing class balancing sweeping\", \"Beginning model selection.\", \"Best run model explanations started\", \"Model explanations data setup completed\", \"Choosing LightGBM as the surrogate model for explanations\", \"Computation of engineered features started\", \"Computation of engineered features completed\", \"Computation of raw features started\"]}]}, {\"name\": \"average_precision_score_micro\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.9999461757339565]}]}, {\"name\": \"f1_score_weighted\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.9993571428056712]}]}, {\"name\": \"AUC_micro\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.9999454808810456]}]}, {\"name\": \"f1_score_micro\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.99934127061579]}]}, {\"name\": \"precision_score_micro\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.99934127061579]}]}, {\"name\": \"precision_score_weighted\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.9993794300401153]}]}, {\"name\": \"matthews_correlation\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.8144168645406298]}]}, {\"name\": \"recall_score_weighted\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.99934127061579]}]}, {\"name\": \"average_precision_score_macro\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.9300352383788006]}]}, {\"name\": \"average_precision_score_weighted\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.9997376883433837]}]}, {\"name\": \"weighted_accuracy\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.9995841341627256]}]}, {\"name\": \"AUC_weighted\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.9865510193167059]}]}, {\"name\": \"AUC_macro\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.9865510193167059]}]}, {\"name\": \"balanced_accuracy\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.9273285044224875]}]}, {\"name\": \"accuracy\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.99934127061579]}]}, {\"name\": \"recall_score_micro\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.99934127061579]}]}, {\"name\": \"norm_macro_recall\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.854657008844975]}]}, {\"name\": \"precision_score_macro\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.8880356812053598]}]}, {\"name\": \"f1_score_macro\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.9067315775700886]}]}, {\"name\": \"log_loss\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.08728756467542635]}]}, {\"name\": \"recall_score_macro\", \"run_id\": \"AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82\", \"categories\": [0], \"series\": [{\"data\": [0.9273285044224875]}]}], \"run_logs\": \"\\nRun is completed.\", \"graph\": {}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": false, \"log_level\": \"INFO\", \"sdk_version\": \"1.34.0\"}, \"loading\": false}"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from azureml.widgets import RunDetails\n",
    "RunDetails(remote_run).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'runId': 'AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82',\n",
       " 'target': 'cpu-cluster',\n",
       " 'status': 'Completed',\n",
       " 'startTimeUtc': '2021-10-25T21:57:36.694565Z',\n",
       " 'endTimeUtc': '2021-10-25T22:13:18.143823Z',\n",
       " 'services': {},\n",
       " 'properties': {'num_iterations': '10',\n",
       "  'training_type': 'TrainFull',\n",
       "  'acquisition_function': 'EI',\n",
       "  'primary_metric': 'average_precision_score_weighted',\n",
       "  'train_split': '0.2',\n",
       "  'acquisition_parameter': '0',\n",
       "  'num_cross_validation': None,\n",
       "  'target': 'cpu-cluster',\n",
       "  'AMLSettingsJsonString': '{\"path\":null,\"name\":\"test-set-support-automl-classification-ccard-remote\",\"subscription_id\":\"381b38e9-9840-4719-a5a0-61d9585e1e91\",\"resource_group\":\"cesardl-automl-ncentralus-demo-ws-resgrp\",\"workspace_name\":\"cesardl-automl-ncentralus-demo-ws\",\"region\":\"northcentralus\",\"compute_target\":\"cpu-cluster\",\"spark_service\":null,\"azure_service\":\"remote\",\"many_models\":false,\"pipeline_fetch_max_batch_size\":1,\"enable_batch_run\":false,\"enable_run_restructure\":false,\"start_auxiliary_runs_before_parent_complete\":true,\"enable_code_generation\":false,\"iterations\":10,\"primary_metric\":\"average_precision_score_weighted\",\"task_type\":\"classification\",\"data_script\":null,\"test_size\":0.1,\"test_include_predictions_only\":false,\"validation_size\":0.2,\"n_cross_validations\":null,\"y_min\":null,\"y_max\":null,\"num_classes\":null,\"featurization\":\"auto\",\"_ignore_package_version_incompatibilities\":false,\"is_timeseries\":false,\"max_cores_per_iteration\":1,\"max_concurrent_iterations\":4,\"iteration_timeout_minutes\":null,\"mem_in_mb\":null,\"enforce_time_on_windows\":false,\"experiment_timeout_minutes\":15,\"experiment_exit_score\":null,\"whitelist_models\":null,\"blacklist_algos\":[\"TensorFlowLinearClassifier\",\"TensorFlowDNN\"],\"supported_models\":[\"BernoulliNaiveBayes\",\"LinearSVM\",\"RandomForest\",\"LogisticRegression\",\"SGD\",\"XGBoostClassifier\",\"AveragedPerceptronClassifier\",\"GradientBoosting\",\"TensorFlowLinearClassifier\",\"MultinomialNaiveBayes\",\"DecisionTree\",\"LightGBM\",\"KNN\",\"SVM\",\"ExtremeRandomTrees\",\"TensorFlowDNN\"],\"private_models\":[],\"auto_blacklist\":true,\"blacklist_samples_reached\":false,\"exclude_nan_labels\":true,\"verbosity\":20,\"_debug_log\":\"azureml_automl.log\",\"show_warnings\":false,\"model_explainability\":true,\"service_url\":null,\"sdk_url\":null,\"sdk_packages\":null,\"enable_onnx_compatible_models\":false,\"enable_split_onnx_featurizer_estimator_models\":false,\"vm_type\":\"STANDARD_D12_V2\",\"telemetry_verbosity\":20,\"send_telemetry\":true,\"enable_dnn\":false,\"scenario\":\"SDK-1.13.0\",\"environment_label\":null,\"save_mlflow\":false,\"force_text_dnn\":false,\"enable_feature_sweeping\":true,\"enable_early_stopping\":true,\"early_stopping_n_iters\":10,\"metrics\":null,\"enable_metric_confidence\":false,\"enable_ensembling\":true,\"enable_stack_ensembling\":true,\"ensemble_iterations\":10,\"enable_tf\":false,\"enable_subsampling\":false,\"subsample_seed\":null,\"enable_nimbusml\":false,\"enable_streaming\":false,\"force_streaming\":false,\"track_child_runs\":true,\"allowed_private_models\":[],\"label_column_name\":\"Class\",\"weight_column_name\":null,\"cv_split_column_names\":null,\"positive_label\":null,\"enable_local_managed\":false,\"_local_managed_run_id\":null,\"cost_mode\":1,\"lag_length\":0,\"metric_operation\":\"maximize\",\"preprocess\":true}',\n",
       "  'DataPrepJsonString': '{\\\\\"training_data\\\\\": {\\\\\"datasetId\\\\\": \\\\\"72710eb4-5208-465d-85dc-6330b6828dcc\\\\\"}, \\\\\"datasets\\\\\": 0}',\n",
       "  'EnableSubsampling': 'False',\n",
       "  'runTemplate': 'AutoML',\n",
       "  'azureml.runsource': 'automl',\n",
       "  'display_task_type': 'classification',\n",
       "  'dependencies_versions': '{\"azureml-widgets\": \"1.34.0\", \"azureml-train\": \"1.34.0\", \"azureml-train-restclients-hyperdrive\": \"1.34.0\", \"azureml-train-core\": \"1.34.0\", \"azureml-train-automl\": \"1.34.0\", \"azureml-train-automl-runtime\": \"1.34.0.post1\", \"azureml-train-automl-client\": \"1.34.0\", \"azureml-tensorboard\": \"1.34.0\", \"azureml-telemetry\": \"1.34.0\", \"azureml-sdk\": \"1.34.0\", \"azureml-samples\": \"0+unknown\", \"azureml-responsibleai\": \"1.34.0\", \"azureml-pipeline\": \"1.34.0\", \"azureml-pipeline-steps\": \"1.34.0\", \"azureml-pipeline-core\": \"1.34.0\", \"azureml-opendatasets\": \"1.34.0\", \"azureml-mlflow\": \"1.34.0\", \"azureml-interpret\": \"1.34.0\", \"azureml-inference-server-http\": \"0.3.1\", \"azureml-explain-model\": \"1.34.0\", \"azureml-defaults\": \"1.34.0\", \"azureml-dataset-runtime\": \"1.34.0\", \"azureml-dataprep\": \"2.22.2\", \"azureml-dataprep-rslex\": \"1.20.2\", \"azureml-dataprep-native\": \"38.0.0\", \"azureml-datadrift\": \"1.34.0\", \"azureml-core\": \"1.34.0\", \"azureml-contrib-services\": \"1.34.0\", \"azureml-contrib-server\": \"1.34.0\", \"azureml-contrib-reinforcementlearning\": \"1.34.0\", \"azureml-contrib-pipeline-steps\": \"1.34.0\", \"azureml-contrib-notebook\": \"1.34.0\", \"azureml-contrib-fairness\": \"1.34.0\", \"azureml-contrib-dataset\": \"1.34.0\", \"azureml-contrib-automl-pipeline-steps\": \"1.34.0\", \"azureml-cli-common\": \"1.34.0\", \"azureml-automl-runtime\": \"1.34.0.post1\", \"azureml-automl-dnn-nlp\": \"1.34.0\", \"azureml-automl-core\": \"1.34.0.post1\", \"azureml-accel-models\": \"1.34.0\"}',\n",
       "  '_aml_system_scenario_identification': 'Remote.Parent',\n",
       "  'ClientType': 'SDK',\n",
       "  'environment_cpu_name': 'AzureML-AutoML',\n",
       "  'environment_cpu_label': 'prod',\n",
       "  'environment_gpu_name': 'AzureML-AutoML-GPU',\n",
       "  'environment_gpu_label': 'prod',\n",
       "  'root_attribution': 'automl',\n",
       "  'attribution': 'AutoML',\n",
       "  'Orchestrator': 'AutoML',\n",
       "  'CancelUri': 'https://northcentralus.api.azureml.ms/jasmine/v1.0/subscriptions/381b38e9-9840-4719-a5a0-61d9585e1e91/resourceGroups/cesardl-automl-ncentralus-demo-ws-resgrp/providers/Microsoft.MachineLearningServices/workspaces/cesardl-automl-ncentralus-demo-ws/experimentids/77f8e477-de26-42cc-a847-09f78a84c07e/cancel/AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82',\n",
       "  'ClientSdkVersion': '1.34.1',\n",
       "  'snapshotId': '00000000-0000-0000-0000-000000000000',\n",
       "  'SetupRunId': 'AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_setup',\n",
       "  'SetupRunContainerId': 'dcid.AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_setup',\n",
       "  'FeaturizationRunJsonPath': 'featurizer_container.json',\n",
       "  'FeaturizationRunId': 'AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_featurize',\n",
       "  'ProblemInfoJsonString': '{\"dataset_num_categorical\": 0, \"is_sparse\": false, \"subsampling\": false, \"has_extra_col\": true, \"dataset_classes\": 2, \"dataset_features\": 30, \"dataset_samples\": 163951, \"single_frequency_class_detected\": false}',\n",
       "  'ModelExplainRunId': 'AutoML_3422f595-4af4-4b2f-872e-1b1bfbb1cb82_ModelExplain',\n",
       "  'azureml.git.repository_uri': 'https://github.com/Azure/automl-testdataset-preview.git',\n",
       "  'mlflow.source.git.repoURL': 'https://github.com/Azure/automl-testdataset-preview.git',\n",
       "  'azureml.git.branch': 'main',\n",
       "  'mlflow.source.git.branch': 'main',\n",
       "  'azureml.git.commit': 'd2a2829489edbed961029aa33af5a3bdd7904698',\n",
       "  'mlflow.source.git.commit': 'd2a2829489edbed961029aa33af5a3bdd7904698',\n",
       "  'azureml.git.dirty': 'True'},\n",
       " 'inputDatasets': [{'dataset': {'id': '72710eb4-5208-465d-85dc-6330b6828dcc'}, 'consumptionDetails': {'type': 'RunInput', 'inputName': 'training_data', 'mechanism': 'Direct'}}],\n",
       " 'outputDatasets': [],\n",
       " 'logFiles': {},\n",
       " 'submittedBy': 'Cesar De la Torre Llorente'}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "remote_run.wait_for_completion(show_output=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explain model\n",
    "\n",
    "Automated ML models can be explained and visualized using the SDK Explainability library. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyze results\n",
    "\n",
    "### Retrieve the Best Model/Run\n",
    "\n",
    "Below we select the best pipeline from our iterations. The `get_output` method returns the best run and the fitted model.  Overloads on `get_output` allow you to retrieve the best run and fitted model for *any* logged metric or for a particular *iteration*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PipelineWithYTransformations(Pipeline={'memory': None,\n",
       "                                       'steps': [('datatransformer',\n",
       "                                                  DataTransformer(enable_dnn=False, enable_feature_sweeping=True, feature_sweeping_config={}, feature_sweeping_timeout=86400, featurization_config=None, force_text_dnn=False, is_cross_validation=False, is_onnx_compatible=False, observer=None, task='classification', working_dir='/m...\n",
       "), random_state=None))], verbose=False)), ('4', Pipeline(memory=None, steps=[('maxabsscaler', MaxAbsScaler(copy=True)), ('lightgbmclassifier', LightGBMClassifier(boosting_type='gbdt', colsample_bytree=0.4955555555555555, learning_rate=0.09473736842105263, max_bin=20, max_depth=4, min_child_weight=0, min_data_in_leaf=0.05517689655172415, min_split_gain=0.7894736842105263, n_estimators=10, n_jobs=1, num_leaves=176, problem_info=ProblemInfo(\n",
       "    gpu_training_param_dict={'processing_unit_type': 'cpu'}\n",
       "), random_state=None, reg_alpha=0.3684210526315789, reg_lambda=0.631578947368421, subsample=0.49526315789473685))], verbose=False))], flatten_transform=None, weights=[0.1, 0.1, 0.1, 0.2, 0.4, 0.1]))],\n",
       "                                       'verbose': False},\n",
       "                             y_transformer={},\n",
       "                             y_transformer_name='LabelEncoder')"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_run, fitted_model = remote_run.get_output()\n",
    "fitted_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Print the properties of the model\n",
    "The fitted_model is a python object and you can read the different properties of the object.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get the Predictions and Metrics Generated by the Test Run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the test run associated with the best run. When a remote test run is requested by passing in a value for `test_data` or `test_size` to `AutoMLConfig`, only the best training run will have a test run associated with it. To start a test run for models which are not associated with the best run or to start another best run test run, use `ModelProxy`. See the example later in the notebook in the __Test the fitted model remotely__ section for more details.\n",
    "\n",
    "To see more details about the test run in Azure Machine Learning Studio (view its metrics, get a preview of the predictions, etc...) follow the link to the __Details Page__ listed in the next cells output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"width:100%\"><tr><th>Experiment</th><th>Id</th><th>Type</th><th>Status</th><th>Details Page</th><th>Docs Page</th></tr><tr><td>test-set-support-automl-classification-ccard-remote</td><td>e2fd2f41-0882-4bd1-877e-6fd8f73ace46</td><td>automl.model_test</td><td>Completed</td><td><a href=\"https://ml.azure.com/runs/e2fd2f41-0882-4bd1-877e-6fd8f73ace46?wsid=/subscriptions/381b38e9-9840-4719-a5a0-61d9585e1e91/resourcegroups/cesardl-automl-ncentralus-demo-ws-resgrp/workspaces/cesardl-automl-ncentralus-demo-ws&amp;tid=72f988bf-86f1-41af-91ab-2d7cd011db47\" target=\"_blank\" rel=\"noopener\">Link to Azure Machine Learning studio</a></td><td><a href=\"https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.run.Run?view=azure-ml-py\" target=\"_blank\" rel=\"noopener\">Link to Documentation</a></td></tr></table>"
      ],
      "text/plain": [
       "Run(Experiment: test-set-support-automl-classification-ccard-remote,\n",
       "Id: e2fd2f41-0882-4bd1-877e-6fd8f73ace46,\n",
       "Type: automl.model_test,\n",
       "Status: Completed)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_run = next(best_run.get_children(type='automl.model_test'))\n",
    "test_run.wait_for_completion(show_output=False, wait_post_processing=True)\n",
    "test_run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the __metrics__ from the test run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "matthews_correlation: 0.7505032095637124\n",
      "average_precision_score_macro: 0.8878577879244975\n",
      "norm_macro_recall: 0.7889458404522789\n",
      "recall_score_macro: 0.8944729202261394\n",
      "recall_score_weighted: 0.9991217284384332\n",
      "AUC_micro: 0.9996670111975998\n",
      "precision_score_macro: 0.856966878260323\n",
      "f1_score_weighted: 0.9991436465915597\n",
      "precision_score_weighted: 0.9991718535619092\n",
      "weighted_accuracy: 0.9994715695224134\n",
      "AUC_weighted: 0.9598653535395628\n",
      "average_precision_score_micro: 0.9995664236282711\n",
      "balanced_accuracy: 0.8944729202261394\n",
      "recall_score_micro: 0.9991217284384332\n",
      "f1_score_macro: 0.874780045750484\n",
      "precision_score_micro: 0.9991217284384332\n",
      "f1_score_micro: 0.9991217284384332\n",
      "accuracy: 0.9991217284384332\n",
      "average_precision_score_weighted: 0.9995316154124612\n",
      "AUC_macro: 0.9598653535395628\n",
      "log_loss: 0.08762644383474373\n",
      "confusion_matrix: aml://artifactId/ExperimentRun/dcid.e2fd2f41-0882-4bd1-877e-6fd8f73ace46/confusion_matrix\n",
      "accuracy_table: aml://artifactId/ExperimentRun/dcid.e2fd2f41-0882-4bd1-877e-6fd8f73ace46/accuracy_table\n"
     ]
    }
   ],
   "source": [
    "test_run_metrics = test_run.get_metrics()\n",
    "for name, value in test_run_metrics.items():\n",
    "    print(f\"{name}: {value}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the __predictions__ from the test run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Class_orig</th>\n",
       "      <th>Class_predicted</th>\n",
       "      <th>False_predicted_proba</th>\n",
       "      <th>True_predicted_proba</th>\n",
       "      <th>Time_orig</th>\n",
       "      <th>V1_orig</th>\n",
       "      <th>V2_orig</th>\n",
       "      <th>V3_orig</th>\n",
       "      <th>V4_orig</th>\n",
       "      <th>V5_orig</th>\n",
       "      <th>...</th>\n",
       "      <th>V20_orig</th>\n",
       "      <th>V21_orig</th>\n",
       "      <th>V22_orig</th>\n",
       "      <th>V23_orig</th>\n",
       "      <th>V24_orig</th>\n",
       "      <th>V25_orig</th>\n",
       "      <th>V26_orig</th>\n",
       "      <th>V27_orig</th>\n",
       "      <th>V28_orig</th>\n",
       "      <th>Amount_orig</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.25</td>\n",
       "      <td>144339.00</td>\n",
       "      <td>0.84</td>\n",
       "      <td>1.25</td>\n",
       "      <td>-2.01</td>\n",
       "      <td>0.46</td>\n",
       "      <td>0.53</td>\n",
       "      <td>...</td>\n",
       "      <td>0.51</td>\n",
       "      <td>-1.17</td>\n",
       "      <td>0.78</td>\n",
       "      <td>-0.07</td>\n",
       "      <td>-0.32</td>\n",
       "      <td>0.66</td>\n",
       "      <td>-0.05</td>\n",
       "      <td>0.04</td>\n",
       "      <td>0.16</td>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>0.91</td>\n",
       "      <td>0.09</td>\n",
       "      <td>99102.00</td>\n",
       "      <td>1.71</td>\n",
       "      <td>-0.37</td>\n",
       "      <td>-2.51</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.58</td>\n",
       "      <td>...</td>\n",
       "      <td>0.25</td>\n",
       "      <td>0.28</td>\n",
       "      <td>0.82</td>\n",
       "      <td>-0.41</td>\n",
       "      <td>0.13</td>\n",
       "      <td>0.68</td>\n",
       "      <td>0.39</td>\n",
       "      <td>-0.17</td>\n",
       "      <td>-0.05</td>\n",
       "      <td>237.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>0.89</td>\n",
       "      <td>0.11</td>\n",
       "      <td>128642.00</td>\n",
       "      <td>-4.10</td>\n",
       "      <td>-6.43</td>\n",
       "      <td>-0.24</td>\n",
       "      <td>1.13</td>\n",
       "      <td>-5.27</td>\n",
       "      <td>...</td>\n",
       "      <td>4.25</td>\n",
       "      <td>1.35</td>\n",
       "      <td>0.49</td>\n",
       "      <td>4.50</td>\n",
       "      <td>-0.14</td>\n",
       "      <td>0.60</td>\n",
       "      <td>0.64</td>\n",
       "      <td>-0.68</td>\n",
       "      <td>0.15</td>\n",
       "      <td>1895.62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>0.94</td>\n",
       "      <td>0.06</td>\n",
       "      <td>31158.00</td>\n",
       "      <td>-1.54</td>\n",
       "      <td>0.22</td>\n",
       "      <td>1.19</td>\n",
       "      <td>-0.79</td>\n",
       "      <td>0.73</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.35</td>\n",
       "      <td>-0.08</td>\n",
       "      <td>-0.10</td>\n",
       "      <td>-0.06</td>\n",
       "      <td>0.08</td>\n",
       "      <td>-0.87</td>\n",
       "      <td>0.65</td>\n",
       "      <td>-0.95</td>\n",
       "      <td>0.24</td>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>0.92</td>\n",
       "      <td>0.08</td>\n",
       "      <td>66866.00</td>\n",
       "      <td>-0.38</td>\n",
       "      <td>1.05</td>\n",
       "      <td>1.26</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.15</td>\n",
       "      <td>...</td>\n",
       "      <td>0.03</td>\n",
       "      <td>-0.25</td>\n",
       "      <td>-0.63</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.00</td>\n",
       "      <td>-0.26</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.26</td>\n",
       "      <td>0.09</td>\n",
       "      <td>0.89</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 34 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Class_orig  Class_predicted  False_predicted_proba  True_predicted_proba  \\\n",
       "0  False       False           0.75                   0.25                    \n",
       "1  False       False           0.91                   0.09                    \n",
       "2  False       False           0.89                   0.11                    \n",
       "3  False       False           0.94                   0.06                    \n",
       "4  False       False           0.92                   0.08                    \n",
       "\n",
       "   Time_orig  V1_orig  V2_orig  V3_orig  V4_orig  V5_orig  ...  V20_orig  \\\n",
       "0 144339.00  0.84     1.25     -2.01    0.46     0.53      ... 0.51        \n",
       "1 99102.00   1.71     -0.37    -2.51    0.50     0.58      ... 0.25        \n",
       "2 128642.00  -4.10    -6.43    -0.24    1.13     -5.27     ... 4.25        \n",
       "3 31158.00   -1.54    0.22     1.19     -0.79    0.73      ... -0.35       \n",
       "4 66866.00   -0.38    1.05     1.26     0.02     0.15      ... 0.03        \n",
       "\n",
       "   V21_orig  V22_orig  V23_orig  V24_orig  V25_orig  V26_orig  V27_orig  \\\n",
       "0 -1.17     0.78      -0.07     -0.32     0.66      -0.05     0.04        \n",
       "1 0.28      0.82      -0.41     0.13      0.68      0.39      -0.17       \n",
       "2 1.35      0.49      4.50      -0.14     0.60      0.64      -0.68       \n",
       "3 -0.08     -0.10     -0.06     0.08      -0.87     0.65      -0.95       \n",
       "4 -0.25     -0.63     0.02      0.00      -0.26     0.10      0.26        \n",
       "\n",
       "   V28_orig  Amount_orig  \n",
       "0 0.16      1.00          \n",
       "1 -0.05     237.00        \n",
       "2 0.15      1895.62       \n",
       "3 0.24      1.00          \n",
       "4 0.09      0.89          \n",
       "\n",
       "[5 rows x 34 columns]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_run_details = test_run.get_details()\n",
    "test_run_predictions = Dataset.get_by_id(ws, test_run_details['outputDatasets'][0]['identifier']['savedId'])\n",
    "test_run_predictions.to_pandas_dataframe().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the fitted model remotely (On-demand)\n",
    "This is testing a model the same way it was tested at the end of the training. However, in this case it's performed on-demand, meaning that you can test the model at any time, not just right after training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:azureml._base_sdk_common._docstring_wrapper:Class ModelProxy: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   Class_orig  Class_predicted  False_predicted_proba  True_predicted_proba  \\\n",
      "0  False       False           0.93                   0.07                    \n",
      "1  False       False           0.91                   0.09                    \n",
      "2  False       False           0.91                   0.09                    \n",
      "3  False       False           0.93                   0.07                    \n",
      "4  False       False           0.86                   0.14                    \n",
      "\n",
      "   Time_orig  V1_orig  V2_orig  V3_orig  V4_orig  V5_orig  ...  V20_orig  \\\n",
      "0 1.00       -0.97    -0.19    1.79     -0.86    -0.01     ... -0.21       \n",
      "1 10.00      1.45     -1.18    0.91     -1.38    -1.97     ... -0.39       \n",
      "2 11.00      1.07     0.29     0.83     2.71     -0.18     ... -0.15       \n",
      "3 13.00      -0.44    0.92     0.92     -0.73    0.92      ... -0.05       \n",
      "4 14.00      -5.40    -5.45    1.19     1.74     3.05      ... -2.20       \n",
      "\n",
      "   V21_orig  V22_orig  V23_orig  V24_orig  V25_orig  V26_orig  V27_orig  \\\n",
      "0 -0.11     0.01      -0.19     -1.18     0.65      -0.22     0.06        \n",
      "1 -0.01     0.31      0.03      0.50      0.25      -0.13     0.04        \n",
      "2 -0.04     0.07      -0.07     0.10      0.55      0.10      0.02        \n",
      "3 -0.19     -0.67     -0.16     -0.89     -0.34     -0.05     0.08        \n",
      "4 -0.50     0.98      2.46      0.04      -0.48     -0.62     0.39        \n",
      "\n",
      "   V28_orig  Amount_orig  \n",
      "0 0.06      123.50        \n",
      "1 0.02      7.80          \n",
      "2 0.02      27.50         \n",
      "3 0.13      0.89          \n",
      "4 0.95      46.80         \n",
      "\n",
      "[5 rows x 34 columns]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>recall_score_weighted</th>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>f1_score_micro</th>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>accuracy</th>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>precision_score_macro</th>\n",
       "      <td>0.89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>f1_score_weighted</th>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>precision_score_micro</th>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>average_precision_score_micro</th>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>balanced_accuracy</th>\n",
       "      <td>0.92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>log_loss</th>\n",
       "      <td>0.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>recall_score_macro</th>\n",
       "      <td>0.92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AUC_weighted</th>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>recall_score_micro</th>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weighted_accuracy</th>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>matthews_correlation</th>\n",
       "      <td>0.82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>confusion_matrix</th>\n",
       "      <td>aml://artifactId/ExperimentRun/dcid.e8cd4d3f-7c27-4b8a-8a7f-f69ee3bcd5b6/confusion_matrix</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>f1_score_macro</th>\n",
       "      <td>0.91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>norm_macro_recall</th>\n",
       "      <td>0.85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>precision_score_weighted</th>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AUC_micro</th>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AUC_macro</th>\n",
       "      <td>0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>average_precision_score_weighted</th>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>average_precision_score_macro</th>\n",
       "      <td>0.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>accuracy_table</th>\n",
       "      <td>aml://artifactId/ExperimentRun/dcid.e8cd4d3f-7c27-4b8a-8a7f-f69ee3bcd5b6/accuracy_table</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                      Value\n",
       "recall_score_weighted            1.00                                                                                      \n",
       "f1_score_micro                   1.00                                                                                      \n",
       "accuracy                         1.00                                                                                      \n",
       "precision_score_macro            0.89                                                                                      \n",
       "f1_score_weighted                1.00                                                                                      \n",
       "precision_score_micro            1.00                                                                                      \n",
       "average_precision_score_micro    1.00                                                                                      \n",
       "balanced_accuracy                0.92                                                                                      \n",
       "log_loss                         0.09                                                                                      \n",
       "recall_score_macro               0.92                                                                                      \n",
       "AUC_weighted                     0.99                                                                                      \n",
       "recall_score_micro               1.00                                                                                      \n",
       "weighted_accuracy                1.00                                                                                      \n",
       "matthews_correlation             0.82                                                                                      \n",
       "confusion_matrix                  aml://artifactId/ExperimentRun/dcid.e8cd4d3f-7c27-4b8a-8a7f-f69ee3bcd5b6/confusion_matrix\n",
       "f1_score_macro                   0.91                                                                                      \n",
       "norm_macro_recall                0.85                                                                                      \n",
       "precision_score_weighted         1.00                                                                                      \n",
       "AUC_micro                        1.00                                                                                      \n",
       "AUC_macro                        0.99                                                                                      \n",
       "average_precision_score_weighted 1.00                                                                                      \n",
       "average_precision_score_macro    0.93                                                                                      \n",
       "accuracy_table                    aml://artifactId/ExperimentRun/dcid.e8cd4d3f-7c27-4b8a-8a7f-f69ee3bcd5b6/accuracy_table  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from azureml.train.automl.model_proxy import ModelProxy\n",
    "\n",
    "model_proxy = ModelProxy(best_run)\n",
    "predictions, test_run_metrics = model_proxy.test(test_data)\n",
    "\n",
    "print(predictions.to_pandas_dataframe().head())\n",
    "pd.DataFrame.from_dict(test_run_metrics, orient='index', columns=['Value'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the fitted model locally\n",
    "\n",
    "Now that the model is trained, split the data in the same way the data was split for training (The difference here is the data is being split locally) and then run the test data through the trained model to get the predicted values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert the test data to dataframe\n",
    "X_test_df = test_data.drop_columns(columns=[label_column_name]).to_pandas_dataframe()\n",
    "y_test_df = test_data.keep_columns(columns=[label_column_name], validate=True).to_pandas_dataframe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# call the predict functions on the model\n",
    "y_pred = fitted_model.predict(X_test_df)\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculate metrics for the prediction\n",
    "\n",
    "Now visualize the data on a scatter plot to show what our truth (actual) values are compared to the predicted values \n",
    "from the trained model that was returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "import numpy as np\n",
    "import itertools\n",
    "\n",
    "cf =confusion_matrix(y_test_df.values,y_pred)\n",
    "plt.imshow(cf,cmap=plt.cm.Blues,interpolation='nearest')\n",
    "plt.colorbar()\n",
    "plt.title('Confusion Matrix')\n",
    "plt.xlabel('Predicted')\n",
    "plt.ylabel('Actual')\n",
    "class_labels = ['False','True']\n",
    "tick_marks = np.arange(len(class_labels))\n",
    "plt.xticks(tick_marks,class_labels)\n",
    "plt.yticks([-0.5,0,1,1.5],['','False','True',''])\n",
    "# plotting text value inside cells\n",
    "thresh = cf.max() / 2.\n",
    "for i,j in itertools.product(range(cf.shape[0]),range(cf.shape[1])):\n",
    "    plt.text(j,i,format(cf[i,j],'d'),horizontalalignment='center',color='white' if cf[i,j] >thresh else 'black')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Acknowledgements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This Credit Card fraud Detection dataset is made available under the Open Database License: http://opendatacommons.org/licenses/odbl/1.0/. Any rights in individual contents of the database are licensed under the Database Contents License: http://opendatacommons.org/licenses/dbcl/1.0/ and is available at: https://www.kaggle.com/mlg-ulb/creditcardfraud\n",
    "\n",
    "The dataset has been collected and analysed during a research collaboration of Worldline and the Machine Learning Group (http://mlg.ulb.ac.be) of ULB (Université Libre de Bruxelles) on big data mining and fraud detection.\n",
    "More details on current and past projects on related topics are available on https://www.researchgate.net/project/Fraud-detection-5 and the page of the DefeatFraud project\n",
    "\n",
    "Please cite the following works:\n",
    "\n",
    "Andrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson and Gianluca Bontempi. Calibrating Probability with Undersampling for Unbalanced Classification. In Symposium on Computational Intelligence and Data Mining (CIDM), IEEE, 2015\n",
    "\n",
    "Dal Pozzolo, Andrea; Caelen, Olivier; Le Borgne, Yann-Ael; Waterschoot, Serge; Bontempi, Gianluca. Learned lessons in credit card fraud detection from a practitioner perspective, Expert systems with applications,41,10,4915-4928,2014, Pergamon\n",
    "\n",
    "Dal Pozzolo, Andrea; Boracchi, Giacomo; Caelen, Olivier; Alippi, Cesare; Bontempi, Gianluca. Credit card fraud detection: a realistic modeling and a novel learning strategy, IEEE transactions on neural networks and learning systems,29,8,3784-3797,2018,IEEE\n",
    "\n",
    "Dal Pozzolo, Andrea Adaptive Machine learning for credit card fraud detection ULB MLG PhD thesis (supervised by G. Bontempi)\n",
    "\n",
    "Carcillo, Fabrizio; Dal Pozzolo, Andrea; Le Borgne, Yann-Aël; Caelen, Olivier; Mazzer, Yannis; Bontempi, Gianluca. Scarff: a scalable framework for streaming credit card fraud detection with Spark, Information fusion,41, 182-194,2018,Elsevier\n",
    "\n",
    "Carcillo, Fabrizio; Le Borgne, Yann-Aël; Caelen, Olivier; Bontempi, Gianluca. Streaming active learning strategies for real-life credit card fraud detection: assessment and visualization, International Journal of Data Science and Analytics, 5,4,285-300,2018,Springer International Publishing\n",
    "\n",
    "Bertrand Lebichot, Yann-Aël Le Borgne, Liyun He, Frederic Oblé, Gianluca Bontempi Deep-Learning Domain Adaptation Techniques for Credit Cards Fraud Detection, INNSBDDL 2019: Recent Advances in Big Data and Deep Learning, pp 78-88, 2019\n",
    "\n",
    "Fabrizio Carcillo, Yann-Aël Le Borgne, Olivier Caelen, Frederic Oblé, Gianluca Bontempi Combining Unsupervised and Supervised Learning in Credit Card Fraud Detection Information Sciences, 2019"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "ratanase"
   }
  ],
  "category": "tutorial",
  "compute": [
   "AML Compute"
  ],
  "datasets": [
   "Creditcard"
  ],
  "deployment": [
   "None"
  ],
  "exclude_from_index": false,
  "file_extension": ".py",
  "framework": [
   "None"
  ],
  "friendly_name": "Classification of credit card fraudulent transactions using Automated ML",
  "index_order": 5,
  "kernel_info": {
   "name": "python3-azureml"
  },
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "mimetype": "text/x-python",
  "name": "python",
  "nbconvert_exporter": "python",
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  },
  "pygments_lexer": "ipython3",
  "tags": [
   "remote_run",
   "AutomatedML"
  ],
  "task": "Classification",
  "version": "3.6.7"
 },
 "nbformat": 4,
 "nbformat_minor": 1
}