Step02_SageMaker_XGBoost_Tuningjob.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用SageMaker+XGBoost，将时间序列转换为监督学习，完成预测性维护的实践\n",
    "https://github.com/liangyimingcom/Use-SageMaker_XGBoost-convert-Time-Series-into-Supervised-Learning-for-predictive-maintenance\n",
    "\n",
    "**关键字**：SageMaker；XGBoost；Python；滑窗；滑动窗口方法；时间序列预测转化为监督学习问题；将多元时间序列数据转换为监督学习问题；如何用Python将时间序列问题转化为有监督学习问题；时间序列预测的机器学习模型；Machine Learning；ML；\n",
    "\n",
    "# SageMaker Hyperparameter Tuning 超参数优化\n",
    "\n",
    "我们已经准备好数据集，就可以训练模型了。\n",
    "在执行此操作之前，要注意的一件事是存在称为“超参数”的算法设置，这些设置可能会严重影响训练后的模型的准确度。\n",
    "例如，XGBoost算法具有几十个超参数，我们需要为这些超参数选择正确的值，以便获得所需的模型训练结果。\n",
    "由于超参数设置复杂并导致了准确度的变化，因此不可能直接得到最佳超参数设置，SageMaker的超参数优化（Hyperparameter_Tuning），以自动的方法搜索到最佳超参数设置。\n",
    "\n",
    "我们将使用SageMaker Hyperparameter Tuning来有效地自动执行搜索最佳参数的过程。具体来说，对于超参数，我们可以指定每个超参数指定一个范围或可能值的列表。 SageMaker超参数优化将自动启动具有不同超参数设置的多个训练作业，基于预定义的“结果指标evaluation metric”评估那些训练作业的结果，并根据先前的结果为以后的尝试选择超参数设置。对于每个超参数调整作业，我们将给它一个预算（最大训练作业数），并且在执行完许多训练作业后该预算将完成。\n",
    "\n",
    "---\n",
    "### 本章节内容Contents\n",
    "\n",
    "1. [算法选择构建模型（XGBoost）]\n",
    "2. [模型训练]\n",
    "3. [超参数优化(反复调参试错)]\n",
    "4. [评估Evaluation]\n",
    "5. [部署模型线上推理]\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker-cn-northwest-1-337575217701\n",
      "xgb-10000871-part_ac.csv-TC58-SW010-WC006-20210405135629\n",
      "TC58-SW010-WC006-20210405151047\n"
     ]
    }
   ],
   "source": [
    "import sagemaker\n",
    "import boto3\n",
    "from sagemaker.tuner import IntegerParameter, CategoricalParameter, ContinuousParameter, HyperparameterTuner\n",
    "\n",
    "import numpy as np                                # For matrix operations and numerical processing\n",
    "import pandas as pd                               # For munging tabular data\n",
    "import os \n",
    "from datetime import datetime\n",
    "from time import gmtime, strftime, sleep\n",
    "\n",
    "region = boto3.Session().region_name    \n",
    "smclient = boto3.Session().client('sagemaker')\n",
    "\n",
    "role = sagemaker.get_execution_role()\n",
    "\n",
    "bucket = sagemaker.Session().default_bucket()\n",
    "\n",
    "# prefix为Step1中，数据预处理后的S3存储桶名称，用于导入训练数据集\n",
    "##prefix = 'xgb-10000871.csv-TC58-SW100-WC10-20201202080714'\n",
    "##prefix = 'xgb-10000871.csv-TC58-SW100-WC20-20201202051756'\n",
    "#prefix = 'xgb-10000871.csv-TC58-SW100-WC40-20201202032834'\n",
    "##prefix = 'xgb-10000871.csv-TC58-SW100-WC60-20201202011847'\n",
    "##prefix = 'xgb-10000871.csv-TC58-SW100-WC80-20201202023456'\n",
    "##prefix = 'xgb-10000871.csv-TC58-SW100-WC99-20201202064356'\n",
    "prefix = 'xgb-10000871-part_ac.csv-TC58-SW010-WC006-20210405135629' ##for demo \n",
    "\n",
    "#获取一个有意义的prefix_tuningjobname，\n",
    "prefix_tuningjobname = prefix[-31:-15]+'-'+strftime(\"%Y%m%d%H%M%S\", gmtime())\n",
    "#prefix_tuningjobname = prefix[-30:-15]#+'-'+strftime(\"%Y%m%d%H%M%S\", gmtime())\n",
    "\n",
    "#for teseting\n",
    "print(bucket) #S3桶名称\n",
    "print(prefix) #S3桶名称\n",
    "print(prefix_tuningjobname) #有意义的直观的训练任务名称\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'get_image_uri' method will be deprecated in favor of 'ImageURIProvider' class in SageMaker Python SDK v2.\n",
      "WARNING:root:There is a more up to date SageMaker XGBoost image. To use the newer image, please set 'repo_version'='1.0-1'. For example:\n",
      "\tget_image_uri(region, 'xgboost', '1.0-1').\n",
      "WARNING:root:Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "\n",
    "sess = sagemaker.Session()\n",
    "\n",
    "container = get_image_uri(region, 'xgboost', repo_version='1')\n",
    "#container = get_image_uri(boto3.Session().region_name, 'xgboost')\n",
    "#display(container)\n",
    "\n",
    "xgb = sagemaker.estimator.Estimator(container,\n",
    "                                    role, \n",
    "                                    train_instance_count=1, \n",
    "                                    train_instance_type='ml.m5.xlarge',\n",
    "                                    output_path='s3://{}/{}/output'.format(bucket, prefix),\n",
    "                                    sagemaker_session=sess)\n",
    "\n",
    "xgb.set_hyperparameters(eval_metric='mae',\n",
    "                        objective='binary:logistic')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### XGBoost使用四个超参数：\n",
    "* * eta *：在更新中使用步长缩小以防止过度拟合。 在每个增强步骤之后，您都可以直接获得新功能的权重。 eta参数实际上缩小了特征权重，以使提升过程更加保守。\n",
    "* * alpha *：关于权重的L1正则化项。 增加此值会使模型更加保守。\n",
    "* * min_child_weight *：子级中实例重量的最小总和（hessian）。 如果树分区步骤导致叶节点的实例权重之和小于min_child_weight，则构建过程将放弃进一步的分区。 在线性回归模型中，这仅对应于每个节点所需的最少实例数。 算法越大，则算法越保守。\n",
    "* * max_depth *：一棵树的最大深度。 增加此值会使模型更复杂，并且可能会过拟合。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparameter_ranges = {'num_round': IntegerParameter(100, 1000)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 评估模型的第一步是要先确立评估的标准——evaluation metric，在这里我们使用XGBoost解决回归问题，选择mae来作为评估标准\n",
    "我们将指定我们想要调整的客观指标及其定义，其中包括从训练任务的CloudWatch日志中提取该指标所需的正则表达式（Regex）。 由于我们在这里使用内置的XGBoost算法，因此它发出两个预定义的指标：* validation：auc *和* train：auc *，我们选择监视* validation：auc *，如下所示。 在这种情况下，我们只需要指定指标名称即可，而无需提供正则表达式。 如果您带上自己的算法，则算法本身会发出指标。 在这种情况下，您需要在此处添加MetricDefinition对象，以通过正则表达式定义那些指标的格式，以便SageMaker知道如何从CloudWatch日志中提取那些指标。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "objective_metric_name = 'validation:mae'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### 创建一个“ HyperparameterTuner”对象，并将其传递给该对象：\n",
    "- The XGBoost estimator we created above\n",
    "- Our hyperparameter ranges\n",
    "- Objective metric name and definition\n",
    "- Tuning resource configurations such as Number of training jobs to run in total and how many training jobs can be run in parallel.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuner = HyperparameterTuner(xgb,\n",
    "                            objective_metric_name,                           \n",
    "                            hyperparameter_ranges,\n",
    "                            objective_type='Minimize',\n",
    "                            base_tuning_job_name=prefix_tuningjobname,\n",
    "                            max_jobs=9,\n",
    "                            max_parallel_jobs=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 运行超参数调优 Hyperparameter_Tuning 任务\n",
    "现在，我们可以通过调用* fit（）*函数来启动超参数调整作业。 创建超参数调整作业后，我们可以转到SageMaker控制台以跟踪超参数调整作业的进度，直到完成为止。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "### for solved issue: <AttributeError: module 'sagemaker' has no attribute 's3_input'> only working on sagemaker=  1.68.0 !!!\n",
    "#!pip list \n",
    "#!pip install sagemaker==1.68.0    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:sagemaker:'s3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.\n",
      "WARNING:sagemaker:'s3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.\n"
     ]
    }
   ],
   "source": [
    "s3_input_train = sagemaker.s3_input(s3_data='s3://{}/{}/train'.format(bucket, prefix), content_type='csv')\n",
    "s3_input_validation = sagemaker.s3_input(s3_data='s3://{}/{}/validation/'.format(bucket, prefix), content_type='csv')\n",
    "\n",
    "tuner.fit({'train': s3_input_train, 'validation': s3_input_validation}, include_cls_metadata=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 让我们对超参数调整作业状态进行 反复 检查，以确保其成功完成。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuningjobname = tuner.latest_tuning_job.job_name\n",
    "#print(tuningjobname) #for testing\n",
    "\n",
    "boto3.client('sagemaker').describe_hyper_parameter_tuning_job(\n",
    "    HyperParameterTuningJobName=tuner.latest_tuning_job.job_name)['HyperParameterTuningJobStatus']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 分析超参数调整作业的结果\n",
    "一旦完成了调优作业，便可以分析结果以了解每个超参数如何影响模型的质量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "import os\n",
    "\n",
    "region = boto3.Session().region_name\n",
    "sage_client = boto3.Session().client('sagemaker')\n",
    "\n",
    "## You must have already run a hyperparameter tuning job to analyze it here.\n",
    "## The Hyperparameter tuning jobs you have run are listed in the Training section on your SageMaker dashboard.\n",
    "## Copy the name of a completed job you want to analyze from that list.\n",
    "## For example: tuning_job_name = 'mxnet-training-201007-0054'.\n",
    "#tuning_job_name = 'YOUR-HYPERPARAMETER-TUNING-JOB-NAME'\n",
    "tuning_job_name = tuningjobname"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 跟踪超参数调整作业进度\n",
    "启动调优作业后，可以通过调用describe_tuning_job API来查看其进度。 describe-tuning-job的输出是一个JSON对象，其中包含有关调整作业的当前状态的信息。 您可以调用list_training_jobs_for_tuning_job来查看调优作业启动的培训作业的详细列表。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9 training jobs have completed\n"
     ]
    }
   ],
   "source": [
    "# run this cell to check current status of hyperparameter tuning job\n",
    "tuning_job_result = sage_client.describe_hyper_parameter_tuning_job(HyperParameterTuningJobName=tuning_job_name)\n",
    "\n",
    "status = tuning_job_result['HyperParameterTuningJobStatus']\n",
    "if status != 'Completed':\n",
    "    print('Reminder: the tuning job has not been completed.')\n",
    "    \n",
    "job_count = tuning_job_result['TrainingJobStatusCounters']['Completed']\n",
    "print(\"%d training jobs have completed\" % job_count)\n",
    "    \n",
    "is_minimize = (tuning_job_result['HyperParameterTuningJobConfig']['HyperParameterTuningJobObjective']['Type'] != 'Maximize')\n",
    "objective_name = tuning_job_result['HyperParameterTuningJobConfig']['HyperParameterTuningJobObjective']['MetricName']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best model found so far:\n",
      "{'CreationTime': datetime.datetime(2021, 4, 5, 14, 47, 30, tzinfo=tzlocal()),\n",
      " 'FinalHyperParameterTuningJobObjectiveMetric': {'MetricName': 'validation:mae',\n",
      "                                                 'Value': 0.13454200327396393},\n",
      " 'ObjectiveStatus': 'Succeeded',\n",
      " 'TrainingEndTime': datetime.datetime(2021, 4, 5, 14, 50, 45, tzinfo=tzlocal()),\n",
      " 'TrainingJobArn': 'arn:aws-cn:sagemaker:cn-northwest-1:337575217701:training-job/tc58-sw010-wc006-210405-1444-005-f9925ce7',\n",
      " 'TrainingJobName': 'TC58-SW010-WC006-210405-1444-005-f9925ce7',\n",
      " 'TrainingJobStatus': 'Completed',\n",
      " 'TrainingStartTime': datetime.datetime(2021, 4, 5, 14, 49, 32, tzinfo=tzlocal()),\n",
      " 'TunedHyperParameters': {'num_round': '517'}}\n"
     ]
    }
   ],
   "source": [
    "from pprint import pprint\n",
    "if tuning_job_result.get('BestTrainingJob',None):\n",
    "    print(\"Best model found so far:\")\n",
    "    pprint(tuning_job_result['BestTrainingJob'])\n",
    "else:\n",
    "    print(\"No training jobs have reported results yet.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 将所有结果提取为DataFrame\n",
    "我们可以列出所有培训工作的超参数和客观指标，并以最佳客观指标来选择培训工作。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training jobs with valid objective: 9\n",
      "{'lowest': 0.13454200327396393, 'highest': 0.13991299271583557}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:13: FutureWarning: Passing a negative integer is deprecated in version 1.0 and will not be supported in future version. Instead, use None to not limit the column width.\n",
      "  del sys.path[0]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>num_round</th>\n",
       "      <th>TrainingJobName</th>\n",
       "      <th>TrainingJobStatus</th>\n",
       "      <th>FinalObjectiveValue</th>\n",
       "      <th>TrainingStartTime</th>\n",
       "      <th>TrainingEndTime</th>\n",
       "      <th>TrainingElapsedTimeSeconds</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>517.0</td>\n",
       "      <td>TC58-SW010-WC006-210405-1444-005-f9925ce7</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.134542</td>\n",
       "      <td>2021-04-05 14:49:32+00:00</td>\n",
       "      <td>2021-04-05 14:50:45+00:00</td>\n",
       "      <td>73.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>522.0</td>\n",
       "      <td>TC58-SW010-WC006-210405-1444-003-ee48a7e9</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.134943</td>\n",
       "      <td>2021-04-05 14:46:28+00:00</td>\n",
       "      <td>2021-04-05 14:47:32+00:00</td>\n",
       "      <td>64.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>354.0</td>\n",
       "      <td>TC58-SW010-WC006-210405-1444-006-698d3951</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.136213</td>\n",
       "      <td>2021-04-05 14:49:51+00:00</td>\n",
       "      <td>2021-04-05 14:50:53+00:00</td>\n",
       "      <td>62.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>506.0</td>\n",
       "      <td>TC58-SW010-WC006-210405-1444-007-7aff253f</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.136344</td>\n",
       "      <td>2021-04-05 14:52:50+00:00</td>\n",
       "      <td>2021-04-05 14:53:54+00:00</td>\n",
       "      <td>64.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>483.0</td>\n",
       "      <td>TC58-SW010-WC006-210405-1444-008-c20c6a15</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.137560</td>\n",
       "      <td>2021-04-05 14:53:10+00:00</td>\n",
       "      <td>2021-04-05 14:54:15+00:00</td>\n",
       "      <td>65.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>408.0</td>\n",
       "      <td>TC58-SW010-WC006-210405-1444-004-2c699c37</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.137659</td>\n",
       "      <td>2021-04-05 14:49:30+00:00</td>\n",
       "      <td>2021-04-05 14:50:18+00:00</td>\n",
       "      <td>48.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>588.0</td>\n",
       "      <td>TC58-SW010-WC006-210405-1444-001-b7524cf3</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.138861</td>\n",
       "      <td>2021-04-05 14:46:17+00:00</td>\n",
       "      <td>2021-04-05 14:47:19+00:00</td>\n",
       "      <td>62.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>144.0</td>\n",
       "      <td>TC58-SW010-WC006-210405-1444-009-8e99177b</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.139664</td>\n",
       "      <td>2021-04-05 14:53:07+00:00</td>\n",
       "      <td>2021-04-05 14:53:50+00:00</td>\n",
       "      <td>43.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>826.0</td>\n",
       "      <td>TC58-SW010-WC006-210405-1444-002-9d602c33</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.139913</td>\n",
       "      <td>2021-04-05 14:46:17+00:00</td>\n",
       "      <td>2021-04-05 14:47:18+00:00</td>\n",
       "      <td>61.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   num_round                            TrainingJobName TrainingJobStatus  \\\n",
       "4  517.0      TC58-SW010-WC006-210405-1444-005-f9925ce7  Completed          \n",
       "6  522.0      TC58-SW010-WC006-210405-1444-003-ee48a7e9  Completed          \n",
       "3  354.0      TC58-SW010-WC006-210405-1444-006-698d3951  Completed          \n",
       "2  506.0      TC58-SW010-WC006-210405-1444-007-7aff253f  Completed          \n",
       "1  483.0      TC58-SW010-WC006-210405-1444-008-c20c6a15  Completed          \n",
       "5  408.0      TC58-SW010-WC006-210405-1444-004-2c699c37  Completed          \n",
       "8  588.0      TC58-SW010-WC006-210405-1444-001-b7524cf3  Completed          \n",
       "0  144.0      TC58-SW010-WC006-210405-1444-009-8e99177b  Completed          \n",
       "7  826.0      TC58-SW010-WC006-210405-1444-002-9d602c33  Completed          \n",
       "\n",
       "   FinalObjectiveValue         TrainingStartTime           TrainingEndTime  \\\n",
       "4  0.134542            2021-04-05 14:49:32+00:00 2021-04-05 14:50:45+00:00   \n",
       "6  0.134943            2021-04-05 14:46:28+00:00 2021-04-05 14:47:32+00:00   \n",
       "3  0.136213            2021-04-05 14:49:51+00:00 2021-04-05 14:50:53+00:00   \n",
       "2  0.136344            2021-04-05 14:52:50+00:00 2021-04-05 14:53:54+00:00   \n",
       "1  0.137560            2021-04-05 14:53:10+00:00 2021-04-05 14:54:15+00:00   \n",
       "5  0.137659            2021-04-05 14:49:30+00:00 2021-04-05 14:50:18+00:00   \n",
       "8  0.138861            2021-04-05 14:46:17+00:00 2021-04-05 14:47:19+00:00   \n",
       "0  0.139664            2021-04-05 14:53:07+00:00 2021-04-05 14:53:50+00:00   \n",
       "7  0.139913            2021-04-05 14:46:17+00:00 2021-04-05 14:47:18+00:00   \n",
       "\n",
       "   TrainingElapsedTimeSeconds  \n",
       "4  73.0                        \n",
       "6  64.0                        \n",
       "3  62.0                        \n",
       "2  64.0                        \n",
       "1  65.0                        \n",
       "5  48.0                        \n",
       "8  62.0                        \n",
       "0  43.0                        \n",
       "7  61.0                        "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "tuner = sagemaker.HyperparameterTuningJobAnalytics(tuning_job_name)\n",
    "\n",
    "full_df = tuner.dataframe()\n",
    "\n",
    "if len(full_df) > 0:\n",
    "    df = full_df[full_df['FinalObjectiveValue'] > -float('inf')]\n",
    "    if len(df) > 0:\n",
    "        df = df.sort_values('FinalObjectiveValue', ascending=is_minimize)\n",
    "        print(\"Number of training jobs with valid objective: %d\" % len(df))\n",
    "        print({\"lowest\":min(df['FinalObjectiveValue']),\"highest\": max(df['FinalObjectiveValue'])})\n",
    "        pd.set_option('display.max_colwidth', -1)  # Don't truncate TrainingJobName        \n",
    "    else:\n",
    "        print(\"No training jobs have reported valid results yet.\")\n",
    "        \n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 查看TuningJob结果与时间的关系\n",
    "\n",
    "接下来，我们将说明随着调整工作的进行，客观指标如何随时间变化。 对于贝叶斯策略，您应该期望看到一个总体趋势，即取得更好的结果，但是这种进展将不会稳定，因为该算法需要平衡参数空间的新区域的探索与已知良好区域的探索之间的平衡。 这样可以使您了解培训工作的数量是否足以满足您搜索空间的复杂性。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div class=\"bk-root\">\n",
       "        <a href=\"https://bokeh.org\" target=\"_blank\" class=\"bk-logo bk-logo-small bk-logo-notebook\"></a>\n",
       "        <span id=\"1184\">Loading BokehJS ...</span>\n",
       "    </div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "(function(root) {\n",
       "  function now() {\n",
       "    return new Date();\n",
       "  }\n",
       "\n",
       "  var force = true;\n",
       "\n",
       "  if (typeof root._bokeh_onload_callbacks === \"undefined\" || force === true) {\n",
       "    root._bokeh_onload_callbacks = [];\n",
       "    root._bokeh_is_loading = undefined;\n",
       "  }\n",
       "\n",
       "  var JS_MIME_TYPE = 'application/javascript';\n",
       "  var HTML_MIME_TYPE = 'text/html';\n",
       "  var EXEC_MIME_TYPE = 'application/vnd.bokehjs_exec.v0+json';\n",
       "  var CLASS_NAME = 'output_bokeh rendered_html';\n",
       "\n",
       "  /**\n",
       "   * Render data to the DOM node\n",
       "   */\n",
       "  function render(props, node) {\n",
       "    var script = document.createElement(\"script\");\n",
       "    node.appendChild(script);\n",
       "  }\n",
       "\n",
       "  /**\n",
       "   * Handle when an output is cleared or removed\n",
       "   */\n",
       "  function handleClearOutput(event, handle) {\n",
       "    var cell = handle.cell;\n",
       "\n",
       "    var id = cell.output_area._bokeh_element_id;\n",
       "    var server_id = cell.output_area._bokeh_server_id;\n",
       "    // Clean up Bokeh references\n",
       "    if (id != null && id in Bokeh.index) {\n",
       "      Bokeh.index[id].model.document.clear();\n",
       "      delete Bokeh.index[id];\n",
       "    }\n",
       "\n",
       "    if (server_id !== undefined) {\n",
       "      // Clean up Bokeh references\n",
       "      var cmd = \"from bokeh.io.state import curstate; print(curstate().uuid_to_server['\" + server_id + \"'].get_sessions()[0].document.roots[0]._id)\";\n",
       "      cell.notebook.kernel.execute(cmd, {\n",
       "        iopub: {\n",
       "          output: function(msg) {\n",
       "            var id = msg.content.text.trim();\n",
       "            if (id in Bokeh.index) {\n",
       "              Bokeh.index[id].model.document.clear();\n",
       "              delete Bokeh.index[id];\n",
       "            }\n",
       "          }\n",
       "        }\n",
       "      });\n",
       "      // Destroy server and session\n",
       "      var cmd = \"import bokeh.io.notebook as ion; ion.destroy_server('\" + server_id + \"')\";\n",
       "      cell.notebook.kernel.execute(cmd);\n",
       "    }\n",
       "  }\n",
       "\n",
       "  /**\n",
       "   * Handle when a new output is added\n",
       "   */\n",
       "  function handleAddOutput(event, handle) {\n",
       "    var output_area = handle.output_area;\n",
       "    var output = handle.output;\n",
       "\n",
       "    // limit handleAddOutput to display_data with EXEC_MIME_TYPE content only\n",
       "    if ((output.output_type != \"display_data\") || (!output.data.hasOwnProperty(EXEC_MIME_TYPE))) {\n",
       "      return\n",
       "    }\n",
       "\n",
       "    var toinsert = output_area.element.find(\".\" + CLASS_NAME.split(' ')[0]);\n",
       "\n",
       "    if (output.metadata[EXEC_MIME_TYPE][\"id\"] !== undefined) {\n",
       "      toinsert[toinsert.length - 1].firstChild.textContent = output.data[JS_MIME_TYPE];\n",
       "      // store reference to embed id on output_area\n",
       "      output_area._bokeh_element_id = output.metadata[EXEC_MIME_TYPE][\"id\"];\n",
       "    }\n",
       "    if (output.metadata[EXEC_MIME_TYPE][\"server_id\"] !== undefined) {\n",
       "      var bk_div = document.createElement(\"div\");\n",
       "      bk_div.innerHTML = output.data[HTML_MIME_TYPE];\n",
       "      var script_attrs = bk_div.children[0].attributes;\n",
       "      for (var i = 0; i < script_attrs.length; i++) {\n",
       "        toinsert[toinsert.length - 1].firstChild.setAttribute(script_attrs[i].name, script_attrs[i].value);\n",
       "      }\n",
       "      // store reference to server id on output_area\n",
       "      output_area._bokeh_server_id = output.metadata[EXEC_MIME_TYPE][\"server_id\"];\n",
       "    }\n",
       "  }\n",
       "\n",
       "  function register_renderer(events, OutputArea) {\n",
       "\n",
       "    function append_mime(data, metadata, element) {\n",
       "      // create a DOM node to render to\n",
       "      var toinsert = this.create_output_subarea(\n",
       "        metadata,\n",
       "        CLASS_NAME,\n",
       "        EXEC_MIME_TYPE\n",
       "      );\n",
       "      this.keyboard_manager.register_events(toinsert);\n",
       "      // Render to node\n",
       "      var props = {data: data, metadata: metadata[EXEC_MIME_TYPE]};\n",
       "      render(props, toinsert[toinsert.length - 1]);\n",
       "      element.append(toinsert);\n",
       "      return toinsert\n",
       "    }\n",
       "\n",
       "    /* Handle when an output is cleared or removed */\n",
       "    events.on('clear_output.CodeCell', handleClearOutput);\n",
       "    events.on('delete.Cell', handleClearOutput);\n",
       "\n",
       "    /* Handle when a new output is added */\n",
       "    events.on('output_added.OutputArea', handleAddOutput);\n",
       "\n",
       "    /**\n",
       "     * Register the mime type and append_mime function with output_area\n",
       "     */\n",
       "    OutputArea.prototype.register_mime_type(EXEC_MIME_TYPE, append_mime, {\n",
       "      /* Is output safe? */\n",
       "      safe: true,\n",
       "      /* Index of renderer in `output_area.display_order` */\n",
       "      index: 0\n",
       "    });\n",
       "  }\n",
       "\n",
       "  // register the mime type if in Jupyter Notebook environment and previously unregistered\n",
       "  if (root.Jupyter !== undefined) {\n",
       "    var events = require('base/js/events');\n",
       "    var OutputArea = require('notebook/js/outputarea').OutputArea;\n",
       "\n",
       "    if (OutputArea.prototype.mime_types().indexOf(EXEC_MIME_TYPE) == -1) {\n",
       "      register_renderer(events, OutputArea);\n",
       "    }\n",
       "  }\n",
       "\n",
       "  \n",
       "  if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n",
       "    root._bokeh_timeout = Date.now() + 5000;\n",
       "    root._bokeh_failed_load = false;\n",
       "  }\n",
       "\n",
       "  var NB_LOAD_WARNING = {'data': {'text/html':\n",
       "     \"<div style='background-color: #fdd'>\\n\"+\n",
       "     \"<p>\\n\"+\n",
       "     \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n",
       "     \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n",
       "     \"</p>\\n\"+\n",
       "     \"<ul>\\n\"+\n",
       "     \"<li>re-rerun `output_notebook()` to attempt to load from CDN again, or</li>\\n\"+\n",
       "     \"<li>use INLINE resources instead, as so:</li>\\n\"+\n",
       "     \"</ul>\\n\"+\n",
       "     \"<code>\\n\"+\n",
       "     \"from bokeh.resources import INLINE\\n\"+\n",
       "     \"output_notebook(resources=INLINE)\\n\"+\n",
       "     \"</code>\\n\"+\n",
       "     \"</div>\"}};\n",
       "\n",
       "  function display_loaded() {\n",
       "    var el = document.getElementById(\"1184\");\n",
       "    if (el != null) {\n",
       "      el.textContent = \"BokehJS is loading...\";\n",
       "    }\n",
       "    if (root.Bokeh !== undefined) {\n",
       "      if (el != null) {\n",
       "        el.textContent = \"BokehJS \" + root.Bokeh.version + \" successfully loaded.\";\n",
       "      }\n",
       "    } else if (Date.now() < root._bokeh_timeout) {\n",
       "      setTimeout(display_loaded, 100)\n",
       "    }\n",
       "  }\n",
       "\n",
       "\n",
       "  function run_callbacks() {\n",
       "    try {\n",
       "      root._bokeh_onload_callbacks.forEach(function(callback) {\n",
       "        if (callback != null)\n",
       "          callback();\n",
       "      });\n",
       "    } finally {\n",
       "      delete root._bokeh_onload_callbacks\n",
       "    }\n",
       "    console.debug(\"Bokeh: all callbacks have finished\");\n",
       "  }\n",
       "\n",
       "  function load_libs(css_urls, js_urls, callback) {\n",
       "    if (css_urls == null) css_urls = [];\n",
       "    if (js_urls == null) js_urls = [];\n",
       "\n",
       "    root._bokeh_onload_callbacks.push(callback);\n",
       "    if (root._bokeh_is_loading > 0) {\n",
       "      console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n",
       "      return null;\n",
       "    }\n",
       "    if (js_urls == null || js_urls.length === 0) {\n",
       "      run_callbacks();\n",
       "      return null;\n",
       "    }\n",
       "    console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n",
       "    root._bokeh_is_loading = css_urls.length + js_urls.length;\n",
       "\n",
       "    function on_load() {\n",
       "      root._bokeh_is_loading--;\n",
       "      if (root._bokeh_is_loading === 0) {\n",
       "        console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n",
       "        run_callbacks()\n",
       "      }\n",
       "    }\n",
       "\n",
       "    function on_error() {\n",
       "      console.error(\"failed to load \" + url);\n",
       "    }\n",
       "\n",
       "    for (var i = 0; i < css_urls.length; i++) {\n",
       "      var url = css_urls[i];\n",
       "      const element = document.createElement(\"link\");\n",
       "      element.onload = on_load;\n",
       "      element.onerror = on_error;\n",
       "      element.rel = \"stylesheet\";\n",
       "      element.type = \"text/css\";\n",
       "      element.href = url;\n",
       "      console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n",
       "      document.body.appendChild(element);\n",
       "    }\n",
       "\n",
       "    for (var i = 0; i < js_urls.length; i++) {\n",
       "      var url = js_urls[i];\n",
       "      var element = document.createElement('script');\n",
       "      element.onload = on_load;\n",
       "      element.onerror = on_error;\n",
       "      element.async = false;\n",
       "      element.src = url;\n",
       "      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n",
       "      document.head.appendChild(element);\n",
       "    }\n",
       "  };var element = document.getElementById(\"1184\");\n",
       "  if (element == null) {\n",
       "    console.error(\"Bokeh: ERROR: autoload.js configured with elementid '1184' but no matching script tag was found. \")\n",
       "    return false;\n",
       "  }\n",
       "\n",
       "  function inject_raw_css(css) {\n",
       "    const element = document.createElement(\"style\");\n",
       "    element.appendChild(document.createTextNode(css));\n",
       "    document.body.appendChild(element);\n",
       "  }\n",
       "\n",
       "  \n",
       "  var js_urls = [\"https://cdn.pydata.org/bokeh/release/bokeh-1.4.0.min.js\", \"https://cdn.pydata.org/bokeh/release/bokeh-widgets-1.4.0.min.js\", \"https://cdn.pydata.org/bokeh/release/bokeh-tables-1.4.0.min.js\", \"https://cdn.pydata.org/bokeh/release/bokeh-gl-1.4.0.min.js\"];\n",
       "  var css_urls = [];\n",
       "  \n",
       "\n",
       "  var inline_js = [\n",
       "    function(Bokeh) {\n",
       "      Bokeh.set_log_level(\"info\");\n",
       "    },\n",
       "    function(Bokeh) {\n",
       "    \n",
       "    \n",
       "    }\n",
       "  ];\n",
       "\n",
       "  function run_inline_js() {\n",
       "    \n",
       "    if (root.Bokeh !== undefined || force === true) {\n",
       "      \n",
       "    for (var i = 0; i < inline_js.length; i++) {\n",
       "      inline_js[i].call(root, root.Bokeh);\n",
       "    }\n",
       "    if (force === true) {\n",
       "        display_loaded();\n",
       "      }} else if (Date.now() < root._bokeh_timeout) {\n",
       "      setTimeout(run_inline_js, 100);\n",
       "    } else if (!root._bokeh_failed_load) {\n",
       "      console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n",
       "      root._bokeh_failed_load = true;\n",
       "    } else if (force !== true) {\n",
       "      var cell = $(document.getElementById(\"1184\")).parents('.cell').data().cell;\n",
       "      cell.output_area.append_execute_result(NB_LOAD_WARNING)\n",
       "    }\n",
       "\n",
       "  }\n",
       "\n",
       "  if (root._bokeh_is_loading === 0) {\n",
       "    console.debug(\"Bokeh: BokehJS loaded, going straight to plotting\");\n",
       "    run_inline_js();\n",
       "  } else {\n",
       "    load_libs(css_urls, js_urls, function() {\n",
       "      console.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n",
       "      run_inline_js();\n",
       "    });\n",
       "  }\n",
       "}(window));"
      ],
      "application/vnd.bokehjs_load.v0+json": "\n(function(root) {\n  function now() {\n    return new Date();\n  }\n\n  var force = true;\n\n  if (typeof root._bokeh_onload_callbacks === \"undefined\" || force === true) {\n    root._bokeh_onload_callbacks = [];\n    root._bokeh_is_loading = undefined;\n  }\n\n  \n\n  \n  if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n    root._bokeh_timeout = Date.now() + 5000;\n    root._bokeh_failed_load = false;\n  }\n\n  var NB_LOAD_WARNING = {'data': {'text/html':\n     \"<div style='background-color: #fdd'>\\n\"+\n     \"<p>\\n\"+\n     \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n     \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n     \"</p>\\n\"+\n     \"<ul>\\n\"+\n     \"<li>re-rerun `output_notebook()` to attempt to load from CDN again, or</li>\\n\"+\n     \"<li>use INLINE resources instead, as so:</li>\\n\"+\n     \"</ul>\\n\"+\n     \"<code>\\n\"+\n     \"from bokeh.resources import INLINE\\n\"+\n     \"output_notebook(resources=INLINE)\\n\"+\n     \"</code>\\n\"+\n     \"</div>\"}};\n\n  function display_loaded() {\n    var el = document.getElementById(\"1184\");\n    if (el != null) {\n      el.textContent = \"BokehJS is loading...\";\n    }\n    if (root.Bokeh !== undefined) {\n      if (el != null) {\n        el.textContent = \"BokehJS \" + root.Bokeh.version + \" successfully loaded.\";\n      }\n    } else if (Date.now() < root._bokeh_timeout) {\n      setTimeout(display_loaded, 100)\n    }\n  }\n\n\n  function run_callbacks() {\n    try {\n      root._bokeh_onload_callbacks.forEach(function(callback) {\n        if (callback != null)\n          callback();\n      });\n    } finally {\n      delete root._bokeh_onload_callbacks\n    }\n    console.debug(\"Bokeh: all callbacks have finished\");\n  }\n\n  function load_libs(css_urls, js_urls, callback) {\n    if (css_urls == null) css_urls = [];\n    if (js_urls == null) js_urls = [];\n\n    root._bokeh_onload_callbacks.push(callback);\n    if (root._bokeh_is_loading > 0) {\n      console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n      return null;\n    }\n    if (js_urls == null || js_urls.length === 0) {\n      run_callbacks();\n      return null;\n    }\n    console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n    root._bokeh_is_loading = css_urls.length + js_urls.length;\n\n    function on_load() {\n      root._bokeh_is_loading--;\n      if (root._bokeh_is_loading === 0) {\n        console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n        run_callbacks()\n      }\n    }\n\n    function on_error() {\n      console.error(\"failed to load \" + url);\n    }\n\n    for (var i = 0; i < css_urls.length; i++) {\n      var url = css_urls[i];\n      const element = document.createElement(\"link\");\n      element.onload = on_load;\n      element.onerror = on_error;\n      element.rel = \"stylesheet\";\n      element.type = \"text/css\";\n      element.href = url;\n      console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n      document.body.appendChild(element);\n    }\n\n    for (var i = 0; i < js_urls.length; i++) {\n      var url = js_urls[i];\n      var element = document.createElement('script');\n      element.onload = on_load;\n      element.onerror = on_error;\n      element.async = false;\n      element.src = url;\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      document.head.appendChild(element);\n    }\n  };var element = document.getElementById(\"1184\");\n  if (element == null) {\n    console.error(\"Bokeh: ERROR: autoload.js configured with elementid '1184' but no matching script tag was found. \")\n    return false;\n  }\n\n  function inject_raw_css(css) {\n    const element = document.createElement(\"style\");\n    element.appendChild(document.createTextNode(css));\n    document.body.appendChild(element);\n  }\n\n  \n  var js_urls = [\"https://cdn.pydata.org/bokeh/release/bokeh-1.4.0.min.js\", \"https://cdn.pydata.org/bokeh/release/bokeh-widgets-1.4.0.min.js\", \"https://cdn.pydata.org/bokeh/release/bokeh-tables-1.4.0.min.js\", \"https://cdn.pydata.org/bokeh/release/bokeh-gl-1.4.0.min.js\"];\n  var css_urls = [];\n  \n\n  var inline_js = [\n    function(Bokeh) {\n      Bokeh.set_log_level(\"info\");\n    },\n    function(Bokeh) {\n    \n    \n    }\n  ];\n\n  function run_inline_js() {\n    \n    if (root.Bokeh !== undefined || force === true) {\n      \n    for (var i = 0; i < inline_js.length; i++) {\n      inline_js[i].call(root, root.Bokeh);\n    }\n    if (force === true) {\n        display_loaded();\n      }} else if (Date.now() < root._bokeh_timeout) {\n      setTimeout(run_inline_js, 100);\n    } else if (!root._bokeh_failed_load) {\n      console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n      root._bokeh_failed_load = true;\n    } else if (force !== true) {\n      var cell = $(document.getElementById(\"1184\")).parents('.cell').data().cell;\n      cell.output_area.append_execute_result(NB_LOAD_WARNING)\n    }\n\n  }\n\n  if (root._bokeh_is_loading === 0) {\n    console.debug(\"Bokeh: BokehJS loaded, going straight to plotting\");\n    run_inline_js();\n  } else {\n    load_libs(css_urls, js_urls, function() {\n      console.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n      run_inline_js();\n    });\n  }\n}(window));"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "  <div class=\"bk-root\" id=\"43fdf4c5-c67d-4193-8764-4891eed355c0\" data-root-id=\"1186\"></div>\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "(function(root) {\n",
       "  function embed_document(root) {\n",
       "    \n",
       "  var docs_json = {\"d16dc644-8014-4c50-ae04-51300c6f0a92\":{\"roots\":{\"references\":[{\"attributes\":{\"below\":[{\"id\":\"1195\",\"type\":\"DatetimeAxis\"}],\"center\":[{\"id\":\"1199\",\"type\":\"Grid\"},{\"id\":\"1204\",\"type\":\"Grid\"}],\"left\":[{\"id\":\"1200\",\"type\":\"LinearAxis\"}],\"plot_height\":400,\"plot_width\":900,\"renderers\":[{\"id\":\"1225\",\"type\":\"GlyphRenderer\"}],\"title\":{\"id\":\"1248\",\"type\":\"Title\"},\"toolbar\":{\"id\":\"1212\",\"type\":\"Toolbar\"},\"x_range\":{\"id\":\"1187\",\"type\":\"DataRange1d\"},\"x_scale\":{\"id\":\"1191\",\"type\":\"LinearScale\"},\"y_range\":{\"id\":\"1189\",\"type\":\"DataRange1d\"},\"y_scale\":{\"id\":\"1193\",\"type\":\"LinearScale\"}},\"id\":\"1186\",\"subtype\":\"Figure\",\"type\":\"Plot\"},{\"attributes\":{\"months\":[0,4,8]},\"id\":\"1264\",\"type\":\"MonthsTicker\"},{\"attributes\":{},\"id\":\"1253\",\"type\":\"UnionRenderers\"},{\"attributes\":{\"months\":[0,6]},\"id\":\"1265\",\"type\":\"MonthsTicker\"},{\"attributes\":{\"formatter\":{\"id\":\"1249\",\"type\":\"BasicTickFormatter\"},\"ticker\":{\"id\":\"1201\",\"type\":\"BasicTicker\"}},\"id\":\"1200\",\"type\":\"LinearAxis\"},{\"attributes\":{},\"id\":\"1251\",\"type\":\"DatetimeTickFormatter\"},{\"attributes\":{\"mantissas\":[1,2,5],\"max_interval\":500.0,\"num_minor_ticks\":0},\"id\":\"1255\",\"type\":\"AdaptiveTicker\"},{\"attributes\":{},\"id\":\"1266\",\"type\":\"YearsTicker\"},{\"attributes\":{\"text\":\"\"},\"id\":\"1248\",\"type\":\"Title\"},{\"attributes\":{},\"id\":\"1249\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{\"callback\":null},\"id\":\"1189\",\"type\":\"DataRange1d\"},{\"attributes\":{},\"id\":\"1205\",\"type\":\"PanTool\"},{\"attributes\":{},\"id\":\"1207\",\"type\":\"WheelZoomTool\"},{\"attributes\":{\"base\":60,\"mantissas\":[1,2,5,10,15,20,30],\"max_interval\":1800000.0,\"min_interval\":1000.0,\"num_minor_ticks\":0},\"id\":\"1256\",\"type\":\"AdaptiveTicker\"},{\"attributes\":{\"active_drag\":\"auto\",\"active_inspect\":\"auto\",\"active_multi\":null,\"active_scroll\":\"auto\",\"active_tap\":\"auto\",\"tools\":[{\"id\":\"1185\",\"type\":\"HoverTool\"},{\"id\":\"1205\",\"type\":\"PanTool\"},{\"id\":\"1206\",\"type\":\"CrosshairTool\"},{\"id\":\"1207\",\"type\":\"WheelZoomTool\"},{\"id\":\"1208\",\"type\":\"ZoomInTool\"},{\"id\":\"1209\",\"type\":\"ZoomOutTool\"},{\"id\":\"1210\",\"type\":\"UndoTool\"},{\"id\":\"1211\",\"type\":\"ResetTool\"}]},\"id\":\"1212\",\"type\":\"Toolbar\"},{\"attributes\":{\"num_minor_ticks\":5,\"tickers\":[{\"id\":\"1255\",\"type\":\"AdaptiveTicker\"},{\"id\":\"1256\",\"type\":\"AdaptiveTicker\"},{\"id\":\"1257\",\"type\":\"AdaptiveTicker\"},{\"id\":\"1258\",\"type\":\"DaysTicker\"},{\"id\":\"1259\",\"type\":\"DaysTicker\"},{\"id\":\"1260\",\"type\":\"DaysTicker\"},{\"id\":\"1261\",\"type\":\"DaysTicker\"},{\"id\":\"1262\",\"type\":\"MonthsTicker\"},{\"id\":\"1263\",\"type\":\"MonthsTicker\"},{\"id\":\"1264\",\"type\":\"MonthsTicker\"},{\"id\":\"1265\",\"type\":\"MonthsTicker\"},{\"id\":\"1266\",\"type\":\"YearsTicker\"}]},\"id\":\"1196\",\"type\":\"DatetimeTicker\"},{\"attributes\":{\"callback\":null,\"data\":{\"FinalObjectiveValue\":{\"__ndarray__\":\"AAAAIKw4wT8AAADgz0XBPwAAAIBtb8E/AAAAYLhzwT8AAADgkJvBPwAAAGDPnsE/AAAAgDLGwT8AAACAguDBPwAAAECr6ME/\",\"dtype\":\"float64\",\"shape\":[9]},\"TrainingElapsedTimeSeconds\":{\"__ndarray__\":\"AAAAAABAUkAAAAAAAABQQAAAAAAAAE9AAAAAAAAAUEAAAAAAAEBQQAAAAAAAAEhAAAAAAAAAT0AAAAAAAIBFQAAAAAAAgE5A\",\"dtype\":\"float64\",\"shape\":[9]},\"TrainingEndTime\":{\"__ndarray__\":\"AICYOyiKd0IAAHoMKIp3QgCAjD0oindCAAC9aSiKd0IAgN1uKIp3QgAAATUoindCAIBNCSiKd0IAAMNoKIp3QgAADwkoindC\",\"dtype\":\"float64\",\"shape\":[9]},\"TrainingJobName\":[\"TC58-SW010-WC006-210405-1444-005-f9925ce7\",\"TC58-SW010-WC006-210405-1444-003-ee48a7e9\",\"TC58-SW010-WC006-210405-1444-006-698d3951\",\"TC58-SW010-WC006-210405-1444-007-7aff253f\",\"TC58-SW010-WC006-210405-1444-008-c20c6a15\",\"TC58-SW010-WC006-210405-1444-004-2c699c37\",\"TC58-SW010-WC006-210405-1444-001-b7524cf3\",\"TC58-SW010-WC006-210405-1444-009-8e99177b\",\"TC58-SW010-WC006-210405-1444-002-9d602c33\"],\"TrainingJobStatus\":[\"Completed\",\"Completed\",\"Completed\",\"Completed\",\"Completed\",\"Completed\",\"Completed\",\"Completed\",\"Completed\"],\"TrainingStartTime\":{\"__ndarray__\":\"AADGKSiKd0IAANr8J4p3QgCAaS4oindCAAAdWiiKd0IAAP9eKIp3QgAASSkoindCAIAq+ieKd0IAgENeKIp3QgCAKvonindC\",\"dtype\":\"float64\",\"shape\":[9]},\"index\":[4,6,3,2,1,5,8,0,7],\"num_round\":{\"__ndarray__\":\"AAAAAAAogEAAAAAAAFCAQAAAAAAAIHZAAAAAAACgf0AAAAAAADB+QAAAAAAAgHlAAAAAAABggkAAAAAAAABiQAAAAAAA0IlA\",\"dtype\":\"float64\",\"shape\":[9]}},\"selected\":{\"id\":\"1254\",\"type\":\"Selection\"},\"selection_policy\":{\"id\":\"1253\",\"type\":\"UnionRenderers\"}},\"id\":\"1221\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"base\":24,\"mantissas\":[1,2,4,6,8,12],\"max_interval\":43200000.0,\"min_interval\":3600000.0,\"num_minor_ticks\":0},\"id\":\"1257\",\"type\":\"AdaptiveTicker\"},{\"attributes\":{},\"id\":\"1254\",\"type\":\"Selection\"},{\"attributes\":{\"dimension\":1,\"ticker\":{\"id\":\"1201\",\"type\":\"BasicTicker\"}},\"id\":\"1204\",\"type\":\"Grid\"},{\"attributes\":{\"callback\":null},\"id\":\"1187\",\"type\":\"DataRange1d\"},{\"attributes\":{},\"id\":\"1206\",\"type\":\"CrosshairTool\"},{\"attributes\":{\"source\":{\"id\":\"1221\",\"type\":\"ColumnDataSource\"}},\"id\":\"1226\",\"type\":\"CDSView\"},{\"attributes\":{\"days\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]},\"id\":\"1258\",\"type\":\"DaysTicker\"},{\"attributes\":{},\"id\":\"1210\",\"type\":\"UndoTool\"},{\"attributes\":{},\"id\":\"1208\",\"type\":\"ZoomInTool\"},{\"attributes\":{\"callback\":null,\"tooltips\":[[\"FinalObjectiveValue\",\"@FinalObjectiveValue\"],[\"TrainingJobName\",\"@TrainingJobName\"],[\"num_round\",\"@{num_round}\"]]},\"id\":\"1185\",\"type\":\"HoverTool\"},{\"attributes\":{},\"id\":\"1209\",\"type\":\"ZoomOutTool\"},{\"attributes\":{\"months\":[0,2,4,6,8,10]},\"id\":\"1263\",\"type\":\"MonthsTicker\"},{\"attributes\":{},\"id\":\"1201\",\"type\":\"BasicTicker\"},{\"attributes\":{\"days\":[1,4,7,10,13,16,19,22,25,28]},\"id\":\"1259\",\"type\":\"DaysTicker\"},{\"attributes\":{\"months\":[0,1,2,3,4,5,6,7,8,9,10,11]},\"id\":\"1262\",\"type\":\"MonthsTicker\"},{\"attributes\":{\"data_source\":{\"id\":\"1221\",\"type\":\"ColumnDataSource\"},\"glyph\":{\"id\":\"1223\",\"type\":\"Circle\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1224\",\"type\":\"Circle\"},\"selection_glyph\":null,\"view\":{\"id\":\"1226\",\"type\":\"CDSView\"}},\"id\":\"1225\",\"type\":\"GlyphRenderer\"},{\"attributes\":{\"fill_color\":{\"value\":\"#1f77b4\"},\"line_color\":{\"value\":\"#1f77b4\"},\"x\":{\"field\":\"TrainingStartTime\"},\"y\":{\"field\":\"FinalObjectiveValue\"}},\"id\":\"1223\",\"type\":\"Circle\"},{\"attributes\":{},\"id\":\"1191\",\"type\":\"LinearScale\"},{\"attributes\":{\"ticker\":{\"id\":\"1196\",\"type\":\"DatetimeTicker\"}},\"id\":\"1199\",\"type\":\"Grid\"},{\"attributes\":{\"days\":[1,8,15,22]},\"id\":\"1260\",\"type\":\"DaysTicker\"},{\"attributes\":{\"formatter\":{\"id\":\"1251\",\"type\":\"DatetimeTickFormatter\"},\"ticker\":{\"id\":\"1196\",\"type\":\"DatetimeTicker\"}},\"id\":\"1195\",\"type\":\"DatetimeAxis\"},{\"attributes\":{\"fill_alpha\":{\"value\":0.1},\"fill_color\":{\"value\":\"#1f77b4\"},\"line_alpha\":{\"value\":0.1},\"line_color\":{\"value\":\"#1f77b4\"},\"x\":{\"field\":\"TrainingStartTime\"},\"y\":{\"field\":\"FinalObjectiveValue\"}},\"id\":\"1224\",\"type\":\"Circle\"},{\"attributes\":{},\"id\":\"1211\",\"type\":\"ResetTool\"},{\"attributes\":{\"days\":[1,15]},\"id\":\"1261\",\"type\":\"DaysTicker\"},{\"attributes\":{},\"id\":\"1193\",\"type\":\"LinearScale\"}],\"root_ids\":[\"1186\"]},\"title\":\"Bokeh Application\",\"version\":\"1.4.0\"}};\n",
       "  var render_items = [{\"docid\":\"d16dc644-8014-4c50-ae04-51300c6f0a92\",\"roots\":{\"1186\":\"43fdf4c5-c67d-4193-8764-4891eed355c0\"}}];\n",
       "  root.Bokeh.embed.embed_items_notebook(docs_json, render_items);\n",
       "\n",
       "  }\n",
       "  if (root.Bokeh !== undefined) {\n",
       "    embed_document(root);\n",
       "  } else {\n",
       "    var attempts = 0;\n",
       "    var timer = setInterval(function(root) {\n",
       "      if (root.Bokeh !== undefined) {\n",
       "        clearInterval(timer);\n",
       "        embed_document(root);\n",
       "      } else {\n",
       "        attempts++;\n",
       "        if (attempts > 100) {\n",
       "          clearInterval(timer);\n",
       "          console.log(\"Bokeh: ERROR: Unable to run BokehJS code because BokehJS library is missing\");\n",
       "        }\n",
       "      }\n",
       "    }, 10, root)\n",
       "  }\n",
       "})(window);"
      ],
      "application/vnd.bokehjs_exec.v0+json": ""
     },
     "metadata": {
      "application/vnd.bokehjs_exec.v0+json": {
       "id": "1186"
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#!pip install bokeh # 如果报错\n",
    "import bokeh\n",
    "import bokeh.io\n",
    "bokeh.io.output_notebook()\n",
    "from bokeh.plotting import figure, show\n",
    "from bokeh.models import HoverTool\n",
    "\n",
    "class HoverHelper():\n",
    "\n",
    "    def __init__(self, tuning_analytics):\n",
    "        self.tuner = tuning_analytics\n",
    "\n",
    "    def hovertool(self):\n",
    "        tooltips = [\n",
    "            (\"FinalObjectiveValue\", \"@FinalObjectiveValue\"),\n",
    "            (\"TrainingJobName\", \"@TrainingJobName\"),\n",
    "        ]\n",
    "        for k in self.tuner.tuning_ranges.keys():\n",
    "            tooltips.append( (k, \"@{%s}\" % k) )\n",
    "\n",
    "        ht = HoverTool(tooltips=tooltips)\n",
    "        return ht\n",
    "\n",
    "    def tools(self, standard_tools='pan,crosshair,wheel_zoom,zoom_in,zoom_out,undo,reset'):\n",
    "        return [self.hovertool(), standard_tools]\n",
    "\n",
    "hover = HoverHelper(tuner)\n",
    "\n",
    "p = figure(plot_width=900, plot_height=400, tools=hover.tools(), x_axis_type='datetime')\n",
    "p.circle(source=df, x='TrainingStartTime', y='FinalObjectiveValue')\n",
    "show(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 分析客观指标和单个超参数之间的相关性\n",
    "您已经完成了调优工作，您可能想知道目标指标和您选择要调整的单个超参数之间的相关性。\n",
    "有了这些见识，可以帮助您确定为某些超参数调整搜索范围并开始另一项调优工作是否有意义。\n",
    "例如，如果您看到客观指标和数值超参数之间呈正趋势，则可能要在下一个调整工作中为该超参数设置更高的调整范围。\n",
    "\n",
    "下面的单元格为每个超参数绘制一个图形，以显示其与目标指标的相关性。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "\n",
       "  <div class=\"bk-root\" id=\"e276403a-03c7-4ed4-aa1d-1169cb972c41\" data-root-id=\"1431\"></div>\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "(function(root) {\n",
       "  function embed_document(root) {\n",
       "    \n",
       "  var docs_json = {\"c8b332a0-46c2-484c-8616-981e15d306ba\":{\"roots\":{\"references\":[{\"attributes\":{\"children\":[{\"id\":\"1388\",\"subtype\":\"Figure\",\"type\":\"Plot\"}]},\"id\":\"1431\",\"type\":\"Column\"},{\"attributes\":{},\"id\":\"1397\",\"type\":\"LinearScale\"},{\"attributes\":{\"dimension\":1,\"ticker\":{\"id\":\"1405\",\"type\":\"BasicTicker\"}},\"id\":\"1408\",\"type\":\"Grid\"},{\"attributes\":{},\"id\":\"1473\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{},\"id\":\"1414\",\"type\":\"UndoTool\"},{\"attributes\":{\"axis_label\":\"num_round\",\"formatter\":{\"id\":\"1475\",\"type\":\"BasicTickFormatter\"},\"ticker\":{\"id\":\"1400\",\"type\":\"BasicTicker\"}},\"id\":\"1399\",\"type\":\"LinearAxis\"},{\"attributes\":{},\"id\":\"1415\",\"type\":\"ResetTool\"},{\"attributes\":{},\"id\":\"1413\",\"type\":\"ZoomOutTool\"},{\"attributes\":{\"ticker\":{\"id\":\"1400\",\"type\":\"BasicTicker\"}},\"id\":\"1403\",\"type\":\"Grid\"},{\"attributes\":{},\"id\":\"1478\",\"type\":\"Selection\"},{\"attributes\":{\"fill_color\":{\"value\":\"#1f77b4\"},\"line_color\":{\"value\":\"#1f77b4\"},\"x\":{\"field\":\"num_round\"},\"y\":{\"field\":\"FinalObjectiveValue\"}},\"id\":\"1427\",\"type\":\"Circle\"},{\"attributes\":{},\"id\":\"1405\",\"type\":\"BasicTicker\"},{\"attributes\":{\"axis_label\":\"validation:mae\",\"formatter\":{\"id\":\"1473\",\"type\":\"BasicTickFormatter\"},\"ticker\":{\"id\":\"1405\",\"type\":\"BasicTicker\"}},\"id\":\"1404\",\"type\":\"LinearAxis\"},{\"attributes\":{\"source\":{\"id\":\"1425\",\"type\":\"ColumnDataSource\"}},\"id\":\"1430\",\"type\":\"CDSView\"},{\"attributes\":{},\"id\":\"1477\",\"type\":\"UnionRenderers\"},{\"attributes\":{\"callback\":null,\"tooltips\":[[\"FinalObjectiveValue\",\"@FinalObjectiveValue\"],[\"TrainingJobName\",\"@TrainingJobName\"],[\"num_round\",\"@{num_round}\"]]},\"id\":\"1387\",\"type\":\"HoverTool\"},{\"attributes\":{\"below\":[{\"id\":\"1399\",\"type\":\"LinearAxis\"}],\"center\":[{\"id\":\"1403\",\"type\":\"Grid\"},{\"id\":\"1408\",\"type\":\"Grid\"}],\"left\":[{\"id\":\"1404\",\"type\":\"LinearAxis\"}],\"plot_height\":500,\"plot_width\":500,\"renderers\":[{\"id\":\"1429\",\"type\":\"GlyphRenderer\"}],\"title\":{\"id\":\"1389\",\"type\":\"Title\"},\"toolbar\":{\"id\":\"1416\",\"type\":\"Toolbar\"},\"x_range\":{\"id\":\"1391\",\"type\":\"DataRange1d\"},\"x_scale\":{\"id\":\"1395\",\"type\":\"LinearScale\"},\"y_range\":{\"id\":\"1393\",\"type\":\"DataRange1d\"},\"y_scale\":{\"id\":\"1397\",\"type\":\"LinearScale\"}},\"id\":\"1388\",\"subtype\":\"Figure\",\"type\":\"Plot\"},{\"attributes\":{\"active_drag\":\"auto\",\"active_inspect\":\"auto\",\"active_multi\":null,\"active_scroll\":\"auto\",\"active_tap\":\"auto\",\"tools\":[{\"id\":\"1387\",\"type\":\"HoverTool\"},{\"id\":\"1409\",\"type\":\"PanTool\"},{\"id\":\"1410\",\"type\":\"CrosshairTool\"},{\"id\":\"1411\",\"type\":\"WheelZoomTool\"},{\"id\":\"1412\",\"type\":\"ZoomInTool\"},{\"id\":\"1413\",\"type\":\"ZoomOutTool\"},{\"id\":\"1414\",\"type\":\"UndoTool\"},{\"id\":\"1415\",\"type\":\"ResetTool\"}]},\"id\":\"1416\",\"type\":\"Toolbar\"},{\"attributes\":{\"callback\":null,\"data\":{\"FinalObjectiveValue\":{\"__ndarray__\":\"AAAAIKw4wT8AAADgz0XBPwAAAIBtb8E/AAAAYLhzwT8AAADgkJvBPwAAAGDPnsE/AAAAgDLGwT8AAACAguDBPwAAAECr6ME/\",\"dtype\":\"float64\",\"shape\":[9]},\"TrainingElapsedTimeSeconds\":{\"__ndarray__\":\"AAAAAABAUkAAAAAAAABQQAAAAAAAAE9AAAAAAAAAUEAAAAAAAEBQQAAAAAAAAEhAAAAAAAAAT0AAAAAAAIBFQAAAAAAAgE5A\",\"dtype\":\"float64\",\"shape\":[9]},\"TrainingEndTime\":{\"__ndarray__\":\"AICYOyiKd0IAAHoMKIp3QgCAjD0oindCAAC9aSiKd0IAgN1uKIp3QgAAATUoindCAIBNCSiKd0IAAMNoKIp3QgAADwkoindC\",\"dtype\":\"float64\",\"shape\":[9]},\"TrainingJobName\":[\"TC58-SW010-WC006-210405-1444-005-f9925ce7\",\"TC58-SW010-WC006-210405-1444-003-ee48a7e9\",\"TC58-SW010-WC006-210405-1444-006-698d3951\",\"TC58-SW010-WC006-210405-1444-007-7aff253f\",\"TC58-SW010-WC006-210405-1444-008-c20c6a15\",\"TC58-SW010-WC006-210405-1444-004-2c699c37\",\"TC58-SW010-WC006-210405-1444-001-b7524cf3\",\"TC58-SW010-WC006-210405-1444-009-8e99177b\",\"TC58-SW010-WC006-210405-1444-002-9d602c33\"],\"TrainingJobStatus\":[\"Completed\",\"Completed\",\"Completed\",\"Completed\",\"Completed\",\"Completed\",\"Completed\",\"Completed\",\"Completed\"],\"TrainingStartTime\":{\"__ndarray__\":\"AADGKSiKd0IAANr8J4p3QgCAaS4oindCAAAdWiiKd0IAAP9eKIp3QgAASSkoindCAIAq+ieKd0IAgENeKIp3QgCAKvonindC\",\"dtype\":\"float64\",\"shape\":[9]},\"index\":[4,6,3,2,1,5,8,0,7],\"num_round\":{\"__ndarray__\":\"AAAAAAAogEAAAAAAAFCAQAAAAAAAIHZAAAAAAACgf0AAAAAAADB+QAAAAAAAgHlAAAAAAABggkAAAAAAAABiQAAAAAAA0IlA\",\"dtype\":\"float64\",\"shape\":[9]}},\"selected\":{\"id\":\"1478\",\"type\":\"Selection\"},\"selection_policy\":{\"id\":\"1477\",\"type\":\"UnionRenderers\"}},\"id\":\"1425\",\"type\":\"ColumnDataSource\"},{\"attributes\":{\"callback\":null},\"id\":\"1391\",\"type\":\"DataRange1d\"},{\"attributes\":{},\"id\":\"1409\",\"type\":\"PanTool\"},{\"attributes\":{\"text\":\"Objective vs num_round\"},\"id\":\"1389\",\"type\":\"Title\"},{\"attributes\":{\"data_source\":{\"id\":\"1425\",\"type\":\"ColumnDataSource\"},\"glyph\":{\"id\":\"1427\",\"type\":\"Circle\"},\"hover_glyph\":null,\"muted_glyph\":null,\"nonselection_glyph\":{\"id\":\"1428\",\"type\":\"Circle\"},\"selection_glyph\":null,\"view\":{\"id\":\"1430\",\"type\":\"CDSView\"}},\"id\":\"1429\",\"type\":\"GlyphRenderer\"},{\"attributes\":{},\"id\":\"1410\",\"type\":\"CrosshairTool\"},{\"attributes\":{},\"id\":\"1400\",\"type\":\"BasicTicker\"},{\"attributes\":{},\"id\":\"1411\",\"type\":\"WheelZoomTool\"},{\"attributes\":{},\"id\":\"1395\",\"type\":\"LinearScale\"},{\"attributes\":{},\"id\":\"1412\",\"type\":\"ZoomInTool\"},{\"attributes\":{\"callback\":null},\"id\":\"1393\",\"type\":\"DataRange1d\"},{\"attributes\":{},\"id\":\"1475\",\"type\":\"BasicTickFormatter\"},{\"attributes\":{\"fill_alpha\":{\"value\":0.1},\"fill_color\":{\"value\":\"#1f77b4\"},\"line_alpha\":{\"value\":0.1},\"line_color\":{\"value\":\"#1f77b4\"},\"x\":{\"field\":\"num_round\"},\"y\":{\"field\":\"FinalObjectiveValue\"}},\"id\":\"1428\",\"type\":\"Circle\"}],\"root_ids\":[\"1431\"]},\"title\":\"Bokeh Application\",\"version\":\"1.4.0\"}};\n",
       "  var render_items = [{\"docid\":\"c8b332a0-46c2-484c-8616-981e15d306ba\",\"roots\":{\"1431\":\"e276403a-03c7-4ed4-aa1d-1169cb972c41\"}}];\n",
       "  root.Bokeh.embed.embed_items_notebook(docs_json, render_items);\n",
       "\n",
       "  }\n",
       "  if (root.Bokeh !== undefined) {\n",
       "    embed_document(root);\n",
       "  } else {\n",
       "    var attempts = 0;\n",
       "    var timer = setInterval(function(root) {\n",
       "      if (root.Bokeh !== undefined) {\n",
       "        clearInterval(timer);\n",
       "        embed_document(root);\n",
       "      } else {\n",
       "        attempts++;\n",
       "        if (attempts > 100) {\n",
       "          clearInterval(timer);\n",
       "          console.log(\"Bokeh: ERROR: Unable to run BokehJS code because BokehJS library is missing\");\n",
       "        }\n",
       "      }\n",
       "    }, 10, root)\n",
       "  }\n",
       "})(window);"
      ],
      "application/vnd.bokehjs_exec.v0+json": ""
     },
     "metadata": {
      "application/vnd.bokehjs_exec.v0+json": {
       "id": "1431"
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ranges = tuner.tuning_ranges\n",
    "figures = []\n",
    "for hp_name, hp_range in ranges.items():\n",
    "    categorical_args = {}\n",
    "    if hp_range.get('Values'):\n",
    "        # This is marked as categorical.  Check if all options are actually numbers.\n",
    "        def is_num(x):\n",
    "            try:\n",
    "                float(x)\n",
    "                return 1\n",
    "            except:\n",
    "                return 0           \n",
    "        vals = hp_range['Values']\n",
    "        if sum([is_num(x) for x in vals]) == len(vals):\n",
    "            # Bokeh has issues plotting a \"categorical\" range that's actually numeric, so plot as numeric\n",
    "            print(\"Hyperparameter %s is tuned as categorical, but all values are numeric\" % hp_name)\n",
    "        else:\n",
    "            # Set up extra options for plotting categoricals.  A bit tricky when they're actually numbers.\n",
    "            categorical_args['x_range'] = vals\n",
    "\n",
    "    # Now plot it\n",
    "    p = figure(plot_width=500, plot_height=500, \n",
    "               title=\"Objective vs %s\" % hp_name,\n",
    "               tools=hover.tools(),\n",
    "               x_axis_label=hp_name, y_axis_label=objective_name,\n",
    "               **categorical_args)\n",
    "    p.circle(source=df, x=hp_name, y='FinalObjectiveValue')\n",
    "    figures.append(p)\n",
    "show(bokeh.layouts.Column(*figures))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "请注意，这里需要手动操作！！！\n",
      "请手动部署最优的模型为 Endpoint，分为两个步骤：\n",
      "步骤1：Create Model，name = prefix_tuningjobname\n",
      "步骤2：Create endpoint，name = prefix_tuningjobname\n",
      "\n",
      "prefix_tuningjobname = TC58-SW010-WC006-20210405151047\n"
     ]
    }
   ],
   "source": [
    "#prefix_tuningjobname\n",
    "print('请注意，这里需要手动操作！！！')\n",
    "print('请手动部署最优的模型为 Endpoint，分为两个步骤：')\n",
    "print('步骤1：Create Model，name = prefix_tuningjobname')\n",
    "print('步骤2：Create endpoint，name = prefix_tuningjobname\\n')\n",
    "print('prefix_tuningjobname = ' + prefix_tuningjobname)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 给上述操作提个醒：请手动部署最优的模型为 Endpoint，\n",
    "* 步骤1：Create Model，name = prefix_tuningjobname\n",
    "* 步骤2：Create endpoint，name = prefix_tuningjobname\n",
    "![image-20210405230902960](https://raw.githubusercontent.com/liangyimingcom/storage/master/uPic/image-20210405230902960.png)\n",
    "![image-20210405231402322](https://raw.githubusercontent.com/liangyimingcom/storage/master/uPic/image-20210405231402322.png)\n",
    "![image-20210405231448328](https://raw.githubusercontent.com/liangyimingcom/storage/master/uPic/image-20210405231448328.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 输入 创建成功创建后的推理EC2，即endpoint name\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "### 输入 创建成功的endpoint name\n",
    "endpoint_name_input = 'TC58-SW010-WC006-20210405151047' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "S3存储桶名称= xgb-10000871-part_ac.csv-TC58-SW010-WC006-20210405135629\n",
      "S3路径名称= TC58-SW010-WC006-20210405151047\n"
     ]
    }
   ],
   "source": [
    "import numpy as np                                # For matrix operations and numerical processing\n",
    "import pandas as pd                               # For munging tabular data\n",
    "import os \n",
    "from datetime import datetime\n",
    "from time import gmtime, strftime, sleep\n",
    "import boto3\n",
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "#from sagemaker.tuner import IntegerParameter, CategoricalParameter, ContinuousParameter, HyperparameterTuner\n",
    "from time import sleep\n",
    "import time\n",
    "\n",
    "region = boto3.Session().region_name    \n",
    "smclient = boto3.Session().client('sagemaker')\n",
    "\n",
    "role = sagemaker.get_execution_role()\n",
    "bucket = sagemaker.Session().default_bucket()\n",
    "\n",
    "s3_wholedata_path = s3_data='s3://{}/{}/wholedata/whole_data.csv'.format(bucket, prefix)\n",
    "\n",
    "endpoint_name = endpoint_name_input #提醒，这里输入\n",
    "\n",
    "#!pip install s3fs -U ## 如果报错请安装SDK\n",
    "role = get_execution_role()\n",
    "model_data = pd.read_csv(s3_wholedata_path)\n",
    "#model_data.head(3) #for testing\n",
    "#model_data.info() #for testing\n",
    "#prefix = 'xgb-10000871.csv-TC58-SW010-WC008-20210111073547'\n",
    "print('S3存储桶名称= '+prefix)\n",
    "print('S3路径名称= '+endpoint_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 模型评估 与 模型推理（预测效果展示）\n",
    "现在我们要使用上图中的部署好的endpoint来做推理。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "#预测 1：准备sagemaker endpoint\n",
    "xgb_predictor = sagemaker.predictor.RealTimePredictor(endpoint=endpoint_name)\n",
    "\n",
    "#from sagemaker.predictor import csv_serializer\n",
    "from sagemaker.predictor import csv_serializer, json_deserializer\n",
    "xgb_predictor.content_type = 'text/csv'\n",
    "xgb_predictor.serializer = csv_serializer\n",
    "xgb_predictor.deserializer = None\n",
    "\n",
    "# inference预测处理：传入modeldata数据和 sagemaker inference handle，获得预测结果\n",
    "def sagemaker_predict(data, xgb_predictor, rows=100):\n",
    "    split_array = np.array_split(data, int(data.shape[0] / float(rows) + 1))    \n",
    "    predictions = ''\n",
    "    for array in split_array:\n",
    "        split_result = xgb_predictor.predict(data=array).decode('utf-8')\n",
    "        #print(split_result)\n",
    "        split_result = str.strip(split_result, '[]') # 去掉多余的前后[]符号\n",
    "        #split_result=split_result.replace(' ', '')\n",
    "        #print('res='+split_result)\n",
    "             \n",
    "        predictions = ','.join([predictions, split_result])\n",
    "            \n",
    "    #print(predictions)\n",
    "    #return np.fromstring(predictions[1:], sep=',', dtype=np.float64) \n",
    "    return np.fromstring(predictions[1:], sep=',') \n",
    "\n",
    "#预测 2：预测并获得预测结果\n",
    "model_data_to_numpy = model_data.to_numpy()[:, 1:] #转换所有的行，从第一列开始（忽略0列）\n",
    "\n",
    "#调用函数\n",
    "predictions = sagemaker_predict(model_data_to_numpy,xgb_predictor, rows=100)\n",
    "#print('predictions=')\n",
    "#print(predictions)\n",
    "#print('predictions round=')\n",
    "#print(np.round(predictions))\n",
    "#print('\\n')\n",
    "#print('expect round=')\n",
    "#print(model_data.to_numpy()[:, 0:1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 检测 模型推理 效果\n",
    "\n",
    "检测 预测效果，我们使用之前放入S3里面的完整数据集作为检测 模型推理 效果\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model_data: 1=246, 0=2264 / \n",
      "predictions Result: 1=245, 0=2265\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>predictions_data</th>\n",
       "      <th>0.0</th>\n",
       "      <th>1.0</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>model_data</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.0</th>\n",
       "      <td>2237</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>28</td>\n",
       "      <td>218</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "predictions_data   0.0  1.0\n",
       "model_data                 \n",
       "0.0               2237  27 \n",
       "1.0               28    218"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVIAAAFNCAYAAABSVeehAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3wU5fb48c9JKKF3G0VAQDqoiCAKKCYgSJVeBMTrhatYuNjL1/qzXcWromJBlF6UIoIURdoFEQWpUgSkiHRCDYTk/P6Yybqkbspmssl5v155ZeeZdmZ29+zzTHlGVBVjjDEZF+Z1AMYYE+oskRpjTCZZIjXGmEyyRGqMMZlkidQYYzLJEqkxxmSSJdIgEJE+IjLf6zi8JiKVROSUiIRn4zori4iKSL7sWmcwichGEWmZgfly7WdQRFqKyF6v4/CX6xOpiOwSkbPuF/ovERkjIkWDuU5VHa+qUcFcR07k7uvbEoZVdbeqFlXVOC/j8oqb0KtlZhmqWkdVf0hjPUl+PPLqZ9AruT6RutqralGgIXAN8ITH8WSIl7Ws3FLDSw/b3yZQeSWRAqCqfwHzcBIqACJSUET+IyK7ReSAiHwoIoX8xncUkbUickJEfheRNm55CRH5VET2i8g+EXkpoQkrIgNEZJn7+kMR+Y9/HCIyU0SGua+vEJEvReSQiOwUkQf8pntORKaJyDgROQEMSLxNbhxfuPP/ISJPi0iYXxzLReRdEYkWkd9EpFWieVPbhuUiMkJEjgLPichVIvK9iBwRkcMiMl5ESrrTjwUqAV+7tf9HE9eUROQHEXnRXe5JEZkvImX94rnL3YYjIvJM4hpuou0uJCJvutNHi8gy//cN6OO+p4dF5Cm/+RqLyAoROe5u93siUsBvvIrIfSKyDdjmlv1XRPa4n4GfReRmv+nDReRJ97Nx0h1fUUSWuJP86u6PHu70d7ifp+Mi8j8Rqe+3rF0i8piIrANOi0g+/33gxr7ajeOAiLzlzpqwruPuupr6fwbdeeuIyAIROerO+2QK+zXF74Mb20q/93OIOIceItzhqeK0+qJFZImI1PFb7hgReV9E5roxLheRy0TkbRE55n42r0m0L54QkU3u+M8S1pNMzCl+h7KNqubqP2AXcJv7ugKwHviv3/i3gVlAaaAY8DXwijuuMRANROL86JQHarrjZgCjgCLAJcAq4J/uuAHAMvd1c2APIO5wKeAscIW7zJ+BZ4ECQFVgB9DanfY5IBbo5E5bKJnt+wKY6cZeGdgKDPKL4wLwMJAf6OFuT+kAt+ECMBTIBxQCqrn7oiBQDucL/HZy+9odrgwokM8d/gH4HajhLu8H4FV3XG3gFHCTuy/+4277bSm8ryPd+csD4cCNblwJ6/zYXUcD4BxQy53vOqCJu02Vgc3AQ37LVWABzuehkFvWFyjjzvNv4C8gwh33CM5n6mpA3PWV8VtWNb9lXwscBG5wY+7v7rOCfvtvLVDRb92+fQqsAPq5r4sCTZLbz8l8BosB+93YI9zhG1LYr6l9H8Lc9/w5oDpwDLjGb9673XkKustZ6zduDHDY3f8RwPfATuAud1+8BCxK9Fna4O6L0sBy4CV3XEtgr19MKX6Hsi3PeJ3ogr6BzhtyCjjpfti+A0q64wQ4DVzlN31TYKf7ehQwIpllXorz5SzkV9Yr4YOQ6EMswG6guTv8D+B79/UNwO5Ey34C+Mx9/RywJJVtC3fjqO1X9k/gB784/sRN4m7ZKqBfgNuwO6V1u9N0AtYk2tdpJdKn/cb/C/jWff0sMNFvXGHgPMkkUvfLcxZokMy4hHVWSLTNPVPYhoeA6X7DCtyaxnYfS1g3sAXomMJ0iRPpB8CLiabZArTw2393J/P5TUikS4DngbIpbHNKibSX//uUynal+n3wW9dRnB+gJ1JZVkk3phLu8BjgY7/xQ4HNfsP1gOOJtnuw33Bb4Hf3dUv+TqSpfoey6y+vHIfppKoLRaQFMAEoCxzHqVUVBn4WkYRpBSdBgfNrOCeZ5V2JU8Pb7zdfGE7N8yKqqiIyCefDvAToDYzzW84VInLcb5ZwYKnfcJJl+imL8yv8h1/ZHzi1tAT71P10+Y2/IsBtuGjdInIJ8A5wM07NIwwnqaTHX36vz+DUrHBj8q1PVc+IyJEUllEWp1bze3rXIyI1gLeARjjvfT6cGo2/xNv9b+AeN0YFirsxgPMZSS0Of1cC/UVkqF9ZAXe5ya47kUHAC8BvIrITeF5VZwew3kBjTOv7gKruEpFFOIltpG8i55DQy0A3dznx7qiyOK0ggAN+6zqbzHDik8D++yLhc5tYIN+hoMtrx0gX4/wyJhyzPIzzBtZR1ZLuXwl1TkyB80Zelcyi9uDU5sr6zVdcVeskMy3ARKCriFyJ8wv6pd9ydvoto6SqFlPVtv5hp7JJh3Gav1f6lVUC9vkNlxe/b4U7/s8AtyHxul9xy+qranGcJq+kMn167Mc59AI4x0BxmtPJOQzEkPx7k5YPgN+A6u42PMnF2wB+2+EeD30M6A6UUtWSOIkhYZ6UPiPJ2QO8nOj9LqyqE5Nbd2Kquk1Ve+EchnkNmCYiRVKbJ50xpvV9QETa4tRSvwPe8Ju3N9ARuA0ogVNzhaT7Nj0q+r1O+NwmFsh3KOjyVCJ1vQ1EikhDVY3HOZY2wq1tISLlRaS1O+2nwEARaSUiYe64mqq6H5gPvCkixd1xV7k13iRUdQ1wCPgEmKeqCb+eq4AT7kH8Qu6Ji7oicn0gG6LOZUVTgJdFpJibqIfxd40XnC/dAyKSX0S6AbWAOendBlcxnMMkx0WkPM7xQX8HcI5RZcQ0oL2I3CjOyZ/nSeFL6L5vo4G33BMN4e4JloIBrKcYcAI4JSI1gSEBTH8B5/3LJyLP4tRIE3wCvCgi1cVRX0QSfgAS74+PgcEicoM7bRERaScixQKIGxHpKyLl3O1P+AzFubHFk/K+nw1cJiIPuSeTionIDYknSuv7IM6JwU9xauf9cd6vhIRVDOeH+QhOrfb/BbJNabhPRCqISGmcH7zJyUyTqe9QVslziVRVD+GcoHnGLXoM2A6sFOfM+EKcEweo6ipgIDACpxaymL9rf3fhNMs24TRvpwGXp7LqiTi/1hP8YokD2uNcRbATp0bwCc4veqCG4hzX2gEsc5c/2m/8jzgnBg7jNL26qmpCkzm92/A8zgmTaOAb4KtE418BnhbnjPTwdGwDqrrR3ZZJOLXTkzgnZs6lMMtwnJM8P+Ecs3uNwD7Pw3FqTydxkkZyX05/84C5OCfx/sCpCfs3Od/C+TGbj5OgP8U5yQXOMe7P3f3RXVVX4xwjfw9nf28nmSsxUtEG2Cgip4D/4hz3jVHVMzjv7XJ3XU38Z1LVkzgnCdvjHPLYBtySwjpS/D4AHwEzVXWO+xkaBHzi/nB84e6ffTifp5Xp2K6UTMDZrzvcv5cST5BF36FMSziTbHIhERkA3KOqN3kdS3qJc9PEcZwm+E6v4zHZS0R24Xx2F3odSyDyXI3U5Fwi0l5ECrvH/f6DU+Pc5W1UxqTNEqnJSTrinFD4E+dwRE+1JpMJAda0N8aYTLIaqTHGZJIlUmOMyaSQu7OpbNmyWrlyZa/DMMbkMj///PNhVS2XkXlDLpFWrlyZ1atXex2GMSaXEZE/0p4qeda0N8aYTLJEaowxmWSJ1BhjMskSqTHGZJIlUmOMySRLpMYYk0mWSI0xJpOClkhFZLSIHBSRDSmMFxF5R0S2i8g6Ebk2WLEYY0wwBbNGOganI9qU3I7Tw0914F6cR0AYY0zICdqdTaq6REQqpzJJR+ALt5u0lSJSUkQudx+BYYzJy07ugw2jIeao15EExMtbRMtz8SMb9rpllkiNCYTGw45v4K9ceMv0yhe8jiBdvEykyT3YLNnOUUXkXpzmP5UqVQpmTMak7vwp2PYlnDnkdSSwYzbsXex1FAZvE+leLn7cagWSf9wqqvoRzoO3aNSokfVEbYJD4+GPBXAilb4rFvwz++IxjuZvQFh4li4y+pSyYl0cbW70T4HDMrw8LxPpLOB+EZmE86z3aDs+GqADa+Dob15HkfuseRf2r/A6iowp1xCqdfI6iqyVLwKu7gElKmfpYg8ePE2bNuNYt+4AM2b05I47arhjcmAiFZGJQEugrIjsBf4PyA+gqh8Cc4C2OI9+PYPz2GOTGlVY/G/4eYTXkRiABv+CfIXSni7YLrseanTN8lpbbvTHH8eJihrH1q3OE8m7d5/K9u0PcMUVxTK13GCete+VxngF7gvW+kPGuWg49Gtg0y55FPb/GNx4jKNW35STZHhBqNUHrmiS/HiTI23efIioqHHs3XsCgLAw4b332mY6iUIIduwccqJ3QszxFMbtgK+7ZnzZhS+FirdkfH6TVL5CUP8fcEVTryMxWWj16j9p02YcR46cBaBAgXAmTryTLl1qZcnyLZFmlsY7yTK5p7EuHg6/zwzOeps8DTe+AJLcxQ/GmASLFu2kQ4dJnDp1HoAiRfIzc2ZPWrWqmmXrsEQaiJjjEH8+aXnsaRh7LZxLocaZXuVvTnua8AJQ926o1Ttr1mlMLjZjxm/07DmNc+fiAChduhBz5/ahcePyWboeS6Rp+fZu2PhZ5pdTupZzbC055epDizehcNnMr8cYA8Dnn6/l7rtnER/vtBbLly/G/Pn9qF07Q8+3S5Ul0tSs/SB9SbTkVUnL8hWG6x6GunZRgjHZSRVfEq1WrTQLFvSjcuWSQVmXJdLEDvwMc/rC8e0Qf+HicYVS+CUrVR0iP4KydYIfnzEmIAMGNOTYsbN88cU6vv22D5deWjRo6xJN7iRJDtaoUSMN2uOYD2+Ez+smP+6mV+CGx4OzXmNM0MTEXCAiIu06o4j8rKqNMrKOvFkj3TYDVvxf0suSTu5OOq2EOyd3LIkak2OdPx/Hiy8uZvjwGylRIuKicYEk0czKe4n0QgzM6hzYtENPONcVhuW93WRMqDhzJpauXacwd+52Fi/+g2+/7UvhwvmzNYa88aiRU3/CtwNhwo3w3wBu6StyGTxwCgoUsyRqTA52/HgMUVFjmTt3OwBLl+5m7NgA7xTMQnkjS0xtlXInH//YlahAoFhFu9DdmBzuwIFTtG49jl9/PeAre+aZ5tx773XZHkvuTqRx52HlSykn0R5LofiV2RuTMSbTdu06TmTkWLZv/7sH/REjWvPQQ970f5A7E2l8nPOYgkUPwoWzF4/r8JVzj3qpGnYBvDEhaNOmQ0RGjuXPP08CEB4ufPppB/r3b+hZTLkvkZ4/CbPudDroTaxWH6ge4IkmY0yOs2rVPm6/fTxHjzoVpIIFw5k8uSsdO9b0NK7cl0iXPZV8Em3yDNz4XLaHY4zJGj/+uJdWrb7g9OlYAIoWLcCsWT255ZYqHkeW287ax8U6vZz7u6oD9FsLzV4AyV2ba0xeUrNmWa6+2jkcV6ZMIRYt6p8jkijktkS64vmLhxs/Dh1nwCUNvInHGJNlSpSI4Ntv+3DbbVVZunQgjRpd4XVIPrmnaa8KP758cdkNT9plTMbkIuXKFWHBgn5eh5FE7qiRnjsBv024uKzteOeCemNMyFFVnnrqOz799BevQwlI6NdIY0/DeyWSllvHx8aEpLi4eO67bw6jRv1MWJhQqlShLHskSLCEfo107QdJy0rkjAPQxpj0OX8+jt69v2LUqJ8Bpz/RCRPWk9N7qQvtGml8HKx65eKyevdAo0e8iccYk2GnT5/nzjunMG/e776yvn3rM3p0BySHn+sI3UR6Lho+rnzx85KqtIWojz0LyRiTMceOnaVduwmsWLHXVzZ0aGPefrsNYWE5O4lCKCZSVfe4aCkgUXXfHudhTMjZv/8krVuPY/36g76y555rwbPPtsjxNdEEoZdID66Bd5J5ZECDIVDNbv80JpTs2HGMyMix7NhxzFf2zjttGDr0Bg+jSr/QS6SJa6EAA7dA6RrZH4oxJsNiYi5wyy2fs3t3NOB0PjJmTCf69q3vcWTpF7pn7fMVhqIVoN1ES6LGhKCIiHy8+morRJzX06f3CMkkCiFZI3U9eNrrCIwxmdSrVz1OnTpPjRplaNGistfhZFhoJtKUHotsjMnRYmPjyJ8//KKyf/wj+3u0z2qh2bS/uofXERhj0unjj3/muus+4siRM16HkuVCM5EaY0LKa68t4957Z7N+/UHatp3AqVPnvQ4pS1kiNcYEjary2GMLePzx73xlFy7EExNzwcOosl5oHiM1xuR4cXHxDB48m08+WeMra9HiSmbN6kXx4gU9jCzrWSI1xmS5c+cu0LfvdKZN2+Qr69DhaiZNupNChfJ7GFlwWCI1xmSpU6fO06XLZBYs2OEru+uuBnz6aQfy5cudRxMtkRpjsszRo07nIytX/t35yIMP3sBbb7UOic5HMip3/jwYYzzx7LOLLkqiL7zQkhEjcncSBUukxpgs9Oqrt9GkSQUA3nvvdp55JnR6cMoMa9obY7JM0aIF+Oab3ixd+gcdO9b0OpxsYzVSY0yG7dt3IklZ6dKF8lQSBUukxpgM+vbb7dSo8R7vv/+T16F4zhKpMSbdJk/eQIcOEzlzJpb775/DlCkbvQ7JU5ZIjTHpMmrUanr1+pLY2HgAKlYsQcOGl3kclbeCmkhFpI2IbBGR7SLyeDLjK4nIIhFZIyLrRKRtMOMxxmScqvLKK0sZPPgbEp6OXLNmWZYvv5saNcp4G5zHgpZIRSQcGAncDtQGeolI7USTPQ1MUdVrgJ7A+8GKxxiTcarKI48s4Mknv/eVNWp0BUuXDqRCheIeRpYzBLNG2hjYrqo7VPU8MAnomGgaBRLehRLAn0GMxxiTARcuxHPPPbN4880VvrJbb63C99/fRdmyhT2MLOcI5nWk5YE9fsN7gcSPBnwOmC8iQ4EiwG1BjMcYk04xMRfo3ftLpk//zVfWqVNNJk68k4gIuww9QTBrpMndzpD4EaC9gDGqWgFoC4wVkSQxici9IrJaRFYHIU5jTArWrTvAN99s8w0PGNCQqVO7WRJNJJiJdC9Q0W+4Akmb7oOAKQCqugKIAMomXpCqfqSqjVS1UZBiNcYko3Hj8kyceCdhYcKwYU1ydQ9OmRHMn5WfgOoiUgXYh3MyqXeiaXYDrYAxIlILJ5EeCmJMxph06tKlFr/8ci/161+aJ+6bz4ig/bSo6gXgfmAesBnn7PxGEXlBRDq4k/0b+IeI/ApMBAaoauLmvzEmm2zbdoTdu6OTlDdocJkl0VQE9UCHqs4B5iQqe9bv9SagWTBjMMYEZu3av2jdehwlS0awdOlALrmkiNchhQw72GGMYdmy3bRsOYaDB0+zdesRunSZjDUOA2eJ1Jg8bs6cbURFjSU6+hwAJUoU5LXXbrOmfDrYNQzG5GETJ67nrrtmcOGCc9/8pZcWYd68vjRokLfvnU8vq5Eak0e9//5P9OnzlS+JVq5ckmXL7rYkmgGWSI3JY1SVl15awn33zfF1PlK7djmWLRtItWqlvQ0uRFnT3pg8JD5e+fe/5/H22z/6yho3Ls+cOb0pU8bum88oq5Eak4fExyvbtx/zDbdqVYXvvrvLkmgmWY3UmDwkX74wpkzpSps24ylbtjATJnShYEFLA5lle9CYPKZQofx8801vIiLy2X3zWcT2ojG52OHDZ/jgg6QPpytatIAl0SxkNVJjcqk9e6KJihrHb78d5vTpWIYPv9HrkHIt+0kyJhfauvUIN930Gb/9dhiAxx5byNatRzyOKveyRGpMLvPLL/u56abRvl6c8ucPY+LEO/P8A+qCyZr2xuQiS5b8Qfv2EzlxwrlvvnDh/Hz1VXdat67mcWS5myVSY3KJ2bO30q3bVGJiLgBQsmQEc+b0pmnTimnMaTLLEqkxucD48evo338GcXHOPZ+XXVaU+fP7Uq/epR5HljdYIjUmxI0atZrBg7/xDVetWooFC/pRtWopD6PKW+xkkzEhrnbtcr6netatewnLlg20JJrNLJEaE+JuvvlKpk3rxs03V2Lx4gFcfnkxr0PKc6xpb0wu0K5dDdq2rW692nvEaqTGhJCzZ2O5555Z/P770STjLIl6J6BEKiIFRMQuRDPGQydOnOP228fz6adriIwcy59/nvQ6JONKM5GKSDtgPbDAHW4oItODHZgx5m+HDp3mlls+Z/HiPwDYufM406Zt8jgqkyCQGukLwA3AcQBVXQtY7dSYbLJ7dzQ33/wZv/yy31f2+uu38cADN3gYlfEXyMmmWFU9nuj4iz3w2phssGXLYSIjx7JnzwkAwsKEUaPu4J57rvU4MuMvkES6WUS6A2EiUgV4EFgZ3LCMMb/8sp/Wrcdx+PAZAAoUCGfChC7ceWdtjyMziQXStL8fuA6IB74CYnCSqTEmSBYv3kXLlmN8SbRIkfzMnt3LkmgOFUiNtLWqPgY8llAgIl1wkqoxJosl1ETPnYsDoFSpCObM6UOTJhU8jsykJJAa6dPJlD2V1YEYYxwNGlzKHXfUAODyy4uyZMlAS6I5XIo1UhFpDbQByovIW36jiuM0840xQRAeHsb48V0oVWoOTz55M1Wq2H3zOV1qTfuDwAacY6Ib/cpPAo8HMyhj8hJV5yIY/ytjChbMx8cfd/AqJJNOKSZSVV0DrBGR8aoak40xGZNnxMcrDzwwl/z5w3jrrdZ2m2eICuRkU3kReRmoDUQkFKpqjaBFZUweEBsbx4ABM5kwYT0AZcoU5umnm3sclcmIQE42jQE+AwS4HZgCTApiTMbkemfOxNK582RfEgXYuPEQ8fF2r0soCiSRFlbVeQCq+ruqPg3cEtywjMm9oqNjaNNmHN98s81XNmRII8aN60xYmDXtQ1EgTftz4hy4+V1EBgP7gEuCG5YxudOBA6do02Y8a9f+5St76qmbefHFW+z4aAgLJJE+DBQFHgBeBkoAdwczKGNyoz/+OE5k5Fi2bfu7L9E334xi2LCmHkZlskKaiVRVf3RfngT6AYiIXR1sTDps2nSIqKix7Nvn9CEaFiZ88kl7Bg68xuPITFZI9RipiFwvIp1EpKw7XEdEvsA6LTEmYHFx8XTuPNmXRAsUCGfatG6WRHORFBOpiLwCjAf6AN+KyFPAIuBXwC59MiZA4eFhfPFFJ4oUyU/RogWYM6c3nTvX8josk4VSa9p3BBqo6lkRKQ386Q5vyZ7QjMk9brihAjNn9qRYsYI0blze63BMFkstkcao6lkAVT0qIr9ZEjUmMMePx1CyZMRFZa1aVfUoGhNsqR0jrSoiX7l/04HKfsPWhZ4xKRgxYgU1arzLb78d9joUk01Sq5HemWj4vfQuXETaAP8FwoFPVPXVZKbpDjyH8/iSX1W1d3rXY0xOoKo888wiXn55KQBRUWNZvvxuKlYs4XFkJthS67Tku8wsWETCgZFAJLAX+ElEZqnqJr9pqgNPAM1U9ZiI2IX+JiTFxyv33z+HDz5Y7Su78sqSFC9e0MOoTHYJ6Ln2GdQY2K6qO1T1PM79+R0TTfMPYKSqHgNQ1YNBjMeYoDh/Po4+fb66KIm2bVudefP6UqJERCpzmtwimIm0PLDHb3ivW+avBlBDRJaLyEr3UEASInKviKwWkdXJjTfGK2fOxNKp0yQmTdrgK+vdux4zZvSgcOH8HkZmslPAiVRE0ttGSe7G4cRd2+QDqgMtgV7AJyJSMslMqh+paiNVbZTOGIwJmuPHY4iKGsvcudt9Zffddz1jx3Ymf/5wDyMz2S3NRCoijUVkPbDNHW4gIu8GsOy9QEW/4Qo416Imnmamqsaq6k5gC05iNSZH++uvU7RsOYbly/9udD3zTHPeffd268EpDwqkRvoOcAdwBEBVfyWwbvR+AqqLSBURKQD0BGYlmmZGwrLc21BrADsCC90Y73z88c/8+usB3/CIEa154QXrwSmvCiSRhqnqH4nK4tKaSVUvAPcD84DNwBRV3SgiL4hIwsNo5gFHRGQTzu2nj6jqkcDDN8YbTz3VnN696xEeLnz+eSceeqiJ1yEZDwXSjd4eEWkMqHtJ01BgayALV9U5wJxEZc/6vVZgmPtnTMgICxPGjOnIkCGNuOmmSl6HYzwWSI10CE6iqwQcAJq4ZcbkGb/++hdxcRc/hTx//nBLogYILJFeUNWeqlrW/eupqnbvm8kzvvxyE40bf8LQoXN9j042xl8gifQnEZkjIv1FpFjQIzImB/n001/o3n0a58/H8cEHq3njjf95HZLJgdJMpKp6FfAScB2wXkRmiEjPoEdmjMfeeGM599zzte/JnjVqlKFnz7oeR2VyooAuyFfV/6nqA8C1wAmcDp+NyZVUlSeeWMijjy70lV177eUsXTqQSpWsAxKTVJpn7UWkKM498j2BWsBM4MYgx2WMJ+Li4vnXv77ho49+8ZW1aHEls2b1sg5ITIoCufxpA/A18LqqLg1yPMZ45vz5OPr2/YqpU30dlNG+fQ0mT+5KoUJ237xJWSCJtKqqxqc9mTGh6/Tp89x55xTmzfvdV9avX30+/bSD3Tdv0pRiIhWRN1X138CXIpLkmg9V7RLUyIzJRn/9dYpfftnvGx46tDFvv93G7ps3AUmtRjrZ/Z/unvGNCTVXXVWab7/tyy23fM6wYU149tkWdt+8CVhqPeSvcl/WUtWLkqmI3A9kqgd9Y3Kaa6+9nM2b7+OKK+xyaZM+gVz+dHcyZYOyOhBjstP69QdYuXJvknJLoiYjUjtG2gPnkqcqiZ4aWgw4HuzAjAmWFSv20LbtBERgyZKB1K1rjwozmZPaMdJVOH2QVsB5iF2Ck8CaYAZlTLDMn/87nTtP5syZWAC6dZvKhg1DCA8P5lN3TG6X2jHSncBOYGFK0xgTSqZO3UifPl8RG+tczVeuXGHGj+9iSdRkWmpN+8Wq2kJEjnHxs5YEpyvR0kGPzpgs8vHHPzN48De+++YrVSrBggX9qFGjjMeRmdwgtaZ9wuNEymZHIMYEy2uvLePxx/++yKRmzbLMn9+XihXtvnmTNVJs0/jdzVQRCFfVOKAp8E+gSDbEZkymqCqPPbbgoiR63XWXs2TJAEuiJksFcnBoBs5jRq4CvsDpuGRCUKMyJpPi45V77/2a11//u//Qli0r8/33/SlXzuoBJmsFkshizjkAACAASURBVEjjVTUW6AK8rapDgfLBDcuYzBGBEiUifMMdOlzN3Ll9rAcnExSBdFpyQUS6Af2ATm6ZdYVjcjQR4Y03Ijl69Czx8conn3QgXz47O2+CI5BEejfwL5xu9HaISBVgYnDDMibzRISPP26PiFjnIyaoAnnUyAbgAWC1iNQE9qjqy0GPzJh02LfvBI88Mp8LFy7u8TE8PMySqAm6QHrIvxkYC+zDuYb0MhHpp6rLgx2cMYHYvv0okZFj2bXrOEeOnOXTTztYz00mWwVy0GgE0FZVm6nqjUA74L/BDcuYwKxbd4CbbhrNrl1O9w9jx65jzZq/PI7K5DWBJNICqup79oKqbgYKBC8kYwKzfPlumjf/jAMHTgMQEZGPmTN7cu21l3scmclrAjnZ9IuIjMJp3gP0wTotMR779tvtdOkymbNnLwBQokRBZs/uzU03VfI4MpMXBZJIB+OcbHoU5xjpEuDdYAZlTGomTdpAv37TfSeWLrmkCPPm9aVhw8s8jszkVakmUhGpB1wFTFfV17MnJGNS9uGHq/nXv75B3W50rrzS6XykenXrfMR4J8VjpCLyJM7toX2ABSKSXE/5xmSbDz74iSFD/k6itWuXY/nyuy2JGs+ldrKpD1BfVbsB1wNDsickY5LXunU1Lr+8KACNG5dnyZIBlC9f3OOojEk9kZ5T1dMAqnoojWmNCbqqVUsxb15funatzcKF/ShTprDXIRkDpH6MtKrfs5oEuMr/2U32XHvjhXr1LmXq1G5eh2HMRVJLpHcmGrbn25tsc/LkOXr3/opHHrmR5s2v9DocY1KV2jOb7Ln1xhNHjpzh9tvH89NPf7JkyR8sWtTfLrI3OZod9zQ5yt69J7j55s/46ac/AThx4hxLlvzhcVTGpC6QC/KNyRbbth0hMnIsf/wRDTidM48c2ZYhQ673ODJjUhdwIhWRgqp6LpjBmLxr7dq/aN16HAcPOvfN58sXxtixnenZs67HkRmTtjSb9iLSWETWA9vc4QYiYreImiyzdOkftGgxxpdECxXKx6xZPS2JmpARyDHSd4A7gCMAqvorfz+q2ZhMmTNnG1FR4zhxwmnslChRkAUL+nH77dU9jsyYwAXStA9T1T8SdZQbF6R4TB6yZcthOnac5Ot85NJLnc5HGjSwzkdMaAmkRrpHRBrjPJI5XEQeArYGOS6TB1x9dVkee6wZAJUrl2TZsrstiZqQFEiNdAhO874ScABYSID33YtIG5ze9MOBT1T11RSm6wpMBa5X1dWBLNvkDi++eAtFiuTnrrsa2H3zJmSlmUhV9SDQM70LFpFwYCQQCewFfhKRWf697bvTFcPp7/TH9K7DhJb4eOX8+TgiIv7+2IkITzxxs4dRGZN5gTz87mNAE5er6r1pzNoY2K6qO9zlTAI6ApsSTfci8DowPJCATWi6cCGeQYNmcfjwGWbM6EH+/OFeh2RMlgnkGOlC4Dv3bzlwCRDI9aTlgT1+w3vdMh8RuQaoqKqzA4rWhKSYmAvceecUvvjiV+bM2Ub//jOIj0/y22xMyAqkaT/Zf1hExgILAlh2cs/D9X17RCQM5wmlA9JckMi9wL0A11UIYM0mxzhx4hwdO07ihx92+coKF86PqpL8R8SY0JORe+2rAIF0x7MXqOg3XAH402+4GFAX+EFEdgFNgFki0ijxglT1I1VtpKpJxpmc69Ch09x66+cXJdFHH72Rjz9uT3i4dfNgco9AjpEe4++aZBhwFHg8gGX/BFQXkSrAPpwTVr0TRqpqNFDWbz0/AMPtrH3usGdPNJGRY9my5Yiv7NVXW/HYYzd5GJUxwZHWw+8EaICTCAHiVTWgg1uqekFE7gfm4Vz+NFpVN4rIC8BqVZ2VibhNDrZly2EiI8eyZ88JwOl8ZNSoO/jHP67zODJjgiPVRKqqKiLTVTVD3wBVnQPMSVT2bArTtszIOkzO8ssv+2nTZhyHDp0BIH/+MMaP70K3bnU8jsyY4AnkQNUqEbk26JGYkKeqDBnyjS+JFi6cn6+/7mVJ1OR6qT2OOaG2ehNOMt0iIr+IyBoR+SV7wjOhRESYOrUbFSsWp2TJCBYu7Efr1tW8DsuYoEutab8KuBbolE2xmFygUqUSLFjQj/Pn46hX71KvwzEmW6SWSAVAVX/PplhMCNqx4xhVq5a6qOzqq8umMLUxuVNqibSciAxLaaSqvhWEeEyIUFWef34xr7yyjNmzexEZeZXXIRnjmdRONoUDRXEunE/uz+RR8fHKgw9+y/PPL+b8+Tg6d57MmjX7vQ7LGM+kViPdr6ovZFskJiTExsZx992zGDduna+sWbNK1KhRxsOojPFWmsdIjUlw9mws3btPY/bsv/v17t69DmPHdqZAAevNyeRdqSXSVtkWhcnxoqNj6NBh0kXPmL/33mt5//12dt+8yfNS/Aao6tHsDMTkXAcPnuaWWz6/KIk+8cRNfPjhHZZEjSEdz7U3edMffxwnKmocW7f+3fnIG29EMnz4jR5GZUzOYonUpGrRol2+JBoWJnz00R0MGmR3DBvjz9plJlUDBjTkpZduoUCBcKZO7WZJ1JhkWI3UpOnJJ2+mW7c6domTMSmwGqm5yHff7eDUqfMXlYmIJVFjUmGJ1Ph8/vlaWrceR+fOkzl37oLX4RgTMiyRGgDefnslAwbMJC5OWbhwB489ttDrkIwJGZZI8zhV5Zlnvufhh+f5yurXv5THH7dnKxkTKDvZlIfFxytDh87h/ff/ft5gs2YVmT27NyVLRngYmTGhxRJpHhUbG0f//jOYOHGDr+z226sxbVp3ChfO72FkxoQeS6R50JkzsXTrNpU5c7b5ynr2rMvnn3eyzkeMyQBLpHnM8eMxtG8/kWXLdvvKhgxpxLvv3m73zRuTQfbNyWPi45Xjx2N8w08/fTMjR7a1JGpMJti3J48pXboQ8+b1pWrVUrz1VhQvvngrItb1rDGZYU37POiKK4qxbt1gihQp4HUoxuQKViPN5X76aR+TJ29IUm5J1JisYzXSXOz773fSseMkzp6NpWjRArRrV8PrkIzJlaxGmktNn76Z228fz6lT54mLUwYP/oaYGLt/3phgsESaC40evYauXady/nwcAOXLF2PevL5ERFgDxJhgsESay7z55v8YNGgW8fEKQPXqpVm+/G5q1y7ncWTG5F6WSHMJVeWpp75j+PAFvrKGDS9j6dKBXHllSQ8jMyb3s7ZeLhAXF899981h1KiffWU331yJr7/uRYkS1vmIMcFmiTTEqSr9+89g/Pj1vrJ27aozZUo363zEmGxiTfsQJyLcemsV33Dv3vWYPr2HJVFjspHVSHOBu+++huPHY9i58xj//e/thIXZLZ/GZCdLpLnEsGFNUVW7b94YD1jTPsTs3HmMrl2nEB0dk2ScJVFjvGE10hCyYcNBoqLGsn//KQ4ePM28eX0pVMiOhRrjNauRhoiVK/fSvPln7N9/CoBVq/bxyy/7PY7KGAOWSEPCwoU7uO22Lzh2zGnOFytWgG+/7UuzZpU8jswYA9a0z/G+/HITvXt/5btvvmzZwnz7bR+uu+4KjyMzxiSwGmkO9umnv9C9+zRfEq1YsThLlw60JGpMDmOJNId6443l3HPP177OR2rUKMOyZXdTs2ZZjyMzxiQW1EQqIm1EZIuIbBeRx5MZP0xENonIOhH5TkSuDGY8oeKTT37h0UcX+oavvfZyli4dSKVKJTyMyhiTkqAlUhEJB0YCtwO1gV4iUjvRZGuARqpaH5gGvB6seEJJt261ueaaywBo0eJKFi3qzyWXFPE4KmNMSoJZI20MbFfVHap6HpgEdPSfQFUXqeoZd3AlUCGI8YSMEiUi+Pbbvgwd2pi5c/tQvHhBr0MyxqQimIm0PLDHb3ivW5aSQcDcIMaTY8XFxScpu+SSIrzzzu12wb0xISCYiTS5+xU12QlF+gKNgDdSGH+viKwWkdVZGF+OcPToWZo3H8Po0Wu8DsUYk0HBvI50L1DRb7gC8GfiiUTkNuApoIWqnktuQar6EfARQKOKkmwyDkX7958kKmocGzYcZOXKvZQsGUGXLrW8DssYk07BrJH+BFQXkSoiUgDoCczyn0BErgFGAR1U9WAQY8lxfv/9KM2ajWbDBmez4+OVv/465XFUxpiMCFqNVFUviMj9wDwgHBitqhtF5AVgtarOwmnKFwWmuj0X7VbVDsGKKadYv/4AUVHjfIkzPFz4/PNO9OlT3+PIjDEZEdRbRFV1DjAnUdmzfq9vC+b6c6L//W8P7dpN4Phx5775iIh8TJ3ajTvuqOFxZMaYjLJ77bPRvHnb6dJlCmfOxAJQvHhBvv66F82b230IxoQyS6TZZMqUjfTt+xWxsc6lTuXKFWbevL5cc83lHkdmjMksS6TZYN++E9x113RfEq1UqQQLFvSjRo0yHkdmjMkK1mlJNihfvjijR3dEBGrWLMvy5XdbEjUmF7EaaTbp3bse+fKFceutVShbtrDX4RhjspAl0iCIi4vn2LGYJAmze/c6HkVkjAkma9pnsXPnLtCjxzRathzD0aNnvQ7HGJMNLJFmoVOnznPHHRP58svNbNx4iHbtJvgudTLG5F7WtM8iR46coV27Cfz44z5fWdOmFYiIsF1sTG5n3/IssG/fCaKixrFp0yFf2Usv3cKTT96Me+urMSYXs0SaSdu3HyUyciy7dh0HQARGjmzLkCHXexyZMSa7WCLNhF9//YvWrcdx4MBpAPLlC+OLLzrRq1c9jyMzxmQnS6QZtHz5btq1m0B0tNOFaqFC+Zg2rTtt21b3ODJjTHazRJpBb7zxP18SLVGiILNn9+ammyp5HJUxxgt2+VMGjRvXhRtuKM8llxRh8eIBlkSNycOsRppBRYsWYM6cPhw9epZq1Up7HY4xxkNWIw2AqrJ8+e4k5aVLF7IkaoyxRJoWVWX48PncdNNnfPDBT16HY4zJgSyRpuLChXgGDZrFW2+tBOC+++awYMHvHkdljMlp7BhpCmJiLtCr15fMmPGbr6xz51r2WBBjTBKWSJNx8uQ5OnWazPff7/SV3X13Q0aNak++fFaJN8ZczBJpIocPn6Ft2/H89NOfvrLhw5vy+uuRdt+8MSZZlkj97N17gqiosWzefNhX9sorrXjssWaWRI0xKbJE6tq69QiRkWPZvTsacDof+eCDdvzzn408jswYk9NZInXt2RPNX3+dAiB//jDGju1Mjx51PY4qd4uNjWXv3r3ExMR4HYrJQyIiIqhQoQL58+fPsmVaInW1alWViRPvZMCAGUyd2o3Wrat5HVKut3fvXooVK0blypXt0InJFqrKkSNH2Lt3L1WqVMmy5Voi9dOlSy1atLiSMmXsKZ/ZISYmxpKoyVYiQpkyZTh06FDaE6dDnr2WZ9q0Tb7jof4siWYvS6ImuwXjM5cnE+l7762iW7epREWN5dCh016HY4wJcXkqkaoqL764mKFD5wKwZcsRhg2b73FUxkvh4eE0bNiQunXr0r59e44fP+4bt3HjRm699VZq1KhB9erVefHFF1FV3/i5c+fSqFEjatWqRc2aNRk+fLgXm5CqNWvWcM8993gdRqpeeeUVqlWrxtVXX828efOSneb777/n2muvpW7duvTv358LFy4AEB0dTfv27WnQoAF16tThs88+A+DQoUO0adMm27YBVQ2pv+sqoLrwfk2vuLh4ffDBuQrP+f6aNPlEjxw5k+5lmayxadMmr0PQIkWK+F7fdddd+tJLL6mq6pkzZ7Rq1ao6b948VVU9ffq0tmnTRt977z1VVV2/fr1WrVpVN2/erKqqsbGxOnLkyCyNLTY2NtPL6Nq1q65duzZb15keGzdu1Pr162tMTIzu2LFDq1atqhcuXLhomri4OK1QoYJu2bJFVVWfeeYZ/eSTT1RV9eWXX9ZHH31UVVUPHjyopUqV0nPnzqmq6oABA3TZsmXJrje5zx6wWjOYl/LEyaaEzke++OJXX1lkZFWmT+9BkSIFPIzM+LwZxGOl/9a0pwGaNm3KunXrAJgwYQLNmjUjKioKgMKFC/Pee+/RsmVL7rvvPl5//XWeeuopatasCUC+fPn417/+lWSZp06dYujQoaxevRoR4f/+7/+48847KVq0KKdOOZfbTZs2jdmzZzNmzBgGDBhA6dKlWbNmDQ0bNmT69OmsXbuWkiVLAlCtWjWWL19OWFgYgwcPZvdup3vHt99+m2bNml207pMnT7Ju3ToaNGgAwKpVq3jooYc4e/YshQoV4rPPPuPqq69mzJgxfPPNN8TExHD69Gm+//573njjDaZMmcK5c+fo3Lkzzz//PACdOnViz549xMTE8OCDD3Lvvfem661IbObMmfTs2ZOCBQtSpUoVqlWrxqpVq2jatKlvmiNHjlCwYEFq1KgBQGRkJK+88gqDBg1CRDh58iSqyqlTpyhdujT58uXzxTp+/Pgk+yUYcn0ijYm5QI8e05g1a4uvrGvX2owb15mCBXP95psAxcXF8d133zFo0CDAadZfd911F01z1VVXcerUKU6cOMGGDRv497//neZyX3zxRUqUKMH69esBOHbsWJrzbN26lYULFxIeHk58fDzTp09n4MCB/Pjjj1SuXJlLL72U3r178/DDD3PTTTexe/duWrduzebNmy9azurVq6lb9+9roWvWrMmSJUvIly8fCxcu5Mknn+TLL78EYMWKFaxbt47SpUszf/58tm3bxqpVq1BVOnTowJIlS2jevDmjR4+mdOnSnD17luuvv54777yTMmXKXLTehx9+mEWLFiXZrp49e/L4449fVLZv3z6aNGniG65QoQL79u27aJqyZcsSGxvL6tWradSoEdOmTWPPnj0A3H///XTo0IErrriCkydPMnnyZMLCnCOWjRo14umnn05zf2eFXJ1JTpw4R8eOk/jhh12+snvuuYYPP7yD8PA8dXjYpODs2bM0bNiQXbt2cd111xEZGQk4h7xSOrubnrO+CxcuZNKkSb7hUqVKpTlPt27dCA8PB6BHjx688MILDBw4kEmTJtGjRw/fcjdt2uSb58SJE5w8eZJixYr5yvbv30+5cuV8w9HR0fTv359t27YhIsTGxvrGRUZGUrq000n5/PnzmT9/Ptdccw3g1Kq3bdtG8+bNeeedd5g+fToAe/bsYdu2bUkS6YgRIwLbOXDRMecEifeviDBp0iQefvhhzp07R1RUlK/WOW/ePBo2bMj333/P77//TmRkJDfffDPFixfnkksu4c8//0yy/GDItYn02LGzREaO5eef9/vKHnusGa+80souucmJAmx+Z7VChQqxdu1aoqOjueOOOxg5ciQPPPAAderUYcmSJRdNu2PHDooWLUqxYsWoU6cOP//8s6/ZnJKUErJ/WeI7u4oUKeJ73bRpU7Zv386hQ4eYMWOGr4YVHx/PihUrKFSoUKrb5r/sZ555hltuuYXp06eza9cuWrZsmew6VZUnnniCf/7znxct74cffmDhwoWsWLGCwoUL07Jly2TvSktPjbRChQq+2iU4N2lcccUVSeZt2rQpS5cuBZxEv3XrVgA+++wzHn/8cUSEatWqUaVKFX777TcaN25MTExMqvsnK+XaalnRogW47LKivuHXXruNV1+9zZKoSVaJEiV45513+M9//kNsbCx9+vRh2bJlLFy4EHBqrg888ACPPvooAI888gj/7//9P98XOj4+nrfeeivJcqOionjvvfd8wwlN+0svvZTNmzf7mu4pERE6d+7MsGHDqFWrlq/2l3i5a9euTTJvrVq12L59u284Ojqa8uXLAzBmzJgU19m6dWtGjx7tO4a7b98+Dh48SHR0NKVKlaJw4cL89ttvrFy5Mtn5R4wYwdq1a5P8JU6iAB06dGDSpEmcO3eOnTt3sm3bNho3bpxkuoMHDwJw7tw5XnvtNQYPHgxApUqV+O677wA4cOAAW7ZsoWrVqoBziMT/0EYw5dpEmj9/OFOmdKNFiyv5+OP2PPpo8A84m9B2zTXX0KBBAyZNmkShQoWYOXMmL730EldffTX16tXj+uuv5/777wegfv36vP322/Tq1YtatWpRt25d9u/fn2SZTz/9NMeOHaNu3bo0aNDAV1N79dVXueOOO7j11lu5/PLLU42rR48ejBs3ztesB3jnnXdYvXo19evXp3bt2nz44YdJ5qtZsybR0dGcPHkSgEcffZQnnniCZs2aERcXl+L6oqKi6N27N02bNqVevXp07dqVkydP0qZNGy5cuED9+vV55plnLjq2mVF16tShe/fu1K5dmzZt2jBy5EjfYY22bdv6muZvvPEGtWrVon79+rRv355bb70VcGrZ//vf/6hXrx6tWrXitddeo2zZsgAsWrSIdu3aZTrGQEhyxyhyskYVRVePuR9avRvQ9PHxSliY1UJzos2bN1OrVi2vw8jVRowYQbFixXL8taTB0Lx5c2bOnJnscenkPnsi8rOqZqi7t1xTI128eBcvv7wkSbklUZOXDRkyhIIFC3odRrY7dOgQw4YNC+jkXlbIFSebvv56C927TyMm5gKFC+fn4Yebpj2TMXlAREQE/fr18zqMbFeuXDk6deqUbesL+RrpuHHr6Nx5MjExzi1j//nPCk6cOOdxVCZQoXZoyYS+YHzmQjqRvvPOj/TrN524OGfHVK1aiqVLB1K8eN5ryoSiiIgIjhw5YsnUZBt1+yONiIjI0uWGZNNeFZ5/7geef36xr6xevUuYN68vl19eLJU5TU5SoUIF9u7dm+V9QxqTmoQe8rNSUBOpiLQB/guEA5+o6quJxhcEvgCuA44APVR1V6oLVXhwZBHenfF3Em3atALffNObUqWy5+JbkzXy58+fpb2UG+OVoDXtRSQcGAncDtQGeolI7USTDQKOqWo1YATwWlrL3XmsJO/O+Dthtm59FQsW9LMkaozxTDCPkTYGtqvqDlU9D0wCOiaapiPwuft6GtBK0rj16OiZvxNmjx51mDWrl/XgZIzxVDATaXlgj9/wXrcs2WlU9QIQDZQhFYJzYuKf/7yO8eO7UKBAeFbFa4wxGRLMY6TJ1SwTn54NZBpE5F4goePDc/DchlGjYNSoTEaYM5UFDnsdRBDl5u3LzdsGuX/7rs7ojMFMpHuBin7DFYDEfVolTLNXRPIBJYCjiRekqh8BHwGIyOqM3sYVCmz7Qldu3jbIG9uX0XmD2bT/CaguIlVEpADQE5iVaJpZQH/3dVfge7WLCo0xISZoNVJVvSAi9wPzcC5/Gq2qG0XkBZxno8wCPgXGish2nJpoz2DFY4wxwRLU60hVdQ4wJ1HZs36vY4Bu6VzsR1kQWk5m2xe6cvO2gW1fikKuGz1jjMlpQvpee2OMyQlybCIVkTYiskVEtotIkmcUiEhBEZnsjv9RRCpnf5QZF8D2DRORTSKyTkS+E5ErvYgzI9LaNr/puoqIikhInQkOZPtEpLv7/m0UkQnZHWNmBPDZrCQii0Rkjfv5bOtFnBkhIqNF5KCIbEhhvIjIO+62rxORawNacFoPvvfiD+fk1O9AVaAA8CtQO9E0/wI+dF/3BCZ7HXcWb98tQGH39ZBQ2b5Ats2drhiwBFgJNPI67ix+76oDa4BS7vAlXsedxdv3ETDEfV0b2OV13OnYvubAtcCGFMa3BebiXOPeBPgxkOXm1BppUG4vzUHS3D5VXaSqZ9zBlTjX4YaCQN47gBeB14Gkj6HM2QLZvn8AI1X1GICqHszmGDMjkO1ToLj7ugRJrw/PsVR1Cclcq+6nI/CFOlYCJUUk9YdqkXOb9kG5vTQHCWT7/A3C+ZUMBWlum4hcA1RU1dnZGVgWCeS9qwHUEJHlIrLS7QUtVASyfc8BfUVkL85VOUOzJ7Rskd7vJpBz+yPNsttLc6iAYxeRvkAjoEVQI8o6qW6biITh9PQ1ILsCymKBvHf5cJr3LXFaEktFpK6qHg9ybFkhkO3rBYxR1TdFpCnOteB1VTU++OEFXYbySk6tkabn9lJSu700hwpk+xCR24CngA6qGirPT0lr24oBdYEfRGQXznGoWSF0winQz+ZMVY1V1Z3AFpzEGgoC2b5BwBQAVV0BRODch58bBPTdTCynJtLcfntpmtvnNn9H4STRUDrGluq2qWq0qpZV1cqqWhnn+G8HVc3wfc7ZLJDP5gyck4WISFmcpv6ObI0y4wLZvt1AKwARqYWTSHPLYw5mAXe5Z++bANGquj/Nubw+i5bK2bW2wFacM4hPuWUv4HzpwHnzpgLbgVVAVa9jzuLtWwgcANa6f7O8jjmrti3RtD8QQmftA3zvBHgL2ASsB3p6HXMWb19tYDnOGf21QJTXMadj2yYC+4FYnNrnIGAwMNjvvRvpbvv6QD+bdmeTMcZkUk5t2htjTMiwRGqMMZlkidQYYzLJEqkxxmSSJVJjjMkkS6QmTSISJyJr/f4qpzJt5ZR61knnOn9weyD61b3VMt0PJhORwSJyl/t6gIhc4TfuExGpncVx/iQiDQOY5yERKZzZdZucwxKpCcRZVW3o97crm9bbR1Ub4HRO80Z6Z1bVD1X1C3dwAHCF37h7VHVTlkT5d5zvE1icDwGWSHMRS6QmQ9ya51IR+cX9uzGZaeqIyCq3FrtORKq75X39ykeJSHgaq1sCVHPnbeX2g7ne7VuyoFv+ql//rf9xy54TkeEi0hWnv4Lx7joLuTXJRiIyRERe94t5gIi8m8E4V+DXwYWIfCAiq8Xpk/R5t+wBnIS+SEQWuWVRIrLC3Y9TRaRoGusxOYwlUhOIQn7N+ulu2UEgUlWvBXoA7yQz32Dgv6raECeR7XVvKewBNHPL44A+aay/PbBeRCKAMUAPVa2H0znIEBEpDXQG6qhqfeAl/5lVdRqwGqfm2FBVz/qNngZ08RvuAUzOYJxtcG4PTfCUOo8vrg+0EJH6qvoOzr3bt6jqLe4tpE8Dt7n7cjUwLI31mBwmp/b+ZHKWs2U5vAAAAiJJREFUs24y8ZcfeM89JhiHcz95YiuAp0SkAvCVqm4TkVbAdcBPbvexhXCScnLGi8hZYBdOV21XAztVdas7/nPgPuA9nH5NPxGRb4CAu+dT1UMissO9r3qbu47l7nLTE2cRnE6R/XtU7y4i9+J8zy7HubVyXaJ5m7jly931FMDZbyaEWCI1GfUwTl8ADXBaNkk6aFbVCSLyI9AOmCci9+Dcy/y5qj4RwDr6qF9nJiKSbH+z6jz6uzFORxo9gfuBW9OxLZOB7sBvwHRVVXGyWsBx4tx3/irOfdpdRKQKMBy4XlWPicgYnP4hEhNggar2Ske8Joexpr3JqBLAfnX6oOyHUxu7iIhUBXa4zdlZOE3c74CuInKJO01pCfx5VL8BlUWkmjvcD1jsHlMsoc7jvx8CkjtzfhKnC7/kfAV0wulnc7Jblq44VTUWp4nexD0sUBw4DUSLyKXA7SnEshJolrBNIlJYRJKr3ZsczBKpyaj3gf4ishKnWX86mWl6ABtEZC1QE+cRDptwEs58EVkHLMBp9qZJVWOAgcBUEVkPxAMf4iSl2e7yFuPUlhMbA3yYcLIp0XKP4fTUdKWqrnLL0h2ne+z1TWC4qv6K89ymjcBonMMFCT4C5orIIlU9hHNFwUR3PStx9pUJIdb7kzHGZJLVSI0xJpMskRpjTCZZIjXGmEyyRGqMMZlkidQYYzLJEqkxxmSSJVJjjMkkS6TGGJNJ/x/R9T4MFNeC5QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sklearn AUC: 0.981115695653423\n",
      "\n",
      "xgb_predictor.endpoint = TC58-SW010-WC006-20210405151047\n",
      "\n",
      "s3_wholedata_path = s3://sagemaker-cn-northwest-1-337575217701/xgb-10000871-part_ac.csv-TC58-SW010-WC006-20210405135629/wholedata/whole_data.csv\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(None,)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from numpy import asarray\n",
    "from pandas import read_csv\n",
    "from pandas import DataFrame\n",
    "from pandas import concat\n",
    "#from xgboost import XGBRegressor\n",
    "%matplotlib inline\n",
    "from matplotlib import pyplot\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "\n",
    "y_labels = model_data.iloc[:, 0]\n",
    "y_scores = predictions\n",
    "\n",
    "#结果 1：测试数据集结果 与 预测结果 的数量对比\n",
    "print('model_data: 1='+ str((y_labels == 1).sum()) + ', 0='+str((y_labels == 0).sum()) +' / '),\n",
    "print('predictions Result: 1='+ str((np.round(y_scores) == 1).sum()) + ', 0='+str((np.round(y_scores) == 0).sum()))\n",
    "\n",
    "#结果 2：交叉表，说明 测试数据集结果 与 预测结果 的正确情况对比\n",
    "display(pd.crosstab(index=model_data.iloc[:, 0], columns=np.round(predictions), \n",
    "            rownames=['model_data'], colnames=['predictions_data'],margins=False))\n",
    "#print('\\n')\n",
    "\n",
    "#结果 4：ROC图，越大越好，无限接近与1\n",
    "from sklearn.metrics import roc_curve, auc                     #导入库\n",
    "fpr, tpr, thresholds  =  roc_curve(y_labels, y_scores)  #计算真正率和假正率\n",
    "roc_auc = auc(fpr,tpr)                                         #计算auc的值\n",
    "plt.figure()\n",
    "lw = 3\n",
    "plt.figure(figsize=(5,5))\n",
    "plt.plot(fpr, tpr, color='darkorange',lw=lw, label='ROC curve (area = %0.2f)' % roc_auc) #假正率为横坐标，真正率为纵坐标做曲线\n",
    "plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Receiver operating characteristic example')\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.show()\n",
    "\n",
    "#结果 3：AUC值，越大越好，无限接近与1\n",
    "import numpy as np\n",
    "from sklearn.metrics import roc_auc_score\n",
    "##y_labels=[0. 1. 0. 0. 1. 1. 1. 1. 1. 1. 0. 1. 1. 1. 0. 0. 1. 0. 1. 0. 0. 0. 1. 1.\n",
    "##y_scores=[4.92438182e-01 1.74370125e-01 8.58997216e-01 1.30175965e-01 \n",
    "# 调用sklearn中的方法计算AUC，与后面自己写的方法作对比\n",
    "print('sklearn AUC:', roc_auc_score(y_labels, y_scores))\n",
    "#打印全局变量值\n",
    "print('\\nxgb_predictor.endpoint = ' + str(xgb_predictor.endpoint)),\n",
    "print('\\ns3_wholedata_path = '+ str(s3_wholedata_path)),"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAQHElEQVR4nO3da4xdV3nG8f/TmEBLAecyiSLbrUGYlggJEo1SIyQKmKLEVDgfkiqoNG5k1YKmiIpKrVs+9PohqVTSRkJpLUJxEJekaWksSGlTJxEtqgOTJuQKzZCm8chpPDSJWxpxSXn74SzDxB57jj3nzGSW/z/paO+99jqz3+UzfmbPOvvsSVUhSerLjyx3AZKk0TPcJalDhrskdchwl6QOGe6S1KFVy10AwJlnnlnr169f7jIkaUW5++67v1lVE/Pte0GE+/r165mamlruMiRpRUnyH0fb57SMJHXIcJekDhnuktQhw12SOmS4S1KHDHdJ6pDhLkkdMtwlqUOGuyR16AXxCdXFWL/j88t27MeueueyHVuSjsUzd0nqkOEuSR0y3CWpQ4a7JHXIcJekDhnuktQhw12SOmS4S1KHDHdJ6pDhLkkdGirck6xOcnOSryV5OMkbk5ye5LYkj7Tlaa1vklybZDrJfUnOH+8QJEmHG/bM/c+AL1TVTwOvBx4GdgB7qmoDsKdtA1wEbGiP7cB1I61YkrSgBcM9ycuBNwPXA1TVd6vqGWALsKt12wVc3Na3ADfUwF5gdZJzRl65JOmohjlzfxUwC/xlknuSfDTJS4Gzq+oJgLY8q/VfA+yb8/yZ1vY8SbYnmUoyNTs7u6hBSJKeb5hwXwWcD1xXVecB/8sPp2Dmk3na6oiGqp1VNVlVkxMTE0MVK0kazjDhPgPMVNVdbftmBmH/5KHplrY8MKf/ujnPXwvsH025kqRhLBjuVfWfwL4kP9WaNgEPAbuBra1tK3BLW98NXN6umtkIHDw0fSNJWhrD/iWm9wOfTHIq8ChwBYMfDDcl2QY8Dlza+t4KbAamgWdbX0nSEhoq3KvqXmBynl2b5ulbwJWLrEuStAh+QlWSOmS4S1KHDHdJ6pDhLkkdMtwlqUOGuyR1yHCXpA4Z7pLUIcNdkjpkuEtShwx3SeqQ4S5JHTLcJalDhrskdchwl6QOGe6S1CHDXZI6ZLhLUocMd0nqkOEuSR0y3CWpQ4a7JHXIcJekDg0V7kkeS3J/knuTTLW205PcluSRtjyttSfJtUmmk9yX5PxxDkCSdKTjOXN/a1W9oaom2/YOYE9VbQD2tG2Ai4AN7bEduG5UxUqShrOYaZktwK62vgu4eE77DTWwF1id5JxFHEeSdJyGDfcC/iHJ3Um2t7azq+oJgLY8q7WvAfbNee5Ma3ueJNuTTCWZmp2dPbHqJUnzWjVkvzdV1f4kZwG3JfnaMfpmnrY6oqFqJ7ATYHJy8oj9kqQTN9SZe1Xtb8sDwGeBC4AnD023tOWB1n0GWDfn6WuB/aMqWJK0sAXDPclLk7zs0DrwDuABYDewtXXbCtzS1ncDl7erZjYCBw9N30iSlsYw0zJnA59Ncqj/p6rqC0m+AtyUZBvwOHBp638rsBmYBp4Frhh51ZKkY1ow3KvqUeD187T/F7BpnvYCrhxJdZKkE+InVCWpQ4a7JHXIcJekDhnuktQhw12SOmS4S1KHDHdJ6pDhLkkdMtwlqUOGuyR1yHCXpA4Z7pLUIcNdkjpkuEtShwx3SeqQ4S5JHTLcJalDhrskdchwl6QOGe6S1CHDXZI6ZLhLUocMd0nq0NDhnuSUJPck+VzbfmWSu5I8kuTGJKe29he37em2f/14SpckHc3xnLl/AHh4zvbVwDVVtQF4GtjW2rcBT1fVq4FrWj9J0hIaKtyTrAXeCXy0bQd4G3Bz67ILuLitb2nbtP2bWn9J0hIZ9sz9T4HfBL7fts8Anqmq59r2DLCmra8B9gG0/Qdb/+dJsj3JVJKp2dnZEyxfkjSfBcM9yc8DB6rq7rnN83StIfb9sKFqZ1VNVtXkxMTEUMVKkoazaog+bwLelWQz8BLg5QzO5FcnWdXOztcC+1v/GWAdMJNkFfAK4KmRVy5JOqoFz9yr6reram1VrQcuA26vql8E7gAuad22Are09d1tm7b/9qo64sxdkjQ+i7nO/beADyaZZjCnfn1rvx44o7V/ENixuBIlScdrmGmZH6iqO4E72/qjwAXz9Pk2cOkIapMknSA/oSpJHTLcJalDhrskdchwl6QOGe6S1CHDXZI6ZLhLUocMd0nqkOEuSR0y3CWpQ4a7JHXIcJekDhnuktQhw12SOmS4S1KHDHdJ6pDhLkkdMtwlqUOGuyR1yHCXpA4Z7pLUIcNdkjpkuEtShxYM9yQvSfLlJF9N8mCS32/tr0xyV5JHktyY5NTW/uK2Pd32rx/vECRJhxvmzP07wNuq6vXAG4ALk2wErgauqaoNwNPAttZ/G/B0Vb0auKb1kyQtoQXDvQa+1TZf1B4FvA24ubXvAi5u61vaNm3/piQZWcWSpAUNNeee5JQk9wIHgNuAbwDPVNVzrcsMsKatrwH2AbT9B4Ez5vma25NMJZmanZ1d3CgkSc8zVLhX1f9V1RuAtcAFwGvn69aW852l1xENVTurarKqJicmJoatV5I0hOO6WqaqngHuBDYCq5OsarvWAvvb+gywDqDtfwXw1CiKlSQNZ5irZSaSrG7rPwq8HXgYuAO4pHXbCtzS1ne3bdr+26vqiDN3SdL4rFq4C+cAu5KcwuCHwU1V9bkkDwGfSfJHwD3A9a3/9cAnkkwzOGO/bAx1S5KOYcFwr6r7gPPmaX+Uwfz74e3fBi4dSXWSpBPiJ1QlqUOGuyR1yHCXpA4Z7pLUIcNdkjpkuEtShwx3SeqQ4S5JHTLcJalDhrskdchwl6QOGe6S1CHDXZI6ZLhLUocMd0nqkOEuSR0y3CWpQ4a7JHXIcJekDhnuktQhw12SOmS4S1KHDHdJ6tCC4Z5kXZI7kjyc5MEkH2jtpye5LckjbXlaa0+Sa5NMJ7kvyfnjHoQk6fmGOXN/DviNqnotsBG4Msm5wA5gT1VtAPa0bYCLgA3tsR24buRVS5KOacFwr6onqupf2/r/AA8Da4AtwK7WbRdwcVvfAtxQA3uB1UnOGXnlkqSjOq459yTrgfOAu4Czq+oJGPwAAM5q3dYA++Y8baa1SZKWyNDhnuTHgb8Gfr2q/vtYXedpq3m+3vYkU0mmZmdnhy1DkjSEocI9yYsYBPsnq+pvWvOTh6Zb2vJAa58B1s15+lpg/+Ffs6p2VtVkVU1OTEycaP2SpHkMc7VMgOuBh6vqw3N27Qa2tvWtwC1z2i9vV81sBA4emr6RJC2NVUP0eRPwS8D9Se5tbb8DXAXclGQb8Dhwadt3K7AZmAaeBa4YacWSpAUtGO5V9c/MP48OsGme/gVcuci6JEmL4CdUJalDhrskdchwl6QOGe6S1CHDXZI6ZLhLUocMd0nqkOEuSR0y3CWpQ4a7JHXIcJekDhnuktQhw12SOmS4S1KHDHdJ6pDhLkkdMtwlqUOGuyR1yHCXpA4Z7pLUIcNdkjpkuEtShwx3SeqQ4S5JHVow3JN8LMmBJA/MaTs9yW1JHmnL01p7klybZDrJfUnOH2fxkqT5DXPm/nHgwsPadgB7qmoDsKdtA1wEbGiP7cB1oylTknQ8Fgz3qvoi8NRhzVuAXW19F3DxnPYbamAvsDrJOaMqVpI0nBOdcz+7qp4AaMuzWvsaYN+cfjOt7QhJtieZSjI1Ozt7gmVIkuYz6jdUM09bzdexqnZW1WRVTU5MTIy4DEk6uZ1ouD95aLqlLQ+09hlg3Zx+a4H9J16eJOlEnGi47wa2tvWtwC1z2i9vV81sBA4emr6RJC2dVQt1SPJp4C3AmUlmgN8FrgJuSrINeBy4tHW/FdgMTAPPAleMoWZJ0gIWDPeqevdRdm2ap28BVy62KEnS4vgJVUnqkOEuSR0y3CWpQ4a7JHVowTdUJal363d8ftmO/dhV7xzL1/XMXZI6ZLhLUocMd0nqkOEuSR0y3CWpQ4a7JHXIcJekDhnuktQhw12SOmS4S1KHDHdJ6pDhLkkdMtwlqUOGuyR1yHCXpA4Z7pLUIf9YxyIs1w3+x3Vzf0n98Mxdkjo0lnBPcmGSryeZTrJjHMeQJB3dyKdlkpwCfAT4OWAG+EqS3VX10KiPdbLq8e89Shqtccy5XwBMV9WjAEk+A2wBDPcOnGzvMyznD9Llspw/wE/Gf+9xGUe4rwH2zdmeAX7m8E5JtgPb2+a3knz9BI93JvDNE3zuSnXSjTlXn3xjZple51y91Ed8npPudV7k9/ZPHm3HOMI987TVEQ1VO4Gdiz5YMlVVk4v9OiuJYz45OOaTw7jGPI43VGeAdXO21wL7x3AcSdJRjCPcvwJsSPLKJKcClwG7x3AcSdJRjHxapqqeS/JrwN8DpwAfq6oHR32cORY9tbMCOeaTg2M+OYxlzKk6YjpckrTC+QlVSeqQ4S5JHVox4b7QLQ2SvDjJjW3/XUnWL32VozXEmD+Y5KEk9yXZk+So17yuFMPeuiLJJUkqyYq/bG6YMSf5hfZaP5jkU0td46gN8b39E0nuSHJP+/7evBx1jkqSjyU5kOSBo+xPkmvbv8d9Sc5f9EGr6gX/YPDG7DeAVwGnAl8Fzj2sz68Cf97WLwNuXO66l2DMbwV+rK2/72QYc+v3MuCLwF5gcrnrXoLXeQNwD3Ba2z5ruetegjHvBN7X1s8FHlvuuhc55jcD5wMPHGX/ZuDvGHxOaCNw12KPuVLO3H9wS4Oq+i5w6JYGc20BdrX1m4FNSeb7QNVKseCYq+qOqnq2be5l8JmClWyY1xngD4E/Br69lMWNyTBj/hXgI1X1NEBVHVjiGkdtmDEX8PK2/gpW+GdlquqLwFPH6LIFuKEG9gKrk5yzmGOulHCf75YGa47Wp6qeAw4CZyxJdeMxzJjn2sbgJ/9KtuCYk5wHrKuqzy1lYWM0zOv8GuA1Sb6UZG+SC5esuvEYZsy/B7wnyQxwK/D+pSlt2Rzv//cFrZQ/1jHMLQ2Guu3BCjL0eJK8B5gEfnasFY3fMcec5EeAa4BfXqqClsAwr/MqBlMzb2Hw29k/JXldVT0z5trGZZgxvxv4eFX9SZI3Ap9oY/7++MtbFiPPr5Vy5j7MLQ1+0CfJKga/yh3r16AXuqFu45Dk7cCHgHdV1XeWqLZxWWjMLwNeB9yZ5DEGc5O7V/ibqsN+b99SVd+rqn8Hvs4g7FeqYca8DbgJoKr+BXgJg5uK9Wrkt21ZKeE+zC0NdgNb2/olwO3V3qlYoRYcc5ui+AsGwb7S52FhgTFX1cGqOrOq1lfVegbvM7yrqqaWp9yRGOZ7+28ZvHlOkjMZTNM8uqRVjtYwY34c2ASQ5LUMwn12SatcWruBy9tVMxuBg1X1xKK+4nK/i3wc7zZvBv6NwbvsH2ptf8DgPzcMXvy/AqaBLwOvWu6al2DM/wg8CdzbHruXu+Zxj/mwvneywq+WGfJ1DvBhBn8T4X7gsuWueQnGfC7wJQZX0twLvGO5a17keD8NPAF8j8FZ+jbgvcB757zGH2n/HveP4vva2w9IUodWyrSMJOk4GO6S1CHDXZI6ZLhLUocMd0nqkOEuSR0y3CWpQ/8P9n6SwmTbRSgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>predictions_data</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>model_data</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.0</th>\n",
       "      <td>663</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>6</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "predictions_data    0   1\n",
       "model_data               \n",
       "0.0               663  2 \n",
       "1.0               6    70"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#结果 5：分布柱状图；用于分析和优化结果（round函数）；\n",
    "plt.hist(predictions, bins=10, range=(0, 1))\n",
    "plt.show()\n",
    "\n",
    "#用于分析和优化结果（round函数）；\n",
    "round_predictions=0.45\n",
    "pd.crosstab(index=model_data.iloc[:, 0], columns=np.where(predictions > round_predictions, 1, 0), \n",
    "            rownames=['model_data'], colnames=['predictions_data'],margins=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 结束"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "instance_type": "ml.m5.4xlarge",
  "kernelspec": {
   "display_name": "Python 3 (arn:aws:sagemaker:cn-northwest-1:390780980154:image/datascience-1.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:cn-northwest-1:390780980154:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}