sagemaker-train-deploy/gpt2-news-classifier-sagemaker-train-deploy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8787d49b",
   "metadata": {},
   "source": [
    "# GPT-2 News Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9a3d8ef",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cab2f762",
   "metadata": {},
   "source": [
    "In this notebook, I will demonstrate how to train and deploy a GPT-2-fine-tuned text classification model using Amazon SageMaker.\n",
    "\n",
    "Here I picked text-classification as a sample use case, but the same approach can be applied to any kind of machine learning model: from RandomForest classification models to YOLO object-detection models. On this news classifier application, a user types/pastes a news paragraph and its category can be predicted with probability. To achieve this we will fine-tune a GPT-2 model and deploy it on this SageMaker Notebook.\n",
    "\n",
    "I used Pytorch and HuggingFace transformers library for the model development. For more information about the PyTorch in SageMaker, please visit [sagemaker-pytorch-training-toolkit](https://github.com/aws/sagemaker-pytorch-training-toolkit) and [sagemaker-python-sdk](https://github.com/aws/sagemaker-python-sdk) github repositories."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "300e607b",
   "metadata": {},
   "source": [
    "## Packages"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "273ddb7f",
   "metadata": {},
   "source": [
    "First, install all of the required Python packages on this SageMaker Notebook instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "44703e42",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install -U sagemaker\n",
    "# !pip3 install torch==1.10.1\n",
    "# !pip3 install transformers==4.15.0\n",
    "# !pip3 install pandas\n",
    "# !pip3 install numpy\n",
    "# !pip3 install tqdm==4.62.3\n",
    "# !pip3 install streamlit"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "039d264e",
   "metadata": {},
   "source": [
    "Then, import all of the required Python packages for this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "af7dfc43",
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports\n",
    "import pandas as pd\n",
    "import torch\n",
    "import sagemaker\n",
    "\n",
    "from sagemaker.pytorch import PyTorch\n",
    "from sagemaker.pytorch.model import PyTorchModel\n",
    "from sagemaker.predictor import json_deserializer, json_serializer\n",
    "from transformers import GPT2Tokenizer\n",
    "\n",
    "import utils"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf22087d",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cc91415",
   "metadata": {},
   "source": [
    "This notebook was created and tested on an ml.t2.medium notebook instance.\n",
    "\n",
    "Let's start by creating a **SageMaker session** and specifying two parameters: **S3 bucket** and **prefix** that you want to use for training and model data. This should be within the same region as the Notebook Instance, training, and hosting.\n",
    "\n",
    "The **IAM role** are used to give training and hosting access to your data. See the documentation for how to create these. Note, if more than one role is required for notebook instances, training, and/or hosting, please replace the sagemaker.get_execution_role() with a the appropriate full IAM role arn string(s)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "a8a6edf9",
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker_session = sagemaker.Session()\n",
    "role = sagemaker.get_execution_role()\n",
    "\n",
    "bucket = \"gpt2-news-classifier\"\n",
    "prefix = \"data\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92b06d2b",
   "metadata": {},
   "source": [
    "## Read Data from S3 bucket"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8adc08e6",
   "metadata": {},
   "source": [
    "Before doing this step, make sure you create a S3 bucket (\"gpt2-news-classifier\") in your AWS console (see [https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html)), and upload our `data` folder (under the same directory of this notebook) to the S3 bucket. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "631f8830",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data_key = \"train/train.csv\"\n",
    "val_data_key = \"val/val.csv\"\n",
    "test_data_key = \"test/test.csv\"\n",
    "\n",
    "train_data_location = 's3://{}/{}/{}'.format(bucket, prefix, train_data_key)\n",
    "val_data_location = 's3://{}/{}/{}'.format(bucket, prefix, val_data_key)\n",
    "test_data_location = 's3://{}/{}/{}'.format(bucket, prefix, test_data_key)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "863e7851",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>category</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>848</td>\n",
       "      <td>tech</td>\n",
       "      <td>slimmer playstation triple sales sony playstat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>899</td>\n",
       "      <td>sport</td>\n",
       "      <td>williams stays on despite dispute matt william...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>553</td>\n",
       "      <td>tech</td>\n",
       "      <td>games win for blu-ray dvd format the next-gene...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2071</td>\n",
       "      <td>business</td>\n",
       "      <td>lesotho textile workers lose jobs six foreign-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1155</td>\n",
       "      <td>politics</td>\n",
       "      <td>bid to cut court witness stress new targets to...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0  category                                               text\n",
       "0         848      tech  slimmer playstation triple sales sony playstat...\n",
       "1         899     sport  williams stays on despite dispute matt william...\n",
       "2         553      tech  games win for blu-ray dvd format the next-gene...\n",
       "3        2071  business  lesotho textile workers lose jobs six foreign-...\n",
       "4        1155  politics  bid to cut court witness stress new targets to..."
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(train_data_location).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "41e355d1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>category</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>820</td>\n",
       "      <td>sport</td>\n",
       "      <td>hantuchova in dubai last eight daniela hantuch...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>800</td>\n",
       "      <td>tech</td>\n",
       "      <td>more women turn to net security older people a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>98</td>\n",
       "      <td>business</td>\n",
       "      <td>japan narrowly escapes recession japan s econo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>310</td>\n",
       "      <td>politics</td>\n",
       "      <td>mps issued with blackberry threat mps will be ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>623</td>\n",
       "      <td>sport</td>\n",
       "      <td>bryan twins keep us hopes alive the united sta...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0  category                                               text\n",
       "0         820     sport  hantuchova in dubai last eight daniela hantuch...\n",
       "1         800      tech  more women turn to net security older people a...\n",
       "2          98  business  japan narrowly escapes recession japan s econo...\n",
       "3         310  politics  mps issued with blackberry threat mps will be ...\n",
       "4         623     sport  bryan twins keep us hopes alive the united sta..."
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(val_data_location).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "da7f2e17",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>category</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1510</td>\n",
       "      <td>business</td>\n",
       "      <td>soros group warns of kazakh close the open soc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>962</td>\n",
       "      <td>entertainment</td>\n",
       "      <td>german music in a  zombie  state the german mu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>588</td>\n",
       "      <td>sport</td>\n",
       "      <td>mourinho takes swipe at arsenal chelsea boss j...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1673</td>\n",
       "      <td>entertainment</td>\n",
       "      <td>mogul wilson backing uk rap band tony wilson  ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2171</td>\n",
       "      <td>sport</td>\n",
       "      <td>palace threat over cantona masks manchester un...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0       category  \\\n",
       "0        1510       business   \n",
       "1         962  entertainment   \n",
       "2         588          sport   \n",
       "3        1673  entertainment   \n",
       "4        2171          sport   \n",
       "\n",
       "                                                text  \n",
       "0  soros group warns of kazakh close the open soc...  \n",
       "1  german music in a  zombie  state the german mu...  \n",
       "2  mourinho takes swipe at arsenal chelsea boss j...  \n",
       "3  mogul wilson backing uk rap band tony wilson  ...  \n",
       "4  palace threat over cantona masks manchester un...  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.read_csv(test_data_location).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb01cf13",
   "metadata": {},
   "source": [
    "## Train GPT-2 model (Fine-tuning a pre-trained GPT-2 model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1a4e902",
   "metadata": {},
   "source": [
    "### Training Script"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5bb1e30",
   "metadata": {},
   "source": [
    "Our training script `train_deploy.py` provides all the code we need for training and hosting a SageMaker model (model_fn function to load a model). The training script is very similar to a training script you might run outside of SageMaker, but you can access useful properties about the training environment through various environment variables, such as:\n",
    "\n",
    "`SM_MODEL_DIR`: A string representing the path to the directory to write model artifacts to. These artifacts are uploaded to S3 for model hosting.\n",
    "`SM_NUM_GPUS`: The number of gpus available in the current container.\n",
    "`SM_CURRENT_HOST`: The name of the current container on the container network.\n",
    "`SM_HOSTS`: JSON encoded list containing all the hosts .\n",
    "Supposing one input channel, 'training', was used in the call to the PyTorch estimator's `fit()` method, the following will be set, following the format `SM_CHANNEL_[channel_name]`:\n",
    "- `SM_CHANNEL_TRAIN`: A string representing the path to the directory containing data in the `train` channel.\n",
    "\n",
    "For more information about training environment variables, please visit [SageMaker Containers](https://github.com/aws/sagemaker-containers).\n",
    "\n",
    "A typical training script loads data from the input channels, configures training with hyperparameters, trains a model, and saves a model to model_dir so that it can be hosted later. Hyperparameters are passed to your script as arguments and can be retrieved with an `argparse.ArgumentParser` instance.\n",
    "\n",
    "Because the SageMaker imports the training script, you should put your training code in a main guard (if `__name__=='__main__'`:) if you are using the same script to host your model as we do in this example, so that SageMaker does not inadvertently run your training code at the wrong point in execution.\n",
    "\n",
    "For example, the script run by this notebook (uncomment to see the code):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "18972725",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment to see the script code. \n",
    "# !pygmentize ./code/train_deploy.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c370e2a4",
   "metadata": {},
   "source": [
    "### Run traning in SageMaker Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24195a33",
   "metadata": {},
   "source": [
    "SageMaker's `PyTorch` class allows us to run our training function as a training job on SageMaker infrastructure. We need to configure it with our training script, an IAM role, the number of training instances, the training instance type, and hyperparameters. In this case we are going to run our training job on 1 ml.m5.4xlarge instance. But this example can be ran on one or multiple, cpu or gpu instances ([see full list of available instances](https://aws.amazon.com/sagemaker/pricing/). The hyperparameters parameter is a dict of values that will be passed to your training script -- you can see how to access these values in the `train_deploy.py` script above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "d67a5b6a",
   "metadata": {},
   "outputs": [],
   "source": [
    "pytorch_estimator = PyTorch(entry_point='train_deploy.py',\n",
    "                            source_dir=\"code\",\n",
    "                            role=role,\n",
    "                            instance_type='ml.m5.4xlarge',\n",
    "                            instance_count=1,\n",
    "                            framework_version='1.10',\n",
    "                            py_version='py38',\n",
    "                            hyperparameters = {\n",
    "                                'epochs': 1, \n",
    "                                'batch-size': 2, \n",
    "                                'lr': 1e-5\n",
    "                            })"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18f0c72e",
   "metadata": {},
   "source": [
    "After we've constructed our PyTorch object, we can fit it using the data we uploaded to S3 bucket. SageMaker makes sure our data is available in the local filesystem, so our training script can simply read the data from disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "56f6ce05",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-01-26 20:00:11 Starting - Starting the training job...\n",
      "2022-01-26 20:00:23 Starting - Launching requested ML instancesProfilerReport-1643227211: InProgress\n",
      "......\n",
      "2022-01-26 20:01:41 Starting - Preparing the instances for training.........\n",
      "2022-01-26 20:03:13 Downloading - Downloading input data...\n",
      "2022-01-26 20:03:38 Training - Downloading the training image.....\u001b[34mbash: cannot set terminal process group (-1): Inappropriate ioctl for device\u001b[0m\n",
      "\u001b[34mbash: no job control in this shell\u001b[0m\n",
      "\u001b[34m2022-01-26 20:04:25,754 sagemaker-training-toolkit INFO     Imported framework sagemaker_pytorch_container.training\u001b[0m\n",
      "\u001b[34m2022-01-26 20:04:25,756 sagemaker-training-toolkit INFO     No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34m2022-01-26 20:04:25,763 sagemaker_pytorch_container.training INFO     Block until all host DNS lookups succeed.\u001b[0m\n",
      "\n",
      "2022-01-26 20:04:38 Training - Training image download completed. Training in progress.\u001b[34m2022-01-26 20:04:31,986 sagemaker_pytorch_container.training INFO     Invoking user training script.\u001b[0m\n",
      "\u001b[34m2022-01-26 20:04:32,412 sagemaker-training-toolkit INFO     Installing dependencies from requirements.txt:\u001b[0m\n",
      "\u001b[34m/opt/conda/bin/python3.8 -m pip install -r requirements.txt\u001b[0m\n",
      "\u001b[34mCollecting torch==1.10.1\u001b[0m\n",
      "\u001b[34mDownloading torch-1.10.1-cp38-cp38-manylinux1_x86_64.whl (881.9 MB)\u001b[0m\n",
      "\u001b[34mCollecting transformers==4.15.0\u001b[0m\n",
      "\u001b[34mDownloading transformers-4.15.0-py3-none-any.whl (3.4 MB)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: pandas in /opt/conda/lib/python3.8/site-packages (from -r requirements.txt (line 3)) (1.3.4)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: numpy in /opt/conda/lib/python3.8/site-packages (from -r requirements.txt (line 4)) (1.21.2)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: tqdm in /opt/conda/lib/python3.8/site-packages (from -r requirements.txt (line 5)) (4.62.3)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: typing-extensions in /opt/conda/lib/python3.8/site-packages (from torch==1.10.1->-r requirements.txt (line 1)) (4.0.0)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.8/site-packages (from transformers==4.15.0->-r requirements.txt (line 2)) (5.4.1)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.8/site-packages (from transformers==4.15.0->-r requirements.txt (line 2)) (21.3)\u001b[0m\n",
      "\u001b[34mCollecting regex!=2019.12.17\u001b[0m\n",
      "\u001b[34mDownloading regex-2022.1.18-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (764 kB)\u001b[0m\n",
      "\u001b[34mCollecting tokenizers<0.11,>=0.10.1\u001b[0m\n",
      "\u001b[34mDownloading tokenizers-0.10.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (3.3 MB)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: requests in /opt/conda/lib/python3.8/site-packages (from transformers==4.15.0->-r requirements.txt (line 2)) (2.26.0)\u001b[0m\n",
      "\u001b[34mCollecting huggingface-hub<1.0,>=0.1.0\u001b[0m\n",
      "\u001b[34mDownloading huggingface_hub-0.4.0-py3-none-any.whl (67 kB)\u001b[0m\n",
      "\u001b[34mCollecting filelock\u001b[0m\n",
      "\u001b[34mDownloading filelock-3.4.2-py3-none-any.whl (9.9 kB)\u001b[0m\n",
      "\u001b[34mCollecting sacremoses\u001b[0m\n",
      "\u001b[34mDownloading sacremoses-0.0.47-py2.py3-none-any.whl (895 kB)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/lib/python3.8/site-packages (from pandas->-r requirements.txt (line 3)) (2.8.2)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.8/site-packages (from pandas->-r requirements.txt (line 3)) (2021.3)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.8/site-packages (from packaging>=20.0->transformers==4.15.0->-r requirements.txt (line 2)) (3.0.6)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: six>=1.5 in /opt/conda/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas->-r requirements.txt (line 3)) (1.16.0)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.8/site-packages (from requests->transformers==4.15.0->-r requirements.txt (line 2)) (2021.10.8)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.8/site-packages (from requests->transformers==4.15.0->-r requirements.txt (line 2)) (3.2)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.8/site-packages (from requests->transformers==4.15.0->-r requirements.txt (line 2)) (1.26.7)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/lib/python3.8/site-packages (from requests->transformers==4.15.0->-r requirements.txt (line 2)) (2.0.4)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: click in /opt/conda/lib/python3.8/site-packages (from sacremoses->transformers==4.15.0->-r requirements.txt (line 2)) (8.0.3)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: joblib in /opt/conda/lib/python3.8/site-packages (from sacremoses->transformers==4.15.0->-r requirements.txt (line 2)) (1.1.0)\u001b[0m\n",
      "\u001b[34mInstalling collected packages: regex, filelock, tokenizers, sacremoses, huggingface-hub, transformers, torch\u001b[0m\n",
      "\u001b[34mAttempting uninstall: torch\u001b[0m\n",
      "\u001b[34mFound existing installation: torch 1.10.0+cpu\u001b[0m\n",
      "\u001b[34mUninstalling torch-1.10.0+cpu:\u001b[0m\n",
      "\u001b[34mSuccessfully uninstalled torch-1.10.0+cpu\u001b[0m\n",
      "\u001b[34mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\u001b[0m\n",
      "\u001b[34mtorchvision 0.11.1+cpu requires torch==1.10.0, but you have torch 1.10.1 which is incompatible.\u001b[0m\n",
      "\u001b[34mSuccessfully installed filelock-3.4.2 huggingface-hub-0.4.0 regex-2022.1.18 sacremoses-0.0.47 tokenizers-0.10.3 torch-1.10.1 transformers-4.15.0\u001b[0m\n",
      "\u001b[34mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\n",
      "\u001b[34m2022-01-26 20:05:09,510 sagemaker-training-toolkit INFO     No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34m2022-01-26 20:05:09,520 sagemaker-training-toolkit INFO     No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34m2022-01-26 20:05:09,529 sagemaker-training-toolkit INFO     No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34m2022-01-26 20:05:09,537 sagemaker-training-toolkit INFO     Invoking user script\u001b[0m\n",
      "\u001b[34mTraining Env:\u001b[0m\n",
      "\u001b[34m{\n",
      "    \"additional_framework_parameters\": {},\n",
      "    \"channel_input_dirs\": {\n",
      "        \"val\": \"/opt/ml/input/data/val\",\n",
      "        \"test\": \"/opt/ml/input/data/test\",\n",
      "        \"train\": \"/opt/ml/input/data/train\"\n",
      "    },\n",
      "    \"current_host\": \"algo-1\",\n",
      "    \"framework_module\": \"sagemaker_pytorch_container.training:main\",\n",
      "    \"hosts\": [\n",
      "        \"algo-1\"\n",
      "    ],\n",
      "    \"hyperparameters\": {\n",
      "        \"batch-size\": 2,\n",
      "        \"lr\": 1e-05,\n",
      "        \"epochs\": 1\n",
      "    },\n",
      "    \"input_config_dir\": \"/opt/ml/input/config\",\n",
      "    \"input_data_config\": {\n",
      "        \"val\": {\n",
      "            \"TrainingInputMode\": \"File\",\n",
      "            \"S3DistributionType\": \"FullyReplicated\",\n",
      "            \"RecordWrapperType\": \"None\"\n",
      "        },\n",
      "        \"test\": {\n",
      "            \"TrainingInputMode\": \"File\",\n",
      "            \"S3DistributionType\": \"FullyReplicated\",\n",
      "            \"RecordWrapperType\": \"None\"\n",
      "        },\n",
      "        \"train\": {\n",
      "            \"TrainingInputMode\": \"File\",\n",
      "            \"S3DistributionType\": \"FullyReplicated\",\n",
      "            \"RecordWrapperType\": \"None\"\n",
      "        }\n",
      "    },\n",
      "    \"input_dir\": \"/opt/ml/input\",\n",
      "    \"is_master\": true,\n",
      "    \"job_name\": \"pytorch-training-2022-01-26-20-00-11-378\",\n",
      "    \"log_level\": 20,\n",
      "    \"master_hostname\": \"algo-1\",\n",
      "    \"model_dir\": \"/opt/ml/model\",\n",
      "    \"module_dir\": \"s3://sagemaker-us-east-1-439373211214/pytorch-training-2022-01-26-20-00-11-378/source/sourcedir.tar.gz\",\n",
      "    \"module_name\": \"train_deploy\",\n",
      "    \"network_interface_name\": \"eth0\",\n",
      "    \"num_cpus\": 16,\n",
      "    \"num_gpus\": 0,\n",
      "    \"output_data_dir\": \"/opt/ml/output/data\",\n",
      "    \"output_dir\": \"/opt/ml/output\",\n",
      "    \"output_intermediate_dir\": \"/opt/ml/output/intermediate\",\n",
      "    \"resource_config\": {\n",
      "        \"current_host\": \"algo-1\",\n",
      "        \"hosts\": [\n",
      "            \"algo-1\"\n",
      "        ],\n",
      "        \"network_interface_name\": \"eth0\"\n",
      "    },\n",
      "    \"user_entry_point\": \"train_deploy.py\"\u001b[0m\n",
      "\u001b[34m}\u001b[0m\n",
      "\u001b[34mEnvironment variables:\u001b[0m\n",
      "\u001b[34mSM_HOSTS=[\"algo-1\"]\u001b[0m\n",
      "\u001b[34mSM_NETWORK_INTERFACE_NAME=eth0\u001b[0m\n",
      "\u001b[34mSM_HPS={\"batch-size\":2,\"epochs\":1,\"lr\":1e-05}\u001b[0m\n",
      "\u001b[34mSM_USER_ENTRY_POINT=train_deploy.py\u001b[0m\n",
      "\u001b[34mSM_FRAMEWORK_PARAMS={}\u001b[0m\n",
      "\u001b[34mSM_RESOURCE_CONFIG={\"current_host\":\"algo-1\",\"hosts\":[\"algo-1\"],\"network_interface_name\":\"eth0\"}\u001b[0m\n",
      "\u001b[34mSM_INPUT_DATA_CONFIG={\"test\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"},\"train\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"},\"val\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"}}\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_DATA_DIR=/opt/ml/output/data\u001b[0m\n",
      "\u001b[34mSM_CHANNELS=[\"test\",\"train\",\"val\"]\u001b[0m\n",
      "\u001b[34mSM_CURRENT_HOST=algo-1\u001b[0m\n",
      "\u001b[34mSM_MODULE_NAME=train_deploy\u001b[0m\n",
      "\u001b[34mSM_LOG_LEVEL=20\u001b[0m\n",
      "\u001b[34mSM_FRAMEWORK_MODULE=sagemaker_pytorch_container.training:main\u001b[0m\n",
      "\u001b[34mSM_INPUT_DIR=/opt/ml/input\u001b[0m\n",
      "\u001b[34mSM_INPUT_CONFIG_DIR=/opt/ml/input/config\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_DIR=/opt/ml/output\u001b[0m\n",
      "\u001b[34mSM_NUM_CPUS=16\u001b[0m\n",
      "\u001b[34mSM_NUM_GPUS=0\u001b[0m\n",
      "\u001b[34mSM_MODEL_DIR=/opt/ml/model\u001b[0m\n",
      "\u001b[34mSM_MODULE_DIR=s3://sagemaker-us-east-1-439373211214/pytorch-training-2022-01-26-20-00-11-378/source/sourcedir.tar.gz\u001b[0m\n",
      "\u001b[34mSM_TRAINING_ENV={\"additional_framework_parameters\":{},\"channel_input_dirs\":{\"test\":\"/opt/ml/input/data/test\",\"train\":\"/opt/ml/input/data/train\",\"val\":\"/opt/ml/input/data/val\"},\"current_host\":\"algo-1\",\"framework_module\":\"sagemaker_pytorch_container.training:main\",\"hosts\":[\"algo-1\"],\"hyperparameters\":{\"batch-size\":2,\"epochs\":1,\"lr\":1e-05},\"input_config_dir\":\"/opt/ml/input/config\",\"input_data_config\":{\"test\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"},\"train\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"},\"val\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"}},\"input_dir\":\"/opt/ml/input\",\"is_master\":true,\"job_name\":\"pytorch-training-2022-01-26-20-00-11-378\",\"log_level\":20,\"master_hostname\":\"algo-1\",\"model_dir\":\"/opt/ml/model\",\"module_dir\":\"s3://sagemaker-us-east-1-439373211214/pytorch-training-2022-01-26-20-00-11-378/source/sourcedir.tar.gz\",\"module_name\":\"train_deploy\",\"network_interface_name\":\"eth0\",\"num_cpus\":16,\"num_gpus\":0,\"output_data_dir\":\"/opt/ml/output/data\",\"output_dir\":\"/opt/ml/output\",\"output_intermediate_dir\":\"/opt/ml/output/intermediate\",\"resource_config\":{\"current_host\":\"algo-1\",\"hosts\":[\"algo-1\"],\"network_interface_name\":\"eth0\"},\"user_entry_point\":\"train_deploy.py\"}\u001b[0m\n",
      "\u001b[34mSM_USER_ARGS=[\"--batch-size\",\"2\",\"--epochs\",\"1\",\"--lr\",\"1e-05\"]\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate\u001b[0m\n",
      "\u001b[34mSM_CHANNEL_VAL=/opt/ml/input/data/val\u001b[0m\n",
      "\u001b[34mSM_CHANNEL_TEST=/opt/ml/input/data/test\u001b[0m\n",
      "\u001b[34mSM_CHANNEL_TRAIN=/opt/ml/input/data/train\u001b[0m\n",
      "\u001b[34mSM_HP_BATCH-SIZE=2\u001b[0m\n",
      "\u001b[34mSM_HP_LR=1e-05\u001b[0m\n",
      "\u001b[34mSM_HP_EPOCHS=1\u001b[0m\n",
      "\u001b[34mPYTHONPATH=/opt/ml/code:/opt/conda/bin:/opt/conda/lib/python38.zip:/opt/conda/lib/python3.8:/opt/conda/lib/python3.8/lib-dynload:/opt/conda/lib/python3.8/site-packages\u001b[0m\n",
      "\u001b[34mInvoking script with the following command:\u001b[0m\n",
      "\u001b[34m/opt/conda/bin/python3.8 train_deploy.py --batch-size 2 --epochs 1 --lr 1e-05\u001b[0m\n",
      "\u001b[34mLoading GPT2Tokenizer.\u001b[0m\n",
      "\u001b[34mDownloading:   0%|          | 0.00/0.99M [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34mDownloading: 100%|██████████| 0.99M/0.99M [00:00<00:00, 65.2MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:   0%|          | 0.00/446k [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34mDownloading: 100%|██████████| 446k/446k [00:00<00:00, 57.4MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:   0%|          | 0.00/1.29M [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34mDownloading: 100%|██████████| 1.29M/1.29M [00:00<00:00, 73.7MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:   0%|          | 0.00/665 [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34mDownloading: 100%|██████████| 665/665 [00:00<00:00, 1.02MB/s]\u001b[0m\n",
      "\u001b[34mNumber of gpus available - 0\u001b[0m\n",
      "\u001b[34mGet train data loader\u001b[0m\n",
      "\u001b[34mGet val data loader\u001b[0m\n",
      "\u001b[34mGet test data loader\u001b[0m\n",
      "\u001b[34mProcesses 1780/1780 (100%) of train data\u001b[0m\n",
      "\u001b[34mProcesses 222/222 (100%) of val data\u001b[0m\n",
      "\u001b[34mProcesses 223/223 (100%) of test data\u001b[0m\n",
      "\u001b[34mDownloading:   0%|          | 0.00/523M [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34mDownloading:   2%|▏         | 9.55M/523M [00:00<00:05, 100MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:   4%|▎         | 19.4M/523M [00:00<00:05, 102MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:   6%|▌         | 29.3M/523M [00:00<00:05, 103MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:   8%|▊         | 39.4M/523M [00:00<00:04, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:   9%|▉         | 49.4M/523M [00:00<00:04, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  11%|█▏        | 59.4M/523M [00:00<00:04, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  13%|█▎        | 69.3M/523M [00:00<00:04, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  15%|█▌        | 79.3M/523M [00:00<00:04, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  17%|█▋        | 89.3M/523M [00:00<00:04, 105MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  19%|█▉        | 99.3M/523M [00:01<00:04, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  21%|██        | 109M/523M [00:01<00:04, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  23%|██▎       | 119M/523M [00:01<00:04, 105MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  25%|██▍       | 129M/523M [00:01<00:03, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  27%|██▋       | 139M/523M [00:01<00:03, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  29%|██▊       | 149M/523M [00:01<00:03, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  30%|███       | 159M/523M [00:01<00:03, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  32%|███▏      | 169M/523M [00:01<00:03, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  34%|███▍      | 179M/523M [00:01<00:03, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  36%|███▌      | 189M/523M [00:01<00:03, 105MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  38%|███▊      | 199M/523M [00:02<00:03, 105MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  40%|████      | 209M/523M [00:02<00:03, 105MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  42%|████▏     | 219M/523M [00:02<00:03, 105MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  44%|████▍     | 229M/523M [00:02<00:02, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  46%|████▌     | 239M/523M [00:02<00:02, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  48%|████▊     | 249M/523M [00:02<00:02, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  50%|████▉     | 259M/523M [00:02<00:02, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  51%|█████▏    | 269M/523M [00:02<00:02, 105MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  53%|█████▎    | 279M/523M [00:02<00:02, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  55%|█████▌    | 289M/523M [00:02<00:02, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  57%|█████▋    | 299M/523M [00:03<00:02, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  59%|█████▉    | 309M/523M [00:03<00:02, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  61%|██████    | 319M/523M [00:03<00:02, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  63%|██████▎   | 329M/523M [00:03<00:01, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  65%|██████▍   | 339M/523M [00:03<00:01, 105MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  67%|██████▋   | 349M/523M [00:03<00:01, 106MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  69%|██████▊   | 359M/523M [00:03<00:01, 106MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  71%|███████   | 369M/523M [00:03<00:01, 99.5MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  73%|███████▎  | 380M/523M [00:03<00:01, 102MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  75%|███████▍  | 390M/523M [00:03<00:01, 104MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  77%|███████▋  | 400M/523M [00:04<00:01, 105MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  79%|███████▊  | 411M/523M [00:04<00:01, 106MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  81%|████████  | 421M/523M [00:04<00:01, 106MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  82%|████████▏ | 431M/523M [00:04<00:00, 107MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  84%|████████▍ | 441M/523M [00:04<00:00, 107MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  86%|████████▋ | 452M/523M [00:04<00:00, 107MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  88%|████████▊ | 462M/523M [00:04<00:00, 107MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  90%|█████████ | 472M/523M [00:04<00:00, 107MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  92%|█████████▏| 482M/523M [00:04<00:00, 107MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  94%|█████████▍| 493M/523M [00:04<00:00, 107MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  96%|█████████▌| 503M/523M [00:05<00:00, 107MB/s]\u001b[0m\n",
      "\u001b[34mDownloading:  98%|█████████▊| 513M/523M [00:05<00:00, 107MB/s]\u001b[0m\n",
      "\u001b[34mDownloading: 100%|██████████| 523M/523M [00:05<00:00, 105MB/s]\u001b[0m\n",
      "\u001b[34m0%|          | 0/890 [00:00<?, ?it/s]\u001b[0m\n",
      "\u001b[34m0%|          | 1/890 [00:00<08:55,  1.66it/s]\u001b[0m\n",
      "\u001b[34m0%|          | 2/890 [00:01<07:54,  1.87it/s]\u001b[0m\n",
      "\u001b[34m0%|          | 3/890 [00:01<07:26,  1.99it/s]\u001b[0m\n",
      "\u001b[34m0%|          | 4/890 [00:02<07:06,  2.08it/s]\u001b[0m\n",
      "\u001b[34m1%|          | 5/890 [00:02<07:04,  2.09it/s]\u001b[0m\n",
      "\u001b[34m1%|          | 6/890 [00:02<07:02,  2.09it/s]\u001b[0m\n",
      "\u001b[34m1%|          | 7/890 [00:03<07:04,  2.08it/s]\u001b[0m\n",
      "\u001b[34m1%|          | 8/890 [00:03<07:14,  2.03it/s]\u001b[0m\n",
      "\u001b[34m1%|          | 9/890 [00:04<07:04,  2.07it/s]\u001b[0m\n",
      "\u001b[34m1%|          | 10/890 [00:04<07:05,  2.07it/s]\u001b[0m\n",
      "\u001b[34m1%|          | 11/890 [00:05<07:08,  2.05it/s]\u001b[0m\n",
      "\u001b[34m1%|▏         | 12/890 [00:05<07:01,  2.08it/s]\u001b[0m\n",
      "\u001b[34m1%|▏         | 13/890 [00:06<07:02,  2.08it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 14/890 [00:06<07:06,  2.06it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 15/890 [00:07<07:17,  2.00it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 16/890 [00:07<07:16,  2.00it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 17/890 [00:08<07:09,  2.03it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 18/890 [00:08<07:09,  2.03it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 19/890 [00:09<07:16,  2.00it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 20/890 [00:09<07:08,  2.03it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 21/890 [00:10<06:57,  2.08it/s]\u001b[0m\n",
      "\u001b[34m2%|▏         | 22/890 [00:10<06:55,  2.09it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 23/890 [00:11<06:52,  2.10it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 24/890 [00:11<06:52,  2.10it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 25/890 [00:12<07:06,  2.03it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 26/890 [00:12<07:23,  1.95it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 27/890 [00:13<07:12,  2.00it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 28/890 [00:13<06:55,  2.08it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 29/890 [00:14<06:58,  2.06it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 30/890 [00:14<06:48,  2.11it/s]\u001b[0m\n",
      "\u001b[34m3%|▎         | 31/890 [00:15<06:50,  2.09it/s]\u001b[0m\n",
      "\u001b[34m4%|▎         | 32/890 [00:15<06:49,  2.10it/s]\u001b[0m\n",
      "\u001b[34m4%|▎         | 33/890 [00:16<06:49,  2.09it/s]\u001b[0m\n",
      "\u001b[34m4%|▍         | 34/890 [00:16<06:54,  2.07it/s]\u001b[0m\n",
      "\u001b[34m4%|▍         | 35/890 [00:17<06:48,  2.09it/s]\u001b[0m\n",
      "\u001b[34m4%|▍         | 36/890 [00:17<06:44,  2.11it/s]\u001b[0m\n",
      "\u001b[34m4%|▍         | 37/890 [00:18<06:52,  2.07it/s]\u001b[0m\n",
      "\u001b[34m4%|▍         | 38/890 [00:18<06:47,  2.09it/s]\u001b[0m\n",
      "\u001b[34m4%|▍         | 39/890 [00:18<06:53,  2.06it/s]\u001b[0m\n",
      "\u001b[34m4%|▍         | 40/890 [00:19<06:48,  2.08it/s]\u001b[0m\n",
      "\u001b[34m5%|▍         | 41/890 [00:19<06:38,  2.13it/s]\u001b[0m\n",
      "\u001b[34m5%|▍         | 42/890 [00:20<06:41,  2.11it/s]\u001b[0m\n",
      "\u001b[34m5%|▍         | 43/890 [00:20<06:39,  2.12it/s]\u001b[0m\n",
      "\u001b[34m5%|▍         | 44/890 [00:21<06:34,  2.14it/s]\u001b[0m\n",
      "\u001b[34m5%|▌         | 45/890 [00:21<06:28,  2.17it/s]\u001b[0m\n",
      "\u001b[34m5%|▌         | 46/890 [00:22<06:28,  2.17it/s]\u001b[0m\n",
      "\u001b[34m5%|▌         | 47/890 [00:22<06:37,  2.12it/s]\u001b[0m\n",
      "\u001b[34m5%|▌         | 48/890 [00:23<06:30,  2.15it/s]\u001b[0m\n",
      "\u001b[34m6%|▌         | 49/890 [00:23<06:29,  2.16it/s]\u001b[0m\n",
      "\u001b[34m6%|▌         | 50/890 [00:24<07:11,  1.95it/s]\u001b[0m\n",
      "\u001b[34m6%|▌         | 51/890 [00:24<07:00,  2.00it/s]\u001b[0m\n",
      "\u001b[34m6%|▌         | 52/890 [00:25<06:52,  2.03it/s]\u001b[0m\n",
      "\u001b[34m6%|▌         | 53/890 [00:25<06:49,  2.04it/s]\u001b[0m\n",
      "\u001b[34m6%|▌         | 54/890 [00:26<06:44,  2.07it/s]\u001b[0m\n",
      "\u001b[34m6%|▌         | 55/890 [00:26<06:36,  2.10it/s]\u001b[0m\n",
      "\u001b[34m6%|▋         | 56/890 [00:27<06:37,  2.10it/s]\u001b[0m\n",
      "\u001b[34m6%|▋         | 57/890 [00:27<06:36,  2.10it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 58/890 [00:28<06:49,  2.03it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 59/890 [00:28<06:49,  2.03it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 60/890 [00:29<06:49,  2.03it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 61/890 [00:29<06:42,  2.06it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 62/890 [00:29<06:31,  2.12it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 63/890 [00:30<06:27,  2.13it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 64/890 [00:30<06:26,  2.14it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 65/890 [00:31<06:17,  2.19it/s]\u001b[0m\n",
      "\u001b[34m7%|▋         | 66/890 [00:31<06:16,  2.19it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 67/890 [00:32<06:13,  2.20it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 68/890 [00:32<06:12,  2.21it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 69/890 [00:33<06:27,  2.12it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 70/890 [00:33<06:19,  2.16it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 71/890 [00:34<06:18,  2.16it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 72/890 [00:34<06:26,  2.12it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 73/890 [00:35<06:21,  2.14it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 74/890 [00:35<06:38,  2.05it/s]\u001b[0m\n",
      "\u001b[34m8%|▊         | 75/890 [00:36<06:28,  2.10it/s]\u001b[0m\n",
      "\u001b[34m9%|▊         | 76/890 [00:36<06:32,  2.07it/s]\u001b[0m\n",
      "\u001b[34m9%|▊         | 77/890 [00:37<06:40,  2.03it/s]\u001b[0m\n",
      "\u001b[34m9%|▉         | 78/890 [00:37<07:12,  1.88it/s]\u001b[0m\n",
      "\u001b[34m9%|▉         | 79/890 [00:38<06:59,  1.93it/s]\u001b[0m\n",
      "\u001b[34m9%|▉         | 80/890 [00:38<06:51,  1.97it/s]\u001b[0m\n",
      "\u001b[34m9%|▉         | 81/890 [00:39<06:52,  1.96it/s]\u001b[0m\n",
      "\u001b[34m9%|▉         | 82/890 [00:39<07:25,  1.81it/s]\u001b[0m\n",
      "\u001b[34m9%|▉         | 83/890 [00:40<07:12,  1.87it/s]\u001b[0m\n",
      "\u001b[34m9%|▉         | 84/890 [00:40<07:39,  1.76it/s]\u001b[0m\n",
      "\u001b[34m10%|▉         | 85/890 [00:41<07:58,  1.68it/s]\u001b[0m\n",
      "\u001b[34m10%|▉         | 86/890 [00:42<08:02,  1.67it/s]\u001b[0m\n",
      "\u001b[34m10%|▉         | 87/890 [00:42<07:27,  1.80it/s]\u001b[0m\n",
      "\u001b[34m10%|▉         | 88/890 [00:43<07:04,  1.89it/s]\u001b[0m\n",
      "\u001b[34m10%|█         | 89/890 [00:43<06:41,  1.99it/s]\u001b[0m\n",
      "\u001b[34m10%|█         | 90/890 [00:44<06:26,  2.07it/s]\u001b[0m\n",
      "\u001b[34m10%|█         | 91/890 [00:44<06:21,  2.09it/s]\u001b[0m\n",
      "\u001b[34m10%|█         | 92/890 [00:44<06:14,  2.13it/s]\u001b[0m\n",
      "\u001b[34m10%|█         | 93/890 [00:45<06:06,  2.17it/s]\u001b[0m\n",
      "\u001b[34m11%|█         | 94/890 [00:45<06:10,  2.15it/s]\u001b[0m\n",
      "\u001b[34m11%|█         | 95/890 [00:46<06:06,  2.17it/s]\u001b[0m\n",
      "\u001b[34m11%|█         | 96/890 [00:46<06:02,  2.19it/s]\u001b[0m\n",
      "\u001b[34m11%|█         | 97/890 [00:47<06:21,  2.08it/s]\u001b[0m\n",
      "\u001b[34m11%|█         | 98/890 [00:47<06:16,  2.10it/s]\u001b[0m\n",
      "\u001b[34m11%|█         | 99/890 [00:48<06:15,  2.11it/s]\u001b[0m\n",
      "\u001b[34m11%|█         | 100/890 [00:48<06:09,  2.14it/s]\u001b[0m\n",
      "\u001b[34m11%|█▏        | 101/890 [00:49<06:07,  2.14it/s]\u001b[0m\n",
      "\u001b[34m11%|█▏        | 102/890 [00:49<06:11,  2.12it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 103/890 [00:50<06:06,  2.15it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 104/890 [00:50<06:02,  2.17it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 105/890 [00:51<06:02,  2.16it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 106/890 [00:51<06:16,  2.08it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 107/890 [00:51<06:12,  2.10it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 108/890 [00:52<06:07,  2.13it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 109/890 [00:52<06:03,  2.15it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 110/890 [00:53<06:10,  2.10it/s]\u001b[0m\n",
      "\u001b[34m12%|█▏        | 111/890 [00:53<06:07,  2.12it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 112/890 [00:54<06:29,  2.00it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 113/890 [00:54<06:24,  2.02it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 114/890 [00:55<06:13,  2.08it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 115/890 [00:55<06:08,  2.10it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 116/890 [00:56<06:09,  2.09it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 117/890 [00:56<06:05,  2.11it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 118/890 [00:57<06:03,  2.12it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 119/890 [00:57<06:03,  2.12it/s]\u001b[0m\n",
      "\u001b[34m13%|█▎        | 120/890 [00:58<06:04,  2.11it/s]\u001b[0m\n",
      "\u001b[34m14%|█▎        | 121/890 [00:58<06:15,  2.05it/s]\u001b[0m\n",
      "\u001b[34m14%|█▎        | 122/890 [00:59<06:10,  2.07it/s]\u001b[0m\n",
      "\u001b[34m14%|█▍        | 123/890 [00:59<06:05,  2.10it/s]\u001b[0m\n",
      "\u001b[34m14%|█▍        | 124/890 [01:00<05:57,  2.14it/s]\u001b[0m\n",
      "\u001b[34m14%|█▍        | 125/890 [01:00<05:55,  2.15it/s]\u001b[0m\n",
      "\u001b[34m14%|█▍        | 126/890 [01:01<05:51,  2.17it/s]\u001b[0m\n",
      "\u001b[34m14%|█▍        | 127/890 [01:01<06:10,  2.06it/s]\u001b[0m\n",
      "\u001b[34m14%|█▍        | 128/890 [01:01<06:00,  2.11it/s]\u001b[0m\n",
      "\u001b[34m14%|█▍        | 129/890 [01:02<05:56,  2.14it/s]\u001b[0m\n",
      "\u001b[34m15%|█▍        | 130/890 [01:02<06:02,  2.10it/s]\u001b[0m\n",
      "\u001b[34m15%|█▍        | 131/890 [01:03<06:06,  2.07it/s]\u001b[0m\n",
      "\u001b[34m15%|█▍        | 132/890 [01:03<06:01,  2.10it/s]\u001b[0m\n",
      "\u001b[34m15%|█▍        | 133/890 [01:04<06:00,  2.10it/s]\u001b[0m\n",
      "\u001b[34m15%|█▌        | 134/890 [01:04<05:55,  2.13it/s]\u001b[0m\n",
      "\u001b[34m15%|█▌        | 135/890 [01:05<05:52,  2.14it/s]\u001b[0m\n",
      "\u001b[34m15%|█▌        | 136/890 [01:05<05:43,  2.19it/s]\u001b[0m\n",
      "\u001b[34m15%|█▌        | 137/890 [01:06<05:43,  2.19it/s]\u001b[0m\n",
      "\u001b[34m16%|█▌        | 138/890 [01:06<05:44,  2.18it/s]\u001b[0m\n",
      "\u001b[34m16%|█▌        | 139/890 [01:07<05:41,  2.20it/s]\u001b[0m\n",
      "\u001b[34m16%|█▌        | 140/890 [01:07<05:39,  2.21it/s]\u001b[0m\n",
      "\u001b[34m16%|█▌        | 141/890 [01:07<05:34,  2.24it/s]\u001b[0m\n",
      "\u001b[34m16%|█▌        | 142/890 [01:08<05:37,  2.21it/s]\u001b[0m\n",
      "\u001b[34m16%|█▌        | 143/890 [01:08<05:48,  2.15it/s]\u001b[0m\n",
      "\u001b[34m16%|█▌        | 144/890 [01:09<05:49,  2.14it/s]\u001b[0m\n",
      "\u001b[34m16%|█▋        | 145/890 [01:09<05:47,  2.14it/s]\u001b[0m\n",
      "\u001b[34m16%|█▋        | 146/890 [01:10<05:41,  2.18it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 147/890 [01:10<05:41,  2.18it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 148/890 [01:11<05:33,  2.22it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 149/890 [01:11<05:33,  2.22it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 150/890 [01:12<05:46,  2.13it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 151/890 [01:12<05:45,  2.14it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 152/890 [01:13<05:41,  2.16it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 153/890 [01:13<05:40,  2.16it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 154/890 [01:14<05:53,  2.08it/s]\u001b[0m\n",
      "\u001b[34m17%|█▋        | 155/890 [01:14<05:51,  2.09it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 156/890 [01:14<05:46,  2.12it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 157/890 [01:15<05:42,  2.14it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 158/890 [01:15<05:41,  2.14it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 159/890 [01:16<05:38,  2.16it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 160/890 [01:16<05:47,  2.10it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 161/890 [01:17<05:51,  2.07it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 162/890 [01:17<05:46,  2.10it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 163/890 [01:18<05:42,  2.12it/s]\u001b[0m\n",
      "\u001b[34m18%|█▊        | 164/890 [01:18<05:50,  2.07it/s]\u001b[0m\n",
      "\u001b[34m19%|█▊        | 165/890 [01:19<05:44,  2.10it/s]\u001b[0m\n",
      "\u001b[34m19%|█▊        | 166/890 [01:19<05:37,  2.15it/s]\u001b[0m\n",
      "\u001b[34m19%|█▉        | 167/890 [01:20<05:38,  2.14it/s]\u001b[0m\n",
      "\u001b[34m19%|█▉        | 168/890 [01:20<05:35,  2.15it/s]\u001b[0m\n",
      "\u001b[34m19%|█▉        | 169/890 [01:21<05:30,  2.18it/s]\u001b[0m\n",
      "\u001b[34m19%|█▉        | 170/890 [01:21<05:35,  2.15it/s]\u001b[0m\n",
      "\u001b[34m19%|█▉        | 171/890 [01:22<05:36,  2.13it/s]\u001b[0m\n",
      "\u001b[34m19%|█▉        | 172/890 [01:22<05:31,  2.17it/s]\u001b[0m\n",
      "\u001b[34m19%|█▉        | 173/890 [01:22<05:34,  2.14it/s]\u001b[0m\n",
      "\u001b[34m20%|█▉        | 174/890 [01:23<05:30,  2.17it/s]\u001b[0m\n",
      "\u001b[34m20%|█▉        | 175/890 [01:23<05:39,  2.10it/s]\u001b[0m\n",
      "\u001b[34m20%|█▉        | 176/890 [01:24<05:35,  2.13it/s]\u001b[0m\n",
      "\u001b[34m20%|█▉        | 177/890 [01:24<05:45,  2.06it/s]\u001b[0m\n",
      "\u001b[34m20%|██        | 178/890 [01:25<05:56,  1.99it/s]\u001b[0m\n",
      "\u001b[34m20%|██        | 179/890 [01:25<05:49,  2.03it/s]\u001b[0m\n",
      "\u001b[34m20%|██        | 180/890 [01:26<05:44,  2.06it/s]\u001b[0m\n",
      "\u001b[34m20%|██        | 181/890 [01:26<05:39,  2.09it/s]\u001b[0m\n",
      "\u001b[34m20%|██        | 182/890 [01:27<05:42,  2.07it/s]\u001b[0m\n",
      "\u001b[34m21%|██        | 183/890 [01:27<05:42,  2.06it/s]\u001b[0m\n",
      "\u001b[34m21%|██        | 184/890 [01:28<05:34,  2.11it/s]\u001b[0m\n",
      "\u001b[34m21%|██        | 185/890 [01:28<05:23,  2.18it/s]\u001b[0m\n",
      "\u001b[34m21%|██        | 186/890 [01:29<05:25,  2.16it/s]\u001b[0m\n",
      "\u001b[34m21%|██        | 187/890 [01:29<05:21,  2.19it/s]\u001b[0m\n",
      "\u001b[34m21%|██        | 188/890 [01:30<05:29,  2.13it/s]\u001b[0m\n",
      "\u001b[34m21%|██        | 189/890 [01:30<05:26,  2.15it/s]\u001b[0m\n",
      "\u001b[34m21%|██▏       | 190/890 [01:31<05:24,  2.15it/s]\u001b[0m\n",
      "\u001b[34m21%|██▏       | 191/890 [01:31<05:28,  2.13it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 192/890 [01:31<05:23,  2.16it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 193/890 [01:32<05:23,  2.15it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 194/890 [01:32<05:27,  2.12it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 195/890 [01:33<05:36,  2.06it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 196/890 [01:33<05:27,  2.12it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 197/890 [01:34<05:25,  2.13it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 198/890 [01:34<05:19,  2.17it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 199/890 [01:35<05:21,  2.15it/s]\u001b[0m\n",
      "\u001b[34m22%|██▏       | 200/890 [01:35<05:16,  2.18it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 201/890 [01:36<05:17,  2.17it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 202/890 [01:36<05:14,  2.18it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 203/890 [01:37<05:28,  2.09it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 204/890 [01:37<05:40,  2.01it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 205/890 [01:38<05:37,  2.03it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 206/890 [01:38<05:30,  2.07it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 207/890 [01:39<05:31,  2.06it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 208/890 [01:39<06:05,  1.87it/s]\u001b[0m\n",
      "\u001b[34m23%|██▎       | 209/890 [01:40<06:10,  1.84it/s]\u001b[0m\n",
      "\u001b[34m24%|██▎       | 210/890 [01:40<06:12,  1.82it/s]\u001b[0m\n",
      "\u001b[34m24%|██▎       | 211/890 [01:41<06:35,  1.72it/s]\u001b[0m\n",
      "\u001b[34m24%|██▍       | 212/890 [01:42<06:48,  1.66it/s]\u001b[0m\n",
      "\u001b[34m24%|██▍       | 213/890 [01:42<06:17,  1.79it/s]\u001b[0m\n",
      "\u001b[34m24%|██▍       | 214/890 [01:43<05:54,  1.90it/s]\u001b[0m\n",
      "\u001b[34m24%|██▍       | 215/890 [01:43<05:48,  1.94it/s]\u001b[0m\n",
      "\u001b[34m24%|██▍       | 216/890 [01:44<05:40,  1.98it/s]\u001b[0m\n",
      "\u001b[34m24%|██▍       | 217/890 [01:44<05:54,  1.90it/s]\u001b[0m\n",
      "\u001b[34m24%|██▍       | 218/890 [01:45<05:38,  1.99it/s]\u001b[0m\n",
      "\u001b[34m25%|██▍       | 219/890 [01:45<05:31,  2.02it/s]\u001b[0m\n",
      "\u001b[34m25%|██▍       | 220/890 [01:46<05:23,  2.07it/s]\u001b[0m\n",
      "\u001b[34m25%|██▍       | 221/890 [01:46<05:17,  2.11it/s]\u001b[0m\n",
      "\u001b[34m25%|██▍       | 222/890 [01:46<05:10,  2.15it/s]\u001b[0m\n",
      "\u001b[34m25%|██▌       | 223/890 [01:47<05:09,  2.15it/s]\u001b[0m\n",
      "\u001b[34m25%|██▌       | 224/890 [01:47<05:23,  2.06it/s]\u001b[0m\n",
      "\u001b[34m25%|██▌       | 225/890 [01:48<05:22,  2.06it/s]\u001b[0m\n",
      "\u001b[34m25%|██▌       | 226/890 [01:48<05:15,  2.11it/s]\u001b[0m\n",
      "\u001b[34m26%|██▌       | 227/890 [01:49<05:13,  2.11it/s]\u001b[0m\n",
      "\u001b[34m26%|██▌       | 228/890 [01:49<05:19,  2.07it/s]\u001b[0m\n",
      "\u001b[34m26%|██▌       | 229/890 [01:50<05:11,  2.12it/s]\u001b[0m\n",
      "\u001b[34m26%|██▌       | 230/890 [01:50<05:11,  2.12it/s]\u001b[0m\n",
      "\u001b[34m26%|██▌       | 231/890 [01:51<05:08,  2.14it/s]\u001b[0m\n",
      "\u001b[34m26%|██▌       | 232/890 [01:51<05:05,  2.15it/s]\u001b[0m\n",
      "\u001b[34m26%|██▌       | 233/890 [01:52<05:01,  2.18it/s]\u001b[0m\n",
      "\u001b[34m26%|██▋       | 234/890 [01:52<05:03,  2.16it/s]\u001b[0m\n",
      "\u001b[34m26%|██▋       | 235/890 [01:53<05:03,  2.16it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 236/890 [01:53<05:00,  2.17it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 237/890 [01:53<04:57,  2.19it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 238/890 [01:54<05:08,  2.11it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 239/890 [01:54<05:04,  2.14it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 240/890 [01:55<05:33,  1.95it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 241/890 [01:55<05:22,  2.01it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 242/890 [01:56<05:20,  2.02it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 243/890 [01:56<05:15,  2.05it/s]\u001b[0m\n",
      "\u001b[34m27%|██▋       | 244/890 [01:57<05:08,  2.09it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 245/890 [01:57<05:18,  2.02it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 246/890 [01:58<05:15,  2.04it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 247/890 [01:58<05:10,  2.07it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 248/890 [01:59<05:09,  2.07it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 249/890 [01:59<05:09,  2.07it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 250/890 [02:00<05:08,  2.07it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 251/890 [02:00<05:05,  2.09it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 252/890 [02:01<05:02,  2.11it/s]\u001b[0m\n",
      "\u001b[34m28%|██▊       | 253/890 [02:01<05:06,  2.08it/s]\u001b[0m\n",
      "\u001b[34m29%|██▊       | 254/890 [02:02<05:02,  2.10it/s]\u001b[0m\n",
      "\u001b[34m29%|██▊       | 255/890 [02:02<05:06,  2.07it/s]\u001b[0m\n",
      "\u001b[34m29%|██▉       | 256/890 [02:03<05:03,  2.09it/s]\u001b[0m\n",
      "\u001b[34m29%|██▉       | 257/890 [02:03<05:04,  2.08it/s]\u001b[0m\n",
      "\u001b[34m29%|██▉       | 258/890 [02:04<04:58,  2.12it/s]\u001b[0m\n",
      "\u001b[34m29%|██▉       | 259/890 [02:04<04:59,  2.11it/s]\u001b[0m\n",
      "\u001b[34m29%|██▉       | 260/890 [02:05<05:05,  2.06it/s]\u001b[0m\n",
      "\u001b[34m29%|██▉       | 261/890 [02:05<05:03,  2.08it/s]\u001b[0m\n",
      "\u001b[34m29%|██▉       | 262/890 [02:06<05:10,  2.03it/s]\u001b[0m\n",
      "\u001b[34m30%|██▉       | 263/890 [02:06<05:04,  2.06it/s]\u001b[0m\n",
      "\u001b[34m30%|██▉       | 264/890 [02:07<05:03,  2.06it/s]\u001b[0m\n",
      "\u001b[34m30%|██▉       | 265/890 [02:07<05:02,  2.06it/s]\u001b[0m\n",
      "\u001b[34m30%|██▉       | 266/890 [02:08<05:06,  2.04it/s]\u001b[0m\n",
      "\u001b[34m30%|███       | 267/890 [02:08<05:07,  2.02it/s]\u001b[0m\n",
      "\u001b[34m30%|███       | 268/890 [02:09<05:13,  1.99it/s]\u001b[0m\n",
      "\u001b[34m30%|███       | 269/890 [02:09<05:03,  2.05it/s]\u001b[0m\n",
      "\u001b[34m30%|███       | 270/890 [02:10<05:06,  2.02it/s]\u001b[0m\n",
      "\u001b[34m30%|███       | 271/890 [02:10<05:08,  2.01it/s]\u001b[0m\n",
      "\u001b[34m31%|███       | 272/890 [02:11<05:00,  2.05it/s]\u001b[0m\n",
      "\u001b[34m31%|███       | 273/890 [02:11<04:57,  2.08it/s]\u001b[0m\n",
      "\u001b[34m31%|███       | 274/890 [02:11<05:00,  2.05it/s]\u001b[0m\n",
      "\u001b[34m31%|███       | 275/890 [02:12<04:56,  2.07it/s]\u001b[0m\n",
      "\u001b[34m31%|███       | 276/890 [02:12<04:54,  2.09it/s]\u001b[0m\n",
      "\u001b[34m31%|███       | 277/890 [02:13<04:54,  2.09it/s]\u001b[0m\n",
      "\u001b[34m31%|███       | 278/890 [02:13<04:57,  2.06it/s]\u001b[0m\n",
      "\u001b[34m31%|███▏      | 279/890 [02:14<04:54,  2.08it/s]\u001b[0m\n",
      "\u001b[34m31%|███▏      | 280/890 [02:14<04:59,  2.04it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 281/890 [02:15<04:58,  2.04it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 282/890 [02:15<04:51,  2.08it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 283/890 [02:16<04:53,  2.07it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 284/890 [02:16<04:52,  2.07it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 285/890 [02:17<04:46,  2.11it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 286/890 [02:17<04:40,  2.15it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 287/890 [02:18<04:43,  2.13it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 288/890 [02:18<04:43,  2.12it/s]\u001b[0m\n",
      "\u001b[34m32%|███▏      | 289/890 [02:19<04:46,  2.10it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 290/890 [02:19<04:44,  2.11it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 291/890 [02:20<04:50,  2.07it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 292/890 [02:20<04:47,  2.08it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 293/890 [02:21<04:46,  2.09it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 294/890 [02:21<04:37,  2.15it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 295/890 [02:21<04:37,  2.14it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 296/890 [02:22<04:33,  2.17it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 297/890 [02:22<04:31,  2.18it/s]\u001b[0m\n",
      "\u001b[34m33%|███▎      | 298/890 [02:23<04:33,  2.16it/s]\u001b[0m\n",
      "\u001b[34m34%|███▎      | 299/890 [02:23<04:34,  2.16it/s]\u001b[0m\n",
      "\u001b[34m34%|███▎      | 300/890 [02:24<04:32,  2.17it/s]\u001b[0m\n",
      "\u001b[34m34%|███▍      | 301/890 [02:24<04:30,  2.18it/s]\u001b[0m\n",
      "\u001b[34m34%|███▍      | 302/890 [02:25<04:42,  2.08it/s]\u001b[0m\n",
      "\u001b[34m34%|███▍      | 303/890 [02:25<04:53,  2.00it/s]\u001b[0m\n",
      "\u001b[34m34%|███▍      | 304/890 [02:26<05:07,  1.91it/s]\u001b[0m\n",
      "\u001b[34m34%|███▍      | 305/890 [02:26<05:00,  1.95it/s]\u001b[0m\n",
      "\u001b[34m34%|███▍      | 306/890 [02:27<04:54,  1.98it/s]\u001b[0m\n",
      "\u001b[34m34%|███▍      | 307/890 [02:27<04:49,  2.01it/s]\u001b[0m\n",
      "\u001b[34m35%|███▍      | 308/890 [02:28<04:48,  2.02it/s]\u001b[0m\n",
      "\u001b[34m35%|███▍      | 309/890 [02:28<04:42,  2.06it/s]\u001b[0m\n",
      "\u001b[34m35%|███▍      | 310/890 [02:29<04:40,  2.07it/s]\u001b[0m\n",
      "\u001b[34m35%|███▍      | 311/890 [02:29<04:36,  2.09it/s]\u001b[0m\n",
      "\u001b[34m35%|███▌      | 312/890 [02:30<04:37,  2.08it/s]\u001b[0m\n",
      "\u001b[34m35%|███▌      | 313/890 [02:30<04:38,  2.07it/s]\u001b[0m\n",
      "\u001b[34m35%|███▌      | 314/890 [02:31<04:40,  2.05it/s]\u001b[0m\n",
      "\u001b[34m35%|███▌      | 315/890 [02:31<04:43,  2.03it/s]\u001b[0m\n",
      "\u001b[34m36%|███▌      | 316/890 [02:32<04:36,  2.08it/s]\u001b[0m\n",
      "\u001b[34m36%|███▌      | 317/890 [02:32<04:37,  2.06it/s]\u001b[0m\n",
      "\u001b[34m36%|███▌      | 318/890 [02:33<04:35,  2.08it/s]\u001b[0m\n",
      "\u001b[34m36%|███▌      | 319/890 [02:33<04:34,  2.08it/s]\u001b[0m\n",
      "\u001b[34m36%|███▌      | 320/890 [02:34<04:35,  2.07it/s]\u001b[0m\n",
      "\u001b[34m36%|███▌      | 321/890 [02:34<04:53,  1.94it/s]\u001b[0m\n",
      "\u001b[34m36%|███▌      | 322/890 [02:35<04:44,  2.00it/s]\u001b[0m\n",
      "\u001b[34m36%|███▋      | 323/890 [02:35<04:41,  2.01it/s]\u001b[0m\n",
      "\u001b[34m36%|███▋      | 324/890 [02:36<04:37,  2.04it/s]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 325/890 [02:36<04:36,  2.04it/s]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 326/890 [02:37<04:40,  2.01it/s]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 327/890 [02:37<04:54,  1.91it/s]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 328/890 [02:38<04:46,  1.96it/s]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 329/890 [02:38<04:36,  2.03it/s]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 330/890 [02:39<04:32,  2.05it/s]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 331/890 [02:39<05:05,  1.83it/s]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 332/890 [02:40<05:13,  1.78it/s]\u001b[0m\n",
      "\u001b[34m37%|███▋      | 333/890 [02:40<05:17,  1.76it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 334/890 [02:41<05:29,  1.69it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 335/890 [02:42<05:43,  1.61it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 336/890 [02:42<05:22,  1.72it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 337/890 [02:43<05:04,  1.82it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 338/890 [02:43<04:55,  1.87it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 339/890 [02:44<04:43,  1.94it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 340/890 [02:44<04:39,  1.97it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 341/890 [02:45<04:33,  2.01it/s]\u001b[0m\n",
      "\u001b[34m38%|███▊      | 342/890 [02:45<04:32,  2.01it/s]\u001b[0m\n",
      "\u001b[34m39%|███▊      | 343/890 [02:46<04:33,  2.00it/s]\u001b[0m\n",
      "\u001b[34m39%|███▊      | 344/890 [02:46<04:26,  2.05it/s]\u001b[0m\n",
      "\u001b[34m39%|███▉      | 345/890 [02:47<04:20,  2.10it/s]\u001b[0m\n",
      "\u001b[34m39%|███▉      | 346/890 [02:47<04:19,  2.09it/s]\u001b[0m\n",
      "\u001b[34m39%|███▉      | 347/890 [02:48<04:17,  2.11it/s]\u001b[0m\n",
      "\u001b[34m39%|███▉      | 348/890 [02:48<04:13,  2.14it/s]\u001b[0m\n",
      "\u001b[34m39%|███▉      | 349/890 [02:48<04:13,  2.14it/s]\u001b[0m\n",
      "\u001b[34m39%|███▉      | 350/890 [02:49<04:14,  2.12it/s]\u001b[0m\n",
      "\u001b[34m39%|███▉      | 351/890 [02:49<04:16,  2.10it/s]\u001b[0m\n",
      "\u001b[34m40%|███▉      | 352/890 [02:50<04:18,  2.08it/s]\u001b[0m\n",
      "\u001b[34m40%|███▉      | 353/890 [02:51<04:30,  1.98it/s]\u001b[0m\n",
      "\u001b[34m40%|███▉      | 354/890 [02:51<04:30,  1.98it/s]\u001b[0m\n",
      "\u001b[34m40%|███▉      | 355/890 [02:51<04:24,  2.02it/s]\u001b[0m\n",
      "\u001b[34m40%|████      | 356/890 [02:52<04:20,  2.05it/s]\u001b[0m\n",
      "\u001b[34m40%|████      | 357/890 [02:52<04:18,  2.06it/s]\u001b[0m\n",
      "\u001b[34m40%|████      | 358/890 [02:53<04:15,  2.08it/s]\u001b[0m\n",
      "\u001b[34m40%|████      | 359/890 [02:53<04:17,  2.06it/s]\u001b[0m\n",
      "\u001b[34m40%|████      | 360/890 [02:54<04:19,  2.04it/s]\u001b[0m\n",
      "\u001b[34m41%|████      | 361/890 [02:54<04:24,  2.00it/s]\u001b[0m\n",
      "\u001b[34m41%|████      | 362/890 [02:55<04:19,  2.03it/s]\u001b[0m\n",
      "\u001b[34m41%|████      | 363/890 [02:55<04:28,  1.96it/s]\u001b[0m\n",
      "\u001b[34m41%|████      | 364/890 [02:56<04:43,  1.85it/s]\u001b[0m\n",
      "\u001b[34m41%|████      | 365/890 [02:57<04:39,  1.88it/s]\u001b[0m\n",
      "\u001b[34m41%|████      | 366/890 [02:57<04:32,  1.92it/s]\u001b[0m\n",
      "\u001b[34m41%|████      | 367/890 [02:58<04:28,  1.94it/s]\u001b[0m\n",
      "\u001b[34m41%|████▏     | 368/890 [02:58<04:24,  1.98it/s]\u001b[0m\n",
      "\u001b[34m41%|████▏     | 369/890 [02:59<04:19,  2.01it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 370/890 [02:59<04:14,  2.04it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 371/890 [03:00<04:16,  2.02it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 372/890 [03:00<04:11,  2.06it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 373/890 [03:00<04:12,  2.05it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 374/890 [03:01<04:10,  2.06it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 375/890 [03:01<04:07,  2.08it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 376/890 [03:02<04:06,  2.09it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 377/890 [03:02<04:07,  2.07it/s]\u001b[0m\n",
      "\u001b[34m42%|████▏     | 378/890 [03:03<04:05,  2.09it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 379/890 [03:03<04:03,  2.10it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 380/890 [03:04<04:02,  2.10it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 381/890 [03:04<04:01,  2.11it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 382/890 [03:05<04:02,  2.09it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 383/890 [03:05<04:01,  2.10it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 384/890 [03:06<03:59,  2.11it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 385/890 [03:06<03:56,  2.13it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 386/890 [03:07<04:02,  2.08it/s]\u001b[0m\n",
      "\u001b[34m43%|████▎     | 387/890 [03:07<04:02,  2.08it/s]\u001b[0m\n",
      "\u001b[34m44%|████▎     | 388/890 [03:08<04:03,  2.06it/s]\u001b[0m\n",
      "\u001b[34m44%|████▎     | 389/890 [03:08<04:01,  2.07it/s]\u001b[0m\n",
      "\u001b[34m44%|████▍     | 390/890 [03:09<03:56,  2.12it/s]\u001b[0m\n",
      "\u001b[34m44%|████▍     | 391/890 [03:09<03:58,  2.09it/s]\u001b[0m\n",
      "\u001b[34m44%|████▍     | 392/890 [03:10<03:56,  2.11it/s]\u001b[0m\n",
      "\u001b[34m44%|████▍     | 393/890 [03:10<04:10,  1.98it/s]\u001b[0m\n",
      "\u001b[34m44%|████▍     | 394/890 [03:11<04:07,  2.00it/s]\u001b[0m\n",
      "\u001b[34m44%|████▍     | 395/890 [03:11<04:07,  2.00it/s]\u001b[0m\n",
      "\u001b[34m44%|████▍     | 396/890 [03:12<04:06,  2.00it/s]\u001b[0m\n",
      "\u001b[34m45%|████▍     | 397/890 [03:12<04:06,  2.00it/s]\u001b[0m\n",
      "\u001b[34m45%|████▍     | 398/890 [03:13<04:04,  2.01it/s]\u001b[0m\n",
      "\u001b[34m45%|████▍     | 399/890 [03:13<04:00,  2.04it/s]\u001b[0m\n",
      "\u001b[34m45%|████▍     | 400/890 [03:14<03:59,  2.05it/s]\u001b[0m\n",
      "\u001b[34m45%|████▌     | 401/890 [03:14<03:56,  2.06it/s]\u001b[0m\n",
      "\u001b[34m45%|████▌     | 402/890 [03:15<04:00,  2.03it/s]\u001b[0m\n",
      "\u001b[34m45%|████▌     | 403/890 [03:15<03:57,  2.05it/s]\u001b[0m\n",
      "\u001b[34m45%|████▌     | 404/890 [03:15<03:53,  2.08it/s]\u001b[0m\n",
      "\u001b[34m46%|████▌     | 405/890 [03:16<03:55,  2.06it/s]\u001b[0m\n",
      "\u001b[34m46%|████▌     | 406/890 [03:16<03:55,  2.06it/s]\u001b[0m\n",
      "\u001b[34m46%|████▌     | 407/890 [03:17<03:55,  2.05it/s]\u001b[0m\n",
      "\u001b[34m46%|████▌     | 408/890 [03:17<03:55,  2.05it/s]\u001b[0m\n",
      "\u001b[34m46%|████▌     | 409/890 [03:18<03:56,  2.03it/s]\u001b[0m\n",
      "\u001b[34m46%|████▌     | 410/890 [03:18<03:54,  2.05it/s]\u001b[0m\n",
      "\u001b[34m46%|████▌     | 411/890 [03:19<03:53,  2.06it/s]\u001b[0m\n",
      "\u001b[34m46%|████▋     | 412/890 [03:19<03:52,  2.06it/s]\u001b[0m\n",
      "\u001b[34m46%|████▋     | 413/890 [03:20<03:48,  2.09it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 414/890 [03:20<03:47,  2.09it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 415/890 [03:21<03:46,  2.10it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 416/890 [03:21<03:49,  2.07it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 417/890 [03:22<03:43,  2.11it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 418/890 [03:22<03:43,  2.11it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 419/890 [03:23<03:42,  2.12it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 420/890 [03:23<03:41,  2.12it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 421/890 [03:24<03:41,  2.12it/s]\u001b[0m\n",
      "\u001b[34m47%|████▋     | 422/890 [03:24<03:41,  2.12it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 423/890 [03:25<03:41,  2.11it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 424/890 [03:25<03:47,  2.05it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 425/890 [03:26<03:43,  2.08it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 426/890 [03:26<03:52,  2.00it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 427/890 [03:27<03:57,  1.95it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 428/890 [03:27<03:54,  1.97it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 429/890 [03:28<03:49,  2.01it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 430/890 [03:28<03:51,  1.99it/s]\u001b[0m\n",
      "\u001b[34m48%|████▊     | 431/890 [03:29<03:47,  2.02it/s]\u001b[0m\n",
      "\u001b[34m49%|████▊     | 432/890 [03:29<03:48,  2.01it/s]\u001b[0m\n",
      "\u001b[34m49%|████▊     | 433/890 [03:30<03:44,  2.04it/s]\u001b[0m\n",
      "\u001b[34m49%|████▉     | 434/890 [03:30<03:38,  2.09it/s]\u001b[0m\n",
      "\u001b[34m49%|████▉     | 435/890 [03:30<03:34,  2.12it/s]\u001b[0m\n",
      "\u001b[34m49%|████▉     | 436/890 [03:31<03:32,  2.14it/s]\u001b[0m\n",
      "\u001b[34m49%|████▉     | 437/890 [03:31<03:33,  2.12it/s]\u001b[0m\n",
      "\u001b[34m49%|████▉     | 438/890 [03:32<03:30,  2.15it/s]\u001b[0m\n",
      "\u001b[34m49%|████▉     | 439/890 [03:32<03:35,  2.10it/s]\u001b[0m\n",
      "\u001b[34m49%|████▉     | 440/890 [03:33<03:32,  2.12it/s]\u001b[0m\n",
      "\u001b[34m50%|████▉     | 441/890 [03:33<03:33,  2.10it/s]\u001b[0m\n",
      "\u001b[34m50%|████▉     | 442/890 [03:34<03:32,  2.11it/s]\u001b[0m\n",
      "\u001b[34m50%|████▉     | 443/890 [03:34<03:35,  2.08it/s]\u001b[0m\n",
      "\u001b[34m50%|████▉     | 444/890 [03:35<03:37,  2.05it/s]\u001b[0m\n",
      "\u001b[34m50%|█████     | 445/890 [03:35<03:38,  2.04it/s]\u001b[0m\n",
      "\u001b[34m50%|█████     | 446/890 [03:36<03:36,  2.05it/s]\u001b[0m\n",
      "\u001b[34m50%|█████     | 447/890 [03:36<03:37,  2.04it/s]\u001b[0m\n",
      "\u001b[34m50%|█████     | 448/890 [03:37<03:41,  2.00it/s]\u001b[0m\n",
      "\u001b[34m50%|█████     | 449/890 [03:37<03:48,  1.93it/s]\u001b[0m\n",
      "\u001b[34m51%|█████     | 450/890 [03:38<03:45,  1.95it/s]\u001b[0m\n",
      "\u001b[34m51%|█████     | 451/890 [03:38<03:41,  1.98it/s]\u001b[0m\n",
      "\u001b[34m51%|█████     | 452/890 [03:39<03:38,  2.01it/s]\u001b[0m\n",
      "\u001b[34m51%|█████     | 453/890 [03:40<04:00,  1.82it/s]\u001b[0m\n",
      "\u001b[34m51%|█████     | 454/890 [03:40<03:57,  1.83it/s]\u001b[0m\n",
      "\u001b[34m51%|█████     | 455/890 [03:41<04:02,  1.79it/s]\u001b[0m\n",
      "\u001b[34m51%|█████     | 456/890 [03:41<04:16,  1.69it/s]\u001b[0m\n",
      "\u001b[34m51%|█████▏    | 457/890 [03:42<04:22,  1.65it/s]\u001b[0m\n",
      "\u001b[34m51%|█████▏    | 458/890 [03:42<04:00,  1.79it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 459/890 [03:43<03:49,  1.88it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 460/890 [03:43<03:42,  1.93it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 461/890 [03:44<03:40,  1.95it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 462/890 [03:44<03:34,  2.00it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 463/890 [03:45<03:28,  2.05it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 464/890 [03:45<03:25,  2.07it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 465/890 [03:46<03:29,  2.03it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 466/890 [03:46<03:24,  2.07it/s]\u001b[0m\n",
      "\u001b[34m52%|█████▏    | 467/890 [03:47<03:23,  2.08it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 468/890 [03:47<03:20,  2.10it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 469/890 [03:48<03:21,  2.09it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 470/890 [03:48<03:18,  2.12it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 471/890 [03:49<03:17,  2.12it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 472/890 [03:49<03:18,  2.10it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 473/890 [03:50<03:16,  2.12it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 474/890 [03:50<03:15,  2.13it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 475/890 [03:50<03:13,  2.14it/s]\u001b[0m\n",
      "\u001b[34m53%|█████▎    | 476/890 [03:51<03:12,  2.15it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▎    | 477/890 [03:51<03:11,  2.16it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▎    | 478/890 [03:52<03:13,  2.13it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▍    | 479/890 [03:52<03:11,  2.15it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▍    | 480/890 [03:53<03:13,  2.12it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▍    | 481/890 [03:53<03:13,  2.11it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▍    | 482/890 [03:54<03:14,  2.10it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▍    | 483/890 [03:54<03:12,  2.11it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▍    | 484/890 [03:55<03:11,  2.12it/s]\u001b[0m\n",
      "\u001b[34m54%|█████▍    | 485/890 [03:55<03:14,  2.08it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▍    | 486/890 [03:56<03:11,  2.11it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▍    | 487/890 [03:56<03:08,  2.14it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▍    | 488/890 [03:57<03:18,  2.03it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▍    | 489/890 [03:57<03:16,  2.04it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▌    | 490/890 [03:58<03:11,  2.09it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▌    | 491/890 [03:58<03:09,  2.11it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▌    | 492/890 [03:59<03:08,  2.12it/s]\u001b[0m\n",
      "\u001b[34m55%|█████▌    | 493/890 [03:59<03:05,  2.14it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▌    | 494/890 [03:59<03:06,  2.12it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▌    | 495/890 [04:00<03:06,  2.11it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▌    | 496/890 [04:00<03:05,  2.12it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▌    | 497/890 [04:01<03:06,  2.10it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▌    | 498/890 [04:01<03:05,  2.11it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▌    | 499/890 [04:02<03:05,  2.10it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▌    | 500/890 [04:02<03:05,  2.10it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▋    | 501/890 [04:03<03:02,  2.13it/s]\u001b[0m\n",
      "\u001b[34m56%|█████▋    | 502/890 [04:03<03:01,  2.14it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 503/890 [04:04<02:59,  2.16it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 504/890 [04:04<02:59,  2.15it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 505/890 [04:05<02:57,  2.16it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 506/890 [04:05<02:57,  2.17it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 507/890 [04:06<02:56,  2.16it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 508/890 [04:06<03:01,  2.11it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 509/890 [04:06<02:59,  2.12it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 510/890 [04:07<02:58,  2.13it/s]\u001b[0m\n",
      "\u001b[34m57%|█████▋    | 511/890 [04:07<03:00,  2.10it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 512/890 [04:08<02:55,  2.15it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 513/890 [04:08<02:54,  2.16it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 514/890 [04:09<02:53,  2.16it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 515/890 [04:09<02:56,  2.13it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 516/890 [04:10<02:54,  2.15it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 517/890 [04:10<02:53,  2.15it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 518/890 [04:11<02:52,  2.16it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 519/890 [04:11<02:55,  2.11it/s]\u001b[0m\n",
      "\u001b[34m58%|█████▊    | 520/890 [04:12<02:53,  2.14it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▊    | 521/890 [04:12<02:51,  2.16it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▊    | 522/890 [04:13<02:54,  2.11it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▉    | 523/890 [04:13<02:54,  2.11it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▉    | 524/890 [04:14<02:52,  2.12it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▉    | 525/890 [04:14<02:49,  2.15it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▉    | 526/890 [04:14<02:47,  2.18it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▉    | 527/890 [04:15<02:47,  2.16it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▉    | 528/890 [04:15<02:49,  2.13it/s]\u001b[0m\n",
      "\u001b[34m59%|█████▉    | 529/890 [04:16<02:48,  2.15it/s]\u001b[0m\n",
      "\u001b[34m60%|█████▉    | 530/890 [04:16<02:49,  2.13it/s]\u001b[0m\n",
      "\u001b[34m60%|█████▉    | 531/890 [04:17<02:52,  2.08it/s]\u001b[0m\n",
      "\u001b[34m60%|█████▉    | 532/890 [04:17<02:51,  2.09it/s]\u001b[0m\n",
      "\u001b[34m60%|█████▉    | 533/890 [04:18<02:50,  2.10it/s]\u001b[0m\n",
      "\u001b[34m60%|██████    | 534/890 [04:18<02:52,  2.06it/s]\u001b[0m\n",
      "\u001b[34m60%|██████    | 535/890 [04:19<02:49,  2.09it/s]\u001b[0m\n",
      "\u001b[34m60%|██████    | 536/890 [04:19<02:47,  2.11it/s]\u001b[0m\n",
      "\u001b[34m60%|██████    | 537/890 [04:20<02:46,  2.12it/s]\u001b[0m\n",
      "\u001b[34m60%|██████    | 538/890 [04:20<02:44,  2.14it/s]\u001b[0m\n",
      "\u001b[34m61%|██████    | 539/890 [04:21<02:45,  2.12it/s]\u001b[0m\n",
      "\u001b[34m61%|██████    | 540/890 [04:21<02:43,  2.14it/s]\u001b[0m\n",
      "\u001b[34m61%|██████    | 541/890 [04:22<02:55,  1.99it/s]\u001b[0m\n",
      "\u001b[34m61%|██████    | 542/890 [04:22<02:50,  2.04it/s]\u001b[0m\n",
      "\u001b[34m61%|██████    | 543/890 [04:23<02:55,  1.98it/s]\u001b[0m\n",
      "\u001b[34m61%|██████    | 544/890 [04:23<02:49,  2.04it/s]\u001b[0m\n",
      "\u001b[34m61%|██████    | 545/890 [04:24<02:47,  2.06it/s]\u001b[0m\n",
      "\u001b[34m61%|██████▏   | 546/890 [04:24<02:43,  2.11it/s]\u001b[0m\n",
      "\u001b[34m61%|██████▏   | 547/890 [04:25<02:48,  2.04it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 548/890 [04:25<02:43,  2.09it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 549/890 [04:25<02:43,  2.09it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 550/890 [04:26<02:45,  2.05it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 551/890 [04:26<02:44,  2.06it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 552/890 [04:27<02:44,  2.05it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 553/890 [04:28<02:52,  1.96it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 554/890 [04:28<02:47,  2.01it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 555/890 [04:28<02:46,  2.01it/s]\u001b[0m\n",
      "\u001b[34m62%|██████▏   | 556/890 [04:29<02:42,  2.06it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 557/890 [04:29<02:39,  2.09it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 558/890 [04:30<02:39,  2.08it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 559/890 [04:30<02:37,  2.11it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 560/890 [04:31<02:36,  2.11it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 561/890 [04:31<02:33,  2.14it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 562/890 [04:32<02:44,  2.00it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 563/890 [04:32<02:41,  2.03it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 564/890 [04:33<02:40,  2.03it/s]\u001b[0m\n",
      "\u001b[34m63%|██████▎   | 565/890 [04:33<02:36,  2.07it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▎   | 566/890 [04:34<02:33,  2.11it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▎   | 567/890 [04:34<02:32,  2.12it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▍   | 568/890 [04:35<02:33,  2.09it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▍   | 569/890 [04:35<02:33,  2.10it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▍   | 570/890 [04:36<02:30,  2.13it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▍   | 571/890 [04:36<02:38,  2.02it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▍   | 572/890 [04:37<02:53,  1.83it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▍   | 573/890 [04:37<03:00,  1.76it/s]\u001b[0m\n",
      "\u001b[34m64%|██████▍   | 574/890 [04:38<02:54,  1.82it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▍   | 575/890 [04:39<03:06,  1.69it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▍   | 576/890 [04:39<03:11,  1.64it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▍   | 577/890 [04:40<02:59,  1.75it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▍   | 578/890 [04:40<02:47,  1.86it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▌   | 579/890 [04:41<02:40,  1.94it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▌   | 580/890 [04:41<02:34,  2.00it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▌   | 581/890 [04:42<02:33,  2.01it/s]\u001b[0m\n",
      "\u001b[34m65%|██████▌   | 582/890 [04:42<02:29,  2.06it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▌   | 583/890 [04:43<02:26,  2.10it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▌   | 584/890 [04:43<02:27,  2.08it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▌   | 585/890 [04:44<02:26,  2.08it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▌   | 586/890 [04:44<02:24,  2.10it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▌   | 587/890 [04:45<02:24,  2.09it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▌   | 588/890 [04:45<02:25,  2.08it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▌   | 589/890 [04:46<02:27,  2.05it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▋   | 590/890 [04:46<02:27,  2.04it/s]\u001b[0m\n",
      "\u001b[34m66%|██████▋   | 591/890 [04:46<02:23,  2.08it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 592/890 [04:47<02:26,  2.03it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 593/890 [04:47<02:24,  2.06it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 594/890 [04:48<02:25,  2.04it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 595/890 [04:48<02:21,  2.09it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 596/890 [04:49<02:20,  2.09it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 597/890 [04:49<02:18,  2.12it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 598/890 [04:50<02:16,  2.14it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 599/890 [04:50<02:14,  2.17it/s]\u001b[0m\n",
      "\u001b[34m67%|██████▋   | 600/890 [04:51<02:13,  2.17it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 601/890 [04:51<02:15,  2.13it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 602/890 [04:52<02:15,  2.12it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 603/890 [04:52<02:17,  2.08it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 604/890 [04:53<02:18,  2.06it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 605/890 [04:53<02:14,  2.12it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 606/890 [04:54<02:14,  2.12it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 607/890 [04:54<02:13,  2.12it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 608/890 [04:55<02:13,  2.11it/s]\u001b[0m\n",
      "\u001b[34m68%|██████▊   | 609/890 [04:55<02:11,  2.14it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▊   | 610/890 [04:56<02:14,  2.07it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▊   | 611/890 [04:56<02:17,  2.03it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 612/890 [04:57<02:17,  2.03it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 613/890 [04:57<02:17,  2.02it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 614/890 [04:58<02:31,  1.82it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 615/890 [04:58<02:29,  1.84it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 616/890 [04:59<02:25,  1.88it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 617/890 [04:59<02:22,  1.91it/s]\u001b[0m\n",
      "\u001b[34m69%|██████▉   | 618/890 [05:00<02:19,  1.95it/s]\u001b[0m\n",
      "\u001b[34m70%|██████▉   | 619/890 [05:00<02:17,  1.97it/s]\u001b[0m\n",
      "\u001b[34m70%|██████▉   | 620/890 [05:01<02:15,  1.99it/s]\u001b[0m\n",
      "\u001b[34m70%|██████▉   | 621/890 [05:01<02:11,  2.05it/s]\u001b[0m\n",
      "\u001b[34m70%|██████▉   | 622/890 [05:02<02:09,  2.07it/s]\u001b[0m\n",
      "\u001b[34m70%|███████   | 623/890 [05:02<02:10,  2.05it/s]\u001b[0m\n",
      "\u001b[34m70%|███████   | 624/890 [05:03<02:12,  2.01it/s]\u001b[0m\n",
      "\u001b[34m70%|███████   | 625/890 [05:03<02:12,  2.00it/s]\u001b[0m\n",
      "\u001b[34m70%|███████   | 626/890 [05:04<02:10,  2.02it/s]\u001b[0m\n",
      "\u001b[34m70%|███████   | 627/890 [05:04<02:08,  2.04it/s]\u001b[0m\n",
      "\u001b[34m71%|███████   | 628/890 [05:05<02:07,  2.06it/s]\u001b[0m\n",
      "\u001b[34m71%|███████   | 629/890 [05:05<02:07,  2.05it/s]\u001b[0m\n",
      "\u001b[34m71%|███████   | 630/890 [05:06<02:05,  2.07it/s]\u001b[0m\n",
      "\u001b[34m71%|███████   | 631/890 [05:06<02:07,  2.03it/s]\u001b[0m\n",
      "\u001b[34m71%|███████   | 632/890 [05:07<02:06,  2.04it/s]\u001b[0m\n",
      "\u001b[34m71%|███████   | 633/890 [05:07<02:03,  2.07it/s]\u001b[0m\n",
      "\u001b[34m71%|███████   | 634/890 [05:08<02:05,  2.03it/s]\u001b[0m\n",
      "\u001b[34m71%|███████▏  | 635/890 [05:08<02:04,  2.04it/s]\u001b[0m\n",
      "\u001b[34m71%|███████▏  | 636/890 [05:09<02:04,  2.04it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 637/890 [05:09<02:01,  2.08it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 638/890 [05:09<01:58,  2.13it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 639/890 [05:10<01:57,  2.13it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 640/890 [05:10<01:57,  2.12it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 641/890 [05:11<01:56,  2.14it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 642/890 [05:11<01:57,  2.11it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 643/890 [05:12<01:54,  2.16it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 644/890 [05:12<01:54,  2.14it/s]\u001b[0m\n",
      "\u001b[34m72%|███████▏  | 645/890 [05:13<01:55,  2.12it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 646/890 [05:13<01:53,  2.14it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 647/890 [05:14<01:54,  2.12it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 648/890 [05:14<01:52,  2.15it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 649/890 [05:15<01:52,  2.14it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 650/890 [05:15<01:53,  2.11it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 651/890 [05:15<01:52,  2.13it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 652/890 [05:16<01:50,  2.15it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 653/890 [05:16<01:50,  2.14it/s]\u001b[0m\n",
      "\u001b[34m73%|███████▎  | 654/890 [05:17<01:49,  2.15it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▎  | 655/890 [05:17<01:51,  2.11it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▎  | 656/890 [05:18<01:49,  2.13it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▍  | 657/890 [05:18<01:49,  2.12it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▍  | 658/890 [05:19<01:49,  2.12it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▍  | 659/890 [05:19<01:48,  2.12it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▍  | 660/890 [05:20<01:49,  2.10it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▍  | 661/890 [05:20<01:48,  2.11it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▍  | 662/890 [05:21<01:49,  2.08it/s]\u001b[0m\n",
      "\u001b[34m74%|███████▍  | 663/890 [05:21<01:47,  2.12it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▍  | 664/890 [05:22<01:46,  2.11it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▍  | 665/890 [05:22<01:48,  2.07it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▍  | 666/890 [05:23<01:47,  2.08it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▍  | 667/890 [05:23<01:48,  2.06it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 668/890 [05:24<01:46,  2.08it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 669/890 [05:24<01:48,  2.03it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 670/890 [05:25<01:45,  2.08it/s]\u001b[0m\n",
      "\u001b[34m75%|███████▌  | 671/890 [05:25<01:44,  2.09it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▌  | 672/890 [05:26<01:43,  2.10it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▌  | 673/890 [05:26<01:42,  2.12it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▌  | 674/890 [05:26<01:39,  2.16it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▌  | 675/890 [05:27<01:40,  2.13it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▌  | 676/890 [05:27<01:39,  2.14it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▌  | 677/890 [05:28<01:43,  2.05it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▌  | 678/890 [05:28<01:47,  1.97it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▋  | 679/890 [05:29<01:45,  2.00it/s]\u001b[0m\n",
      "\u001b[34m76%|███████▋  | 680/890 [05:29<01:42,  2.05it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 681/890 [05:30<01:41,  2.07it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 682/890 [05:30<01:40,  2.07it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 683/890 [05:31<01:37,  2.11it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 684/890 [05:31<01:39,  2.07it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 685/890 [05:32<01:40,  2.05it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 686/890 [05:32<01:37,  2.09it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 687/890 [05:33<01:36,  2.10it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 688/890 [05:33<01:35,  2.12it/s]\u001b[0m\n",
      "\u001b[34m77%|███████▋  | 689/890 [05:34<01:33,  2.16it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 690/890 [05:34<01:32,  2.17it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 691/890 [05:35<01:31,  2.18it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 692/890 [05:35<01:34,  2.10it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 693/890 [05:36<01:33,  2.11it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 694/890 [05:36<01:34,  2.08it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 695/890 [05:37<01:37,  1.99it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 696/890 [05:37<01:37,  1.98it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 697/890 [05:38<01:34,  2.03it/s]\u001b[0m\n",
      "\u001b[34m78%|███████▊  | 698/890 [05:38<01:36,  1.99it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▊  | 699/890 [05:39<01:33,  2.05it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▊  | 700/890 [05:39<01:33,  2.02it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▉  | 701/890 [05:40<01:42,  1.84it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▉  | 702/890 [05:40<01:38,  1.91it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▉  | 703/890 [05:41<01:42,  1.82it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▉  | 704/890 [05:41<01:48,  1.71it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▉  | 705/890 [05:42<01:51,  1.66it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▉  | 706/890 [05:43<01:42,  1.79it/s]\u001b[0m\n",
      "\u001b[34m79%|███████▉  | 707/890 [05:43<01:37,  1.88it/s]\u001b[0m\n",
      "\u001b[34m80%|███████▉  | 708/890 [05:43<01:33,  1.94it/s]\u001b[0m\n",
      "\u001b[34m80%|███████▉  | 709/890 [05:44<01:29,  2.01it/s]\u001b[0m\n",
      "\u001b[34m80%|███████▉  | 710/890 [05:44<01:27,  2.05it/s]\u001b[0m\n",
      "\u001b[34m80%|███████▉  | 711/890 [05:45<01:26,  2.08it/s]\u001b[0m\n",
      "\u001b[34m80%|████████  | 712/890 [05:45<01:24,  2.11it/s]\u001b[0m\n",
      "\u001b[34m80%|████████  | 713/890 [05:46<01:22,  2.14it/s]\u001b[0m\n",
      "\u001b[34m80%|████████  | 714/890 [05:46<01:21,  2.15it/s]\u001b[0m\n",
      "\u001b[34m80%|████████  | 715/890 [05:47<01:21,  2.16it/s]\u001b[0m\n",
      "\u001b[34m80%|████████  | 716/890 [05:47<01:19,  2.19it/s]\u001b[0m\n",
      "\u001b[34m81%|████████  | 717/890 [05:48<01:18,  2.21it/s]\u001b[0m\n",
      "\u001b[34m81%|████████  | 718/890 [05:48<01:20,  2.15it/s]\u001b[0m\n",
      "\u001b[34m81%|████████  | 719/890 [05:49<01:19,  2.16it/s]\u001b[0m\n",
      "\u001b[34m81%|████████  | 720/890 [05:49<01:19,  2.14it/s]\u001b[0m\n",
      "\u001b[34m81%|████████  | 721/890 [05:49<01:19,  2.13it/s]\u001b[0m\n",
      "\u001b[34m81%|████████  | 722/890 [05:50<01:17,  2.16it/s]\u001b[0m\n",
      "\u001b[34m81%|████████  | 723/890 [05:50<01:18,  2.14it/s]\u001b[0m\n",
      "\u001b[34m81%|████████▏ | 724/890 [05:51<01:18,  2.11it/s]\u001b[0m\n",
      "\u001b[34m81%|████████▏ | 725/890 [05:51<01:18,  2.10it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 726/890 [05:52<01:17,  2.12it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 727/890 [05:52<01:15,  2.17it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 728/890 [05:53<01:15,  2.14it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 729/890 [05:53<01:15,  2.13it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 730/890 [05:54<01:15,  2.12it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 731/890 [05:54<01:13,  2.17it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 732/890 [05:55<01:13,  2.15it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 733/890 [05:55<01:12,  2.16it/s]\u001b[0m\n",
      "\u001b[34m82%|████████▏ | 734/890 [05:56<01:11,  2.17it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 735/890 [05:56<01:14,  2.08it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 736/890 [05:57<01:16,  2.03it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 737/890 [05:57<01:13,  2.07it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 738/890 [05:58<01:12,  2.11it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 739/890 [05:58<01:12,  2.08it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 740/890 [05:59<01:20,  1.87it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 741/890 [05:59<01:15,  1.96it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 742/890 [06:00<01:14,  1.99it/s]\u001b[0m\n",
      "\u001b[34m83%|████████▎ | 743/890 [06:00<01:11,  2.05it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▎ | 744/890 [06:01<01:10,  2.08it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▎ | 745/890 [06:01<01:08,  2.12it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▍ | 746/890 [06:01<01:07,  2.13it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▍ | 747/890 [06:02<01:07,  2.11it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▍ | 748/890 [06:02<01:06,  2.14it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▍ | 749/890 [06:03<01:05,  2.15it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▍ | 750/890 [06:03<01:06,  2.11it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▍ | 751/890 [06:04<01:06,  2.09it/s]\u001b[0m\n",
      "\u001b[34m84%|████████▍ | 752/890 [06:04<01:04,  2.13it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▍ | 753/890 [06:05<01:06,  2.07it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▍ | 754/890 [06:05<01:04,  2.10it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▍ | 755/890 [06:06<01:04,  2.08it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▍ | 756/890 [06:06<01:03,  2.12it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▌ | 757/890 [06:07<01:02,  2.12it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▌ | 758/890 [06:07<01:04,  2.05it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▌ | 759/890 [06:08<01:03,  2.08it/s]\u001b[0m\n",
      "\u001b[34m85%|████████▌ | 760/890 [06:08<01:04,  2.03it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▌ | 761/890 [06:09<01:03,  2.03it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▌ | 762/890 [06:09<01:03,  2.02it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▌ | 763/890 [06:10<01:01,  2.07it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▌ | 764/890 [06:10<01:00,  2.09it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▌ | 765/890 [06:11<00:59,  2.11it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▌ | 766/890 [06:11<00:58,  2.13it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▌ | 767/890 [06:11<00:57,  2.14it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▋ | 768/890 [06:12<00:56,  2.16it/s]\u001b[0m\n",
      "\u001b[34m86%|████████▋ | 769/890 [06:12<00:55,  2.18it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 770/890 [06:13<00:55,  2.16it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 771/890 [06:13<00:55,  2.16it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 772/890 [06:14<00:54,  2.16it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 773/890 [06:14<00:53,  2.18it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 774/890 [06:15<00:53,  2.18it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 775/890 [06:15<00:53,  2.16it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 776/890 [06:16<00:52,  2.17it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 777/890 [06:16<00:52,  2.17it/s]\u001b[0m\n",
      "\u001b[34m87%|████████▋ | 778/890 [06:17<00:51,  2.18it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 779/890 [06:17<00:51,  2.15it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 780/890 [06:17<00:51,  2.14it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 781/890 [06:18<00:51,  2.12it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 782/890 [06:18<00:49,  2.17it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 783/890 [06:19<00:49,  2.14it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 784/890 [06:19<00:50,  2.12it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 785/890 [06:20<00:49,  2.13it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 786/890 [06:20<00:51,  2.02it/s]\u001b[0m\n",
      "\u001b[34m88%|████████▊ | 787/890 [06:21<00:49,  2.07it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▊ | 788/890 [06:21<00:48,  2.09it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▊ | 789/890 [06:22<00:48,  2.09it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▉ | 790/890 [06:22<00:47,  2.12it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▉ | 791/890 [06:23<00:46,  2.13it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▉ | 792/890 [06:23<00:45,  2.14it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▉ | 793/890 [06:24<00:45,  2.15it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▉ | 794/890 [06:24<00:45,  2.12it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▉ | 795/890 [06:25<00:45,  2.08it/s]\u001b[0m\n",
      "\u001b[34m89%|████████▉ | 796/890 [06:25<00:45,  2.09it/s]\u001b[0m\n",
      "\u001b[34m90%|████████▉ | 797/890 [06:26<00:44,  2.09it/s]\u001b[0m\n",
      "\u001b[34m90%|████████▉ | 798/890 [06:26<00:44,  2.08it/s]\u001b[0m\n",
      "\u001b[34m90%|████████▉ | 799/890 [06:27<00:45,  2.01it/s]\u001b[0m\n",
      "\u001b[34m90%|████████▉ | 800/890 [06:27<00:43,  2.07it/s]\u001b[0m\n",
      "\u001b[34m90%|█████████ | 801/890 [06:28<00:43,  2.04it/s]\u001b[0m\n",
      "\u001b[34m90%|█████████ | 802/890 [06:28<00:47,  1.87it/s]\u001b[0m\n",
      "\u001b[34m90%|█████████ | 803/890 [06:29<00:44,  1.94it/s]\u001b[0m\n",
      "\u001b[34m90%|█████████ | 804/890 [06:29<00:47,  1.82it/s]\u001b[0m\n",
      "\u001b[34m90%|█████████ | 805/890 [06:30<00:45,  1.86it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████ | 806/890 [06:30<00:43,  1.94it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████ | 807/890 [06:31<00:41,  1.98it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████ | 808/890 [06:31<00:40,  2.02it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████ | 809/890 [06:32<00:39,  2.06it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████ | 810/890 [06:32<00:37,  2.11it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████ | 811/890 [06:33<00:37,  2.12it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████ | 812/890 [06:33<00:36,  2.13it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████▏| 813/890 [06:33<00:35,  2.15it/s]\u001b[0m\n",
      "\u001b[34m91%|█████████▏| 814/890 [06:34<00:36,  2.09it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 815/890 [06:34<00:35,  2.10it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 816/890 [06:35<00:34,  2.14it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 817/890 [06:35<00:33,  2.15it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 818/890 [06:36<00:33,  2.14it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 819/890 [06:36<00:32,  2.16it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 820/890 [06:37<00:32,  2.13it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 821/890 [06:37<00:34,  1.99it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 822/890 [06:38<00:33,  2.05it/s]\u001b[0m\n",
      "\u001b[34m92%|█████████▏| 823/890 [06:38<00:32,  2.07it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 824/890 [06:39<00:31,  2.09it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 825/890 [06:39<00:32,  2.02it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 826/890 [06:40<00:34,  1.84it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 827/890 [06:40<00:32,  1.93it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 828/890 [06:41<00:35,  1.77it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 829/890 [06:42<00:36,  1.68it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 830/890 [06:42<00:33,  1.78it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 831/890 [06:43<00:32,  1.84it/s]\u001b[0m\n",
      "\u001b[34m93%|█████████▎| 832/890 [06:43<00:30,  1.93it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▎| 833/890 [06:44<00:28,  1.99it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▎| 834/890 [06:44<00:27,  2.02it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 835/890 [06:45<00:27,  2.01it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 836/890 [06:45<00:26,  2.05it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 837/890 [06:46<00:25,  2.09it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 838/890 [06:46<00:24,  2.09it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 839/890 [06:47<00:24,  2.07it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 840/890 [06:47<00:24,  2.06it/s]\u001b[0m\n",
      "\u001b[34m94%|█████████▍| 841/890 [06:48<00:23,  2.06it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▍| 842/890 [06:48<00:23,  2.04it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▍| 843/890 [06:48<00:22,  2.09it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▍| 844/890 [06:49<00:21,  2.14it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▍| 845/890 [06:49<00:21,  2.07it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▌| 846/890 [06:50<00:20,  2.10it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▌| 847/890 [06:50<00:20,  2.12it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▌| 848/890 [06:51<00:19,  2.11it/s]\u001b[0m\n",
      "\u001b[34m95%|█████████▌| 849/890 [06:51<00:18,  2.16it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▌| 850/890 [06:52<00:18,  2.13it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▌| 851/890 [06:52<00:17,  2.17it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▌| 852/890 [06:53<00:17,  2.16it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▌| 853/890 [06:53<00:17,  2.16it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▌| 854/890 [06:54<00:16,  2.17it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▌| 855/890 [06:54<00:16,  2.16it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▌| 856/890 [06:55<00:16,  2.05it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▋| 857/890 [06:55<00:15,  2.07it/s]\u001b[0m\n",
      "\u001b[34m96%|█████████▋| 858/890 [06:56<00:15,  2.03it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 859/890 [06:56<00:15,  2.05it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 860/890 [06:57<00:14,  2.08it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 861/890 [06:57<00:13,  2.08it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 862/890 [06:57<00:13,  2.12it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 863/890 [06:58<00:12,  2.11it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 864/890 [06:58<00:12,  2.11it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 865/890 [06:59<00:11,  2.10it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 866/890 [06:59<00:11,  2.03it/s]\u001b[0m\n",
      "\u001b[34m97%|█████████▋| 867/890 [07:00<00:11,  1.99it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 868/890 [07:00<00:10,  2.04it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 869/890 [07:01<00:10,  2.09it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 870/890 [07:01<00:09,  2.12it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 871/890 [07:02<00:08,  2.13it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 872/890 [07:02<00:08,  2.14it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 873/890 [07:03<00:07,  2.14it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 874/890 [07:03<00:07,  2.13it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 875/890 [07:04<00:07,  2.12it/s]\u001b[0m\n",
      "\u001b[34m98%|█████████▊| 876/890 [07:04<00:06,  2.16it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▊| 877/890 [07:05<00:06,  2.13it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▊| 878/890 [07:05<00:05,  2.10it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▉| 879/890 [07:06<00:05,  2.13it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▉| 880/890 [07:06<00:04,  2.08it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▉| 881/890 [07:07<00:04,  2.04it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▉| 882/890 [07:07<00:03,  2.08it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▉| 883/890 [07:08<00:03,  2.03it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▉| 884/890 [07:08<00:02,  2.08it/s]\u001b[0m\n",
      "\u001b[34m99%|█████████▉| 885/890 [07:08<00:02,  2.07it/s]\u001b[0m\n",
      "\u001b[34m100%|█████████▉| 886/890 [07:09<00:01,  2.10it/s]\u001b[0m\n",
      "\u001b[34m100%|█████████▉| 887/890 [07:09<00:01,  2.09it/s]\u001b[0m\n",
      "\u001b[34m100%|█████████▉| 888/890 [07:10<00:00,  2.06it/s]\u001b[0m\n",
      "\u001b[34m100%|█████████▉| 889/890 [07:10<00:00,  2.09it/s]\u001b[0m\n",
      "\u001b[34m100%|██████████| 890/890 [07:11<00:00,  2.09it/s]\u001b[0m\n",
      "\u001b[34m100%|██████████| 890/890 [07:11<00:00,  2.06it/s]\u001b[0m\n",
      "\u001b[34mEpochs: 1 | Train Loss:  0.465             | Train Accuracy:  0.855             | Val Loss:  0.024             | Val Accuracy:  0.986\u001b[0m\n",
      "\n",
      "2022-01-26 20:13:00 Uploading - Uploading generated training model\u001b[34mSaving the model.\u001b[0m\n",
      "\u001b[34mINFO:__main__:Saving the model.\u001b[0m\n",
      "\u001b[34m2022-01-26 20:12:56,169 sagemaker-training-toolkit INFO     Reporting training SUCCESS\u001b[0m\n",
      "\n",
      "2022-01-26 20:14:03 Completed - Training job completed\n",
      "Training seconds: 650\n",
      "Billable seconds: 650\n"
     ]
    }
   ],
   "source": [
    "pytorch_estimator.fit({\n",
    "    'train': train_data_location,\n",
    "    'val': val_data_location,\n",
    "    'test': test_data_location\n",
    "                      })"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48856f88",
   "metadata": {},
   "source": [
    "## Deploy the model to a SageMaker Endpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b03baaef",
   "metadata": {},
   "source": [
    "After training, the model is saved automatically to a S3 bucket accessible using `.model_data` method. We can either go to this bucket on AWS console and download the model for deployment elsewhere (e.g. an EC2 instance), or we can deploy the model right here within this notebook. I will show the latter here.\n",
    "\n",
    "As mentioned above we have implementation of `model_fn` in the `train_deploy.py` script that is required. We also adapted `input_fn`, `predict_fn` and `output_fn` functions according to our specific text processing steps as instructed [here](https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/using_pytorch.html#serve-a-pytorch-model).\n",
    "\n",
    "The arguments to the deploy function allow us to set the number and type of instances that will be used for the Endpoint. These do not need to be the same as the values we used for the training job. For example, you can train a model on a set of GPU-based instances, and then deploy the Endpoint to a fleet of CPU-based instances, but you need to make sure that you return or save your model as a cpu model similar to what we did in `train_deploy.py`. Here we will deploy the model to a single `ml.t2.medium` instance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "352a3345",
   "metadata": {},
   "source": [
    "To deploy the model within this notebook instance, we first need to use SageMaker's `PytorchModel` object to load the saved model. This step can be skipped since we can directly use the `pytorch_estimator` object trained in the last step. However, in practice, we often need to rerun this notebook for debugging/test, and to avoid having to train the model every time we open this notebook, I created this \"breakpoint\" here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "7831d59b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'pytorch_estimator' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-12-aeda050096cc>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmodel_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpytorch_estimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel_data\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel_data\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'pytorch_estimator' is not defined"
     ]
    }
   ],
   "source": [
    "model_data = pytorch_estimator.model_data\n",
    "print(model_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "bc458565",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_data = \"s3://sagemaker-us-east-1-439373211214/pytorch-training-2022-01-26-20-00-11-378/output/model.tar.gz\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "ba05c3fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "pytorch_model = PyTorchModel(model_data=model_data,\n",
    "                             role=role,\n",
    "                             framework_version=\"1.10\",\n",
    "                             source_dir=\"code\",\n",
    "                             py_version=\"py38\",\n",
    "                             entry_point=\"train_deploy.py\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d905b27",
   "metadata": {},
   "source": [
    "Then, we use the loaded model object to deploy a `PyTorchPredictor`. This creates a Sagemaker Endpoint -- a hosted prediction service that we can use to perform inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c2733e30",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------!"
     ]
    }
   ],
   "source": [
    "predictor = pytorch_model.deploy(initial_instance_count=1, \n",
    "                                 instance_type=\"ml.t2.medium\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcd93ab6",
   "metadata": {},
   "source": [
    "## Invoke the Endpoint within the SageMaker Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08f253ce",
   "metadata": {},
   "source": [
    "After deployment, we can evaluate the Endpoint using example input text. In our `input_fn` function in `train_deploy.py`, we require the input data format to be `JSON`, so we need to convert our example input text to `JSON` format and then feed it to our predictor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "9f545cec",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor.serializer = json_serializer\n",
    "predictor.deserializer = json_deserializer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10f400f1",
   "metadata": {},
   "source": [
    "We use a clipped news from [BBC Politics](https://www.bbc.com/news/uk-60095459) as an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "0279aee0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "example_text = \"\"\"\n",
    "The UK has accused President Putin of plotting to install a pro-Moscow figure to lead Ukraine's government.\n",
    "\n",
    "The Foreign Office took the unusual step of naming former Ukrainian MP Yevhen Murayev as a potential Kremlin candidate.\n",
    "\n",
    "Russia has moved 100,000 troops near to its border with Ukraine but denies it is planning an invasion.\n",
    "\n",
    "UK ministers have warned that the Russian government will face serious consequences if there is an incursion.\n",
    "\n",
    "In a statement, Foreign Secretary Liz Truss said: \"The information being released today shines a light on the extent of Russian activity designed to subvert Ukraine, and is an insight into Kremlin thinking.\n",
    "\n",
    "\"Russia must de-escalate, end its campaigns of aggression and disinformation, and pursue a path of diplomacy.\"\n",
    "\n",
    "The Russian Ministry of Foreign Affairs tweeted that the Foreign Office was \"circulating disinformation\" and urged it to \"cease these provocative activities\" and \"stop spreading nonsense\".\n",
    "\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ca0b3a0",
   "metadata": {},
   "source": [
    "Let's see if our predictor can correctly classify this news:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "0f3f0c0a",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "The json_serializer has been renamed in sagemaker>=2.\n",
      "See: https://sagemaker.readthedocs.io/en/stable/v2.html for details.\n",
      "The json_deserializer has been renamed in sagemaker>=2.\n",
      "See: https://sagemaker.readthedocs.io/en/stable/v2.html for details.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'politics': 0.9923005104064941, 'business': 0.005291244480758905, 'sport': 0.0014019005466252565, 'entertainment': 0.0005219860468059778, 'tech': 0.00048434155178256333}\n"
     ]
    }
   ],
   "source": [
    "result = predictor.predict({\"text\": example_text})\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cbfab6d",
   "metadata": {},
   "source": [
    "Great! It did classify the news as \"politics\"! \"politics\" has the highest probability of 0.99. Good job, GPT-2!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3134f0e2",
   "metadata": {},
   "source": [
    "## Invoke the endpoint from a Streamlit app"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2c74032",
   "metadata": {},
   "source": [
    "With our model endpoint deployed on SageMaker Notebook Instance, we can also develop a Streamlit app on this instance for quick prototyping. [Streamlit](https://streamlit.io/) is a really nice tool for this purpose because of its simplicity, flexibility and integrations with a wide variety of visualisation tools such as [Altair](https://altair-viz.github.io/), [Bokeh](https://docs.bokeh.org/en/latest/index.html) and [Plotly](https://plotly.com/). The source code for our Streamlit app can be found at `./streamlit_app/src/app.py`.\n",
    "\n",
    "To avoid differences between development and deployment environments, we use Docker containers to simplify the deployment. Our Dockerfile is defined at `./streamlit_app/Dockerfile` and has an `CMD` set to `streamlit run src/app.py` to start the app server within the container. We start off by building the Docker image, and setting the SAGEMAKER_ENDPOINT_NAME argument to `predictor.endpoint_name`, so the app invokes the correct model endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f328d790",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pytorch-inference-2022-01-27-03-52-16-369\n"
     ]
    }
   ],
   "source": [
    "print(predictor.endpoint_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "803c5c6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker_endpoint_name = \"pytorch-inference-2022-01-27-03-52-16-369\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "fcd979ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "!export SAGEMAKER_ENDPOINT_NAME=\"pytorch-inference-2022-01-27-03-52-16-369\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "bebce4d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "image_name = \"streamlit:gpt2-news-classifier-app\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "06556e8c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sending build context to Docker daemon  8.192kB\n",
      "Step 1/15 : FROM python:3.7.4-slim-stretch\n",
      " ---> fad2b9f06d3b\n",
      "Step 2/15 : ARG SAGEMAKER_ENDPOINT_NAME\n",
      " ---> Running in 5d1d27361b30\n",
      "Removing intermediate container 5d1d27361b30\n",
      " ---> fa39be725832\n",
      "Step 3/15 : ENV SAGEMAKER_ENDPOINT_NAME=$SAGEMAKER_ENDPOINT_NAME\n",
      " ---> Running in a487ce73a924\n",
      "Removing intermediate container a487ce73a924\n",
      " ---> f5e513440b92\n",
      "Step 4/15 : WORKDIR /usr/src/app\n",
      " ---> Running in c8fde7602857\n",
      "Removing intermediate container c8fde7602857\n",
      " ---> 002d541ef861\n",
      "Step 5/15 : COPY requirements.txt requirements.txt\n",
      " ---> 4271a85c9653\n",
      "Step 6/15 : COPY src/ src/\n",
      " ---> 7f41933884a1\n",
      "Step 7/15 : RUN pip install --upgrade pip\n",
      " ---> Running in efe8d37135a5\n",
      "Collecting pip\n",
      "  Downloading https://files.pythonhosted.org/packages/a4/6d/6463d49a933f547439d6b5b98b46af8742cc03ae83543e4d7688c2420f8b/pip-21.3.1-py3-none-any.whl (1.7MB)\n",
      "Installing collected packages: pip\n",
      "  Found existing installation: pip 19.3\n",
      "    Uninstalling pip-19.3:\n",
      "      Successfully uninstalled pip-19.3\n",
      "Successfully installed pip-21.3.1\n",
      "Removing intermediate container efe8d37135a5\n",
      " ---> 91d04f0bdeaa\n",
      "Step 8/15 : RUN pip install -r requirements.txt\n",
      " ---> Running in c1acc7f8723b\n",
      "Collecting streamlit==1.3.0\n",
      "  Downloading streamlit-1.3.0-py2.py3-none-any.whl (9.2 MB)\n",
      "Collecting boto3\n",
      "  Downloading boto3-1.20.44-py3-none-any.whl (131 kB)\n",
      "Collecting astor\n",
      "  Downloading astor-0.8.1-py2.py3-none-any.whl (27 kB)\n",
      "Collecting numpy\n",
      "  Downloading numpy-1.21.5-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)\n",
      "Collecting cachetools>=4.0\n",
      "  Downloading cachetools-5.0.0-py3-none-any.whl (9.1 kB)\n",
      "Collecting pydeck>=0.1.dev5\n",
      "  Downloading pydeck-0.7.1-py2.py3-none-any.whl (4.3 MB)\n",
      "Collecting gitpython!=3.1.19\n",
      "  Downloading GitPython-3.1.26-py3-none-any.whl (180 kB)\n",
      "Collecting pympler>=0.9\n",
      "  Downloading Pympler-1.0.1-py3-none-any.whl (164 kB)\n",
      "Collecting toml\n",
      "  Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)\n",
      "Collecting blinker\n",
      "  Downloading blinker-1.4.tar.gz (111 kB)\n",
      "  Preparing metadata (setup.py): started\n",
      "  Preparing metadata (setup.py): finished with status 'done'\n",
      "Collecting pandas>=0.21.0\n",
      "  Downloading pandas-1.3.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.3 MB)\n",
      "Collecting base58\n",
      "  Downloading base58-2.1.1-py3-none-any.whl (5.6 kB)\n",
      "Collecting validators\n",
      "  Downloading validators-0.18.2-py3-none-any.whl (19 kB)\n",
      "Collecting packaging\n",
      "  Downloading packaging-21.3-py3-none-any.whl (40 kB)\n",
      "Collecting pillow>=6.2.0\n",
      "  Downloading Pillow-9.0.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB)\n",
      "Collecting requests\n",
      "  Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)\n",
      "Collecting altair>=3.2.0\n",
      "  Downloading altair-4.2.0-py3-none-any.whl (812 kB)\n",
      "Collecting python-dateutil\n",
      "  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)\n",
      "Collecting tzlocal\n",
      "  Downloading tzlocal-4.1-py3-none-any.whl (19 kB)\n",
      "Collecting watchdog\n",
      "  Downloading watchdog-2.1.6-py3-none-manylinux2014_x86_64.whl (76 kB)\n",
      "Collecting pyarrow\n",
      "  Downloading pyarrow-6.0.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25.6 MB)\n",
      "Collecting tornado>=5.0\n",
      "  Downloading tornado-6.1-cp37-cp37m-manylinux2010_x86_64.whl (428 kB)\n",
      "Collecting protobuf!=3.11,>=3.6.0\n",
      "  Downloading protobuf-3.19.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)\n",
      "Collecting click<8.0,>=7.0\n",
      "  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)\n",
      "Collecting attrs\n",
      "  Downloading attrs-21.4.0-py2.py3-none-any.whl (60 kB)\n",
      "Collecting s3transfer<0.6.0,>=0.5.0\n",
      "  Downloading s3transfer-0.5.0-py3-none-any.whl (79 kB)\n",
      "Collecting jmespath<1.0.0,>=0.7.1\n",
      "  Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)\n",
      "Collecting botocore<1.24.0,>=1.23.44\n",
      "  Downloading botocore-1.23.44-py3-none-any.whl (8.5 MB)\n",
      "Collecting jinja2\n",
      "  Downloading Jinja2-3.0.3-py3-none-any.whl (133 kB)\n",
      "Collecting toolz\n",
      "  Downloading toolz-0.11.2-py3-none-any.whl (55 kB)\n",
      "Collecting jsonschema>=3.0\n",
      "  Downloading jsonschema-4.4.0-py3-none-any.whl (72 kB)\n",
      "Collecting entrypoints\n",
      "  Downloading entrypoints-0.3-py2.py3-none-any.whl (11 kB)\n",
      "Collecting urllib3<1.27,>=1.25.4\n",
      "  Downloading urllib3-1.26.8-py2.py3-none-any.whl (138 kB)\n",
      "Collecting typing-extensions>=3.7.4.3\n",
      "  Downloading typing_extensions-4.0.1-py3-none-any.whl (22 kB)\n",
      "Collecting gitdb<5,>=4.0.1\n",
      "  Downloading gitdb-4.0.9-py3-none-any.whl (63 kB)\n",
      "Collecting pytz>=2017.3\n",
      "  Downloading pytz-2021.3-py2.py3-none-any.whl (503 kB)\n",
      "Collecting ipywidgets>=7.0.0\n",
      "  Downloading ipywidgets-7.6.5-py2.py3-none-any.whl (121 kB)\n",
      "Collecting traitlets>=4.3.2\n",
      "  Downloading traitlets-5.1.1-py3-none-any.whl (102 kB)\n",
      "Collecting ipykernel>=5.1.2\n",
      "  Downloading ipykernel-6.7.0-py3-none-any.whl (127 kB)\n",
      "Collecting six>=1.5\n",
      "  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)\n",
      "Collecting pyparsing!=3.0.5,>=2.0.2\n",
      "  Downloading pyparsing-3.0.7-py3-none-any.whl (98 kB)\n",
      "Collecting charset-normalizer~=2.0.0\n",
      "  Downloading charset_normalizer-2.0.10-py3-none-any.whl (39 kB)\n",
      "Collecting certifi>=2017.4.17\n",
      "  Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB)\n",
      "Collecting idna<4,>=2.5\n",
      "  Downloading idna-3.3-py3-none-any.whl (61 kB)\n",
      "Collecting backports.zoneinfo\n",
      "  Downloading backports.zoneinfo-0.2.1-cp37-cp37m-manylinux1_x86_64.whl (70 kB)\n",
      "Collecting pytz-deprecation-shim\n",
      "  Downloading pytz_deprecation_shim-0.1.0.post0-py2.py3-none-any.whl (15 kB)\n",
      "Collecting decorator>=3.4.0\n",
      "  Downloading decorator-5.1.1-py3-none-any.whl (9.1 kB)\n",
      "Collecting smmap<6,>=3.0.1\n",
      "  Downloading smmap-5.0.0-py3-none-any.whl (24 kB)\n",
      "Collecting jupyter-client<8.0\n",
      "  Downloading jupyter_client-7.1.2-py3-none-any.whl (130 kB)\n",
      "Collecting matplotlib-inline<0.2.0,>=0.1.0\n",
      "  Downloading matplotlib_inline-0.1.3-py3-none-any.whl (8.2 kB)\n",
      "Collecting nest-asyncio\n",
      "  Downloading nest_asyncio-1.5.4-py3-none-any.whl (5.1 kB)\n",
      "Collecting debugpy<2.0,>=1.0.0\n",
      "  Downloading debugpy-1.5.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.9 MB)\n",
      "Collecting ipython>=7.23.1\n",
      "  Downloading ipython-7.31.1-py3-none-any.whl (792 kB)\n",
      "Collecting nbformat>=4.2.0\n",
      "  Downloading nbformat-5.1.3-py3-none-any.whl (178 kB)\n",
      "Collecting jupyterlab-widgets>=1.0.0\n",
      "  Downloading jupyterlab_widgets-1.0.2-py3-none-any.whl (243 kB)\n",
      "Collecting ipython-genutils~=0.2.0\n",
      "  Downloading ipython_genutils-0.2.0-py2.py3-none-any.whl (26 kB)\n",
      "Collecting widgetsnbextension~=3.5.0\n",
      "  Downloading widgetsnbextension-3.5.2-py2.py3-none-any.whl (1.6 MB)\n",
      "Collecting MarkupSafe>=2.0\n",
      "  Downloading MarkupSafe-2.0.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (31 kB)\n",
      "Collecting importlib-resources>=1.4.0\n",
      "  Downloading importlib_resources-5.4.0-py3-none-any.whl (28 kB)\n",
      "Collecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0\n",
      "  Downloading pyrsistent-0.18.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (117 kB)\n",
      "Collecting importlib-metadata\n",
      "  Downloading importlib_metadata-4.10.1-py3-none-any.whl (17 kB)\n",
      "Collecting tzdata\n",
      "  Downloading tzdata-2021.5-py2.py3-none-any.whl (339 kB)\n",
      "Collecting zipp>=3.1.0\n",
      "  Downloading zipp-3.7.0-py3-none-any.whl (5.3 kB)\n",
      "Collecting pickleshare\n",
      "  Downloading pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)\n",
      "Collecting prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0\n",
      "  Downloading prompt_toolkit-3.0.24-py3-none-any.whl (374 kB)\n",
      "Collecting jedi>=0.16\n",
      "  Downloading jedi-0.18.1-py2.py3-none-any.whl (1.6 MB)\n",
      "Collecting pexpect>4.3\n",
      "  Downloading pexpect-4.8.0-py2.py3-none-any.whl (59 kB)\n",
      "Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.7/site-packages (from ipython>=7.23.1->ipykernel>=5.1.2->pydeck>=0.1.dev5->streamlit==1.3.0->-r requirements.txt (line 1)) (41.4.0)\n",
      "Collecting backcall\n",
      "  Downloading backcall-0.2.0-py2.py3-none-any.whl (11 kB)\n",
      "Collecting pygments\n",
      "  Downloading Pygments-2.11.2-py3-none-any.whl (1.1 MB)\n",
      "Collecting jupyter-core>=4.6.0\n",
      "  Downloading jupyter_core-4.9.1-py3-none-any.whl (86 kB)\n",
      "Collecting pyzmq>=13\n",
      "  Downloading pyzmq-22.3.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.1 MB)\n",
      "Collecting notebook>=4.4.1\n",
      "  Downloading notebook-6.4.8-py3-none-any.whl (9.9 MB)\n",
      "Collecting parso<0.9.0,>=0.8.0\n",
      "  Downloading parso-0.8.3-py2.py3-none-any.whl (100 kB)\n",
      "Collecting argon2-cffi\n",
      "  Downloading argon2_cffi-21.3.0-py3-none-any.whl (14 kB)\n",
      "Collecting Send2Trash>=1.8.0\n",
      "  Downloading Send2Trash-1.8.0-py3-none-any.whl (18 kB)\n",
      "Collecting terminado>=0.8.3\n",
      "  Downloading terminado-0.13.0-py3-none-any.whl (14 kB)\n",
      "Collecting prometheus-client\n",
      "  Downloading prometheus_client-0.13.0-py3-none-any.whl (56 kB)\n",
      "Collecting nbconvert\n",
      "  Downloading nbconvert-6.4.1-py3-none-any.whl (557 kB)\n",
      "Collecting ptyprocess>=0.5\n",
      "  Downloading ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)\n",
      "Collecting wcwidth\n",
      "  Downloading wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)\n",
      "Collecting argon2-cffi-bindings\n",
      "  Downloading argon2_cffi_bindings-21.2.0-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (86 kB)\n",
      "Collecting mistune<2,>=0.8.1\n",
      "  Downloading mistune-0.8.4-py2.py3-none-any.whl (16 kB)\n",
      "Collecting defusedxml\n",
      "  Downloading defusedxml-0.7.1-py2.py3-none-any.whl (25 kB)\n",
      "Collecting nbclient<0.6.0,>=0.5.0\n",
      "  Downloading nbclient-0.5.10-py3-none-any.whl (69 kB)\n",
      "Collecting testpath\n",
      "  Downloading testpath-0.5.0-py3-none-any.whl (84 kB)\n",
      "Collecting pandocfilters>=1.4.1\n",
      "  Downloading pandocfilters-1.5.0-py2.py3-none-any.whl (8.7 kB)\n",
      "Collecting bleach\n",
      "  Downloading bleach-4.1.0-py2.py3-none-any.whl (157 kB)\n",
      "Collecting jupyterlab-pygments\n",
      "  Downloading jupyterlab_pygments-0.1.2-py2.py3-none-any.whl (4.6 kB)\n",
      "Collecting cffi>=1.0.1\n",
      "  Downloading cffi-1.15.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (427 kB)\n",
      "Collecting webencodings\n",
      "  Downloading webencodings-0.5.1-py2.py3-none-any.whl (11 kB)\n",
      "Collecting pycparser\n",
      "  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)\n",
      "Building wheels for collected packages: blinker\n",
      "  Building wheel for blinker (setup.py): started\n",
      "  Building wheel for blinker (setup.py): finished with status 'done'\n",
      "  Created wheel for blinker: filename=blinker-1.4-py3-none-any.whl size=13451 sha256=3a2ee58a6169cbcbee8fa818b16f42c989e98620f484241b88a183b345054e3f\n",
      "  Stored in directory: /root/.cache/pip/wheels/22/f5/18/df711b66eb25b21325c132757d4314db9ac5e8dabeaf196eab\n",
      "Successfully built blinker\n",
      "Installing collected packages: zipp, typing-extensions, traitlets, six, pyrsistent, importlib-resources, importlib-metadata, attrs, wcwidth, tornado, pyzmq, python-dateutil, pyparsing, pycparser, ptyprocess, parso, nest-asyncio, jupyter-core, jsonschema, ipython-genutils, entrypoints, webencodings, pygments, prompt-toolkit, pickleshare, pexpect, packaging, nbformat, matplotlib-inline, MarkupSafe, jupyter-client, jedi, decorator, cffi, backcall, testpath, pandocfilters, nbclient, mistune, jupyterlab-pygments, jinja2, ipython, defusedxml, debugpy, bleach, argon2-cffi-bindings, terminado, Send2Trash, prometheus-client, nbconvert, ipykernel, argon2-cffi, notebook, widgetsnbextension, urllib3, tzdata, smmap, pytz, numpy, jupyterlab-widgets, jmespath, backports.zoneinfo, toolz, pytz-deprecation-shim, pandas, ipywidgets, idna, gitdb, charset-normalizer, certifi, botocore, watchdog, validators, tzlocal, toml, s3transfer, requests, pympler, pydeck, pyarrow, protobuf, pillow, gitpython, click, cachetools, blinker, base58, astor, altair, streamlit, boto3\n",
      "Successfully installed MarkupSafe-2.0.1 Send2Trash-1.8.0 altair-4.2.0 argon2-cffi-21.3.0 argon2-cffi-bindings-21.2.0 astor-0.8.1 attrs-21.4.0 backcall-0.2.0 backports.zoneinfo-0.2.1 base58-2.1.1 bleach-4.1.0 blinker-1.4 boto3-1.20.44 botocore-1.23.44 cachetools-5.0.0 certifi-2021.10.8 cffi-1.15.0 charset-normalizer-2.0.10 click-7.1.2 debugpy-1.5.1 decorator-5.1.1 defusedxml-0.7.1 entrypoints-0.3 gitdb-4.0.9 gitpython-3.1.26 idna-3.3 importlib-metadata-4.10.1 importlib-resources-5.4.0 ipykernel-6.7.0 ipython-7.31.1 ipython-genutils-0.2.0 ipywidgets-7.6.5 jedi-0.18.1 jinja2-3.0.3 jmespath-0.10.0 jsonschema-4.4.0 jupyter-client-7.1.2 jupyter-core-4.9.1 jupyterlab-pygments-0.1.2 jupyterlab-widgets-1.0.2 matplotlib-inline-0.1.3 mistune-0.8.4 nbclient-0.5.10 nbconvert-6.4.1 nbformat-5.1.3 nest-asyncio-1.5.4 notebook-6.4.8 numpy-1.21.5 packaging-21.3 pandas-1.3.5 pandocfilters-1.5.0 parso-0.8.3 pexpect-4.8.0 pickleshare-0.7.5 pillow-9.0.0 prometheus-client-0.13.0 prompt-toolkit-3.0.24 protobuf-3.19.3 ptyprocess-0.7.0 pyarrow-6.0.1 pycparser-2.21 pydeck-0.7.1 pygments-2.11.2 pympler-1.0.1 pyparsing-3.0.7 pyrsistent-0.18.1 python-dateutil-2.8.2 pytz-2021.3 pytz-deprecation-shim-0.1.0.post0 pyzmq-22.3.0 requests-2.27.1 s3transfer-0.5.0 six-1.16.0 smmap-5.0.0 streamlit-1.3.0 terminado-0.13.0 testpath-0.5.0 toml-0.10.2 toolz-0.11.2 tornado-6.1 traitlets-5.1.1 typing-extensions-4.0.1 tzdata-2021.5 tzlocal-4.1 urllib3-1.26.8 validators-0.18.2 watchdog-2.1.6 wcwidth-0.2.5 webencodings-0.5.1 widgetsnbextension-3.5.2 zipp-3.7.0\n",
      "\u001b[91mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\n",
      "\u001b[0mRemoving intermediate container c1acc7f8723b\n",
      " ---> bf4193cb3e32\n",
      "Step 9/15 : ENV LC_ALL=C.UTF-8\n",
      " ---> Running in 0a4dff28add0\n",
      "Removing intermediate container 0a4dff28add0\n",
      " ---> f3bd7fad57e4\n",
      "Step 10/15 : ENV LANG=C.UTF-8\n",
      " ---> Running in b9a14720d718\n",
      "Removing intermediate container b9a14720d718\n",
      " ---> 7a161397a17c\n",
      "Step 11/15 : RUN mkdir -p /root/.streamlit\n",
      " ---> Running in ab1c1d0ced7c\n",
      "Removing intermediate container ab1c1d0ced7c\n",
      " ---> b4e6594f74e9\n",
      "Step 12/15 : RUN bash -c 'echo -e \"[general]\\nemail = \\\"\\\"\\n\" > /root/.streamlit/credentials.toml'\n",
      " ---> Running in 78c94d062099\n",
      "Removing intermediate container 78c94d062099\n",
      " ---> 3ca1275a7bc4\n",
      "Step 13/15 : RUN bash -c 'echo -e \"[server]\\nenableCORS = false\\nenableXsrfProtection = false\\nenableWebsocketCompression = false\\n\" > /root/.streamlit/config.toml'\n",
      " ---> Running in 6837898cc3fd\n",
      "Removing intermediate container 6837898cc3fd\n",
      " ---> 10c525a6a54e\n",
      "Step 14/15 : EXPOSE 80\n",
      " ---> Running in 838d0f97bd6b\n",
      "Removing intermediate container 838d0f97bd6b\n",
      " ---> cc46b8a55739\n",
      "Step 15/15 : ENTRYPOINT [ \"streamlit\", \"run\", \"src/app.py\",              \"--browser.serverAddress\", \"0.0.0.0\",              \"--server.enableCORS\", \"false\",              \"--server.enableXsrfProtection\", \"false\",             \"--server.enableWebsocketCompression\", \"false\",             \"--server.port\", \"80\"]\n",
      " ---> Running in 035a9d99fb08\n",
      "Removing intermediate container 035a9d99fb08\n",
      " ---> d5788bb54f59\n",
      "Successfully built d5788bb54f59\n",
      "Successfully tagged streamlit_gpt2_news_classifier_app:latest\n"
     ]
    }
   ],
   "source": [
    "!(cd streamlit_app && docker build -t {image_name} --build-arg SAGEMAKER_ENDPOINT_NAME={sagemaker_endpoint_name} .)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "fbaeb16e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REPOSITORY   TAG                        IMAGE ID       CREATED       SIZE\n",
      "streamlit    gpt2-news-classifier-app   ddd0674e030c   6 hours ago   737MB\n",
      "python       3.7.4-slim-stretch         fad2b9f06d3b   2 years ago   155MB\n"
     ]
    }
   ],
   "source": [
    "!docker images"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4450bc9e",
   "metadata": {},
   "source": [
    "With our Docker image built, we can now choose a local port that will be used by the dashboard server. Although the Streamlit server will be running on port 80 inside the container (defined in the `Dockerfile`), we can map this to a different local port on our Amazon SageMaker Notebook Instance. We'll use port 8501 in our example, which is the Streamlit default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "1e20a81d",
   "metadata": {},
   "outputs": [],
   "source": [
    "port = 8501"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4118d116",
   "metadata": {},
   "source": [
    "We're now ready to start the Docker container. We have a defined a utility function, called `get_docker_run_command`, that can be used to construct the correct `docker run` command. It handles a number of different things including:\n",
    "\n",
    "- port forwarding: access the server running inside the Docker container.\n",
    "- local directory mounts: edit source files without having to rebuild the Docker container.\n",
    "- debug modes: change the verbosity of error messages.\n",
    "- permissions: pass the IAM role from the SageMaker Notebook Instance to the Docker container.\n",
    "\n",
    "After running the cell below, you should click on the `Dashboard URL` link that appears at the top, rather than the `URL` output by the Streamlit server. All local ports must be accessed via the [Jupyter Server Proxy](https://github.com/jupyterhub/jupyter-server-proxy) at `https://{notebook-url}/proxy/{port}`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "566f5ee8",
   "metadata": {},
   "outputs": [],
   "source": [
    "url = utils.get_dashboard_url(port)\n",
    "command = utils.get_docker_run_command(port, image_name, local_dir_mount='./streamlit_app', debug=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "139a4f33",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://gpt2-news-classifier-2.notebook.us-east-1.sagemaker.aws/proxy/8501/\n"
     ]
    }
   ],
   "source": [
    "print(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "f291de23",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "docker run -p 8501:80 \\\n",
      "-v /home/ec2-user/SageMaker/streamlit_app:/usr/src/app/src \\\n",
      "--env AWS_DEFAULT_REGION=us-east-1 \\\n",
      "--env AWS_ACCESS_KEY_ID=ASIAWMTFPEZHKPYPY6PH \\\n",
      "--env AWS_SECRET_ACCESS_KEY=1KLhtxlzmJ2Q00q9oeB00rDbVy3AfJo0MCmI+NML \\\n",
      "--env AWS_SESSION_TOKEN=IQoJb3JpZ2luX2VjELb//////////wEaCXVzLWVhc3QtMSJHMEUCIQDLGkHPFOGAERxtP6sPNqNyXSD2KJYe6XVAy6OUy8rwRwIgY4IDynnUdYdnSFF/YqDH/2u+TW1XfFwZaXhC811DTfkqvQII3///////////ARAAGgw0MzkzNzMyMTEyMTQiDA8yHM9EN5c5pHR9pSqRAoERxzU0V17qUiVG3D1XJb4glHjnMIIbsrOn/WIPUYHcA3V9QAbW6zFIvViiTDKSUpSF1vWHA6AuL9K4ZfVjqI+Pn/r7l+WRLbY3AIaA+ljeWimtmskDc8cucNva1cBfbRB54FcTMWw9Leqj7YZYNULbqTg6FLEZfah3gpAPvqfKEtjxJe+darLX/FLpIPckIjwKdOjh6OyILLlzEgkLnbVR18OAnku9LUk7TRKnGffqGE+TK8yBnuUUJ/32EcQm8KRDLmjgKdwpE6eszQhXojnguAi63FUMx5w6+EUCJl7y9Q244f6vTV844kc8sfjmuS0BvfD2WuybpFZYZwkVaYMwe3masOBBCcqn4zsry7A58DD9wNGPBjqTATD3/QiYGq/NRKVO0BQR0+Tt01WduJmZsfOlN5kg1wQOhfo+TDx+J3e2HCx1XvZFyU98ZuIJCes3IgRiwD2MHGUMnwF6UlUsTNHWngnCAuPPBcaJEMEsMixDlIzFbXPamiQpF+R1OgyTv8SpZNqt3fFxvTD2PRoOXyGJn83v0mDBPtjP5lvMqwd/5/cVWi2d6kTf3Q== \\\n",
      "--env DASHBOARD_DEBUG=true \\\n",
      "streamlit:gpt2-news-classifier-app\n"
     ]
    }
   ],
   "source": [
    "print(command)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6edb6b91",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "APP URL: https://gpt2-news-classifier.notebook.us-east-1.sagemaker.aws/proxy/8501/\n",
      "\n",
      "  You can now view your Streamlit app in your browser.\n",
      "\n",
      "  URL: http://0.0.0.0:80\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!echo APP URL: {url}\n",
    "!{command}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1df7b9f",
   "metadata": {},
   "source": [
    "You should be able to try our GPT-2 News Classifier in this web app!\n",
    "\n",
    "When you're finished developing the Streamlit app, interrupt the Jupyter kernel by clicking on the stop button or clicking 'Kernel' -> 'Interrupt'."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26c3101e",
   "metadata": {},
   "source": [
    "## Clean Up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4e7d584",
   "metadata": {},
   "source": [
    "After you have finished with this example, make sure to delete the prediction endpoint to release the instance(s) associated with it and avoid being over-charged!\n",
    "\n",
    "**Caution**: You also need to manually delete any extra resources that you may have created in these notebooks. Some examples include, extra Amazon S3 buckets (to the solution's default bucket), extra Amazon SageMaker endpoints (using a custom name), and extra Amazon ECR repositories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "d78324fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# delete endpoint to release instance\n",
    "# sagemaker_session.delete_endpoint(\n",
    "#     endpoint_name = sagemaker_endpoint_name\n",
    "# )\n",
    "# sagemaker_session.delete_model(\n",
    "#     model_name=sagemaker_endpoint_name\n",
    "# )"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_latest_p36",
   "language": "python",
   "name": "conda_pytorch_latest_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}