bert_finetune/bert_pytorch_finetuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finetuning PyTorch BERT\n",
    "\n",
    "Pre-trained language representations have been shown to improve many downstream NLP tasks such as question answering, and natural language inference. Devlin et al proposed [BERT](https://arxiv.org/abs/1706.03762) (Bidirectional Encoder Representations from Transformers), which fine-tunes deep bidirectional representations on a wide range of tasks with minimal task-specific parameters, and obtained state-of-the-art results.\n",
    "\n",
    "In this tutorial, we will focus on adapting the BERT model for the question answering task on the SQuAD dataset. Specifically, we will:\n",
    "\n",
    "- learn how to pre-process the [SQuAD dataset](https://rajpurkar.github.io/SQuAD-explorer/) to leverage the learnt representation in BERT;\n",
    "- adapt the BERT model to the question answering task;\n",
    "- load a pretrained BERT model and finetune it;\n",
    "- inference on the SQuAD test dataset.\n",
    "\n",
    "[**NGC**](https://www.nvidia.com/en-us/gpu-cloud/) is the hub for GPU-optimized software for deep learning and high-performance computing (HPC) that takes care of all the plumbing, so that researchers can focus on building solutions, gathering insights, and delivering business value. We will be finetuning on the pretrained BERT model provided by NGC.\n",
    "\n",
    "Now let's get started and we first import the packages for this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex.\n"
     ]
    }
   ],
   "source": [
    "import collections, datetime, json, math, os, tarfile, time\n",
    "from io import StringIO\n",
    "\n",
    "import boto3\n",
    "import numpy as np\n",
    "import torch\n",
    "import sagemaker\n",
    "from sagemaker.pytorch import estimator, PyTorch, PyTorchModel, PyTorchPredictor\n",
    "from sagemaker.utils import name_from_base\n",
    "\n",
    "from file_utils import PYTORCH_PRETRAINED_BERT_CACHE\n",
    "from helper_funcs import *\n",
    "from modeling import BertForQuestionAnswering, BertConfig, WEIGHTS_NAME, CONFIG_NAME\n",
    "from tokenization import (BasicTokenizer, BertTokenizer, whitespace_tokenize)\n",
    "from types import SimpleNamespace\n",
    "\n",
    "sagemaker_session = sagemaker.Session()\n",
    "bucket = sagemaker_session.default_bucket() # can replace with your own S3 bucket\n",
    "\n",
    "with open('s3_bucket.txt','w') as f:\n",
    "    f.write(f's3://{bucket}')\n",
    "with open('hyperparameters.json', 'r') as f:\n",
    "    params = json.load(f)\n",
    "params['save_to_s3'] = bucket\n",
    "with open('hyperparameters.json', 'w') as f:\n",
    "    json.dump(params, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Building and training the BERT model from scratch is expensive, most of the time we only need to finetune on the pretrained model. To reuse the pretrained parameters from NGC, we first download the `bert_base` model. The downloading might take a little bit time to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.12 s, sys: 176 ms, total: 1.3 s\n",
      "Wall time: 1min 36s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "!wget -q https://api.ngc.nvidia.com/v2/models/nvidia/bert_base_pyt_amp_ckpt_pretraining_lamb/versions/1/files/bert_base.pt -O bert_base.pt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## I. Create training docker container\n",
    "\n",
    "### Step 1. Build the docker container\n",
    "\n",
    "To build a end-to-end training environment, we first initiate a docker container. The container will install packages, setup PATH, copy the scripts (such as `train` and `serve`) and BERT base model to the container. The explicit steps are writing in `Dockerfile` in the same directory as this notebook.\n",
    "\n",
    "Now we create a custom docker container based on the NGC Bert container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "build_docker_start = time.time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Login Succeeded\n",
      "Sending build context to Docker daemon  1.941GB\r",
      "\r\n",
      "Step 1/15 : ARG FROM_IMAGE_NAME=nvcr.io/nvidia/pytorch:19.12-py3\n",
      "Step 2/15 : FROM ${FROM_IMAGE_NAME}\n",
      " ---> be021446e08c\n",
      "Step 3/15 : RUN apt-get update && apt-get install -y pbzip2 pv bzip2 cabextract nginx wget\n",
      " ---> Using cache\n",
      " ---> 22b1d5b1aec7\n",
      "Step 4/15 : ENV BERT_PREP_WORKING_DIR /workspace/bert/data\n",
      " ---> Using cache\n",
      " ---> 24fdfd86f5f2\n",
      "Step 5/15 : WORKDIR /workspace\n",
      " ---> Using cache\n",
      " ---> 7507b34aa969\n",
      "Step 6/15 : RUN git clone https://github.com/attardi/wikiextractor.git\n",
      " ---> Using cache\n",
      " ---> 40cf9d0d4d6e\n",
      "Step 7/15 : RUN git clone https://github.com/soskek/bookcorpus.git\n",
      " ---> Using cache\n",
      " ---> 74dea5f19474\n",
      "Step 8/15 : WORKDIR /workspace/bert\n",
      " ---> Using cache\n",
      " ---> e2923624a6a9\n",
      "Step 9/15 : RUN pip install --upgrade --no-cache-dir pip  && pip install --no-cache-dir  gevent flask pathlib gunicorn tqdm boto3 requests six ipdb h5py html2text nltk progressbar onnxruntime git+https://github.com/NVIDIA/dllogger\n",
      " ---> Using cache\n",
      " ---> 6d9f905f8951\n",
      "Step 10/15 : RUN apt-get install -y iputils-ping\n",
      " ---> Using cache\n",
      " ---> fae7ae19522e\n",
      "Step 11/15 : ENV PATH=\"/workspace/bert:${PATH}\"\n",
      " ---> Using cache\n",
      " ---> d0fa5c7fb3ab\n",
      "Step 12/15 : RUN mkdir /opt/ml && mkdir /opt/ml/input && mkdir /opt/ml/output && mkdir /opt/ml/model && mkdir /opt/ml/input/config\n",
      " ---> Using cache\n",
      " ---> 178662ffd298\n",
      "Step 13/15 : COPY . .\n",
      " ---> e31aee04e721\n",
      "Step 14/15 : RUN chmod +x /workspace/bert/train\n",
      " ---> Running in b0ff818dc51d\n",
      "Removing intermediate container b0ff818dc51d\n",
      " ---> c48733f7303d\n",
      "Step 15/15 : RUN chmod +x /workspace/bert/serve\n",
      " ---> Running in 9f1a94d33c6d\n",
      "Removing intermediate container 9f1a94d33c6d\n",
      " ---> fa62975c7b84\n",
      "Successfully built fa62975c7b84\n",
      "Successfully tagged bert-train-kdd:latest\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING! Using --password via the CLI is insecure. Use --password-stdin.\n",
      "WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.\n",
      "Configure a credential helper to remove this warning. See\n",
      "https://docs.docker.com/engine/reference/commandline/login/#credentials-store\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "\n",
    "# The name of our algorithm\n",
    "algorithm_name=bert-train-kdd\n",
    "\n",
    "chmod +x train\n",
    "chmod +x serve\n",
    "\n",
    "account=$(aws sts get-caller-identity --query Account --output text)\n",
    "\n",
    "# Get the region defined in the current configuration (default to us-west-2 if none defined)\n",
    "region=$(aws configure get region)\n",
    "region=${region:-us-east-1}\n",
    "\n",
    "fullname=\"${account}.dkr.ecr.${region}.amazonaws.com/${algorithm_name}:latest\"\n",
    "\n",
    "# If the repository doesn't exist in ECR, create it.\n",
    "aws ecr describe-repositories --repository-names \"${algorithm_name}\" > /dev/null 2>&1\n",
    "\n",
    "if [ $? -ne 0 ]\n",
    "then\n",
    "    aws ecr create-repository --repository-name \"${algorithm_name}\" > /dev/null\n",
    "fi\n",
    "\n",
    "# Get the login command from ECR and execute it directly\n",
    "$(aws ecr get-login --region ${region} --no-include-email)\n",
    "\n",
    "# Build the docker image locally with the image name and then push it to ECR\n",
    "# with the full name.\n",
    "docker build  -t ${algorithm_name} .\n",
    "docker tag ${algorithm_name} ${fullname}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time for building docker : 73.68929195404053 s\n"
     ]
    }
   ],
   "source": [
    "print(\"Time for building docker : {} s\".format(time.time() - build_docker_start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2. Edit ECR permission\n",
    "\n",
    "Once the built container is done, we can push it to the AWS Elastic Container Registry (ECR). However, for the security consideration, you well need to access ECR and change the permission. To do that, we first create the individual **unique** Json file similar as below:\n",
    "\n",
    "```{`json}\n",
    "{\n",
    "  \"Version\": \"2008-10-17\",\n",
    "  \"Statement\": [\n",
    "    {\n",
    "      \"Sid\": \"All-Allow\",\n",
    "      \"Effect\": \"Allow\",\n",
    "      \"Principal\": {\n",
    "        \"AWS\": [\n",
    "          \"arn:aws:iam::XXXXXXXXXXXXXXXXXXXXXXXXXX\"\n",
    "        ]\n",
    "      },\n",
    "      \"Action\": \"*\"\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we [**access this link**](https://console.aws.amazon.com/ecr/repositories/bert-train-kdd/permissions?region=us-east-1) and click `Edit Policy JSON`, then paste the above Json text and click `Save`.\n",
    "\n",
    "\n",
    "\n",
    "(**NOTE: You only need to replace** the sample Json's `arn:aws:iam` with your own as shown below:)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'arn:aws:iam::383827541835:role/service-role/AmazonSageMaker-ExecutionRole-20200409T103675'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "role = sagemaker.get_execution_role()\n",
    "role"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3. Push the docker container\n",
    "\n",
    "With the ECR permission, let's push our built docker container to ECR."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "push_docker_start = time.time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The push refers to repository [383827541835.dkr.ecr.us-east-1.amazonaws.com/bert-train-kdd]\n",
      "c2b8caee5adc: Preparing\n",
      "abd8764d9cee: Preparing\n",
      "28a690722ab3: Preparing\n",
      "7e55eb3ddb14: Preparing\n",
      "3a667acb45c0: Preparing\n",
      "19a858f1c4dd: Preparing\n",
      "3fac541413a1: Preparing\n",
      "d61b297ff92b: Preparing\n",
      "9b5db43f614f: Preparing\n",
      "3e96f70c3868: Preparing\n",
      "fddcdacefca4: Preparing\n",
      "6c9a5f2bcdc9: Preparing\n",
      "97f77c2bf551: Preparing\n",
      "0ad644802067: Preparing\n",
      "0988452d60ad: Preparing\n",
      "22b2247d543c: Preparing\n",
      "3fac541413a1: Waiting\n",
      "5130ccfce7b2: Preparing\n",
      "d61b297ff92b: Waiting\n",
      "005c189102b1: Preparing\n",
      "19a858f1c4dd: Waiting\n",
      "9b5db43f614f: Waiting\n",
      "b2541313126e: Preparing\n",
      "3e96f70c3868: Waiting\n",
      "7a631d1de8a8: Preparing\n",
      "563ea1e7989f: Preparing\n",
      "fddcdacefca4: Waiting\n",
      "05b737b70379: Preparing\n",
      "461c94146b25: Preparing\n",
      "6c9a5f2bcdc9: Waiting\n",
      "89f14d452cdc: Preparing\n",
      "221c639fb572: Preparing\n",
      "97f77c2bf551: Waiting\n",
      "094a55ed8561: Preparing\n",
      "0ad644802067: Waiting\n",
      "c727ce4f07f0: Preparing\n",
      "0988452d60ad: Waiting\n",
      "5f4f32dbd55d: Preparing\n",
      "577dd6013185: Preparing\n",
      "22b2247d543c: Waiting\n",
      "78c62c90c01c: Preparing\n",
      "6d5f1e49ad99: Preparing\n",
      "96a6eb08694f: Preparing\n",
      "5130ccfce7b2: Waiting\n",
      "b0404397b1f6: Preparing\n",
      "b2541313126e: Waiting\n",
      "c558708f95ac: Preparing\n",
      "7a631d1de8a8: Waiting\n",
      "c003f7d80d34: Preparing\n",
      "005c189102b1: Waiting\n",
      "9f34fb2ba40d: Preparing\n",
      "563ea1e7989f: Waiting\n",
      "a9268194e7cd: Preparing\n",
      "461c94146b25: Waiting\n",
      "8e893f1677ca: Preparing\n",
      "89f14d452cdc: Waiting\n",
      "cc0b30658be2: Preparing\n",
      "05b737b70379: Waiting\n",
      "77db1fc0f8a5: Preparing\n",
      "5cb994f962f2: Preparing\n",
      "221c639fb572: Waiting\n",
      "045b335db3d7: Preparing\n",
      "ffa33d177975: Preparing\n",
      "c7b9adf34c73: Preparing\n",
      "e385f28a9371: Preparing\n",
      "96a6eb08694f: Waiting\n",
      "c558708f95ac: Waiting\n",
      "aa98ab4614ff: Preparing\n",
      "aa42716fcc2f: Preparing\n",
      "b0404397b1f6: Waiting\n",
      "c003f7d80d34: Waiting\n",
      "2fcfd47c7b80: Preparing\n",
      "40f9283f52a3: Preparing\n",
      "c727ce4f07f0: Waiting\n",
      "577dd6013185: Waiting\n",
      "9f34fb2ba40d: Waiting\n",
      "21fa43656dc2: Preparing\n",
      "5f4f32dbd55d: Waiting\n",
      "519d10164fb4: Preparing\n",
      "a9268194e7cd: Waiting\n",
      "b4e68a0c3294: Preparing\n",
      "ffa33d177975: Waiting\n",
      "4fc18f064f08: Preparing\n",
      "8e893f1677ca: Waiting\n",
      "c7b9adf34c73: Waiting\n",
      "7a0c9aa36cb6: Preparing\n",
      "911e33e3bdb5: Preparing\n",
      "045b335db3d7: Waiting\n",
      "5cb994f962f2: Waiting\n",
      "a12854a1091b: Preparing\n",
      "77db1fc0f8a5: Waiting\n",
      "d489d7212cf0: Preparing\n",
      "aa98ab4614ff: Waiting\n",
      "d9d8412a92db: Preparing\n",
      "519d10164fb4: Waiting\n",
      "40f9283f52a3: Waiting\n",
      "aa42716fcc2f: Waiting\n",
      "21fa43656dc2: Waiting\n",
      "7cdd75a5beec: Preparing\n",
      "5f514c1078a9: Preparing\n",
      "52d0beb8ba85: Preparing\n",
      "b4e68a0c3294: Waiting\n",
      "7a0c9aa36cb6: Waiting\n",
      "4fc18f064f08: Waiting\n",
      "bef829be1d9b: Preparing\n",
      "47a60ca21bc2: Preparing\n",
      "911e33e3bdb5: Waiting\n",
      "43e75c4a74b7: Preparing\n",
      "d5a82cd15754: Preparing\n",
      "d489d7212cf0: Waiting\n",
      "e0b3afb09dc3: Preparing\n",
      "6c01b5a53aac: Preparing\n",
      "5f514c1078a9: Waiting\n",
      "2c6ac8e5063e: Preparing\n",
      "52d0beb8ba85: Waiting\n",
      "7cdd75a5beec: Waiting\n",
      "cc967c529ced: Preparing\n",
      "47a60ca21bc2: Waiting\n",
      "e0b3afb09dc3: Waiting\n",
      "43e75c4a74b7: Waiting\n",
      "d5a82cd15754: Waiting\n",
      "6c01b5a53aac: Waiting\n",
      "cc967c529ced: Waiting\n",
      "2c6ac8e5063e: Waiting\n",
      "094a55ed8561: Waiting\n",
      "e385f28a9371: Waiting\n",
      "c2b8caee5adc: Pushed\n",
      "abd8764d9cee: Pushed\n",
      "7e55eb3ddb14: Pushed\n",
      "3fac541413a1: Pushed\n",
      "3a667acb45c0: Pushed\n",
      "9b5db43f614f: Pushed\n",
      "d61b297ff92b: Pushed\n",
      "6c9a5f2bcdc9: Pushed\n",
      "fddcdacefca4: Pushed\n",
      "97f77c2bf551: Pushed\n",
      "19a858f1c4dd: Pushed\n",
      "0988452d60ad: Pushed\n",
      "5130ccfce7b2: Pushed\n",
      "005c189102b1: Pushed\n",
      "b2541313126e: Pushed\n",
      "3e96f70c3868: Pushed\n",
      "22b2247d543c: Pushed\n",
      "0ad644802067: Pushed\n",
      "05b737b70379: Pushed\n",
      "461c94146b25: Pushed\n",
      "563ea1e7989f: Pushed\n",
      "89f14d452cdc: Pushed\n",
      "c727ce4f07f0: Pushed\n",
      "094a55ed8561: Pushed\n",
      "577dd6013185: Pushed\n",
      "78c62c90c01c: Pushed\n",
      "7a631d1de8a8: Pushed\n",
      "6d5f1e49ad99: Pushed\n",
      "b0404397b1f6: Pushed\n",
      "c558708f95ac: Pushed\n",
      "5f4f32dbd55d: Pushed\n",
      "9f34fb2ba40d: Pushed\n",
      "a9268194e7cd: Pushed\n",
      "8e893f1677ca: Pushed\n",
      "cc0b30658be2: Pushed\n",
      "c003f7d80d34: Pushed\n",
      "77db1fc0f8a5: Pushed\n",
      "045b335db3d7: Pushed\n",
      "96a6eb08694f: Pushed\n",
      "ffa33d177975: Pushed\n",
      "e385f28a9371: Pushed\n",
      "aa98ab4614ff: Pushed\n",
      "aa42716fcc2f: Pushed\n",
      "2fcfd47c7b80: Pushed\n",
      "40f9283f52a3: Pushed\n",
      "21fa43656dc2: Pushed\n",
      "519d10164fb4: Pushed\n",
      "b4e68a0c3294: Pushed\n",
      "4fc18f064f08: Pushed\n",
      "7a0c9aa36cb6: Pushed\n",
      "221c639fb572: Pushed\n",
      "a12854a1091b: Pushed\n",
      "d489d7212cf0: Pushed\n",
      "d9d8412a92db: Pushed\n",
      "7cdd75a5beec: Pushed\n",
      "5f514c1078a9: Pushed\n",
      "52d0beb8ba85: Pushed\n",
      "bef829be1d9b: Pushed\n",
      "47a60ca21bc2: Pushed\n",
      "c7b9adf34c73: Pushed\n",
      "28a690722ab3: Pushed\n",
      "e0b3afb09dc3: Pushed\n",
      "6c01b5a53aac: Pushed\n",
      "2c6ac8e5063e: Pushed\n",
      "d5a82cd15754: Pushed\n",
      "cc967c529ced: Pushed\n",
      "911e33e3bdb5: Pushed\n",
      "5cb994f962f2: Pushed\n",
      "43e75c4a74b7: Pushed\n",
      "latest: digest: sha256:43bd5239d29570cf62f4504133e4f41612abf32e6b008c69568234d1a6a631f7 size: 14805\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "\n",
    "algorithm_name=bert-train-kdd\n",
    "account=$(aws sts get-caller-identity --query Account --output text)\n",
    "region=$(aws configure get region)\n",
    "# region=${region:-us-east-1}\n",
    "fullname=\"${account}.dkr.ecr.${region}.amazonaws.com/${algorithm_name}:latest\"\n",
    "\n",
    "docker push ${fullname}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time for pushing docker : 181.9604253768921 s\n"
     ]
    }
   ],
   "source": [
    "print(\"Time for pushing docker : {} s\".format(time.time() - push_docker_start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## II. Instantiate the model\n",
    "\n",
    "It's the time to instantiate our BERT model, we first specify our hyperparameters for training as below. Note that here `save_to_s3` is the place where the finetuned model is going to be living in, i.e., \"s3://{`bucket`}/model.tar.gz\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set our hyperparameters\n",
    "hyperparameters = {'bert_model': 'bert-base-uncased',  \n",
    "                   'vocab_file': '/workspace/bert/data/bert_vocab.txt',\n",
    "                   'config_file':'/workspace/bert/bert_config.json', \n",
    "                   'output_dir': '/opt/ml/model',\n",
    "                   'train_file': '/workspace/bert/data/squad/v1.1/train-v1.1.json',\n",
    "                   'num_gpus':4, \n",
    "                   'num_train_epochs': 1, \n",
    "                   'train_batch_size':16, \n",
    "                   'max_seq_length':512, \n",
    "                   'doc_stride':128, \n",
    "                   'seed':1,\n",
    "                   'learning_rate':3e-5,\n",
    "                   'save_to_s3':bucket}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Once we have set our hyperparameters, we will instantiate a Sagemaker Estimator `PyTorch` that we will use to run our training job. Here are some other parameters we need to specify:\n",
    "\n",
    "- The GPUs (or `train_instance_type`) we are going to use will be the AWS [`ml.p3.8xlarge` instance](https://aws.amazon.com/sagemaker/pricing/instance-types/). It contains four *V100* volta GPUs, making them ideal for this heavy duty deep learning training. \n",
    "\n",
    "\n",
    "- We specify the Docker image we just pushed to ECR with `image_name`. Our Docker container has two commands, train and serve. When we instantiate a training job, behind the scenes Sagemaker is running our Docker container and telling it to run the train command.\n",
    "\n",
    "\n",
    "- We illustrate an entrypoint algorithm by `entry_point`. The entrypoint file will give instructions for what operations our container should perform when it starts up. \n",
    "\n",
    "\n",
    "- We use AWS Deep Learning Containers for PyTorch 1.4.0 for the `framework_version`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.\n"
     ]
    }
   ],
   "source": [
    "account=!aws sts get-caller-identity --query Account --output text\n",
    "region=!aws configure get region\n",
    "algoname = 'bert-train-kdd'\n",
    "\n",
    "# instantiate model\n",
    "torch_model = PyTorch(role=role,\n",
    "                      train_instance_count=1,\n",
    "                      train_instance_type='ml.p3.8xlarge',\n",
    "                      entry_point='transform_script.py',\n",
    "                      image_name=\"{}.dkr.ecr.{}.amazonaws.com/{}\".format(account[0], region[0], algoname),\n",
    "                      framework_version='1.4.0',\n",
    "                      hyperparameters=hyperparameters,\n",
    "#                       model_dir=f's3://{bucket}/model.tar.gz'\n",
    "                     )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## III. Fine-tune the model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you use an [`ml.p3.8xlarge` instance](https://aws.amazon.com/sagemaker/pricing/instance-types/) with 4 GPUs and a batch size of 16, this process will take around 15 minutes to complete for this particular finetuning task with 1 epoch. It's recommended to use at minimum use a training instance with 4 GPUs, although you will likely get better performance with one of the `ml.p3.16xlarge` or `ml.p3dn.24xlarge` instances. \n",
    "\n",
    "Let's start the training!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-08-23 20:41:16 Starting - Starting the training job...\n",
      "2020-08-23 20:41:18 Starting - Launching requested ML instances......\n",
      "2020-08-23 20:42:30 Starting - Preparing the instances for training......\n",
      "2020-08-23 20:43:38 Downloading - Downloading input data\n",
      "2020-08-23 20:43:38 Training - Downloading the training image.....................\n",
      "2020-08-23 20:47:09 Training - Training image download completed. Training in progress.\u001b[34m=============\u001b[0m\n",
      "\u001b[34m== PyTorch ==\u001b[0m\n",
      "\u001b[34m=============\n",
      "\u001b[0m\n",
      "\u001b[34mNVIDIA Release 19.12 (build 9142930)\u001b[0m\n",
      "\u001b[34mPyTorch Version 1.4.0a0+a5b4d78\n",
      "\u001b[0m\n",
      "\u001b[34mContainer image Copyright (c) 2019, NVIDIA CORPORATION.  All rights reserved.\n",
      "\u001b[0m\n",
      "\u001b[34mCopyright (c) 2014-2019 Facebook Inc.\u001b[0m\n",
      "\u001b[34mCopyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)\u001b[0m\n",
      "\u001b[34mCopyright (c) 2012-2014 Deepmind Technologies    (Koray Kavukcuoglu)\u001b[0m\n",
      "\u001b[34mCopyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)\u001b[0m\n",
      "\u001b[34mCopyright (c) 2011-2013 NYU                      (Clement Farabet)\u001b[0m\n",
      "\u001b[34mCopyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)\u001b[0m\n",
      "\u001b[34mCopyright (c) 2006      Idiap Research Institute (Samy Bengio)\u001b[0m\n",
      "\u001b[34mCopyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)\u001b[0m\n",
      "\u001b[34mCopyright (c) 2015      Google Inc.\u001b[0m\n",
      "\u001b[34mCopyright (c) 2015      Yangqing Jia\u001b[0m\n",
      "\u001b[34mCopyright (c) 2013-2016 The Caffe contributors\u001b[0m\n",
      "\u001b[34mAll rights reserved.\n",
      "\u001b[0m\n",
      "\u001b[34mVarious files include modifications (c) NVIDIA CORPORATION.  All rights reserved.\u001b[0m\n",
      "\u001b[34mNVIDIA modifications are covered by the license terms that apply to the underlying project or file.\u001b[0m\n",
      "\u001b[34mNOTE: MOFED driver for multi-node communication was not detected.\n",
      "      Multi-node communication performance may be reduced.\u001b[0m\n",
      "\u001b[34mDownloading dataset for squad...\u001b[0m\n",
      "\u001b[34mmkdir: cannot create directory ‘v1.1’: File exists\u001b[0m\n",
      "\u001b[34m--2020-08-23 20:47:12--  https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json\u001b[0m\n",
      "\u001b[34mResolving rajpurkar.github.io (rajpurkar.github.io)... 185.199.109.153, 185.199.110.153, 185.199.111.153, ...\u001b[0m\n",
      "\u001b[34mConnecting to rajpurkar.github.io (rajpurkar.github.io)|185.199.109.153|:443... connected.\u001b[0m\n",
      "\u001b[34mHTTP request sent, awaiting response... 200 OK\u001b[0m\n",
      "\u001b[34mLength: 30288272 (29M) [application/json]\u001b[0m\n",
      "\u001b[34mSaving to: ‘v1.1/train-v1.1.json’\n",
      "\n",
      "     0K .......... .......... .......... .......... ..........  0% 20.3M 1s\n",
      "    50K .......... .......... .......... .......... ..........  0% 18.8M 1s\n",
      "   100K .......... .......... .......... .......... ..........  0% 22.2M 1s\n",
      "   150K .......... .......... .......... .......... ..........  0% 26.7M 1s\n",
      "   200K .......... .......... .......... .......... ..........  0% 28.1M 1s\n",
      "   250K .......... .......... .......... .......... ..........  1% 30.6M 1s\n",
      "   300K .......... .......... .......... .......... ..........  1% 36.6M 1s\n",
      "   350K .......... .......... .......... .......... ..........  1% 33.6M 1s\n",
      "   400K .......... .......... .......... .......... ..........  1% 28.1M 1s\n",
      "   450K .......... .......... .......... .......... ..........  1% 40.6M 1s\n",
      "   500K .......... .......... .......... .......... ..........  1% 32.7M 1s\n",
      "   550K .......... .......... .......... .......... ..........  2% 35.1M 1s\n",
      "   600K .......... .......... .......... .......... ..........  2% 42.2M 1s\n",
      "   650K .......... .......... .......... .......... ..........  2% 36.8M 1s\n",
      "   700K .......... .......... .......... .......... ..........  2% 99.6M 1s\n",
      "   750K .......... .......... .......... .......... ..........  2% 43.1M 1s\n",
      "   800K .......... .......... .......... .......... ..........  2% 50.4M 1s\n",
      "   850K .......... .......... .......... .......... ..........  3% 59.2M 1s\n",
      "   900K .......... .......... .......... .......... ..........  3% 42.3M 1s\n",
      "   950K .......... .......... .......... .......... ..........  3% 45.4M 1s\n",
      "  1000K .......... .......... .......... .......... ..........  3% 59.1M 1s\n",
      "  1050K .......... .......... .......... .......... ..........  3% 79.1M 1s\n",
      "  1100K .......... .......... .......... .......... ..........  3% 52.7M 1s\n",
      "  1150K .......... .......... .......... .......... ..........  4% 47.3M 1s\n",
      "  1200K .......... .......... .......... .......... ..........  4% 76.4M 1s\n",
      "  1250K .......... .......... .......... .......... ..........  4% 49.6M 1s\n",
      "  1300K .......... .......... .......... .......... ..........  4% 75.0M 1s\n",
      "  1350K .......... .......... .......... .......... ..........  4% 59.6M 1s\n",
      "  1400K .......... .......... .......... .......... ..........  4% 53.6M 1s\n",
      "  1450K .......... .......... .......... .......... ..........  5%  123M 1s\n",
      "  1500K .......... .......... .......... .......... ..........  5% 43.6M 1s\n",
      "  1550K .......... .......... .......... .......... ..........  5%  162M 1s\n",
      "  1600K .......... .......... .......... .......... ..........  5% 67.9M 1s\n",
      "  1650K .......... .......... .......... .......... ..........  5% 97.5M 1s\n",
      "  1700K .......... .......... .......... .......... ..........  5% 42.9M 1s\n",
      "  1750K .......... .......... .......... .......... ..........  6%  129M 1s\n",
      "  1800K .......... .......... .......... .......... ..........  6% 55.1M 1s\n",
      "  1850K .......... .......... .......... .......... ..........  6%  101M 1s\n",
      "  1900K .......... .......... .......... .......... ..........  6% 52.1M 1s\n",
      "  1950K .......... .......... .......... .......... ..........  6%  157M 1s\n",
      "  2000K .......... .......... .......... .......... ..........  6%  103M 1s\n",
      "  2050K .......... .......... .......... .......... ..........  7% 59.7M 1s\n",
      "  2100K .......... .......... .......... .......... ..........  7% 89.7M 1s\n",
      "  2150K .......... .......... .......... .......... ..........  7% 83.1M 1s\n",
      "  2200K .......... .......... .......... .......... ..........  7%  145M 1s\n",
      "  2250K .......... .......... .......... .......... ..........  7% 61.5M 1s\n",
      "  2300K .......... .......... .......... .......... ..........  7% 99.8M 1s\n",
      "  2350K .......... .......... .......... .......... ..........  8% 63.2M 1s\n",
      "  2400K .......... .......... .......... .......... ..........  8%  195M 1s\n",
      "  2450K .......... .......... .......... .......... ..........  8%  142M 1s\n",
      "  2500K .......... .......... .......... .......... ..........  8% 67.1M 1s\n",
      "  2550K .......... .......... .......... .......... ..........  8% 95.2M 1s\n",
      "  2600K .......... .......... .......... .......... ..........  8% 83.0M 1s\n",
      "  2650K .......... .......... .......... .......... ..........  9%  120M 1s\n",
      "  2700K .......... .......... .......... .......... ..........  9%  102M 1s\n",
      "  2750K .......... .......... .......... .......... ..........  9%  197M 0s\n",
      "  2800K .......... .......... .......... .......... ..........  9% 92.8M 0s\n",
      "  2850K .......... .......... .......... .......... ..........  9% 71.6M 0s\n",
      "  2900K .......... .......... .......... .......... ..........  9%  126M 0s\n",
      "  2950K .......... .......... .......... .......... .......... 10% 55.2M 0s\n",
      "  3000K .......... .......... .......... .......... .......... 10%  117M 0s\n",
      "  3050K .......... .......... .......... .......... .......... 10%  209M 0s\n",
      "  3100K .......... .......... .......... .......... .......... 10%  218M 0s\n",
      "  3150K .......... .......... .......... .......... .......... 10% 97.0M 0s\n",
      "  3200K .......... .......... .......... .......... .......... 10%  139M 0s\n",
      "  3250K .......... .......... .......... .......... .......... 11%  148M 0s\n",
      "  3300K .......... .......... .......... .......... .......... 11% 69.9M 0s\n",
      "  3350K .......... .......... .......... .......... .......... 11% 98.6M 0s\n",
      "  3400K .......... .......... .......... .......... .......... 11%  123M 0s\n",
      "  3450K .......... .......... .......... .......... .......... 11%  191M 0s\n",
      "  3500K .......... .......... .......... .......... .......... 12%  121M 0s\n",
      "  3550K .......... .......... .......... .......... .......... 12%  130M 0s\n",
      "  3600K .......... .......... .......... .......... .......... 12%  116M 0s\n",
      "  3650K .......... .......... .......... .......... .......... 12%  170M 0s\n",
      "  3700K .......... .......... .......... .......... .......... 12% 82.8M 0s\n",
      "  3750K .......... .......... .......... .......... .......... 12%  195M 0s\n",
      "  3800K .......... .......... .......... .......... .......... 13%  127M 0s\n",
      "  3850K .......... .......... .......... .......... .......... 13%  201M 0s\n",
      "  3900K .......... .......... .......... .......... .......... 13%  102M 0s\n",
      "  3950K .......... .......... .......... .......... .......... 13%  169M 0s\n",
      "  4000K .......... .......... .......... .......... .......... 13%  125M 0s\n",
      "  4050K .......... .......... .......... .......... .......... 13%  138M 0s\n",
      "  4100K .......... .......... .......... .......... .......... 14% 99.0M 0s\n",
      "  4150K .......... .......... .......... .......... .......... 14%  115M 0s\n",
      "  4200K .......... .......... .......... .......... .......... 14%  132M 0s\n",
      "  4250K .......... .......... .......... .......... .......... 14%  207M 0s\n",
      "  4300K .......... .......... .......... .......... .......... 14%  113M 0s\n",
      "  4350K .......... .......... .......... .......... .......... 14%  129M 0s\n",
      "  4400K .......... .......... .......... .......... .......... 15%  147M 0s\n",
      "  4450K .......... .......... .......... .......... .......... 15%  123M 0s\n",
      "  4500K .......... .......... .......... .......... .......... 15% 62.0M 0s\n",
      "  4550K .......... .......... .......... .......... .......... 15%  170M 0s\n",
      "  4600K .......... .......... .......... .......... .......... 15%  118M 0s\n",
      "  4650K .......... .......... .......... .......... .......... 15%  179M 0s\n",
      "  4700K .......... .......... .......... .......... .......... 16%  123M 0s\n",
      "  4750K .......... .......... .......... .......... .......... 16%  204M 0s\n",
      "  4800K .......... .......... .......... .......... .......... 16%  165M 0s\n",
      "  4850K .......... .......... .......... .......... .......... 16%  174M 0s\n",
      "  4900K .......... .......... .......... .......... .......... 16%  106M 0s\n",
      "  4950K .......... .......... .......... .......... .......... 16% 95.6M 0s\n",
      "  5000K .......... .......... .......... .......... .......... 17%  189M 0s\n",
      "  5050K .......... .......... .......... .......... .......... 17%  103M 0s\n",
      "  5100K .......... .......... .......... .......... .......... 17%  119M 0s\n",
      "  5150K .......... .......... .......... .......... .......... 17%  224M 0s\n",
      "  5200K .......... .......... .......... .......... .......... 17%  117M 0s\n",
      "  5250K .......... .......... .......... .......... .......... 17%  181M 0s\n",
      "  5300K .......... .......... .......... .......... .......... 18%  119M 0s\n",
      "  5350K .......... .......... .......... .......... .......... 18%  149M 0s\n",
      "  5400K .......... .......... .......... .......... .......... 18%  144M 0s\n",
      "  5450K .......... .......... .......... .......... .......... 18%  130M 0s\n",
      "  5500K .......... .......... .......... .......... .......... 18%  113M 0s\n",
      "  5550K .......... .......... .......... .......... .......... 18% 89.3M 0s\n",
      "  5600K .......... .......... .......... .......... .......... 19%  113M 0s\n",
      "  5650K .......... .......... .......... .......... .......... 19%  240M 0s\n",
      "  5700K .......... .......... .......... .......... .......... 19%  132M 0s\n",
      "  5750K .......... .......... .......... .......... .......... 19%  198M 0s\n",
      "  5800K .......... .......... .......... .......... .......... 19%  131M 0s\n",
      "  5850K .......... .......... .......... .......... .......... 19%  177M 0s\n",
      "  5900K .......... .......... .......... .......... .......... 20%  111M 0s\n",
      "  5950K .......... .......... .......... .......... .......... 20%  178M 0s\n",
      "  6000K .......... .......... .......... .......... .......... 20% 83.4M 0s\n",
      "  6050K .......... .......... .......... .......... .......... 20% 79.5M 0s\n",
      "  6100K .......... .......... .......... .......... .......... 20%  109M 0s\n",
      "  6150K .......... .......... .......... .......... .......... 20%  101M 0s\n",
      "  6200K .......... .......... .......... .......... .......... 21%  177M 0s\n",
      "  6250K .......... .......... .......... .......... .......... 21%  197M 0s\n",
      "  6300K .......... .......... .......... .......... .......... 21%  107M 0s\n",
      "  6350K .......... .......... .......... .......... .......... 21%  155M 0s\n",
      "  6400K .......... .......... .......... .......... .......... 21%  118M 0s\n",
      "  6450K .......... .......... .......... .......... .......... 21% 97.7M 0s\n",
      "  6500K .......... .......... .......... .......... .......... 22%  122M 0s\n",
      "  6550K .......... .......... .......... .......... .......... 22%  111M 0s\n",
      "  6600K .......... .......... .......... .......... .......... 22%  184M 0s\n",
      "  6650K .......... .......... .......... .......... .......... 22%  187M 0s\n",
      "  6700K .......... .......... .......... .......... .......... 22% 79.4M 0s\n",
      "  6750K .......... .......... .......... .......... .......... 22%  141M 0s\n",
      "  6800K .......... .......... .......... .......... .......... 23%  126M 0s\n",
      "  6850K .......... .......... .......... .......... .......... 23%  114M 0s\n",
      "  6900K .......... .......... .......... .......... .......... 23%  118M 0s\n",
      "  6950K .......... .......... .......... .......... .......... 23%  132M 0s\n",
      "  7000K .......... .......... .......... .......... .......... 23%  113M 0s\n",
      "  7050K .......... .......... .......... .......... .......... 24%  110M 0s\n",
      "  7100K .......... .......... .......... .......... .......... 24%  104M 0s\n",
      "  7150K .......... .......... .......... .......... .......... 24%  112M 0s\n",
      "  7200K .......... .......... .......... .......... .......... 24%  105M 0s\n",
      "  7250K .......... .......... .......... .......... .......... 24%  185M 0s\n",
      "  7300K .......... .......... .......... .......... .......... 24%  204M 0s\n",
      "  7350K .......... .......... .......... .......... .......... 25%  203M 0s\n",
      "  7400K .......... .......... .......... .......... .......... 25%  111M 0s\n",
      "  7450K .......... .......... .......... .......... .......... 25%  204M 0s\n",
      "  7500K .......... .......... .......... .......... .......... 25%  209M 0s\n",
      "  7550K .......... .......... .......... .......... .......... 25% 55.1M 0s\n",
      "  7600K .......... .......... .......... .......... .......... 25%  202M 0s\n",
      "  7650K .......... .......... .......... .......... .......... 26%  138M 0s\n",
      "  7700K .......... .......... .......... .......... .......... 26%  111M 0s\n",
      "  7750K .......... .......... .......... .......... .......... 26%  160M 0s\n",
      "  7800K .......... .......... .......... .......... .......... 26%  146M 0s\n",
      "  7850K .......... .......... .......... .......... .......... 26% 97.2M 0s\n",
      "  7900K .......... .......... .......... .......... .......... 26%  195M 0s\n",
      "  7950K .......... .......... .......... .......... .......... 27% 78.4M 0s\n",
      "  8000K .......... .......... .......... .......... .......... 27%  118M 0s\n",
      "  8050K .......... .......... .......... .......... .......... 27%  114M 0s\n",
      "  8100K .......... .......... .......... .......... .......... 27%  116M 0s\n",
      "  8150K .......... .......... .......... .......... .......... 27%  200M 0s\n",
      "  8200K .......... .......... .......... .......... .......... 27%  231M 0s\n",
      "  8250K .......... .......... .......... .......... .......... 28%  148M 0s\n",
      "  8300K .......... .......... .......... .......... .......... 28%  237M 0s\n",
      "  8350K .......... .......... .......... .......... .......... 28%  101M 0s\n",
      "  8400K .......... .......... .......... .......... .......... 28% 98.9M 0s\n",
      "  8450K .......... .......... .......... .......... .......... 28%  151M 0s\n",
      "  8500K .......... .......... .......... .......... .......... 28% 89.1M 0s\n",
      "  8550K .......... .......... .......... .......... .......... 29%  198M 0s\n",
      "  8600K .......... .......... .......... .......... .......... 29%  228M 0s\n",
      "  8650K .......... .......... .......... .......... .......... 29%  184M 0s\n",
      "  8700K .......... .......... .......... .......... .......... 29%  148M 0s\n",
      "  8750K .......... .......... .......... .......... .......... 29%  143M 0s\n",
      "  8800K .......... .......... .......... .......... .......... 29%  120M 0s\n",
      "  8850K .......... .......... .......... .......... .......... 30%  113M 0s\n",
      "  8900K .......... .......... .......... .......... .......... 30%  157M 0s\n",
      "  8950K .......... .......... .......... .......... .......... 30%  219M 0s\n",
      "  9000K .......... .......... .......... .......... .......... 30%  233M 0s\n",
      "  9050K .......... .......... .......... .......... .......... 30%  151M 0s\n",
      "  9100K .......... .......... .......... .......... .......... 30%  235M 0s\n",
      "  9150K .......... .......... .......... .......... .......... 31%  107M 0s\n",
      "  9200K .......... .......... .......... .......... .......... 31%  205M 0s\n",
      "  9250K .......... .......... .......... .......... .......... 31%  129M 0s\n",
      "  9300K .......... .......... .......... .......... .......... 31%  228M 0s\n",
      "  9350K .......... .......... .......... .......... .......... 31%  108M 0s\n",
      "  9400K .......... .......... .......... .......... .......... 31%  114M 0s\n",
      "  9450K .......... .......... .......... .......... .......... 32%  234M 0s\n",
      "  9500K .......... .......... .......... .......... .......... 32%  120M 0s\n",
      "  9550K .......... .......... .......... .......... .......... 32%  179M 0s\n",
      "  9600K .......... .......... .......... .......... .......... 32%  157M 0s\n",
      "  9650K .......... .......... .......... .......... .......... 32%  244M 0s\n",
      "  9700K .......... .......... .......... .......... .......... 32%  160M 0s\n",
      "  9750K .......... .......... .......... .......... .......... 33%  208M 0s\n",
      "  9800K .......... .......... .......... .......... .......... 33%  151M 0s\n",
      "  9850K .......... .......... .......... .......... .......... 33%  231M 0s\n",
      "  9900K .......... .......... .......... .......... .......... 33%  182M 0s\n",
      "  9950K .......... .......... .......... .......... .......... 33% 81.5M 0s\n",
      " 10000K .......... .......... .......... .......... .......... 33%  121M 0s\n",
      " 10050K .......... .......... .......... .......... .......... 34%  135M 0s\n",
      " 10100K .......... .......... .......... .......... .......... 34%  158M 0s\n",
      " 10150K .......... .......... .......... .......... .......... 34%  154M 0s\n",
      " 10200K .......... .......... .......... .......... .......... 34%  149M 0s\n",
      " 10250K .......... .......... .......... .......... .......... 34%  247M 0s\n",
      " 10300K .......... .......... .......... .......... .......... 34%  149M 0s\n",
      " 10350K .......... .......... .......... .......... .......... 35%  133M 0s\n",
      " 10400K .......... .......... .......... .......... .......... 35%  251M 0s\n",
      " 10450K .......... .......... .......... .......... .......... 35%  238M 0s\n",
      " 10500K .......... .......... .......... .......... .......... 35%  198M 0s\n",
      " 10550K .......... .......... .......... .......... .......... 35% 92.0M 0s\n",
      " 10600K .......... .......... .......... .......... .......... 36%  232M 0s\n",
      " 10650K .......... .......... .......... .......... .......... 36%  253M 0s\n",
      " 10700K .......... .......... .......... .......... .......... 36%  246M 0s\n",
      " 10750K .......... .......... .......... .......... .......... 36%  200M 0s\n",
      " 10800K .......... .......... .......... .......... .......... 36% 29.5M 0s\n",
      " 10850K .......... .......... .......... .......... .......... 36% 95.4M 0s\n",
      " 10900K .......... .......... .......... .......... .......... 37%  216M 0s\n",
      " 10950K .......... .......... .......... .......... .......... 37%  206M 0s\n",
      " 11000K .......... .......... .......... .......... .......... 37%  240M 0s\n",
      " 11050K .......... .......... .......... .......... .......... 37% 14.7M 0s\n",
      " 11100K .......... .......... .......... .......... .......... 37%  115M 0s\n",
      " 11150K .......... .......... .......... .......... .......... 37%  207M 0s\n",
      " 11200K .......... .......... .......... .......... .......... 38%  246M 0s\n",
      " 11250K .......... .......... .......... .......... .......... 38% 39.3M 0s\n",
      " 11300K .......... .......... .......... .......... .......... 38% 97.8M 0s\n",
      " 11350K .......... .......... .......... .......... .......... 38%  222M 0s\n",
      " 11400K .......... .......... .......... .......... .......... 38%  240M 0s\n",
      " 11450K .......... .......... .......... .......... .......... 38%  114M 0s\n",
      " 11500K .......... .......... .......... .......... .......... 39%  140M 0s\n",
      " 11550K .......... .......... .......... .......... .......... 39% 93.4M 0s\n",
      " 11600K .......... .......... .......... .......... .......... 39%  235M 0s\n",
      " 11650K .......... .......... .......... .......... .......... 39%  241M 0s\n",
      " 11700K .......... .......... .......... .......... .......... 39%  107M 0s\n",
      " 11750K .......... .......... .......... .......... .......... 39%  128M 0s\n",
      " 11800K .......... .......... .......... .......... .......... 40%  224M 0s\n",
      " 11850K .......... .......... .......... .......... .......... 40%  214M 0s\n",
      " 11900K .......... .......... .......... .......... .......... 40% 94.2M 0s\n",
      " 11950K .......... .......... .......... .......... .......... 40%  147M 0s\n",
      " 12000K .......... .......... .......... .......... .......... 40%  247M 0s\n",
      " 12050K .......... .......... .......... .......... .......... 40%  152M 0s\n",
      " 12100K .......... .......... .......... .......... .......... 41% 95.1M 0s\n",
      " 12150K .......... .......... .......... .......... .......... 41%  206M 0s\n",
      " 12200K .......... .......... .......... .......... .......... 41%  235M 0s\n",
      " 12250K .......... .......... .......... .......... .......... 41%  256M 0s\n",
      " 12300K .......... .......... .......... .......... .......... 41%  231M 0s\n",
      " 12350K .......... .......... .......... .......... .......... 41%  183M 0s\n",
      " 12400K .......... .......... .......... .......... .......... 42% 14.9M 0s\n",
      " 12450K .......... .......... .......... .......... .......... 42%  185M 0s\n",
      " 12500K .......... .......... .......... .......... .......... 42%  242M 0s\n",
      " 12550K .......... .......... .......... .......... .......... 42%  218M 0s\n",
      " 12600K .......... .......... .......... .......... .......... 42% 24.3M 0s\n",
      " 12650K .......... .......... .......... .......... .......... 42% 83.8M 0s\n",
      " 12700K .......... .......... .......... .......... .......... 43% 99.3M 0s\n",
      " 12750K .......... .......... .......... .......... .......... 43%  194M 0s\n",
      " 12800K .......... .......... .......... .......... .......... 43%  245M 0s\n",
      " 12850K .......... .......... .......... .......... .......... 43%  246M 0s\n",
      " 12900K .......... .......... .......... .......... .......... 43%  236M 0s\n",
      " 12950K .......... .......... .......... .......... .......... 43%  187M 0s\n",
      " 13000K .......... .......... .......... .......... .......... 44%  246M 0s\n",
      " 13050K .......... .......... .......... .......... .......... 44% 24.5M 0s\n",
      " 13100K .......... .......... .......... .......... .......... 44%  119M 0s\n",
      " 13150K .......... .......... .......... .......... .......... 44%  128M 0s\n",
      " 13200K .......... .......... .......... .......... .......... 44%  246M 0s\n",
      " 13250K .......... .......... .......... .......... .......... 44%  235M 0s\n",
      " 13300K .......... .......... .......... .......... .......... 45% 21.9M 0s\n",
      " 13350K .......... .......... .......... .......... .......... 45% 91.6M 0s\n",
      " 13400K .......... .......... .......... .......... .......... 45%  135M 0s\n",
      " 13450K .......... .......... .......... .......... .......... 45%  151M 0s\n",
      " 13500K .......... .......... .......... .......... .......... 45%  238M 0s\n",
      " 13550K .......... .......... .......... .......... .......... 45%  196M 0s\n",
      " 13600K .......... .......... .......... .......... .......... 46%  232M 0s\n",
      " 13650K .......... .......... .......... .......... .......... 46%  235M 0s\n",
      " 13700K .......... .......... .......... .......... .......... 46% 31.9M 0s\n",
      " 13750K .......... .......... .......... .......... .......... 46%  114M 0s\n",
      " 13800K .......... .......... .......... .......... .......... 46%  237M 0s\n",
      " 13850K .......... .......... .......... .......... .......... 46%  243M 0s\n",
      " 13900K .......... .......... .......... .......... .......... 47%  240M 0s\n",
      " 13950K .......... .......... .......... .......... .......... 47% 33.1M 0s\n",
      " 14000K .......... .......... .......... .......... .......... 47%  215M 0s\n",
      " 14050K .......... .......... .......... .......... .......... 47%  225M 0s\n",
      " 14100K .......... .......... .......... .......... .......... 47%  237M 0s\n",
      " 14150K .......... .......... .......... .......... .......... 48% 42.9M 0s\n",
      " 14200K .......... .......... .......... .......... .......... 48%  153M 0s\n",
      " 14250K .......... .......... .......... .......... .......... 48%  231M 0s\n",
      " 14300K .......... .......... .......... .......... .......... 48%  240M 0s\n",
      " 14350K .......... .......... .......... .......... .......... 48% 71.2M 0s\n",
      " 14400K .......... .......... .......... .......... .......... 48%  221M 0s\n",
      " 14450K .......... .......... .......... .......... .......... 49%  238M 0s\n",
      " 14500K .......... .......... .......... .......... .......... 49%  212M 0s\n",
      " 14550K .......... .......... .......... .......... .......... 49% 74.3M 0s\n",
      " 14600K .......... .......... .......... .......... .......... 49%  229M 0s\n",
      " 14650K .......... .......... .......... .......... .......... 49%  245M 0s\n",
      " 14700K .......... .......... .......... .......... .......... 49%  248M 0s\n",
      " 14750K .......... .......... .......... .......... .......... 50%  202M 0s\n",
      " 14800K .......... .......... .......... .......... .......... 50% 64.8M 0s\n",
      " 14850K .......... .......... .......... .......... .......... 50% 85.3M 0s\n",
      " 14900K .......... .......... .......... .......... .......... 50%  239M 0s\n",
      " 14950K .......... .......... .......... .......... .......... 50%  221M 0s\n",
      " 15000K .......... .......... .......... .......... .......... 50%  234M 0s\n",
      " 15050K .......... .......... .......... .......... .......... 51% 22.8M 0s\n",
      " 15100K .......... .......... .......... .......... .......... 51% 81.1M 0s\n",
      " 15150K .......... .......... .......... .......... .......... 51%  126M 0s\n",
      " 15200K .......... .......... .......... .......... .......... 51% 95.2M 0s\n",
      " 15250K .......... .......... .......... .......... .......... 51% 85.6M 0s\n",
      " 15300K .......... .......... .......... .......... .......... 51%  100M 0s\n",
      " 15350K .......... .......... .......... .......... .......... 52% 71.0M 0s\n",
      " 15400K .......... .......... .......... .......... .......... 52%  199M 0s\n",
      " 15450K .......... .......... .......... .......... .......... 52%  222M 0s\n",
      " 15500K .......... .......... .......... .......... .......... 52%  241M 0s\n",
      " 15550K .......... .......... .......... .......... .......... 52%  195M 0s\n",
      " 15600K .......... .......... .......... .......... .......... 52%  227M 0s\n",
      " 15650K .......... .......... .......... .......... .......... 53%  146M 0s\n",
      " 15700K .......... .......... .......... .......... .......... 53%  117M 0s\n",
      " 15750K .......... .......... .......... .......... .......... 53%  162M 0s\n",
      " 15800K .......... .......... .......... .......... .......... 53%  229M 0s\n",
      " 15850K .......... .......... .......... .......... .......... 53% 94.0M 0s\n",
      " 15900K .......... .......... .......... .......... .......... 53%  156M 0s\n",
      " 15950K .......... .......... .......... .......... .......... 54%  187M 0s\n",
      " 16000K .......... .......... .......... .......... .......... 54%  232M 0s\n",
      " 16050K .......... .......... .......... .......... .......... 54% 91.1M 0s\n",
      " 16100K .......... .......... .......... .......... .......... 54%  182M 0s\n",
      " 16150K .......... .......... .......... .......... .......... 54%  206M 0s\n",
      " 16200K .......... .......... .......... .......... .......... 54%  178M 0s\n",
      " 16250K .......... .......... .......... .......... .......... 55%  220M 0s\n",
      " 16300K .......... .......... .......... .......... .......... 55%  235M 0s\n",
      " 16350K .......... .......... .......... .......... .......... 55%  192M 0s\n",
      " 16400K .......... .......... .......... .......... .......... 55%  228M 0s\n",
      " 16450K .......... .......... .......... .......... .......... 55%  224M 0s\n",
      " 16500K .......... .......... .......... .......... .......... 55%  152M 0s\n",
      " 16550K .......... .......... .......... .......... .......... 56% 85.4M 0s\n",
      " 16600K .......... .......... .......... .......... .......... 56%  228M 0s\n",
      " 16650K .......... .......... .......... .......... .......... 56%  229M 0s\n",
      " 16700K .......... .......... .......... .......... .......... 56%  221M 0s\n",
      " 16750K .......... .......... .......... .......... .......... 56% 79.9M 0s\n",
      " 16800K .......... .......... .......... .......... .......... 56%  232M 0s\n",
      " 16850K .......... .......... .......... .......... .......... 57%  250M 0s\n",
      " 16900K .......... .......... .......... .......... .......... 57%  234M 0s\n",
      " 16950K .......... .......... .......... .......... .......... 57%  214M 0s\n",
      " 17000K .......... .......... .......... .......... .......... 57%  247M 0s\n",
      " 17050K .......... .......... .......... .......... .......... 57%  224M 0s\n",
      " 17100K .......... .......... .......... .......... .......... 57%  230M 0s\n",
      " 17150K .......... .......... .......... .......... .......... 58% 77.6M 0s\n",
      " 17200K .......... .......... .......... .......... .......... 58%  218M 0s\n",
      " 17250K .......... .......... .......... .......... .......... 58%  244M 0s\n",
      " 17300K .......... .......... .......... .......... .......... 58%  234M 0s\n",
      " 17350K .......... .......... .......... .......... .......... 58%  213M 0s\n",
      " 17400K .......... .......... .......... .......... .......... 58% 75.5M 0s\n",
      " 17450K .......... .......... .......... .......... .......... 59%  204M 0s\n",
      " 17500K .......... .......... .......... .......... .......... 59%  226M 0s\n",
      " 17550K .......... .......... .......... .......... .......... 59%  205M 0s\n",
      " 17600K .......... .......... .......... .......... .......... 59%  226M 0s\n",
      " 17650K .......... .......... .......... .......... .......... 59%  239M 0s\n",
      " 17700K .......... .......... .......... .......... .......... 60%  225M 0s\n",
      " 17750K .......... .......... .......... .......... .......... 60% 81.9M 0s\n",
      " 17800K .......... .......... .......... .......... .......... 60%  188M 0s\n",
      " 17850K .......... .......... .......... .......... .......... 60%  233M 0s\n",
      " 17900K .......... .......... .......... .......... .......... 60%  115M 0s\n",
      " 17950K .......... .......... .......... .......... .......... 60%  112M 0s\n",
      " 18000K .......... .......... .......... .......... .......... 61%  226M 0s\n",
      " 18050K .......... .......... .......... .......... .......... 61%  235M 0s\n",
      " 18100K .......... .......... .......... .......... .......... 61%  233M 0s\n",
      " 18150K .......... .......... .......... .......... .......... 61%  105M 0s\n",
      " 18200K .......... .......... .......... .......... .......... 61%  124M 0s\n",
      " 18250K .......... .......... .......... .......... .......... 61%  255M 0s\n",
      " 18300K .......... .......... .......... .......... .......... 62%  152M 0s\n",
      " 18350K .......... .......... .......... .......... .......... 62% 93.4M 0s\n",
      " 18400K .......... .......... .......... .......... .......... 62%  206M 0s\n",
      " 18450K .......... .......... .......... .......... .......... 62% 83.9M 0s\n",
      " 18500K .......... .......... .......... .......... .......... 62%  142M 0s\n",
      " 18550K .......... .......... .......... .......... .......... 62% 89.6M 0s\n",
      " 18600K .......... .......... .......... .......... .......... 63%  231M 0s\n",
      " 18650K .......... .......... .......... .......... .......... 63%  229M 0s\n",
      " 18700K .......... .......... .......... .......... .......... 63%  221M 0s\n",
      " 18750K .......... .......... .......... .......... .......... 63%  192M 0s\n",
      " 18800K .......... .......... .......... .......... .......... 63%  235M 0s\n",
      " 18850K .......... .......... .......... .......... .......... 63% 90.4M 0s\n",
      " 18900K .......... .......... .......... .......... .......... 64% 99.0M 0s\n",
      " 18950K .......... .......... .......... .......... .......... 64%  108M 0s\n",
      " 19000K .......... .......... .......... .......... .......... 64% 77.9M 0s\n",
      " 19050K .......... .......... .......... .......... .......... 64%  147M 0s\n",
      " 19100K .......... .......... .......... .......... .......... 64%  247M 0s\n",
      " 19150K .......... .......... .......... .......... .......... 64%  188M 0s\n",
      " 19200K .......... .......... .......... .......... .......... 65%  240M 0s\n",
      " 19250K .......... .......... .......... .......... .......... 65%  227M 0s\n",
      " 19300K .......... .......... .......... .......... .......... 65%  173M 0s\n",
      " 19350K .......... .......... .......... .......... .......... 65% 97.0M 0s\n",
      " 19400K .......... .......... .......... .......... .......... 65%  171M 0s\n",
      " 19450K .......... .......... .......... .......... .......... 65%  234M 0s\n",
      " 19500K .......... .......... .......... .......... .......... 66%  238M 0s\n",
      " 19550K .......... .......... .......... .......... .......... 66%  199M 0s\n",
      " 19600K .......... .......... .......... .......... .......... 66%  130M 0s\n",
      " 19650K .......... .......... .......... .......... .......... 66%  119M 0s\n",
      " 19700K .......... .......... .......... .......... .......... 66%  183M 0s\n",
      " 19750K .......... .......... .......... .......... .......... 66% 93.0M 0s\n",
      " 19800K .......... .......... .......... .......... .......... 67%  149M 0s\n",
      " 19850K .......... .......... .......... .......... .......... 67%  240M 0s\n",
      " 19900K .......... .......... .......... .......... .......... 67%  131M 0s\n",
      " 19950K .......... .......... .......... .......... .......... 67% 90.6M 0s\n",
      " 20000K .......... .......... .......... .......... .......... 67%  242M 0s\n",
      " 20050K .......... .......... .......... .......... .......... 67%  239M 0s\n",
      " 20100K .......... .......... .......... .......... .......... 68%  244M 0s\n",
      " 20150K .......... .......... .......... .......... .......... 68%  213M 0s\n",
      " 20200K .......... .......... .......... .......... .......... 68%  241M 0s\n",
      " 20250K .......... .......... .......... .......... .......... 68% 86.4M 0s\n",
      " 20300K .......... .......... .......... .......... .......... 68%  158M 0s\n",
      " 20350K .......... .......... .......... .......... .......... 68%  201M 0s\n",
      " 20400K .......... .......... .......... .......... .......... 69%  234M 0s\n",
      " 20450K .......... .......... .......... .......... .......... 69%  243M 0s\n",
      " 20500K .......... .......... .......... .......... .......... 69%  130M 0s\n",
      " 20550K .......... .......... .......... .......... .......... 69%  101M 0s\n",
      " 20600K .......... .......... .......... .......... .......... 69%  236M 0s\n",
      " 20650K .......... .......... .......... .......... .......... 69%  239M 0s\n",
      " 20700K .......... .......... .......... .......... .......... 70%  239M 0s\n",
      " 20750K .......... .......... .......... .......... .......... 70%  204M 0s\n",
      " 20800K .......... .......... .......... .......... .......... 70%  131M 0s\n",
      " 20850K .......... .......... .......... .......... .......... 70%  110M 0s\n",
      " 20900K .......... .......... .......... .......... .......... 70%  246M 0s\n",
      " 20950K .......... .......... .......... .......... .......... 70%  211M 0s\n",
      " 21000K .......... .......... .......... .......... .......... 71%  240M 0s\n",
      " 21050K .......... .......... .......... .......... .......... 71%  226M 0s\n",
      " 21100K .......... .......... .......... .......... .......... 71%  141M 0s\n",
      " 21150K .......... .......... .......... .......... .......... 71% 83.9M 0s\n",
      " 21200K .......... .......... .......... .......... .......... 71%  248M 0s\n",
      " 21250K .......... .......... .......... .......... .......... 72%  244M 0s\n",
      " 21300K .......... .......... .......... .......... .......... 72%  233M 0s\n",
      " 21350K .......... .......... .......... .......... .......... 72% 70.6M 0s\n",
      " 21400K .......... .......... .......... .......... .......... 72%  240M 0s\n",
      " 21450K .......... .......... .......... .......... .......... 72%  235M 0s\n",
      " 21500K .......... .......... .......... .......... .......... 72%  241M 0s\n",
      " 21550K .......... .......... .......... .......... .......... 73%  207M 0s\n",
      " 21600K .......... .......... .......... .......... .......... 73%  236M 0s\n",
      " 21650K .......... .......... .......... .......... .......... 73%  223M 0s\n",
      " 21700K .......... .......... .......... .......... .......... 73% 91.8M 0s\n",
      " 21750K .......... .......... .......... .......... .......... 73% 70.5M 0s\n",
      " 21800K .......... .......... .......... .......... .......... 73%  175M 0s\n",
      " 21850K .......... .......... .......... .......... .......... 74%  249M 0s\n",
      " 21900K .......... .......... .......... .......... .......... 74%  240M 0s\n",
      " 21950K .......... .......... .......... .......... .......... 74%  201M 0s\n",
      " 22000K .......... .......... .......... .......... .......... 74%  125M 0s\n",
      " 22050K .......... .......... .......... .......... .......... 74% 88.3M 0s\n",
      " 22100K .......... .......... .......... .......... .......... 74%  170M 0s\n",
      " 22150K .......... .......... .......... .......... .......... 75%  219M 0s\n",
      " 22200K .......... .......... .......... .......... .......... 75%  244M 0s\n",
      " 22250K .......... .......... .......... .......... .......... 75%  243M 0s\n",
      " 22300K .......... .......... .......... .......... .......... 75%  108M 0s\n",
      " 22350K .......... .......... .......... .......... .......... 75%  127M 0s\n",
      " 22400K .......... .......... .......... .......... .......... 75%  248M 0s\n",
      " 22450K .......... .......... .......... .......... .......... 76%  254M 0s\n",
      " 22500K .......... .......... .......... .......... .......... 76%  230M 0s\n",
      " 22550K .......... .......... .......... .......... .......... 76%  204M 0s\n",
      " 22600K .......... .......... .......... .......... .......... 76% 79.6M 0s\n",
      " 22650K .......... .......... .......... .......... .......... 76%  238M 0s\n",
      " 22700K .......... .......... .......... .......... .......... 76%  248M 0s\n",
      " 22750K .......... .......... .......... .......... .......... 77%  101M 0s\n",
      " 22800K .......... .......... .......... .......... .......... 77%  157M 0s\n",
      " 22850K .......... .......... .......... .......... .......... 77%  246M 0s\n",
      " 22900K .......... .......... .......... .......... .......... 77%  251M 0s\n",
      " 22950K .......... .......... .......... .......... .......... 77%  126M 0s\n",
      " 23000K .......... .......... .......... .......... .......... 77%  113M 0s\n",
      " 23050K .......... .......... .......... .......... .......... 78%  242M 0s\n",
      " 23100K .......... .......... .......... .......... .......... 78%  235M 0s\n",
      " 23150K .......... .......... .......... .......... .......... 78% 75.1M 0s\n",
      " 23200K .......... .......... .......... .......... .......... 78%  248M 0s\n",
      " 23250K .......... .......... .......... .......... .......... 78% 79.1M 0s\n",
      " 23300K .......... .......... .......... .......... .......... 78%  244M 0s\n",
      " 23350K .......... .......... .......... .......... .......... 79%  205M 0s\n",
      " 23400K .......... .......... .......... .......... .......... 79% 89.6M 0s\n",
      " 23450K .......... .......... .......... .......... .......... 79%  188M 0s\n",
      " 23500K .......... .......... .......... .......... .......... 79%  167M 0s\n",
      " 23550K .......... .......... .......... .......... .......... 79%  123M 0s\n",
      " 23600K .......... .......... .......... .......... .......... 79%  233M 0s\n",
      " 23650K .......... .......... .......... .......... .......... 80%  234M 0s\n",
      " 23700K .......... .......... .......... .......... .......... 80%  109M 0s\n",
      " 23750K .......... .......... .......... .......... .......... 80%  160M 0s\n",
      " 23800K .......... .......... .......... .......... .......... 80%  251M 0s\n",
      " 23850K .......... .......... .......... .......... .......... 80%  232M 0s\n",
      " 23900K .......... .......... .......... .......... .......... 80%  230M 0s\n",
      " 23950K .......... .......... .......... .......... .......... 81%  199M 0s\n",
      " 24000K .......... .......... .......... .......... .......... 81%  244M 0s\n",
      " 24050K .......... .......... .......... .......... .......... 81%  252M 0s\n",
      " 24100K .......... .......... .......... .......... .......... 81%  246M 0s\n",
      " 24150K .......... .......... .......... .......... .......... 81%  210M 0s\n",
      " 24200K .......... .......... .......... .......... .......... 81%  235M 0s\n",
      " 24250K .......... .......... .......... .......... .......... 82%  227M 0s\n",
      " 24300K .......... .......... .......... .......... .......... 82%  244M 0s\n",
      " 24350K .......... .......... .......... .......... .......... 82%  197M 0s\n",
      " 24400K .......... .......... .......... .......... .......... 82%  247M 0s\n",
      " 24450K .......... .......... .......... .......... .......... 82%  245M 0s\n",
      " 24500K .......... .......... .......... .......... .......... 82%  108M 0s\n",
      " 24550K .......... .......... .......... .......... .......... 83% 68.0M 0s\n",
      " 24600K .......... .......... .......... .......... .......... 83% 83.4M 0s\n",
      " 24650K .......... .......... .......... .......... .......... 83%  101M 0s\n",
      " 24700K .......... .......... .......... .......... .......... 83% 67.1M 0s\n",
      " 24750K .......... .......... .......... .......... .......... 83%  116M 0s\n",
      " 24800K .......... .......... .......... .......... .......... 84%  246M 0s\n",
      " 24850K .......... .......... .......... .......... .......... 84%  241M 0s\n",
      " 24900K .......... .......... .......... .......... .......... 84%  236M 0s\n",
      " 24950K .......... .......... .......... .......... .......... 84% 81.9M 0s\n",
      " 25000K .......... .......... .......... .......... .......... 84%  225M 0s\n",
      " 25050K .......... .......... .......... .......... .......... 84%  248M 0s\n",
      " 25100K .......... .......... .......... .......... .......... 85%  181M 0s\n",
      " 25150K .......... .......... .......... .......... .......... 85%  112M 0s\n",
      " 25200K .......... .......... .......... .......... .......... 85%  244M 0s\n",
      " 25250K .......... .......... .......... .......... .......... 85%  167M 0s\n",
      " 25300K .......... .......... .......... .......... .......... 85%  230M 0s\n",
      " 25350K .......... .......... .......... .......... .......... 85%  213M 0s\n",
      " 25400K .......... .......... .......... .......... .......... 86%  237M 0s\n",
      " 25450K .......... .......... .......... .......... .......... 86%  235M 0s\n",
      " 25500K .......... .......... .......... .......... .......... 86%  119M 0s\n",
      " 25550K .......... .......... .......... .......... .......... 86% 79.0M 0s\n",
      " 25600K .......... .......... .......... .......... .......... 86%  250M 0s\n",
      " 25650K .......... .......... .......... .......... .......... 86%  117M 0s\n",
      " 25700K .......... .......... .......... .......... .......... 87% 89.3M 0s\n",
      " 25750K .......... .......... .......... .......... .......... 87%  220M 0s\n",
      " 25800K .......... .......... .......... .......... .......... 87%  131M 0s\n",
      " 25850K .......... .......... .......... .......... .......... 87%  238M 0s\n",
      " 25900K .......... .......... .......... .......... .......... 87%  242M 0s\n",
      " 25950K .......... .......... .......... .......... .......... 87%  145M 0s\n",
      " 26000K .......... .......... .......... .......... .......... 88%  190M 0s\n",
      " 26050K .......... .......... .......... .......... .......... 88%  185M 0s\n",
      " 26100K .......... .......... .......... .......... .......... 88%  174M 0s\n",
      " 26150K .......... .......... .......... .......... .......... 88%  140M 0s\n",
      " 26200K .......... .......... .......... .......... .......... 88%  109M 0s\n",
      " 26250K .......... .......... .......... .......... .......... 88%  136M 0s\n",
      " 26300K .......... .......... .......... .......... .......... 89%  106M 0s\n",
      " 26350K .......... .......... .......... .......... .......... 89%  204M 0s\n",
      " 26400K .......... .......... .......... .......... .......... 89%  243M 0s\n",
      " 26450K .......... .......... .......... .......... .......... 89%  133M 0s\n",
      " 26500K .......... .......... .......... .......... .......... 89%  242M 0s\n",
      " 26550K .......... .......... .......... .......... .......... 89%  175M 0s\n",
      " 26600K .......... .......... .......... .......... .......... 90%  157M 0s\n",
      " 26650K .......... .......... .......... .......... .......... 90%  114M 0s\n",
      " 26700K .......... .......... .......... .......... .......... 90%  233M 0s\n",
      " 26750K .......... .......... .......... .......... .......... 90%  113M 0s\n",
      " 26800K .......... .......... .......... .......... .......... 90%  125M 0s\n",
      " 26850K .......... .......... .......... .......... .......... 90%  235M 0s\n",
      " 26900K .......... .......... .......... .......... .......... 91%  196M 0s\n",
      " 26950K .......... .......... .......... .......... .......... 91% 99.6M 0s\n",
      " 27000K .......... .......... .......... .......... .......... 91%  175M 0s\n",
      " 27050K .......... .......... .......... .......... .......... 91%  235M 0s\n",
      " 27100K .......... .......... .......... .......... .......... 91%  157M 0s\n",
      " 27150K .......... .......... .......... .......... .......... 91%  117M 0s\n",
      " 27200K .......... .......... .......... .......... .......... 92%  233M 0s\n",
      " 27250K .......... .......... .......... .......... .......... 92%  238M 0s\n",
      " 27300K .......... .......... .......... .......... .......... 92%  218M 0s\n",
      " 27350K .......... .......... .......... .......... .......... 92% 82.9M 0s\n",
      " 27400K .......... .......... .......... .......... .......... 92%  246M 0s\n",
      " 27450K .......... .......... .......... .......... .......... 92%  231M 0s\n",
      " 27500K .......... .......... .......... .......... .......... 93%  137M 0s\n",
      " 27550K .......... .......... .......... .......... .......... 93% 82.7M 0s\n",
      " 27600K .......... .......... .......... .......... .......... 93%  247M 0s\n",
      " 27650K .......... .......... .......... .......... .......... 93%  240M 0s\n",
      " 27700K .......... .......... .......... .......... .......... 93% 98.6M 0s\n",
      " 27750K .......... .......... .......... .......... .......... 93%  210M 0s\n",
      " 27800K .......... .......... .......... .......... .......... 94%  242M 0s\n",
      " 27850K .......... .......... .......... .......... .......... 94%  234M 0s\n",
      " 27900K .......... .......... .......... .......... .......... 94%  124M 0s\n",
      " 27950K .......... .......... .......... .......... .......... 94%  144M 0s\n",
      " 28000K .......... .......... .......... .......... .......... 94%  209M 0s\n",
      " 28050K .......... .......... .......... .......... .......... 95%  108M 0s\n",
      " 28100K .......... .......... .......... .......... .......... 95%  238M 0s\n",
      " 28150K .......... .......... .......... .......... .......... 95%  211M 0s\n",
      " 28200K .......... .......... .......... .......... .......... 95%  242M 0s\n",
      " 28250K .......... .......... .......... .......... .......... 95%  226M 0s\n",
      " 28300K .......... .......... .......... .......... .......... 95%  216M 0s\n",
      " 28350K .......... .......... .......... .......... .......... 96% 99.8M 0s\n",
      " 28400K .......... .......... .......... .......... .......... 96%  128M 0s\n",
      " 28450K .......... .......... .......... .......... .......... 96%  245M 0s\n",
      " 28500K .......... .......... .......... .......... .......... 96%  232M 0s\n",
      " 28550K .......... .......... .......... .......... .......... 96%  208M 0s\n",
      " 28600K .......... .......... .......... .......... .......... 96% 89.3M 0s\n",
      " 28650K .......... .......... .......... .......... .......... 97%  163M 0s\n",
      " 28700K .......... .......... .......... .......... .......... 97%  247M 0s\n",
      " 28750K .......... .......... .......... .......... .......... 97%  197M 0s\n",
      " 28800K .......... .......... .......... .......... .......... 97%  264M 0s\n",
      " 28850K .......... .......... .......... .......... .......... 97%  239M 0s\n",
      " 28900K .......... .......... .......... .......... .......... 97%  233M 0s\n",
      " 28950K .......... .......... .......... .......... .......... 98%  202M 0s\n",
      " 29000K .......... .......... .......... .......... .......... 98%  228M 0s\n",
      " 29050K .......... .......... .......... .......... .......... 98%  242M 0s\n",
      " 29100K .......... .......... .......... .......... .......... 98%  228M 0s\n",
      " 29150K .......... .......... .......... .......... .......... 98%  200M 0s\n",
      " 29200K .......... .......... .......... .......... .......... 98%  248M 0s\n",
      " 29250K .......... .......... .......... .......... .......... 99%  238M 0s\n",
      " 29300K .......... .......... .......... .......... .......... 99% 28.1M 0s\n",
      " 29350K .......... .......... .......... .......... .......... 99%  188M 0s\n",
      " 29400K .......... .......... .......... .......... .......... 99%  238M 0s\n",
      " 29450K .......... .......... .......... .......... .......... 99%  240M 0s\n",
      " 29500K .......... .......... .......... .......... .......... 99% 20.0M 0s\n",
      " 29550K .......... .......... ........                        100%  208M=0.2s\n",
      "\u001b[0m\n",
      "\u001b[34m2020-08-23 20:47:13 (116 MB/s) - ‘v1.1/train-v1.1.json’ saved [30288272/30288272]\n",
      "\u001b[0m\n",
      "\u001b[34m--2020-08-23 20:47:13--  https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json\u001b[0m\n",
      "\u001b[34mResolving rajpurkar.github.io (rajpurkar.github.io)... 185.199.108.153, 185.199.109.153, 185.199.110.153, ...\u001b[0m\n",
      "\u001b[34mConnecting to rajpurkar.github.io (rajpurkar.github.io)|185.199.108.153|:443... connected.\u001b[0m\n",
      "\u001b[34mHTTP request sent, awaiting response... 200 OK\u001b[0m\n",
      "\u001b[34mLength: 4854279 (4.6M) [application/json]\u001b[0m\n",
      "\u001b[34mSaving to: ‘v1.1/dev-v1.1.json’\n",
      "\n",
      "     0K .......... .......... .......... .......... ..........  1% 18.7M 0s\n",
      "    50K .......... .......... .......... .......... ..........  2% 20.4M 0s\n",
      "   100K .......... .......... .......... .......... ..........  3% 23.1M 0s\n",
      "   150K .......... .......... .......... .......... ..........  4% 20.6M 0s\n",
      "   200K .......... .......... .......... .......... ..........  5% 26.8M 0s\n",
      "   250K .......... .......... .......... .......... ..........  6% 22.3M 0s\n",
      "   300K .......... .......... .......... .......... ..........  7% 36.2M 0s\n",
      "   350K .......... .......... .......... .......... ..........  8% 28.7M 0s\n",
      "   400K .......... .......... .......... .......... ..........  9% 32.1M 0s\n",
      "   450K .......... .......... .......... .......... .......... 10% 36.5M 0s\n",
      "   500K .......... .......... .......... .......... .......... 11% 38.5M 0s\n",
      "   550K .......... .......... .......... .......... .......... 12% 36.0M 0s\n",
      "   600K .......... .......... .......... .......... .......... 13% 36.2M 0s\n",
      "   650K .......... .......... .......... .......... .......... 14% 33.7M 0s\n",
      "   700K .......... .......... .......... .......... .......... 15% 46.2M 0s\n",
      "   750K .......... .......... .......... .......... .......... 16% 49.4M 0s\n",
      "   800K .......... .......... .......... .......... .......... 17% 58.8M 0s\n",
      "   850K .......... .......... .......... .......... .......... 18% 37.3M 0s\n",
      "   900K .......... .......... .......... .......... .......... 20% 53.0M 0s\n",
      "   950K .......... .......... .......... .......... .......... 21% 36.8M 0s\n",
      "  1000K .......... .......... .......... .......... .......... 22% 37.1M 0s\n",
      "  1050K .......... .......... .......... .......... .......... 23% 98.6M 0s\n",
      "  1100K .......... .......... .......... .......... .......... 24% 53.3M 0s\n",
      "  1150K .......... .......... .......... .......... .......... 25% 71.6M 0s\n",
      "  1200K .......... .......... .......... .......... .......... 26% 51.1M 0s\n",
      "  1250K .......... .......... .......... .......... .......... 27%  106M 0s\n",
      "  1300K .......... .......... .......... .......... .......... 28% 41.1M 0s\n",
      "  1350K .......... .......... .......... .......... .......... 29% 41.2M 0s\n",
      "  1400K .......... .......... .......... .......... .......... 30%  174M 0s\n",
      "  1450K .......... .......... .......... .......... .......... 31% 46.3M 0s\n",
      "  1500K .......... .......... .......... .......... .......... 32% 73.8M 0s\n",
      "  1550K .......... .......... .......... .......... .......... 33% 58.4M 0s\n",
      "  1600K .......... .......... .......... .......... .......... 34% 84.2M 0s\n",
      "  1650K .......... .......... .......... .......... .......... 35% 61.4M 0s\n",
      "  1700K .......... .......... .......... .......... .......... 36% 76.1M 0s\n",
      "  1750K .......... .......... .......... .......... .......... 37% 96.3M 0s\n",
      "  1800K .......... .......... .......... .......... .......... 39% 81.0M 0s\n",
      "  1850K .......... .......... .......... .......... .......... 40% 59.9M 0s\n",
      "  1900K .......... .......... .......... .......... .......... 41%  114M 0s\n",
      "  1950K .......... .......... .......... .......... .......... 42% 63.8M 0s\n",
      "  2000K .......... .......... .......... .......... .......... 43% 77.5M 0s\n",
      "  2050K .......... .......... .......... .......... .......... 44%  168M 0s\n",
      "  2100K .......... .......... .......... .......... .......... 45% 48.6M 0s\n",
      "  2150K .......... .......... .......... .......... .......... 46%  133M 0s\n",
      "  2200K .......... .......... .......... .......... .......... 47%  135M 0s\n",
      "  2250K .......... .......... .......... .......... .......... 48% 64.1M 0s\n",
      "  2300K .......... .......... .......... .......... .......... 49%  151M 0s\n",
      "  2350K .......... .......... .......... .......... .......... 50% 47.1M 0s\n",
      "  2400K .......... .......... .......... .......... .......... 51%  158M 0s\n",
      "  2450K .......... .......... .......... .......... .......... 52%  205M 0s\n",
      "  2500K .......... .......... .......... .......... .......... 53% 52.0M 0s\n",
      "  2550K .......... .......... .......... .......... .......... 54%  139M 0s\n",
      "  2600K .......... .......... .......... .......... .......... 55%  151M 0s\n",
      "  2650K .......... .......... .......... .......... .......... 56% 56.9M 0s\n",
      "  2700K .......... .......... .......... .......... .......... 58%  174M 0s\n",
      "  2750K .......... .......... .......... .......... .......... 59%  101M 0s\n",
      "  2800K .......... .......... .......... .......... .......... 60% 68.5M 0s\n",
      "  2850K .......... .......... .......... .......... .......... 61% 75.7M 0s\n",
      "  2900K .......... .......... .......... .......... .......... 62%  118M 0s\n",
      "  2950K .......... .......... .......... .......... .......... 63%  219M 0s\n",
      "  3000K .......... .......... .......... .......... .......... 64% 55.0M 0s\n",
      "  3050K .......... .......... .......... .......... .......... 65%  156M 0s\n",
      "  3100K .......... .......... .......... .......... .......... 66%  159M 0s\n",
      "  3150K .......... .......... .......... .......... .......... 67%  104M 0s\n",
      "  3200K .......... .......... .......... .......... .......... 68% 82.3M 0s\n",
      "  3250K .......... .......... .......... .......... .......... 69%  144M 0s\n",
      "  3300K .......... .......... .......... .......... .......... 70% 55.7M 0s\n",
      "  3350K .......... .......... .......... .......... .......... 71%  147M 0s\n",
      "  3400K .......... .......... .......... .......... .......... 72%  156M 0s\n",
      "  3450K .......... .......... .......... .......... .......... 73%  216M 0s\n",
      "  3500K .......... .......... .......... .......... .......... 74% 70.8M 0s\n",
      "  3550K .......... .......... .......... .......... .......... 75%  172M 0s\n",
      "  3600K .......... .......... .......... .......... .......... 76%  149M 0s\n",
      "  3650K .......... .......... .......... .......... .......... 78%  237M 0s\n",
      "  3700K .......... .......... .......... .......... .......... 79% 71.5M 0s\n",
      "  3750K .......... .......... .......... .......... .......... 80%  142M 0s\n",
      "  3800K .......... .......... .......... .......... .......... 81%  154M 0s\n",
      "  3850K .......... .......... .......... .......... .......... 82%  227M 0s\n",
      "  3900K .......... .......... .......... .......... .......... 83% 55.4M 0s\n",
      "  3950K .......... .......... .......... .......... .......... 84%  152M 0s\n",
      "  4000K .......... .......... .......... .......... .......... 85%  178M 0s\n",
      "  4050K .......... .......... .......... .......... .......... 86%  208M 0s\n",
      "  4100K .......... .......... .......... .......... .......... 87%  163M 0s\n",
      "  4150K .......... .......... .......... .......... .......... 88%  124M 0s\n",
      "  4200K .......... .......... .......... .......... .......... 89%  149M 0s\n",
      "  4250K .......... .......... .......... .......... .......... 90%  145M 0s\n",
      "  4300K .......... .......... .......... .......... .......... 91%  206M 0s\n",
      "  4350K .......... .......... .......... .......... .......... 92%  107M 0s\n",
      "  4400K .......... .......... .......... .......... .......... 93% 91.3M 0s\n",
      "  4450K .......... .......... .......... .......... .......... 94%  152M 0s\n",
      "  4500K .......... .......... .......... .......... .......... 95%  241M 0s\n",
      "  4550K .......... .......... .......... .......... .......... 97%  216M 0s\n",
      "  4600K .......... .......... .......... .......... .......... 98% 80.5M 0s\n",
      "  4650K .......... .......... .......... .......... .......... 99%  240M 0s\n",
      "  4700K .......... .......... .......... ..........           100%  222M=0.07s\n",
      "\u001b[0m\n",
      "\u001b[34m2020-08-23 20:47:14 (65.4 MB/s) - ‘v1.1/dev-v1.1.json’ saved [4854279/4854279]\n",
      "\u001b[0m\n",
      "\u001b[34m--2020-08-23 20:47:14--  https://worksheets.codalab.org/rest/bundles/0xbcd57bee090b421c982906709c8c27e1/contents/blob/\u001b[0m\n",
      "\u001b[34mResolving worksheets.codalab.org (worksheets.codalab.org)... 40.114.41.203\u001b[0m\n",
      "\u001b[34mConnecting to worksheets.codalab.org (worksheets.codalab.org)|40.114.41.203|:443... connected.\u001b[0m\n",
      "\u001b[34mHTTP request sent, awaiting response... 200 OK\u001b[0m\n",
      "\u001b[34mSyntax error in Set-Cookie: codalab_session=\"\"; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=-1; Path=/ at position 70.\u001b[0m\n",
      "\u001b[34mLength: unspecified [text/x-python]\u001b[0m\n",
      "\u001b[34mSaving to: ‘v1.1/evaluate-v1.1.py’\n",
      "\n",
      "     0K ...                                                     287M=0s\n",
      "\u001b[0m\n",
      "\u001b[34m2020-08-23 20:47:14 (287 MB/s) - ‘v1.1/evaluate-v1.1.py’ saved [3419]\n",
      "\u001b[0m\n",
      "\u001b[34mSquad data download done!\u001b[0m\n",
      "\u001b[34mVerifying Dataset....\u001b[0m\n",
      "\u001b[34mComplete!\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34mdevice: cuda:2 n_gpu: 1, distributed training: True, 16-bits training: True\u001b[0m\n",
      "\u001b[34mdevice: cuda:3 n_gpu: 1, distributed training: True, 16-bits training: True\u001b[0m\n",
      "\u001b[34mdevice: cuda:1 n_gpu: 1, distributed training: True, 16-bits training: True\u001b[0m\n",
      "\u001b[34mdevice: cuda:0 n_gpu: 1, distributed training: True, 16-bits training: True\u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:47:19.117129 - PARAMETER Config : [\"Namespace(bert_model='bert-base-uncased', config_file='/workspace/bert/bert_config.json', do_eval=False, do_lower_case=True, do_predict=False, do_train=True, doc_stride=128, eval_script='evaluate.py', fp16=True, gradient_accumulation_steps=1, init_checkpoint='/workspace/bert/bert_base.pt', json_summary='dllogger.json', learning_rate=3e-05, local_rank=0, log_freq=50, loss_scale=0, max_answer_length=30, max_query_length=64, max_seq_length=512, max_steps=-1.0, n_best_size=20, no_cuda=False, null_score_diff_threshold=0.0, num_train_epochs=1.0, output_dir='/opt/ml/model', predict_batch_size=8, predict_file=None, save_to_s3='sagemaker-us-east-1-383827541835', save_to_sagemaker_dir=None, seed=1, skip_checkpoint=False, train_batch_size=16, train_file='/workspace/bert/data/squad/v1.1/train-v1.1.json', use_env=False, verbose_logging=False, version_2_with_negative=False, vocab_file='/workspace/bert/data/bert_vocab.txt', warmup_proportion=0.1)\"] \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:47:19.117359 - PARAMETER SEED : 1 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:47:30.173659 - PARAMETER loading_checkpoint : True \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:47:42.559456 - PARAMETER loaded_checkpoint : True \u001b[0m\n",
      "\u001b[34mSelected optimization level O2:  FP16 training with FP32 batchnorm and FP32 master weights.\n",
      "\u001b[0m\n",
      "\u001b[34mDefaults for this optimization level are:\u001b[0m\n",
      "\u001b[34menabled                : True\u001b[0m\n",
      "\u001b[34mopt_level              : O2\u001b[0m\n",
      "\u001b[34mcast_model_type        : torch.float16\u001b[0m\n",
      "\u001b[34mpatch_torch_functions  : False\u001b[0m\n",
      "\u001b[34mkeep_batchnorm_fp32    : True\u001b[0m\n",
      "\u001b[34mmaster_weights         : True\u001b[0m\n",
      "\u001b[34mloss_scale             : dynamic\u001b[0m\n",
      "\u001b[34mProcessing user overrides (additional kwargs that are not None)...\u001b[0m\n",
      "\u001b[34mAfter processing overrides, optimization options are:\u001b[0m\n",
      "\u001b[34menabled                : True\u001b[0m\n",
      "\u001b[34mopt_level              : O2\u001b[0m\n",
      "\u001b[34mcast_model_type        : torch.float16\u001b[0m\n",
      "\u001b[34mpatch_torch_functions  : False\u001b[0m\n",
      "\u001b[34mkeep_batchnorm_fp32    : False\u001b[0m\n",
      "\u001b[34mmaster_weights         : True\u001b[0m\n",
      "\u001b[34mloss_scale             : dynamic\u001b[0m\n",
      "\u001b[34mloading cached features ... ...\u001b[0m\n",
      "\u001b[34mloading cached features ... ...\u001b[0m\n",
      "\u001b[34mconverting features ... ...\u001b[0m\n",
      "\u001b[34mconverting features ... ...\u001b[0m\n",
      "\u001b[34mloading cached features ... ...\u001b[0m\n",
      "\u001b[34mconverting features ... ...\u001b[0m\n",
      "\u001b[34mloading cached features ... ...\u001b[0m\n",
      "\u001b[34mconverting features ... ...\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   *** Example ***\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   *** Example ***\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   unique_id: 1000000000\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   unique_id: 1000000000\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   example_index: 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   example_index: 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   doc_span_index: 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   doc_span_index: 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   tokens: [CLS] to whom did the virgin mary allegedly appear in 1858 in lou ##rdes france ? [SEP] architectural ##ly , the school has a catholic character . atop the main building ' s gold dome is a golden statue of the virgin mary . immediately in front of the main building and facing it , is a copper statue of christ with arms up ##rai ##sed with the legend \" ve ##ni ##te ad me om ##nes \" . next to the main building is the basilica of the sacred heart . immediately behind the basilica is the gr ##otto , a marian place of prayer and reflection . it is a replica of the gr ##otto at lou ##rdes , france where the virgin mary reputed ##ly appeared to saint bern ##ade ##tte so ##ub ##iro ##us in 1858 . at the end of the main drive ( and in a direct line that connects through 3 statues and the gold dome ) , is a simple , modern stone statue of mary . [SEP]\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   tokens: [CLS] to whom did the virgin mary allegedly appear in 1858 in lou ##rdes france ? [SEP] architectural ##ly , the school has a catholic character . atop the main building ' s gold dome is a golden statue of the virgin mary . immediately in front of the main building and facing it , is a copper statue of christ with arms up ##rai ##sed with the legend \" ve ##ni ##te ad me om ##nes \" . next to the main building is the basilica of the sacred heart . immediately behind the basilica is the gr ##otto , a marian place of prayer and reflection . it is a replica of the gr ##otto at lou ##rdes , france where the virgin mary reputed ##ly appeared to saint bern ##ade ##tte so ##ub ##iro ##us in 1858 . at the end of the main drive ( and in a direct line that connects through 3 statues and the gold dome ) , is a simple , modern stone statue of mary . [SEP]\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   token_to_orig_map: 17:0 18:0 19:0 20:1 21:2 22:3 23:4 24:5 25:6 26:6 27:7 28:8 29:9 30:10 31:10 32:10 33:11 34:12 35:13 36:14 37:15 38:16 39:17 40:18 41:19 42:20 43:20 44:21 45:22 46:23 47:24 48:25 49:26 50:27 51:28 52:29 53:30 54:30 55:31 56:32 57:33 58:34 59:35 60:36 61:37 62:38 63:39 64:39 65:39 66:40 67:41 68:42 69:43 70:43 71:43 72:43 73:44 74:45 75:46 76:46 77:46 78:46 79:47 80:48 81:49 82:50 83:51 84:52 85:53 86:54 87:55 88:56 89:57 90:58 91:58 92:59 93:60 94:61 95:62 96:63 97:64 98:65 99:65 100:65 101:66 102:67 103:68 104:69 105:70 106:71 107:72 108:72 109:73 110:74 111:75 112:76 113:77 114:78 115:79 116:79 117:80 118:81 119:81 120:81 121:82 122:83 123:84 124:85 125:86 126:87 127:87 128:88 129:89 130:90 131:91 132:91 133:91 134:92 135:92 136:92 137:92 138:93 139:94 140:94 141:95 142:96 143:97 144:98 145:99 146:100 147:101 148:102 149:102 150:103 151:104 152:105 153:106 154:107 155:108 156:109 157:110 158:111 159:112 160:113 161:114 162:115 163:115 164:115 165:116 166:117 167:118 168:118 169:119 170:120 171:121 172:122 173:123 174:123\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   token_to_orig_map: 17:0 18:0 19:0 20:1 21:2 22:3 23:4 24:5 25:6 26:6 27:7 28:8 29:9 30:10 31:10 32:10 33:11 34:12 35:13 36:14 37:15 38:16 39:17 40:18 41:19 42:20 43:20 44:21 45:22 46:23 47:24 48:25 49:26 50:27 51:28 52:29 53:30 54:30 55:31 56:32 57:33 58:34 59:35 60:36 61:37 62:38 63:39 64:39 65:39 66:40 67:41 68:42 69:43 70:43 71:43 72:43 73:44 74:45 75:46 76:46 77:46 78:46 79:47 80:48 81:49 82:50 83:51 84:52 85:53 86:54 87:55 88:56 89:57 90:58 91:58 92:59 93:60 94:61 95:62 96:63 97:64 98:65 99:65 100:65 101:66 102:67 103:68 104:69 105:70 106:71 107:72 108:72 109:73 110:74 111:75 112:76 113:77 114:78 115:79 116:79 117:80 118:81 119:81 120:81 121:82 122:83 123:84 124:85 125:86 126:87 127:87 128:88 129:89 130:90 131:91 132:91 133:91 134:92 135:92 136:92 137:92 138:93 139:94 140:94 141:95 142:96 143:97 144:98 145:99 146:100 147:101 148:102 149:102 150:103 151:104 152:105 153:106 154:107 155:108 156:109 157:110 158:111 159:112 160:113 161:114 162:115 163:115 164:115 165:116 166:117 167:118 168:118 169:119 170:120 171:121 172:122 173:123 174:123\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   token_is_max_context: 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True 156:True 157:True 158:True 159:True 160:True 161:True 162:True 163:True 164:True 165:True 166:True 167:True 168:True 169:True 170:True 171:True 172:True 173:True 174:True\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   token_is_max_context: 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True 156:True 157:True 158:True 159:True 160:True 161:True 162:True 163:True 164:True 165:True 166:True 167:True 168:True 169:True 170:True 171:True 172:True 173:True 174:True\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   input_ids: 101 2000 3183 2106 1996 6261 2984 9382 3711 1999 8517 1999 10223 26371 2605 1029 102 6549 2135 1010 1996 2082 2038 1037 3234 2839 1012 10234 1996 2364 2311 1005 1055 2751 8514 2003 1037 3585 6231 1997 1996 6261 2984 1012 3202 1999 2392 1997 1996 2364 2311 1998 5307 2009 1010 2003 1037 6967 6231 1997 4828 2007 2608 2039 14995 6924 2007 1996 5722 1000 2310 3490 2618 4748 2033 18168 5267 1000 1012 2279 2000 1996 2364 2311 2003 1996 13546 1997 1996 6730 2540 1012 3202 2369 1996 13546 2003 1996 24665 23052 1010 1037 14042 2173 1997 7083 1998 9185 1012 2009 2003 1037 15059 1997 1996 24665 23052 2012 10223 26371 1010 2605 2073 1996 6261 2984 22353 2135 2596 2000 3002 16595 9648 4674 2061 12083 9711 2271 1999 8517 1012 2012 1996 2203 1997 1996 2364 3298 1006 1998 1999 1037 3622 2240 2008 8539 2083 1017 11342 1998 1996 2751 8514 1007 1010 2003 1037 3722 1010 2715 2962 6231 1997 2984 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   input_ids: 101 2000 3183 2106 1996 6261 2984 9382 3711 1999 8517 1999 10223 26371 2605 1029 102 6549 2135 1010 1996 2082 2038 1037 3234 2839 1012 10234 1996 2364 2311 1005 1055 2751 8514 2003 1037 3585 6231 1997 1996 6261 2984 1012 3202 1999 2392 1997 1996 2364 2311 1998 5307 2009 1010 2003 1037 6967 6231 1997 4828 2007 2608 2039 14995 6924 2007 1996 5722 1000 2310 3490 2618 4748 2033 18168 5267 1000 1012 2279 2000 1996 2364 2311 2003 1996 13546 1997 1996 6730 2540 1012 3202 2369 1996 13546 2003 1996 24665 23052 1010 1037 14042 2173 1997 7083 1998 9185 1012 2009 2003 1037 15059 1997 1996 24665 23052 2012 10223 26371 1010 2605 2073 1996 6261 2984 22353 2135 2596 2000 3002 16595 9648 4674 2061 12083 9711 2271 1999 8517 1012 2012 1996 2203 1997 1996 2364 3298 1006 1998 1999 1037 3622 2240 2008 8539 2083 1017 11342 1998 1996 2751 8514 1007 1010 2003 1037 3722 1010 2715 2962 6231 1997 2984 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   start_position: 130\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   end_position: 137\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   answer: saint bern ##ade ##tte so ##ub ##iro ##us\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   start_position: 130\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   end_position: 137\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   answer: saint bern ##ade ##tte so ##ub ##iro ##us\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   *** Example ***\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   unique_id: 1000000000\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   example_index: 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   doc_span_index: 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   tokens: [CLS] to whom did the virgin mary allegedly appear in 1858 in lou ##rdes france ? [SEP] architectural ##ly , the school has a catholic character . atop the main building ' s gold dome is a golden statue of the virgin mary . immediately in front of the main building and facing it , is a copper statue of christ with arms up ##rai ##sed with the legend \" ve ##ni ##te ad me om ##nes \" . next to the main building is the basilica of the sacred heart . immediately behind the basilica is the gr ##otto , a marian place of prayer and reflection . it is a replica of the gr ##otto at lou ##rdes , france where the virgin mary reputed ##ly appeared to saint bern ##ade ##tte so ##ub ##iro ##us in 1858 . at the end of the main drive ( and in a direct line that connects through 3 statues and the gold dome ) , is a simple , modern stone statue of mary . [SEP]\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   token_to_orig_map: 17:0 18:0 19:0 20:1 21:2 22:3 23:4 24:5 25:6 26:6 27:7 28:8 29:9 30:10 31:10 32:10 33:11 34:12 35:13 36:14 37:15 38:16 39:17 40:18 41:19 42:20 43:20 44:21 45:22 46:23 47:24 48:25 49:26 50:27 51:28 52:29 53:30 54:30 55:31 56:32 57:33 58:34 59:35 60:36 61:37 62:38 63:39 64:39 65:39 66:40 67:41 68:42 69:43 70:43 71:43 72:43 73:44 74:45 75:46 76:46 77:46 78:46 79:47 80:48 81:49 82:50 83:51 84:52 85:53 86:54 87:55 88:56 89:57 90:58 91:58 92:59 93:60 94:61 95:62 96:63 97:64 98:65 99:65 100:65 101:66 102:67 103:68 104:69 105:70 106:71 107:72 108:72 109:73 110:74 111:75 112:76 113:77 114:78 115:79 116:79 117:80 118:81 119:81 120:81 121:82 122:83 123:84 124:85 125:86 126:87 127:87 128:88 129:89 130:90 131:91 132:91 133:91 134:92 135:92 136:92 137:92 138:93 139:94 140:94 141:95 142:96 143:97 144:98 145:99 146:100 147:101 148:102 149:102 150:103 151:104 152:105 153:106 154:107 155:108 156:109 157:110 158:111 159:112 160:113 161:114 162:115 163:115 164:115 165:116 166:117 167:118 168:118 169:119 170:120 171:121 172:122 173:123 174:123\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   *** Example ***\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   token_is_max_context: 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True 156:True 157:True 158:True 159:True 160:True 161:True 162:True 163:True 164:True 165:True 166:True 167:True 168:True 169:True 170:True 171:True 172:True 173:True 174:True\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   unique_id: 1000000000\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   example_index: 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   doc_span_index: 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   tokens: [CLS] to whom did the virgin mary allegedly appear in 1858 in lou ##rdes france ? [SEP] architectural ##ly , the school has a catholic character . atop the main building ' s gold dome is a golden statue of the virgin mary . immediately in front of the main building and facing it , is a copper statue of christ with arms up ##rai ##sed with the legend \" ve ##ni ##te ad me om ##nes \" . next to the main building is the basilica of the sacred heart . immediately behind the basilica is the gr ##otto , a marian place of prayer and reflection . it is a replica of the gr ##otto at lou ##rdes , france where the virgin mary reputed ##ly appeared to saint bern ##ade ##tte so ##ub ##iro ##us in 1858 . at the end of the main drive ( and in a direct line that connects through 3 statues and the gold dome ) , is a simple , modern stone statue of mary . [SEP]\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   input_ids: 101 2000 3183 2106 1996 6261 2984 9382 3711 1999 8517 1999 10223 26371 2605 1029 102 6549 2135 1010 1996 2082 2038 1037 3234 2839 1012 10234 1996 2364 2311 1005 1055 2751 8514 2003 1037 3585 6231 1997 1996 6261 2984 1012 3202 1999 2392 1997 1996 2364 2311 1998 5307 2009 1010 2003 1037 6967 6231 1997 4828 2007 2608 2039 14995 6924 2007 1996 5722 1000 2310 3490 2618 4748 2033 18168 5267 1000 1012 2279 2000 1996 2364 2311 2003 1996 13546 1997 1996 6730 2540 1012 3202 2369 1996 13546 2003 1996 24665 23052 1010 1037 14042 2173 1997 7083 1998 9185 1012 2009 2003 1037 15059 1997 1996 24665 23052 2012 10223 26371 1010 2605 2073 1996 6261 2984 22353 2135 2596 2000 3002 16595 9648 4674 2061 12083 9711 2271 1999 8517 1012 2012 1996 2203 1997 1996 2364 3298 1006 1998 1999 1037 3622 2240 2008 8539 2083 1017 11342 1998 1996 2751 8514 1007 1010 2003 1037 3722 1010 2715 2962 6231 1997 2984 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   token_to_orig_map: 17:0 18:0 19:0 20:1 21:2 22:3 23:4 24:5 25:6 26:6 27:7 28:8 29:9 30:10 31:10 32:10 33:11 34:12 35:13 36:14 37:15 38:16 39:17 40:18 41:19 42:20 43:20 44:21 45:22 46:23 47:24 48:25 49:26 50:27 51:28 52:29 53:30 54:30 55:31 56:32 57:33 58:34 59:35 60:36 61:37 62:38 63:39 64:39 65:39 66:40 67:41 68:42 69:43 70:43 71:43 72:43 73:44 74:45 75:46 76:46 77:46 78:46 79:47 80:48 81:49 82:50 83:51 84:52 85:53 86:54 87:55 88:56 89:57 90:58 91:58 92:59 93:60 94:61 95:62 96:63 97:64 98:65 99:65 100:65 101:66 102:67 103:68 104:69 105:70 106:71 107:72 108:72 109:73 110:74 111:75 112:76 113:77 114:78 115:79 116:79 117:80 118:81 119:81 120:81 121:82 122:83 123:84 124:85 125:86 126:87 127:87 128:88 129:89 130:90 131:91 132:91 133:91 134:92 135:92 136:92 137:92 138:93 139:94 140:94 141:95 142:96 143:97 144:98 145:99 146:100 147:101 148:102 149:102 150:103 151:104 152:105 153:106 154:107 155:108 156:109 157:110 158:111 159:112 160:113 161:114 162:115 163:115 164:115 165:116 166:117 167:118 168:118 169:119 170:120 171:121 172:122 173:123 174:123\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   token_is_max_context: 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True 156:True 157:True 158:True 159:True 160:True 161:True 162:True 163:True 164:True 165:True 166:True 167:True 168:True 169:True 170:True 171:True 172:True 173:True 174:True\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   start_position: 130\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   input_ids: 101 2000 3183 2106 1996 6261 2984 9382 3711 1999 8517 1999 10223 26371 2605 1029 102 6549 2135 1010 1996 2082 2038 1037 3234 2839 1012 10234 1996 2364 2311 1005 1055 2751 8514 2003 1037 3585 6231 1997 1996 6261 2984 1012 3202 1999 2392 1997 1996 2364 2311 1998 5307 2009 1010 2003 1037 6967 6231 1997 4828 2007 2608 2039 14995 6924 2007 1996 5722 1000 2310 3490 2618 4748 2033 18168 5267 1000 1012 2279 2000 1996 2364 2311 2003 1996 13546 1997 1996 6730 2540 1012 3202 2369 1996 13546 2003 1996 24665 23052 1010 1037 14042 2173 1997 7083 1998 9185 1012 2009 2003 1037 15059 1997 1996 24665 23052 2012 10223 26371 1010 2605 2073 1996 6261 2984 22353 2135 2596 2000 3002 16595 9648 4674 2061 12083 9711 2271 1999 8517 1012 2012 1996 2203 1997 1996 2364 3298 1006 1998 1999 1037 3622 2240 2008 8539 2083 1017 11342 1998 1996 2751 8514 1007 1010 2003 1037 3722 1010 2715 2962 6231 1997 2984 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   end_position: 137\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   answer: saint bern ##ade ##tte so ##ub ##iro ##us\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   start_position: 130\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   end_position: 137\u001b[0m\n",
      "\u001b[34m08/23/2020 20:47:46 - INFO - __main__ -   answer: saint bern ##ade ##tte so ##ub ##iro ##us\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34mDLL 2020-08-23 20:53:37.641220 - PARAMETER Cached_train features_file : /workspace/bert/data/squad/v1.1/train-v1.1.json_bert-base-uncased_512_128_64 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:53:47.434952 - PARAMETER train_start : True \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:53:47.435093 - PARAMETER training_samples : 87599 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:53:47.435141 - PARAMETER training_features : 87748 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:53:47.435178 - PARAMETER train_batch_size : 16 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:53:47.435213 - PARAMETER steps : 1368.0 \u001b[0m\n",
      "\u001b[34mGradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0\u001b[0m\n",
      "\u001b[34mGradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0\u001b[0m\n",
      "\u001b[34mGradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0\u001b[0m\n",
      "\u001b[34mGradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 32768.0\u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:53:51.069260 - Training Epoch: 0 Training Iteration: 1  step_loss : 6.265625  learning_rate : 2.1929824561403507e-07 \u001b[0m\n",
      "\u001b[34mGradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0\u001b[0m\n",
      "\u001b[34mGradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0\u001b[0m\n",
      "\u001b[34mGradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0\u001b[0m\n",
      "\u001b[34mGradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 16384.0\u001b[0m\n",
      "\u001b[34mGradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0\u001b[0m\n",
      "\u001b[34mGradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0\u001b[0m\n",
      "\u001b[34mGradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0\u001b[0m\n",
      "\u001b[34mGradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 8192.0\u001b[0m\n",
      "\u001b[34mGradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0\u001b[0m\n",
      "\u001b[34mGradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0\u001b[0m\n",
      "\u001b[34mGradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0\u001b[0m\n",
      "\u001b[34mGradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 4096.0\u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:53:59.952869 - Training Epoch: 0 Training Iteration: 51  step_loss : 2.54296875  learning_rate : 1.118421052631579e-05 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:54:08.630988 - Training Epoch: 0 Training Iteration: 101  step_loss : 1.5283203125  learning_rate : 2.2149122807017543e-05 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:54:17.310736 - Training Epoch: 0 Training Iteration: 151  step_loss : 1.751953125  learning_rate : 2.965399610136452e-05 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:54:25.992293 - Training Epoch: 0 Training Iteration: 201  step_loss : 1.4931640625  learning_rate : 2.8435672514619884e-05 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:54:34.676956 - Training Epoch: 0 Training Iteration: 251  step_loss : 1.658203125  learning_rate : 2.7217348927875245e-05 \u001b[0m\n",
      "\u001b[34m#015Iteration:   0%|          | 0/1372 [00:00<?, ?it/s]#015Iteration:   0%|          | 1/1372 [00:01<25:35,  1.12s/it]#015Iteration:   0%|          | 2/1372 [00:01<20:28,  1.12it/s]#015Iteration:   0%|          | 3/1372 [00:01<15:28,  1.47it/s]#015Iteration:   0%|          | 4/1372 [00:01<11:58,  1.90it/s]#015Iteration:   0%|          | 5/1372 [00:02<09:35,  2.37it/s]#015Iteration:   0%|          | 6/1372 [00:02<07:53,  2.89it/s]#015Iteration:   1%|          | 7/1372 [00:02<06:42,  3.39it/s]#015Iteration:   1%|          | 8/1372 [00:02<05:52,  3.87it/s]#015Iteration:   1%|          | 9/1372 [00:02<05:17,  4.30it/s]#015Iteration:   1%|          | 10/1372 [00:02<04:52,  4.65it/s]#015Iteration:   1%|          | 11/1372 [00:03<04:35,  4.94it/s]#015Iteration:   1%|          | 12/1372 [00:03<04:23,  5.16it/s]#015Iteration:   1%|          | 13/1372 [00:03<04:15,  5.32it/s]#015Iteration:   1%|          | 14/1372 [00:03<04:09,  5.44it/s]#015Iteration:   1%|          | 15/1372 [00:03<04:04,  5.54it/s]#015Iteration:   1%|          | 16/1372 [00:03<04:01,  5.61it/s]#015Iteration:   1%|          | 17/1372 [00:04<03:59,  5.65it/s]#015Iteration:   1%|▏         | 18/1372 [00:04<03:58,  5.69it/s]#015Iteration:   1%|▏         | 19/1372 [00:04<03:56,  5.71it/s]#015Iteration:   1%|▏         | 20/1372 [00:04<03:56,  5.72it/s]#015Iteration:   2%|▏         | 21/1372 [00:04<03:55,  5.74it/s]#015Iteration:   2%|▏         | 22/1372 [00:04<03:55,  5.74it/s]#015Iteration:   2%|▏         | 23/1372 [00:05<03:54,  5.74it/s]#015Iteration:   2%|▏         | 24/1372 [00:05<03:54,  5.75it/s]#015Iteration:   2%|▏         | 25/1372 [00:05<03:54,  5.75it/s]#015Iteration:   2%|▏         | 26/1372 [00:05<03:54,  5.74it/s]#015Iteration:   2%|▏         | 27/1372 [00:05<03:53,  5.75it/s]#015Iteration:   2%|▏         | 28/1372 [00:06<03:53,  5.75it/s]#015Iteration:   2%|▏         | 29/1372 [00:06<03:53,  5.75it/s]#015Iteration:   2%|▏         | 30/1372 [00:06<03:54,  5.73it/s]#015Iteration:   2%|▏         | 31/1372 [00:06<03:53,  5.73it/s]#015Iteration:   2%|▏         | 32/1372 [00:06<03:53,  5.74it/s]#015Iteration:   2%|▏         | 33/1372 [00:06<03:53,  5.75it/s]#015Iteration:   2%|▏         | 34/1372 [00:07<03:53,  5.72it/s]#015Iteration:   3%|▎         | 35/1372 [00:07<03:53,  5.73it/s]#015Iteration:   3%|▎         | 36/1372 [00:07<03:53,  5.73it/s]#015Iteration:   3%|▎         | 37/1372 [00:07<03:52,  5.74it/s]#015Iteration:   3%|▎         | 38/1372 [00:07<03:52,  5.74it/s]#015Iteration:   3%|▎         | 39/1372 [00:07<03:51,  5.75it/s]#015Iteration:   3%|▎         | 40/1372 [00:08<03:51,  5.75it/s]#015Iteration:   3%|▎         | 41/1372 [00:08<03:51,  5.76it/s]#015Iteration:   3%|▎         | 42/1372 [00:08<03:51,  5.76it/s]#015Iteration:   3%|▎         | 43/1372 [00:08<03:51,  5.75it/s]#015Iteration:   3%|▎         | 44/1372 [00:08<03:50,  5.76it/s]#015Iteration:   3%|▎         | 45/1372 [00:08<03:50,  5.75it/s]#015Iteration:   3%|▎         | 46/1372 [00:09<03:50,  5.76it/s]#015Iteration:   3%|▎         | 47/1372 [00:09<03:50,  5.76it/s]#015Iteration:   3%|▎         | 48/1372 [00:09<03:49,  5.76it/s]#015Iteration:   4%|▎         | 49/1372 [00:09<03:49,  5.76it/s]#015Iteration:   4%|▎         | 50/1372 [00:09<03:49,  5.76it/s]#015Iteration:   4%|▎         | 51/1372 [00:10<03:49,  5.76it/s]#015Iteration:   4%|▍         | 52/1372 [00:10<03:49,  5.76it/s]#015Iteration:   4%|▍         | 53/1372 [00:10<03:49,  5.76it/s]#015Iteration:   4%|▍         | 54/1372 [00:10<03:48,  5.76it/s]#015Iteration:   4%|▍         | 55/1372 [00:10<03:48,  5.76it/s]#015Iteration:   4%|▍         | 56/1372 [00:10<03:48,  5.76it/s]#015Iteration:   4%|▍         | 57/1372 [00:11<03:48,  5.77it/s]#015Iteration:   4%|▍         | 58/1372 [00:11<03:48,  5.76it/s]#015Iteration:   4%|▍         | 59/1372 [00:11<03:47,  5.76it/s]#015Iteration:   4%|▍         | 60/1372 [00:11<03:47,  5.76it/s]#015Iteration:   4%|▍         | 61/1372 [00:11<03:47,  5.76it/s]#015Iteration:   5%|▍         | 62/1372 [00:11<03:47,  5.76it/s]#015Iteration:   5%|▍         | 63/1372 [00:12<03:47,  5.76it/s]#015Iteration:   5%|▍         | 64/1372 [00:12<03:47,  5.76it/s]#015Iteration:   5%|▍         | 65/1372 [00:12<03:47,  5.75it/s]#015Iteration:   5%|▍         | 66/1372 [00:12<03:46,  5.75it/s]#015Iteration:   5%|▍         | 67/1372 [00:12<03:46,  5.75it/s]#015Iteration:   5%|▍         | 68/1372 [00:12<03:46,  5.76it/s]#015Iteration:   5%|▌         | 69/1372 [00:13<03:46,  5.76it/s]#015Iteration:   5%|▌         | 70/1372 [00:13<03:45,  5.76it/s]#015Iteration:   5%|▌         | 71/1372 [00:13<03:45,  5.76it/s]#015Iteration:   5%|▌         | 72/1372 [00:13<03:45,  5.76it/s]#015Iteration:   5%|▌         | 73/1372 [00:13<03:45,  5.76it/s]#015Iteration:   5%|▌         | 74/1372 [00:13<03:45,  5.76it/s]#015Iteration:   5%|▌         | 75/1372 [00:14<03:45,  5.76it/s]#015Iteration:   6%|▌         | 76/1372 [00:14<03:44,  5.76it/s]#015Iteration:   6%|▌         | 77/1372 [00:14<03:44,  5.76it/s]#015Iteration:   6%|▌         | 78/1372 [00:14<03:44,  5.77it/s]#015Iteration:   6%|▌         | 79/1372 [00:14<03:44,  5.77it/s]#015Iteration:   6%|▌         | 80/1372 [00:15<03:44,  5.77it/s]#015Iteration:   6%|▌         | 81/1372 [00:15<03:43,  5.77it/s]#015Iteration:   6%|▌         | 82/1372 [00:15<03:43,  5.77it/s]#015Iteration:   6%|▌         | 83/1372 [00:15<03:43,  5.77it/s]#015Iteration:   6%|▌         | 84/1372 [00:15<03:43,  5.76it/s]#015Iteration:   6%|▌         | 85/1372 [00:15<03:43,  5.76it/s]#015Iteration:   6%|▋         | 86/1372 [00:16<03:43,  5.76it/s]#015Iteration:   6%|▋         | 87/1372 [00:16<03:42,  5.76it/s]#015Iteration:   6%|▋         | 88/1372 [00:16<03:42,  5.76it/s]#015Iteration:   6%|▋         | 89/1372 [00:16<03:42,  5.76it/s]#015Iteration:   7%|▋         | 90/1372 [00:16<03:42,  5.76it/s]#015Iteration:   7%|▋         | 91/1372 [00:16<03:42,  5.76it/s]#015Iteration:   7%|▋         | 92/1372 [00:17<03:42,  5.76it/s]#015Iteration:   7%|▋         | 93/1372 [00:17<03:41,  5.76it/s]#015Iteration:   7%|▋         | 94/1372 [00:17<03:41,  5.76it/s]#015Iteration:   7%|▋         | 95/1372 [00:17<03:41,  5.76it/s]#015Iteration:   7%|▋         | 96/1372 [00:17<03:41,  5.76it/s]#015Iteration:   7%|▋         | 97/1372 [00:17<03:41,  5.76it/s]#015Iteration:   7%|▋         | 98/1372 [00:18<03:41,  5.76it/s]#015Iteration:   7%|▋         | 99/1372 [00:18<03:40,  5.76it/s]#015Iteration:   7%|▋         | 100/1372 [00:18<03:40,  5.77it/s]#015Iteration:   7%|▋         | 101/1372 [00:18<03:40,  5.76it/s]#015Iteration:   7%|▋         | 102/1372 [00:18<03:40,  5.76it/s]#015Iteration:   8%|▊         | 103/1372 [00:19<03:40,  5.76it/s]#015Iteration:   8%|▊         | 104/1372 [00:19<03:40,  5.76it/s]#015Iteration:   8%|▊         | 105/1372 [00:19<03:39,  5.76it/s]#015Iteration:   8%|▊         | 106/1372 [00:19<03:39,  5.76it/s]#015Iteration:   8%|▊         | 107/1372 [00:19<03:39,  5.76it/s]#015Iteration:   8%|▊         | 108/1372 [00:19<03:39,  5.76it/s]#015Iteration:   8%|▊         | 109/1372 [00:20<03:39,  5.76it/s]#015Iteration:   8%|▊         | 110/1372 [00:20<03:39,  5.76it/s]#015Iteration:   8%|▊         | 111/1372 [00:20<03:39,  5.76it/s]#015Iteration:   8%|▊         | 112/1372 [00:20<03:38,  5.76it/s]#015Iteration:   8%|▊         | 113/1372 [00:20<03:38,  5.76it/s]#015Iteration:   8%|▊         | 114/1372 [00:20<03:38,  5.76it/s]#015Iteration:   8%|▊         | 115/1372 [00:21<03:38,  5.76it/s]#015Iteration:   8%|▊         | 116/1372 [00:21<03:37,  5.76it/s]#015Iteration:   9%|▊         | 117/1372 [00:21<03:37,  5.76it/s]#015Iteration:   9%|▊         | 118/1372 [00:21<03:37,  5.76it/s]#015Iteration:   9%|▊         | 119/1372 [00:21<03:37,  5.76it/s]#015Iteration:   9%|▊         | 120/1372 [00:21<03:37,  5.76it/s]#015Iteration:   9%|▉         | 121/1372 [00:22<03:37,  5.76it/s]#015Iteration:   9%|▉         | 122/1372 [00:22<03:36,  5.77it/s]#015Iteration:   9%|▉         | 123/1372 [00:22<03:36,  5.76it/s]#015Iteration:   9%|▉         | 124/1372 [00:22<03:36,  5.76it/s]#015Iteration:   9%|▉         | 125/1372 [00:22<03:36,  5.76it/s]#015Iteration:   9%|▉         | 126/1372 [00:23<03:36,  5.76it/s]#015Iteration:   9%|▉         | 127/1372 [00:23<03:36,  5.76it/s]#015Iteration:   9%|▉         | 128/1372 [00:23<03:35,  5.76it/s]#015Iteration:   9%|▉         | 129/1372 [00:23<03:35,  5.76it/s]#015Iteration:   9%|▉         | 130/1372 [00:23<03:35,  5.76it/s]#015Iteration:  10%|▉         | 131/1372 [00:23<03:35,  5.76it/s]#015Iteration:  10%|▉         | 132/1372 [00:24<03:35,  5.76it/s]#015Iteration:  10%|▉         | 133/1372 [00:24<03:35,  5.76it/s]#015Iteration:  10%|▉         | 134/1372 [00:24<03:34,  5.76it/s]#015Iteration:  10%|▉         | 135/1372 [00:24<03:34,  5.76it/s]#015Iteration:  10%|▉         | 136/1372 [00:24<03:34,  5.76it/s]#015Iteration:  10%|▉         | 137/1372 [00:24<03:34,  5.76it/s]#015Iteration:  10%|█         | 138/1372 [00:25<03:34,  5.76it/s]#015Iteration:  10%|█         | 139/1372 [00:25<03:34,  5.76it/s]#015Iteration:  10%|█         | 140/1372 [00:25<03:33,  5.76it/s]#015Iteration:  10%|█         | 141/1372 [00:25<03:33,  5.76it/s]#015Iteration:  10%|█         | 142/1372 [00:25<03:33,  5.76it/s]#015Iteration:  10%|█         | 143/1372 [00:25<03:33,  5.76it/s]#015Iteration:  10%|█         | 144/1372 [00:26<03:33,  5.76it/s]#015Iteration:  11%|█         | 145/1372 [00:26<03:32,  5.76it/s]#015Iteration:  11%|█         | 146/1372 [00:26<03:32,  5.76it/s]#015Iteration:  11%|█         | 147/1372 [00:26<03:32,  5.76it/s]#015Iteration:  11%|█         | 148/1372 [00:26<03:32,  5.76it/s]#015Iteration:  11%|█         | 149/1372 [00:27<03:32,  5.76it/s]#015Iteration:  11%|█         | 150/1372 [00:27<03:32,  5.76it/s]#015Iteration:  11%|█         | 151/1372 [00:27<03:32,  5.76it/s]#015Iteration:  11%|█         | 152/1372 [00:27<03:31,  5.76it/s]#015Iteration:  11%|█         | 153/1372 [00:27<03:31,  5.76it/s]#015Iteration:  11%|█         | 154/1372 [00:27<03:31,  5.76it/s]#015Iteration:  11%|█▏        | 155/1372 [00:28<03:31,  5.75it/s]#015Iteration:  11%|█▏        | 156/1372 [00:28<03:31,  5.75it/s]#015Iteration:  11%|█▏        | 157/1372 [00:28<03:31,  5.76it/s]#015Iteration:  12%|█▏        | 158/1372 [00:28<03:30,  5.76it/s]#015Iteration:  12%|█▏        | 159/1372 [00:28<03:30,  5.75it/s]#015Iteration:  12%|█▏        | 160/1372 [00:28<03:30,  5.76it/s]#015Iteration:  12%|█▏        | 161/1372 [00:29<03:30,  5.76it/s]#015Iteration:  12%|█▏        | 162/1372 [00:29<03:30,  5.76it/s]#015Iteration:  12%|█▏        | 163/1372 [00:29<03:29,  5.76it/s]#015Iteration:  12%|█▏        | 164/1372 [00:29<03:29,  5.76it/s]#015Iteration:  12%|█▏        | 165/1372 [00:29<03:29,  5.76it/s]#015Iteration:  12%|█▏        | 166/1372 [00:29<03:29,  5.76it/s]#015Iteration:  12%|█▏        | 167/1372 [00:30<03:29,  5.76it/s]#015Iteration:  12%|█▏        | 168/1372 [00:30<03:28,  5.76it/s]#015Iteration:  12%|█▏        | 169/1372 [00:30<03:28,  5.76it/s]#015Iteration:  12%|█▏        | 170/1372 [00:30<03:28,  5.76it/s]#015Iteration:  12%|█▏        | 171/1372 [00:30<03:28,  5.76it/s]#015Iteration:  13%|█▎        | 172/1372 [00:31<03:28,  5.76it/s]#015Iteration:  13%|█▎        | 173/1372 [00:31<03:28,  5.76it/s]#015Iteration:  13%|█▎        | 174/1372 [00:31<03:28,  5.76it/s]#015Iteration:  13%|█▎        | 175/1372 [00:31<03:27,  5.76it/s]#015Iteration:  13%|█▎        | 176/1372 [00:31<03:27,  5.76it/s]#015Iteration:  13%|█▎        | 177/1372 [00:31<03:27,  5.76it/s]#015Iteration:  13%|█▎        | 178/1372 [00:32<03:27,  5.76it/s]#015Iteration:  13%|█▎        | 179/1372 [00:32<03:27,  5.76it/s]#015Iteration:  13%|█▎        | 180/1372 [00:32<03:26,  5.76it/s]#015Iteration:  13%|█▎        | 181/1372 [00:32<03:26,  5.76it/s]#015Iteration:  13%|█▎        | 182/1372 [00:32<03:26,  5.76it/s]#015Iteration:  13%|█▎        | 183/1372 [00:32<03:26,  5.76it/s]#015Iteration:  13%|█▎        | 184/1372 [00:33<03:26,  5.76it/s]#015Iteration:  13%|█▎        | 185/1372 [00:33<03:25,  5.76it/s]#015Iteration:  14%|█▎        | 186/1372 [00:33<03:25,  5.76it/s]#015Iteration:  14%|█▎        | 187/1372 [00:33<03:25,  5.76it/s]#015Iteration:  14%|█▎        | 188/1372 [00:33<03:25,  5.76it/s]#015Iteration:  14%|█▍        | 189/1372 [00:33<03:25,  5.76it/s]#015Iteration:  14%|█▍        | 190/1372 [00:34<03:25,  5.76it/s]#015Iteration:  14%|█▍        | 191/1372 [00:34<03:25,  5.76it/s]#015Iteration:  14%|█▍        | 192/1372 [00:34<03:24,  5.76it/s]#015Iteration:  14%|█▍        | 193/1372 [00:34<03:24,  5.76it/s]#015Iteration:  14%|█▍        | 194/1372 [00:34<03:24,  5.76it/s]#015Iteration:  14%|█▍        | 195/1372 [00:35<03:24,  5.76it/s]#015Iteration:  14%|█▍        | 196/1372 [00:35<03:24,  5.76it/s]#015Iteration:  14%|█▍        | 197/1372 [00:35<03:23,  5.76it/s]#015Iteration:  14%|█▍        | 198/1372 [00:35<03:23,  5.76it/s]#015Iteration:  15%|█▍        | 199/1372 [00:35<03:23,  5.76it/s]#015Iteration:  15%|█▍        | 200/1372 [00:35<03:23,  5.77it/s]#015Iteration:  15%|█▍        | 201/1372 [00:36<03:23,  5.76it/s]#015Iteration:  15%|█▍        | 202/1372 [00:36<03:23,  5.76it/s]#015Iteration:  15%|█▍        | 203/1372 [00:36<03:23,  5.76it/s]#015Iteration:  15%|█▍        | 204/1372 [00:36<03:22,  5.76it/s]#015Iteration:  15%|█▍        | 205/1372 [00:36<03:22,  5.76it/s]#015Iteration:  15%|█▌        | 206/1372 [00:36<03:22,  5.76it/s]#015Iteration:  15%|█▌        | 207/1372 [00:37<03:22,  5.76it/s]#015Iteration:  15%|█▌        | 208/1372 [00:37<03:21,  5.76it/s]#015Iteration:  15%|█▌        | 209/1372 [00:37<03:21,  5.76it/s]#015Iteration:  15%|█▌        | 210/1372 [00:37<03:21,  5.76it/s]#015Iteration:  15%|█▌        | 211/1372 [00:37<03:21,  5.76it/s]#015Iteration:  15%|█▌        | 212/1372 [00:37<03:21,  5.76it/s]#015Iteration:  16%|█▌        | 213/1372 [00:38<03:21,  5.75it/s]#015Iteration:  16%|█▌        | 214/1372 [00:38<03:21,  5.75it/s]#015Iteration:  16%|█▌        | 215/1372 [00:38<03:21,  5.75it/s]#015Iteration:  16%|█▌        | 216/1372 [00:38<03:21,  5.75it/s]#015Iteration:  16%|█▌        | 217/1372 [00:38<03:20,  5.76it/s]#015Iteration:  16%|█▌        | 218/1372 [00:38<03:20,  5.76it/s]#015Iteration:  16%|█▌        | 219/1372 [00:39<03:20,  5.76it/s]#015Iteration:  16%|█▌        | 220/1372 [00:39<03:19,  5.76it/s]#015Iteration:  16%|█▌        | 221/1372 [00:39<03:19,  5.76it/s]#015Iteration:  16%|█▌        | 222/1372 [00:39<03:19,  5.76it/s]#015Iteration:  16%|█▋        | 223/1372 [00:39<03:19,  5.76it/s]#015Iteration:  16%|█▋        | 224/1372 [00:40<03:19,  5.76it/s]#015Iteration:  16%|█▋        | 225/1372 [00:40<03:19,  5.76it/s]#015Iteration:  16%|█▋        | 226/1372 [00:40<03:19,  5.76it/s]#015Iteration:  17%|█▋        | 227/1372 [00:40<03:18,  5.76it/s]#015Iteration:  17%|█▋        | 228/1372 [00:40<03:18,  5.75it/s]#015Iteration:  17%|█▋        | 229/1372 [00:40<03:18,  5.75it/s]#015Iteration:  17%|█▋        | 230/1372 [00:41<03:18,  5.75it/s]#015Iteration:  17%|█▋        | 231/1372 [00:41<03:18,  5.76it/s]#015Iteration:  17%|█▋        | 232/1372 [00:41<03:18,  5.76it/s]#015Iteration:  17%|█▋        | 233/1372 [00:41<03:17,  5.76it/s]#015Iteration:  17%|█▋        | 234/1372 [00:41<03:17,  5.76it/s]#015Iteration:  17%|█▋        | 235/1372 [00:41<03:17,  5.76it/s]#015Iteration:  17%|█▋        | 236/1372 [00:42<03:17,  5.76it/s]#015Iteration:  17%|█▋        | 237/1372 [00:42<03:17,  5.76it/s]#015Iteration:  17%|█▋        | 238/1372 [00:42<03:16,  5.76it/s]#015Iteration:  17%|█▋        | 239/1372 [00:42<03:16,  5.76it/s]#015Iteration:  17%|█▋        | 240/1372 [00:42<03:16,  5.76it/s]#015Iteration:  18%|█▊        | 241/1372 [00:42<03:16,  5.76it/s]#015Iteration:  18%|█▊        | 242/1372 [00:43<03:16,  5.76it/s]#015Iteration:  18%|█▊        | 243/1372 [00:43<03:16,  5.75it/s]#015Iteration:  18%|█▊        | 244/1372 [00:43<03:16,  5.75it/s]#015Iteration:  18%|█▊        | 245/1372 [00:43<03:15,  5.75it/s]#015Iteration:  18%|█▊        | 246/1372 [00:43<03:15,  5.75it/s]#015Iteration:  18%|█▊        | 247/1372 [00:44<03:15,  5.75it/s]#015Iteration:  18%|█▊        | 248/1372 [00:44<03:15,  5.75it/s]#015Iteration:  18%|█▊        | 249/1372 [00:44<03:15,  5.76it/s]#015Iteration:  18%|█▊        | 250/1372 [00:44<03:14,  5.76it/s]#015Iteration:  18%|█▊        | 251/1372 [00:44<03:14,  5.76it/s]#015Iteration:  18%|█▊        | 252/1372 [00:44<03:14,  5.76it/s]#015Iteration:  18%|█▊        | 253/1372 [00:45<03:14,  5.76it/s]#015Iteration:  19%|█▊        | 254/1372 [00:45<03:14,  5.76it/s]#015Iteration:  19%|�\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34mDLL 2020-08-23 20:54:43.362326 - Training Epoch: 0 Training Iteration: 301  step_loss : 2.06640625  learning_rate : 2.59990253411306e-05 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:54:52.050100 - Training Epoch: 0 Training Iteration: 351  step_loss : 1.203125  learning_rate : 2.478070175438597e-05 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:55:00.742000 - Training Epoch: 0 Training Iteration: 401  step_loss : 1.94921875  learning_rate : 2.3562378167641325e-05 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:55:09.443747 - Training Epoch: 0 Training Iteration: 451  step_loss : 0.939453125  learning_rate : 2.2344054580896682e-05 \u001b[0m\n",
      "\u001b[34m��▊        | 255/1372 [00:45<03:13,  5.76it/s]#015Iteration:  19%|█▊        | 256/1372 [00:45<03:13,  5.76it/s]#015Iteration:  19%|█▊        | 257/1372 [00:45<03:13,  5.76it/s]#015Iteration:  19%|█▉        | 258/1372 [00:45<03:13,  5.76it/s]#015Iteration:  19%|█▉        | 259/1372 [00:46<03:13,  5.76it/s]#015Iteration:  19%|█▉        | 260/1372 [00:46<03:12,  5.76it/s]#015Iteration:  19%|█▉        | 261/1372 [00:46<03:12,  5.76it/s]#015Iteration:  19%|█▉        | 262/1372 [00:46<03:12,  5.76it/s]#015Iteration:  19%|█▉        | 263/1372 [00:46<03:12,  5.75it/s]#015Iteration:  19%|█▉        | 264/1372 [00:46<03:12,  5.75it/s]#015Iteration:  19%|█▉        | 265/1372 [00:47<03:12,  5.76it/s]#015Iteration:  19%|█▉        | 266/1372 [00:47<03:12,  5.76it/s]#015Iteration:  19%|█▉        | 267/1372 [00:47<03:11,  5.76it/s]#015Iteration:  20%|█▉        | 268/1372 [00:47<03:11,  5.75it/s]#015Iteration:  20%|█▉        | 269/1372 [00:47<03:11,  5.75it/s]#015Iteration:  20%|█▉        | 270/1372 [00:48<03:11,  5.76it/s]#015Iteration:  20%|█▉        | 271/1372 [00:48<03:11,  5.75it/s]#015Iteration:  20%|█▉        | 272/1372 [00:48<03:11,  5.75it/s]#015Iteration:  20%|█▉        | 273/1372 [00:48<03:10,  5.75it/s]#015Iteration:  20%|█▉        | 274/1372 [00:48<03:10,  5.76it/s]#015Iteration:  20%|██        | 275/1372 [00:48<03:10,  5.76it/s]#015Iteration:  20%|██        | 276/1372 [00:49<03:10,  5.76it/s]#015Iteration:  20%|██        | 277/1372 [00:49<03:10,  5.76it/s]#015Iteration:  20%|██        | 278/1372 [00:49<03:09,  5.76it/s]#015Iteration:  20%|██        | 279/1372 [00:49<03:09,  5.76it/s]#015Iteration:  20%|██        | 280/1372 [00:49<03:09,  5.76it/s]#015Iteration:  20%|██        | 281/1372 [00:49<03:09,  5.76it/s]#015Iteration:  21%|██        | 282/1372 [00:50<03:09,  5.76it/s]#015Iteration:  21%|██        | 283/1372 [00:50<03:08,  5.76it/s]#015Iteration:  21%|██        | 284/1372 [00:50<03:08,  5.76it/s]#015Iteration:  21%|██        | 285/1372 [00:50<03:08,  5.76it/s]#015Iteration:  21%|██        | 286/1372 [00:50<03:08,  5.76it/s]#015Iteration:  21%|██        | 287/1372 [00:50<03:08,  5.76it/s]#015Iteration:  21%|██        | 288/1372 [00:51<03:08,  5.76it/s]#015Iteration:  21%|██        | 289/1372 [00:51<03:08,  5.76it/s]#015Iteration:  21%|██        | 290/1372 [00:51<03:07,  5.76it/s]#015Iteration:  21%|██        | 291/1372 [00:51<03:07,  5.76it/s]#015Iteration:  21%|██▏       | 292/1372 [00:51<03:07,  5.76it/s]#015Iteration:  21%|██▏       | 293/1372 [00:52<03:07,  5.76it/s]#015Iteration:  21%|██▏       | 294/1372 [00:52<03:07,  5.75it/s]#015Iteration:  22%|██▏       | 295/1372 [00:52<03:07,  5.76it/s]#015Iteration:  22%|██▏       | 296/1372 [00:52<03:06,  5.75it/s]#015Iteration:  22%|██▏       | 297/1372 [00:52<03:06,  5.75it/s]#015Iteration:  22%|██▏       | 298/1372 [00:52<03:06,  5.75it/s]#015Iteration:  22%|██▏       | 299/1372 [00:53<03:06,  5.75it/s]#015Iteration:  22%|██▏       | 300/1372 [00:53<03:06,  5.75it/s]#015Iteration:  22%|██▏       | 301/1372 [00:53<03:06,  5.75it/s]#015Iteration:  22%|██▏       | 302/1372 [00:53<03:05,  5.75it/s]#015Iteration:  22%|██▏       | 303/1372 [00:53<03:05,  5.75it/s]#015Iteration:  22%|██▏       | 304/1372 [00:53<03:05,  5.76it/s]#015Iteration:  22%|██▏       | 305/1372 [00:54<03:05,  5.76it/s]#015Iteration:  22%|██▏       | 306/1372 [00:54<03:05,  5.76it/s]#015Iteration:  22%|██▏       | 307/1372 [00:54<03:04,  5.76it/s]#015Iteration:  22%|██▏       | 308/1372 [00:54<03:04,  5.76it/s]#015Iteration:  23%|██▎       | 309/1372 [00:54<03:04,  5.76it/s]#015Iteration:  23%|██▎       | 310/1372 [00:54<03:04,  5.76it/s]#015Iteration:  23%|██▎       | 311/1372 [00:55<03:04,  5.76it/s]#015Iteration:  23%|██▎       | 312/1372 [00:55<03:04,  5.75it/s]#015Iteration:  23%|██▎       | 313/1372 [00:55<03:03,  5.76it/s]#015Iteration:  23%|██▎       | 314/1372 [00:55<03:03,  5.76it/s]#015Iteration:  23%|██▎       | 315/1372 [00:55<03:03,  5.76it/s]#015Iteration:  23%|██▎       | 316/1372 [00:56<03:03,  5.76it/s]#015Iteration:  23%|██▎       | 317/1372 [00:56<03:03,  5.76it/s]#015Iteration:  23%|██▎       | 318/1372 [00:56<03:02,  5.76it/s]#015Iteration:  23%|██▎       | 319/1372 [00:56<03:02,  5.76it/s]#015Iteration:  23%|██▎       | 320/1372 [00:56<03:02,  5.76it/s]#015Iteration:  23%|██▎       | 321/1372 [00:56<03:02,  5.76it/s]#015Iteration:  23%|██▎       | 322/1372 [00:57<03:02,  5.76it/s]#015Iteration:  24%|██▎       | 323/1372 [00:57<03:02,  5.76it/s]#015Iteration:  24%|██▎       | 324/1372 [00:57<03:02,  5.76it/s]#015Iteration:  24%|██▎       | 325/1372 [00:57<03:01,  5.76it/s]#015Iteration:  24%|██▍       | 326/1372 [00:57<03:01,  5.76it/s]#015Iteration:  24%|██▍       | 327/1372 [00:57<03:01,  5.75it/s]#015Iteration:  24%|██▍       | 328/1372 [00:58<03:01,  5.75it/s]#015Iteration:  24%|██▍       | 329/1372 [00:58<03:01,  5.75it/s]#015Iteration:  24%|██▍       | 330/1372 [00:58<03:01,  5.74it/s]#015Iteration:  24%|██▍       | 331/1372 [00:58<03:01,  5.75it/s]#015Iteration:  24%|██▍       | 332/1372 [00:58<03:00,  5.75it/s]#015Iteration:  24%|██▍       | 333/1372 [00:58<03:00,  5.75it/s]#015Iteration:  24%|██▍       | 334/1372 [00:59<03:00,  5.76it/s]#015Iteration:  24%|██▍       | 335/1372 [00:59<03:00,  5.76it/s]#015Iteration:  24%|██▍       | 336/1372 [00:59<02:59,  5.76it/s]#015Iteration:  25%|██▍       | 337/1372 [00:59<02:59,  5.75it/s]#015Iteration:  25%|██▍       | 338/1372 [00:59<02:59,  5.75it/s]#015Iteration:  25%|██▍       | 339/1372 [01:00<02:59,  5.75it/s]#015Iteration:  25%|██▍       | 340/1372 [01:00<02:59,  5.75it/s]#015Iteration:  25%|██▍       | 341/1372 [01:00<02:59,  5.75it/s]#015Iteration:  25%|██▍       | 342/1372 [01:00<02:59,  5.75it/s]#015Iteration:  25%|██▌       | 343/1372 [01:00<02:58,  5.75it/s]#015Iteration:  25%|██▌       | 344/1372 [01:00<02:58,  5.75it/s]#015Iteration:  25%|██▌       | 345/1372 [01:01<02:58,  5.75it/s]#015Iteration:  25%|██▌       | 346/1372 [01:01<02:58,  5.75it/s]#015Iteration:  25%|██▌       | 347/1372 [01:01<02:58,  5.75it/s]#015Iteration:  25%|██▌       | 348/1372 [01:01<02:57,  5.75it/s]#015Iteration:  25%|██▌       | 349/1372 [01:01<02:57,  5.75it/s]#015Iteration:  26%|██▌       | 350/1372 [01:01<02:57,  5.76it/s]#015Iteration:  26%|██▌       | 351/1372 [01:02<02:57,  5.75it/s]#015Iteration:  26%|██▌       | 352/1372 [01:02<02:57,  5.76it/s]#015Iteration:  26%|██▌       | 353/1372 [01:02<02:57,  5.75it/s]#015Iteration:  26%|██▌       | 354/1372 [01:02<02:56,  5.75it/s]#015Iteration:  26%|██▌       | 355/1372 [01:02<02:56,  5.75it/s]#015Iteration:  26%|██▌       | 356/1372 [01:02<02:56,  5.75it/s]#015Iteration:  26%|██▌       | 357/1372 [01:03<02:56,  5.75it/s]#015Iteration:  26%|██▌       | 358/1372 [01:03<02:56,  5.75it/s]#015Iteration:  26%|██▌       | 359/1372 [01:03<02:56,  5.75it/s]#015Iteration:  26%|██▌       | 360/1372 [01:03<02:56,  5.75it/s]#015Iteration:  26%|██▋       | 361/1372 [01:03<02:55,  5.75it/s]#015Iteration:  26%|██▋       | 362/1372 [01:04<02:55,  5.75it/s]#015Iteration:  26%|██▋       | 363/1372 [01:04<02:55,  5.75it/s]#015Iteration:  27%|██▋       | 364/1372 [01:04<02:55,  5.75it/s]#015Iteration:  27%|██▋       | 365/1372 [01:04<02:54,  5.75it/s]#015Iteration:  27%|██▋       | 366/1372 [01:04<02:54,  5.75it/s]#015Iteration:  27%|██▋       | 367/1372 [01:04<02:54,  5.75it/s]#015Iteration:  27%|██▋       | 368/1372 [01:05<02:54,  5.75it/s]#015Iteration:  27%|██▋       | 369/1372 [01:05<02:54,  5.76it/s]#015Iteration:  27%|██▋       | 370/1372 [01:05<02:54,  5.75it/s]#015Iteration:  27%|██▋       | 371/1372 [01:05<02:53,  5.75it/s]#015Iteration:  27%|██▋       | 372/1372 [01:05<02:53,  5.75it/s]#015Iteration:  27%|██▋       | 373/1372 [01:05<02:53,  5.75it/s]#015Iteration:  27%|██▋       | 374/1372 [01:06<02:53,  5.76it/s]#015Iteration:  27%|██▋       | 375/1372 [01:06<02:53,  5.76it/s]#015Iteration:  27%|██▋       | 376/1372 [01:06<02:52,  5.76it/s]#015Iteration:  27%|██▋       | 377/1372 [01:06<02:52,  5.75it/s]#015Iteration:  28%|██▊       | 378/1372 [01:06<02:52,  5.75it/s]#015Iteration:  28%|██▊       | 379/1372 [01:06<02:52,  5.75it/s]#015Iteration:  28%|██▊       | 380/1372 [01:07<02:52,  5.75it/s]#015Iteration:  28%|██▊       | 381/1372 [01:07<02:52,  5.75it/s]#015Iteration:  28%|██▊       | 382/1372 [01:07<02:52,  5.76it/s]#015Iteration:  28%|██▊       | 383/1372 [01:07<02:51,  5.75it/s]#015Iteration:  28%|██▊       | 384/1372 [01:07<02:51,  5.75it/s]#015Iteration:  28%|██▊       | 385/1372 [01:08<02:51,  5.75it/s]#015Iteration:  28%|██▊       | 386/1372 [01:08<02:51,  5.75it/s]#015Iteration:  28%|██▊       | 387/1372 [01:08<02:51,  5.75it/s]#015Iteration:  28%|██▊       | 388/1372 [01:08<02:51,  5.75it/s]#015Iteration:  28%|██▊       | 389/1372 [01:08<02:50,  5.75it/s]#015Iteration:  28%|██▊       | 390/1372 [01:08<02:50,  5.75it/s]#015Iteration:  28%|██▊       | 391/1372 [01:09<02:50,  5.75it/s]#015Iteration:  29%|██▊       | 392/1372 [01:09<02:50,  5.75it/s]#015Iteration:  29%|██▊       | 393/1372 [01:09<02:50,  5.75it/s]#015Iteration:  29%|██▊       | 394/1372 [01:09<02:49,  5.76it/s]#015Iteration:  29%|██▉       | 395/1372 [01:09<02:49,  5.76it/s]#015Iteration:  29%|██▉       | 396/1372 [01:09<02:49,  5.76it/s]#015Iteration:  29%|██▉       | 397/1372 [01:10<02:49,  5.76it/s]#015Iteration:  29%|██▉       | 398/1372 [01:10<02:49,  5.75it/s]#015Iteration:  29%|██▉       | 399/1372 [01:10<02:49,  5.75it/s]#015Iteration:  29%|██▉       | 400/1372 [01:10<02:49,  5.75it/s]#015Iteration:  29%|██▉       | 401/1372 [01:10<02:48,  5.75it/s]#015Iteration:  29%|██▉       | 402/1372 [01:10<02:48,  5.74it/s]#015Iteration:  29%|██▉       | 403/1372 [01:11<02:48,  5.74it/s]#015Iteration:  29%|██▉       | 404/1372 [01:11<02:48,  5.74it/s]#015Iteration:  30%|██▉       | 405/1372 [01:11<02:48,  5.75it/s]#015Iteration:  30%|██▉       | 406/1372 [01:11<02:48,  5.74it/s]#015Iteration:  30%|██▉       | 407/1372 [01:11<02:48,  5.74it/s]#015Iteration:  30%|██▉       | 408/1372 [01:12<02:48,  5.74it/s]#015Iteration:  30%|██▉       | 409/1372 [01:12<02:47,  5.74it/s]#015Iteration:  30%|██▉       | 410/1372 [01:12<02:47,  5.74it/s]#015Iteration:  30%|██▉       | 411/1372 [01:12<02:47,  5.75it/s]#015Iteration:  30%|███       | 412/1372 [01:12<02:47,  5.74it/s]#015Iteration:  30%|███       | 413/1372 [01:12<02:47,  5.74it/s]#015Iteration:  30%|███       | 414/1372 [01:13<02:46,  5.74it/s]#015Iteration:  30%|███       | 415/1372 [01:13<02:46,  5.74it/s]#015Iteration:  30%|███       | 416/1372 [01:13<02:46,  5.75it/s]#015Iteration:  30%|███       | 417/1372 [01:13<02:46,  5.75it/s]#015Iteration:  30%|███       | 418/1372 [01:13<02:45,  5.75it/s]#015Iteration:  31%|███       | 419/1372 [01:13<02:45,  5.74it/s]#015Iteration:  31%|███       | 420/1372 [01:14<02:45,  5.74it/s]#015Iteration:  31%|███       | 421/1372 [01:14<02:45,  5.75it/s]#015Iteration:  31%|███       | 422/1372 [01:14<02:45,  5.75it/s]#015Iteration:  31%|███       | 423/1372 [01:14<02:45,  5.75it/s]#015Iteration:  31%|███       | 424/1372 [01:14<02:44,  5.75it/s]#015Iteration:  31%|███       | 425/1372 [01:14<02:44,  5.75it/s]#015Iteration:  31%|███       | 426/1372 [01:15<02:44,  5.74it/s]#015Iteration:  31%|███       | 427/1372 [01:15<02:44,  5.75it/s]#015Iteration:  31%|███       | 428/1372 [01:15<02:44,  5.75it/s]#015Iteration:  31%|███▏      | 429/1372 [01:15<02:43,  5.75it/s]#015Iteration:  31%|███▏      | 430/1372 [01:15<02:43,  5.75it/s]#015Iteration:  31%|███▏      | 431/1372 [01:16<02:43,  5.75it/s]#015Iteration:  31%|███▏      | 432/1372 [01:16<02:43,  5.75it/s]#015Iteration:  32%|███▏      | 433/1372 [01:16<02:43,  5.75it/s]#015Iteration:  32%|███▏      | 434/1372 [01:16<02:43,  5.75it/s]#015Iteration:  32%|███▏      | 435/1372 [01:16<02:42,  5.75it/s]#015Iteration:  32%|███▏      | 436/1372 [01:16<02:42,  5.75it/s]#015Iteration:  32%|███▏      | 437/1372 [01:17<02:42,  5.75it/s]#015Iteration:  32%|███▏      | 438/1372 [01:17<02:42,  5.74it/s]#015Iteration:  32%|███▏      | 439/1372 [01:17<02:42,  5.74it/s]#015Iteration:  32%|███▏      | 440/1372 [01:17<02:42,  5.74it/s]#015Iteration:  32%|███▏      | 441/1372 [01:17<02:42,  5.75it/s]#015Iteration:  32%|███▏      | 442/1372 [01:17<02:41,  5.74it/s]#015Iteration:  32%|███▏      | 443/1372 [01:18<02:41,  5.75it/s]#015Iteration:  32%|███▏      | 444/1372 [01:18<02:41,  5.75it/s]#015Iteration:  32%|███▏      | 445/1372 [01:18<02:41,  5.75it/s]#015Iteration:  33%|███▎      | 446/1372 [01:18<02:41,  5.75it/s]#015Iteration:  33%|███▎      | 447/1372 [01:18<02:40,  5.75it/s]#015Iteration:  33%|███▎      | 448/1372 [01:18<02:40,  5.74it/s]#015Iteration:  33%|███▎      | 449/1372 [01:19<02:40,  5.75it/s]#015Iteration:  33%|███▎      | 450/1372 [01:19<02:40,  5.75it/s]#015Iteration:  33%|███▎      | 451/1372 [01:19<02:40,  5.75it/s]#015Iteration:  33%|███▎      | 452/1372 [01:19<02:39,  5.75it/s]#015Iteration:  33%|███▎      | 453/1372 [01:19<02:39,  5.75it/s]#015Iteration:  33%|███▎      | 454/1372 [01:20<02:39,  5.75it/s]#015Iteration:  33%|███▎      | 455/1372 [01:20<02:39,  5.75it/s]#015Iteration:  33%|███▎      | 456/1372 [01:20<02:39,  5.75it/s]#015Iteration:  33%|███▎      | 457/1372 [01:20<02:38,  5.76it/s]#015Iteration:  33%|███▎      | 458/1372 [01:20<02:38,  5.75it/s]#015Iteration:  33%|███▎      | 459/1372 [01:20<02:38,  5.75it/s]#015Iteration:  34%|███▎      | 460/1372 [01:21<02:38,  5.75it/s]#015Iteration:  34%|███▎      | 461/1372 [01:21<02:38,  5.75it/s]#015Iteration:  34%|███▎      | 462/1372 [01:21<02:38,  5.75it/s]#015Iteration:  34%|███▎      | 463/1372 [01:21<02:38,  5.75it/s]#015Iteration:  34%|███▍      | 464/1372 [01:21<02:37,  5.75it/s]#015Iteration:  34%|███▍      | 465/1372 [01:21<02:37,  5.75it/s]#015Iteration:  34%|███▍      | 466/1372 [01:22<02:37,  5.75it/s]#015Iteration:  34%|███▍      | 467/1372 [01:22<02:37,  5.75it/s]#015Iteration:  34%|███▍      | 468/1372 [01:22<02:37,  5.75it/s]#015Iteration:  34%|███▍      | 469/1372 [01:22<02:37,  5.75it/s]#015Iteration:  34%|███▍      | 470/1372 [01:22<02:37,  5.74it/s]#015Iteration:  34%|███▍      | 471/1372 [01:22<02:36,  5.74it/s]#015Iteration:  34%|███▍      | 472/1372 [01:23<02:36,  5.74it/s]#015Iteration:  34%|███▍      | 473/1372 [01:23<02:36,  5.75it/s]#015Iteration:  35%|███▍      | 474/1372 [01:23<02:36,  5.74it/s]#015Iteration:  35%|███▍      | 475/1372 [01:23<02:36,  5.75it/s]#015Iteration:  35%|███▍      | 476/1372 [01:23<02:35,  5.75it/s]#015Iteration:  35%|███▍      | 477/1372 [01:24<02:35,  5.75it/s]#015Iteration:  35%|███▍      | 478/1372 [01:24<02:35,  5.75it/s]#015Iteration:  35%|███▍      | 479/1372 [01:24<02:35,  5.75it/s]#015Iteration:  35%|███▍      | 480/1372 [01:24<02:35,  5.75it/s]#015Iteration:  35%|███▌      | 481/1372 [01:24<02:34,  5.75it/s]#015Iteration:  35%|███▌      | 482/1372 [01:24<02:34,  5.75it/s]#015Iteration:  35%|███▌      | 483/1372 [01:25<02:34,  5.74it/s]#015Iteration:  35%|███▌      | 484/1372 [01:25<02:34,  5.74it/s]#015Iteration:  35%|███▌      | 485/1372 [01:25<02:34,  5.74it/s]#015Iteration:  35%|███▌      | 486/1372 [01:25<02:34,  5.74it/s]#015Iteration:  35%|███▌      | 487/1372 [01:25<02:34,  5.74it/s]#015Iteration:  36%|███▌      | 488/1372 [01:25<02:33,  5.74it/s]#015Iteration:  36%|███▌      | 489/1372 [01:26<02:33,  5.75it/s]#015Iteration:  36%|███▌      | 490/1372 [01:26<02:33,  5.75it/s]#015Iteration:  36%|███▌      | 491/1372 [01:26<02:33,  5.75it/s]#015Iteration:  36%|███▌      | 492/1372 [01:26<02:32,  5.75it/s]#015Iteration:  36%|███▌      | 493/1372 [01:26<02:32,  5.75it/s]#015Iteration:  36%|███▌      | 494/1372 [01:26<02:32,  5.75it/s]#015Iteration:  36%|██�\u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:55:18.144132 - Training Epoch: 0 Training Iteration: 501  step_loss : 1.91796875  learning_rate : 2.112573099415205e-05 \u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34mDLL 2020-08-23 20:55:26.843576 - Training Epoch: 0 Training Iteration: 551  step_loss : 1.244140625  learning_rate : 1.9907407407407406e-05 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:55:35.542806 - Training Epoch: 0 Training Iteration: 601  step_loss : 1.1591796875  learning_rate : 1.868908382066277e-05 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:55:44.244250 - Training Epoch: 0 Training Iteration: 651  step_loss : 1.5859375  learning_rate : 1.747076023391813e-05 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:55:52.968846 - Training Epoch: 0 Training Iteration: 701  step_loss : 1.10546875  learning_rate : 1.6252436647173486e-05 \u001b[0m\n",
      "\u001b[34m��▌      | 495/1372 [01:27<02:32,  5.75it/s]#015Iteration:  36%|███▌      | 496/1372 [01:27<02:32,  5.75it/s]#015Iteration:  36%|███▌      | 497/1372 [01:27<02:32,  5.75it/s]#015Iteration:  36%|███▋      | 498/1372 [01:27<02:32,  5.74it/s]#015Iteration:  36%|███▋      | 499/1372 [01:27<02:31,  5.74it/s]#015Iteration:  36%|███▋      | 500/1372 [01:28<02:31,  5.74it/s]#015Iteration:  37%|███▋      | 501/1372 [01:28<02:31,  5.74it/s]#015Iteration:  37%|███▋      | 502/1372 [01:28<02:31,  5.74it/s]#015Iteration:  37%|███▋      | 503/1372 [01:28<02:31,  5.74it/s]#015Iteration:  37%|███▋      | 504/1372 [01:28<02:31,  5.74it/s]#015Iteration:  37%|███▋      | 505/1372 [01:28<02:30,  5.74it/s]#015Iteration:  37%|███▋      | 506/1372 [01:29<02:30,  5.75it/s]#015Iteration:  37%|███▋      | 507/1372 [01:29<02:30,  5.75it/s]#015Iteration:  37%|███▋      | 508/1372 [01:29<02:30,  5.75it/s]#015Iteration:  37%|███▋      | 509/1372 [01:29<02:30,  5.75it/s]#015Iteration:  37%|███▋      | 510/1372 [01:29<02:30,  5.75it/s]#015Iteration:  37%|███▋      | 511/1372 [01:29<02:29,  5.75it/s]#015Iteration:  37%|███▋      | 512/1372 [01:30<02:29,  5.75it/s]#015Iteration:  37%|███▋      | 513/1372 [01:30<02:29,  5.75it/s]#015Iteration:  37%|███▋      | 514/1372 [01:30<02:29,  5.75it/s]#015Iteration:  38%|███▊      | 515/1372 [01:30<02:29,  5.74it/s]#015Iteration:  38%|███▊      | 516/1372 [01:30<02:29,  5.74it/s]#015Iteration:  38%|███▊      | 517/1372 [01:30<02:28,  5.74it/s]#015Iteration:  38%|███▊      | 518/1372 [01:31<02:28,  5.74it/s]#015Iteration:  38%|███▊      | 519/1372 [01:31<02:28,  5.74it/s]#015Iteration:  38%|███▊      | 520/1372 [01:31<02:28,  5.75it/s]#015Iteration:  38%|███▊      | 521/1372 [01:31<02:28,  5.75it/s]#015Iteration:  38%|███▊      | 522/1372 [01:31<02:27,  5.75it/s]#015Iteration:  38%|███▊      | 523/1372 [01:32<02:27,  5.75it/s]#015Iteration:  38%|███▊      | 524/1372 [01:32<02:27,  5.75it/s]#015Iteration:  38%|███▊      | 525/1372 [01:32<02:27,  5.75it/s]#015Iteration:  38%|███▊      | 526/1372 [01:32<02:27,  5.75it/s]#015Iteration:  38%|███▊      | 527/1372 [01:32<02:26,  5.75it/s]#015Iteration:  38%|███▊      | 528/1372 [01:32<02:26,  5.75it/s]#015Iteration:  39%|███▊      | 529/1372 [01:33<02:26,  5.75it/s]#015Iteration:  39%|███▊      | 530/1372 [01:33<02:26,  5.75it/s]#015Iteration:  39%|███▊      | 531/1372 [01:33<02:26,  5.75it/s]#015Iteration:  39%|███▉      | 532/1372 [01:33<02:26,  5.75it/s]#015Iteration:  39%|███▉      | 533/1372 [01:33<02:25,  5.75it/s]#015Iteration:  39%|███▉      | 534/1372 [01:33<02:25,  5.75it/s]#015Iteration:  39%|███▉      | 535/1372 [01:34<02:25,  5.75it/s]#015Iteration:  39%|███▉      | 536/1372 [01:34<02:25,  5.75it/s]#015Iteration:  39%|███▉      | 537/1372 [01:34<02:25,  5.75it/s]#015Iteration:  39%|███▉      | 538/1372 [01:34<02:25,  5.75it/s]#015Iteration:  39%|███▉      | 539/1372 [01:34<02:24,  5.75it/s]#015Iteration:  39%|███▉      | 540/1372 [01:34<02:24,  5.75it/s]#015Iteration:  39%|███▉      | 541/1372 [01:35<02:24,  5.75it/s]#015Iteration:  40%|███▉      | 542/1372 [01:35<02:24,  5.75it/s]#015Iteration:  40%|███▉      | 543/1372 [01:35<02:24,  5.75it/s]#015Iteration:  40%|███▉      | 544/1372 [01:35<02:24,  5.75it/s]#015Iteration:  40%|███▉      | 545/1372 [01:35<02:23,  5.75it/s]#015Iteration:  40%|███▉      | 546/1372 [01:36<02:23,  5.75it/s]#015Iteration:  40%|███▉      | 547/1372 [01:36<02:23,  5.75it/s]#015Iteration:  40%|███▉      | 548/1372 [01:36<02:23,  5.75it/s]#015Iteration:  40%|████      | 549/1372 [01:36<02:23,  5.75it/s]#015Iteration:  40%|████      | 550/1372 [01:36<02:22,  5.75it/s]#015Iteration:  40%|████      | 551/1372 [01:36<02:22,  5.75it/s]#015Iteration:  40%|████      | 552/1372 [01:37<02:22,  5.75it/s]#015Iteration:  40%|████      | 553/1372 [01:37<02:22,  5.75it/s]#015Iteration:  40%|████      | 554/1372 [01:37<02:22,  5.75it/s]#015Iteration:  40%|████      | 555/1372 [01:37<02:22,  5.75it/s]#015Iteration:  41%|████      | 556/1372 [01:37<02:21,  5.75it/s]#015Iteration:  41%|████      | 557/1372 [01:37<02:21,  5.75it/s]#015Iteration:  41%|████      | 558/1372 [01:38<02:21,  5.75it/s]#015Iteration:  41%|████      | 559/1372 [01:38<02:21,  5.74it/s]#015Iteration:  41%|████      | 560/1372 [01:38<02:21,  5.74it/s]#015Iteration:  41%|████      | 561/1372 [01:38<02:21,  5.74it/s]#015Iteration:  41%|████      | 562/1372 [01:38<02:20,  5.75it/s]#015Iteration:  41%|████      | 563/1372 [01:38<02:20,  5.75it/s]#015Iteration:  41%|████      | 564/1372 [01:39<02:20,  5.75it/s]#015Iteration:  41%|████      | 565/1372 [01:39<02:20,  5.75it/s]#015Iteration:  41%|████▏     | 566/1372 [01:39<02:20,  5.75it/s]#015Iteration:  41%|████▏     | 567/1372 [01:39<02:20,  5.75it/s]#015Iteration:  41%|████▏     | 568/1372 [01:39<02:19,  5.75it/s]#015Iteration:  41%|████▏     | 569/1372 [01:40<02:19,  5.75it/s]#015Iteration:  42%|████▏     | 570/1372 [01:40<02:19,  5.75it/s]#015Iteration:  42%|████▏     | 571/1372 [01:40<02:19,  5.75it/s]#015Iteration:  42%|████▏     | 572/1372 [01:40<02:19,  5.75it/s]#015Iteration:  42%|████▏     | 573/1372 [01:40<02:18,  5.75it/s]#015Iteration:  42%|████▏     | 574/1372 [01:40<02:18,  5.75it/s]#015Iteration:  42%|████▏     | 575/1372 [01:41<02:18,  5.75it/s]#015Iteration:  42%|████▏     | 576/1372 [01:41<02:18,  5.75it/s]#015Iteration:  42%|████▏     | 577/1372 [01:41<02:18,  5.75it/s]#015Iteration:  42%|████▏     | 578/1372 [01:41<02:18,  5.75it/s]#015Iteration:  42%|████▏     | 579/1372 [01:41<02:17,  5.75it/s]#015Iteration:  42%|████▏     | 580/1372 [01:41<02:17,  5.75it/s]#015Iteration:  42%|████▏     | 581/1372 [01:42<02:17,  5.75it/s]#015Iteration:  42%|████▏     | 582/1372 [01:42<02:17,  5.75it/s]#015Iteration:  42%|████▏     | 583/1372 [01:42<02:17,  5.75it/s]#015Iteration:  43%|████▎     | 584/1372 [01:42<02:17,  5.75it/s]#015Iteration:  43%|████▎     | 585/1372 [01:42<02:16,  5.75it/s]#015Iteration:  43%|████▎     | 586/1372 [01:42<02:16,  5.75it/s]#015Iteration:  43%|████▎     | 587/1372 [01:43<02:16,  5.75it/s]#015Iteration:  43%|████▎     | 588/1372 [01:43<02:16,  5.75it/s]#015Iteration:  43%|████▎     | 589/1372 [01:43<02:16,  5.75it/s]#015Iteration:  43%|████▎     | 590/1372 [01:43<02:16,  5.74it/s]#015Iteration:  43%|████▎     | 591/1372 [01:43<02:15,  5.74it/s]#015Iteration:  43%|████▎     | 592/1372 [01:44<02:15,  5.74it/s]#015Iteration:  43%|████▎     | 593/1372 [01:44<02:15,  5.75it/s]#015Iteration:  43%|████▎     | 594/1372 [01:44<02:15,  5.75it/s]#015Iteration:  43%|████▎     | 595/1372 [01:44<02:15,  5.75it/s]#015Iteration:  43%|████▎     | 596/1372 [01:44<02:15,  5.74it/s]#015Iteration:  44%|████▎     | 597/1372 [01:44<02:14,  5.74it/s]#015Iteration:  44%|████▎     | 598/1372 [01:45<02:14,  5.74it/s]#015Iteration:  44%|████▎     | 599/1372 [01:45<02:14,  5.75it/s]#015Iteration:  44%|████▎     | 600/1372 [01:45<02:14,  5.75it/s]#015Iteration:  44%|████▍     | 601/1372 [01:45<02:14,  5.75it/s]#015Iteration:  44%|████▍     | 602/1372 [01:45<02:14,  5.75it/s]#015Iteration:  44%|████▍     | 603/1372 [01:45<02:13,  5.75it/s]#015Iteration:  44%|████▍     | 604/1372 [01:46<02:13,  5.75it/s]#015Iteration:  44%|████▍     | 605/1372 [01:46<02:13,  5.75it/s]#015Iteration:  44%|████▍     | 606/1372 [01:46<02:13,  5.75it/s]#015Iteration:  44%|████▍     | 607/1372 [01:46<02:13,  5.75it/s]#015Iteration:  44%|████▍     | 608/1372 [01:46<02:12,  5.75it/s]#015Iteration:  44%|████▍     | 609/1372 [01:46<02:12,  5.75it/s]#015Iteration:  44%|████▍     | 610/1372 [01:47<02:12,  5.75it/s]#015Iteration:  45%|████▍     | 611/1372 [01:47<02:12,  5.75it/s]#015Iteration:  45%|████▍     | 612/1372 [01:47<02:12,  5.75it/s]#015Iteration:  45%|████▍     | 613/1372 [01:47<02:12,  5.75it/s]#015Iteration:  45%|████▍     | 614/1372 [01:47<02:11,  5.75it/s]#015Iteration:  45%|████▍     | 615/1372 [01:48<02:11,  5.75it/s]#015Iteration:  45%|████▍     | 616/1372 [01:48<02:11,  5.73it/s]#015Iteration:  45%|████▍     | 617/1372 [01:48<02:11,  5.74it/s]#015Iteration:  45%|████▌     | 618/1372 [01:48<02:11,  5.74it/s]#015Iteration:  45%|████▌     | 619/1372 [01:48<02:11,  5.74it/s]#015Iteration:  45%|████▌     | 620/1372 [01:48<02:10,  5.74it/s]#015Iteration:  45%|████▌     | 621/1372 [01:49<02:10,  5.75it/s]#015Iteration:  45%|████▌     | 622/1372 [01:49<02:10,  5.74it/s]#015Iteration:  45%|████▌     | 623/1372 [01:49<02:10,  5.75it/s]#015Iteration:  45%|████▌     | 624/1372 [01:49<02:10,  5.75it/s]#015Iteration:  46%|████▌     | 625/1372 [01:49<02:09,  5.75it/s]#015Iteration:  46%|████▌     | 626/1372 [01:49<02:09,  5.75it/s]#015Iteration:  46%|████▌     | 627/1372 [01:50<02:09,  5.75it/s]#015Iteration:  46%|████▌     | 628/1372 [01:50<02:09,  5.75it/s]#015Iteration:  46%|████▌     | 629/1372 [01:50<02:09,  5.75it/s]#015Iteration:  46%|████▌     | 630/1372 [01:50<02:09,  5.74it/s]#015Iteration:  46%|████▌     | 631/1372 [01:50<02:09,  5.74it/s]#015Iteration:  46%|████▌     | 632/1372 [01:50<02:08,  5.75it/s]#015Iteration:  46%|████▌     | 633/1372 [01:51<02:08,  5.75it/s]#015Iteration:  46%|████▌     | 634/1372 [01:51<02:08,  5.75it/s]#015Iteration:  46%|████▋     | 635/1372 [01:51<02:08,  5.75it/s]#015Iteration:  46%|████▋     | 636/1372 [01:51<02:08,  5.75it/s]#015Iteration:  46%|████▋     | 637/1372 [01:51<02:07,  5.75it/s]#015Iteration:  47%|████▋     | 638/1372 [01:52<02:07,  5.75it/s]#015Iteration:  47%|████▋     | 639/1372 [01:52<02:07,  5.75it/s]#015Iteration:  47%|████▋     | 640/1372 [01:52<02:07,  5.75it/s]#015Iteration:  47%|████▋     | 641/1372 [01:52<02:07,  5.75it/s]#015Iteration:  47%|████▋     | 642/1372 [01:52<02:06,  5.75it/s]#015Iteration:  47%|████▋     | 643/1372 [01:52<02:06,  5.75it/s]#015Iteration:  47%|████▋     | 644/1372 [01:53<02:06,  5.75it/s]#015Iteration:  47%|████▋     | 645/1372 [01:53<02:06,  5.74it/s]#015Iteration:  47%|████▋     | 646/1372 [01:53<02:06,  5.74it/s]#015Iteration:  47%|████▋     | 647/1372 [01:53<02:06,  5.75it/s]#015Iteration:  47%|████▋     | 648/1372 [01:53<02:05,  5.75it/s]#015Iteration:  47%|████▋     | 649/1372 [01:53<02:05,  5.74it/s]#015Iteration:  47%|████▋     | 650/1372 [01:54<02:05,  5.75it/s]#015Iteration:  47%|████▋     | 651/1372 [01:54<02:05,  5.75it/s]#015Iteration:  48%|████▊     | 652/1372 [01:54<02:05,  5.75it/s]#015Iteration:  48%|████▊     | 653/1372 [01:54<02:05,  5.74it/s]#015Iteration:  48%|████▊     | 654/1372 [01:54<02:05,  5.74it/s]#015Iteration:  48%|████▊     | 655/1372 [01:54<02:04,  5.74it/s]#015Iteration:  48%|████▊     | 656/1372 [01:55<02:04,  5.74it/s]#015Iteration:  48%|████▊     | 657/1372 [01:55<02:04,  5.74it/s]#015Iteration:  48%|████▊     | 658/1372 [01:55<02:04,  5.74it/s]#015Iteration:  48%|████▊     | 659/1372 [01:55<02:04,  5.74it/s]#015Iteration:  48%|████▊     | 660/1372 [01:55<02:04,  5.74it/s]#015Iteration:  48%|████▊     | 661/1372 [01:56<02:03,  5.74it/s]#015Iteration:  48%|████▊     | 662/1372 [01:56<02:03,  5.74it/s]#015Iteration:  48%|████▊     | 663/1372 [01:56<02:03,  5.73it/s]#015Iteration:  48%|████▊     | 664/1372 [01:56<02:03,  5.73it/s]#015Iteration:  48%|████▊     | 665/1372 [01:56<02:03,  5.73it/s]#015Iteration:  49%|████▊     | 666/1372 [01:56<02:03,  5.74it/s]#015Iteration:  49%|████▊     | 667/1372 [01:57<02:02,  5.73it/s]#015Iteration:  49%|████▊     | 668/1372 [01:57<02:02,  5.73it/s]#015Iteration:  49%|████▉     | 669/1372 [01:57<02:02,  5.74it/s]#015Iteration:  49%|████▉     | 670/1372 [01:57<02:02,  5.74it/s]#015Iteration:  49%|████▉     | 671/1372 [01:57<02:02,  5.74it/s]#015Iteration:  49%|████▉     | 672/1372 [01:57<02:02,  5.73it/s]#015Iteration:  49%|████▉     | 673/1372 [01:58<02:01,  5.73it/s]#015Iteration:  49%|████▉     | 674/1372 [01:58<02:01,  5.73it/s]#015Iteration:  49%|████▉     | 675/1372 [01:58<02:01,  5.73it/s]#015Iteration:  49%|████▉     | 676/1372 [01:58<02:01,  5.73it/s]#015Iteration:  49%|████▉     | 677/1372 [01:58<02:01,  5.73it/s]#015Iteration:  49%|████▉     | 678/1372 [01:59<02:01,  5.73it/s]#015Iteration:  49%|████▉     | 679/1372 [01:59<02:00,  5.73it/s]#015Iteration:  50%|████▉     | 680/1372 [01:59<02:00,  5.73it/s]#015Iteration:  50%|████▉     | 681/1372 [01:59<02:00,  5.73it/s]#015Iteration:  50%|████▉     | 682/1372 [01:59<02:00,  5.73it/s]#015Iteration:  50%|████▉     | 683/1372 [01:59<02:00,  5.73it/s]#015Iteration:  50%|████▉     | 684/1372 [02:00<02:00,  5.73it/s]#015Iteration:  50%|████▉     | 685/1372 [02:00<01:59,  5.73it/s]#015Iteration:  50%|█████     | 686/1372 [02:00<01:59,  5.73it/s]#015Iteration:  50%|█████     | 687/1372 [02:00<01:59,  5.72it/s]#015Iteration:  50%|█████     | 688/1372 [02:00<01:59,  5.72it/s]#015Iteration:  50%|█████     | 689/1372 [02:00<01:59,  5.73it/s]#015Iteration:  50%|█████     | 690/1372 [02:01<01:59,  5.72it/s]#015Iteration:  50%|█████     | 691/1372 [02:01<01:58,  5.73it/s]#015Iteration:  50%|█████     | 692/1372 [02:01<01:58,  5.72it/s]#015Iteration:  51%|█████     | 693/1372 [02:01<01:58,  5.72it/s]#015Iteration:  51%|█████     | 694/1372 [02:01<01:58,  5.72it/s]#015Iteration:  51%|█████     | 695/1372 [02:01<01:58,  5.72it/s]#015Iteration:  51%|█████     | 696/1372 [02:02<01:58,  5.72it/s]#015Iteration:  51%|█████     | 697/1372 [02:02<01:57,  5.73it/s]#015Iteration:  51%|█████     | 698/1372 [02:02<01:57,  5.73it/s]#015Iteration:  51%|█████     | 699/1372 [02:02<01:57,  5.73it/s]#015Iteration:  51%|█████     | 700/1372 [02:02<01:57,  5.73it/s]#015Iteration:  51%|█████     | 701/1372 [02:03<01:57,  5.73it/s]#015Iteration:  51%|█████     | 702/1372 [02:03<01:57,  5.72it/s]#015Iteration:  51%|█████     | 703/1372 [02:03<01:56,  5.72it/s]#015Iteration:  51%|█████▏    | 704/1372 [02:03<01:56,  5.73it/s]#015Iteration:  51%|█████▏    | 705/1372 [02:03<01:56,  5.73it/s]#015Iteration:  51%|█████▏    | 706/1372 [02:03<01:56,  5.73it/s]#015Iteration:  52%|█████▏    | 707/1372 [02:04<01:56,  5.73it/s]#015Iteration:  52%|█████▏    | 708/1372 [02:04<01:55,  5.73it/s]#015Iteration:  52%|█████▏    | 709/1372 [02:04<01:55,  5.73it/s]#015Iteration:  52%|█████▏    | 710/1372 [02:04<01:55,  5.73it/s]#015Iteration:  52%|█████▏    | 711/1372 [02:04<01:55,  5.73it/s]#015Iteration:  52%|█████▏    | 712/1372 [02:04<01:55,  5.73it/s]#015Iteration:  52%|█████▏    | 713/1372 [02:05<01:54,  5.73it/s]#015Iteration:  52%|█████▏    | 714/1372 [02:05<01:54,  5.73it/s]#015Iteration:  52%|█████▏    | 715/1372 [02:05<01:54,  5.73it/s]#015Iteration:  52%|█████▏    | 716/1372 [02:05<01:54,  5.74it/s]#015Iteration:  52%|█████▏    | 717/1372 [02:05<01:54,  5.74it/s]#015Iteration:  52%|█████▏    | 718/1372 [02:05<01:53,  5.74it/s]#015Iteration:  52%|█████▏    | 719/1372 [02:06<01:53,  5.74it/s]#015Iteration:  52%|█████▏    | 720/1372 [02:06<01:53,  5.74it/s]#015Iteration:  53%|█████▎    | 721/1372 [02:06<01:53,  5.74it/s]#015Iteration:  53%|█████▎    | 722/1372 [02:06<01:53,  5.74it/s]#015Iteration:  53%|█████▎    | 723/1372 [02:06<01:53,  5.74it/s]#015Iteration:  53%|█�\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34mDLL 2020-08-23 20:56:01.688924 - Training Epoch: 0 Training Iteration: 751  step_loss : 0.8291015625  learning_rate : 1.5034113060428853e-05 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:56:10.410643 - Training Epoch: 0 Training Iteration: 801  step_loss : 1.28125  learning_rate : 1.3815789473684211e-05 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:56:19.130821 - Training Epoch: 0 Training Iteration: 851  step_loss : 1.0673828125  learning_rate : 1.259746588693957e-05 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:56:27.852123 - Training Epoch: 0 Training Iteration: 901  step_loss : 0.845703125  learning_rate : 1.1379142300194933e-05 \u001b[0m\n",
      "\u001b[34m�███▎    | 724/1372 [02:07<01:52,  5.74it/s]#015Iteration:  53%|█████▎    | 725/1372 [02:07<01:52,  5.74it/s]#015Iteration:  53%|█████▎    | 726/1372 [02:07<01:52,  5.74it/s]#015Iteration:  53%|█████▎    | 727/1372 [02:07<01:52,  5.74it/s]#015Iteration:  53%|█████▎    | 728/1372 [02:07<01:52,  5.74it/s]#015Iteration:  53%|█████▎    | 729/1372 [02:07<01:52,  5.74it/s]#015Iteration:  53%|█████▎    | 730/1372 [02:08<01:51,  5.74it/s]#015Iteration:  53%|█████▎    | 731/1372 [02:08<01:51,  5.74it/s]#015Iteration:  53%|█████▎    | 732/1372 [02:08<01:51,  5.74it/s]#015Iteration:  53%|█████▎    | 733/1372 [02:08<01:51,  5.73it/s]#015Iteration:  53%|█████▎    | 734/1372 [02:08<01:51,  5.73it/s]#015Iteration:  54%|█████▎    | 735/1372 [02:08<01:51,  5.73it/s]#015Iteration:  54%|█████▎    | 736/1372 [02:09<01:50,  5.73it/s]#015Iteration:  54%|█████▎    | 737/1372 [02:09<01:50,  5.73it/s]#015Iteration:  54%|█████▍    | 738/1372 [02:09<01:50,  5.73it/s]#015Iteration:  54%|█████▍    | 739/1372 [02:09<01:50,  5.73it/s]#015Iteration:  54%|█████▍    | 740/1372 [02:09<01:50,  5.73it/s]#015Iteration:  54%|█████▍    | 741/1372 [02:09<01:50,  5.74it/s]#015Iteration:  54%|█████▍    | 742/1372 [02:10<01:49,  5.74it/s]#015Iteration:  54%|█████▍    | 743/1372 [02:10<01:49,  5.74it/s]#015Iteration:  54%|█████▍    | 744/1372 [02:10<01:49,  5.74it/s]#015Iteration:  54%|█████▍    | 745/1372 [02:10<01:49,  5.73it/s]#015Iteration:  54%|█████▍    | 746/1372 [02:10<01:49,  5.73it/s]#015Iteration:  54%|█████▍    | 747/1372 [02:11<01:48,  5.73it/s]#015Iteration:  55%|█████▍    | 748/1372 [02:11<01:48,  5.74it/s]#015Iteration:  55%|█████▍    | 749/1372 [02:11<01:48,  5.73it/s]#015Iteration:  55%|█████▍    | 750/1372 [02:11<01:48,  5.73it/s]#015Iteration:  55%|█████▍    | 751/1372 [02:11<01:48,  5.73it/s]#015Iteration:  55%|█████▍    | 752/1372 [02:11<01:48,  5.73it/s]#015Iteration:  55%|█████▍    | 753/1372 [02:12<01:48,  5.73it/s]#015Iteration:  55%|█████▍    | 754/1372 [02:12<01:47,  5.73it/s]#015Iteration:  55%|█████▌    | 755/1372 [02:12<01:47,  5.73it/s]#015Iteration:  55%|█████▌    | 756/1372 [02:12<01:47,  5.73it/s]#015Iteration:  55%|█████▌    | 757/1372 [02:12<01:47,  5.73it/s]#015Iteration:  55%|█████▌    | 758/1372 [02:12<01:47,  5.73it/s]#015Iteration:  55%|█████▌    | 759/1372 [02:13<01:46,  5.74it/s]#015Iteration:  55%|█████▌    | 760/1372 [02:13<01:46,  5.73it/s]#015Iteration:  55%|█████▌    | 761/1372 [02:13<01:46,  5.73it/s]#015Iteration:  56%|█████▌    | 762/1372 [02:13<01:46,  5.73it/s]#015Iteration:  56%|█████▌    | 763/1372 [02:13<01:46,  5.73it/s]#015Iteration:  56%|█████▌    | 764/1372 [02:14<01:46,  5.73it/s]#015Iteration:  56%|█████▌    | 765/1372 [02:14<01:45,  5.73it/s]#015Iteration:  56%|█████▌    | 766/1372 [02:14<01:45,  5.73it/s]#015Iteration:  56%|█████▌    | 767/1372 [02:14<01:45,  5.73it/s]#015Iteration:  56%|█████▌    | 768/1372 [02:14<01:45,  5.73it/s]#015Iteration:  56%|█████▌    | 769/1372 [02:14<01:45,  5.73it/s]#015Iteration:  56%|█████▌    | 770/1372 [02:15<01:45,  5.73it/s]#015Iteration:  56%|█████▌    | 771/1372 [02:15<01:44,  5.73it/s]#015Iteration:  56%|█████▋    | 772/1372 [02:15<01:44,  5.74it/s]#015Iteration:  56%|█████▋    | 773/1372 [02:15<01:44,  5.74it/s]#015Iteration:  56%|█████▋    | 774/1372 [02:15<01:44,  5.74it/s]#015Iteration:  56%|█████▋    | 775/1372 [02:15<01:44,  5.74it/s]#015Iteration:  57%|█████▋    | 776/1372 [02:16<01:43,  5.74it/s]#015Iteration:  57%|█████▋    | 777/1372 [02:16<01:43,  5.74it/s]#015Iteration:  57%|█████▋    | 778/1372 [02:16<01:43,  5.73it/s]#015Iteration:  57%|█████▋    | 779/1372 [02:16<01:43,  5.73it/s]#015Iteration:  57%|█████▋    | 780/1372 [02:16<01:43,  5.74it/s]#015Iteration:  57%|█████▋    | 781/1372 [02:16<01:42,  5.74it/s]#015Iteration:  57%|█████▋    | 782/1372 [02:17<01:42,  5.74it/s]#015Iteration:  57%|█████▋    | 783/1372 [02:17<01:42,  5.74it/s]#015Iteration:  57%|█████▋    | 784/1372 [02:17<01:42,  5.74it/s]#015Iteration:  57%|█████▋    | 785/1372 [02:17<01:42,  5.74it/s]#015Iteration:  57%|█████▋    | 786/1372 [02:17<01:42,  5.73it/s]#015Iteration:  57%|█████▋    | 787/1372 [02:18<01:42,  5.73it/s]#015Iteration:  57%|█████▋    | 788/1372 [02:18<01:41,  5.73it/s]#015Iteration:  58%|█████▊    | 789/1372 [02:18<01:41,  5.73it/s]#015Iteration:  58%|█████▊    | 790/1372 [02:18<01:41,  5.73it/s]#015Iteration:  58%|█████▊    | 791/1372 [02:18<01:41,  5.73it/s]#015Iteration:  58%|█████▊    | 792/1372 [02:18<01:41,  5.73it/s]#015Iteration:  58%|█████▊    | 793/1372 [02:19<01:41,  5.73it/s]#015Iteration:  58%|█████▊    | 794/1372 [02:19<01:40,  5.73it/s]#015Iteration:  58%|█████▊    | 795/1372 [02:19<01:40,  5.73it/s]#015Iteration:  58%|█████▊    | 796/1372 [02:19<01:40,  5.73it/s]#015Iteration:  58%|█████▊    | 797/1372 [02:19<01:40,  5.73it/s]#015Iteration:  58%|█████▊    | 798/1372 [02:19<01:40,  5.73it/s]#015Iteration:  58%|█████▊    | 799/1372 [02:20<01:39,  5.73it/s]#015Iteration:  58%|█████▊    | 800/1372 [02:20<01:39,  5.73it/s]#015Iteration:  58%|█████▊    | 801/1372 [02:20<01:39,  5.73it/s]#015Iteration:  58%|█████▊    | 802/1372 [02:20<01:39,  5.73it/s]#015Iteration:  59%|█████▊    | 803/1372 [02:20<01:39,  5.73it/s]#015Iteration:  59%|█████▊    | 804/1372 [02:20<01:39,  5.73it/s]#015Iteration:  59%|█████▊    | 805/1372 [02:21<01:38,  5.73it/s]#015Iteration:  59%|█████▊    | 806/1372 [02:21<01:38,  5.73it/s]#015Iteration:  59%|█████▉    | 807/1372 [02:21<01:38,  5.73it/s]#015Iteration:  59%|█████▉    | 808/1372 [02:21<01:38,  5.74it/s]#015Iteration:  59%|█████▉    | 809/1372 [02:21<01:38,  5.74it/s]#015Iteration:  59%|█████▉    | 810/1372 [02:22<01:38,  5.73it/s]#015Iteration:  59%|█████▉    | 811/1372 [02:22<01:37,  5.73it/s]#015Iteration:  59%|█████▉    | 812/1372 [02:22<01:37,  5.74it/s]#015Iteration:  59%|█████▉    | 813/1372 [02:22<01:37,  5.74it/s]#015Iteration:  59%|█████▉    | 814/1372 [02:22<01:37,  5.74it/s]#015Iteration:  59%|█████▉    | 815/1372 [02:22<01:37,  5.74it/s]#015Iteration:  59%|█████▉    | 816/1372 [02:23<01:36,  5.73it/s]#015Iteration:  60%|█████▉    | 817/1372 [02:23<01:36,  5.74it/s]#015Iteration:  60%|█████▉    | 818/1372 [02:23<01:36,  5.74it/s]#015Iteration:  60%|█████▉    | 819/1372 [02:23<01:36,  5.74it/s]#015Iteration:  60%|█████▉    | 820/1372 [02:23<01:36,  5.74it/s]#015Iteration:  60%|█████▉    | 821/1372 [02:23<01:36,  5.74it/s]#015Iteration:  60%|█████▉    | 822/1372 [02:24<01:35,  5.74it/s]#015Iteration:  60%|█████▉    | 823/1372 [02:24<01:35,  5.74it/s]#015Iteration:  60%|██████    | 824/1372 [02:24<01:35,  5.74it/s]#015Iteration:  60%|██████    | 825/1372 [02:24<01:35,  5.73it/s]#015Iteration:  60%|██████    | 826/1372 [02:24<01:35,  5.73it/s]#015Iteration:  60%|██████    | 827/1372 [02:24<01:35,  5.73it/s]#015Iteration:  60%|██████    | 828/1372 [02:25<01:34,  5.74it/s]#015Iteration:  60%|██████    | 829/1372 [02:25<01:34,  5.74it/s]#015Iteration:  60%|██████    | 830/1372 [02:25<01:34,  5.73it/s]#015Iteration:  61%|██████    | 831/1372 [02:25<01:34,  5.73it/s]#015Iteration:  61%|██████    | 832/1372 [02:25<01:34,  5.73it/s]#015Iteration:  61%|██████    | 833/1372 [02:26<01:33,  5.74it/s]#015Iteration:  61%|██████    | 834/1372 [02:26<01:33,  5.74it/s]#015Iteration:  61%|██████    | 835/1372 [02:26<01:33,  5.74it/s]#015Iteration:  61%|██████    | 836/1372 [02:26<01:33,  5.73it/s]#015Iteration:  61%|██████    | 837/1372 [02:26<01:33,  5.73it/s]#015Iteration:  61%|██████    | 838/1372 [02:26<01:33,  5.73it/s]#015Iteration:  61%|██████    | 839/1372 [02:27<01:32,  5.74it/s]#015Iteration:  61%|██████    | 840/1372 [02:27<01:32,  5.74it/s]#015Iteration:  61%|██████▏   | 841/1372 [02:27<01:32,  5.74it/s]#015Iteration:  61%|██████▏   | 842/1372 [02:27<01:32,  5.74it/s]#015Iteration:  61%|██████▏   | 843/1372 [02:27<01:32,  5.74it/s]#015Iteration:  62%|██████▏   | 844/1372 [02:27<01:32,  5.73it/s]#015Iteration:  62%|██████▏   | 845/1372 [02:28<01:31,  5.73it/s]#015Iteration:  62%|██████▏   | 846/1372 [02:28<01:31,  5.73it/s]#015Iteration:  62%|██████▏   | 847/1372 [02:28<01:31,  5.73it/s]#015Iteration:  62%|██████▏   | 848/1372 [02:28<01:31,  5.73it/s]#015Iteration:  62%|██████▏   | 849/1372 [02:28<01:31,  5.73it/s]#015Iteration:  62%|██████▏   | 850/1372 [02:29<01:31,  5.73it/s]#015Iteration:  62%|██████▏   | 851/1372 [02:29<01:31,  5.72it/s]#015Iteration:  62%|██████▏   | 852/1372 [02:29<01:30,  5.73it/s]#015Iteration:  62%|██████▏   | 853/1372 [02:29<01:30,  5.73it/s]#015Iteration:  62%|██████▏   | 854/1372 [02:29<01:30,  5.73it/s]#015Iteration:  62%|██████▏   | 855/1372 [02:29<01:30,  5.73it/s]#015Iteration:  62%|██████▏   | 856/1372 [02:30<01:30,  5.73it/s]#015Iteration:  62%|██████▏   | 857/1372 [02:30<01:29,  5.73it/s]#015Iteration:  63%|██████▎   | 858/1372 [02:30<01:29,  5.73it/s]#015Iteration:  63%|██████▎   | 859/1372 [02:30<01:29,  5.73it/s]#015Iteration:  63%|██████▎   | 860/1372 [02:30<01:29,  5.73it/s]#015Iteration:  63%|██████▎   | 861/1372 [02:30<01:29,  5.73it/s]#015Iteration:  63%|██████▎   | 862/1372 [02:31<01:28,  5.73it/s]#015Iteration:  63%|██████▎   | 863/1372 [02:31<01:28,  5.73it/s]#015Iteration:  63%|██████▎   | 864/1372 [02:31<01:28,  5.74it/s]#015Iteration:  63%|██████▎   | 865/1372 [02:31<01:28,  5.73it/s]#015Iteration:  63%|██████▎   | 866/1372 [02:31<01:28,  5.73it/s]#015Iteration:  63%|██████▎   | 867/1372 [02:31<01:28,  5.73it/s]#015Iteration:  63%|██████▎   | 868/1372 [02:32<01:27,  5.73it/s]#015Iteration:  63%|██████▎   | 869/1372 [02:32<01:27,  5.73it/s]#015Iteration:  63%|██████▎   | 870/1372 [02:32<01:27,  5.74it/s]#015Iteration:  63%|██████▎   | 871/1372 [02:32<01:27,  5.74it/s]#015Iteration:  64%|██████▎   | 872/1372 [02:32<01:27,  5.73it/s]#015Iteration:  64%|██████▎   | 873/1372 [02:33<01:27,  5.73it/s]#015Iteration:  64%|██████▎   | 874/1372 [02:33<01:26,  5.73it/s]#015Iteration:  64%|██████▍   | 875/1372 [02:33<01:26,  5.73it/s]#015Iteration:  64%|██████▍   | 876/1372 [02:33<01:26,  5.74it/s]#015Iteration:  64%|██████▍   | 877/1372 [02:33<01:26,  5.74it/s]#015Iteration:  64%|██████▍   | 878/1372 [02:33<01:26,  5.74it/s]#015Iteration:  64%|██████▍   | 879/1372 [02:34<01:25,  5.73it/s]#015Iteration:  64%|██████▍   | 880/1372 [02:34<01:25,  5.73it/s]#015Iteration:  64%|██████▍   | 881/1372 [02:34<01:25,  5.73it/s]#015Iteration:  64%|██████▍   | 882/1372 [02:34<01:25,  5.74it/s]#015Iteration:  64%|██████▍   | 883/1372 [02:34<01:25,  5.74it/s]#015Iteration:  64%|██████▍   | 884/1372 [02:34<01:25,  5.73it/s]#015Iteration:  65%|██████▍   | 885/1372 [02:35<01:24,  5.73it/s]#015Iteration:  65%|██████▍   | 886/1372 [02:35<01:24,  5.73it/s]#015Iteration:  65%|██████▍   | 887/1372 [02:35<01:24,  5.73it/s]#015Iteration:  65%|██████▍   | 888/1372 [02:35<01:24,  5.73it/s]#015Iteration:  65%|██████▍   | 889/1372 [02:35<01:24,  5.73it/s]#015Iteration:  65%|██████▍   | 890/1372 [02:35<01:24,  5.73it/s]#015Iteration:  65%|██████▍   | 891/1372 [02:36<01:23,  5.73it/s]#015Iteration:  65%|██████▌   | 892/1372 [02:36<01:23,  5.73it/s]#015Iteration:  65%|██████▌   | 893/1372 [02:36<01:23,  5.73it/s]#015Iteration:  65%|██████▌   | 894/1372 [02:36<01:23,  5.73it/s]#015Iteration:  65%|██████▌   | 895/1372 [02:36<01:23,  5.74it/s]#015Iteration:  65%|██████▌   | 896/1372 [02:37<01:23,  5.73it/s]#015Iteration:  65%|██████▌   | 897/1372 [02:37<01:22,  5.73it/s]#015Iteration:  65%|██████▌   | 898/1372 [02:37<01:22,  5.73it/s]#015Iteration:  66%|██████▌   | 899/1372 [02:37<01:22,  5.73it/s]#015Iteration:  66%|██████▌   | 900/1372 [02:37<01:22,  5.73it/s]#015Iteration:  66%|██████▌   | 901/1372 [02:37<01:22,  5.73it/s]#015Iteration:  66%|██████▌   | 902/1372 [02:38<01:22,  5.73it/s]#015Iteration:  66%|██████▌   | 903/1372 [02:38<01:21,  5.73it/s]#015Iteration:  66%|██████▌   | 904/1372 [02:38<01:21,  5.73it/s]#015Iteration:  66%|██████▌   | 905/1372 [02:38<01:21,  5.73it/s]#015Iteration:  66%|██████▌   | 906/1372 [02:38<01:21,  5.73it/s]#015Iteration:  66%|██████▌   | 907/1372 [02:38<01:21,  5.73it/s]#015Iteration:  66%|██████▌   | 908/1372 [02:39<01:20,  5.73it/s]#015Iteration:  66%|██████▋   | 909/1372 [02:39<01:20,  5.73it/s]#015Iteration:  66%|██████▋   | 910/1372 [02:39<01:20,  5.73it/s]#015Iteration:  66%|██████▋   | 911/1372 [02:39<01:20,  5.73it/s]#015Iteration:  66%|██████▋   | 912/1372 [02:39<01:20,  5.73it/s]#015Iteration:  67%|██████▋   | 913/1372 [02:39<01:20,  5.73it/s]#015Iteration:  67%|██████▋   | 914/1372 [02:40<01:19,  5.73it/s]#015Iteration:  67%|██████▋   | 915/1372 [02:40<01:19,  5.73it/s]#015Iteration:  67%|██████▋   | 916/1372 [02:40<01:19,  5.73it/s]#015Iteration:  67%|██████▋   | 917/1372 [02:40<01:19,  5.73it/s]#015Iteration:  67%|██████▋   | 918/1372 [02:40<01:19,  5.74it/s]#015Iteration:  67%|██████▋   | 919/1372 [02:41<01:19,  5.73it/s]#015Iteration:  67%|██████▋   | 920/1372 [02:41<01:18,  5.73it/s]#015Iteration:  67%|██████▋   | 921/1372 [02:41<01:18,  5.73it/s]#015Iteration:  67%|██████▋   | 922/1372 [02:41<01:18,  5.73it/s]#015Iteration:  67%|██████▋   | 923/1372 [02:41<01:18,  5.73it/s]#015Iteration:  67%|██████▋   | 924/1372 [02:41<01:18,  5.73it/s]#015Iteration:  67%|██████▋   | 925/1372 [02:42<01:17,  5.73it/s]#015Iteration:  67%|██████▋   | 926/1372 [02:42<01:17,  5.73it/s]#015Iteration:  68%|██████▊   | 927/1372 [02:42<01:17,  5.73it/s]#015Iteration:  68%|██████▊   | 928/1372 [02:42<01:17,  5.73it/s]#015Iteration:  68%|██████▊   | 929/1372 [02:42<01:17,  5.73it/s]#015Iteration:  68%|██████▊   | 930/1372 [02:42<01:17,  5.73it/s]#015Iteration:  68%|██████▊   | 931/1372 [02:43<01:16,  5.73it/s]#015Iteration:  68%|██████▊   | 932/1372 [02:43<01:16,  5.73it/s]#015Iteration:  68%|██████▊   | 933/1372 [02:43<01:16,  5.73it/s]#015Iteration:  68%|██████▊   | 934/1372 [02:43<01:16,  5.73it/s]#015Iteration:  68%|██████▊   | 935/1372 [02:43<01:16,  5.73it/s]#015Iteration:  68%|██████▊   | 936/1372 [02:44<01:16,  5.73it/s]#015Iteration:  68%|██████▊   | 937/1372 [02:44<01:15,  5.73it/s]#015Iteration:  68%|██████▊   | 938/1372 [02:44<01:15,  5.73it/s]#015Iteration:  68%|██████▊   | 939/1372 [02:44<01:15,  5.73it/s]#015Iteration:  69%|██████▊   | 940/1372 [02:44<01:15,  5.74it/s]#015Iteration:  69%|██████▊   | 941/1372 [02:44<01:15,  5.74it/s]#015Iteration:  69%|██████▊   | 942/1372 [02:45<01:14,  5.73i\u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:56:36.573969 - Training Epoch: 0 Training Iteration: 951  step_loss : 1.076171875  learning_rate : 1.0160818713450294e-05 \u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34mDLL 2020-08-23 20:56:45.295516 - Training Epoch: 0 Training Iteration: 1001  step_loss : 1.4111328125  learning_rate : 8.942495126705652e-06 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:56:54.015710 - Training Epoch: 0 Training Iteration: 1051  step_loss : 0.8193359375  learning_rate : 7.724171539961016e-06 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:57:02.741450 - Training Epoch: 0 Training Iteration: 1101  step_loss : 1.345703125  learning_rate : 6.505847953216375e-06 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:57:11.467380 - Training Epoch: 0 Training Iteration: 1151  step_loss : 1.14453125  learning_rate : 5.287524366471733e-06 \u001b[0m\n",
      "\u001b[34mt/s]#015Iteration:  69%|██████▊   | 943/1372 [02:45<01:14,  5.73it/s]#015Iteration:  69%|██████▉   | 944/1372 [02:45<01:14,  5.73it/s]#015Iteration:  69%|██████▉   | 945/1372 [02:45<01:14,  5.73it/s]#015Iteration:  69%|██████▉   | 946/1372 [02:45<01:14,  5.74it/s]#015Iteration:  69%|██████▉   | 947/1372 [02:45<01:14,  5.73it/s]#015Iteration:  69%|██████▉   | 948/1372 [02:46<01:13,  5.73it/s]#015Iteration:  69%|██████▉   | 949/1372 [02:46<01:13,  5.73it/s]#015Iteration:  69%|██████▉   | 950/1372 [02:46<01:13,  5.73it/s]#015Iteration:  69%|██████▉   | 951/1372 [02:46<01:13,  5.73it/s]#015Iteration:  69%|██████▉   | 952/1372 [02:46<01:13,  5.73it/s]#015Iteration:  69%|██████▉   | 953/1372 [02:46<01:13,  5.73it/s]#015Iteration:  70%|██████▉   | 954/1372 [02:47<01:12,  5.74it/s]#015Iteration:  70%|██████▉   | 955/1372 [02:47<01:12,  5.74it/s]#015Iteration:  70%|██████▉   | 956/1372 [02:47<01:12,  5.74it/s]#015Iteration:  70%|██████▉   | 957/1372 [02:47<01:12,  5.74it/s]#015Iteration:  70%|██████▉   | 958/1372 [02:47<01:12,  5.74it/s]#015Iteration:  70%|██████▉   | 959/1372 [02:48<01:12,  5.73it/s]#015Iteration:  70%|██████▉   | 960/1372 [02:48<01:11,  5.74it/s]#015Iteration:  70%|███████   | 961/1372 [02:48<01:11,  5.73it/s]#015Iteration:  70%|███████   | 962/1372 [02:48<01:11,  5.74it/s]#015Iteration:  70%|███████   | 963/1372 [02:48<01:11,  5.74it/s]#015Iteration:  70%|███████   | 964/1372 [02:48<01:11,  5.74it/s]#015Iteration:  70%|███████   | 965/1372 [02:49<01:10,  5.73it/s]#015Iteration:  70%|███████   | 966/1372 [02:49<01:10,  5.73it/s]#015Iteration:  70%|███████   | 967/1372 [02:49<01:10,  5.73it/s]#015Iteration:  71%|███████   | 968/1372 [02:49<01:10,  5.73it/s]#015Iteration:  71%|███████   | 969/1372 [02:49<01:10,  5.73it/s]#015Iteration:  71%|███████   | 970/1372 [02:49<01:10,  5.73it/s]#015Iteration:  71%|███████   | 971/1372 [02:50<01:09,  5.73it/s]#015Iteration:  71%|███████   | 972/1372 [02:50<01:09,  5.73it/s]#015Iteration:  71%|███████   | 973/1372 [02:50<01:09,  5.73it/s]#015Iteration:  71%|███████   | 974/1372 [02:50<01:09,  5.73it/s]#015Iteration:  71%|███████   | 975/1372 [02:50<01:09,  5.73it/s]#015Iteration:  71%|███████   | 976/1372 [02:50<01:09,  5.73it/s]#015Iteration:  71%|███████   | 977/1372 [02:51<01:08,  5.73it/s]#015Iteration:  71%|███████▏  | 978/1372 [02:51<01:08,  5.73it/s]#015Iteration:  71%|███████▏  | 979/1372 [02:51<01:08,  5.74it/s]#015Iteration:  71%|███████▏  | 980/1372 [02:51<01:08,  5.74it/s]#015Iteration:  72%|███████▏  | 981/1372 [02:51<01:08,  5.73it/s]#015Iteration:  72%|███████▏  | 982/1372 [02:52<01:08,  5.73it/s]#015Iteration:  72%|███████▏  | 983/1372 [02:52<01:07,  5.73it/s]#015Iteration:  72%|███████▏  | 984/1372 [02:52<01:07,  5.74it/s]#015Iteration:  72%|███████▏  | 985/1372 [02:52<01:07,  5.73it/s]#015Iteration:  72%|███████▏  | 986/1372 [02:52<01:07,  5.73it/s]#015Iteration:  72%|███████▏  | 987/1372 [02:52<01:07,  5.73it/s]#015Iteration:  72%|███████▏  | 988/1372 [02:53<01:06,  5.73it/s]#015Iteration:  72%|███████▏  | 989/1372 [02:53<01:06,  5.73it/s]#015Iteration:  72%|███████▏  | 990/1372 [02:53<01:06,  5.73it/s]#015Iteration:  72%|███████▏  | 991/1372 [02:53<01:06,  5.73it/s]#015Iteration:  72%|███████▏  | 992/1372 [02:53<01:06,  5.73it/s]#015Iteration:  72%|███████▏  | 993/1372 [02:53<01:06,  5.73it/s]#015Iteration:  72%|███████▏  | 994/1372 [02:54<01:05,  5.73it/s]#015Iteration:  73%|███████▎  | 995/1372 [02:54<01:05,  5.73it/s]#015Iteration:  73%|███████▎  | 996/1372 [02:54<01:05,  5.73it/s]#015Iteration:  73%|███████▎  | 997/1372 [02:54<01:05,  5.73it/s]#015Iteration:  73%|███████▎  | 998/1372 [02:54<01:05,  5.73it/s]#015Iteration:  73%|███████▎  | 999/1372 [02:54<01:05,  5.73it/s]#015Iteration:  73%|███████▎  | 1000/1372 [02:55<01:04,  5.73it/s]#015Iteration:  73%|███████▎  | 1001/1372 [02:55<01:04,  5.73it/s]#015Iteration:  73%|███████▎  | 1002/1372 [02:55<01:04,  5.73it/s]#015Iteration:  73%|███████▎  | 1003/1372 [02:55<01:04,  5.73it/s]#015Iteration:  73%|███████▎  | 1004/1372 [02:55<01:04,  5.73it/s]#015Iteration:  73%|███████▎  | 1005/1372 [02:56<01:03,  5.74it/s]#015Iteration:  73%|███████▎  | 1006/1372 [02:56<01:03,  5.73it/s]#015Iteration:  73%|███████▎  | 1007/1372 [02:56<01:03,  5.73it/s]#015Iteration:  73%|███████▎  | 1008/1372 [02:56<01:03,  5.73it/s]#015Iteration:  74%|███████▎  | 1009/1372 [02:56<01:03,  5.73it/s]#015Iteration:  74%|███████▎  | 1010/1372 [02:56<01:03,  5.73it/s]#015Iteration:  74%|███████▎  | 1011/1372 [02:57<01:02,  5.73it/s]#015Iteration:  74%|███████▍  | 1012/1372 [02:57<01:02,  5.73it/s]#015Iteration:  74%|███████▍  | 1013/1372 [02:57<01:02,  5.74it/s]#015Iteration:  74%|███████▍  | 1014/1372 [02:57<01:02,  5.73it/s]#015Iteration:  74%|███████▍  | 1015/1372 [02:57<01:02,  5.73it/s]#015Iteration:  74%|███████▍  | 1016/1372 [02:57<01:02,  5.73it/s]#015Iteration:  74%|███████▍  | 1017/1372 [02:58<01:01,  5.73it/s]#015Iteration:  74%|███████▍  | 1018/1372 [02:58<01:01,  5.73it/s]#015Iteration:  74%|███████▍  | 1019/1372 [02:58<01:01,  5.73it/s]#015Iteration:  74%|███████▍  | 1020/1372 [02:58<01:01,  5.73it/s]#015Iteration:  74%|███████▍  | 1021/1372 [02:58<01:01,  5.73it/s]#015Iteration:  74%|███████▍  | 1022/1372 [02:59<01:01,  5.73it/s]#015Iteration:  75%|███████▍  | 1023/1372 [02:59<01:00,  5.73it/s]#015Iteration:  75%|███████▍  | 1024/1372 [02:59<01:00,  5.73it/s]#015Iteration:  75%|███████▍  | 1025/1372 [02:59<01:00,  5.73it/s]#015Iteration:  75%|███████▍  | 1026/1372 [02:59<01:00,  5.73it/s]#015Iteration:  75%|███████▍  | 1027/1372 [02:59<01:00,  5.74it/s]#015Iteration:  75%|███████▍  | 1028/1372 [03:00<00:59,  5.74it/s]#015Iteration:  75%|███████▌  | 1029/1372 [03:00<00:59,  5.74it/s]#015Iteration:  75%|███████▌  | 1030/1372 [03:00<00:59,  5.73it/s]#015Iteration:  75%|███████▌  | 1031/1372 [03:00<00:59,  5.73it/s]#015Iteration:  75%|███████▌  | 1032/1372 [03:00<00:59,  5.73it/s]#015Iteration:  75%|███████▌  | 1033/1372 [03:00<00:59,  5.73it/s]#015Iteration:  75%|███████▌  | 1034/1372 [03:01<00:58,  5.73it/s]#015Iteration:  75%|███████▌  | 1035/1372 [03:01<00:58,  5.73it/s]#015Iteration:  76%|███████▌  | 1036/1372 [03:01<00:58,  5.74it/s]#015Iteration:  76%|███████▌  | 1037/1372 [03:01<00:58,  5.74it/s]#015Iteration:  76%|███████▌  | 1038/1372 [03:01<00:58,  5.73it/s]#015Iteration:  76%|███████▌  | 1039/1372 [03:01<00:58,  5.73it/s]#015Iteration:  76%|███████▌  | 1040/1372 [03:02<00:57,  5.74it/s]#015Iteration:  76%|███████▌  | 1041/1372 [03:02<00:57,  5.73it/s]#015Iteration:  76%|███████▌  | 1042/1372 [03:02<00:57,  5.73it/s]#015Iteration:  76%|███████▌  | 1043/1372 [03:02<00:57,  5.73it/s]#015Iteration:  76%|███████▌  | 1044/1372 [03:02<00:57,  5.73it/s]#015Iteration:  76%|███████▌  | 1045/1372 [03:03<00:57,  5.73it/s]#015Iteration:  76%|███████▌  | 1046/1372 [03:03<00:56,  5.73it/s]#015Iteration:  76%|███████▋  | 1047/1372 [03:03<00:56,  5.73it/s]#015Iteration:  76%|███████▋  | 1048/1372 [03:03<00:56,  5.73it/s]#015Iteration:  76%|███████▋  | 1049/1372 [03:03<00:56,  5.74it/s]#015Iteration:  77%|███████▋  | 1050/1372 [03:03<00:56,  5.74it/s]#015Iteration:  77%|███████▋  | 1051/1372 [03:04<00:55,  5.74it/s]#015Iteration:  77%|███████▋  | 1052/1372 [03:04<00:55,  5.73it/s]#015Iteration:  77%|███████▋  | 1053/1372 [03:04<00:55,  5.73it/s]#015Iteration:  77%|███████▋  | 1054/1372 [03:04<00:55,  5.73it/s]#015Iteration:  77%|███████▋  | 1055/1372 [03:04<00:55,  5.73it/s]#015Iteration:  77%|███████▋  | 1056/1372 [03:04<00:55,  5.73it/s]#015Iteration:  77%|███████▋  | 1057/1372 [03:05<00:54,  5.73it/s]#015Iteration:  77%|███████▋  | 1058/1372 [03:05<00:54,  5.73it/s]#015Iteration:  77%|███████▋  | 1059/1372 [03:05<00:54,  5.73it/s]#015Iteration:  77%|███████▋  | 1060/1372 [03:05<00:54,  5.73it/s]#015Iteration:  77%|███████▋  | 1061/1372 [03:05<00:54,  5.73it/s]#015Iteration:  77%|███████▋  | 1062/1372 [03:05<00:54,  5.73it/s]#015Iteration:  77%|███████▋  | 1063/1372 [03:06<00:53,  5.73it/s]#015Iteration:  78%|███████▊  | 1064/1372 [03:06<00:53,  5.73it/s]#015Iteration:  78%|███████▊  | 1065/1372 [03:06<00:53,  5.73it/s]#015Iteration:  78%|███████▊  | 1066/1372 [03:06<00:53,  5.73it/s]#015Iteration:  78%|███████▊  | 1067/1372 [03:06<00:53,  5.73it/s]#015Iteration:  78%|███████▊  | 1068/1372 [03:07<00:53,  5.73it/s]#015Iteration:  78%|███████▊  | 1069/1372 [03:07<00:52,  5.73it/s]#015Iteration:  78%|███████▊  | 1070/1372 [03:07<00:52,  5.73it/s]#015Iteration:  78%|███████▊  | 1071/1372 [03:07<00:52,  5.73it/s]#015Iteration:  78%|███████▊  | 1072/1372 [03:07<00:52,  5.73it/s]#015Iteration:  78%|███████▊  | 1073/1372 [03:07<00:52,  5.73it/s]#015Iteration:  78%|███████▊  | 1074/1372 [03:08<00:51,  5.73it/s]#015Iteration:  78%|███████▊  | 1075/1372 [03:08<00:51,  5.73it/s]#015Iteration:  78%|███████▊  | 1076/1372 [03:08<00:51,  5.73it/s]#015Iteration:  78%|███████▊  | 1077/1372 [03:08<00:51,  5.73it/s]#015Iteration:  79%|███████▊  | 1078/1372 [03:08<00:51,  5.73it/s]#015Iteration:  79%|███████▊  | 1079/1372 [03:08<00:51,  5.73it/s]#015Iteration:  79%|███████▊  | 1080/1372 [03:09<00:50,  5.73it/s]#015Iteration:  79%|███████▉  | 1081/1372 [03:09<00:50,  5.74it/s]#015Iteration:  79%|███████▉  | 1082/1372 [03:09<00:50,  5.73it/s]#015Iteration:  79%|███████▉  | 1083/1372 [03:09<00:50,  5.74it/s]#015Iteration:  79%|███████▉  | 1084/1372 [03:09<00:50,  5.74it/s]#015Iteration:  79%|███████▉  | 1085/1372 [03:09<00:50,  5.74it/s]#015Iteration:  79%|███████▉  | 1086/1372 [03:10<00:49,  5.74it/s]#015Iteration:  79%|███████▉  | 1087/1372 [03:10<00:49,  5.74it/s]#015Iteration:  79%|███████▉  | 1088/1372 [03:10<00:49,  5.73it/s]#015Iteration:  79%|███████▉  | 1089/1372 [03:10<00:49,  5.73it/s]#015Iteration:  79%|███████▉  | 1090/1372 [03:10<00:49,  5.74it/s]#015Iteration:  80%|███████▉  | 1091/1372 [03:11<00:48,  5.74it/s]#015Iteration:  80%|███████▉  | 1092/1372 [03:11<00:48,  5.73it/s]#015Iteration:  80%|███████▉  | 1093/1372 [03:11<00:48,  5.73it/s]#015Iteration:  80%|███████▉  | 1094/1372 [03:11<00:48,  5.72it/s]#015Iteration:  80%|███████▉  | 1095/1372 [03:11<00:48,  5.72it/s]#015Iteration:  80%|███████▉  | 1096/1372 [03:11<00:48,  5.72it/s]#015Iteration:  80%|███████▉  | 1097/1372 [03:12<00:48,  5.72it/s]#015Iteration:  80%|████████  | 1098/1372 [03:12<00:47,  5.72it/s]#015Iteration:  80%|████████  | 1099/1372 [03:12<00:47,  5.72it/s]#015Iteration:  80%|████████  | 1100/1372 [03:12<00:47,  5.72it/s]#015Iteration:  80%|████████  | 1101/1372 [03:12<00:47,  5.72it/s]#015Iteration:  80%|████████  | 1102/1372 [03:12<00:47,  5.72it/s]#015Iteration:  80%|████████  | 1103/1372 [03:13<00:46,  5.73it/s]#015Iteration:  80%|████████  | 1104/1372 [03:13<00:46,  5.72it/s]#015Iteration:  81%|████████  | 1105/1372 [03:13<00:46,  5.73it/s]#015Iteration:  81%|████████  | 1106/1372 [03:13<00:46,  5.73it/s]#015Iteration:  81%|████████  | 1107/1372 [03:13<00:46,  5.73it/s]#015Iteration:  81%|████████  | 1108/1372 [03:14<00:46,  5.73it/s]#015Iteration:  81%|████████  | 1109/1372 [03:14<00:45,  5.73it/s]#015Iteration:  81%|████████  | 1110/1372 [03:14<00:45,  5.72it/s]#015Iteration:  81%|████████  | 1111/1372 [03:14<00:45,  5.73it/s]#015Iteration:  81%|████████  | 1112/1372 [03:14<00:45,  5.73it/s]#015Iteration:  81%|████████  | 1113/1372 [03:14<00:45,  5.73it/s]#015Iteration:  81%|████████  | 1114/1372 [03:15<00:45,  5.73it/s]#015Iteration:  81%|████████▏ | 1115/1372 [03:15<00:44,  5.73it/s]#015Iteration:  81%|████████▏ | 1116/1372 [03:15<00:44,  5.73it/s]#015Iteration:  81%|████████▏ | 1117/1372 [03:15<00:44,  5.73it/s]#015Iteration:  81%|████████▏ | 1118/1372 [03:15<00:44,  5.73it/s]#015Iteration:  82%|████████▏ | 1119/1372 [03:15<00:44,  5.73it/s]#015Iteration:  82%|████████▏ | 1120/1372 [03:16<00:44,  5.72it/s]#015Iteration:  82%|████████▏ | 1121/1372 [03:16<00:43,  5.73it/s]#015Iteration:  82%|████████▏ | 1122/1372 [03:16<00:43,  5.73it/s]#015Iteration:  82%|████████▏ | 1123/1372 [03:16<00:43,  5.73it/s]#015Iteration:  82%|████████▏ | 1124/1372 [03:16<00:43,  5.73it/s]#015Iteration:  82%|████████▏ | 1125/1372 [03:16<00:43,  5.73it/s]#015Iteration:  82%|████████▏ | 1126/1372 [03:17<00:42,  5.73it/s]#015Iteration:  82%|████████▏ | 1127/1372 [03:17<00:42,  5.73it/s]#015Iteration:  82%|████████▏ | 1128/1372 [03:17<00:42,  5.73it/s]#015Iteration:  82%|████████▏ | 1129/1372 [03:17<00:42,  5.73it/s]#015Iteration:  82%|████████▏ | 1130/1372 [03:17<00:42,  5.73it/s]#015Iteration:  82%|████████▏ | 1131/1372 [03:18<00:42,  5.73it/s]#015Iteration:  83%|████████▎ | 1132/1372 [03:18<00:41,  5.73it/s]#015Iteration:  83%|████████▎ | 1133/1372 [03:18<00:41,  5.73it/s]#015Iteration:  83%|████████▎ | 1134/1372 [03:18<00:41,  5.73it/s]#015Iteration:  83%|████████▎ | 1135/1372 [03:18<00:41,  5.73it/s]#015Iteration:  83%|████████▎ | 1136/1372 [03:18<00:41,  5.73it/s]#015Iteration:  83%|████████▎ | 1137/1372 [03:19<00:40,  5.73it/s]#015Iteration:  83%|████████▎ | 1138/1372 [03:19<00:40,  5.73it/s]#015Iteration:  83%|████████▎ | 1139/1372 [03:19<00:40,  5.73it/s]#015Iteration:  83%|████████▎ | 1140/1372 [03:19<00:40,  5.73it/s]#015Iteration:  83%|████████▎ | 1141/1372 [03:19<00:40,  5.73it/s]#015Iteration:  83%|████████▎ | 1142/1372 [03:19<00:40,  5.73it/s]#015Iteration:  83%|████████▎ | 1143/1372 [03:20<00:39,  5.74it/s]#015Iteration:  83%|████████▎ | 1144/1372 [03:20<00:39,  5.73it/s]#015Iteration:  83%|████████▎ | 1145/1372 [03:20<00:39,  5.73it/s]#015Iteration:  84%|████████▎ | 1146/1372 [03:20<00:39,  5.73it/s]#015Iteration:  84%|████████▎ | 1147/1372 [03:20<00:39,  5.73it/s]#015Iteration:  84%|████████▎ | 1148/1372 [03:20<00:39,  5.73it/s]#015Iteration:  84%|████████▎ | 1149/1372 [03:21<00:38,  5.73it/s]#015Iteration:  84%|████████▍ | 1150/1372 [03:21<00:38,  5.72it/s]#015It\u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:57:20.197378 - Training Epoch: 0 Training Iteration: 1201  step_loss : 0.84716796875  learning_rate : 4.069200779727097e-06 \u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34mDLL 2020-08-23 20:57:28.925476 - Training Epoch: 0 Training Iteration: 1251  step_loss : 1.166015625  learning_rate : 2.8508771929824557e-06 \u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:57:37.655220 - Training Epoch: 0 Training Iteration: 1301  step_loss : 0.8642578125  learning_rate : 1.632553606237815e-06 \u001b[0m\n",
      "\u001b[34meration:  84%|████████▍ | 1151/1372 [03:21<00:38,  5.73it/s]#015Iteration:  84%|████████▍ | 1152/1372 [03:21<00:38,  5.73it/s]#015Iteration:  84%|████████▍ | 1153/1372 [03:21<00:38,  5.73it/s]#015Iteration:  84%|████████▍ | 1154/1372 [03:22<00:38,  5.72it/s]#015Iteration:  84%|████████▍ | 1155/1372 [03:22<00:37,  5.73it/s]#015Iteration:  84%|████████▍ | 1156/1372 [03:22<00:37,  5.73it/s]#015Iteration:  84%|████████▍ | 1157/1372 [03:22<00:37,  5.73it/s]#015Iteration:  84%|████████▍ | 1158/1372 [03:22<00:37,  5.73it/s]#015Iteration:  84%|████████▍ | 1159/1372 [03:22<00:37,  5.73it/s]#015Iteration:  85%|████████▍ | 1160/1372 [03:23<00:37,  5.73it/s]#015Iteration:  85%|████████▍ | 1161/1372 [03:23<00:36,  5.73it/s]#015Iteration:  85%|████████▍ | 1162/1372 [03:23<00:36,  5.73it/s]#015Iteration:  85%|████████▍ | 1163/1372 [03:23<00:36,  5.73it/s]#015Iteration:  85%|████████▍ | 1164/1372 [03:23<00:36,  5.73it/s]#015Iteration:  85%|████████▍ | 1165/1372 [03:23<00:36,  5.73it/s]#015Iteration:  85%|████████▍ | 1166/1372 [03:24<00:35,  5.73it/s]#015Iteration:  85%|████████▌ | 1167/1372 [03:24<00:35,  5.73it/s]#015Iteration:  85%|████████▌ | 1168/1372 [03:24<00:35,  5.73it/s]#015Iteration:  85%|████████▌ | 1169/1372 [03:24<00:35,  5.73it/s]#015Iteration:  85%|████████▌ | 1170/1372 [03:24<00:35,  5.73it/s]#015Iteration:  85%|████████▌ | 1171/1372 [03:25<00:35,  5.73it/s]#015Iteration:  85%|████████▌ | 1172/1372 [03:25<00:34,  5.73it/s]#015Iteration:  85%|████████▌ | 1173/1372 [03:25<00:34,  5.73it/s]#015Iteration:  86%|████████▌ | 1174/1372 [03:25<00:34,  5.73it/s]#015Iteration:  86%|████████▌ | 1175/1372 [03:25<00:34,  5.73it/s]#015Iteration:  86%|████████▌ | 1176/1372 [03:25<00:34,  5.73it/s]#015Iteration:  86%|████████▌ | 1177/1372 [03:26<00:34,  5.72it/s]#015Iteration:  86%|████████▌ | 1178/1372 [03:26<00:33,  5.73it/s]#015Iteration:  86%|████████▌ | 1179/1372 [03:26<00:33,  5.73it/s]#015Iteration:  86%|████████▌ | 1180/1372 [03:26<00:33,  5.73it/s]#015Iteration:  86%|████████▌ | 1181/1372 [03:26<00:33,  5.73it/s]#015Iteration:  86%|████████▌ | 1182/1372 [03:26<00:33,  5.73it/s]#015Iteration:  86%|████████▌ | 1183/1372 [03:27<00:32,  5.73it/s]#015Iteration:  86%|████████▋ | 1184/1372 [03:27<00:32,  5.73it/s]#015Iteration:  86%|████████▋ | 1185/1372 [03:27<00:32,  5.72it/s]#015Iteration:  86%|████████▋ | 1186/1372 [03:27<00:32,  5.73it/s]#015Iteration:  87%|████████▋ | 1187/1372 [03:27<00:32,  5.73it/s]#015Iteration:  87%|████████▋ | 1188/1372 [03:27<00:32,  5.73it/s]#015Iteration:  87%|████████▋ | 1189/1372 [03:28<00:31,  5.72it/s]#015Iteration:  87%|████████▋ | 1190/1372 [03:28<00:31,  5.72it/s]#015Iteration:  87%|████████▋ | 1191/1372 [03:28<00:31,  5.72it/s]#015Iteration:  87%|████████▋ | 1192/1372 [03:28<00:31,  5.72it/s]#015Iteration:  87%|████████▋ | 1193/1372 [03:28<00:31,  5.72it/s]#015Iteration:  87%|████████▋ | 1194/1372 [03:29<00:31,  5.72it/s]#015Iteration:  87%|████████▋ | 1195/1372 [03:29<00:30,  5.72it/s]#015Iteration:  87%|████████▋ | 1196/1372 [03:29<00:30,  5.72it/s]#015Iteration:  87%|████████▋ | 1197/1372 [03:29<00:30,  5.72it/s]#015Iteration:  87%|████████▋ | 1198/1372 [03:29<00:30,  5.72it/s]#015Iteration:  87%|████████▋ | 1199/1372 [03:29<00:30,  5.73it/s]#015Iteration:  87%|████████▋ | 1200/1372 [03:30<00:30,  5.73it/s]#015Iteration:  88%|████████▊ | 1201/1372 [03:30<00:29,  5.73it/s]#015Iteration:  88%|████████▊ | 1202/1372 [03:30<00:29,  5.72it/s]#015Iteration:  88%|████████▊ | 1203/1372 [03:30<00:29,  5.72it/s]#015Iteration:  88%|████████▊ | 1204/1372 [03:30<00:29,  5.72it/s]#015Iteration:  88%|████████▊ | 1205/1372 [03:30<00:29,  5.73it/s]#015Iteration:  88%|████████▊ | 1206/1372 [03:31<00:28,  5.73it/s]#015Iteration:  88%|████████▊ | 1207/1372 [03:31<00:28,  5.73it/s]#015Iteration:  88%|████████▊ | 1208/1372 [03:31<00:28,  5.73it/s]#015Iteration:  88%|████████▊ | 1209/1372 [03:31<00:28,  5.73it/s]#015Iteration:  88%|████████▊ | 1210/1372 [03:31<00:28,  5.73it/s]#015Iteration:  88%|████████▊ | 1211/1372 [03:31<00:28,  5.73it/s]#015Iteration:  88%|████████▊ | 1212/1372 [03:32<00:27,  5.73it/s]#015Iteration:  88%|████████▊ | 1213/1372 [03:32<00:27,  5.73it/s]#015Iteration:  88%|████████▊ | 1214/1372 [03:32<00:27,  5.73it/s]#015Iteration:  89%|████████▊ | 1215/1372 [03:32<00:27,  5.73it/s]#015Iteration:  89%|████████▊ | 1216/1372 [03:32<00:27,  5.73it/s]#015Iteration:  89%|████████▊ | 1217/1372 [03:33<00:27,  5.73it/s]#015Iteration:  89%|████████▉ | 1218/1372 [03:33<00:26,  5.73it/s]#015Iteration:  89%|████████▉ | 1219/1372 [03:33<00:26,  5.73it/s]#015Iteration:  89%|████████▉ | 1220/1372 [03:33<00:26,  5.73it/s]#015Iteration:  89%|████████▉ | 1221/1372 [03:33<00:26,  5.73it/s]#015Iteration:  89%|████████▉ | 1222/1372 [03:33<00:26,  5.73it/s]#015Iteration:  89%|████████▉ | 1223/1372 [03:34<00:26,  5.72it/s]#015Iteration:  89%|████████▉ | 1224/1372 [03:34<00:25,  5.73it/s]#015Iteration:  89%|████████▉ | 1225/1372 [03:34<00:25,  5.73it/s]#015Iteration:  89%|████████▉ | 1226/1372 [03:34<00:25,  5.73it/s]#015Iteration:  89%|████████▉ | 1227/1372 [03:34<00:25,  5.73it/s]#015Iteration:  90%|████████▉ | 1228/1372 [03:34<00:25,  5.73it/s]#015Iteration:  90%|████████▉ | 1229/1372 [03:35<00:24,  5.73it/s]#015Iteration:  90%|████████▉ | 1230/1372 [03:35<00:24,  5.73it/s]#015Iteration:  90%|████████▉ | 1231/1372 [03:35<00:24,  5.73it/s]#015Iteration:  90%|████████▉ | 1232/1372 [03:35<00:24,  5.73it/s]#015Iteration:  90%|████████▉ | 1233/1372 [03:35<00:24,  5.73it/s]#015Iteration:  90%|████████▉ | 1234/1372 [03:36<00:24,  5.73it/s]#015Iteration:  90%|█████████ | 1235/1372 [03:36<00:23,  5.73it/s]#015Iteration:  90%|█████████ | 1236/1372 [03:36<00:23,  5.73it/s]#015Iteration:  90%|█████████ | 1237/1372 [03:36<00:23,  5.73it/s]#015Iteration:  90%|█████████ | 1238/1372 [03:36<00:23,  5.73it/s]#015Iteration:  90%|█████████ | 1239/1372 [03:36<00:23,  5.73it/s]#015Iteration:  90%|█████████ | 1240/1372 [03:37<00:23,  5.73it/s]#015Iteration:  90%|█████████ | 1241/1372 [03:37<00:22,  5.73it/s]#015Iteration:  91%|█████████ | 1242/1372 [03:37<00:22,  5.73it/s]#015Iteration:  91%|█████████ | 1243/1372 [03:37<00:22,  5.73it/s]#015Iteration:  91%|█████████ | 1244/1372 [03:37<00:22,  5.73it/s]#015Iteration:  91%|█████████ | 1245/1372 [03:37<00:22,  5.73it/s]#015Iteration:  91%|█████████ | 1246/1372 [03:38<00:21,  5.73it/s]#015Iteration:  91%|█████████ | 1247/1372 [03:38<00:21,  5.73it/s]#015Iteration:  91%|█████████ | 1248/1372 [03:38<00:21,  5.73it/s]#015Iteration:  91%|█████████ | 1249/1372 [03:38<00:21,  5.73it/s]#015Iteration:  91%|█████████ | 1250/1372 [03:38<00:21,  5.73it/s]#015Iteration:  91%|█████████ | 1251/1372 [03:38<00:21,  5.73it/s]#015Iteration:  91%|█████████▏| 1252/1372 [03:39<00:20,  5.73it/s]#015Iteration:  91%|█████████▏| 1253/1372 [03:39<00:20,  5.73it/s]#015Iteration:  91%|█████████▏| 1254/1372 [03:39<00:20,  5.72it/s]#015Iteration:  91%|█████████▏| 1255/1372 [03:39<00:20,  5.72it/s]#015Iteration:  92%|█████████▏| 1256/1372 [03:39<00:20,  5.72it/s]#015Iteration:  92%|█████████▏| 1257/1372 [03:40<00:20,  5.73it/s]#015Iteration:  92%|█████████▏| 1258/1372 [03:40<00:19,  5.73it/s]#015Iteration:  92%|█████████▏| 1259/1372 [03:40<00:19,  5.73it/s]#015Iteration:  92%|█████████▏| 1260/1372 [03:40<00:19,  5.72it/s]#015Iteration:  92%|█████████▏| 1261/1372 [03:40<00:19,  5.72it/s]#015Iteration:  92%|█████████▏| 1262/1372 [03:40<00:19,  5.72it/s]#015Iteration:  92%|█████████▏| 1263/1372 [03:41<00:19,  5.73it/s]#015Iteration:  92%|█████████▏| 1264/1372 [03:41<00:18,  5.73it/s]#015Iteration:  92%|█████████▏| 1265/1372 [03:41<00:18,  5.73it/s]#015Iteration:  92%|█████████▏| 1266/1372 [03:41<00:18,  5.73it/s]#015Iteration:  92%|█████████▏| 1267/1372 [03:41<00:18,  5.73it/s]#015Iteration:  92%|█████████▏| 1268/1372 [03:41<00:18,  5.73it/s]#015Iteration:  92%|█████████▏| 1269/1372 [03:42<00:17,  5.73it/s]#015Iteration:  93%|█████████▎| 1270/1372 [03:42<00:17,  5.73it/s]#015Iteration:  93%|█████████▎| 1271/1372 [03:42<00:17,  5.73it/s]#015Iteration:  93%|█████████▎| 1272/1372 [03:42<00:17,  5.73it/s]#015Iteration:  93%|█████████▎| 1273/1372 [03:42<00:17,  5.73it/s]#015Iteration:  93%|█████████▎| 1274/1372 [03:42<00:17,  5.73it/s]#015Iteration:  93%|█████████▎| 1275/1372 [03:43<00:16,  5.73it/s]#015Iteration:  93%|█████████▎| 1276/1372 [03:43<00:16,  5.73it/s]#015Iteration:  93%|█████████▎| 1277/1372 [03:43<00:16,  5.73it/s]#015Iteration:  93%|█████████▎| 1278/1372 [03:43<00:16,  5.73it/s]#015Iteration:  93%|█████████▎| 1279/1372 [03:43<00:16,  5.73it/s]#015Iteration:  93%|█████████▎| 1280/1372 [03:44<00:16,  5.73it/s]#015Iteration:  93%|█████████▎| 1281/1372 [03:44<00:15,  5.73it/s]#015Iteration:  93%|█████████▎| 1282/1372 [03:44<00:15,  5.73it/s]#015Iteration:  94%|█████████▎| 1283/1372 [03:44<00:15,  5.73it/s]#015Iteration:  94%|█████████▎| 1284/1372 [03:44<00:15,  5.73it/s]#015Iteration:  94%|█████████▎| 1285/1372 [03:44<00:15,  5.73it/s]#015Iteration:  94%|█████████▎| 1286/1372 [03:45<00:15,  5.73it/s]#015Iteration:  94%|█████████▍| 1287/1372 [03:45<00:14,  5.73it/s]#015Iteration:  94%|█████████▍| 1288/1372 [03:45<00:14,  5.73it/s]#015Iteration:  94%|█████████▍| 1289/1372 [03:45<00:14,  5.73it/s]#015Iteration:  94%|█████████▍| 1290/1372 [03:45<00:14,  5.73it/s]#015Iteration:  94%|█████████▍| 1291/1372 [03:45<00:14,  5.73it/s]#015Iteration:  94%|█████████▍| 1292/1372 [03:46<00:13,  5.73it/s]#015Iteration:  94%|█████████▍| 1293/1372 [03:46<00:13,  5.73it/s]#015Iteration:  94%|█████████▍| 1294/1372 [03:46<00:13,  5.73it/s]#015Iteration:  94%|█████████▍| 1295/1372 [03:46<00:13,  5.73it/s]#015Iteration:  94%|█████████▍| 1296/1372 [03:46<00:13,  5.73it/s]#015Iteration:  95%|█████████▍| 1297/1372 [03:47<00:13,  5.73it/s]#015Iteration:  95%|█████████▍| 1298/1372 [03:47<00:12,  5.73it/s]#015Iteration:  95%|█████████▍| 1299/1372 [03:47<00:12,  5.73it/s]#015Iteration:  95%|█████████▍| 1300/1372 [03:47<00:12,  5.73it/s]#015Iteration:  95%|█████████▍| 1301/1372 [03:47<00:12,  5.73it/s]#015Iteration:  95%|█████████▍| 1302/1372 [03:47<00:12,  5.72it/s]#015Iteration:  95%|█████████▍| 1303/1372 [03:48<00:12,  5.72it/s]#015Iteration:  95%|█████████▌| 1304/1372 [03:48<00:11,  5.72it/s]#015Iteration:  95%|█████████▌| 1305/1372 [03:48<00:11,  5.73it/s]#015Iteration:  95%|█████████▌| 1306/1372 [03:48<00:11,  5.73it/s]#015Iteration:  95%|█████████▌| 1307/1372 [03:48<00:11,  5.73it/s]#015Iteration:  95%|█████████▌| 1308/1372 [03:48<00:11,  5.73it/s]#015Iteration:  95%|█████████▌| 1309/1372 [03:49<00:10,  5.73it/s]#015Iteration:  95%|█████████▌| 1310/1372 [03:49<00:10,  5.73it/s]#015Iteration:  96%|█████████▌| 1311/1372 [03:49<00:10,  5.73it/s]#015Iteration:  96%|█████████▌| 1312/1372 [03:49<00:10,  5.73it/s]#015Iteration:  96%|█████████▌| 1313/1372 [03:49<00:10,  5.73it/s]#015Iteration:  96%|█████████▌| 1314/1372 [03:49<00:10,  5.73it/s]#015Iteration:  96%|█████████▌| 1315/1372 [03:50<00:09,  5.73it/s]#015Iteration:  96%|█████████▌| 1316/1372 [03:50<00:09,  5.73it/s]#015Iteration:  96%|█████████▌| 1317/1372 [03:50<00:09,  5.73it/s]#015Iteration:  96%|█████████▌| 1318/1372 [03:50<00:09,  5.72it/s]#015Iteration:  96%|█████████▌| 1319/1372 [03:50<00:09,  5.72it/s]#015Iteration:  96%|█████████▌| 1320/1372 [03:51<00:09,  5.72it/s]#015Iteration:  96%|█████████▋| 1321/1372 [03:51<00:08,  5.72it/s]#015Iteration:  96%|█████████▋| 1322/1372 [03:51<00:08,  5.72it/s]#015Iteration:  96%|█████████▋| 1323/1372 [03:51<00:08,  5.73it/s]#015Iteration:  97%|█████████▋| 1324/1372 [03:51<00:08,  5.73it/s]#015Iteration:  97%|█████████▋| 1325/1372 [03:51<00:08,  5.73it/s]#015Iteration:  97%|█████████▋| 1326/1372 [03:52<00:08,  5.73it/s]#015Iteration:  97%|█████████▋| 1327/1372 [03:52<00:07,  5.73it/s]#015Iteration:  97%|█████████▋| 1328/1372 [03:52<00:07,  5.73it/s]#015Iteration:  97%|█████████▋| 1329/1372 [03:52<00:07,  5.73it/s]#015Iteration:  97%|█████████▋| 1330/1372 [03:52<00:07,  5.73it/s]#015Iteration:  97%|█████████▋| 1331/1372 [03:52<00:07,  5.73it/s]#015Iteration:  97%|█████████▋| 1332/1372 [03:53<00:06,  5.73it/s]#015Iteration:  97%|█████████▋| 1333/1372 [03:53<00:06,  5.73it/s]#015Iteration:  97%|█████████▋| 1334/1372 [03:53<00:06,  5.73it/s]#015Iteration:  97%|█████████▋| 1335/1372 [03:53<00:06,  5.73it/s]#015Iteration:  97%|█████████▋| 1336/1372 [03:53<00:06,  5.73it/s]#015Iteration:  97%|█████████▋| 1337/1372 [03:53<00:06,  5.73it/s]#015Iteration:  98%|█████████▊| 1338/1372 [03:54<00:05,  5.73it/s]#015Iteration:  98%|█████████▊| 1339/1372 [03:54<00:05,  5.73it/s]#015Iteration:  98%|█████████▊| 1340/1372 [03:54<00:05,  5.73it/s]#015Iteration:  98%|█████████▊| 1341/1372 [03:54<00:05,  5.73it/s]#015Iteration:  98%|█████████▊| 1342/1372 [03:54<00:05,  5.73it/s]#015Iteration:  98%|█████████▊| 1343/1372 [03:55<00:05,  5.73it/s]#015Iteration:  98%|█████████▊| 1344/1372 [03:55<00:04,  5.73it/s]#015Iteration:  98%|█████████▊| 1345/1372 [03:55<00:04,  5.73it/s]#015Iteration:  98%|█████████▊| 1346/1372 [03:55<00:04,  5.73it/s]#015Iteration:  98%|█████████▊| 1347/1372 [03:55<00:04,  5.73it/s]#015Iteration:  98%|█████████▊| 1348/1372 [03:55<00:04,  5.73it/s]#015Iteration:  98%|█████████▊| 1349/1372 [03:56<00:04,  5.73it/s]#015Iteration:  98%|█████████▊| 1350/1372 [03:56<00:03,\u001b[0m\n",
      "\u001b[34mDLL 2020-08-23 20:57:46.384450 - Training Epoch: 0 Training Iteration: 1351  step_loss : 0.7880859375  learning_rate : 4.1423001949317804e-07 \u001b[0m\n",
      "\u001b[34m  5.73it/s]#015Iteration:  98%|█████████▊| 1351/1372 [03:56<00:03,  5.73it/s]#015Iteration:  99%|█████████▊| 1352/1372 [03:56<00:03,  5.73it/s]#015Iteration:  99%|█████████▊| 1353/1372 [03:56<00:03,  5.73it/s]#015Iteration:  99%|█████████▊| 1354/1372 [03:56<00:03,  5.73it/s]#015Iteration:  99%|█████████▉| 1355/1372 [03:57<00:02,  5.73it/s]#015Iteration:  99%|█████████▉| 1356/1372 [03:57<00:02,  5.73it/s]#015Iteration:  99%|█████████▉| 1357/1372 [03:57<00:02,  5.73it/s]#015Iteration:  99%|█████████▉| 1358/1372 [03:57<00:02,  5.73it/s]#015Iteration:  99%|█████████▉| 1359/1372 [03:57<00:02,  5.73it/s]#015Iteration:  99%|█████████▉| 1360/1372 [03:58<00:02,  5.73it/s]#015Iteration:  99%|█████████▉| 1361/1372 [03:58<00:01,  5.73it/s]#015Iteration:  99%|█████████▉| 1362/1372 [03:58<00:01,  5.73it/s]#015Iteration:  99%|█████████▉| 1363/1372 [03:58<00:01,  5.73it/s]#015Iteration:  99%|█████████▉| 1364/1372 [03:58<00:01,  5.73it/s]#015Iteration:  99%|█████████▉| 1365/1372 [03:58<00:01,  5.73it/s]#015Iteration: 100%|█████████▉| 1366/1372 [03:59<00:01,  5.72it/s]#015Iteration: 100%|█████████▉| 1367/1372 [03:59<00:00,  5.72it/s]#015Iteration: 100%|█████████▉| 1368/1372 [03:59<00:00,  5.72it/s]#015Iteration: 100%|█████████▉| 1369/1372 [03:59<00:00,  5.72it/s]#015Iteration: 100%|█████████▉| 1370/1372 [03:59<00:00,  5.72it/s]#015Iteration: 100%|█████████▉| 1371/1372 [03:59<00:00,  5.73it/s]#015Iteration: 100%|██████████| 1372/1372 [04:00<00:00,  5.59it/s]\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "2020-08-23 21:00:55 Uploading - Uploading generated training model\u001b[34mDLL 2020-08-23 21:00:51.943415 -  e2e_train_time : 240.11868405342102  training_sequences_per_second : 365.4359524162561  final_loss : 0.125 \u001b[0m\n",
      "\u001b[34m*****************************************\u001b[0m\n",
      "\u001b[34mSetting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. \u001b[0m\n",
      "\u001b[34m*****************************************\u001b[0m\n",
      "\u001b[34mstart squad_download.sh downloading ... ...\u001b[0m\n",
      "\u001b[34m{'sagemaker_program': '\"transform_script.py\"', 'seed': '1', 'num_gpus': '4', 'bert_model': '\"bert-base-uncased\"', 'sagemaker_region': '\"us-east-1\"', 'vocab_file': '\"/workspace/bert/data/bert_vocab.txt\"', 'sagemaker_submit_directory': '\"s3://sagemaker-us-east-1-383827541835/bert-train-kdd-2020-08-23-20-41-15-988/source/sourcedir.tar.gz\"', 'train_batch_size': '16', 'num_train_epochs': '1', 'sagemaker_container_log_level': '20', 'train_file': '\"/workspace/bert/data/squad/v1.1/train-v1.1.json\"', 'sagemaker_enable_cloudwatch_metrics': 'false', 'doc_stride': '128', 'output_dir': '\"/opt/ml/model\"', 'max_seq_length': '512', 'sagemaker_job_name': '\"bert-train-kdd-2020-08-23-20-41-15-988\"', 'save_to_s3': '\"sagemaker-us-east-1-383827541835\"', 'config_file': '\"/workspace/bert/bert_config.json\"', 'learning_rate': '3e-05'}\u001b[0m\n",
      "\u001b[34mtrain_cmd:  ['/opt/conda/bin/python', '-m', 'torch.distributed.launch', '--nproc_per_node=4', 'run_squad.py', '--init_checkpoint', '/workspace/bert/bert_base.pt', '--do_train', '--do_lower_case', '--fp16', '--seed', '1', '--bert_model', '\"bert-base-uncased\"', '--vocab_file', '\"/workspace/bert/data/bert_vocab.txt\"', '--train_batch_size', '16', '--num_train_epochs', '1', '--train_file', '\"/workspace/bert/data/squad/v1.1/train-v1.1.json\"', '--doc_stride', '128', '--output_dir', '\"/opt/ml/model\"', '--max_seq_length', '512', '--save_to_s3', '\"sagemaker-us-east-1-383827541835\"', '--config_file', '\"/workspace/bert/bert_config.json\"', '--learning_rate', '3e-05']\u001b[0m\n",
      "\u001b[34mTraining complete.\u001b[0m\n",
      "\n",
      "2020-08-23 21:05:08 Completed - Training job completed\n",
      "Training seconds: 1297\n",
      "Billable seconds: 1297\n"
     ]
    }
   ],
   "source": [
    "training_start = time.time()\n",
    "torch_model.fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time for training : 1467.5661361217499 s\n"
     ]
    }
   ],
   "source": [
    "training_end = time.time()\n",
    "print(\"Time for training : {} s\".format(training_end - training_start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## IV. Deploy our trained model\n",
    "\n",
    "After finetuning the BERT base model, we are ready to deploy the trained model to an Sagemaker endpoint and test it with some question answering tast data.\n",
    "\n",
    "Let's first deploy the model to the inference instance `'ml.g4dn.4xlarge'`. Since the deploy needs to launch new instance and upload the model, it will take about 10-12 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.predictor import RealTimePredictor, json_serializer, json_deserializer\n",
    "\n",
    "class JSONPredictor(RealTimePredictor):\n",
    "    def __init__(self, endpoint_name, sagemaker_session):\n",
    "        super(JSONPredictor, self).__init__(endpoint_name, sagemaker_session, json_serializer, json_deserializer)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Parameter image will be renamed to image_uri in SageMaker Python SDK v2.\n",
      "'create_image_uri' will be deprecated in favor of 'ImageURIProvider' class in SageMaker Python SDK v2.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------------!Time for deploying 'bert-kdd' : 635.0710117816925\n"
     ]
    }
   ],
   "source": [
    "deploy_start = time.time()\n",
    "\n",
    "## endpoint name must satisfy regular expression pattern: ^[a-zA-Z0-9](-*[a-zA-Z0-9])*\n",
    "endpoint_name = 'bert-kdd' \n",
    "        \n",
    "        \n",
    "torch_model = PyTorchModel(model_data=f's3://{bucket}/model.tar.gz',\n",
    "                           role=role,\n",
    "                           entry_point='transform_script.py',\n",
    "                           framework_version='1.4.0',\n",
    "                           predictor_cls=JSONPredictor)\n",
    "\n",
    "bert_end = torch_model.deploy(instance_type='ml.g4dn.4xlarge', \n",
    "                              initial_instance_count=1, \n",
    "                              endpoint_name=endpoint_name)\n",
    "\n",
    "print(\"Time for deploying '{}' : {}\".format(endpoint_name, time.time() - deploy_start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the deployment, you should be able to see your endpoint [here](https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/endpoints)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that our endpoint has been deployed, let's send it some questions and see how it will responde us?! For the NLP QA setting, we need to packed the model both \"question\" and \"context\" (to answer the question) in a Json file as below:\n",
    "\n",
    "```{`json}\n",
    "{\n",
    "  \"context\": \"Rachel is a girl who really loves her tiger, Kimi. Kimi is a large tiger with a very furry belly. He gets very excited by the prospect of eating chicken covered in gravy.\",\n",
    "\n",
    "  \"question\": \"who loves Kimi?\"\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "context = \"Rachel is a girl who really loves her tiger, Kimi. Kimi is a large tiger with a very furry belly. \\\n",
    "        He gets very excited by the prospect of eating chicken covered in gravy.\"\n",
    "question = \"who loves Kimi?\"\n",
    "pass_in_data = {\"context\": context, \"question\": question}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 13.1 ms, sys: 8 µs, total: 13.1 ms\n",
      "Wall time: 5.78 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Rachel'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "response = bert_end.predict(pass_in_data, initial_args={'ContentType':'application/json'}) \n",
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !rm bert_base.pt\n",
    "# !rm s3_bucket.txt\n",
    "bert_end.delete_endpoint()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_p36",
   "language": "python",
   "name": "conda_pytorch_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}